Variable sweepback wing aircraft attitude control technology based on reinforcement learning and switching sliding mode strategy in complex environment

Abstract

The invention discloses a variable sweepback wing aircraft attitude control technology based on reinforcement learning and a self-adaptive switching sliding mode strategy in a complex environment. The method mainly comprises the following steps: firstly, modeling a variable sweepback wing aircraft with a global working mode into a nonlinear switching system form; secondly, designing a reinforcement learning controller, solving a control law for maximizing a strategy objective function gradient by using Actor, and approaching an action value function by using Critic; finally, a nonlinear switching disturbance observer is used for estimating composite disturbance composed of discontinuous disturbance and parameter uncertainty, a dead zone right inverse function is introduced to enable composite nonlinearity composed of a dead zone and saturation nonlinearity to be equivalent to a new saturation link, and an anti-saturation auxiliary system is constructed for processing. And a self-adaptive switching sliding mode controller is provided on the basis of backstepping. The method provided by the invention has effectiveness and superiority.

Description

[0001] The invention relates to a variable-sweep-wing aircraft attitude control technology based on reinforcement learning and switching sliding mode strategy in a complex environment, and belongs to the technical field of aircraft flight control. Background Art

[0002] Since variable-sweep wing aircraft have the characteristics of fast time-varying, strong coupling and strong nonlinearity, changes in its sweep angle will cause changes in structural parameters and aerodynamic parameters, and it is very difficult to establish an accurate global mathematical model. Therefore, a single model can be used within a certain range to describe the flight motion of the system, and then multiple models can be used to approximate the global motion characteristics of the entire system, that is, a multi-model switching system is used to describe the entire large envelope flight process of the variable-sweep wing aircraft.

[0003] The flight envelope of variable-sweep wing aircraft is large and the flight environment is complex. In actual flight, it is inevitable to be affected by unknown interference and uncertainty of modeling parameters, which brings great challenges to the control problem during flight. However, there are few studies on the flight control of variable-sweep wing aircraft in existing research, and few consider the problem of discontinuous external interference caused by changes in wing structure, and few consider the adverse effects of the existence of dead zone and saturation compound nonlinearity of the actuator on the control system; not only that, most of the research on sliding mode control is to design a global unified sliding mode surface for the system. For a complex nonlinear switching system with multiple working modes such as variable-sweep wing aircraft, each subsystem shares the same sliding mode surface, which has certain limitations, that is, it is impossible to design the corresponding sliding mode surface according to the needs of each subsystem. In addition, in order to simplify the design, the existing research only considers its deformation between two working modes, while the present invention will consider its switching between three modes. The control rudder surface will have one more dimension than the former, resulting in a more complex model and increased control difficulty. Therefore, it is necessary to design a new control technology to realize attitude tracking control. Summary of the invention

[0004] The purpose of the present invention is to design a new composite control strategy based on the combination of adaptive switching sliding mode and reinforcement learning to achieve better attitude tracking control effect for the attitude tracking control problem of variable-sweep wing aircraft with discontinuous interference, uncertainty, actuator dead zone and saturation nonlinear characteristics.

[0005] 1. In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] Step 1: Based on the theory of nonlinear switching systems, establish an attitude control model for full-envelope flight of a variable-sweep wing aircraft, and analyze its switching characteristics between subsonic, supersonic, and hypersonic speeds;

[0007] Step 2: Design a disturbance observer for complex environments with discontinuous disturbances, uncertainties, actuator dead zones, and saturation compound nonlinear characteristics, and construct an anti-saturation auxiliary system to address the adverse effects they bring;

[0008] Step 3: Design a composite controller, which includes two parts: one is to design an adaptive switching sliding mode attitude controller based on backstepping based on the content of step 2;

[0009] Step 4: Composite Controller The second step is to design a reinforcement learning controller and use it together with the controller in step 3 to realize attitude control on the variable-sweep wing aircraft.

[0010] Furthermore, the specific operations of step 1 include the following:

[0011] Step 1-1) Establish the force and moment model of the variable-sweep wing aircraft and decompose the aerodynamic force into resistance D σ(t) , lateral force Y σ(t) and lift L σ(t) , the specific expression is as follows:

[0012]

[0013] Aerodynamic moment decomposed into rolling moment Pitch moment and yaw moment The specific expression is as follows:

[0014]

[0015] in, is the dynamic pressure, S σ(t) is the wing reference area, b σ(t) is the wingspan length, c σ(t) is the average aerodynamic chord length, σ(t):[0,+∞)→S={1,2,3} is the flight mode switching signal, X cg is the distance between the reference moment center and the center of mass of the aircraft, α is the angle of attack, β is the sideslip angle, p is the roll angular velocity, q is the pitch angular velocity, r is the yaw angular velocity, V is the flight speed, δ c is the horizontal canard rudder deflection angle, δ e is the left aileron rudder deflection angle, δ a is the right aileron rudder deflection angle, δ r is the rudder deflection angle, is the drag coefficient, is the lift coefficient, is the lateral force coefficient, is the rolling moment coefficient, is the pitching moment coefficient, is the yaw moment coefficient;

[0016] Step 1-2) Establish a nonlinear mathematical model of the variable-sweep wing aircraft, select six physical quantities α, β, μ, p, q, and r to describe the flight motion process, and establish the attitude state equation as follows:

[0017]

[0018] Wherein, μ is the roll angle, M is the mass of the variable-sweep wing aircraft, g is the gravitational acceleration, T x ,T y ,T z are the components of thrust T in the x, y, and z axes respectively, γ is the track angle, are the moments of inertia of the three axes, l T ,m T ,n T are the components of thrust torque in three directions;

[0019] Step 1-3) Establish the attitude affine nonlinear switching system model of the variable-sweep wing aircraft, considering the attitude angle Ω = [α, β, μ] T and attitude angular velocity ω=[p,q,r] T According to the time scale separation principle, the above equation can be rewritten as:

[0020]

[0021] in, represents the control torque vector, l c ,m c ,n c are the control moments in the roll, pitch and yaw directions respectively, is the control allocation matrix, δ=[δ c ,δ e ,δ a ,δ r ] T To control the deflection angle of the rudder surface, is the control gain matrix, and the specific expression is:

[0022]

[0023] in, and represents the basic lift coefficient and the basic side force coefficient; They represent the moment of inertia of the three axes, They represent the control moments excluding those generated by aerodynamic control surfaces;

[0024] Step 1-4) Analyze the characteristics of the variable-sweep wing aircraft switching between subsonic, supersonic, and hypersonic modes. The control of the variable-sweep wing aircraft is mainly achieved by aerodynamic control surfaces and thrust vectors. Among them, the aerodynamic control surfaces mainly include horizontal canards, left and right elevator rudders, and the rudder installed on the vertical tail. The corresponding control surface deflection angles are δ c ,δ e ,δ a ,δ r Indicates. There are three main working modes of variable-sweep wing aircraft. The sweep angles and aerodynamic control surfaces are different in different modes. In the subsonic stage, the wing sweep angle is 40°, the NSV horizontal canard is fully deployed, and the aerodynamic control surfaces are realized by the horizontal canard, left and right elevator rudders and rudders; in the supersonic stage, the wing sweep angle increases to 60°, the horizontal canard is completely retracted into the fuselage, and the flight is not controlled. The aerodynamic control surfaces are completed by the left and right elevator rudders and rudders; in the hypersonic stage, the wing sweep angle reaches 75°, the horizontal canard is still retracted, and the aerodynamic control surfaces are also realized by the left and right elevator rudders and rudders. This multi-working mode deformation method ensures that the NSV exhibits excellent performance in each flight stage and meets the flight requirements of multiple tasks and large envelopes.

[0025] Furthermore, the specific operations of step 2 include the following:

[0026] Step 2-1) Consider the attitude nonlinear switching system that is subject to compound disturbances and has compound nonlinear characteristics in the actuator:

[0027]

[0028] in, is the state vector of the system, representing the attitude angle and attitude angular velocity respectively; are all sufficiently smooth functions of the system state; d1 σ(t) ,d2 σ(t) are the unknown time-varying composite disturbances of the attitude loop and angular velocity loop, and there is a constant satisfy where d1 σ(t) =Δf1 σ(t) +Δd1 σ(t) , d2 σ(t) =Δf2 σ(t) +Δd2 σ(t) , is the uncertainty of the system, is an external unknown bounded disturbance term; SatDead(M c ) indicates that the control input has both dead zone and saturation nonlinear characteristics, and its form is as follows:

[0029] The actuator saturation nonlinearity is:

[0030]

[0031] Among them, M ci is the ith input of the actuator, M cmini is the minimum value that the actuator’s i-th input can reach, M cmaxi is the maximum value that the i-th input of the actuator can reach. Due to the physical limitations of the actuator, the range of the deflection angle of each rudder surface is -30°≤δ e ,δ a ,δ r ≤30°.

[0032] The actuator dead zone nonlinearity is:

[0033]

[0034] Among them, b li <0 and b ri >0 represents the left and right boundaries of the dead zone, m li >0 and m ri >0 represents the left and right slopes of the dead zone respectively.

[0035] Step 2-2) Analyze the actuator composite nonlinearity. Since the composite nonlinearity is relatively complex and difficult to handle directly, the right inverse function of the dead zone is introduced It satisfies D b D + b =I, defined as:

[0036]

[0037] The above dead zone right inverse function D + b (·) is placed before the entire actuator compound nonlinear link to obtain a new cascade nonlinear link, such as Figure 1 shown.

[0038] The mathematical expression of the cascade nonlinear link after the introduction of the dead zone right inverse function can be summarized as:

[0039]

[0040] This means that after the dead zone right inverse function is introduced before the system's composite nonlinear link, the composite nonlinear link can be equivalently regarded as a new saturation link, and then the traditional limited control method can be used to deal with the actuator composite nonlinear problem. The processed attitude nonlinear switching system model can be rewritten as:

[0041]

[0042] Step 2-3) Design a disturbance observer to estimate the composite disturbance consisting of discontinuous disturbance and uncertainty.

[0043] The attitude loop disturbance observer is:

[0044]

[0045] The angular velocity loop disturbance observer is:

[0046]

[0047] Where, s1=[s 11 ,s 12 ,s 13 ] T , s2=[s 21 ,s 22 ,s 23 ] T are the auxiliary sliding surfaces of the attitude loop and angular velocity loop respectively; z d1 =[z d11 ,z d12 ,z d13 ] T , z d2 =[z d21 ,z d22 ,z d23 ] T They are the internal states of the attitude loop and angular velocity loop interference observers respectively;

[0048] are the interference estimation values ​​of the attitude loop and angular velocity loop respectively; They are the gain matrices to be designed for the attitude loop and angular velocity loop, respectively, and satisfy

[0049] Furthermore, the specific operations of step 3 include the following:

[0050] Step 3-1) This study designs an adaptive switching sliding mode controller based on disturbance observer and anti-saturation auxiliary system to generate δ e ,δ a ,δ r , and design a reinforcement learning controller to generate the horizontal canard rudder δ c The two are simultaneously input into the variable sweep wing aircraft model to jointly control the flight attitude. The overall structure of the control system is shown in the figure Figure 2 As shown. First, an adaptive switching sliding mode controller is designed. Figure 3 The structure diagram of the attitude control system of a variable-sweep-wing aircraft based on the backstepping adaptive switching sliding mode method with anti-saturation auxiliary system and disturbance observer is given.

[0051] Step 3-2) Construct the following anti-saturation auxiliary system to deal with the actuator complex nonlinear problem:

[0052]

[0053] Among them, ΔM c =sat new (M c )-M c ,ρ1∈R 3 ,ρ2∈R 3 is the state vector of the auxiliary system, c1>0, c2>0 are the matrix parameters to be designed.

[0054] Combined with the vector of the anti-saturation auxiliary system, the auxiliary variables are formed in the following form:

[0055]

[0056] Among them, z1∈R 3 ,z2∈R 3 is an auxiliary variable, x d is the attitude angle reference signal, The virtual control rate to be designed later is used to replace the traditional error vector with the auxiliary variable to dynamically compensate the controller, which can reduce or offset the influence of the actuator's compound nonlinearity.

[0057] Step 3-3) Design of an adaptive switching sliding mode controller for the attitude angle loop based on backstepping.

[0058] To ensure the stability of the switching system, the designed switching sliding surface must satisfy that all sub-sliding surfaces intersect at the same equilibrium point, that is, satisfy S1 σ(t) (0) = S2 σ(t) (0) = 0. Sliding surface S1 with synchronous switching of modes σ(t) Designed for S1 σ(t) =P1 σ(t) z1=[S 11 σ(t) ,S 12 σ(t) ,S 13 σ(t) ] T ,in, is a positive definite diagonal matrix to be designed, satisfying (P1 σ(t) g1) -1 Existence and

[0059] In order to achieve smooth switching control between different subsystems, the following transition sliding surface is designed to construct a common coordinate transformation: S cm =C cmz1. Among them, C cm =diag{C cm1 ,C cm2 ,C cm3} is the positive definite diagonal matrix to be designed, and

[0060] In order to ensure that the system state can approach and eventually reach the predetermined sliding surface, and considering that the sign function in the traditional sliding mode reaching law is prone to cause chattering, this study changes the sign function sign to the hyperbolic tangent function tanh, and introduces adaptive dynamic compensation to enhance the robustness of the control system. Therefore, the improved sliding mode reaching law is designed as follows:

[0061]

[0062] Among them, k1 σ(t) >0 is the parameter to be designed, a>0 is a small constant used to adjust the smoothness, As follows

[0063] The adaptive law of the formula:

[0064]

[0065] in, δ1 σ(t) >0 is the parameter to be designed.

[0066] Virtual control rate for mode σ(t)=k Designed for:

[0067]

[0068] Step 3-4) Design of adaptive switching sliding mode controller for angular velocity loop based on backstepping.

[0069] Sliding surface S2 with synchronous mode switching σ(t) Designed for: S2 σ(t) =P2 σ(t) z2=[S 21 σ(t) ,S 22 σ(t) ,S 23 σ(t) ] T .in, is a positive definite diagonal matrix to be designed, satisfying (P2 σ(t) g2 σ(t) ) -1 Existence and

[0070] Similar to step 3-3), in order to reduce the chattering caused by the sign function, the hyperbolic tangent function is used to replace the sign function, and an adaptive term is introduced to improve the robustness of the control system. Therefore, the improved sliding mode reaching law is designed as follows:

[0071]

[0072] Among them, k2 σ(t) >0 is the parameter to be designed, b>0 is a small constant used to adjust the smoothness, The adaptive law is as follows:

[0073]

[0074] in, δ2 σ(t) >0 is the parameter to be designed.

[0075] Adaptive multi-mode switching sliding mode control law for mode σ(t)=k Designed for:

[0076]

[0077] Furthermore, the specific operations of step 4 include the following:

[0078] Step 4-1) Design the state space of the reinforcement learning controller.

[0079] To use reinforcement learning for attitude control of variable-sweep wing aircraft, we first need to select appropriate state variables, which can make the system model conform to the Markov property and fully describe the flight state of the system. The goal of variable-sweep wing aircraft control is to make the attitude angle track the target value Ω c =[α c ,β c ,μ c ] T , the posture tracking error is ΔΩ=[Δα,Δβ,Δμ] T , where Δα=α c -α,Δβ=β c -β,Δμ=μ c -μ. The state variables are designed as follows:

[0080] s t =[α,β,μ,p,q,r] T (twenty two)

[0081] Step 4-2) Design the action space of the reinforcement learning controller.

[0082] Considering that when the variable-sweep wing aircraft operates in subsonic mode, the horizontal canard rudder of the fuselage will be deployed and have a control effect on the system, while in hypersonic and hypersonic modes, the horizontal canard will be retracted into the fuselage, that is, the subsonic mode will have one more control rudder than other modes. Therefore, the design action space is:

[0083] A μ (s t )=δ c (twenty three)

[0084] Since the deep deterministic policy gradient algorithm uses a deterministic strategy when generating actions, it does not have the ability to explore. In order to balance the exploration and utilization mechanisms and avoid falling into the local optimal solution, this paper superimposes a Gaussian noise n on the actions output by the Actor network. t ~N(0,σ 2 ), the final exploration strategy is:

[0085] a=A μ (s t )+n t (twenty four)

[0086] In the formula, a is the actual action, A μ (s t ) is the neural network parameter μ in state s t The actions obtained according to the deterministic policy.

[0087] Step 4-3) Design the reward function of the reinforcement learning controller.

[0088] For the attitude tracking control problem of variable-sweep wing aircraft, the control objective is to minimize the attitude angle tracking error in the shortest time, that is, to minimize the sum of the attitude angle errors at each time step. The sum of attitude errors is defined as:

[0089] e 2 (t) = (α c -α) 2 +(β c -β) 2 +(μ c -μ) 2 The reward function is designed as a combination of the following two parts: R = w1r1 + w2r2, where w1 and w2 are the weights of each part, which are designed to be 0.75 and 0.25 respectively, and:

[0090]

[0091] Among them, ε is the upper bound of the tolerable error.

[0092] Step 4-4) Design the neural network structure of the reinforcement learning controller.

[0093] A fully connected multi-hidden layer deep neural network structure is used. The specific design is as follows: the Actor network has a total of 5 layers, of which the input layer is a 6-dimensional state vector, the output layer is a 1-dimensional action vector, and there are 3 hidden layers in the middle. The number of hidden layer neurons is 32, 64, and 128 respectively. The first 4 layers use the Relu activation function, and the output layer uses the Tanh activation function; the Critic network has a total of 5 layers, of which the input layer is a 7-dimensional vector connected by a 6-dimensional state vector and a 1-dimensional action vector, and the output layer is a 1-dimensional action value function estimate. The number of intermediate layers and neurons is the same as the Actor network. The first 4 layers use the Relu activation function, and the output layer is a linear function. The network structure diagram is as follows: Figure 4 shown.

[0094] Compared with the prior art, the present invention is characterized in that the variable-sweep wing aircraft model studied is more complex, the control surfaces are more complex, and the attitude tracking control under the influence of multiple adverse factors is considered. In addition, the proposed new composite control strategy consists of an adaptive switching sliding mode controller and a reinforcement learning controller. The former estimates the composite disturbance by introducing a disturbance observer, and introduces a dead zone right inverse function to convert the composite nonlinearity into a new saturated nonlinearity, and constructs an anti-saturation auxiliary system to handle it.

[0095] The beneficial effects of the present invention are:

[0096] 1. The designed disturbance observer can quickly and accurately estimate the complex

[0097] The proposed method of introducing the dead zone right inverse function and constructing an anti-saturation auxiliary system can effectively deal with the adverse effects brought by composite nonlinearity.

[0098] 2. Compared with traditional control methods, the designed composite control strategy has higher tracking accuracy and faster response speed, and its control performance is superior. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 This is the structural diagram of the cascaded nonlinear links that introduce the dead zone right inverse function.

[0100] Figure 2 This is the overall structure diagram of the control system.

[0101] Figure 3 This is the structure diagram of the attitude control of a variable-sweep wing aircraft based on the backstepping adaptive switching sliding mode method.

[0102] Figure 4 A schematic diagram of the neural network structure.

[0103] Figure 5This is a graph of the wing sweep angle and switching signal of a variable-sweep wing aircraft.

[0104] Figure 6 This is the composite interference estimation curve of the attitude loop and angular velocity loop.

[0105] Figure 7 The comparison diagram of the attitude angle, attitude angle tracking error, and attitude angular velocity response curves of the variable swept-wing aircraft under the influence of different factors using the composite control strategy based on disturbance observer and anti-saturation auxiliary system proposed in this paper.

[0106] Figure 8 This is a comparison chart of the attitude angle, attitude tracking error, and attitude angular velocity response curves of a variable-sweep wing aircraft using the composite controller based on disturbance observer and anti-saturation auxiliary system proposed in this paper and the adaptive switching sliding mode control method alone under the existence of dead zone and saturation composite nonlinearity.

[0107] Fig. 9 The rudder deflection curve generated for the compound control strategy.

[0108] Fig.10 Reward function, action value function estimation, Critic network and Actor network weight curve graph for the reinforcement learning controller. DETAILED DESCRIPTION

[0109] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0110] See also Figure 1-10 The working mode of the variable-sweep wing aircraft of this embodiment switches between subsonic, supersonic and hypersonic speeds, that is, the wing sweep angle switches between 40° and 75°. Three modes are used to represent its flight characteristics: when the switching signal σ(t)=1, the sweep angle changes between 67°<Λ≤75°; when the switching signal σ(t)=2, the sweep angle changes between 50°<Λ≤67°; when the switching signal σ(t)=3, the sweep angle changes between 40°≤Λ≤50°. The variable-sweep wing aircraft wing sweep angle and switching signal curve diagram is shown in the figure Figure 5 shown.

[0111] The force and moment model of the variable-sweep wing aircraft is established. The aerodynamic force can be decomposed into the drag force D σ(t) , lateral force Y σ(t) and lift L σ(t) , the specific expression is as follows:

[0112]

[0113] The aerodynamic moment can also be decomposed into rolling moment, pitching moment and yaw moment. The specific expressions are as follows:

[0114]

[0115] in, is the dynamic pressure, S σ(t) is the wing reference area, b σ(t) is the wingspan length, c σ(t) is the average aerodynamic chord length, σ(t):[0,+∞)→S={1,2,3} is the flight mode switching signal, X cg is the distance between the reference moment center and the center of mass of the aircraft, α is the angle of attack, β is the sideslip angle, p is the roll angular velocity, q is the pitch angular velocity, r is the yaw angular velocity, V is the flight speed, δ c is the horizontal canard rudder deflection angle, δ e is the left aileron rudder deflection angle, δ a is the right aileron rudder deflection angle, δ r is the rudder deflection angle, is the drag coefficient, is the lift coefficient, is the lateral force coefficient, is the rolling moment coefficient, is the pitching moment coefficient, is the yaw moment coefficient;

[0116] The attitude affine nonlinear switching system model of variable-sweep wing aircraft is established. Considering the attitude angle Ω=[α,β,μ] T and attitude angular velocity ω=[p,q,r] T , according to the time scale separation principle:

[0117]

[0118] in, represents the control torque vector, l c ,m c ,n c are the control moments in the roll, pitch and yaw directions respectively, is the control allocation matrix, δ=[δ c ,δ e ,δ a ,δ r ] T To control the deflection angle of the rudder surface, it can be derived from the control torque vector and the control allocation matrix; is the control gain matrix, and the specific expression is:

[0119]

[0120] in, and represents the basic lift coefficient and basic side force coefficient under the switching signal σ(t); They represent the moment of inertia of the three axes under the switching signal σ(t), They respectively represent the control torque generated by the aerodynamic control surface without the switching signal σ(t).

[0121] Consider the attitude nonlinear switching system which is subject to compound disturbances and has compound nonlinear characteristics of the actuator:

[0122]

[0123] in, is the state vector of the system, representing the attitude angle and attitude angular velocity respectively; are all sufficiently smooth functions of the system state; d1 σ(t) ,d2 σ(t) are the unknown time-varying composite disturbances of the attitude loop and angular velocity loop, and there is a constant satisfy where d1 σ(t) =Δf1 σ(t) +Δd1 σ(t) , d2 σ(t) =Δf2 σ(t) +Δd2 σ(t) , is the uncertainty of the system, is an external unknown bounded disturbance term; SatDead(M c ) indicates that the control input has both dead zone and saturation nonlinear characteristics, and its form is as follows:

[0124] The actuator saturation nonlinearity is:

[0125]

[0126] Among them, M ci is the ith input of the actuator, M cmini is the minimum value that the actuator’s i-th input can reach, M cmaxi is the maximum value that the i-th input of the actuator can reach. Due to the physical limitations of the actuator, the range of the deflection angle of each rudder surface is -30°≤δ e ,δ a ,δ r ≤30°.

[0127] The actuator dead zone nonlinearity is:

[0128]

[0129] Among them, bli <0 and b ri >0 represents the left and right boundaries of the dead zone, m li >0 and m ri >0 represents the left and right slopes of the dead zone respectively.

[0130] Step 2-2) Analyze the actuator composite nonlinearity. Since the composite nonlinearity is relatively complex and difficult to handle directly, the right inverse function of the dead zone is introduced It satisfies D b D + b =I, defined as:

[0131]

[0132] The above dead zone right inverse function D + b (·) is placed before the entire actuator compound nonlinear link, and a new cascade nonlinear link is obtained. Next, the cascade link is analyzed in different situations.

[0133] Case 1.M ci >0

[0134] According to the dead zone right inverse function D + b (·) and the definition of saturation function sat(·), we can get the mathematical expression of the dead zone right inverse link and the saturation link in series:

[0135]

[0136] because According to the dead zone function

[0137] According to the definition of , the mathematical expression of the entire cascade nonlinear link after the introduction of the dead zone right inverse function can be obtained as:

[0138]

[0139] Case 2.M ci <0

[0140] Similar to case 1, the mathematical expression of the dead zone right inverse link and the saturation link in series is:

[0141]

[0142] because The mathematical expression of the entire cascade nonlinear link after the introduction of the dead zone right inverse function can be obtained as follows:

[0143]

[0144] Case 3.M ci =0

[0145] In this case, due to Furthermore, Therefore, we can get:

[0146]

[0147] In summary, the mathematical expression of the cascade nonlinear link after the introduction of the dead zone right inverse function can be summarized as:

[0148]

[0149] This means that after the dead zone right inverse function is introduced before the system's composite nonlinear link, the composite nonlinear link can be equivalently regarded as a new saturation link, and then the traditional limited control method can be used to deal with the actuator composite nonlinear problem. The processed attitude nonlinear switching system model can be rewritten as:

[0150]

[0151] The disturbance observer is designed to estimate the composite disturbance composed of discontinuous disturbance and uncertainty. Considering that when the switching signal σ(t) changes, the working mode of the system changes, the composite disturbance d1 σ(t) ,d2 σ(t) Also changes accordingly, then at the moment when the mode switches from i to j, the composite interference also changes from d1 i ,d2 i Instantly jump to d1 j ,d2 j , that is, the composite disturbance is discontinuous at the switching moment. In order to reduce the adverse effect of discontinuous composite disturbance on the system and improve the tracking control accuracy, this embodiment designs a disturbance observer that switches synchronously with the switching signal to estimate the composite disturbance.

[0152] The attitude loop disturbance observer is:

[0153]

[0154] The angular velocity loop disturbance observer is:

[0155]

[0156] Where, s1=[s 11 ,s 12 ,s 13 ] T , s2=[s 21 ,s 22 ,s 23 ] Tare the auxiliary sliding surfaces of the attitude loop and angular velocity loop respectively; z d1 =[z d11 ,z d12 ,z d13 ] T , z d2 =[z d21 ,z d22 ,z d23 ] T They are the internal states of the attitude loop and angular velocity loop interference observers respectively;

[0157] are the interference estimation values ​​of the attitude loop and angular velocity loop respectively; They are the gain matrices to be designed for the attitude loop and angular velocity loop, respectively, and satisfy

[0158] This embodiment designs an adaptive switching sliding mode controller based on a disturbance observer and an anti-saturation auxiliary system to generate δ e ,δ a ,δ r , and design a reinforcement learning controller to generate the horizontal canard rudder δ c The two are simultaneously input into the variable sweep wing aircraft model to jointly control the flight attitude. The overall structure of the control system is shown in the figure Figure 2 As shown. First, an adaptive switching sliding mode controller is designed. Figure 3 The structure diagram of the attitude control system of a variable-sweep-wing aircraft based on the backstepping adaptive switching sliding mode method with anti-saturation auxiliary system and disturbance observer is given.

[0159] The following anti-saturation auxiliary system is constructed to deal with the complex nonlinear problem of the actuator:

[0160]

[0161] Among them, ΔM c =sat new (M c )-M c ,ρ1∈R 3 ,ρ2∈R 3 is the state vector of the auxiliary system, c1>0, c2>0 are the matrix parameters to be designed.

[0162] Combined with the vector of the anti-saturation auxiliary system, the auxiliary variables are formed in the following form:

[0163]

[0164] Among them, z1∈R 3 ,z2∈R 3 is an auxiliary variable, x d is the attitude angle reference signal, The virtual control rate to be designed later is used to replace the traditional error vector with the auxiliary variable to dynamically compensate the controller, which can reduce or offset the influence of the actuator's compound nonlinearity.

[0165] An adaptive switching sliding mode controller for the attitude angle loop is designed based on backstepping.

[0166] To ensure the stability of the switching system, the designed switching sliding surface must satisfy that all sub-sliding surfaces intersect at the same equilibrium point, that is, satisfy S1 σ(t) (0) = S2 σ(t) (0) = 0. Sliding surface S1 with synchronous switching of modes σ(t) Designed for S1 σ(t) =P1 σ(t) z1=[S 11 σ(t) ,S 12 σ(t) ,S 13 σ(t) ] T ,in, is a positive definite diagonal matrix to be designed, satisfying (P1 σ(t) g1) -1 Existence and

[0167] In order to achieve smooth switching control between different subsystems, the following transition sliding surface is designed to construct a common coordinate transformation: S cm =C cm z1. Among them, C cm =diag{C cm1 ,C cm2 ,C cm3} is the positive definite diagonal matrix to be designed, and

[0168] In order to ensure that the system state can approach and eventually reach the predetermined sliding surface, and considering that the sign function in the traditional sliding mode reaching law is prone to cause chattering, this study changes the sign function sign to the hyperbolic tangent function tanh, and introduces adaptive dynamic compensation to enhance the robustness of the control system. Therefore, the improved sliding mode reaching law is designed as follows:

[0169]

[0170] Among them, k1 σ(t) >0 is the parameter to be designed, a>0 is a small constant used to adjust the smoothness, The adaptive law is as follows:

[0171]

[0172] in, δ1 σ(t) >0 is the parameter to be designed.

[0173] Virtual control rate for mode σ(t)=k Designed for:

[0174]

[0175] An adaptive switching sliding mode controller for the angular velocity loop is designed based on backstepping.

[0176] Sliding surface S2 with synchronous mode switching σ(t) Designed for: S2 σ(t) =P2 σ(t) z2=[S 21 σ(t) ,S 22 σ(t) ,S 23 σ(t) ] T .in,

[0177] is a positive definite diagonal matrix to be designed, satisfying (P2 σ(t) g2 σ(t) ) -1 Existence and

[0178]

[0179] In order to reduce the chattering caused by the sign function, the hyperbolic tangent function is used to replace the sign function, and an adaptive term is introduced to improve the robustness of the control system. Therefore, the improved sliding mode reaching law is designed as follows:

[0180]

[0181] Among them, k2 σ(t) >0 is the parameter to be designed, b>0 is a small constant used to adjust the smoothness, The adaptive law is as follows:

[0182]

[0183] in, δ2 σ(t) >0 is the parameter to be designed.

[0184] Adaptive multi-mode switching sliding mode control law for mode σ(t)=k Designed for:

[0185]

[0186] Design the state space of the reinforcement learning controller. To use reinforcement learning for attitude control of variable-sweep wing aircraft, we first need to select appropriate state variables, which can make the system model conform to the Markov property and fully describe the flight state of the system. The goal of variable-sweep wing aircraft control is to make the attitude angle track the target value Ω c =[α c ,β c ,μ c ] T , the posture tracking error is ΔΩ=[Δα,Δβ,Δμ] T , where Δα=α c -α,Δβ=β c -β,Δμ=μ c -μ. Then, the state variables are designed as follows:

[0187] s t =[α,β,μ,p,q,r] T (50)

[0188] Design the action space of the reinforcement learning controller. Considering that when the variable-sweep wing aircraft operates in subsonic mode, the horizontal canard rudder of the fuselage will be deployed and have a control effect on the system, while in hypersonic and hypersonic modes, the horizontal canard will be retracted into the fuselage, that is, the subsonic mode will have one more control rudder than other modes. For this reason, the action space is designed as:

[0189] A μ (s t )=δ c (51)

[0190] Since the deep deterministic policy gradient algorithm uses a deterministic strategy when generating actions, it does not have the ability to explore. In order to balance the exploration and utilization mechanisms and avoid falling into the local optimal solution, this paper superimposes a Gaussian noise n on the actions output by the Actor network. t ~N(0,σ 2 ), the final exploration strategy is:

[0191] a=A μ (s t )+n t (52)

[0192] In the formula, a is the actual action, A μ (s t ) is the neural network parameter μ in state s t The actions obtained according to the deterministic policy.

[0193] Design the reward function of the reinforcement learning controller. For the attitude tracking control problem of a variable-sweep wing aircraft, the control objective is to minimize the attitude angle tracking error in the shortest time, that is, to minimize the sum of the attitude angle errors at each time step. The sum of the attitude errors is defined as: e 2 (t) = (α c -α) 2 +(β c -β) 2 +(μ c -μ) 2 The reward function is designed as a combination of the following two parts: R = w1r1 + w2r2, where w1 and w2 are the weights of each part, which are designed to be 0.75 and 0.25 respectively, and:

[0194]

[0195] Among them, ε is the upper bound of the tolerable error. The advantages of designing the reward function in this way are: r1 represents the reward and punishment for the posture tracking error. If the sum of squared errors is less than a certain threshold, a reward is given, otherwise a penalty is given, which can prompt the agent to quickly learn a strategy that meets the tracking accuracy requirements; r2 represents the reward and punishment for whether the tracking effect is moving in a good direction. If the tracking error meets the accuracy requirements or the sum of squared errors at the current moment is smaller than that at the previous moment, a reward is given, otherwise a penalty is given, which can prompt the agent to learn a strategy that continuously reduces the tracking error.

[0196] Design the neural network structure of the reinforcement learning controller. A fully connected multi-hidden layer deep neural network structure is used. The specific design is as follows: the Actor network has a total of 5 layers, of which the input layer is a 6-dimensional state vector, the output layer is a 1-dimensional action vector, and there are 3 hidden layers in the middle. The number of hidden layer neurons is 32, 64, and 128 respectively. The first 4 layers all use the Relu activation function, and the output layer uses the Tanh activation function; the Critic network has a total of 5 layers, of which the input layer is a 7-dimensional vector connected by a 6-dimensional state vector and a 1-dimensional action vector, and the output layer is a 1-dimensional action value function estimate. The number of intermediate layers and neurons is the same as that of the Actor network. The first 4 layers all use the Relu activation function, and the output layer is a linear function.

[0197] In order to verify the effectiveness of the preset performance controller, the initial conditions of the simulation are: wing sweep angle Λ0 = 40°, flight altitude H0 = 24000m, flight speed V0 = 2500m / s, thrust size T = 160000N, and no thrust vector control is used. The initial attitude angles are: α0 = -1°, β0 = -2°, μ0 = 3°, the initial attitude angular velocities are: p0 = q0 = r0 = 0° / s, and the expected attitude angle command is: α d =2°(1-e -3t ),βd =0°,μ d =3°sin(0.3t), the simulation time is 30s. During the simulation, parameter uncertainty and discontinuous numerical interference are added to the attitude loop and angular velocity loop respectively. The uncertainty of the aerodynamic coefficient is between ±20%, and the interference Δd1 of the attitude loop σ(t) and the disturbance Δd2 of the angular velocity loop σ(t) They are: Among them, ΔM c1 and ΔM c2 The specific values ​​are:

[0198]

[0199] Considering the existence of saturation and dead zone compound nonlinear characteristics of the actuator, the saturation of the control surface deflection angle is -30°≤δ e ,δ a ,δ r ≤30°, and then the distribution matrix can be controlled Calculate the control torque vector M c The dead zone slope of the control surface deflection angle is m ri =m li =1, i = 1, 2, 3, the dead zone range is b ri =-b li =1°.

[0200] Figure 6 The estimation curve and error curve of the disturbance observer for the composite disturbance of the attitude loop and angular velocity loop are shown. It can be seen that the designed disturbance observer can quickly and accurately estimate the unknown composite disturbance in the system.

[0201] Figure 7The comparison of attitude angle, attitude angle tracking error, and attitude angular velocity response curves of the variable-sweep wing aircraft under the influence of different factors using the composite control strategy based on disturbance observer and anti-saturation auxiliary system proposed in this paper is shown. In the figure, the purple dotted line (subscript d) represents the attitude angle reference signal, the blue solid line (subscript 0) represents the system response curve when no treatment is done on the dead zone and saturation composite nonlinearity, the red solid line (subscript 1) represents the system response curve when only measures are taken to deal with saturation nonlinearity but not dead zone nonlinearity, and the yellow solid line represents the system response curve when measures are taken to deal with composite nonlinearity at the same time. It can be seen that the existence of dead zone and saturation composite nonlinearity in the system will seriously affect the control performance. The method of introducing the right inverse function of the dead zone and constructing the auxiliary system proposed in this paper can effectively deal with the adverse effects brought by composite nonlinearity, which not only improves the system response speed, but also greatly improves the tracking control accuracy. There is basically no steady-state error in the attitude tracking of the variable-sweep wing aircraft, and the angular velocity fluctuates slightly near the steady-state value, but it is still within an acceptable range.

[0202] Figure 8 The attitude angle, attitude tracking error, and attitude angular velocity response curves of the variable-sweep wing aircraft using the composite controller based on the disturbance observer and anti-saturation auxiliary system proposed in this paper and the adaptive switching sliding mode control method alone are compared under the presence of dead zone and saturation composite nonlinearity. In the figure, the yellow dotted line (subscript d) represents the attitude angle reference signal, the blue solid line (subscript 0) represents the system response curve when the switching sliding mode control strategy is adopted, and the red solid line represents the system response curve when the composite control strategy proposed in this paper is adopted. It can be seen that the composite control scheme combining the adaptive switching sliding mode controller and the reinforcement learning controller proposed in this paper makes the response speed and convergence speed of the control system significantly faster, the attitude tracking accuracy is higher, the steady-state error is smaller, and the control performance is better.

[0203] Fig. 9 The control rudder deflection curve generated by the control strategy proposed in this paper is shown. The left elevator, right elevator and rudder are all generated by the adaptive switching sliding mode controller, while the horizontal canard rudder is generated by the reinforcement learning controller, and it only plays a control role in the subsonic mode (about the first 8s).

[0204] Fig.10 The reward function curve, action value function estimation curve, and Critic and Actor network weight curves of the reinforcement learning controller are shown respectively. Fig.10 (a) It can be seen that the reward value is low and fluctuates in the early stage of training, but reaches the maximum value and remains stable in the middle and late stages of training. Fig.10 (b) It can be seen that the estimated value of the value function gradually increases from a large negative number and finally stabilizes near zero. Fig.10(c) It can be seen that the weights of the Critic and Actor neural networks converge to steady-state values ​​in a relatively short time.

[0205] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. The variable-sweep-wing aircraft attitude control technology based on reinforcement learning and switching sliding mode strategy in complex environments is characterized by: The following steps are involved: Step 1: Based on the theory of nonlinear switching systems, establish the attitude control model of variable-sweep wing aircraft in full-envelope flight; Step 2: Design a disturbance observer for complex environments with discontinuous disturbances, uncertainties, actuator dead zones, and saturation compound nonlinearities, and construct an anti-saturation auxiliary system to address adverse effects; Step 3: Based on the content of step 2, design an adaptive switching sliding mode controller based on backstepping; Step 4: Design a reinforcement learning controller and combine it with the adaptive switching sliding mode controller described in step 3 to form a composite controller, which acts together on the variable-sweep wing aircraft to achieve attitude control.

2. The variable-sweep wing aircraft attitude control technology based on reinforcement learning and switching sliding mode strategy in a complex environment according to claim 1 is characterized in that: The specific operations of step 1 include the following: Step 1-1) Establish the force and moment model of the variable-sweep wing aircraft and decompose the aerodynamic force into resistance D σ(t) , lateral force Y σ(t) and lift L σ(t) , the specific expression is as follows: Aerodynamic moment decomposed into rolling moment Pitch moment and yaw moment The specific expression is as follows: in, is the dynamic pressure, S σ(t) is the wing reference area, b σ(t) is the wingspan length, c σ(t) is the average aerodynamic chord length, σ(t):[0,+∞)→S={1,2,3} is the flight mode switching signal, X cg is the distance between the reference moment center and the center of mass of the aircraft, α is the angle of attack, β is the sideslip angle, p is the roll angular velocity, q is the pitch angular velocity, r is the yaw angular velocity, V is the flight speed, δ c is the horizontal canard rudder deflection angle, δ e is the left aileron rudder deflection angle, δ a is the right aileron rudder deflection angle, δ r is the rudder deflection angle, is the drag coefficient, is the lift coefficient, is the lateral force coefficient, is the rolling moment coefficient, is the pitching moment coefficient, is the yaw moment coefficient; Step 1-2) Establish a nonlinear mathematical model of the variable-sweep wing aircraft, select six physical quantities α, β, μ, p, q, and r to describe the flight motion process, and establish the attitude state equation as follows: Wherein, μ is the roll angle, M is the mass of the variable-sweep wing aircraft, g is the gravitational acceleration, T x ,T y ,T z are the components of thrust T in the x, y, and z axes respectively, γ is the track angle, are the moments of inertia of the three axes, l T ,m T ,n T are the components of thrust torque in three directions; Step 1-3) Establish the attitude affine nonlinear switching system model of the variable-sweep wing aircraft, considering the attitude angle Ω = [α, β, μ] T and attitude angular velocity ω=[p,q,r] T According to the time scale separation principle, the above equation can be rewritten as: in, represents the control torque vector, l c ,m c ,n c are the control moments in the roll, pitch and yaw directions respectively, is the control allocation matrix, δ=[δ c ,δ e ,δ a ,δ r ] T To control the deflection angle of the rudder surface, is the control gain matrix, and the specific expression is: in, and Indicates the basic lift coefficient and basic side force coefficient; They represent the moment of inertia of the three axes, They respectively represent the control torque excluding the aerodynamic control surfaces.

3. The variable-sweep wing aircraft attitude control technology based on reinforcement learning and switching sliding mode strategy in complex environments according to claim 2 is characterized in that The specific operations of step 2 include the following: Step 2-1) Consider the attitude nonlinear switching system that is subject to compound disturbances and has compound nonlinear characteristics in the actuator: in, is the state vector of the system, representing the attitude angle and attitude angular velocity respectively; f1 σ(t) , are all sufficiently smooth functions of the system state; d1 σ(t) ,d2 σ(t) are the unknown time-varying composite disturbances of the attitude loop and angular velocity loop, and there is a constant satisfy where d1 σ(t) =Δf1 σ(t) +Δd1 σ(t) , d2 σ(t) =Δf2 σ(t) +Δd2 σ(t) , is the uncertainty of the system, is an external unknown bounded disturbance term; SatDead(M c ) indicates that the control input has both dead zone and saturation nonlinear characteristics, and its form is as follows: The actuator saturation nonlinearity is: Among them, M ci is the ith input of the actuator, M cmini is the minimum value that the actuator’s i-th input can reach, M cmaxi is the maximum value that the i-th input of the actuator can reach, and the range of the deflection angle of each rudder surface is -30°≤δ e ,δ a ,δ r ≤30°; The actuator dead zone nonlinearity is: Among them, b li <0 and b ri >0 represents the left and right boundaries of the dead zone, m li >0 and m ri >0 indicates the left and right slopes of the dead zone respectively; Step 2-2) Introduce the dead zone right inverse function It satisfies D b D + b =I, which is: The dead zone right inverse function is placed before the actuator compound nonlinear link to obtain a new cascade nonlinear link, and the mathematical expression is: The processed attitude nonlinear switching system model can be rewritten as: Step 2-3) Design a disturbance observer to estimate the composite disturbance consisting of discontinuous disturbance and uncertainty. The attitude loop disturbance observer is: The angular velocity loop disturbance observer is: Where, s1=[s 11 ,s 12 ,s 13 ] T , s2=[s 21 ,s 22 ,s 23 ] T are the auxiliary sliding surfaces of the attitude loop and angular velocity loop respectively; z d1 =[z d11 ,z d12 ,z d13 ] T , z d2 =[z d21 ,z d22 ,z d23 ] T They are the internal states of the attitude loop and angular velocity loop interference observers respectively; are the interference estimation values ​​of the attitude loop and angular velocity loop respectively; They are the gain matrices to be designed for the attitude loop and angular velocity loop, respectively, and satisfy l j2i σ(t) >1.1η ji σ(t) ,i=1,2,3,j=1,2.

4. The variable-sweep wing aircraft attitude control technology based on reinforcement learning and switching sliding mode strategy in complex environments according to claim 3 is characterized in that The specific operations of step 3 include the following: Step 3-1) Construct the following anti-saturation auxiliary system: Among them, ΔM c =sat new (M c )-M c ,ρ1∈R 3 ,ρ2∈R 3 is the state vector of the anti-saturation auxiliary system, c1>0, c2>0 are the matrix parameters to be designed; Combined with the vector of the anti-saturation auxiliary system, the auxiliary variables of the following form are designed: Among them, z1∈R 3 ,z2∈R 3 is an auxiliary variable, x d is the attitude angle reference signal, This is the virtual control rate to be designed later; Step 3-2) Design the adaptive switching sliding mode controller of the attitude angle loop based on backstepping: Sliding surface S1 with synchronous switching of modes σ(t) Designed for S1 σ(t) =P1 σ(t) z1=[S 11 σ(t) ,S 12 σ(t) ,S 13 σ(t) ] T ;in, is a positive definite diagonal matrix to be designed, satisfying (P1 σ(t) g1) -1 Existence and Design transition sliding surface: S cm =C cm z1; among them, C cm =diag{C cm1 ,C cm2 ,C cm3 } is the positive definite diagonal matrix to be designed, and The sign function sign is changed to the hyperbolic tangent function tanh, and the adaptive term is introduced. The improved sliding mode reaching law is designed as follows: Among them, k1 σ(t) >0 is the parameter to be designed, a>0 is a small constant used to adjust the smoothness, The adaptive law is as follows: in, δ1 σ(t) >0 is the parameter to be designed; Virtual control rate for mode σ(t)=k Designed for: Step 3-3) Design the adaptive switching sliding mode controller of the angular velocity loop based on backstepping: Sliding surface S2 with synchronous switching of modes σ(t) Designed for: S2 σ(t) =P2 σ(t) z2=[S 21 σ(t) ,S 22 σ(t) ,S 23 σ(t) ] T ;in, is a positive definite diagonal matrix to be designed, satisfying (P2 σ(t) g2 σ(t) ) -1 Existence and The improved sliding mode reaching law is designed as follows: Among them, k2 σ(t) >0 is the parameter to be designed, b>0 is a small constant used to adjust the smoothness, The adaptive law is as follows: in, δ2 σ(t) >0 is the parameter to be designed; Adaptive switching sliding mode control law for mode σ(t)=k Designed for:

5. The variable-sweep-wing aircraft attitude control technology based on reinforcement learning and switching sliding mode strategy in complex environments according to claim 4 is characterized in that The specific operations of step 4 include the following: Step 4-1) Design the state space of the reinforcement learning controller: The goal is to make the attitude angle track the target value Ω c =[α c ,β c ,μ c ] T , the posture tracking error is ΔΩ=[Δα,Δβ,Δμ] T , where Δα=α c -α,Δβ=β c -β,Δμ=μ c -μ, then the state space is designed as: s t =[α,β,μ,p,q,r] T ; Step 4-2) Design the action space of the reinforcement learning controller: The designed action space is: A μ (s t )=δ c ; Superimpose a Gaussian noise n on the action output by the Actor network t ~N(0,σ 2 ), the final exploration strategy is: a=A μ (s t )+n t ; where a is the actual action, A μ (s t ) is the neural network parameter μ in state s t The actions obtained according to the deterministic strategy; Step 4-3) Design the reward function of the reinforcement learning controller: Define the sum of attitude errors: e 2 (t) = (α c -α) 2 +(β c -β) 2 +(μ c -μ) 2 The reward function is designed as: R = w1r1 + w2r2, where w1 and w2 are the weights of each part, which are designed to be 0.75 and 0.25 respectively, and: Among them, ε is the upper bound of the tolerable error; r1 represents the reward and punishment for the posture tracking error. If the sum of squared errors is less than a certain threshold, a reward is given, otherwise a penalty is given; r2 represents the reward and punishment for whether the tracking effect is moving in a good direction. If the tracking error meets the accuracy requirement or the sum of squared errors at the current moment is smaller than that at the previous moment, a reward is given, otherwise a penalty is given; Step 4-4) Design the neural network structure of the reinforcement learning controller: A fully connected multi-hidden layer deep neural network structure is adopted, and the specific design is as follows: the Actor network has a total of 5 layers, of which the input layer is a 6-dimensional state vector, the output layer is a 1-dimensional action vector, there are 3 hidden layers in the middle, the number of hidden layer neurons are 32, 64 and 128 respectively, the first 4 layers all use Relu activation function, and the output layer uses Tanh activation function; the Critic network has a total of 5 layers, of which the input layer is a 7-dimensional vector connected by a 6-dimensional state vector and a 1-dimensional action vector, the output layer is a 1-dimensional action value function estimate, the number of intermediate layers and neurons are the same as the Actor network, the first 4 layers all use Relu activation function, and the output layer is a linear function.

Images