Reinforcement learning-based intelligent decision control integration method for variable-elongation aircraft

Abstract

The invention provides a reinforcement learning-based intelligent decision-making control integrated design method for a variable-elongation aircraft, and the method comprises the steps: designing an attitude control law from the perspective of improving the comprehensive flight performance index of an aircraft in a gliding stage, formulating a deformation decision-making index, building an intelligent deformation decision-making model, and carrying out the intelligent decision-making control. The problems of deformation decision and coordination control of the variable-elongation aircraft in the gliding stage are solved. Compared with a fixed shape, the aircraft configuration is adjusted in real time according to the action strategy given by the intelligent agent, the lift-drag ratio can be effectively increased, meanwhile, the attitude tracking error is reduced, and the optimal comprehensive performance index is achieved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent decision-making control of variable-length aircraft, and in particular to an integrated intelligent decision-making control method for variable-length aircraft based on reinforcement learning. Background Art

[0002] Deformable aircraft can autonomously change their aerodynamic configuration according to different flight stages and tasks, thereby improving their own dynamic characteristics and flight performance. Therefore, compared with fixed-shape aircraft, deformable aircraft can adapt to more complex environments, perform more diverse tasks, have a larger flight envelope and stronger economic benefits, and have become one of the research hotspots in the aerospace field. Among the various deformation scheme designs, the telescopic wing method provides a large range of wing area and aspect ratio changes in a single degree of freedom, and the deformation mechanism design is simple and reliable, so it is widely used in the overall design of deformable aircraft.

[0003] How to formulate deformation strategies, when to deform and to what extent are important factors in achieving the optimal global performance of the aircraft. The development of deformation decisions needs to be based on the relevant information about the aircraft's own motion state and flight environment fed back by the sensors installed on the aircraft, and the performance indicators derived from the gliding phase mission requirements. The traditional naive idea of ​​deformation decision is to calibrate the flight mission offline and deploy the corresponding configuration at different stages. However, this method lacks real-time performance and is difficult to adapt to emergencies. In existing literature, many numerical optimization algorithms such as PSO and GPM are applied to the formulation of deformation strategies for deformable aircraft. However, the above methods simplify the aircraft model to a certain extent during the application process, ignoring the coupling effect of deformation decision and attitude control, that is, ignoring the direct or indirect relationship between flight performance and attitude motion. Summary of the invention

[0004] In view of the problem that the prior art cannot comprehensively consider the decision-making and control effects, and taking into account that the attitude movement of the deformable aircraft is affected by the change of the appearance, and that there is a certain relationship between the goal of the deformation decision and the attitude state, the present invention proposes an integrated intelligent decision-making and control method for a variable-length aircraft based on reinforcement learning, which can provide an effective deformation strategy and stable attitude tracking control during the gliding stage of the aircraft, thereby solving the problem of improving the comprehensive performance of the deformable aircraft during the gliding stage.

[0005] The technical solution of the present invention is:

[0006] An integrated intelligent decision-making and control method for a variable-length aircraft based on reinforcement learning comprises the following steps:

[0007] Step 1: For the variable-length aircraft, establish an attitude dynamics model, use the rudder deflection as the control variable, and combine the adaptive dynamic programming method to design the optimal tracking control law:

[0008] Step 1.1: Establish the attitude dynamics model of the variable-length aircraft as follows:

[0009]

[0010] In the formula, α, β, γ V are the angle of attack, sideslip angle and roll angle respectively; ω x ,ω y ,ω z is the three-axis rotation angular velocity; θ represents the track inclination; F a =[L,D,N] T are lift, drag and lateral force respectively; M a =[M ax ,M ay ,M az ] T is the aerodynamic moment of the three axes; F s =[F sx ,F sy ,F sz ] T Indicates the additional force of the three-axis deformation; M s =[M sx ,M sy ,M sz ] T is the additional moment of deformation of the three axes; J x ,J y ,J z is the three-axis principal moment of inertia; m is the total mass of the aircraft; V is the flight speed; g is the gravitational acceleration;

[0011] Step 1.2: Rewrite the attitude dynamics model of the variable-length aircraft into the attitude control system model form:

[0012]

[0013] In the formula, the state variable is selected as x1 = [α, β, γ V ] T ,x2=[ω x ,ω y ,ω z ] T , control input u=[δ x ,δ y ,δ z ] T ;

[0014] Step 1.3: Divide the attitude control system model into an angle slow loop and an angular velocity fast loop. First, design a virtual controller for the slow loop and use it as the desired tracking trajectory of the angular velocity fast loop. Then, design a controller for the fast loop.

[0015] Slow loop controller design:

[0016] Assume that the angle subsystem satisfies the following desired trajectory:

[0017]

[0018] where x 1c is the expected value of the state variable x1; is the steady-state part of the virtual control input, is the estimated value of the comprehensive disturbance d1(x1,x2,ξ);

[0019] Get the steady-state part of the virtual control input as follows:

[0020]

[0021] The optimal feedback part of the slow loop virtual controller is defined as x 2d (t) is the virtual control input of the angle subsystem, and the tracking error is e1(t) = x1(t) - x 1c (t), then:

[0022]

[0023] Among them, f1 e (e1) = f1(x1) - f1(x 1c );

[0024] Fast loop controller design:

[0025] Assume that the desired trajectory of the angular velocity subsystem satisfies the following equation:

[0026]

[0027] where u d is the steady-state part of the actual control input, is the estimated value of the comprehensive disturbance d2(x1,x2,ξ);

[0028] The steady-state control input part of the rudder deflection is as follows:

[0029]

[0030] Define the optimal feedback part of the fast loop controller as u e (t) = u(t) - u d (t), the tracking error is e2(t) = x2(t)-x 2d (t), the tracking error equation of the fast loop can be written as

[0031]

[0032] in,

[0033] Step 1.4: Perform approximate optimal control based on adaptive dynamic programming method:

[0034] The tracking error equations of the slow loop and the fast loop are unified as follows:

[0035]

[0036] The optimal cost function is expressed as follows:

[0037]

[0038] In the formula, Q, is a positive definite symmetric matrix;

[0039] According to the optimal control theory, by solving the HJB equation

[0040]

[0041] Get the optimal control input u * ;

[0042] Step 2: According to the mission requirements of the variable-length aircraft during the gliding phase, the performance indicators of the comprehensive lift-to-drag ratio and attitude tracking error are set for the deformation decision-making problem of the deformable aircraft, and the deep deterministic policy gradient algorithm is used to realize the intelligent deformation decision of the variable-length aircraft; the Actor-Critic framework in deep reinforcement learning is used to realize real-time adjustment of the deformation rate according to the flight status and flight mission, thereby realizing integrated decision-making and control.

[0043] Furthermore, the variable-length aircraft is a variable-length aircraft with wings that can be symmetrically and continuously extended and retracted on both sides.

[0044] Furthermore, in the attitude dynamics model of the variable-length aircraft in step 1.1, the calculation form of the force and moment acting on the aircraft is as follows:

[0045] (1) Aerodynamic force F a

[0046] The calculation method of lift L, drag D and side force N is as follows:

[0047]

[0048] Where q is the dynamic pressure; S is the reference area of ​​the wing when it is not deformed; is the equivalent lift coefficient, is the equivalent resistance coefficient, is the equivalent lateral force coefficient;

[0049] (2) Deformation additional force F s

[0050] The analytical expression of the additional force generated by deformation in the body coordinate system is:

[0051]

[0052] In the formula, ω=[ω x ,ω y ,ω z ] T ;m i Indicates the mass of the left or right wing of the aircraft; v si is the velocity vector of the deformed wing in the body coordinate system; s i Indicates the coordinates of the center of mass of the left or right wing of the aircraft in the body coordinate system;

[0053] (3) Aerodynamic moment M a

[0054] The rolling moment, yaw moment, and pitch moment are calculated as follows:

[0055]

[0056] Where b and c are the reference lengths of the aircraft in the horizontal and vertical directions, respectively. is the equivalent rolling moment coefficient, is the equivalent yaw moment coefficient, is the equivalent yaw moment coefficient;

[0057] (4) Deformation additional moment M s

[0058] The deformation additional moment characterizes the influence of the relative motion of the wing on the fuselage, and its analytical expression is as follows:

[0059]

[0060] where v f is the velocity vector of the aircraft.

[0061] Furthermore, the equivalent aerodynamic force coefficient and the equivalent aerodynamic moment coefficient are expressed as polynomial functions of the deformation rate ξ and the flight state, as shown below:

[0062]

[0063] where δ x ,δ y ,δ z represents the rudder deflection angle in the three-axis direction; Ma is the Mach number; S(ξ) is the function that reflects the change of the reference area; C Lis the lift coefficient, C D is the drag coefficient, C N is the lateral force coefficient, m x is the rolling moment coefficient, m y is the yaw moment coefficient, m z is the yaw moment coefficient.

[0064] Furthermore, in step 1.3, The following nonlinear disturbance observer is obtained:

[0065]

[0066] The following nonlinear disturbance observer is obtained:

[0067]

[0068] where p 11 ,p 12 ,p 21 ,p 22 is the auxiliary variable of the observer; L 11 ,L 12 ,L 21 ,L 22 is the design parameter of the disturbance observer; L ij =diag(l ijk ),l ijk >0, where k=1,2,3.

[0069] Furthermore, in step 1.4, an evaluation network consisting of a single-layer neural network is used to approximate the optimal cost function

[0070]

[0071] Among them, W represents the ideal weight of the evaluation network; Φ(e) represents the activation function to be designed; ε(e) represents the approximate error of the neural network; and the evaluation network is approximated in the following form:

[0072]

[0073] In the formula represents the approximation of the ideal weights of the evaluation network;

[0074] The final control input is obtained based on the approximate result of the evaluation network:

[0075]

[0076] Furthermore, the evaluation network is trained, and the training goal is to minimize the loss function According to the gradient descent method, the weights of the evaluation network are updated according to the following formula

[0077]

[0078] In the formula, α c ,α s >0 is the learning rate; And normalize it J s (e) is the Lyapunov function; The definition is as follows:

[0079]

[0080] Furthermore, in step 2, the performance indicators of the comprehensive lift-to-drag ratio and attitude tracking error are:

[0081]

[0082] Where γ∈[0,1] represents the discount factor; e λ =λ-λ c is the lift-to-drag ratio λ and the maximum lift-to-drag ratio λ that the aircraft can achieve c The error of e α =α-α c is the angle of attack tracking error; R t (·) is the reward function.

[0083] Furthermore, in step 2, in the deep deterministic policy gradient algorithm:

[0084] The state data in the observation space is:

[0085] S t =[s t-4 ,s t-3 ,s t-2 ,s t-1 ,s t ] T

[0086] where s t =[e λ ,e α ,ω z ] T ;

[0087] The action space uses positional instructions to change the expected deformation rate ξ of the extended aircraft c As action output;

[0088] The reward function takes the following form:

[0089]

[0090] in:

[0091]

[0092] In the formula, is the weight of each item in the reward function.

[0093] Beneficial Effects

[0094] The present invention proposes a method for integrated design of intelligent decision-making and control of variable-length aircraft based on reinforcement learning. From the perspective of improving the comprehensive flight performance index of the aircraft in the gliding stage, the attitude control law is designed, the deformation decision index is formulated, and the intelligent deformation decision model is established to solve the deformation decision and coordination control problems of the variable-length aircraft in the gliding stage. Compared with the fixed shape, the present invention can effectively improve the lift-to-drag ratio by adjusting the aircraft configuration in real time according to the action strategy given by the intelligent agent, while reducing the attitude tracking error, and achieving the optimal comprehensive performance index.

[0095] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0097] Figure 1 Design a flow chart for the integrated decision-making and control method of the variable-length aircraft of the present invention;

[0098] Figure 2 It is a deformation schematic diagram of the variable-length aircraft of the present invention;

[0099] Figure 3 Set up the structure of the Actor network;

[0100] Figure 4 The structure setting of the Critic network;

[0101] Figure 5 This is a diagram of the integrated training results of intelligent deformation decision control of the present invention;

[0102] Figure 6 This is the output result diagram of the intelligent deformation decision of the present invention;

[0103] Figure 7 This is a diagram showing the lift-to-drag ratio variation of the variable-length aircraft of the present invention;

[0104] Figure 8 This is a graph showing the change in posture tracking error during the deformation decision stage of the present invention;

[0105] Fig. 9This is a diagram showing the rudder deviation change results during the deformation decision stage of the present invention. DETAILED DESCRIPTION

[0106] Embodiments of the present invention are described in detail below. The embodiments are exemplary and intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0107] This embodiment provides a variable-length aircraft decision-making and control integrated method based on reinforcement learning, and the variable-length aircraft decision-making and control design process reference Figure 1 As shown, the variable-length aircraft decision-making and control integration method based on reinforcement learning includes:

[0108] Step 1: Face Figure 2 For the variable-length aircraft shown in the figure, an attitude dynamics model is established, the rudder deflection is used as the control variable, and the optimal tracking control law is designed in combination with the adaptive dynamic programming method;

[0109] Step 1.1: According to the multi-rigid body dynamics modeling method and Newton-Euler method, the attitude dynamics model of the variable-length aircraft with symmetrical and continuous extension of the wings on both sides can be expressed as follows:

[0110]

[0111] In the formula, α, β, γ V are the angle of attack, sideslip angle and roll angle respectively; ω x ,ω y ,ω z is the three-axis rotation angular velocity; θ represents the track inclination; F a =[L,D,N] T are lift, drag and lateral force respectively; M a =[M ax ,M ay ,M az ] T is the aerodynamic moment of the three axes; F s =[F sx ,F sy ,F sz ] T Indicates the additional force of the three-axis deformation; M s =[M sx ,M sy ,M sz ] T is the additional moment of deformation of the three axes; J x ,J y ,J z is the three-axis principal moment of inertia; m is the total mass of the aircraft; V is the flight speed; g is the gravitational acceleration.

[0112] The calculation form of the force and moment on the aircraft is as follows:

[0113] (1) Aerodynamic force F a calculate

[0114] The calculation method of lift L, drag D and side force N is as follows:

[0115]

[0116] In the formula, q is the dynamic pressure; S is the reference area of ​​the wing when it is not deformed. In order to facilitate the expression of the equation, the influence of the deformation is uniformly considered in the force coefficient, and the equivalent aerodynamic coefficient expression is obtained as follows:

[0117]

[0118] In the formula, δ x ,δ y ,δ z Indicates the rudder deflection angle in three-axis directions; Ma is the Mach number; is the equivalent lift coefficient, is the equivalent resistance coefficient, is the equivalent side force coefficient; the change of the reference area is expressed by the function S(ξ) of the deformation rate ξ, and the deformation rate ξ is defined as the normalized parameter of the deformation state quantity, that is, the wing span, that is:

[0119]

[0120] Where b represents the wingspan, and the subscripts represent the maximum and minimum values ​​of the wingspan.

[0121] (2) Deformation additional force F s calculate

[0122] The analytical expression of the additional force generated by deformation in the body coordinate system is:

[0123]

[0124] In the formula, ω=[ω x ,ω y ,ω z ] T ;m i Indicates the mass of the left or right wing of the aircraft; v si is the velocity vector of the deformed wing in the body coordinate system; s i Indicates the coordinates of the center of mass of the left or right wing of the aircraft in the body coordinate system.

[0125] (3) Aerodynamic moment M a calculate

[0126] Similar to the aerodynamic calculations, the rolling moment, yaw moment, and pitch moment are calculated as follows:

[0127]

[0128] Where b and c are the reference lengths of the aircraft in the horizontal and vertical directions, respectively. is the equivalent rolling moment coefficient, is the equivalent yaw moment coefficient, is the equivalent yaw moment coefficient.

[0129] The equivalent aerodynamic moment coefficient is expressed as a polynomial function of the deformation rate and the flight state as follows:

[0130]

[0131] (4) Deformation additional moment M s calculate

[0132] The deformation additional moment characterizes the influence of the relative motion of the wing on the fuselage, and its analytical expression is as follows:

[0133]

[0134] where v f is the velocity vector of the aircraft.

[0135] Step 1.2: Write the attitude dynamics model of the variable-length aircraft into the following attitude control system model form:

[0136]

[0137] In the formula, the state variable is selected as x1 = [α, β, γ V ] T ,x2=[ω x ,ω y ,ω z ] T , control input u=[δ x ,δ y ,δ z ] T .

[0138] Step 1.3: The attitude control system shown in formula (9) can be divided into an angle slow loop and an angular velocity fast loop. Based on the backstepping method, a virtual controller is first designed for the slow loop and used as the expected tracking trajectory of the angular velocity fast loop. Then, the controller of the fast loop is designed.

[0139] Slow loop controller design:

[0140] Assume that the angle subsystem satisfies the following desired trajectory:

[0141]

[0142] where x 1c is the expected value of the state variable x1; is the steady-state part of the virtual control input, The estimated value of the comprehensive disturbance d1(x1,x2,ξ) is obtained by the following nonlinear disturbance observer:

[0143]

[0144] where p 11 ,p 12 is the auxiliary variable of the observer; L 11 ,L 12 is the design parameter of the disturbance observer, L ij =diag(l ijk ),l ijk >0, where k=1,2,3.

[0145] Then, the steady-state part of the virtual control input can be obtained as follows:

[0146]

[0147] The optimal feedback part of the slow loop virtual controller is defined as x 2d (t) is the virtual control input of the angle subsystem, and the tracking error is e1(t) = x1(t) - x 1c (t), then:

[0148]

[0149] Among them, f1 e (e1) = f1(x1) - f1(x 1c ), the superscript d and superscript e represent the optimal feedback part.

[0150] Fast loop controller design:

[0151] Assume that the desired trajectory of the angular velocity subsystem satisfies the following equation:

[0152]

[0153] where u d is the steady-state part of the actual control input, The estimated value of the comprehensive disturbance d2(x1,x2,ξ) is obtained by the following nonlinear disturbance observer:

[0154]

[0155] where p 21 ,p22 is the auxiliary variable of the observer; L 21 ,L 22 are the design parameters of the disturbance observer,

[0156] L ij =diag(l ijk ),l ijk >0, where k=1,2,3.

[0157] Then, the steady-state control input part of the rudder deflection can be obtained as follows:

[0158]

[0159] Define the optimal feedback part of the fast loop controller as u e (t) = u(t) - u d (t), the tracking error is e2(t) = x2(t)-x 2d (t), the tracking error equation of the fast loop can be written as

[0160]

[0161] in,

[0162] Step 1.4: Approximate optimal control based on adaptive dynamic programming method:

[0163] The error systems shown in formula (13) and formula (17) can be uniformly expressed as:

[0164]

[0165] The optimal control input part ensures the stability of the error system (18) while minimizing the cost function V(e,u). Then the optimal cost function is expressed as follows:

[0166]

[0167] In the formula, Q, is a positive definite symmetric matrix.

[0168] According to the optimal control theory, the optimal control input u * It can be obtained by solving the following HJB equation:

[0169]

[0170] The HJB equation is a nonlinear differential equation, and it is usually difficult to obtain an analytical solution. In this embodiment, an evaluation network consisting of a single-layer neural network is used to approximate V * (e)

[0171]

[0172] in, represents the ideal weight of the evaluation network; represents the activation function to be designed; l is the number of neurons; ε(e) represents the approximate error of the neural network.

[0173] The ideal weights of the neural network are unknown, and the network is approximated using the following form:

[0174]

[0175] In the formula, Represents an approximation of the ideal weights for the evaluation network.

[0176] According to the approximate results of the evaluation network, the final control input can be obtained:

[0177]

[0178] Then, the approximation of the Hamiltonian function is:

[0179]

[0180] The training goal of the evaluation network is to minimize the loss function According to the gradient descent method, the weights of the evaluation network are updated as follows:

[0181]

[0182] In the formula, α c ,α s >0 is the learning rate; And normalize it J s (e) is the Lyapunov function to be designed; The definition is as follows:

[0183]

[0184] The stability analysis is given below:

[0185] Lemma: There exists a positive definite matrix and So that the following is true:

[0186]

[0187] Define the network weight estimation error as Select the Lyapunov function as:

[0188]

[0189] but:

[0190]

[0191] in, B=g(x)R -1 g T (x); ε H represents the residual error caused by the neural network approximation, as shown in (29).

[0192]

[0193] Considering (28) can be written as follows:

[0194]

[0195] Assume a m ≤||A||≤a M , b m ≤||B||≤b M , ||ε(e)||≤E M , ||Φ(e)||≤Φ M ,||f(e)+g(x)u * ||≤δ(e), and the following inequality holds:

[0196]

[0197] According to (31) and (32), we can get:

[0198]

[0199] in,

[0200] Scenario 1: When When, then

[0201] Assume that there exists K>0, such that And combined with You can get:

[0202]

[0203] Therefore, when When (35) or (36) holds,

[0204]

[0205] Situation 2: When When, then

[0206] According to (33) and (26), we can get:

[0207]

[0208] Therefore, when the following inequality holds,

[0209]

[0210] According to Lyapunov theory, the attitude tracking error and weight estimation error are always ultimately bounded.

[0211] Step 2: According to the mission requirements of the variable-length aircraft in the gliding phase, the deformation decision-making problem of the deformable aircraft is solved by using a deep deterministic policy gradient algorithm to realize the intelligent deformation decision of the variable-length aircraft. The Actor-Critic framework in deep reinforcement learning is used to adjust the deformation rate in real time according to the flight status and flight mission, thus realizing the integration of decision-making and control.

[0212] Lift-to-drag ratio is a flight performance indicator that is focused on during the gliding phase of an aircraft. Generally speaking, an aerodynamic shape with a larger lift-to-drag ratio can achieve a greater range, thereby increasing the flexibility of the aircraft in performing tasks. Therefore, appropriate wing deformation can ensure that the lift-to-drag ratio is as large as possible while the aircraft is tracking instructions. When designing traditional deformation decision methods, only the lift-to-drag ratio as a single indicator is generally considered, but the impact of the deformation process on attitude tracking accuracy is ignored. Therefore, the goal of the deformation decision in this step is to enable the aircraft to obtain the largest possible lift-to-drag ratio while reducing the attitude tracking error. In this embodiment, a performance indicator that combines lift-to-drag ratio and attitude tracking error is selected, and compared to focusing on the performance at a certain moment, further attention is paid to the performance within a certain flight time period, specifically:

[0213]

[0214] Where γ∈[0,1] represents the discount factor; e λ =λ-λ c is the lift-to-drag ratio λ and the maximum lift-to-drag ratio λ that the aircraft can achieve c The error of e α =α-α c is the angle of attack tracking error; R t (·) is the reward function.

[0215] According to the deformation decision goal shown in formula (40), the intelligent deformation decision design of the variable-length aircraft is performed based on the deep deterministic policy gradient algorithm:

[0216] (1) Observation space

[0217] The wing deformation decision is expected to obtain the optimal lift-to-drag ratio while minimizing the attitude tracking error as much as possible. As a continuous system, the aircraft system has a certain inertia. Adding observations of error changes or state changes is conducive to improving the fitting effect of the neural network to the actual situation. Therefore, the state quantities in the observation space should include lift-to-drag ratio error, attitude tracking error and three-axis rotation angular velocity.

[0218] Combined with the decision-making task objectives, the observation space contains the following state information:

[0219] s t =[e λ ,e α ,ω z ] T (41)

[0220] Considering that the aircraft system has a certain inertia, the state data of five historical cycles are further introduced into the observation space:

[0221] S t =[s t-4 ,s t-3 ,s t-2 ,s t-1 ,s t ] T (42)

[0222] When training a neural network, the above state quantities are normalized to [0,1] in order to avoid problems such as different magnitudes of different feature quantities and gradient explosion.

[0223] (2) Action Space

[0224] The action output of the agent is the wing deformation instruction of the hypersonic shape-changing aircraft. In this embodiment, the action space adopts the form of positional instructions, that is, the expected deformation rate of the variable-length aircraft:

[0225] a t =ξ c (43)

[0226] (3) Reward Function

[0227] The reward function takes the following form:

[0228]

[0229] in:

[0230]

[0231] In the formula, is the weight of each item in the reward function. The first and second items on the right side of the reward function represent the penalties for the angle of attack tracking error and lift-to-drag ratio error, respectively. Since the above errors vary in different ranges during the deformation process, in order to facilitate the adjustment of the weights and ensure that the Q function is within a reasonable range to improve the fitting effect of the critic network, Represents the absolute value of the error normalized to [0,1]; the third item is used to reward the situation where there is a trend of reducing the angle of attack tracking error when the agent takes actions according to the strategy, so as to encourage the agent to make decisions towards the trend of stable posture tracking.

[0232] The execution process of the deep deterministic policy gradient algorithm is shown in Table 1:

[0233] Table 1 Execution process of deep deterministic policy gradient algorithm

[0234]

[0235]

[0236] (4) Neural network structure design

[0237] Both Actor and Critic networks use multi-hidden layer back-propagation feed-forward neural networks, and the structure is set as follows Figure 3 and Figure 4 shown.

[0238] The input layer of the Actor network has 15 neurons corresponding to the 15-dimensional environment input; after three fully connected hidden layers, each has 64 neurons, and the activation function is ReLU; one neuron in the output layer corresponds to the action of the 1-dimensional agent, that is, the wing deformation instruction, and the activation function is set to tanh type. Adding a bias to it ensures that the output action is within the set range, which helps the rapid convergence of training.

[0239] The input layer of the Critic network has 26 neurons, including 15-dimensional environmental input and 1-dimensional action; the observation state passes through 2 layers of fully connected layers consisting of 64 neurons and then sums with the action input through a layer of fully connected layers consisting of 64 neurons in the same dimension, and the activation function is ReLU; then it passes through another layer of fully connected layers consisting of 64 neurons and is output. The output layer is a fully connected layer, which outputs a 1-dimensional action cost function corresponding to the initial state and strategy, namely the Q function.

[0240] For the decision-making and control integration method of this embodiment, the effectiveness of the agent strategy is verified through MATLAB using the following simulation scenario.

[0241] The simulation scenario is set as follows: 0 to 5 seconds is the aircraft attitude stabilization stage, and the intelligent deformation decision is added from the 5th second.

[0242] Figure 5 Represents the change in the average reward curve during the training of a deep deterministic policy gradient agent.

[0243] Figure 6 The deformation rate change results of the variable-span aircraft are given. The preset deformation instructions are used to test the tracking process of attitude control during the change of the wingspan of the variable-span aircraft within 0 to 5 seconds. After 5 seconds, the deformation rate is output online by the intelligent deformation decision.

[0244] Figure 7 It represents the change result of the lift-to-drag ratio of the aircraft. The intelligent deformation decision can effectively improve the lift-to-drag ratio.

[0245] Figure 8 The comparison results of attitude tracking errors in the intelligent deformation decision stage are given. Compared with the basic decision instructions (uniform deformation), the design of comprehensive performance indicators effectively improves the attitude tracking accuracy of the aircraft.

[0246] Fig. 9 Indicates the rudder deviation change result in the intelligent deformation decision stage.

[0247] Based on the above technical solution, the integrated method of intelligent decision-making and control of a variable-length aircraft based on reinforcement learning provided by the present invention achieves an improvement in the comprehensive performance of the aircraft in the gliding stage by designing attitude control laws and deformation decision methods.

[0248] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.

Claims

1. An integrated intelligent decision-making and control method for a variable-length aircraft based on reinforcement learning, characterized in that: The following steps are involved: Step 1: For the variable-length aircraft, establish an attitude dynamics model, use the rudder deflection as the control variable, and combine the adaptive dynamic programming method to design the optimal tracking control law: Step 1.1: Establish the attitude dynamics model of the variable-length aircraft as follows: In the formula, α, β, γ V are the angle of attack, sideslip angle and roll angle respectively; ω x ,ω y ,ω z is the three-axis rotation angular velocity; θ represents the track inclination; F a =[L,D,N] T are lift, drag and lateral force respectively; M a =[M ax ,M ay ,M az ] T is the aerodynamic moment of the three axes; F s =[F sx ,F sy ,F sz ] T Indicates the additional force of the three-axis deformation; M s =[M sx ,M sy ,M sz ] T is the additional moment of deformation of the three axes; J x ,J y ,J z is the three-axis principal moment of inertia; m is the total mass of the aircraft; V is the flight speed; g is the gravitational acceleration; Step 1.2: Rewrite the attitude dynamics model of the variable-length aircraft into the attitude control system model form: In the formula, the state variable is selected as x1 = [α, β, γ V ] T ,x2=[ω x ,ω y ,ω z ] T , control input u=[δ x ,δ y ,δ z ] T ; Step 1.3: Divide the attitude control system model into an angle slow loop and an angular velocity fast loop. First, design a virtual controller for the slow loop and use it as the desired tracking trajectory of the angular velocity fast loop. Then, design a controller for the fast loop. Slow loop controller design: Assume that the angle subsystem satisfies the following desired trajectory: where x 1c is the expected value of the state variable x1; is the steady-state part of the virtual control input, is the estimated value of the comprehensive disturbance d1(x1,x2,ξ); Get the steady-state part of the virtual control input as follows: The optimal feedback part of the slow loop virtual controller is defined as x 2d (t) is the virtual control input of the angle subsystem, and the tracking error is e1(t) = x1(t) - x 1c (t), then: in, Fast loop controller design: Assume that the desired trajectory of the angular velocity subsystem satisfies the following equation: where u d is the steady-state part of the actual control input, is the estimated value of the comprehensive disturbance d2(x1,x2,ξ); The steady-state control input part of the rudder deflection is as follows: Define the optimal feedback part of the fast loop controller as u e (t) = u(t) - u d (t), the tracking error is e2(t) = x2(t)-x 2d (t), the tracking error equation of the fast loop can be written as in, Step 1.4: Perform approximate optimal control based on adaptive dynamic programming method: The tracking error equations of the slow loop and the fast loop are unified as follows: The optimal cost function is expressed as follows: In the formula, is a positive definite symmetric matrix; According to the optimal control theory, by solving the HJB equation Get the optimal control input u * ; Step 2: According to the mission requirements of the variable-length aircraft during the gliding phase, the performance indicators of the comprehensive lift-to-drag ratio and attitude tracking error are set for the deformation decision-making problem of the deformable aircraft, and the deep deterministic policy gradient algorithm is used to realize the intelligent deformation decision of the variable-length aircraft; the Actor-Critic framework in deep reinforcement learning is used to realize real-time adjustment of the deformation rate according to the flight status and flight mission, thereby realizing integrated decision-making and control.

2. The intelligent decision-making and control integrated method for a variable-length aircraft based on reinforcement learning according to claim 1, characterized in that: The variable-length aircraft is a variable-length aircraft with wings that are symmetrically and continuously retractable on both sides.

3. The intelligent decision-making and control integrated method for a variable-length aircraft based on reinforcement learning according to claim 1, characterized in that: In the attitude dynamics model of the variable-length aircraft in step 1.1, the calculation form of the force and moment acting on the aircraft is as follows: (1) Aerodynamic force F a The calculation method of lift L, drag D and side force N is as follows: Where q is the dynamic pressure; S is the reference area of ​​the wing when it is not deformed; is the equivalent lift coefficient, is the equivalent resistance coefficient, is the equivalent lateral force coefficient; (2) Deformation additional force F s The analytical expression of the additional force generated by deformation in the body coordinate system is: In the formula, ω=[ω x ,ω y ,ω z ] T ;m i Indicates the mass of the left or right wing of the aircraft; v si is the velocity vector of the deformed wing in the body coordinate system; s i Indicates the coordinates of the center of mass of the left or right wing of the aircraft in the body coordinate system; (3) Aerodynamic moment M a The rolling moment, yaw moment, and pitch moment are calculated as follows: Where b and c are the reference lengths of the aircraft in the horizontal and vertical directions, respectively. is the equivalent rolling moment coefficient, is the equivalent yaw moment coefficient, is the equivalent yaw moment coefficient; (4) Deformation additional moment M s The deformation additional moment characterizes the influence of the relative motion of the wing on the fuselage, and its analytical expression is as follows: where v f is the velocity vector of the aircraft.

4. The intelligent decision-making and control integrated method for a variable-length aircraft based on reinforcement learning according to claim 3 is characterized by: The equivalent aerodynamic force coefficient and the equivalent aerodynamic moment coefficient are expressed as polynomial functions of the deformation rate ξ and the flight state as follows: where δ x ,δ y ,δ z represents the rudder deflection angle in the three-axis direction; Ma is the Mach number; S(ξ) is the function that reflects the change of the reference area; C L is the lift coefficient, C D is the drag coefficient, C N is the lateral force coefficient, m x is the rolling moment coefficient, m y is the yaw moment coefficient, m z is the yaw moment coefficient.

5. The intelligent decision-making and control integrated method for a variable-length aircraft based on reinforcement learning according to claim 1, characterized in that: In step 1.3, The following nonlinear disturbance observer is obtained: The following nonlinear disturbance observer is obtained: where p 11 ,p 12 ,p 21 ,p 22 is the auxiliary variable of the observer; L 11 ,L 12 ,L 21 ,L 22 is the design parameter of the disturbance observer; L ij =diag(l ijk ),l ijk >0, where k=1,2,3.

6. The intelligent decision-making and control integrated method for a variable-length aircraft based on reinforcement learning according to claim 1, characterized in that: In step 1.4, an evaluation network consisting of a single-layer neural network is used to approximate the optimal cost function Among them, W represents the ideal weight of the evaluation network; Φ(e) represents the activation function to be designed; ε(e) represents the approximate error of the neural network; and the evaluation network is approximated in the following form: In the formula represents the approximation of the ideal weights of the evaluation network; The final control input is obtained based on the approximate result of the evaluation network:

7. The intelligent decision-making and control integrated method for a variable-length aircraft based on reinforcement learning according to claim 6, characterized in that: Train the evaluation network, and the training goal is to minimize the loss function According to the gradient descent method, the weights of the evaluation network are updated according to the following formula In the formula, α c ,α s >0 is the learning rate; And normalize it J s (e) is the Lyapunov function; The definition is as follows:

8. The intelligent decision-making and control integrated method for a variable-length aircraft based on reinforcement learning according to claim 1, characterized in that: In step 2, the performance indicators of the comprehensive lift-to-drag ratio and attitude tracking error are: Where γ∈[0,1] represents the discount factor; e λ =λ-λ c is the lift-to-drag ratio λ and the maximum lift-to-drag ratio λ that the aircraft can achieve c The error of e α =α-α c is the angle of attack tracking error; R t (·) is the reward function.

9. The intelligent decision-making and control integrated method for a variable-length aircraft based on reinforcement learning according to claim 1, characterized in that: In step 2, in the deep deterministic policy gradient algorithm: The state data in the observation space is: S t =[s t-4 ,s t-3 ,s t-2 ,s t-1 ,s t ] T where s t =[e λ ,e α ,ω z ] T ; The action space uses positional instructions to change the expected deformation rate ξ of the extended aircraft c As action output; The reward function takes the following form: in: In the formula, is the weight of each item in the reward function.

Images