Morphing aircraft level flight deceleration maneuver decision-making method based on reinforcement learning

Abstract

The invention relates to a variant aircraft level flight deceleration maneuver decision-making method based on reinforcement learning, and belongs to the technical field of aviation. Comprising the following steps: establishing a kinematics model and a dynamics model of a morphing aircraft; establishing an aerodynamic force and aerodynamic moment model of the morphing aircraft, and considering aerodynamic force and aerodynamic moment changes caused by morphing; an integral control structure of level flight deceleration maneuver is designed, PI control is used for the speed, backstepping control is used for the attitude angle to keep the attitude stable, and PID control is used for the height at the outer ring of the pitch angle to keep the height unchanged; designing a reward and punishment function of level flight speed reducer maneuver; designing an intelligent agent based on a reinforcement learning DQN algorithm, and discretizing possible degrees of left and right sweepback angles and V tail dihedral angles to adapt to a discrete action space of the DQN algorithm; according to the method, a reinforcement learning control method is used so as to ensure that the optimal aerodynamic force, flight efficiency and control efficiency can be achieved when the aircraft executes specific maneuver.

Description

Technical Field

[0001] The invention belongs to the field of aviation technology and relates to a variant aircraft level flight deceleration maneuver decision method based on reinforcement learning. Background Art

[0002] Aircraft with efficient aerodynamic performance and the ability to quickly change shape to gain maneuverability are an important development trend for future new fighters. The intelligent adaptive variant strategy and adaptive control method of variant aircraft are important measures to improve the maneuverability of new variant aircraft. Although research on variant aircraft has made some progress in recent years, it is still limited by the development of related technologies such as aerodynamic characteristics, variant characteristics and maneuverability requirements.

[0003] There are many methods for studying variant control mechanisms. The variant structure can be used as system input, the variant aircraft model input can be expanded, and the traditional control method can be used directly for control. At the same time, the characteristics of the variant aircraft under different configurations can be analyzed separately, and then control laws with different parameters can be designed for different variant configurations. However, no matter which of the above control methods is used, the variant aircraft is controlled as a conventional aircraft, and the maximum performance of the variant aircraft cannot be guaranteed.

[0004] As one of the most popular control methods, reinforcement learning has made breakthroughs in many fields after the improvement of computer computing power. Reinforcement learning is a method of continuous trial and error. Most of them determine the gradient of the output of the variant controller by the size of the current reward and punishment, and ultimately ensure that the system gets the maximum reward. Therefore, as long as the reward and punishment function is designed well, after a large number of sample training, it can ensure that the variant control mechanism achieves the desired effect. Summary of the invention

[0005] The technical problems to be solved by the present invention are:

[0006] In order to avoid the shortcomings of the prior art, the present invention provides a variant aircraft level flight deceleration maneuver decision method based on reinforcement learning, which is used to solve the problem of the optimal variant configuration of the variant aircraft in performing maneuvering tasks. For the level flight deceleration maneuver, the variant strategy method is studied through reinforcement learning to ensure that the aircraft can achieve the optimal aerodynamic force, flight efficiency and control efficiency when performing the level flight deceleration maneuver.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A method for decision-making of a variant aircraft level flight deceleration maneuver based on reinforcement learning, characterized by comprising:

[0009] S1. Establish nonlinear model, aerodynamic force and aerodynamic moment model of variant aircraft;

[0010] S2, designing an overall control structure of a variant aircraft for the variant aircraft model of the nonlinear model, aerodynamic force and aerodynamic moment model established in step S1, including a speed control method, an altitude control method and an attitude control method;

[0011] S3, based on step S2, establish an environment for the level flight deceleration maneuver task, so as to obtain the state information of the variant aircraft and the reward and punishment information for the level flight deceleration maneuver in reinforcement learning;

[0012] S4. Design the reward and penalty function for the level flight deceleration maneuver task;

[0013] S5. Design a reinforcement learning agent based on the DQN algorithm and train the agent to obtain the final variant decision algorithm.

[0014] A further technical solution of the present invention is as follows: In step S1, a nonlinear model, aerodynamic force and aerodynamic moment model of the variant aircraft are established, specifically:

[0015] (a) Establishment of kinematic and dynamic models of variant aircraft

[0016] The force equations for the morphing aircraft are:

[0017]

[0018] The above formula can be used to obtain the kinetic model:

[0019]

[0020] Where V is the aircraft speed, α is the aircraft angle of attack, β is the aircraft sideslip angle, T is the engine thrust, [pqr] T are the three-axis angular velocity, [DYL] T are drag, side force and lift respectively, [F Ix F Iy F Iz ] T is the component of the inertial force caused by the deformation process in the airflow coordinate system,

[0021]

[0022] Among them, S x is the component of the aircraft static moment on the x-axis of the aircraft system;

[0023] The angular velocity equation of the variant aircraft:

[0024]

[0025] where [φ θ ψ] T is the aircraft attitude angle, are rolling moment, pitching moment and yaw moment respectively, [M Ix M Iy M Iz ] T is the component of the inertia moment caused by the deformation process in the body coordinate axis system, g is the gravitational acceleration,

[0026] The attitude angle equation and navigation equation of the variant aircraft are the same as those of the ordinary aircraft, and are expressed as follows:

[0027]

[0028] (b) Establishment of the aerodynamic force and aerodynamic moment model of the variant aircraft

[0029] The aerodynamic force and aerodynamic moment are expressed as:

[0030] Lift:

[0031] L=C L QS w

[0032] resistance:

[0033] D=C D QS w

[0034] Side Force:

[0035] Y=C Y QS w

[0036] Rolling moment:

[0037]

[0038] Pitch moment:

[0039] M A =C m QS w c A

[0040] Yaw moment:

[0041] N A =C n QS w b

[0042] Among them, C L , C D and C Y are lift coefficient, drag coefficient and side force coefficient respectively; C l , C m and C nThey are the rolling moment coefficient, the pitching moment coefficient and the yaw moment coefficient respectively; is the dynamic pressure, ρ is the air density;

[0043] Let the left and right elevator deflection angles be δ el and δ er , downward deflection is positive, the pitching moment M A is negative, that is, a nose-down moment is generated. At the same time, due to the V-tail configuration, the left and right elevators simultaneously generate yaw moments. The yaw moment generated when the right yaw is negative is negative. Therefore, when the left elevator is negative and the right elevator is positive, a negative yaw moment N is generated. A ; Let the left and right aileron deflection angles be δ al and δ ar , downward deflection is positive, the left aileron is negatively deflected, the right aileron is positively deflected, and the rolling moment generated is is negative; in addition, when the sweep angle Λ is 60°, it is considered that the force and moment generated by the sweep angle change to 0, and the influence of the sweep angle on the aerodynamic force and aerodynamic moment can be calculated;

[0044] The aerodynamic force coefficient and aerodynamic moment coefficient are expressed as:

[0045] Lift coefficient:

[0046]

[0047] Drag coefficient:

[0048]

[0049] Side force coefficient:

[0050]

[0051] Rolling moment coefficient:

[0052]

[0053] Pitch moment coefficient:

[0054]

[0055] Yaw moment coefficient:

[0056]

[0057] Among them, C L0 is the lift coefficient at zero angle of attack, C Lα is the slope of the lift line; C D0 is the drag coefficient at zero angle of attack, C Dα , C Dβ are the derivatives of the drag coefficient with respect to the angle of attack and the sideslip angle respectively; C m0 is the pitching moment coefficient at zero angle of attack, Cmα is the derivative of the pitch moment coefficient with respect to the angle of attack, C mq is the derivative of the pitch moment coefficient with respect to the pitch angular velocity; C Yβ , C lβ , C nδ are the derivatives of the side force coefficient, rolling moment coefficient, and yaw moment coefficient to the sideslip angle; C lp , C np are the derivatives of the rolling moment coefficient and the yaw moment coefficient to the rolling angular velocity respectively; C lr , C nr are the derivatives of the rolling moment coefficient and the yaw moment coefficient to the yaw angular velocity respectively; C Lδe , C Dδe , C Yδe , C lδe , C mδe , C nδe is the elevator control derivative; C Lδa , C Dδa , C Yδa , C lδa , C mδa , C nδa is the control derivative of the aileron; C LΛ Λ、C DΛ Λ、C YΛ Λ、C lΛ Λ、C mΛ Λ、C nΛ Λ is the change of aerodynamic force coefficient and aerodynamic moment coefficient caused by the wing sweep angle and tail dihedral angle.

[0058] A further technical solution of the present invention is as follows: in step S2, the overall control structure of the variant aircraft is designed for the variant aircraft model established in step S1, specifically:

[0059] First, PI control is used for speed, and the control law is as follows:

[0060] T=k p,V (V ref -V)+k i,V ∫(V ref -V)dt

[0061] Among them, V ref is the speed command, k p,V is the proportionality coefficient, k i,V is the integration coefficient;

[0062] The altitude control law consists of two parts, altitude to pitch angle, and pitch angle to elevator. The control from pitch angle to elevator belongs to attitude control. First, the PID control law from altitude to pitch angle is given in the following form:

[0063]

[0064] Among them, θ ref is the pitch angle command, h ref is the height instruction, k p,h , k i,h , k d,h are proportional, integral and differential coefficients respectively;

[0065] Attitude control and sideslip angle control are used in the lateral heading, and the attitude control uses the backstepping method as follows:

[0066]

[0067] The sideslip angle control uses PI control. When controlling the sideslip angle, the yaw rate command in the attitude control is replaced by the sideslip angle, as follows:

[0068] r ref =k p,β (β ref -β)+k i,β ∫(β ref -β)dt

[0069] Among them, β ref is the sideslip angle command, k p,β , k i,β are the proportional coefficient and the integral coefficient respectively.

[0070] A further technical solution of the present invention is as follows: In step S3, an environment for a level flight deceleration maneuvering task is established on the basis of step S2, so as to obtain the state information of the variant aircraft and the reward and punishment information for the level flight deceleration maneuvering in reinforcement learning, specifically:

[0071] Use level flight deceleration maneuvers to train a variant structure agent;

[0072] By setting corresponding rewards and penalties, the agent is trained to converge in the direction of maximum reward, and finally the desired control of the variant mechanism in the level flight deceleration maneuver is obtained; in order to ensure that the application range of reinforcement learning control is wider, it is necessary to add some randomness to each training task, including the randomness of the instructions and the randomness of the initial value;

[0073] The observed state of the environment has 9 quantities, namely [V, α, β, φ, θ, ψ, p, q, r] T These 9 state quantities; and not only the current state needs to be obtained, but also the state at the next moment needs to be calculated; for these 9 state quantities, a variant aircraft nonlinear model is used, and the Runge-Kutta algorithm is used to update and calculate the state of the next shot;

[0074] The input of the environment, i.e. the output of the agent, is the three angle quantities of the variant structure, namely the left sweep angle Λ l , right sweep angle Λ r and the tail wing's dihedral angle Λ d ; Before each simulation starts, set an initialization operation, that is, randomize the speed command signal in the speed control loop;

[0075] In the reinforcement learning training of the level flight deceleration maneuver, the current reward and punishment r is calculated according to the formula of the reward and punishment function. t , and then by r t The return u can be calculated t size; current return u t Represents the sum of rewards at time t and all subsequent rounds; in order to emphasize that future rewards are not as important as current rewards, the discounted return can be defined by the discount rate γ; if the return is larger, it means that the aircraft can track the command signal faster; conversely, the smaller the return, it means that the aircraft tracks the command signal slower; in addition, a certain amount of randomness will be added to the final control output to ensure that the strategy does not fall into a local optimum;

[0076] The entire training process is as follows: at the beginning of the simulation, the aircraft receives a random speed instruction and performs a corresponding deceleration maneuver. During this process, the potential energy of the aircraft does not change; then a single-step update is performed through the time penalty of each step. When the deceleration maneuver task is finally completed, there will be a large reward. At this time, there is a large reward but it is not updated; in this way, the dynamic performance of the aircraft in performing the deceleration maneuver task can be indirectly obtained, and the policy network Actor and the value network Critic are updated accordingly; the update of the policy network will change the output of the aircraft variant structure in real time. If the reward is larger, the network will update more in this direction, and vice versa. After a large number of rounds of training, it can be ensured that the policy network can control the variant structure well according to the current state.

[0077] A further technical solution of the present invention is: in step S4, a reward and penalty function for the level flight deceleration maneuvering task is designed, specifically:

[0078] r t =r s (t)+r d (t)+r tp (t)

[0079]

[0080] r tp (t)=-1

[0081] Among them, r d (t) represents dynamic performance rewards and penalties, r s(t) represents steady-state performance rewards and penalties, r tp (t) represents the time penalty, c1 and c2 are positive constants.

[0082] A further technical solution of the present invention is: in step S5, a reinforcement learning agent based on the DQN algorithm is designed, and the agent is trained to obtain the final variant decision algorithm, specifically:

[0083] In the network structure of DQN, the input layer observation has 9 nodes, which are the 9 state quantities of the aircraft [V, α, β, φ, θ, ψ, p, q, r] T ; Then it passes through a fully connected layer, a relu activation function, a fully connected layer, a relu activation function, and then to the output layer;

[0084] Since the DQN algorithm can only target discrete action spaces, and the variant aircraft variant output uses a continuous action space, it is necessary to discretize the possible actions, that is, to discretize the left and right sweep angles and the V-tail dihedral angle;

[0085] In addition, DQN, as an off-policy method, designs a buffer to store past experience:

[0086] D←D∪(s t ,a t ,r t ,s t+1 )

[0087] Each time when training, a group of data packets (s t ,a t ,r t ,s t+1 ), and then start to update the DQN network; the update of the DQN network parameters is:

[0088]

[0089] Among them, α represents the learning rate, Represents the optimization function J Q The gradient of (θ);

[0090] The TD algorithm is used to update the network, so

[0091]

[0092] The TD error updated by the DQN network is:

[0093]

[0094] Among them, Q(s t ,at θ t ) represents the neural network structure.

[0095] A computer system, characterized in that it comprises: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0096] A computer-readable storage medium, characterized in that it stores computer-executable instructions, which are used to implement the above method when executed.

[0097] A computer program product, characterized by comprising computer executable instructions, wherein the instructions are used to implement the above method when executed.

[0098] The beneficial effects of the present invention are:

[0099] The present invention provides a variant aircraft level flight deceleration maneuver decision method based on reinforcement learning, designs a new variant wing strategy and variant layout to improve the stability of the variant aircraft and enhance its maneuverability and maneuverability. The present invention uses a reinforcement learning control method to ensure that the aircraft can achieve optimal aerodynamic force, flight efficiency and control efficiency when performing specific maneuvers. In addition, the asymmetric variable sweep wing structure can further increase the aircraft's lateral maneuverability, and has advantages such as a smaller turning radius; the tail with a variable dihedral angle can take into account the aircraft's longitudinal and lateral stability. For the level flight deceleration maneuver, a variant control method is studied. Through the DQN algorithm of reinforcement learning, the problem of the optimal variant configuration of the variant aircraft in performing the level flight deceleration maneuver task is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0101] Figure 1 Design a framework diagram for the reinforcement learning-based variant aircraft deceleration maneuver decision algorithm.

[0102] Figure 2 Graph of variant aircraft objects for reinforcement learning applications.

[0103] Figure 3 This is the overall control structure diagram for level flight deceleration maneuvers.

[0104] Figure 4 This is the structural diagram of the reward and punishment function.

[0105] Figure 5 This is the environmental design diagram for level flight deceleration maneuvers.

[0106] Figure 6 This is the DQN network structure diagram.

[0107] Figure 7 This is the reward and punishment chart for each round during training.

[0108] Figure 8 This is the output diagram of the reinforcement learning agent variant structure during the level flight deceleration maneuver.

[0109] Fig. 9 This is the speed change diagram.

[0110] Fig.10 This is the attitude angle change diagram. DETAILED DESCRIPTION

[0111] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0112] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0113] In the process of reinforcement learning, it is necessary to magnify the influence of the variant structure on the flight performance as much as possible, which requires ensuring the stable flight of the aircraft; when the aircraft is in an unstable flight state, the change in the flight state may easily make it difficult for reinforcement learning to converge. Therefore, in the content of the present invention, the aircraft will be controlled to fly at a fixed altitude to ensure that the gravitational potential energy of the aircraft remains unchanged, prevent the mutual conversion of potential energy and kinetic energy from affecting the convergence of reinforcement learning, and make reinforcement learning only learn and update the variant strategy of the aircraft.

[0114] Since the altitude needs to be kept constant during the deceleration maneuver, and the attitude needs to be kept constant as much as possible, it is necessary to design the speed control law, altitude control law and attitude control law respectively, and then design the environment and reward and punishment function of the level flight deceleration maneuver. Finally, a DQN-based reinforcement learning agent is designed to learn the variant decisions of the deformable mechanism (wing sweep angle, V-tail dihedral angle) of the variant aircraft during the deceleration maneuver, and obtain the optimal variant configuration for the maneuver. Finally, a simulation experiment is carried out on the variant aircraft level flight deceleration maneuver decision algorithm based on reinforcement learning.

[0115] like Figure 1 As shown, the present invention provides a variant aircraft level flight deceleration maneuver decision method based on reinforcement learning, comprising the following steps:

[0116] (1) Establish nonlinear models, aerodynamic forces, and aerodynamic moment models of the morphing aircraft;

[0117] (2) Design the speed control method, altitude control method and attitude control method of the variant aircraft;

[0118] (3) Establish an environment for the level flight deceleration task to obtain the state information of the variant aircraft and the reward and punishment information for the level flight deceleration maneuver in reinforcement learning;

[0119] (4) Design the reward and penalty function for the level flight deceleration maneuver task;

[0120] (5) Design a reinforcement learning agent based on the DQN algorithm and train the agent to obtain the final variant decision algorithm.

[0121] In order to enable those skilled in the art to better understand the present invention, the present invention is described in detail below in conjunction with specific embodiments.

[0122] Embodiment 1:

[0123] The object of reinforcement learning application in this embodiment is Figure 2 The variant aircraft shown has a variable structure of left and right wing sweep angles and V-tail dihedral angle, and the sweep angles of the left and right wings can be changed asymmetrically.

[0124] (a) Establishment of kinematic and dynamic models of variant aircraft

[0125] The force equations for the morphing aircraft are:

[0126]

[0127] The above formula can be obtained:

[0128]

[0129] Where V is the aircraft speed, α is the aircraft angle of attack, β is the aircraft sideslip angle, T is the engine thrust, [pqr] T are the three-axis angular velocity, [DYL] T are drag, side force and lift respectively, [F Ix F Iy F Iz ] T is the component of the inertial force caused by the deformation process in the airflow coordinate system,

[0130]

[0131] Among them, S x is the component of the aircraft static moment on the x-axis of the aircraft system.

[0132] The angular velocity equation of the variant aircraft:

[0133]

[0134] where [φ θ ψ] T is the aircraft attitude angle, are rolling moment, pitching moment and yaw moment respectively, [M Ix M Iy M Iz ] T is the component of the inertia moment caused by the deformation process in the body coordinate axis system, g is the gravitational acceleration,

[0135] The attitude angle equation and navigation equation of the variant aircraft are the same as those of the ordinary aircraft and are expressed as follows:

[0136]

[0137] (b) Establishment of the aerodynamic force and aerodynamic moment model of the variant aircraft

[0138] The aerodynamic force and aerodynamic torque can be expressed as:

[0139] Lift:

[0140] L=C L QS w

[0141] resistance:

[0142] D=C D QS w

[0143] Side Force:

[0144] Y=C Y QS w

[0145] Rolling moment:

[0146]

[0147] Pitch moment:

[0148] M A =C m QS w c A

[0149] Yaw moment:

[0150] N A =C n QS w b

[0151] Among them, C L , C D and C Y are lift coefficient, drag coefficient and side force coefficient respectively; C l , C m and C n They are the rolling moment coefficient, the pitching moment coefficient and the yaw moment coefficient respectively; is the dynamic pressure and ρ is the air density.

[0152] Let the left and right elevator deflection angles be δ el and δ er , downward deflection is positive, the pitching moment M A is negative, that is, a nose-down moment is generated. At the same time, due to the V-tail configuration, the left and right elevators simultaneously generate yaw moments. The yaw moment generated when the right yaw is negative is negative. Therefore, when the left elevator is negative and the right elevator is positive, a negative yaw moment N is generated. A Let the left and right aileron deflection angles be δ al and δ ar , downward deflection is positive, the left aileron is negatively deflected, the right aileron is positively deflected, and the resulting rolling moment L A In addition, when the sweep angle Λ is 60°, the force and moment generated by the sweep angle are considered to be 0, and the influence of the sweep angle on the aerodynamic force and aerodynamic moment can be calculated.

[0153] The aerodynamic force coefficient and aerodynamic moment coefficient can be expressed as:

[0154] Lift coefficient:

[0155]

[0156] Drag coefficient:

[0157]

[0158] Side force coefficient:

[0159]

[0160] Rolling moment coefficient:

[0161]

[0162] Pitch moment coefficient:

[0163]

[0164] Yaw moment coefficient:

[0165]

[0166] Among them, C L0 is the lift coefficient at zero angle of attack, C Lα is the slope of the lift line; C D0 is the drag coefficient at zero angle of attack, C Dα , C Dβ are the derivatives of the drag coefficient with respect to the angle of attack and the sideslip angle respectively; C m0 is the pitching moment coefficient at zero angle of attack, C mα is the derivative of the pitch moment coefficient with respect to the angle of attack, C mq is the derivative of the pitch moment coefficient with respect to the pitch angular velocity; C Yβ , C lβ , C nβ are the derivatives of the side force coefficient, rolling moment coefficient, and yaw moment coefficient to the sideslip angle; C lp , C np are the derivatives of the rolling moment coefficient and the yaw moment coefficient to the rolling angular velocity; C lr , C nr are the derivatives of the rolling moment coefficient and the yaw moment coefficient to the yaw angular velocity respectively; C Lδe , C Dδe , C Yδe , C lδe , C mδe , C nδe is the elevator control derivative; C Lδa , C Dδa , C Yδa , C lδa , C mδa , C nδa is the control derivative of the aileron; C LΛ Λ、C DΛ Λ、C YΛ Λ、C lΛ Λ、C mΛ Λ、C nΛ Λ is the change of aerodynamic force coefficient and aerodynamic moment coefficient caused by the wing sweep angle and tail dihedral angle.

[0167] (c) Control structure for level flight deceleration maneuvers

[0168] The main purpose of this invention is to study the performance of the variant aircraft in level flight deceleration maneuvers under different configurations. Therefore, in the reinforcement learning process, it is necessary to maximize the impact of the variant structure on the flight performance, which requires ensuring the aircraft's stable flight. When the aircraft is in an unstable flight, changes in the flight state may easily make it difficult for reinforcement learning to converge. The overall control structure is as follows Figure 3 As shown, the content of the present invention will control the aircraft to fly at a fixed altitude, ensure that the gravitational potential energy of the aircraft remains unchanged, and prevent the mutual conversion of potential energy and kinetic energy from affecting the convergence of reinforcement learning. In order to achieve a stable flight of the aircraft, it is necessary to ensure that the aircraft has good control performance. In the longitudinal direction, the aircraft is controlled to maintain a fixed altitude flight to ensure that the gravitational potential energy of the aircraft does not change much; in the lateral direction, the aircraft is initially guaranteed to fly without sideslip and roll. However, in the case of stable flight only, the performance changes reflected by the variant structure cannot be fully reflected. Therefore, in the reinforcement learning training, it is necessary to let the aircraft perform a level flight deceleration maneuver, and then obtain the optimal control of the variant structure under this maneuver.

[0169] First, PI control is used for speed, and the control law is as follows:

[0170] T=k p,V (V ref -V)+k i,V ∫(V ref -V)dt

[0171] Among them, V ref is the speed command, k p,V is the proportionality coefficient, k i,V is the integration coefficient.

[0172] The altitude control law consists of two parts, altitude to pitch angle, and pitch angle to elevator. The control from pitch angle to elevator belongs to attitude control. Here we first give the PID control law from altitude to pitch angle, which is as follows:

[0173]

[0174] Among them, θ ref is the pitch angle command, h ref is the height instruction, k p,h , k i,h , k d,h are the proportional, integral, and differential coefficients respectively.

[0175] Attitude control and sideslip angle control are used in the lateral heading, and the attitude control uses the backstepping method as follows:

[0176]

[0177] The sideslip angle control uses PI control. When controlling the sideslip angle, the yaw rate command in the attitude control is replaced by the sideslip angle, as follows:

[0178] r ref =k p,β (β ref -β)+k i,β ∫(β ref -β)dt.

[0179] Among them, β ref is the sideslip angle command, k p,β , k i,β are the proportional coefficient and the integral coefficient respectively.

[0180] (d) Design of reward and penalty functions for level flight deceleration maneuvers

[0181] The main purpose of the reinforcement learning algorithm is to enable the intelligent agent to obtain the maximum cumulative reward, and the rewards and penalties at each step will guide the convergence of the action network. Generally speaking, there is no big rule in the design of the reward and punishment function, and it is almost always designed and adjusted based on experience. For some complex behaviors, reinforcement learning may lead to some "strange consequences" due to the designed reward and punishment function. These consequences maximize the benefits as much as possible, but deviate from our goal. Therefore, when designing the reward and punishment function, we will adjust the parameters or change its structure according to the results.

[0182] In the present invention, the design process of the reward and penalty function for the level flight deceleration maneuver is as follows. The structural block diagram of the reward and penalty function is as follows: Figure 4 shown.

[0183] For the control task of acceleration and deceleration, we expect it to be able to track the command speed quickly and accurately. According to the previous content, the speed control already has the corresponding control law, that is, by controlling the thrust, the speed command is tracked. This part uses the control of the variant structure to change the aerodynamic parameters of the fuselage to make the original control more efficient. Therefore, the design standard of the reward and punishment function is the same as the traditional control, including two parts: dynamic performance reward and punishment and steady-state performance reward and punishment.

[0184] For dynamic performance, we mainly hope that the speed can reach the instruction as soon as possible. Therefore, we can give a "-1" penalty to each step, and then give a positive reward when the speed is close to the instruction.

[0185] For steady-state performance, we hope that the speed can stabilize near the target as soon as possible and remain stable for a period of time. When the conditions are met, a positive reward is given and the round ends.

[0186] Therefore, according to the above requirements, the reward and punishment function can be expressed as follows:

[0187] r(t)=r s (t)+r d (t)+r tp (t)

[0188]

[0189] r tp (t)=-1

[0190] Among them, r d (t) represents dynamic performance rewards and penalties, r s (t) represents steady-state performance rewards and penalties, r tp (t) represents the time penalty, c1 and c2 are positive constants.

[0191] (e) Level flight deceleration maneuver environment design

[0192] like Figure 5 As shown, in the present invention, the variant structure intelligent agent is trained using the level flight deceleration maneuver. By setting the corresponding reward and punishment, the intelligent agent is trained to converge in the direction of the maximum reward, and finally the desired variant mechanism is controlled in the level flight deceleration maneuver. During the training process, it is necessary to ensure that the vertical altitude flight is still maintained. In order to ensure a wider range of applications for reinforcement learning control, it is necessary to add some randomness to each training task, including the randomness of the instructions and the randomness of the initial value.

[0193] The observed state of the environment mainly consists of 9 quantities, namely V, α, β, φ, θ, ψ, p, q, and r. And not only the state at the current moment needs to be obtained, but also the state at the next moment needs to be calculated. For these 9 state quantities, the variant aircraft kinematic model is used, and the Runge-Kutta algorithm is used to update and calculate the state of the next shot. The input of the environment, that is, the output of the intelligent agent, is the 3 angle quantities of the variant structure, namely the left sweep angle Λ l , right sweep angle Λ r and the tail wing's dihedral angle Λ d Finally, before each simulation starts, an initialization operation is set, that is, in the speed control loop, the speed command signal is randomized, so that the training results will adapt to different speed targets, thereby making the level flight deceleration task tend to be optimal.

[0194] In the reinforcement learning training of level flight deceleration, the current reward and punishment r is calculated according to the formula of the reward and punishment function. t , and then by r t The return u can be calculated t Size. Current return u tRepresents the sum of rewards at time t and in all subsequent rounds. In order to emphasize that future rewards are not as important as current rewards, the discounted return can be defined by the discount rate γ. If the return is larger, it means that the speed of the aircraft can track the command signal faster; conversely, the smaller the return, the slower the aircraft tracks the command signal. Therefore, the training goal of reinforcement learning can be understood as maximizing the return. As a random variable, the return is related to the behavior and current state at the current moment. In order to evaluate the quality of the current behavior, the action value can be defined as the expectation of the return on the current behavior. The larger the action value, the more effective the control strategy of the current variant structure can be to improve the performance of the aircraft. The action value can help update the policy network of the variant structure and ultimately obtain a satisfactory control strategy.

[0195] The task of the level flight deceleration maneuver is to find a variant strategy that makes the aircraft's deceleration maneuver better and faster. Therefore, in the initialization of the simulation, the speed command will be randomly processed to ensure that the data during training is the data of the aircraft under different degrees of deceleration. In order to highlight the impact of the variant structure on the flight state, the conversion between potential energy and kinetic energy of the aircraft due to altitude changes can be eliminated by ensuring constant altitude flight, which interferes with the impact of the variant structure on the aircraft's deceleration. In addition, a certain amount of randomness will be added to the final control output to ensure that the strategy will not fall into a local optimum.

[0196] The entire training process is as follows: At the beginning of the simulation, the aircraft receives a random speed instruction and performs a corresponding deceleration maneuver. During this process, the potential energy of the aircraft does not change. Then, a single-step update is performed through the time penalty of each step. When the deceleration maneuver task is finally completed, there will be a large reward. At this time, there is a large reward but it is not updated. In this way, the dynamic performance of the aircraft in performing the deceleration maneuver task can be indirectly obtained, and the policy network Actor and the value network Critic are updated accordingly. The update of the policy network will change the output of the aircraft variant structure in real time. If the reward is larger, the network will update more in this direction, and vice versa. After a large number of rounds of training, it can be ensured that the policy network can control the variant structure well according to the current state.

[0197] (f) Design of DQN-based reinforcement learning agent

[0198] The network structure of DQN is as follows Figure 6 In the DQN algorithm, there is only one deep neural network, namely the optimal action value network. When the DQN algorithm is actually applied, in order to prevent the adverse effects of bootstrapping, a target network with the same value network structure is also designed.

[0199] In the network structure of the DQN of the present invention, the input layer observation has 9 nodes, which are the 9 state quantities of the aircraft V, α, β, φ, θ, ψ, p, q, r, and then pass through a fully connected layer with 882 nodes, and after passing through a layer of relu activation function, pass through another fully connected layer with 882 nodes, and then pass through a layer of relu activation function to the output layer with 294 nodes. Since the DQN algorithm can only target discrete action spaces, and the variant aircraft variant output uses a continuous action space, it is necessary to discretize the possible actions. Considering the problem of computing power, the left and right sweep angles and the V tail dihedral angle are discretized every 6°, as shown below:

[0200] Λ l =24:6:60

[0201] Λ r =24:6:60

[0202] Λ d =30:6:60

[0203] The code is as follows:

[0204]

[0205] Therefore, the number of action spaces finally obtained by this discrete method is:

[0206] N action =7*7*6=294

[0207] This is why the output layer nodes in this DQN have 294. In addition, as an off-policy method, DQN can design a buffer to store past experience.

[0208] D←D∪(s t ,a t ,r t ,s t+1 )

[0209] Each time when training, a group of data packets (s t ,a t ,r t ,s t+1 ), and then start to update the DQN network. The update of the DQN network parameters is:

[0210]

[0211] Among them, α represents the learning rate, Represents the optimization function J Q The gradient of (θ).

[0212] Here, the TD algorithm is used to update the network, so

[0213]

[0214] The TD error updated by the DQN network is:

[0215]

[0216] Among them, Q(s t ,a t θ t ) represents the neural network structure.

[0217] The designed variant aircraft deceleration maneuver decision algorithm based on reinforcement learning is simulated and verified as follows:

[0218] The initial condition of the aircraft is steady level flight, with an altitude of 1000m, a speed of 60m / s and an angle of attack of 4.23°.

[0219] First, we give a graph of the single-step reward and penalty for 1000 trainings and the average reward and penalty for 5 steps, as shown in Figure 7 As shown. Figure 7 It can be seen from the figure that the round reward and punishment gradually converge to a larger value after training. When the training continues, the effect is not very good and begins to deteriorate. Finally, when the number of training times is adjusted to 1000, the performance of DQN is better. The following will use the trained agent for simulation verification. The simulation results are as follows Figure 8 shown.

[0220] Use the trained agent to simulate and get the speed and posture responses, such as Fig. 9 and Fig.10 shown.

[0221] According to the simulation results of level flight deceleration, it can be seen that when the aircraft decelerates, the external thrust is 0. After training, the reinforcement learning method makes the aircraft do a rolling maneuver through the asymmetric swept wings. Then, the attitude control of the aircraft makes the aircraft roll back to the center through the control surfaces. At the same time, these control surfaces will also increase the aircraft's resistance, thereby causing the aircraft to decelerate significantly. Specifically, in 10 seconds, the aircraft's speed decreases from 60m / s to 43.3m / s.

[0222] Based on the above simulation results, the effectiveness of the variant aircraft level flight deceleration maneuver decision algorithm designed by the present invention based on reinforcement learning is proved, and the variant strategy of the variant structure can be successfully trained and optimized. Finally, it can ensure that the variant aircraft can achieve a rapid reduction in flight speed only through changes in the variant structure while the altitude and attitude remain basically unchanged.

[0223] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. A variant aircraft level flight deceleration maneuver decision method based on reinforcement learning, characterized in that: include: S1. Establish nonlinear model, aerodynamic force and aerodynamic moment model of variant aircraft; S2, designing an overall control structure of a variant aircraft for the variant aircraft model of the nonlinear model, aerodynamic force and aerodynamic moment model established in step S1, including a speed control method, an altitude control method and an attitude control method; S3, based on step S2, establish an environment for the level flight deceleration maneuver task, so as to obtain the state information of the variant aircraft and the reward and punishment information for the level flight deceleration maneuver in reinforcement learning; S4. Design the reward and penalty function for the level flight deceleration maneuver task; S5. Design a reinforcement learning agent based on the DQN algorithm and train the agent to obtain the final variant decision algorithm.

2. The method for decision-making of a variant aircraft level flight deceleration maneuver based on reinforcement learning according to claim 1, characterized in that: In step S1, a nonlinear model, aerodynamic force and aerodynamic moment model of the variant aircraft are established, specifically: (a) Establishment of kinematic and dynamic models of variant aircraft The force equations for the morphing aircraft are: The above formula can be used to obtain the kinetic model: Where V is the aircraft speed, α is the aircraft angle of attack, β is the aircraft sideslip angle, T is the engine thrust, [pqr] T are the three-axis angular velocity, [DYL] T are drag, side force and lift respectively, [F Ix F Iy F Iz ] T is the component of the inertial force caused by the deformation process in the airflow coordinate system, Among them, S x is the component of the aircraft static moment on the x-axis of the aircraft system; The angular velocity equation of the variant aircraft: where [φ θ ψ] T is the aircraft attitude angle, are rolling moment, pitching moment and yaw moment respectively, [M Ix M Iy M Iz ] T is the component of the inertia moment caused by the deformation process in the body coordinate axis system, g is the gravitational acceleration, The attitude angle equation and navigation equation of the variant aircraft are the same as those of the ordinary aircraft, and are expressed as follows: (b) Establishment of the aerodynamic force and aerodynamic moment model of the variant aircraft The aerodynamic force and aerodynamic moment are expressed as: Lift: L=C L QS w resistance: D=C D QS w Side Force: Y=C Y QS w Rolling moment: Pitch moment: M A =C m QS w c A Yaw moment: N A =C n QS w b Among them, C L , C D and C Y are lift coefficient, drag coefficient and side force coefficient respectively; C l , C m and C n They are the rolling moment coefficient, the pitching moment coefficient and the yaw moment coefficient respectively; is the dynamic pressure, ρ is the air density; Let the left and right elevator deflection angles be δ el and δ er , downward deflection is positive, the pitching moment M A is negative, that is, a nose-down moment is generated. At the same time, due to the V-tail configuration, the left and right elevators simultaneously generate yaw moments. The yaw moment generated when the right yaw is negative is negative. Therefore, when the left elevator is negative and the right elevator is positive, a negative yaw moment N is generated. A ; Let the left and right aileron deflection angles be δ al and δ ar , downward deflection is positive, the left aileron is negatively deflected, the right aileron is positively deflected, and the resulting rolling moment L A is negative; in addition, when the sweep angle Λ is 60°, it is considered that the force and moment generated by the sweep angle change to 0, and the influence of the sweep angle on the aerodynamic force and aerodynamic moment can be calculated; The aerodynamic force coefficient and aerodynamic moment coefficient are expressed as: Lift coefficient: Drag coefficient: Side force coefficient: Rolling moment coefficient: Pitch moment coefficient: Yaw moment coefficient: Among them, C L0 is the lift coefficient at zero angle of attack, C Lα is the slope of the lift line; C D0 is the drag coefficient at zero angle of attack, C Dα , C Dβ are the derivatives of the drag coefficient with respect to the angle of attack and the sideslip angle respectively; C m0 is the pitching moment coefficient at zero angle of attack, C mα is the derivative of the pitch moment coefficient with respect to the angle of attack, C mq is the derivative of the pitch moment coefficient with respect to the pitch angular velocity; C Yβ , C lβ , C nβ are the derivatives of the side force coefficient, rolling moment coefficient, and yaw moment coefficient to the sideslip angle; C lp , C np are the derivatives of the rolling moment coefficient and the yaw moment coefficient to the rolling angular velocity respectively; C lr , C nr are the derivatives of the rolling moment coefficient and the yaw moment coefficient to the yaw angular velocity respectively; C Lδe , C Dδe , C Yδe , C lδe , C mδe , C nδe is the elevator control derivative; C Lδa , C Dδa , C Yδa , C lδa , C mδa , C nδa is the control derivative of the aileron; C LΛ Λ、C DΛ Λ、C YΛ Λ、C lΛ Λ、C mΛ Λ、C nΛ Λ is the change of aerodynamic force coefficient and aerodynamic moment coefficient caused by the wing sweep angle and tail dihedral angle.

3. The method for decision-making of a variant aircraft level flight deceleration maneuver based on reinforcement learning according to claim 2, characterized in that: In step S2, the overall control structure of the variant aircraft is designed for the variant aircraft model established in step S1, specifically: First, PI control is used for speed, and the control law is as follows: T=k p,V (V ref -V)+k i,V ∫(V ref -V)dt Among them, V ref is the speed command, k p,V is the proportionality coefficient, k i,V is the integration coefficient; The altitude control law consists of two parts, altitude to pitch angle, and pitch angle to elevator. The control from pitch angle to elevator belongs to attitude control. First, the PID control law from altitude to pitch angle is given in the following form: Among them, θ ref is the pitch angle command, h ref is the height instruction, k p,h , k i,h , k d,h are proportional, integral and differential coefficients respectively; Attitude control and sideslip angle control are used in the lateral heading, and the attitude control uses the backstepping method as follows: The sideslip angle control uses PI control. When controlling the sideslip angle, the yaw rate command in the attitude control is replaced by the sideslip angle, as follows: r ref =k p,β (β ref -β)+k i,β ∫(β ref -β)dt Among them, β ref is the sideslip angle command, k p,β , k i,β are the proportional coefficient and the integral coefficient respectively.

4. The method for decision-making of a variant aircraft level flight deceleration maneuver based on reinforcement learning according to claim 3, characterized in that: In step S3, an environment for the level flight deceleration maneuver task is established on the basis of step S2, so as to obtain the state information of the variant aircraft and the reward and punishment information for the level flight deceleration maneuver in reinforcement learning, specifically: Use level flight deceleration maneuvers to train a variant structure agent; By setting corresponding rewards and penalties, the agent is trained to converge in the direction of maximum reward, and finally the desired control of the variant mechanism in the level flight deceleration maneuver is obtained; in order to ensure that the application range of reinforcement learning control is wider, it is necessary to add some randomness to each training task, including the randomness of the instructions and the randomness of the initial value; The observed state of the environment has 9 quantities, namely [V, α, β, φ, θ, ψ, p, q, r] T These 9 state quantities; and not only need to get the current state, but also need to calculate the state at the next moment; For these 9 state quantities, the variant aircraft nonlinear model is used, and the Runge-Kutta algorithm is used to update and calculate the state of the next shot; The input of the environment, i.e. the output of the agent, is the three angle quantities of the variant structure, namely the left sweep angle Λ l , right sweep angle Λ r and the tail wing's dihedral angle Λ d ; Before each simulation starts, set an initialization operation, that is, randomize the speed command signal in the speed control loop; In the reinforcement learning training of the level flight deceleration maneuver, the current reward and punishment r is calculated according to the formula of the reward and punishment function. t , and then by r t The return u can be calculated t size; current return u t Represents the sum of rewards at time t and all subsequent rounds; in order to emphasize that future rewards are not as important as current rewards, the discounted return can be defined by the discount rate γ; if the return is larger, it means that the aircraft can track the command signal faster; conversely, the smaller the return, it means that the aircraft tracks the command signal slower; in addition, a certain amount of randomness will be added to the final control output to ensure that the strategy does not fall into a local optimum; The entire training process is as follows: at the beginning of the simulation, the aircraft receives a random speed instruction and performs a corresponding deceleration maneuver. During this process, the potential energy of the aircraft does not change; then a single-step update is performed through the time penalty of each step. When the deceleration maneuver task is finally completed, there will be a large reward. At this time, there is a large reward but it is not updated; in this way, the dynamic performance of the aircraft in performing the deceleration maneuver task can be indirectly obtained, and the policy network Actor and the value network Critic are updated accordingly; the update of the policy network will change the output of the aircraft variant structure in real time. If the reward is larger, the network will update more in this direction, and vice versa. After a large number of rounds of training, it can be ensured that the policy network can control the variant structure well according to the current state.

5. The method for decision-making of a variant aircraft level flight deceleration maneuver based on reinforcement learning according to claim 4, characterized in that: In step S4, the reward and penalty function of the level flight deceleration maneuver task is designed, specifically: r t =r s (t)+r d (t)+r tp (t) r tp (t)＝-1 Among them, r d (t) represents dynamic performance rewards and penalties, r s (t) represents steady-state performance rewards and penalties, r tp (t) represents the time penalty, c1 and c2 are positive constants.

6. The method for decision-making of a variant aircraft level flight deceleration maneuver based on reinforcement learning according to claim 5, characterized in that: In step S5, a reinforcement learning agent based on the DQN algorithm is designed, and the agent is trained to obtain the final variant decision algorithm, which is specifically: In the network structure of DQN, the input layer observation has 9 nodes, which are the 9 state quantities of the aircraft [V, α, β, φ, θ, ψ, p, q, r] T ; Then it passes through a fully connected layer, a relu activation function, a fully connected layer, a relu activation function, and then to the output layer; Since the DQN algorithm can only target discrete action spaces, and the variant aircraft variant output uses a continuous action space, it is necessary to discretize the possible actions, that is, to discretize the left and right sweep angles and the V-tail dihedral angle; In addition, DQN, as an off-policy method, designs a buffer to store past experience: D←D∪(s t ,a t ,r t ,s t+1 ) Each time when training, a group of data packets (s t ,a t ,r t ,s t+1 ), and then start to update the DQN network; the update of the DQN network parameters is: Among them, α represents the learning rate, Represents the optimization function J Q The gradient of (θ); The TD algorithm is used to update the network, so The TD error updated by the DQN network is: Among them, Q(s t ,a t θ t ) represents the neural network structure.

7. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

8. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

9. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images