A hypersonic vehicle attitude control method based on robust adversarial reinforcement learning

Abstract

The present invention belongs to the technical field of reinforcement learning and aircraft attitude control, and specifically relates to a hypersonic aircraft attitude control method based on robust adversarial reinforcement learning, comprising the following steps: Step 1: Establish an aircraft attitude motion model to determine the control relationship between the control torque and the attitude angle; Step 2: Establish an optimization model for aircraft attitude control to determine the reward function that considers steady-state and dynamic performance in the attitude control process; Step 3: Establish an aircraft attitude control model, design an aircraft attitude control network and interference network with autonomous learning ability based on judgment-action, and realize offline training of the algorithm. The present invention can introduce an interference network during aircraft simulation training to simulate deviation interference to the greatest extent, improve control robustness, and realize multi-channel attitude decoupling-free reinforcement learning control of aircraft with strong robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of reinforcement learning and aircraft attitude control, and specifically relates to a hypersonic aircraft attitude control method based on robust adversarial reinforcement learning. Background Art

[0002] Hypersonic technology is a strategic commanding height in the field of aerospace technology in the 21st century. Its rapid strike and long-range delivery capabilities have attracted widespread attention from powerful countries around the world and have become the focus of current research. Due to the interference of the harsh external operating environment of hypersonic aircraft, it is very easy to cause the actual flight trajectory to suddenly deviate significantly from the predetermined reference trajectory, etc., which brings huge challenges to the safe operation of the aircraft. Therefore, ensuring the safe operation of the aircraft under the requirements of the mission under complex flight interference environment is the core issue of aircraft control. During the operation of the aircraft, due to the mutual coupling between the attitude channels and the uncertainty of the aerodynamic environment, the attitude control of the aircraft is greatly challenged. The traditional model-based attitude control method relies on accurate mathematical models and is very sensitive to model errors and external disturbances. The control method based on reinforcement learning cannot fully simulate the impact of disturbances on the environment. Therefore, how to improve the robustness of the aircraft to interference and enhance the generalization ability of the controller has become a difficult problem that needs to be solved in the intelligent control of aircraft.

[0003] Based on the above problems, the present invention conducts research on the attitude control method of hypersonic aircraft based on robust adversarial reinforcement learning. Considering the strong external interference problem caused by the complex operating environment of the aircraft, a robust controller design method for the aircraft based on control-disturbance reinforcement learning is proposed. By introducing an interference network to simulate the interference effect in the offline training stage to the greatest extent, and considering the influence of the aircraft's steady state and dynamic performance, a reward function including the aircraft's three-channel attitude tracking error is established, thereby increasing the reward in the direction of reducing the aircraft's attitude tracking error and improving the dynamic performance, and increasing the penalty in the opposite direction, so as to achieve robust, safe and stable control of the aircraft, improve the robustness of the aircraft control, and realize the aircraft attitude control based on robust adversarial reinforcement learning. Summary of the invention

[0004] The purpose of the present invention is to provide a hypersonic aircraft attitude control method based on robust adversarial reinforcement learning, which can introduce an interference network during the aircraft simulation training process to simulate deviation interference to the greatest extent, improve control robustness, and realize aircraft multi-channel attitude decoupling-free reinforcement learning control with strong robustness.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A hypersonic vehicle attitude control method based on robust adversarial reinforcement learning comprises the following steps:

[0007] Step 1: Establish the aircraft attitude motion model and determine the control relationship between the control torque and attitude angle;

[0008] Step 2: Establish an optimization model for the attitude control of the aircraft and determine the reward function that takes into account the steady-state and dynamic performance during the attitude control process;

[0009] Step 3: Establish an aircraft attitude control model, design an aircraft attitude control network and interference network with autonomous learning capabilities based on judgment-action, and implement offline training of the algorithm.

[0010] Furthermore, the aircraft attitude motion model is a three-degree-of-freedom reentry attitude motion model in a hypersonic aircraft body coordinate system, and the three-degree-of-freedom reentry attitude motion model in a hypersonic aircraft body coordinate system can be expressed as:

[0011]

[0012] Wherein, p, q, r, α, β, σ represent the roll angle rate, pitch angle rate, yaw angle rate, angle of attack, sideslip angle and roll angle respectively; M x ,M y ,M z are rolling moment, pitching moment and yaw moment respectively; I ij (i=x,y,z,j=x,y,z) represents the moment of inertia of the aircraft. Since the rotational motion of a hypersonic aircraft is faster than its translational motion, the orbital motion term and angular velocity described in the aircraft attitude motion equation are ignored in the controller design process based on the principle of model simplification.

[0013] Simplifying expression (1), we can obtain the attitude control equation of the aircraft rotating around the center of mass:

[0014]

[0015] Where Θ = [α, β, σ] T ,ω=[p,q,r] T is the attitude angle vector and rotation angular velocity vector of the hypersonic vehicle, M=[M x ,M y ,M z ] T is the control torque vector, H is the matrix related to the system state, I is the system inertia matrix, ω × is a skew-symmetric matrix about ω, which can be expressed as:

[0016]

[0017] The above control-oriented model mainly gives the control relationship between the control torque M and the attitude angle θ; further, the attitude angle tracking error is defined as:

[0018] e1=Θ-Θ d (3)

[0019] In the formula, Θ d is the reference attitude angle vector; based on equations (2) and (3), the error dynamic equation is established:

[0020]

[0021] In the formula, is the attitude angular rate tracking error, In the formula, h(ω)=-HI -1 ω × Iω is the nominal term of the system.

[0022] Furthermore, the optimization model for aircraft attitude control in step 2 is a Markov model including a state set, an action set, a zero-sum reward function, a state transition probability, a discount factor, etc.

[0023] Furthermore, the three-channel attitude angle, acceleration, etc. obtained by the aircraft using sensors are taken as the state s, and the state set S is defined as:

[0024] S={Θ,ω} (5)

[0025] Where Θ = [α, β, σ] T is the three-channel attitude angle, where α is the angle of attack, β is the sideslip angle, σ is the roll angle, and ω=[p,q,r] T They are the angular velocities of the three channels: roll, pitch and yaw.

[0026] Furthermore, the reward function:

[0027] R=R(e1,δ,t s ,κ) (8)

[0028] Among them, e1 is the three-channel attitude tracking error of the aircraft, δ is the overshoot of the aircraft control, and t s is the adjustment time, κ is the weight coefficient;

[0029] During offline training, the reward function for the interference control network is R as shown in equation (8), while the reward function for the interference network is set to -R.

[0030] Furthermore, control network action set A c Defined as:

[0031] A c ={M x,M y ,M z} (6)

[0032] Among them, M x ,M y ,M z They are the three-channel control torques of roll, pitch and yaw respectively;

[0033] The interference network action set is defined as:

[0034] A D ={D x ,D y ,D z} (7)

[0035] Where D x ,D y ,D z They are the three-channel comprehensive deviation interferences respectively.

[0036] Furthermore, the aircraft control network and the interference network are both designed using a reinforcement learning algorithm based on a double-delayed deep deterministic policy gradient.

[0037] Furthermore, the aircraft control network architecture includes four networks: Actor, Actor target, Critic, and Critic target;

[0038] The input of the Actor network of the control network is the system status information obtained by the aircraft using sensors, and the system status information includes the three-channel attitude angle of the aircraft and the three-channel angular velocity of roll, yaw and pitch. The Actor network of the control network outputs a deterministic strategy, and the output is the three-channel control torque of the aircraft;

[0039] The input of the Critic network of the control network is the state information obtained by the aircraft sensor and the action information output by the Actor network. The output of the Critic network is the Q value corresponding to the action taken in this state, which is used to evaluate the quality of the action.

[0040] The Actor target network and the Critic target network of the control network do not update the network parameters through back propagation, but copy some parameters in the Actor and Critic networks to the target network in a soft update manner;

[0041] The aircraft jamming network architecture includes four networks: Actor, Actor target, Critic, and Critic target;

[0042] The input of the Actor network of the interference network is the system status information obtained by the aircraft using sensors, and the interference network includes the three-channel attitude angle of the aircraft and the three-channel angular velocity of roll, yaw and pitch; the output of the Actor network of the interference network is a deterministic strategy, and the output is the interference amount of the aircraft system;

[0043] The input of the Critic network of the interference network is the state information obtained by the aircraft sensor and the interference amount output by the Actor network of the interference network. The output of the Critic network is the Q value corresponding to the action taken in this state, which is used to evaluate the quality of the action.

[0044] The Actor target network and the Critic target network of the interference network do not participate in the training process, and the parameters of the Actor and the Critic are copied to the target network in a soft update manner.

[0045] Furthermore, the training method of the aircraft attitude control network and the interference network comprises the following steps:

[0046] A1: Fixed the interference network to train the control network until the optimal control performance is obtained;

[0047] A2: Fix the control network and train the interference network until the strongest interference performance is achieved;

[0048] A3: Based on alternating iterative training of two networks, Nash equilibrium is finally reached.

[0049] Furthermore, the control network and the interference network both use the TD3 algorithm to implement offline training of the algorithm. A complete training process of the TD3 algorithm includes the following steps:

[0050] S1: Data collection: During the data collection process, the aircraft observes the environment to obtain the current state s, sends it to the Actor network to output the corresponding operation a, and obtains the immediate reward value R and the next state s'; after performing the above operations, each iteration round will store the data in the following form:

[0051] (s,s',a,R) (9)

[0052] This stage does not involve the Critic evaluation network, but only includes the interaction between the Actor network and the environment;

[0053] S2: Parameter training: θ represents the set of agent policy network parameters, and μ represents the set of aircraft control policies. During the training of the Critic network, the network update scheme of the TD3 algorithm is as follows, and the gradient descent method is performed with the goal of reducing the loss function loss:

[0054]

[0055] In the formula, is the target value generated by the target network and is defined as follows:

[0056] y j =R j +ymin(Q i ,(s j ,,a j ,)) (11)

[0057] in, is the current profit value, min(Q i ′(s′ j ,a′ j )) is the smaller Q value at the future moment.

[0058] In the training of the Actor network, the TD3 algorithm uses the deterministic policy gradient formula. The Actor gradient formula is:

[0059]

[0060] In the formula, ρ π is the sampling space of the state, and E[g] represents the mean square error function.

[0061] The technical effects achieved by the present invention are:

[0062] (1) The hypersonic aircraft attitude control method based on robust adversarial reinforcement learning of the present invention is designed to meet the high anti-interference requirements of aircraft control. An interference network is introduced during the aircraft simulation training process to simulate deviation interference to the greatest extent and improve control robustness. A three-channel aircraft attitude control network and an interference network based on action-comment with autonomous learning capabilities are separately built, and a dual-network training method is studied to achieve a highly robust aircraft multi-channel attitude decoupling-free reinforcement learning control. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a framework diagram of a hypersonic aircraft algorithm based on robust adversarial reinforcement learning of the present invention;

[0064] Figure 2 It is a control network design diagram based on the TD3 algorithm structure of the present invention;

[0065] Figure 3 is a reward function curve diagram of the present invention;

[0066] Figure 4 It is a curve diagram of the completion rate of every 50 rounds of the present invention;

[0067] Figure 5It is a curve diagram of the number of steps per round of the present invention;

[0068] Figure 6 It is a posture error curve diagram of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is specifically described below in conjunction with embodiments. It should be understood that the following text is only used to describe one or several specific embodiments of the present invention, and does not strictly limit the scope of protection of the specific claims of the present invention.

[0070] Considering the strong external interference problem caused by the complex operating environment of the aircraft, the present invention aims to propose a design method of an aircraft robust controller based on control-interference reinforcement learning to ensure the strong robustness and control performance of the aircraft control. Specifically, considering the requirements of aircraft attitude stability control, firstly, a reward function including the aircraft three-channel attitude tracking error and dynamic performance indicators is established, and a decision model for aircraft attitude control is established; secondly, an aircraft attitude control network and an interference network with autonomous learning ability based on actor-critic are respectively built, wherein the interference network is used to learn the optimal disturbance strategy, thereby affecting the control network learning, and the control network is used to learn the optimal control strategy under the counter-disturbance, and then complete the given flight mission; finally, a dual-network training method is studied, and the interference network is continuously and iteratively trained through the two networks to ensure that the interference network maximizes the disturbance effects such as environmental interference, while the control network achieves optimal control under the maximum disturbance, thereby achieving safe and stable control under the interference level, and improving the robustness of aircraft control.

[0071] like Figure 1-6 As shown, a hypersonic aircraft attitude control method based on robust adversarial reinforcement learning includes the following steps:

[0072] Step 1: Establish the aircraft attitude motion model and determine the control relationship between the control torque and attitude angle;

[0073] Specifically, it establishes a three-degree-of-freedom reentry attitude motion model in the hypersonic aircraft body coordinate system, comprehensively considers external disturbance factors, integrates the attitude motion model into the state transfer function, and enhances the performance and reliability of the flight control system.

[0074] The three-degree-of-freedom reentry attitude motion model in the hypersonic vehicle body coordinate system can be expressed as:

[0075]

[0076] Wherein, p, q, r, α, β, σ represent the roll angle rate, pitch angle rate, yaw angle rate, angle of attack, sideslip angle and roll angle respectively; Mx ,M y ,M z are rolling moment, pitching moment and yaw moment respectively; I ij (i=x,y,z,j=x,y,z) represents the moment of inertia of the aircraft. Since the rotational motion of a hypersonic aircraft is much faster than its translational motion, the orbital motion term and angular velocity described in the aircraft attitude motion equation are ignored in the controller design process based on the principle of model simplification.

[0077] Further, by simplifying expression (1), we can obtain the attitude control equation of the aircraft rotating around the center of mass:

[0078]

[0079] Where Θ = [α, β, σ] T ,ω=[p,q,r] T is the attitude angle vector and rotation angular velocity vector of the hypersonic vehicle, M=[M x ,M y ,M z ] T is the control torque vector, H is the matrix related to the system state, I is the system inertia matrix, ω × is a skew-symmetric matrix about ω, which can be expressed as:

[0080]

[0081] The above control-oriented model mainly gives the control relationship between the control torque M and the attitude angle θ; further, the attitude angle tracking error is defined as:

[0082] e1=Θ-Θ d (3)

[0083] In the formula, Θ d is the reference attitude angle vector; based on equations (2) and (3), the error dynamic equation is established:

[0084]

[0085] In the formula, is the attitude angular rate tracking error, In the formula, h(ω)=-HI -1 ω × Iω is the nominal term of the system.

[0086] Step 2: Establish an optimization model for the attitude control of the aircraft and determine the reward function that takes into account the steady-state and dynamic performance during the attitude control process;

[0087] Specifically, it considers the stable tracking control requirements of the aircraft under the influence of unknown external interference, establishes the state set and action set of the aircraft attitude control, and in the reward function setting, on the one hand, considers the stable control performance of the aircraft attitude, on the other hand, considers the dynamic performance indicators such as overshoot during the operation of the aircraft, so as to take into account both steady-state and dynamic performance changes in the aircraft control process, thereby improving the control performance of the aircraft.

[0088] Considering the flight characteristics of the aircraft in a large airspace and wide speed range, and based on the requirements for stable control of the aircraft under the influence of complex external interference, the following Markov model including state set, action set, zero-sum reward function, state transition probability, discount factor, etc. is established;

[0089] Among them, the state set is introduced here:

[0090] The three-channel attitude angle, acceleration, etc. obtained by the aircraft using sensors are taken as the state s, and the state set S is defined as:

[0091] S={Θ,ω} (5)

[0092] Where Θ = [α, β, σ] T is the three-channel attitude angle, where α is the angle of attack, β is the sideslip angle, σ is the roll angle, and ω=[p,q,r] T They are the angular velocities of the three channels: roll, pitch and yaw.

[0093] Among them, the action set is introduced here:

[0094] For the control network, considering the stable tracking control requirements of the aircraft, based on the aircraft attitude control model, the action set includes the basic control quantity of the aircraft - the three-channel control torque, so the control network action set A c Defined as:

[0095] A c ={M x ,M y ,M z} (6)

[0096] Among them, M x ,M y ,M z They are the three-channel control torques of roll, pitch and yaw.

[0097] For the interference network, its network output is the system interference torque, which is used to counter the control network to destroy the stability of the aircraft control. Therefore, the interference network action set is defined as:

[0098] A D ={D x ,D y ,Dz} (7)

[0099] Where D x ,D y ,D z They are three-channel comprehensive deviation interference, including comprehensive interference including external interference, model uncertainty, etc.

[0100] Among them, the zero-sum reward function is introduced here:

[0101] In order to simulate deviation interference to the greatest extent and improve control robustness during offline training, an interference network is introduced during the aircraft simulation training process to simulate deviation interference. Therefore, a zero-sum reward function for the aircraft control network and the interference network is established.

[0102] The present invention comprehensively considers two parts of requirements when designing the reward function: on the one hand, the demand for stable tracking control of the aircraft under the influence of unknown external interference of the aircraft is considered, and the three-channel attitude tracking error is used as a reference. The smaller the tracking error, the greater the reward, and the greater the tracking error, the greater the penalty, so as to ensure the steady-state control performance of the aircraft under the influence of interference; on the other hand, the dynamic characteristics such as overshoot during the operation of the aircraft are also crucial to the control performance of the aircraft, so that dynamic characteristics indicators (such as aircraft control overshoot, response time, etc.) are further added to the reward function setting. The better the dynamic performance indicators, the greater the reward, so as to ensure the dynamic performance changes during the operation of the aircraft, and then improve the control performance of the aircraft. In summary, based on the above requirements to ensure the steady-state performance and dynamic performance of the aircraft, a reward and punishment mechanism is established, the benefit value obtained by single-step decision control is determined, and the reward function shown below is designed:

[0103] R=R(e1,δ,t s ,κ) (8)

[0104] Among them, e1 is the three-channel attitude tracking error of the aircraft, δ is the overshoot of the aircraft control, and t s is the adjustment time, and κ is the weight coefficient.

[0105] In addition, because the aircraft is disturbed by complex environment during operation, in order to improve the simulation degree of disturbance during offline training and improve the robustness of aircraft controller, the interference network is introduced in the aircraft simulation training process to simulate external unknown disturbance, so the corresponding zero-sum reward function for aircraft control network and interference network is established. The purpose of designing interference network is to form a complete confrontation with control network, and it is expected that it will simulate the strongest disturbance to deal with control network, that is, the strongest opponent is trained to confront the control network we need, so as to obtain the control under the strongest disturbance. Therefore, the reward function of control network is designed to be R as shown in formula (8), while the reward function of interference network is set to -R, that is, the reward function of control network and interference network is set to be completely opposite, that is, the maximum reward for one party corresponds to the maximum punishment for the other party, thus forming a zero-sum confrontation, reversely optimizing control target, improving robustness to disturbance and generalization ability of model, and laying a foundation for subsequent network training.

[0106] Among them, the state transition probability is introduced here:

[0107] For the aircraft attitude control problem studied in the present invention, the attitude control model of the aircraft is used as a state transfer function, and the state transfer probability is calculated by the state transfer function;

[0108] Among them, the discount factor is introduced here:

[0109] The discount factor γ indicates the importance of future benefits relative to current benefits. When γ = 0, it is equivalent to only considering current benefits without considering future benefits. When γ = 1, future benefits and current benefits are considered equally important.

[0110] By defining the aircraft control Markov model, the aircraft control process can be re-described as follows: for the state set s(t) consisting of the aircraft attitude angle and angular velocity obtained by the sensor at time t, according to the designed immediate benefit value R i (t), determine the optimal action A under the premise of maximum benefit c (t), and interact with the environment, and obtain the vehicle state s(t+1) at the next moment by the state transfer function, and repeat the process until the desired stable tracking control goal is achieved.

[0111] Step 3: Establish an aircraft attitude control model, design an aircraft attitude control network and interference network with autonomous learning capabilities based on judgment and action, and implement offline training of the algorithm;

[0112] like Figure 1-2As shown in the figure, here, we first consider the flight characteristics of the aircraft in a large airspace and a wide speed range. Aiming at the problems of poor performance and stability of traditional reinforcement learning in the face of uncertainty and potential confrontational environments, an aircraft attitude controller with autonomous learning ability is designed. By introducing an interference network in the aircraft simulation training process to simulate deviation interference to the greatest extent and improve control robustness, a three-channel attitude control network and an interference network for the aircraft with autonomous learning ability based on action-comment are built respectively, and the dual-network training method is studied to realize the aircraft multi-channel attitude decoupling-free reinforcement learning control with strong robustness.

[0113] Among them, considering the influence of external interference on the aircraft, the design strategy of robust controller for hypersonic aircraft based on control-disturbance reinforcement learning is studied. The structure diagram is as follows Figure 1 shown.

[0114] Among them, the design of control-interference network for robust attitude control of aircraft is introduced here;

[0115] First, determine the aircraft attitude control state set (three-channel attitude angle, angular velocity, etc.), action set (control torque), etc., based on the established aircraft control reward function that considers dynamic and steady-state control performance, and introduce the interference network to autonomously train and simulate deviation interference to improve the generalization of control, and build an aircraft attitude control network and interference network with autonomous learning ability based on judgment-action, respectively. The interference network is used to learn the optimal disturbance strategy, thereby affecting the learning of the control network; and the control network is used to learn the optimal control strategy under the counter-disturbance, and then complete the given flight mission. Considering that the Critic target network in the traditional DDPG (Deep Deterministic Policy Gradient) reinforcement learning algorithm does not interact with the environment when calculating the behavioral value of the target action, there is an over-estimation problem of the state value, resulting in poor performance of the learned strategy. Therefore, the present invention adopts a reinforcement learning algorithm based on Twin Delayed Deep Deterministic Policy Gradient (TD3) for network design for both the aircraft control network and the interference network, and its structure diagram is shown as follows Figure 2 As shown, Figure 2 Design diagram of the control network (control and interference network) based on the TD3 algorithm structure.

[0116] Among them, the aircraft control network architecture consists of four networks, including Actor, Actor target, Critic, and Critic target. The input of the Actor network is the system status information obtained by the aircraft using sensors, including the three-channel attitude angle of the aircraft and the three-channel angular velocity of roll, yaw and pitch; the Actor network outputs a deterministic strategy, and the output is the three-channel control torque of the aircraft to ensure the stable tracking control and dynamic performance of the aircraft; the Critic network of the control network inputs the status information including the attitude angle and angular velocity obtained by the aircraft sensor and the action information output by the Actor network, and the output of the Critic network is the Q value corresponding to the action taken in this state, which is used to evaluate the quality of the action; the Actor target network and the Critic target network do not update the network parameters through back propagation, but copy some parameters in the Actor and Critic networks to the target network in a soft update manner, thereby reducing the oscillation during network training and improving the convergence ability of the network.

[0117] Among them, the aircraft interference network architecture is also composed of four networks, including Actor, Actor target, Critic, and Critic target. The input of the Actor network is the system state information obtained by the aircraft using sensors, including the three-channel attitude angle of the aircraft and the three-channel angular velocity of roll, yaw and pitch; the output of the Actor network is also a deterministic strategy, and the output is the interference amount of the aircraft system, in order to maximize the resistance to the stable control of the aircraft and simulate the comprehensive deviation interference effects including external interference and model uncertainty; the input of the Critic network of the interference network is the state information obtained by the aircraft sensor and the interference amount output by the Actor network, and the output of the Critic network is the Q value corresponding to this action taken under this state, which is used to evaluate the quality of the action; like the control network, the Actor target network and the Critic target network of the interference network do not participate in the training process, and the parameters of the Actor and Critic are copied to the target network in a soft update manner.

[0118] Based on the above-established aircraft control network and interference network architecture, a foundation is laid for the subsequent research on network training methods and the realization of aircraft multi-channel decoupling-free attitude control.

[0119] Under the established aircraft control-interference network framework, the network back-propagation algorithm is studied, and aircraft control training is carried out in the constructed simulation training environment. During the training process, the learning characteristics of the control network are also given to the interference network;

[0120] Specifically, the training method of the aircraft attitude control network and the interference network includes the following steps:

[0121] A1: Fixed the interference network to train the control network until the optimal control performance is obtained;

[0122] A2: Fix the control network and train the interference network until the strongest interference performance is achieved;

[0123] A3: Based on the alternating iterative training of the two networks, Nash equilibrium is eventually reached to ensure that the interference network maximizes the impact of disturbances such as environmental interference, while the control network achieves optimal control under the maximum disturbance. This ensures the steady-state and control performance of the aircraft while improving generalization, and achieves theoretical proof of optimality, thus realizing stable tracking control of the aircraft's multi-channel attitude without decoupling.

[0124] In the process of training the control network and the interference network separately, such as Figure 2 As shown, the TD3 algorithm is adopted and offline training of the algorithm is realized;

[0125] The TD3 algorithm used has two decision networks and four evaluation networks. The decision network is represented by π and the evaluation network is described by Q. The Critic network (which only runs during the training phase) can obtain the aircraft status (attitude angle, acceleration, etc.) and action information (control torque), so during the training phase, it can guide the Actor it is responsible for from a bird's eye view.

[0126] A complete training process of the TD3 algorithm includes data collection and parameter training, which specifically includes the following steps:

[0127] S1: Data collection: During the data collection process, the aircraft observes the environment to obtain the current state s, sends it to the Actor network to output the corresponding operation a, and obtains the immediate reward value R and the next state s'. After performing the above operations, each iteration round will store the data in the following form:

[0128] (s,s',a,R) (9)

[0129] This process requires a large number of rounds to obtain enough experience vectors, all of which are stored in the experience pool. This stage does not involve the Critic evaluation network, but only includes the interaction between the Actor network and the environment. One idea is to conduct a large number of episodes before parameter training to obtain a large enough experience pool, but this method has a problem: the Actors used to obtain data use the same strategy, and the resulting control strategies are correlated, which will affect the training effect. Therefore, the present invention adopts a method of collecting and training the network at the same time, using an updated Actor network to collect data to ensure the independence of the collected data.

[0130] S2: Parameter training: θ represents the set of agent policy network parameters, and μ represents the set of aircraft control policies. During the Critic network training process, the network update scheme of the TD3 algorithm is as follows, and the gradient descent method is performed with the goal of reducing the loss function loss:

[0131]

[0132] In the formula, is the target value generated by the target network and is defined as follows:

[0133] y j =R j +ymin(Q i ,(s j ,,a j ,)) (11)

[0134] in, is the current profit value, min(Q′ i (s′ j ,a′ j )) is the smaller Q value at the future moment.

[0135] TD3 has two innovations in the Critic network: 1. In response to the overestimation problem of the Critic network in the DDPG algorithm, a dual network is designed to reduce overestimation; 2. The update frequency of the Critic network is increased to make the Critic network's estimate of the Q value more accurate. In the training of the Actor network, the TD3 algorithm still uses the deterministic policy gradient formula. The Actor gradient formula is:

[0136]

[0137] In the formula, ρ π is the sampling space of the state, and E[g] represents the mean square error function.

[0138] The aircraft control network and the interference network both use the above data collection and parameter training process to complete network training. In the training process, in order to improve the ability of the interference network and the control network's confrontation (robustness) ability, the learning characteristics of the control network are also given to the interference network. The interference network even has characteristics that the control network does not have, such as the system state mutation allowed in the interference network. The training of the two networks first fixes the interference network to train the control network until the optimal control performance is obtained. The control network is further fixed and the interference network is trained until the strongest interference performance is obtained at the training point. Based on the alternating iterative training of the two networks, Nash equilibrium is finally achieved to ensure that the interference network maximizes the influence of environmental interference, training and actual deviations and other disturbances to hinder the stable control of the aircraft to the greatest extent; while the control network achieves optimal control under the maximum disturbance to ensure the stable control of the aircraft to the greatest extent, thereby achieving stable control of the aircraft's multi-channel attitude, ensuring the dynamic and steady-state performance in the control process, and effectively improving the robustness of the controller.

[0139] In order to verify the effectiveness of the aircraft control method based on robust adversarial reinforcement learning proposed in this invention, the aircraft physical model is first designed in the Python environment and a simulation experiment is carried out. The main simulation process is as follows:

[0140] (1) Parameter setting

[0141] Aircraft physical parameter settings: dynamic pressure is q = 5189.62528, aircraft characteristic area S = 4.17, aircraft characteristic length L = 4.12, system set inertial parameter is I xx =214,I yy =3190,I zz =3083,I xz =0, the initial value of the attitude angle is Θ0 = [α0, β0, σ0] T =[0.170.050.34] T rad, the initial value of the attitude angular velocity is designed to be ω0 = [p0, q0, r0] T =

[000] T rad; reference attitude angle Θ d ~Uniform(-0.5,0.5)rad represents Θ d is a random number uniformly distributed between -0.5 and 0.5; the number of steps per round of reinforcement learning is 3000, after which the environment is reset to start the next round of iteration;

[0142] (2) Results Analysis

[0143] Figure 3It represents the reward value curve obtained by the control network. As can be seen from the figure, as the number of iterations increases, the reward value becomes larger and larger, and finally tends to be stable, indicating that the control network has converged and has good control performance; Figure 4 It represents the completion rate of the designed robust adversarial reinforcement learning method every 50 rounds. It can be seen that the completion rate is above 96%; Figure 5 The number of rounds required to achieve the required control accuracy during training. It can be seen that only 1000 steps are needed to achieve the control requirement under the interference network. Figure 6 In order to use the trained control network to verify the attitude control, it can be seen from the figure that the attitude error tends to be stable within 15 seconds and meets the control accuracy requirement. Then, interference is added within 20-25 seconds. It can be seen that the attitude error first increases slightly, then decreases and stabilizes, verifying the effectiveness of the aircraft control method based on robust adversarial reinforcement learning proposed in the present invention.

[0144] In summary, this technical solution takes into account the demand for stable attitude control of the aircraft, establishes a reward function including the three-channel attitude tracking error of the aircraft, and considers the engineering demand for improving the dynamic performance of the operation process. Dynamic performance indicators such as attitude control overshoot are further added to the reward function, thereby increasing the reward in the direction of reducing the attitude tracking error of the aircraft and improving the dynamic performance, and increasing the penalty conversely, laying the foundation for the subsequent training of a strong robust control network. In response to the high anti-interference requirements of aircraft control, an interference network is introduced during the aircraft simulation training process to simulate deviation interference to the greatest extent and improve control robustness. By separately building an action-comment-based three-channel attitude control network and an interference network with autonomous learning capabilities for the aircraft, a dual-network training method is studied to achieve a strong and robust multi-channel attitude decoupling-free reinforcement learning control for aircraft.

[0145] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A hypersonic vehicle attitude control method based on robust adversarial reinforcement learning, characterized in that: The following steps are involved: Step 1: Establish the aircraft attitude motion model and determine the control relationship between the control torque and attitude angle; Step 2: Establish an optimization model for the attitude control of the aircraft and determine the reward function that takes into account the steady-state and dynamic performance during the attitude control process; Step 3: Establish an aircraft attitude control model, design an aircraft attitude control network and interference network with autonomous learning capabilities based on judgment and action, and implement offline training of the algorithm; The aircraft attitude motion model is a three-degree-of-freedom reentry attitude motion model in the hypersonic aircraft body coordinate system. The three-degree-of-freedom reentry attitude motion model in the hypersonic aircraft body coordinate system can be expressed as: Wherein, p, q, r, α, β, σ represent the roll angle rate, pitch angle rate, yaw angle rate, angle of attack, sideslip angle and roll angle respectively; M x ,M y ,M z are rolling moment, pitching moment and yaw moment respectively; I ij (i=x,y,z,j=x,y,z) represents the moment of inertia of the aircraft. Since the rotational motion of a hypersonic aircraft is faster than its translational motion, the orbital motion term and angular velocity described in the aircraft attitude motion equation are ignored in the controller design process based on the principle of model simplification. Simplifying expression (1), we can obtain the attitude control equation of the aircraft rotating around the center of mass: Where Θ = [α, β, σ] T ,ω=[p,q,r] T is the attitude angle vector and rotation angular velocity vector of the hypersonic vehicle, M=[M x ,M y ,M z ] T is the control torque vector, H is the matrix related to the system state, I is the system inertia matrix, ω × is a skew-symmetric matrix about ω, which can be expressed as: The above control-oriented model mainly gives the control relationship between the control torque M and the attitude angle θ; further, the attitude angle tracking error is defined as: e1=Θ-Θ d (3) In the formula, Θ d is the reference attitude angle vector; based on equations (2) and (3), the error dynamic equation is established: In the formula, is the attitude angular rate tracking error, In the formula, h(ω)=-HI -1 ω × Iω is the nominal term of the system.

2. The hypersonic vehicle attitude control method based on robust adversarial reinforcement learning according to claim 1, characterized in that: The optimization model for aircraft attitude control in step 2 is a Markov model including a state set, an action set, a zero-sum reward function, a state transition probability, and a discount factor.

3. The hypersonic vehicle attitude control method based on robust adversarial reinforcement learning according to claim 2, characterized in that: The three-channel attitude angle, acceleration, etc. obtained by the aircraft using sensors are taken as the state s, and the state set S is defined as: S={Θ,ω} (5) Where Θ = [α, β, σ] T is the three-channel attitude angle, where α is the angle of attack, β is the sideslip angle, σ is the roll angle, and ω=[p,q,r] T They are the angular velocities of the three channels: roll, pitch and yaw.

4. The hypersonic vehicle attitude control method based on robust adversarial reinforcement learning according to claim 2, characterized in that: The reward function: R=R(e1,δ,t s ,κ) (8) Among them, e1 is the three-channel attitude tracking error of the aircraft, δ is the overshoot of the aircraft control, and t s is the adjustment time, κ is the weight coefficient; During offline training, the reward function for the interference control network is R as shown in equation (8), while the reward function for the interference network is set to -R.

5. The hypersonic vehicle attitude control method based on robust adversarial reinforcement learning according to claim 2, characterized in that: Control network action set A c Defined as: A c ={M x ,M y ,M z } (6) Among them, M x ,M y ,M z They are the three-channel control torques of roll, pitch and yaw respectively; The interference network action set is defined as: A D ={D x ,D y ,D z } (7) Where D x ,D y ,D z They are the three-channel comprehensive deviation interferences respectively.

6. The hypersonic vehicle attitude control method based on robust adversarial reinforcement learning according to claim 1, characterized in that: The aircraft control network and the interference network are both designed using a reinforcement learning algorithm based on a double-delayed deep deterministic policy gradient.

7. The hypersonic vehicle attitude control method based on robust adversarial reinforcement learning according to claim 1, characterized in that: The aircraft control network architecture includes four networks: Actor, Actor target, Critic, and Critic target; The input of the Actor network of the control network is the system status information obtained by the aircraft using sensors, and the system status information includes the three-channel attitude angle of the aircraft and the three-channel angular velocity of roll, yaw and pitch. The Actor network of the control network outputs a deterministic strategy, and the output is the three-channel control torque of the aircraft; The input of the Critic network of the control network is the state information obtained by the aircraft sensor and the action information output by the Actor network. The output of the Critic network is the Q value corresponding to the action taken in this state, which is used to evaluate the quality of the action. The Actor target network and the Critic target network of the control network do not update the network parameters through back propagation, but copy some parameters in the Actor and Critic networks to the target network in a soft update manner; The aircraft jamming network architecture includes four networks: Actor, Actor target, Critic, and Critic target; The input of the Actor network of the interference network is the system status information obtained by the aircraft using sensors, and the interference network includes the three-channel attitude angle of the aircraft and the three-channel angular velocity of roll, yaw and pitch; the output of the Actor network of the interference network is a deterministic strategy, and the output is the interference amount of the aircraft system; The input of the Critic network of the interference network is the state information obtained by the aircraft sensor and the interference amount output by the Actor network of the interference network. The output of the Critic network is the Q value corresponding to the action taken in this state, which is used to evaluate the quality of the action. The Actor target network and the Critic target network of the interference network do not participate in the training process, and the parameters of the Actor and the Critic are copied to the target network in a soft update manner.

8. The hypersonic vehicle attitude control method based on robust adversarial reinforcement learning according to claim 1, characterized in that: The training method of the aircraft attitude control network and the interference network comprises the following steps: A1: Fixed the interference network to train the control network until the optimal control performance is obtained; A2: Fix the control network and train the interference network until the strongest interference performance is achieved; A3: Based on alternating iterative training of two networks, Nash equilibrium is finally reached.

9. The hypersonic vehicle attitude control method based on robust adversarial reinforcement learning according to claim 8, characterized in that: The control network and the interference network both use the TD3 algorithm to implement offline training of the algorithm. A complete training process of the TD3 algorithm includes the following steps: S1: Data collection: During the data collection process, the aircraft observes the environment to obtain the current state s, sends it to the Actor network to output the corresponding operation a, and obtains the immediate reward value R and the next state s'; after performing the above operations, each iteration round will store the data in the following form: (s,s',a,R) (9) This stage does not involve the Critic evaluation network, but only includes the interaction between the Actor network and the environment; S2: Parameter training: θ represents the set of agent policy network parameters, and μ represents the set of aircraft control policies. During the training of the Critic network, the network update scheme of the TD3 algorithm is as follows, and the gradient descent method is performed with the goal of reducing the loss function loss: In the formula, is the target value generated by the target network and is defined as follows: y j =R j +ymin(Q i ,(s j ,,a j ,)) (11) in, is the current profit value, min(Q i ′(s′ j ,a′ j )) is the smaller Q value at the future moment. In the training of the Actor network, the TD3 algorithm uses the deterministic policy gradient formula. The Actor gradient formula is: In the formula, ρ π is the sampling space of the state, and E[g] represents the mean square error function.

Images