Hypersonic flight vehicle adaptive attitude control method and system based on near-end strategy optimization

Abstract

The invention discloses a hypersonic flight vehicle adaptive attitude control method based on near-end strategy optimization, and belongs to the technical field of hypersonic flight vehicle control, and the method comprises the steps: building an air-breathing hypersonic flight vehicle longitudinal dynamic model considering power; performing reinforcement learning training on the obtained environment model by using a PPO algorithm to obtain an optimized control strategy; selecting a proper state representation and action space, and designing a strategy network and a value network for the processed model, so that an original complex attitude control problem is converted into a reinforcement learning problem which can be effectively learned; and training the reinforcement learning model by using an improved PPO algorithm to obtain a final control instruction. According to the method, the model is trained by using the PPO algorithm, so that the defects of complex controller design, strict requirement on model precision and the like of a traditional control method can be effectively overcome. A simulation experiment verifies that pitch angle tracking can be effectively realized, and the stability of an execution mechanism is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of hypersonic aircraft control, and in particular, relates to a hypersonic aircraft adaptive attitude control method and system based on proximal strategy optimization. Background Art

[0002] Hypersonic vehicles have attracted much attention in the aerospace field due to their excellent flight performance. However, in complex and changeable extreme flight environments, such as high speed, high temperature and strong impact, hypersonic vehicles will cause their aerodynamic characteristics to change dramatically, which brings great challenges to the design of flight control systems. In order to ensure the flight stability and controllability of hypersonic vehicles, the design of attitude control systems is crucial.

[0003] Existing hypersonic vehicle attitude control technologies mainly include PID control, sliding mode control and robust control. Although these methods have achieved the attitude control of hypersonic vehicles to a certain extent, there are still many limitations. As a linear control method, PID control is difficult to adapt to the complex nonlinear aerodynamic characteristics of hypersonic vehicles. Although sliding mode control has strong robustness, it is prone to buffeting problems, which may be difficult for actual systems to withstand. Although robust control can handle modeling uncertainties, the controller design is complex and the control performance needs to be improved. In addition, the above traditional methods usually rely on the acquisition of an accurate mathematical model of the system, while the aerodynamic model of a hypersonic vehicle is highly nonlinear and strongly coupled, and it is extremely difficult to obtain its accurate model. Therefore, it is urgent to develop new control methods to break through the limitations of traditional methods and improve the adaptability, robustness and control performance of the attitude control system of hypersonic vehicles.

[0004] In recent years, the rapid development of artificial intelligence technology has provided new ideas for the attitude control of hypersonic aircraft. Reinforcement learning, as a data-driven intelligent control method, can autonomously learn the optimal control strategy through interactive learning between the agent and the environment, without relying on the precise model of the system, and has strong adaptability and robustness. Previous researchers have proved that reinforcement learning has achieved good results in the attitude control of aircraft such as spacecraft and drones. Applying reinforcement learning to the attitude control of hypersonic aircraft is expected to break through the limitations of traditional methods and significantly improve the control performance. However, reinforcement learning still faces many challenges in the attitude control of hypersonic aircraft, such as high-dimensional state space, continuous action space and low sample efficiency. Further improvement and optimization of reinforcement learning algorithms are needed. As an advanced policy gradient method, the PPO algorithm has the advantages of high sample efficiency, easy implementation and debugging, adaptation to continuous action space, and good generalization ability. It is very suitable for solving complex, high-dimensional and continuous control problems such as attitude control of hypersonic aircraft.

[0005] In view of the importance of hypersonic aircraft in the field of aerospace and the limitations of traditional attitude control methods, the development of a hypersonic aircraft attitude control method based on reinforcement learning has important theoretical significance and practical application value. This invention aims to break through the limitations of traditional methods, improve the adaptability, robustness and control performance of the hypersonic aircraft attitude control system, and provide a new solution for achieving stable and controllable flight of hypersonic aircraft. Summary of the invention

[0006] The present invention aims to design an efficient, adaptive and robust hypersonic aircraft attitude control strategy by applying and improving the existing PPO algorithm's reinforcement learning method, so as to break through the limitations of traditional control methods and significantly improve the control accuracy, adaptability and robustness of the hypersonic aircraft attitude control system.

[0007] In a first aspect, the present invention provides a hypersonic vehicle adaptive attitude control method based on proximal strategy optimization, the control method comprising the following steps:

[0008] Step 1: Based on the complex and changeable aerodynamic characteristics of hypersonic aircraft, a longitudinal dynamic model of hypersonic aircraft is established;

[0009] Step 2: Design the PPO algorithm strategy network structure and train the PPO algorithm strategy network structure;

[0010] Step 3: Use the trained PPO algorithm strategy network structure to interact with the hypersonic aircraft environment to achieve hypersonic aircraft attitude control.

[0011] Furthermore, the longitudinal dynamics model of the air-breathing hypersonic vehicle established in step 1 is:

[0012]

[0013] In the formula, there are 5 state variables X = [V, h, γ, α, Q] T , V, h, γ, α and Q represent the speed, altitude, track angle, angle of attack and pitch velocity of the aircraft respectively; m and g represent the mass and gravitational acceleration of the aircraft respectively, I yy represents the moment of inertia; T, D and L represent the engine thrust, drag and lift respectively, M yy represents the pitching moment, and the specific expression is

[0014]

[0015] In the formula, it contains three control inputs U=[Φ,δ c ,δ e ] T , Φ represents the fuel valve opening, δc and δ e denote the canard deflection angle and the elevator deflection angle respectively; represents dynamic pressure; S, z T Represents parameters related to the hypersonic vehicle fuselage; C T ,C L ,C D ,C M It is an aerodynamic coefficient determined according to the specific aircraft shape and flight conditions, and is used to calculate aerodynamic forces and moments under different flight conditions.

[0016] Furthermore, the specific steps of step 2 include:

[0017] Step 201: Setting the environment, including determining the state space and action space;

[0018] Step 202: Set up the agent and build a PPO algorithm strategy network framework, which mainly includes an Actor Network structure and a Critic Network structure;

[0019] Step 203: The old strategy in the PPO algorithm strategy network Interact multiple times with the hypersonic vehicle longitudinal dynamics model established in step 1 to generate data for strengthening the training of the PPO algorithm strategy network structure;

[0020] Step 204: Use the data generated in step 203 to train the PPO algorithm strategy network structure.

[0021] Furthermore, the Actor Network structure in step 202 is represented by its objective function:

[0022]

[0023] In the formula, is the probability ratio, indicating that the new and old strategies are in state s t Take action a t The probability ratio of is the generalized advantage estimate, used to estimate action a t The advantage function is γ is the discount factor, λ is the parameter that controls the bias-variance tradeoff, is the TD error, V(s) is the state value function, and ε is a hyperparameter used to control the magnitude of the policy update. It means to average all time steps t in the trajectory data Ω, clip(r t (θ), 1-ε, 1+ε) represents the confidence region constraint, and the expression is as follows:

[0024]

[0025] Furthermore, the Critic Network structure in step 202 is represented by its objective function:

[0026]

[0027] In the formula, is the TD error, λ is the L2 regularization coefficient, and ω represents the jth weight in the Critic Network.

[0028] Furthermore, in step 203, during the interaction between the agent and the environment, ActorNetwor receives the current state s t As input, and generate the corresponding action a t , action a t Will affect the environment, causing it to move to the next state s t+1 , and the environment will give the current reward r t , the data set generated by this process [s t ,a t ,r t ,s t+1 ] is stored in the mini-match and used to calculate the advantage function

[0029] Furthermore, step 204 includes the following steps:

[0030] Step 2041: setting the PPO algorithm strategy network parameters and confidence region, and initializing the simulation environment of the hypersonic aircraft, including aerodynamic parameters, initial states of control surfaces, and simulation time steps;

[0031] Step 2042: Execute the current PPO algorithm policy network in the initialized simulation environment, run multiple time steps to collect samples, and store the collected aircraft attitude control data, including the state, action, reward, next state and completion flag of each time step;

[0032] Step 2043: Use the Critic network to predict the current state and value state, and calculate the TD error and generalized advantage estimate This step is performed once per training cycle to normalize the advantage function to stabilize the optimization process;

[0033] Step 2044: Use the data in mini-match to analyze the objective function L CLIP and L Critic Optimize, use the pruned policy objective function to optimize the PPO algorithm policy network parameters, update the Actor network and use the TD target to optimize the Critic network according to the loss function;

[0034] Step 2045: Adopting the experience replay mechanism, repeatedly using the same batch of training samples in each training cycle, and accelerating the training convergence process of the PPO algorithm strategy network;

[0035] Step 2046: Use a validation set or a specific evaluation metric to measure the performance of the PPO algorithm policy network. After meeting performance expectations or following the evaluation cycle, save the current policy network parameters for further analysis and deployment.

[0036] Furthermore, during the training process of the PPO algorithm strategy network, by designing a reasonable reward function, the agent learns the desired actions, which will be passed to the target aircraft as elevator control instructions, thereby achieving control of the aircraft. Considering the physical background of the aircraft, the following infinite time domain objective function is established:

[0037]

[0038] Where the previous step distance pre_distance=|θ t-1 -θ desired |, Current gap

[0039] now_distance=|θ t -θ desired |.

[0040] Furthermore, in step 2402, exploratory noise is introduced into the collected aircraft attitude control data, and exploratory sample data is generated by interacting with the hypersonic aircraft environment to jump out of the local optimal strategy, specifically including:

[0041] Given a state s t Under this condition, the action distribution dist obtained by the PPO algorithm policy network is Beta(α,β), and action a is sampled from the distribution t ~Beta(α,β), and restricting the actions to the range of [0, 1]. Introducing exploration noise can help the agent explore the environment during training, find better strategies, and escape from the local optimal solution.

[0042] Furthermore, the specific steps of step 2045 include:

[0043] (1) Define an experience replay buffer to store the transfer samples collected during the interaction between the agent and the environment. Each transfer sample includes the current state, action, reward, next state, and termination flag. Set the buffer capacity of the experience replay buffer. If the buffer capacity is exceeded, the new sample will overwrite the earliest added sample.

[0044] (2) During the interaction between the agent and the environment, the transfer samples of each step are stored in the experience replay buffer;

[0045] (3) During each training process, a batch of transfer samples is randomly sampled from the experience replay buffer, and the batch sample size is set to represent the number of transfer samples sampled from the buffer each time;

[0046] (4) Using the transfer sample batches sampled from the experience replay buffer, calculate the loss function and perform gradient descent to update the parameters of the PPO algorithm policy network;

[0047] (5) The transfer samples stored in the experience replay buffer can be reused in the training step.

[0048] As an intelligent attitude control method, based on the hypersonic aircraft dynamics model established in step one, the attitude control of the aircraft is realized based on the PPO reinforcement learning algorithm, targeting the characteristics of strong coupling and strong nonlinearity of hypersonic aircraft control, aerodynamics, and motion, effectively overcoming the problem of complex design of hypersonic aircraft attitude controller.

[0049] As an efficient sample utilization method, step 204 introduces a priority-based experience replay mechanism. This mechanism dynamically adjusts the sampling probability of each sample according to the time difference error, and gives priority to samples with high learning value for training. This method significantly improves the sample efficiency of the algorithm, effectively solves the problem of difficult sample acquisition in the complex dynamic environment of hypersonic aircraft, and accelerates the convergence speed of the control strategy.

[0050] As a robustness enhancement strategy, an adaptive exploration noise mechanism is designed in step 204. This mechanism dynamically adjusts the amplitude of the exploration noise according to the performance of the control strategy, maintains a large exploration intensity in the early stage of training to quickly discover effective control strategies, and gradually reduces the exploration amplitude in the later stage of training to achieve fine tuning. This method significantly improves the algorithm's adaptability to uncertainties and disturbances in the hypersonic flight environment and enhances the robustness of the control strategy.

[0051] In a second aspect, the present invention provides a hypersonic vehicle adaptive attitude control system based on proximal strategy optimization, the system comprising:

[0052] The first building module is used to build a longitudinal dynamics model of a hypersonic aircraft according to the complex and changeable aerodynamic characteristics of the hypersonic aircraft;

[0053] The second building module is used to build the PPO algorithm strategy network structure and convert the old strategy in the PPO algorithm strategy network Interact with the hypersonic vehicle longitudinal dynamics model established in step 1 multiple times, and use the generated data to train the PPO algorithm strategy network;

[0054] The attitude control module is used to call the trained PPO algorithm strategy network structure to interact with the hypersonic aircraft environment to achieve hypersonic aircraft attitude control.

[0055] Compared with the prior art, the present invention has the following advantages:

[0056] 1. The present invention uses a reinforcement learning method based on the PPO algorithm to achieve attitude control of the aircraft based on the characteristics of hypersonic aircraft control, aerodynamics, strong coupling and strong nonlinearity, and effectively overcomes the problem of complex design of hypersonic aircraft attitude controller. Simulation experiments show the effectiveness of the method proposed in the present invention.

[0057] 2. The present invention introduces a priority-based experience playback mechanism. The sampling probability of each sample is dynamically adjusted according to the time differential error, and samples with high learning value are preferentially selected for training. This method significantly improves the sample efficiency of the algorithm, effectively solves the problem of difficulty in obtaining samples in the complex dynamic environment of hypersonic aircraft, and accelerates the convergence speed of the control strategy.

[0058] 3. The present invention effectively improves the expressiveness and generalization of the control strategy by designing a policy network structure based on a deep neural network. The network structure can adaptively learn the complex nonlinear dynamic characteristics of hypersonic vehicles and achieve efficient mapping of the vehicle state space, thereby greatly improving the accuracy and stability of attitude control. The simulation experimental results show that the designed network structure exhibits excellent control performance when achieving mission objectives.

[0059] 4. The present invention introduces a confidence region constraint mechanism, which effectively solves the problem of unstable policy updates. By limiting the amplitude of each policy update, the mechanism ensures the stability of the learning process and avoids drastic performance fluctuations caused by excessive policy updates. Simulation results show that the mechanism significantly improves the convergence speed and final performance of the algorithm, enabling the control strategy to work stably and reliably in a complex hypersonic flight environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of the training process of the PPO algorithm strategy network structure in the present invention;

[0061] Figure 2 It is the reward value convergence curve of the aircraft attitude control PPO algorithm obtained by the method proposed in the present invention;

[0062] Figure 3This is the pitch angle tracking situation of the method proposed by the present invention. DETAILED DESCRIPTION

[0063] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0064] This embodiment specifically provides a hypersonic aircraft adaptive attitude control method based on proximal strategy optimization, which is introduced in detail from the following aspects.

[0065] 1. Kinetic model

[0066] The longitudinal dynamics model of the hypersonic vehicle is established based on the velocity coordinate system:

[0067]

[0068] In the formula, there are 5 state variables X = [V, h, γ, α, Q] T , V, h, γ, α and Q represent the speed, altitude, track angle, angle of attack and pitch velocity of the aircraft respectively; m and g represent the mass and gravitational acceleration of the aircraft respectively, I yy represents the moment of inertia; T, D and L represent the engine thrust, drag and lift respectively, M yy represents the pitching moment, and the specific expression is

[0069]

[0070] In the formula, it contains three control inputs U=[Φ,δ c ,δ e ] T , Φ represents the fuel valve opening, δ c and δ e denote the canard deflection angle and the elevator deflection angle respectively; represents dynamic pressure; S, z T Represents parameters related to the hypersonic vehicle fuselage; C T ,C L ,C D ,C M It is an aerodynamic coefficient determined according to the specific aircraft shape and flight conditions, and is used to calculate aerodynamic forces and moments under different flight conditions.

[0071] 2. Design the PPO algorithm strategy network structure, with the input being the aircraft status and environment information, and the output being the control instructions. The specific steps are:

[0072] 2.1 Determine the state space and action space

[0073] (1) State space: aircraft state information (angle of attack, pitch angle, and pitch velocity)

[0074] (2) Action space: aircraft command control (rudder deflection angle)

[0075] 2.2 Design ActorNetwork Structure

[0076] First, the activation function softplus function is defined. The present invention uses σ(·) to represent the softplus function:

[0077] σ(x)=ln(1+e x )

[0078] Next, define the activation function tanh function, usually expressed as tanh(·):

[0079]

[0080] Therefore, σ′(·) represents the derivative of the softplus function:

[0081]

[0082] tanh′(·) represents the derivative of the tanh function:

[0083] tanh'(x)=1-tanh 2 (x)

[0084] Using a structure with two hidden layers and one output layer, given the state The forward propagation process is:

[0085]

[0086] Where W is the weight matrix of each layer, b is the bias vector, and n s is the dimension of the state space, h1 is the output of the first hidden layer, h2 is the output of the second hidden layer, and α and β represent the outputs of the final two output heads, respectively.

[0087] Calculate the gradient of the loss function with respect to the Alpha and Beta outputs:

[0088]

[0089] Compute the gradients for the second hidden layer:

[0090]

[0091] Compute the gradients for the first hidden layer:

[0092]

[0093] For each parameter (W1, b1, W2, b2, W α ,b α ,W β ,b β ), using the BP gradient descent method, the correction formula for evaluating the network weights can be obtained as:

[0094]

[0095] Among them, χ c Represents the above parameters, l c is the step size factor. So far, the complete action network weight function correction formula is obtained.

[0096] 2.3 Designing CriticNetwork Structure

[0097] Using a structure with two hidden layers and one output layer, given the state The complete forward propagation formula is:

[0098]

[0099] Where W is the weight matrix of each layer, b is the bias vector, and n s is the dimension of the state space, h1 is the output of the first hidden layer, and h2 is the output of the first hidden layer.

[0100] tanh′(·) represents the derivative of the activation function tanh:

[0101] tanh'(x)=1-tanh 2 (x)

[0102] Calculate the gradient of the loss function with respect to the output:

[0103]

[0104] Calculate the gradient from the second hidden layer to the first hidden layer:

[0105]

[0106] Calculate the gradient from the first hidden layer to the input layer:

[0107]

[0108]

[0109] For each parameter (W1, b1, W2, b2, W3, b3), using the BP gradient descent method, the correction formula for evaluating the network weight can be obtained as follows:

[0110]

[0111] Among them, χ c Represents the above parameters, l c is the step size factor. So far, the complete evaluation network weight function correction formula is obtained.

[0112] 2.4 Based on the proximal policy optimization (PPO) algorithm framework, by introducing confidence region constraints and generalized advantage estimation, the sample utilization efficiency is improved while ensuring the stability of strategy update. The specific steps are as follows:

[0113] 2.4.1 In order to ensure the stability of strategy updates, PPO introduces a confidence region constraint. This constraint limits the new strategy π θ With the old strategy The difference between them ensures that the strategy update will not deviate too far.

[0114] PPO cuts the probability ratio r t (θ) to approximately satisfy the confidence region constraint:

[0115]

[0116] 2.4.2 PPO uses generalized advantage estimation (GAE) to estimate the advantage function GAE introduces an additional parameter λ to balance the bias and variance and improve the sample utilization efficiency. Mathematically, GAE can be expressed as:

[0117]

[0118] Where γ is the discount factor and λ is a parameter that controls the bias-variance tradeoff. is the TD error, and V(s) is the state value function.

[0119] 2.4.3 Objective function of Actor network:

[0120]

[0121] In the formula, is the probability ratio, indicating that the new and old strategies are in state s t Take action a t The probability ratio of is the generalized advantage estimate, used to estimate action a t The advantage function of ε is a hyperparameter used to control the magnitude of the strategy update. It means averaging all time steps t in the trajectory data Ω.

[0122] 2.4.4 Objective function of the Critic network:

[0123]

[0124] In the formula, error, λ is the L2 regularization coefficient, and ω represents the jth weight in the Critic network.

[0125] Next, use the data in mini-match to analyze the objective function L CLIP and L Critic Optimize and update the parameters ω of the Actor network θ and the Critic network.

[0126] 2.4.5 By designing a reasonable reward function, the agent can learn the desired actions, which will be passed to the target aircraft as elevator control instructions, thereby achieving control of the aircraft. Different from the traditional quadratic form in control, and considering the physical background of the aircraft, the following infinite time domain objective function is established:

[0127]

[0128] Where the previous step distance pre_distance=|θ t-1 -θ desired |, current distance now_distance = |θ t -θ desired |.

[0129] Compared with the traditional quadratic error form, the reward function designed by the present invention only gives discrete rewards or penalties under specific conditions. This sparse reward can better guide the agent towards the target state instead of paying too much attention to the intermediate process, and takes into account the limitations of the actual physical environment, such as angle limitations.

[0130] 3.5 Introduce exploration noise during the training process, generate exploratory sample data by interacting with the environment, enhance the exploration ability of the PPO algorithm, and jump out of the local optimal strategy. The specific contents are:

[0131] Given a state s t Under this condition, the action distribution dist obtained by the policy network is Beta(α,β), and action a is sampled from the distribution t ~Beta(α,β), and limit the action to the range of [0,1]. Using exploration noise can help the agent explore the environment during training, find better strategies, and help escape from the local optimal solution.

[0132] 3.6 Adopt the experience replay mechanism, by sampling and reusing samples from historical trajectory data, accelerate the training convergence process of the policy network and improve the efficiency of sample utilization. The specific steps are as follows:

[0133] 3.6.1 Define an experience replay buffer to store the transfer samples collected during the interaction between the agent and the environment. Each transfer sample includes the current state (state), action (action), reward (reward), next state (next_state) and termination flag (done). Set the size of the buffer to, after exceeding this size, the new sample will overwrite the earliest added sample.

[0134] 3.6.2 During the interaction between the agent and the environment, the transfer samples (state, action, reward, next_state, done) of each step are stored in the experience replay buffer. This approach makes it easier to reuse previously collected samples during training, rather than having to interact with the environment again to generate new samples each time.

[0135] 3.6.3 In each training step, randomly sample a batch of transfer samples from the experience replay buffer, and set the batch size to represent the number of transfer samples sampled from the buffer each time. Random sampling can break the temporal correlation between transfer samples, making the training process more stable.

[0136] 3.6.4 Use the batch of transfer samples sampled from the experience replay buffer to calculate the loss function and perform gradient descent to update the parameters of the policy network.

[0137] 3.6.5 We can reuse the transfer samples stored in the experience replay buffer in multiple training steps. Each transfer sample can be sampled multiple times and used to update the policy network, accelerating the training convergence process of the policy network and improving the utilization efficiency of samples.

[0138] 3. Use the trained PPO algorithm strategy network structure to interact with the hypersonic aircraft environment to achieve hypersonic aircraft attitude control.

[0139] In summary, the training process of the PPO algorithm strategy network is as follows:

[0140] Step 2041: Set the PPO algorithm strategy network parameters and confidence region. The parameter settings are explained later, and initialize the simulation environment of the hypersonic aircraft, including aerodynamic parameters, initial states of control surfaces, and simulation time steps.

[0141] Step 2042: Collect attitude control data in the aircraft environment. Execute the current policy network in the initialized environment and run multiple time steps to collect samples. Store the collected data, including the state, action, reward, next state, and completion flag for each time step.

[0142] Step 2043: Use the Critic network to predict the current state and value state, and calculate the TD error and generalized advantage estimate This step is performed once per training cycle to normalize the advantage function to stabilize the optimization process.

[0143] Step 2044: Use the data in mini-match to analyze the objective function L CLIP and L Critic Optimize, in each training iteration, the sample blocks are divided into: Calculate the ratio of the new log probability of the action to the old log probability. Use the pruned policy objective function to optimize the policy network parameters, update the Actor network and use the TD objective to optimize the Critic network according to the loss function.

[0144] Step 2045: Adopt the experience replay mechanism to repeatedly use the same batch of training samples in each training cycle to accelerate the training convergence process of the PPO algorithm strategy network.

[0145] Step 2046: Use a validation set or a specific evaluation metric to measure the performance of the policy network. After meeting performance expectations or following a certain evaluation cycle, save the current policy network parameters for further analysis and deployment.

[0146] Simulation experiment

[0147] The simulation experiment is introduced below.

[0148] Combined with the hypersonic vehicle attitude dynamics model established in the first section, a large number of simulation experiments are carried out, and the PPO algorithm hyperparameters are tuned according to the control performance indicators. The specific steps are as follows:

[0149] Step 1: Set up the environment, including necessary features such as state space, action space, reward function, etc.

[0150] Step 2: Set up the agent and build the PPO deep learning framework, which mainly includes the Actor and Critic networks;

[0151] Step 3: Train the network for a total of 20,000 rounds of training, with each round requiring M iterations. The specific training iteration steps are:

[0152]

[0153] Step 4: Use the trained PPO model to interact with the environment to achieve hypersonic vehicle attitude control, such as Figure 1 During the training process, we observed the convergence of the reward value of the PPO algorithm, as shown in the attached Figure 2As shown in the figure, as the number of training rounds increases, the reward value shows an upward trend and eventually stabilizes, which shows that the PPO algorithm continuously improves its strategy during the learning process and finally achieves a good performance level.

[0154] In order to further verify the effectiveness of the trained model, we tested the pitch angle tracking performance of the hypersonic vehicle. Figure 3 The performance of the proposed method in the pitch angle tracking task is shown. It can be observed from the figure that the model can accurately track the given pitch angle command, showing good control accuracy and response speed. This result proves the effectiveness and superiority of the attitude control method based on the PPO algorithm proposed in the present invention in the control task of hypersonic aircraft.

[0155] These experimental results not only verify the application potential of the PPO algorithm in complex control tasks, but also demonstrate the contribution of the present invention in improving the attitude control performance of hypersonic aircraft.

[0156] The simulation hyperparameters are given below

[0157] Table 1 PPO algorithm hyper parameters

[0158]

[0159]

[0160] So far, the design process of a hypersonic aircraft attitude control method based on reinforcement learning designed by the present invention is as shown above.

[0161] According to simulation experiments, the attitude control method of a hypersonic aircraft based on reinforcement learning proposed in the present invention can achieve the expected attitude control effect.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the protection scope of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A hypersonic vehicle adaptive attitude control method based on proximal strategy optimization, characterized in that: The control method comprises the following steps: Step 1: Based on the complex and changeable aerodynamic characteristics of hypersonic aircraft, a longitudinal dynamic model of hypersonic aircraft is established; Step 2: Design the PPO algorithm strategy network structure and train the PPO algorithm strategy network structure; Step 3: Use the trained PPO algorithm strategy network structure to interact with the hypersonic aircraft environment to achieve hypersonic aircraft attitude control.

2. The hypersonic vehicle adaptive attitude control method based on proximal strategy optimization according to claim 1 is characterized in that: The longitudinal dynamics model of the air-breathing hypersonic vehicle established in step 1 is: In the formula, there are 5 state variables X = [V, h, γ, α, Q] T , V, h, γ, α and Q represent the speed, altitude, track angle, angle of attack and pitch velocity of the aircraft respectively; m and g represent the mass and gravitational acceleration of the aircraft respectively, I yy represents the moment of inertia; T, D and L represent the engine thrust, drag and lift respectively, M yy represents the pitching moment, and the specific expression is In the formula, it contains three control inputs U=[Φ,δ c ,δ e ] T , Φ represents the fuel valve opening, δ c and δ e denote the canard deflection angle and the elevator deflection angle respectively; represents dynamic pressure; S, z T Represents parameters related to the hypersonic vehicle fuselage; C T ,C L ,C D ,C M It is an aerodynamic coefficient determined according to the specific aircraft shape and flight conditions, and is used to calculate aerodynamic forces and moments under different flight conditions.

3. The hypersonic vehicle adaptive attitude control method based on proximal strategy optimization according to claim 2 is characterized in that: The specific steps of step 2 include: Step 201: Determine the state space and action space; Step 202: Set up the intelligent agent and build the PPO algorithm strategy network framework, which mainly includes the Actor Network structure and the Critic Network structure; Step 203: The old strategy in the PPO algorithm strategy network Interact multiple times with the hypersonic vehicle longitudinal dynamics model established in step 1 to generate data for strengthening the training of the PPO algorithm strategy network structure; Step 204: Use the data generated in step 203 to train the PPO algorithm strategy network structure.

4. The hypersonic vehicle adaptive attitude control method based on proximal strategy optimization according to claim 3 is characterized in that: In step 202, the Actor Network structure is represented by its objective function: In the formula, is the probability ratio, indicating that the new and old strategies are in state s t Take action a t The probability ratio of is the generalized advantage estimate, used to estimate action a t The advantage function is γ is the discount factor, λ is the parameter that controls the bias-variance tradeoff, is the TD error, V(s) is the state value function, and ε is a hyperparameter used to control the magnitude of the policy update. It means to average all time steps t in the trajectory data Ω, clip(r t (θ), 1-ε, 1+ε) represents the confidence region constraint, and the expression is as follows:

5. The hypersonic vehicle adaptive attitude control method based on proximal strategy optimization according to claim 4 is characterized in that: In step 202, the Critic Network structure is represented by its objective function: In the formula, is the TD error, λ is the L2 regularization coefficient, and ω represents the jth weight in the Critic Network.

6. The hypersonic vehicle adaptive attitude control method based on proximal strategy optimization according to claim 5 is characterized in that: In step 203, during the interaction between the agent and the environment, the Actor Network receives the current state s t As input, and generate the corresponding action a t , action a t Will affect the environment, causing it to move to the next state s t+1 , and the environment will give the current reward r t , the data set generated by this process [s t ,a t ,r t ,s t+1 ] is stored in the mini-match and used to calculate the advantage function 7. The hypersonic vehicle adaptive attitude control method based on proximal strategy optimization according to claim 6 is characterized in that: Step 204 includes the following steps: Step 2041: setting the PPO algorithm strategy network parameters and confidence region, and initializing the simulation environment of the hypersonic aircraft, including aerodynamic parameters, initial states of control surfaces, and simulation time steps; Step 2042: Execute the current PPO algorithm policy network in the initialized simulation environment, run multiple time steps to collect samples, and store the collected aircraft attitude control data, including the state, action, reward, next state and completion flag of each time step; Step 2043: Use the Critic network to predict the current state and value state, and calculate the TD error and generalized advantage estimate This step is performed once per training cycle to normalize the advantage function to stabilize the optimization process; Step 2044: Use the data in mini-match to analyze the objective function L CLIP and L Critic Optimize, use the pruned policy objective function to optimize the PPO algorithm policy network parameters, update the Actor network and use the TD target to optimize the Critic network according to the loss function; Step 2045: Adopting the experience replay mechanism, repeatedly using the same batch of training samples in each training cycle, and accelerating the training convergence process of the PPO algorithm strategy network; Step 2046: Use a validation set or a specific evaluation metric to measure the performance of the PPO algorithm policy network. After meeting performance expectations or following the evaluation cycle, save the current policy network parameters for further analysis and deployment.

8. The hypersonic vehicle adaptive attitude control method based on proximal strategy optimization according to claim 7 is characterized in that: During the training process of the PPO algorithm strategy network, by designing a reasonable reward function, the agent learns the desired actions, which will be passed to the target aircraft as elevator control instructions, thereby achieving control of the aircraft. Considering the physical background of the aircraft, the following infinite time domain objective function is established: Where the previous step distance pre_distance=|θ t-1 -θ desired |, Current gap now_distance=|θ t -θ desired |。 9. The hypersonic vehicle adaptive attitude control method based on proximal strategy optimization according to claim 8 is characterized in that: In step 2402, exploratory noise is introduced into the collected aircraft attitude control data, and exploratory sample data is generated by interacting with the hypersonic aircraft environment to jump out of the local optimal strategy, which specifically includes: Given a state s t Under this condition, the action distribution dist obtained by the PPO algorithm policy network is Beta(α,β), and action a is sampled from the distribution t ~Beta(α,β), and restricting the actions to the range of [0, 1]. Introducing exploration noise can help the agent explore the environment during training, find better strategies, and escape from the local optimal solution.

10. The hypersonic vehicle adaptive attitude control method based on proximal strategy optimization according to claim 9, characterized in that: The specific steps of step 2045 include: (1) Define an experience replay buffer to store the transfer samples collected during the interaction between the agent and the environment. Each transfer sample includes the current state, action, reward, next state, and termination flag. Set the buffer capacity of the experience replay buffer. If the buffer capacity is exceeded, the new sample will overwrite the earliest added sample. (2) During the interaction between the agent and the environment, the transfer samples of each step are stored in the experience replay buffer; (3) During each training process, a batch of transfer samples is randomly sampled from the experience replay buffer, and the batch sample size is set to represent the number of transfer samples sampled from the buffer each time; (4) Using the transfer sample batches sampled from the experience replay buffer, calculate the loss function and perform gradient descent to update the parameters of the PPO algorithm policy network; (5) The transfer samples stored in the experience replay buffer can be reused in the training step.

Images