A stall recovery method for an aircraft based on a proximal policy optimization algorithm

Abstract

The present application belongs to the field of aviation technology, and relates to a kind of airplane stall recovery method based on proximal policy optimization algorithm.The present application collects the different trajectories of the airplane by running flight simulator, and the improved proximal policy optimization agent uses the flight trajectory of the airplane to learn the control strategy, and the state parameters of the airplane are transmitted to the agent through UDP protocol.The control strategy of the agent is transmitted to the flight simulator through UDP, and the ratio of beta distribution parameters is calculated using beta distribution to provide help for training planning.A smooth airplane stall recovery reward function is designed to improve the training efficiency of the system.Finally, combined with adaptive learning rate, the sample efficiency of flight trajectory is fully utilized to improve the learning ability of the system.

Description

Technical Field

[0001] This invention belongs to the field of aviation technology and relates to aircraft stall recovery based on deep reinforcement learning, and particularly to an aircraft stall recovery method based on a near-end policy optimization algorithm. Background Technology

[0002] Aircraft generate lift during flight through relative motion with the air. The relatively smooth flow of air over the wings creates lift. In level flight, lift balances gravity. Aircraft speed directly affects the lift coefficient. As speed decreases, the lift coefficient drops rapidly. Angle of attack and altitude are also related to the lift coefficient. As the angle of attack increases, the lift coefficient increases. However, when the angle of attack exceeds a certain value, the airflow near the wing becomes less smooth, and air begins to separate prematurely from the wing surface at the boundary layer. If the angle of attack continues to rise, the lift coefficient suddenly decreases, resulting in a sudden decrease in lift. Lift and gravity lose their balance, and the aircraft's altitude drops sharply. At sufficiently high angles of attack, the aircraft may stall at any speed and altitude. Therefore, the most important factor influencing aircraft stall is the angle of attack. Different types of aircraft, due to differences in airfoil and control surfaces, may have different stall angles of attack.

[0003] The primary goal of stall recovery is to reduce the aircraft's angle of attack and achieve a safe speed in the shortest possible time.

[0004] Stall recovery from an aircraft is a continuous control problem, and reinforcement learning is a good solution for solving continuous decision-making problems. The initial state of the aircraft at the time of stall is used as the first state after the reinforcement learning network's environment is initialized. Then, the output value (i.e., the action value) of the reinforcement learning network is used as the control input on the aircraft's control stick. After the aircraft obtains a new control input, it updates its current state and obtains a new state. At this point, the environment will give a reward value for this control decision based on the new state of the aircraft. Subsequently, this new state is used as the next input value for the reinforcement learning network. In each cycle, the aircraft will complete the entire process of recognizing the current state, executing control commands, reaching the next state, and obtaining a reward value. The purpose of the reinforcement learning network is to maximize the cumulative reward value obtained by the aircraft in each round. By designing a reasonable reward function, the network can be guided to make decisions that are beneficial to stall recovery. In this way, the purpose of stall recovery from an aircraft is achieved using reinforcement learning.

[0005] The literature employs a deep deterministic policy gradient algorithm for stall recovery, while other literature uses a double-delay deep deterministic policy gradient algorithm. Algorithms such as deep deterministic policy gradient and double-delay deep deterministic policy gradient fall under the category of off-track learning. These algorithms address the problem of high sample complexity by storing samples in a replay buffer, which allows data to be reused to compute multiple policy updates. The ability to reuse samples improves learning speed. However, this reuse process can cause the data distribution to deviate from the distribution generated by the current policy. This distribution shift invalidates the standard performance guarantees used in the policy method and may lead to instability in the training process. The learning efficiency of off-track learning algorithms differs significantly from theoretical analysis results. The literature using deep deterministic policy gradient algorithms does not actually solve the stall recovery problem; they actually add an imitation learning stage to achieve stall recovery. Furthermore, the literature employs a two-stage, double-delay deep deterministic policy gradient algorithm for policy recovery, which has a very complex configuration and cannot guarantee the generalization ability of the policy.

[0006] Proximal policy optimization (PFO) deep reinforcement learning algorithms fall under the category of on-orbit learning. PFO updates network parameters using samples generated by the current policy, which theoretically guarantees the stability of policy training. However, due to its on-orbit learning nature, PFO exhibits high variance during reinforcement learning, requiring the collection of a large number of flight trajectories to address this issue. Therefore, traditional PFO methods suffer from high sample complexity and a relatively slow learning process. Summary of the Invention

[0007] The main objective of this invention is to propose an aircraft stall recovery method based on a near-end strategy optimization algorithm to solve the above problems.

[0008] The technical solution of this invention is a method for aircraft stall recovery based on a near-end strategy optimization algorithm, which specifically includes the following steps:

[0009] Step 1: Input the initial parameters of the agent: Based on the aircraft stall conditions, induce the aircraft into a stall state; simultaneously, set the initial parameters. The near-end policy optimization algorithm adopts an actor-critic framework, initializing both the actor network and the critic network separately, using orthogonal initialization for both networks; the learning rate is set to l. rate =0.00003; Adaptive coefficients are initialized to l adaptive =0.01.

[0010] Step 2: The agent generates a strategy for the flight simulator based on the improved beta distribution;

[0011] Step 3: Based on the control strategy provided by the intelligent agent, the flight simulator obtains different flight trajectories, and the flight trajectories are stored in the trajectory storage pool;

[0012] Step 4: Calculate the agent's advantage estimation function;

[0013] Step 5: Based on the trajectory re-entry sum-value function estimation results, optimize the near-end strategy;

[0014] Step 6: Fit the value function using the gradient descent method;

[0015] Step 7: Update the learning rate based on trajectory reuse data;

[0016] Step 8: Based on the flight trajectories obtained from different flight strategies, perform iterative learning and determine whether the aircraft has recovered from a stall; the recovery condition is that the aircraft's angle of attack is 6° > α. A >3°; Pitch angle 5° >θ A >3°; Roll angle 1° >φ A >-1°.

[0017] The improved beta distribution in step 2 is πk=π(θ) k ), through πk=π(θ k Collect the corresponding trajectory D k =τ i .

[0018] The control strategy provided by the intelligent agent in step 3 is as follows: Where Γ(·) is the gamma function, and the random variable x follows a beta distribution with parameters α and β.

[0019] The specific steps in step 4, calculating the agent's advantage estimation function, are as follows: in The value function obtained using the generalized advantage estimator; V(s) t ) is in state s t The value function at time.

[0020] Step 5 is based on a method that combines trajectory reentry and value function estimation. Where 0 < λ < 1 are hyperparameters; It is the value of the value function at time t; the parameter k controls the larger bias-variance tradeoff of the estimator. A larger value leads to the estimator being closer to the empirical return and having less bias and greater variance.

[0021] The method for near-end policy optimization in step 5 is the Adam (Adaptive Moment Estimation) optimization algorithm. The Adam optimization algorithm uses the same learning rate for all parameters and adapts independently as learning progresses.

[0022] Step 6 involves fitting the value function using the gradient descent method, specifically as follows: Where |D k | represents the number of trajectories; T represents the number of interactions between the agent and the flight simulator; V represents the number of interactions between the agent and the flight simulator. φ The value function in state s t The value at time; To enter state s t Expected rewards at that time.

[0023] Step 7, updating the learning rate based on trajectory reuse, specifically involves: Among them l rate This is the initial value for the learning rate; l abaptive δ is the adaptive coefficient; d The distance between trajectories; ∈ i The threshold for trajectory distance is set to 0.6.

[0024] The trajectory reuse is as follows: based on the learning rate update result in step 7, the updated learning rate is used in the optimization process in step 7.

[0025] The learning method in step 8 is as follows: determine whether the aircraft has recovered from the stall based on the recovery conditions. If the recovery conditions are met, the agent obtains the maximum reward value of 1 for this round of learning.

[0026] The beneficial effects of this invention are as follows: This invention collects different aircraft trajectories by running a flight simulator. An agent based on an improved proximal policy optimization learns control strategies using these flight trajectories. The aircraft's state parameters are transmitted to the agent via UDP. The agent's control strategy is transmitted to the flight simulator via UDP. A beta distribution is used to calculate the ratio of beta distribution parameters, thus aiding in training planning. A smooth aircraft stall recovery reward function is designed to improve system training efficiency. Finally, an adaptive learning rate is combined to fully utilize the sample efficiency of the flight trajectories, enhancing the system's learning ability. The proximal policy optimization algorithm outputs a random strategy, traditionally using a Gaussian distribution learning approach. Since the control range of the aircraft control surfaces is limited to a certain interval, and the Gaussian distribution has a probability distribution from negative infinity to positive infinity, this application uses a beta distribution to calculate the ratio of distribution parameters. Combined with an adaptive learning rate, the sample efficiency of the flight trajectories is fully utilized to improve the system's learning ability. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of an aircraft stall recovery system.

[0028] Figure 2 This is a block diagram of the improved near-end policy optimization algorithm. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings:

[0030] The entire reinforcement learning system environment of this invention consists of a set of states S provided by the aircraft, a set of actions A provided by the agents to the aircraft, and an initial state distribution p(s) describing the entire system. i The reward function is in For the reward. The transition probability is p(s). t +1|s t a t ), where the discount factor γ∈[0,1] is used to control the reward discount at different step sizes.

[0031] The agent's policy π is a mapping from states to action distributions. Each training session begins with sampling initial samples s0. At each time step t, the agent generates an action based on the current state: at ~ π(·|s t Subsequently, the intelligent agent received a reward r. t =r(s t a t ), and the new state of the environment. t+1 s t+1 From the distribution p(·|s t a t Sampling in ) . The total discount of future rewards, also known as returns, is defined as

[0032]

[0033] The goal of intelligent agents is to maximize their expected returns. The expectation here is based on dynamic response, replacing the initial state distribution, strategy, and environment transformation.

[0034] Based on the above dynamics, the Q function or action-value function is defined as follows:

[0035]

[0036] The V function (or state-value function is defined as follows)

[0037] A π (s t a t )=Q π (s t a t )-V π (s t (3)

[0038] This function is the dominance function, used to represent the action 'a' at this point. t Compared to policy π in state s t The quality of average actions taken in the process.

[0039] Generalized advantage function estimation extends the advantage function. Let V be an approximator of the value function of a certain policy, i.e., V ≈ V π ,but

[0040]

[0041] This function is a k-step return estimator. The parameter k controls the estimator to have a larger bias-variance tradeoff. A larger value results in an estimator that is closer to the empirical return and has less bias and greater variance. The generalized advantage estimator (GAE) [2] is the method that combines multi-step returns as described below.

[0042]

[0043] Where 0 < λ < 1 are hyperparameters. The advantage can be estimated using the following formula:

[0044]

[0045] The value of this estimator can be calculated in linear time for all states encountered in a plot.

[0046] During training, a flight simulator was first used to simulate aircraft stall conditions under various altitudes and speeds. By changing the elevator control parameters to increase the aircraft's angle of attack, the aircraft entered a stall state when the angle of attack exceeded the critical angle of attack. If effective control was not implemented during the stall, the aircraft would continue to descend, and its altitude would decrease. Simultaneously, due to the influence of asymmetrical airflow, the aircraft might also enter a more dangerous spin.

[0047] An improved near-end policy optimization algorithm receives the aircraft's state from the flight simulator, calculates the corresponding control policy through a policy network, and transmits it to the aircraft in the flight simulator via the UDP network protocol. The aircraft performs corresponding control based on the policy provided by the improved near-end policy optimization, and obtains an effective aircraft stall recovery strategy by exploring and utilizing different control policies.

[0048] Figure 1 The diagram illustrates the aircraft stall recovery system. Different aircraft trajectories can be collected by running a flight simulator. An agent based on an improved near-end policy optimization learns control strategies using these flight trajectories. Aircraft state parameters are transmitted to the agent via UDP; the agent's control strategy is transmitted to the flight simulator via UDP.

[0049] During the initialization phase of stall recovery, the aircraft increases its angle of attack by controlling the elevators, subsequently entering a stall state. The agent outputs control commands to the flight simulator via a beta policy and calculates the reward function and other parameters during flight.

[0050] Figure 2This is an improved deep reinforcement learning algorithm for proximal policy optimization. The algorithm framework mainly consists of initialization and learning processes. The learning process is the core of the improved proximal policy optimization algorithm.

[0051] In the initialization of the improved near-end policy optimization, it is necessary to initialize network parameters, value parameters, learning rate, adaptive coefficients, policy parameters, etc.

[0052] The proximal policy optimization adopts the critic-actor framework. The critic network can adopt a 5-layer structure, and its activation layer uses the hyperbolic tangent function; the actor network adopts a 4-layer structure, where the first and second layers are fully connected networks, and the third and fourth layers output the alpha and beta parameters of the beta distribution, respectively, and the activation layer uses the hyperbolic tangent function.

[0053] The critic and actor networks are initialized using orthogonal initialization. This initialization method is beneficial for function learning. The first state variable received by the control network after environment initialization is the initial state. The table below lists some of the initial conditions used in training; the remaining conditions can be calculated based on the given conditions and the aircraft's state at stall time.

[0054] During the initialization phase of training, the initial settings for each state of the aircraft are as follows:

[0055] Angle of attack αA 3°~6°

[0056] Pitch angle θA 3°~5°

[0057] Roll angle φA -1°~1°

[0058] When the aircraft is in level flight, the elevator is controlled to increase the aircraft's angle of attack until it exceeds the critical angle of attack, at which point the aircraft enters a stall state. The intelligent agent obtains effective recovery strategies through exploration and utilization.

[0059] Aircraft Flight Status Space Settings

[0060] In the PPO reinforcement learning process for aircraft stall recovery, the variables in the reinforcement learning state space consist of 10 aircraft flight state variables: airspeed VA, angle of attack αA, sideslip angle βA, roll angle φA, pitch angle θA, yaw angle ψA, roll rate pA, pitch rate qA, yaw rate rA, and vertical speed Vh.

[0061] During the training of a reinforcement learning network, the state information is represented as a 10-dimensional vector:

[0062] (VA,αA,βA,φA,θA,ψA,pA,qA,rA,VhA)(7)

[0063] Wherein, airspeed V is the relative speed between the aircraft and the air, and is also the speed displayed on the airspeed indicator; angle of attack αA is the projection of the velocity vector onto the longitudinal plane of symmetry, and the angle between it and the longitudinal axis of the aircraft, with the projection being positive when it is below the aircraft's coordinate axis; sideslip angle βA is the angle between the velocity vector and the aircraft's plane of symmetry, with the projection being positive when it is to the right of the aircraft's plane of symmetry; roll angle φA is the angle between the z-axis of the aircraft's coordinate system and the vertical plane passing through the x-axis, with right roll being positive; pitch angle θA is the angle between the x-axis of the aircraft's coordinate system and the ground plane, with pitch being positive; yaw angle ψA is the angle between the aircraft's x-axis and the ground plane. The x-axis of the coordinate system is projected onto the ground plane at an angle between itself and the xg-axis of the ground, with rightward deviation being positive; the roll rate pA is the projection of the rotational angular velocity of the body coordinate system relative to the ground coordinate system onto the x-axis, and is positive in the same direction as the x-axis; the pitch rate qA is the projection of the rotational angular velocity of the body coordinate system relative to the ground coordinate system onto the y-axis, and is positive in the same direction as the y-axis; the yaw rate rA is the projection of the rotational angular velocity of the body coordinate system relative to the ground coordinate system onto the z-axis, and is positive in the same direction as the z-axis; the vertical velocity Vh is the velocity of the aircraft perpendicular to the ground, with upward deviation being positive.

[0064] Action construction based on beta distribution

[0065] In a stall recovery scenario, motion control consists of three components: elevator operation, used to control the aircraft's pitch maneuvers, with an action range of [-1,1]; aileron control, used to control the aircraft's roll maneuvers, with an action range of [-1,1]; and throttle operation, used to control the aircraft's throttle power [0,1]. These actions are within a normalized motion space. The algorithm calculates the normalized motion, which requires further control surface transformation based on the actual control surface parameters to obtain the actual control surface angles.

[0066] In stochastic policy algorithms, the traditional approach to action generation is through Gaussian distributions. Due to the ease of sampling, well-defined parameters, and simple gradient calculation, Gaussian distributions have been extensively studied and widely used as a stochastic policy for continuous control. They have become the primary choice for stochastic action spaces. However, in most continuous control reinforcement learning applications, physical limitations restrict actions to a finite range. During aircraft stall recovery, each control surface faces practical constraints, contradicting the Gaussian distribution's assumption that action sampling is possible from negative infinity to positive infinity. This introduces a non-negligible bias caused by boundary effects. Because of this error, the learned policy may differ from the actual policy required.

[0067] To address this issue, a beta distribution can be used for sampling [2,3,4]. The beta distribution is highly expressive and utilizes two easily interpretable parameters. Beta describes the initial belief in the probability of success for each trial.

[0068] Although the beta distribution can partially solve the problems associated with the Gaussian distribution, directly using the beta distribution still results in a discrepancy between the Fisher information matrix and the actual variance distribution, thus affecting the learning process [3,4]. To address this issue, this invention utilizes the annealing algorithm. In the early stages of near-end policy gradient learning, the policy uncertainty is relatively high; therefore, the beta distribution should have a wider range to increase the policy exploration capability. When the random policy reaches a certain level, it is necessary to utilize more of the area near the known policy. To this end, this invention introduces annealing parameters to control the parameter learning process of the beta distribution.

[0069] The beta probability distribution can be described by two parameters α and β:

[0070]

[0071] Where Γ(·) is the gamma function, and the random variable x follows a beta distribution with parameters α and β. The expected value of the beta probability distribution is:

[0072]

[0073] The variance is:

[0074]

[0075] In the near-end policy optimization process, this invention introduces the ratio parameters α and β as annealing parameters during the learning process. In the early stages of near-end policy optimization, it is desirable for the algorithm to explore more, thus making α and β closer together. During policy learning, as the parameters are continuously optimized, more policy utilization is needed, allowing α and β to differ more significantly. Here, we define:

[0076]

[0077] Where ε beta For small real numbers, we define it as 0.001. It's the difference between α and β. During the learning process, utilize... Control the learning rate.

[0078] Reward function construction

[0079] The reward function primarily reflects the intelligent agent's objective. Here, the intelligent agent's objective is to recover the aircraft from a stall state. Consider the initial state s0 after stall recovery, the action s0 taken, and the resulting state s0. end This, along with combinations of other random variables, forms the basis for considering the reward function setting. These data enable the agent's interaction with the environment to form a Markov Decision Process (MDP).

[0080] During aircraft stall recovery, an analysis should be conducted to identify both desired and undesirable events. The relationship between system states and these events should be extracted to determine the appropriate reward function.

[0081] For aircraft stall recovery, the PPO reinforcement learning reward function is designed to guide the network to make optimal decisions, thereby controlling the aircraft to complete the recovery. In actual training, 10 variables will be designed, including airspeed V, angle of attack α, sideslip angle β, roll angle φ, pitch angle θ, yaw angle ψ, roll rate pA, pitch rate qA, yaw rate rA, and vertical speed Vh.

[0082] The reward function should be designed as a series of functions related to the state variables. To make the target network more likely to converge, the reward function related to each state variable should be uniform, and the reward function should take the following form:

[0083]

[0084] in

[0085]

[0086] in It is the current observation of the state; The control target value for this state; b i To adjust the parameters; i is the code for the corresponding state.

[0087] According to the reward function setting in formula (1), the closer the current aircraft state observation is to the target state value, the greater the reward value obtained by this observation. Meanwhile, the change in reward value is smooth overall, and the second derivative of the function value increases rapidly near the target value. This can help the near-end optimization algorithm obtain a larger reward difference when making decisions near the target value, making it easier for the network to converge to the optimal point. Formula (2) can be adjusted by b... i This is used to modify the positive action reward. A positive action reward means that if the current decision makes the state component closer to the target value than the previous decision, a positive reward is given; otherwise, a negative penalty is incurred. This guides the agent to tend to make positive decisions in each state. In this reward function, the scaling factor b is adjusted... i This allows us to change the intersection point of the curve and the x-axis; by adjusting the position of the intersection point, we can partially replace positive action rewards while avoiding the drawback of positive action rewards oscillating around the target value. Experiments have shown that the optimal training effect can be obtained when the intersection point is around (40, 0), and the corresponding scaling factor is b. i =0.02.

[0088] Formula (1) also normalizes the reward value to [-1, 1]. Normalizing the reward value helps reduce the impact of different scalar values ​​of individual rewards on the overall reward. It also makes it easier to set different weights for rewards with different parameters, and to set more explicit reward functions and weights.

[0089] The total reward is the weighted sum of the rewards related to each state component:

[0090]

[0091] Where w i This represents the weight of the i-th state component. During actual training, these weights are adjusted according to different training stages. In actual training, 10 state variables are calculated: airspeed VA, angle of attack αA, sideslip angle βA, roll angle φA, pitch angle θA, yaw angle ψA, roll rate pA, pitch rate qA, yaw rate rA, and vertical velocity Vh. Subsequently, the reward function values ​​from each training round are summed to obtain the final reward function value.

[0092] Value function estimation:

[0093] Based on the mean squared error, a value function is fitted using regression analysis (usually using the gradient descent algorithm).

[0094]

[0095] Flight trajectory retraining

[0096] Flight trajectory retraining Figure 2 This is commonly referred to as trajectory reentry. It primarily addresses the issue of low trajectory utilization efficiency in traditional near-end optimization strategies.

[0097] Traditional proximal policy optimization (PFO) algorithms can employ a fixed learning rate. If the flight paths used for reinforcement learning are abundant, a smaller learning rate allows PFO to maintain relatively stable policy updates throughout the reinforcement learning process. However, high variance is a significant problem in reinforcement learning, especially for in-orbit learning approaches like PFO. A greater number of flight paths are generally needed to accurately estimate the objective as the true objective.

[0098] In this embodiment, the initial learning rate of the reinforcement learning algorithm can be determined to be 0.00003. The proximal policy optimization algorithm uses a pruning mechanism during updates. When the learning rate of the policy update is extremely small, the learning rate only approximately constrains the penalty term in the corresponding policy improvement lower bound. Because each probability ratio starts from the center of the pruning range, the pruning mechanism has no effect at the beginning of each policy update. The beta distribution policy actor used in this invention can ensure that the pruning mechanism starts from the center, just like the Gaussian distribution policy. If the learning rate is too large, the initial gradient step of the policy update may cause the probability ratio to far exceed the pruning range. In addition, the sensitivity of the probability ratio to gradient updates may change as training progresses, indicating that the learning rate may need to change over time so that the pruning mechanism approximately enforces the total change distance from the trust region throughout the training process.

[0099] After determining the initial learning rate, during reinforcement learning, the expected total variation of interest is approximated based on an estimate of the samples. If the estimated total variation distance δ... d >>∈ i The learning rate will adaptively decrease. In this embodiment, the decrease is based on the estimated total variation δ. d <<∈ i We will increase the learning rate. This approach more closely links the implementation of PPO to the lower bound of policy improvement on which the algorithm is based. Furthermore, the adaptive learning rate prevents large policy updates that could lead to instability, while increasing the learning speed during policy updates.

[0100] The learning rate is updated based on trajectory reuse as follows:

[0101]

[0102] Among them l rate This is the initial value for the learning rate; l adaptive δ is the adaptive coefficient; d The distance between trajectories; ∈ i The threshold for trajectory distance is set to 0.6.

[0103] Training ended

[0104] Reaching the maximum number of control output steps: The current cycle will automatically end after the number of aircraft control steps reaches a given number (specified by the hyperparameter max_step). If the last state still does not meet the stall recovery criteria, the control will be considered a recovery failure.

[0105] The given stall recovery success condition is met: if the control output steps do not reach the maximum number of execution steps in one inner loop and the aircraft state reaches the given stall recovery success state value, then this control is considered a successful recovery.

[0106] After successfully modifying the aircraft control settings, the parameters are set as follows:

[0107] Angle of attack αA 3°~6°

[0108] Pitch angle θA 3°~5°

[0109] Roll angle φA -1°~1°.

Claims

1. A method for aircraft stall recovery based on a near-end strategy optimization algorithm, characterized in that: Specifically, the following steps are included: Step 1: Input the initial parameters of the agent: Based on the aircraft stall conditions, induce the aircraft into a stall state; simultaneously, set the initial parameters. The near-end policy optimization algorithm adopts an actor-critic framework, initializing both the actor network and the critic network separately, using orthogonal initialization for both networks; the learning rate is set to... ; The adaptive coefficients are initialized to ; Step 2: The agent generates a strategy for the flight simulator based on the improved beta distribution; Step 3: Based on the control strategy provided by the intelligent agent, the flight simulator obtains different flight trajectories, and the flight trajectories are stored in the trajectory storage pool; Step 4: Calculate the agent's advantage estimation function; Step 5: Based on the trajectory re-entry sum-value function estimation results, optimize the near-end strategy; Step 6: Fit the value function using the gradient descent method; Step 7: Update the learning rate based on trajectory reuse data; Step 8: Based on the flight trajectories obtained from different flight strategies, perform iterative learning and determine whether the aircraft has recovered from a stall; the recovery condition is the aircraft's angle of attack. Pitch angle Roll angle .

2. The aircraft stall recovery method based on the near-end strategy optimization algorithm as described in claim 1, characterized in that, The improved beta distribution in step 2 is as follows: ,pass Collect corresponding trajectories .

3. The aircraft stall recovery method based on the near-end strategy optimization algorithm as described in claim 1, characterized in that, The control strategy provided by the intelligent agent in step 3 is as follows: ;in It is a gamma function, a random variable. Obtain the parameter as The beta distribution.

4. The aircraft stall recovery method based on the near-end strategy optimization algorithm as described in claim 1, characterized in that, The specific steps in step 4, calculating the agent's advantage estimation function, are as follows: ,in The value function obtained using the generalized advantage estimator; It is in state The value function at time.

5. The aircraft stall recovery method based on the near-end strategy optimization algorithm as described in claim 1, characterized in that, Step 5 is based on a method that combines trajectory reentry and value function estimation. ,in It's a hyperparameter; Is a value function in Values ​​at any time; parameters A larger bias-variance tradeoff in controlling for the estimator results in an estimator that is closer to the empirical return and has less bias and greater variance.

6. The aircraft stall recovery method based on the near-end strategy optimization algorithm as described in claim 1, characterized in that, The method for near-end policy optimization in step 5 is the Adam (Adaptive Moment Estimation) optimization algorithm. The Adam optimization algorithm uses the same learning rate for the parameters and adapts independently as learning progresses.

7. The aircraft stall recovery method based on the near-end strategy optimization algorithm as described in claim 1, characterized in that, Step 6 involves fitting the value function using the gradient descent method, specifically as follows: ,in The number of trajectories; The number of interactions between the intelligent agent and the flight simulator; Value function in state The value at time; To enter the state Expected rewards at that time.

8. The aircraft stall recovery method based on the near-end strategy optimization algorithm as described in claim 1, characterized in that, Step 7, updating the learning rate based on trajectory reuse, specifically involves: ,in This is the initial value for the learning rate; These are adaptive coefficients; The distance between trajectories; The threshold for trajectory distance is set to 0.

6.

9. The aircraft stall recovery method based on the near-end strategy optimization algorithm as described in claim 1, characterized in that, The trajectory reuse is as follows: based on the learning rate update result in step 7, the updated learning rate is used in the optimization process in step 7.

10. The aircraft stall recovery method based on the near-end strategy optimization algorithm as described in claim 1, characterized in that, The learning method in step 8 is as follows: determine whether the aircraft has recovered from the stall based on the recovery conditions. If the recovery conditions are met, the agent obtains the maximum reward value of 1 for this round of learning.

Images