A reinforcement learning-based method for aircraft stall recovery at high angle of attack

Abstract

This invention belongs to the field of artificial intelligence technology and relates to a method for aircraft stall recovery at high angles of attack based on reinforcement learning. Building upon the TD3 algorithm, this invention employs a reward function that better reflects the actual aircraft stall scenario. In a simulation environment, a reinforcement learning interactive environment is constructed based on the elements of the reinforcement learning algorithm. Simultaneously, a reward mechanism based on a hybrid model is used. Within this interactive environment, the reinforcement learning algorithm interacts with the simulation environment to complete training, solidifying the obtained model parameters to form a stall recovery algorithm model; achieving effective stall recovery control. This invention utilizes reinforcement learning algorithms, improving the TD3 algorithm, and designing relevant parameters and models for aircraft stall recovery scenarios to achieve intelligent aircraft stall recovery methods in a simulation environment and improve recovery efficiency.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to a method for aircraft stall recovery at high angle of attack based on reinforcement learning. Background Technology

[0002] When an aircraft flies at a certain speed and its angle of attack exceeds the critical angle of attack, the aircraft's lift coefficient decreases, and the lift and gravity become unbalanced, potentially causing a stall. An aircraft stall can lead to dangerous situations, even fatal accidents, posing a serious threat to flight. For a long time, stall recovery has been a major focus of research in aviation safety. Modern aircraft employ numerous methods to recover from stalls and maximize crew safety.

[0003] Stall recovery is related to many factors and is very complex. The Boeing 737 Max is equipped with a stall prevention system called MCAS (Manuever Characteristics Augmentation System). MCAS is activated when the aircraft is in autopilot mode, flaps are retracted, and the crew has not issued stabilizer trim commands, and the angle of attack exceeds the critical angle of attack. The critical angle of attack is calculated based on airspeed and altitude. MCAS outputs a stabilizer trim command that causes the aircraft to pitch down, controlling the stabilizer in high-speed mode and enhancing the pitch control system. Simultaneously, after MCAS activation, the trim command bypasses the stop stick command and is sent directly to the trim limit switch. Therefore, even if pulling the stick cannot cut off the nose-down trim signal from MCAS, the aircraft will still be able to recover from a stall.

[0004] Besides the Boeing 737 Max, Airbus aircraft are also equipped with angle-of-attack protection devices. These devices instruct the aircraft to pitch down at high angles of attack. If the angle-of-attack detector malfunctions at this point and fails to send a signal to reduce the angle of attack, the aircraft control computer may interpret this as a failure to achieve the intended effect, leading to further instructing the aircraft to pitch down, ultimately resulting in a sustained pitch and a fatal accident.

[0005] Due to the complexity of aircraft stalls, automatic control systems cannot completely prevent them. In some situations, what may not seem dangerous can lead to catastrophic failure and loss of life due to misjudgments by automatic stall protection devices. With the development of intelligent technologies, particularly reinforcement learning, computers have made significant progress in perceptual representation and policy learning. To fully utilize these advancements, we investigate the application of reinforcement learning to address aircraft stall problems.

[0006] Reinforcement learning, a technique for intelligent agents to learn strategies through interaction with their environment, shows great promise in solving aircraft stall recovery problems. Reinforcement learning acquires better strategies through the interaction between the agent and its environment.

[0007] The primary goal of aircraft stall recovery is to reduce the angle of attack and reach a safe speed as quickly as possible, while minimizing altitude loss. In reinforcement learning, we can improve efficiency by selecting a specific neural network structure, designing a reward function, and continuously learning and training the algorithm to find the optimal recovery method. Summary of the Invention

[0008] This invention primarily utilizes reinforcement learning to build a reinforcement learning algorithm for aircraft stall scenarios, leveraging a simulation simulator to solve the aircraft stall recovery problem. Because aircraft stall testing is characterized by high cost and high risk, and relevant data is generally difficult to obtain, this invention uses a simulation simulator to input algorithm parameters. Through continuous training and learning, the algorithm parameters converge, achieving the desired recovery effect.

[0009] In terms of algorithms, this invention improves upon the TD3 algorithm by designing a reward function that better fits the actual aircraft stall scenario. It also proposes a reward mechanism based on a hybrid model, achieving effective stall recovery control and improving recovery efficiency.

[0010] The technical solution of this invention: A method for recovering from stall at high angle of attack of an aircraft based on reinforcement learning, comprising the following steps:

[0011] S1. In the simulation simulator environment, construct the reinforcement learning interactive environment based on the elements of the reinforcement learning algorithm, including the state parameter space and action space of the input algorithm.

[0012] S2. Build a reinforcement learning algorithm framework for stall recovery, including a reward function and a reward mechanism based on a hybrid model;

[0013] S3. In the reinforcement learning interactive environment, the reinforcement learning algorithm completes the training by interacting with the simulation environment, solidifies the obtained model parameters, and forms a stall recovery algorithm model.

[0014] S4. In simulated flight, when a stall occurs, the state parameters are input into the stall recovery algorithm model. The obtained action commands are output through communication interaction and executed by the simulator to gradually realize stall recovery.

[0015] The S1 state parameter space includes the simulation simulator's angle of attack, pitch angle, yaw angle, roll angle, pitch rate, and yaw rate.

[0016] The action space output by the S1 algorithm includes elevator, rudder, and throttle surface operation data.

[0017] The S2 reward function is a series of functions related to state variables, and the reward value associated with each state component is composed of a weighted sum of local and global components.

[0018] The state variables are angle of attack, pitch angle, yaw angle, roll angle, pitch rate, and yaw rate.

[0019] The reward function takes the following form:

[0020] Reward=γReward global +(1-γ)Reward local

[0021] The reward value associated with each state component is composed of a weighted sum of the local and global components. The weight γ determines the importance of the global and local components in this calculation. The global component is calculated as follows:

[0022]

[0023] The function variable state t The coefficient "state" represents the value of the agent's current state component. target This represents the target value of the agent's state component, k is the scaling factor, and ε is the function slope scaling factor. By adjusting its value, the steepness of the function slope can be controlled.

[0024] The reward mechanism based on the hybrid model uses two models to make decisions: one that focuses only on global rewards and the other that focuses only on local rewards. The two models calculate an action for the agent's current state and input these two actions into a comparator. The comparator outputs the better action, which is then used as the action that the agent actually executes in the next moment.

[0025] The global reward model is based on a global reward function.

[0026]

[0027] The constructed model, namely the neural network, calculates an action value through a global reward function.

[0028] The local reward model is based on a local reward function.

[0029]

[0030] The constructed model, namely the neural network, calculates an action value through a local reward function.

[0031] S3 further includes:

[0032] S31, Initialization, i.e., aircraft trim: Based on the aircraft control law, configure the relevant aerodynamic parameters of the aircraft to make the aircraft reach the balance of the specified state;

[0033] S32, Obtain state parameters s: State data, i.e., aircraft aerodynamic parameters, are used as input to the reinforcement learning algorithm neural network. At this point, the state data is obtained through communication with the simulation simulator.

[0034] S33, Neural network output action A: The neural network calculates and outputs the action value;

[0035] S34, Execute Action A: Send the action value to the simulator through communication with the simulator, and the simulator will execute the action;

[0036] S35, the aircraft transitions to the next state s′: Since the simulator has its own flight control system, it can provide new state data based on action values. Simultaneously, it obtains a reward value according to the reward function and reward mechanism.

[0037] S36, Store in the experience pool: Store <current state s, next state s′, action value A, reward value R> as a set of data into the experience pool:

[0038] S37, State Update: Send the latest state data of the simulator to the stall recovery algorithm based on reinforcement learning to prepare for the next round;

[0039] S38, Parameter Update: Randomly select a set of data from the experience pool and update the neural network parameters according to the data update strategy;

[0040] S39, Model Stabilization: When the algorithm training meets the termination condition, the model parameters are stored in the specified data model file format to obtain a usable algorithm model.

[0041] The beneficial effects of this invention are as follows: This invention uses reinforcement learning algorithms as a means, and by improving the reinforcement learning algorithm (TD3), it designs relevant parameters and models for aircraft stall recovery scenarios, so as to realize the intelligentization of aircraft stall recovery methods in simulation environment and improve recovery efficiency.

[0042] This invention proposes a reinforcement learning-based method for aircraft stall recovery at high angles of attack. Based on the TD3 algorithm, it adopts a reward function that is more closely related to the actual aircraft stall scenario and uses a reward mechanism based on a hybrid model to achieve effective stall recovery control, which is helpful for aircraft stall recovery research. Attached Figure Description

[0043] Figure 1 This is a schematic flowchart of a high angle-of-attack stall recovery method for aircraft based on reinforcement learning according to an embodiment of the present invention.

[0044] Figure 2 To enhance the learning interaction environment diagram;

[0045] Figure 3The image of the reward function proposed in this invention;

[0046] Figure 4 This is a flowchart of the action decision-making process based on a hybrid model proposed in this invention;

[0047] Figure 5 This is a flowchart illustrating the training process of the reward mechanism based on a hybrid model proposed in this invention.

[0048] Figure 6 This is the effect of the original TD3 algorithm;

[0049] Figure 7 To improve algorithm performance and action decision model distribution;

[0050] Figure 8 A diagram showing the effect of stalling after using exponentially weighted moving average;

[0051] Figure 9 This describes the algorithm training process. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] like Figure 1 The diagram shown is a flowchart illustrating a reinforcement learning-based method for recovering from stall at high angles of attack in aircraft, according to an embodiment of the present invention. The method includes:

[0054] S1. Construct a reinforcement learning interactive environment, including a state parameter space and an action space.

[0055] In a reinforcement learning system, an agent represents the overall decision-making process of the system. It observes its current environment, acquires its state, makes a decision based on that state, and executes an action. Subsequently, it updates its decision-dependent policy based on feedback from the environment. Figure 2 This process is described.

[0056] S2. Build a reinforcement learning algorithm framework for stall recovery, including the reward function and the reward mechanism based on the hybrid model.

[0057] S3. In the reinforcement learning interactive environment, the reinforcement learning algorithm completes the training by interacting with the simulation environment, solidifies the obtained model parameters, and forms a stall recovery algorithm model.

[0058] S4. In simulated flight, when a stall occurs, the state parameters are input into the stall recovery algorithm model. The obtained action commands are output through communication interaction and executed by the simulator to gradually realize stall recovery.

[0059] In this embodiment, the state parameters consist of seven components. During network training, the state information is represented as a seven-dimensional vector. The table below details the name and meaning of each component.

[0060] Table 1 State-space component table

[0061]

[0062] Motion control consists of three components. The table below details the name, meaning, and value range of each component.

[0063] Table 2 Motion Spatial Components

[0064] Action value name Brief explanation of the meaning of action values Range of values Pitch axis control lever Controlling the pitch maneuver of the aircraft [-1,1] Roll axis control lever Controlling the aircraft's roll maneuvers [-1,1] throttle lever Controlling aircraft throttle power [0,1]

[0065] S2 is further described as follows:

[0066] This invention proposes an improved reward function that fits the aircraft stall scenario. Based on this, it adopts a reward mechanism based on a hybrid model, drawing on the idea of ​​transfer learning, and achieves good results.

[0067] The reward function takes the following form:

[0068] Reward = γReward global +(1-γ)Reward local

[0069] The reward value associated with each state component is composed of a weighted sum of the local and global components. The weight γ determines the importance of the global and local components in this calculation. The global component is calculated as follows:

[0070]

[0071] The function variable state t The coefficient "state" represents the value of the agent's current state component. target This represents the target value of the agent's state component, where k is the scaling factor, and the coefficient ε determines the range of the function; when k = 1, state target When ε = 0 and ε = 0.5, the graph of this function is as follows: Figure 3 As shown:

[0072] As can be seen from the figure, the closer the current state value is to the target value, the larger the function value is. Moreover, the function value changes drastically when it is near the target value. This is to ensure that the reinforcement learning network can still obtain a large reward difference when making decisions near the target value, making it easier for the network to converge to the optimal point.

[0073] The calculation of local components is structured as follows:

[0074]

[0075] variable state t-1 Where represents the state component value of the agent at the time of its last decision, and the variable `state` is... t The state component value at the time of the current decision, with coefficients 'state'. target The target value of the state component is ε, and the parameter ε determines the range of the function. The main role of the local component is that if the current decision makes the state component closer to the target value than the previous decision, a positive reward is obtained; otherwise, a negative penalty is obtained. This will guide the agent to tend to make a positive decision in the current state in each decision.

[0076] The total reward is the weighted sum of the rewards related to each state component, expressed as follows:

[0077]

[0078] Weight W i This indicates the degree of importance the network attaches to the i-th state component. In the actual training process, the weights will be adjusted according to different training stages.

[0079] In actual training, the angle of attack (AoA) and vertical velocity V will be used. vartital Roll, pitch, and indicated airspeed ind These five state components are used for calculation. The final actual reward function depends on the number of training epochs:

[0080]

[0081] The function coefficients of each component are shown in the table below:

[0082] Table 3 Reward Function Parameter Settings

[0083]

[0084] Further simulation experiments revealed that although the aircraft's state curves showed a convergence trend during the recovery process, the curves still exhibited significant jitter after convergence. The reason for this jitter is that the network is affected by both global and local reward functions when making decisions. The local reward is mainly obtained by comparing the current action decision with the previous action decision relative to the target state. If the aircraft oscillates around the target state after reaching it, it can continuously obtain high local rewards, which in turn allows the network to obtain more cumulative rewards.

[0085] This invention proposes a reward mechanism based on a hybrid model, which uses two models—one focusing on global rewards and the other on local rewards—to jointly make decisions. Each model calculates an action based on the agent's current state. These two actions are then input into a comparator, which outputs the better action according to designed rules. This better action is then chosen as the action the agent actually executes in the next time step. The improved action decision is as follows: Figure 4 Methods for obtaining hybrid models include... Figure 5 As shown.

[0086] Furthermore, based on the hybrid model, an exponentially weighted moving average method is used to improve recovery efficiency. Although the state and action curves show less jitter compared to the basic model, the reward function curve shows that the aircraft cannot maintain a given target state after reaching it (i.e., the reward function curve shows a peak). Observing the action quantity curve, the rudder still exhibits slight jitter at the end of the recovery phase, thus the aircraft cannot maintain a stable state with a strict requirement of zero roll angle. To solve this problem, an exponentially weighted moving average method for smoothing filtering in signal processing is introduced to filter the rudder surface action decision sequence controlling the roll angle.

[0087] Before introducing the hybrid model mechanism, the actions output by the neural network were directly sent to the simulator, unlike actual aircraft control which involves a delay. After introducing the exponentially weighted moving average, the action quantity m sent to the aircraft... t The action 'a' currently output by the network t and the quantity m related to actions at historical moments t-1 The result is obtained by exponentially weighted summation, which approximates the actual aircraft control process. The calculation formula is as follows:

[0088] m t =γmt -1 +(1-γ)a t

[0089] A stall recovery simulation experiment was conducted using the hybrid model, and the aircraft's recovery state curve is shown below. Figure 7 As shown on the left, its performance curve is compared with that of the original TD3 algorithm. Figure 6It can be clearly seen that the amplitude of the curve has decreased significantly, and at the same time, from Figure 7 The scatter plot of the model decision (right) shows that in the early stage of stall recovery, when the state curve changes drastically, the decision-making is mainly carried out by the local model. When the curve shows a convergence trend, the decision-making subject shifts to the global model. This indicates that the local model plays a role in state convergence while the global model plays a role in state stabilization.

[0090] After applying the exponentially weighted moving average method, the aircraft's stall recovery curve is as follows: Figure 8 As shown in the figure, the aircraft's roll angle (Roll) and the control surface in that direction converge to a constant value at the end of the recovery phase. Observing its reward function curve, frequent spikes or even stable high threshold rewards appear in the later stages of the recovery phase, indicating that the aircraft is maintaining a stable state.

[0091] S3, in the reinforcement learning interactive environment, the reinforcement learning algorithm completes the training by interacting with the simulation environment, solidifies the obtained model parameters, and forms a stall recovery algorithm model.

[0092] Specifically, the training process described in S3 is as follows: Figure 9 As shown, this is a method based on an embodiment of the present invention. Figure 1 The flowchart for step S3 includes:

[0093] S31, Initialization. This refers to aircraft trim. Based on the aircraft control law, the relevant aerodynamic parameters are configured to bring the aircraft to equilibrium under the specified state. Reinforcement learning algorithm training is a multi-round cyclic process. Each round of training begins by initializing the aircraft state as the initial state input to the neural network. The initial parameter values ​​in this embodiment are shown in the table below.

[0094] Table 4 Initial State Values

[0095] Initial state name State value Angle of Attack 14.5° Indicative airspeed 100 sections high 15,000 feet Roll angle -25°～25° Aircraft weight 47000kg wind speed 0 sections

[0096] S32, Acquire state parameters. The state data, i.e., the aircraft's aerodynamic parameters, is fed into the neural network as input to the algorithm. At this stage, the state data is acquired through communication with the simulation simulator.

[0097] S33, Neural Network Output Action. The neural network calculates and outputs action values.

[0098] S34, Execute Action. The action value is sent to the simulator via communication with the simulator, and the simulator executes the action.

[0099] S35, State Transition. Since the simulator has its own flight controller, it can provide new state data based on action values, and simultaneously obtain reward values ​​according to the reward function and reward mechanism.

[0100] S36, Store the experience pool. Store <current state, next state, action value, reward value> as a set of data into the experience pool.

[0101] S37, State Update. The simulator sends the latest state data to the algorithm to prepare for the next round.

[0102] S38, Parameter Update. Randomly select a set of data from the experience pool and update the neural network parameters according to the data update strategy.

[0103] S39, Model Stabilization. When the algorithm training meets the termination condition, the model parameters are stored in a specified data model file format, resulting in a usable algorithm model.

[0104] Training termination conditions:

[0105] 1. Reaching the maximum number of control output steps: Once the number of control steps of the aircraft reaches a given number in a certain training round, the current cycle will automatically end. If the last state still does not meet the stall recovery standard, the control will be considered a recovery failure.

[0106] 2. Meeting the given stall recovery success condition: If the control output steps do not reach the maximum number of execution steps in one inner loop and the aircraft state reaches the given stall recovery success state value, then this control is considered a successful recovery. Table 5 is the stall recovery success state value table.

[0107] Table 5. Successful Stall Recovery Status Values

[0108] Aircraft status items Given a value Angle of Attack 3°～6° Pitch angle 3°～5° Indicative airspeed Sections 200-240 Vertical velocity -50 fpm~50 fpm Roll angle -1°～1°

[0109] S4. In simulated flight, when a stall occurs, the state parameters are input into the stall recovery algorithm model. The obtained action commands are output through communication interaction and executed by the simulator to gradually realize stall recovery.

[0110] In simulated flight, the aircraft is gradually set to a stall state. At this time, the state data can be input into the algorithm model to achieve the effect of recovering the aircraft from stall through intelligent means.

Claims

1. A method for recovering from stall at high angle of attack of an aircraft based on reinforcement learning, characterized in that, Includes the following steps: S1. In the simulation simulator environment, construct the reinforcement learning interactive environment based on the elements of the reinforcement learning algorithm, including the state parameter space and action space of the input algorithm; S2. Build a reinforcement learning algorithm framework for stall recovery: reward function and reward mechanism based on hybrid model; S3. In the reinforcement learning interactive environment, the reinforcement learning algorithm completes training by interacting with the simulation environment, and solidifies the obtained model parameters to form a stall recovery algorithm model. S4. In simulated flight, when a stall occurs, the state parameters are input into the stall recovery algorithm model. The obtained action commands are output through communication interaction and executed by the simulator to gradually realize stall recovery. The reward function is a series of functions related to state variables, and the reward value associated with each state component is composed of a weighted sum of local and global components; The reward function takes the following form: The reward value associated with each state component is obtained by weighting the local and global components, where the weights are... This determines the relative importance of global and local components in this calculation. The calculation of the global component consists of the following: Among function variables This represents the value of the agent's current state component, and the coefficient is... This represents the target value of the agent's state component. This is the scaling factor. This is the scaling factor for the function's slope; by adjusting its value, the steepness of the function's slope can be controlled. The reward mechanism based on the hybrid model uses a model that focuses only on global rewards and another model that focuses only on local rewards to make decisions together. The two models calculate an action for the agent's current state, input the two actions into a comparator, and the comparator outputs the better action, which is then used as the action that the agent actually executes in the next moment. The global reward model is based on a global reward function. The constructed model, namely the neural network, calculates an action value through a global reward function; The local reward model is based on a local reward function. The constructed model, namely the neural network, calculates an action value through a local reward function.

2. The reinforcement learning-based aircraft stall recovery method at high angles of attack according to claim 1, characterized in that, The S1 state parameter space includes the simulation simulator's angle of attack, pitch angle, yaw angle, roll angle, pitch rate, and yaw rate.

3. The reinforcement learning-based aircraft stall recovery method at high angles of attack according to claim 1, characterized in that, The action space output by the S1 algorithm includes elevator, rudder, and throttle surface operation data.

4. The reinforcement learning-based method for recovering from high angle-of-attack stall in aircraft according to claim 1, characterized in that, The state variables are angle of attack, pitch angle, yaw angle, roll angle, pitch rate, and yaw rate.

5. The reinforcement learning-based method for aircraft stall recovery at high angles of attack as described in claim 1, characterized in that, S3 further includes: S31, Initialization, i.e., aircraft trim: Based on the aircraft control law, configure the relevant aerodynamic parameters of the aircraft to make the aircraft reach the balance of the specified state; S32, Obtain status parameters State data, i.e., aircraft aerodynamic parameters, is used as input to the reinforcement learning algorithm neural network; at this time, the state data is obtained through communication with the simulation simulator. S33, Neural network output action A: The neural network calculates and outputs the action value; S34, Execute Action A: Send the action value to the simulator through communication with the simulator, and the simulator will execute the action; S35, the aircraft is transitioning to the next state. Since the simulator has a built-in flight controller, it can provide new status data based on action values. Simultaneously, it obtains reward values ​​according to the reward function and reward mechanism. S36, Storing the experience pool: <Current state> Next state Action value A and reward value R are stored as a set of data in the experience pool. S37, State Update: Send the latest state data of the simulator to the stall recovery algorithm based on reinforcement learning to prepare for the next round; S38, Parameter Update: Randomly select a set of data from the experience pool and update the neural network parameters according to the data update strategy; S39, Model Stabilization: When the algorithm training meets the termination condition, the model parameters are stored in the specified data model file format to obtain a usable algorithm model.

Images