An aircraft fault-tolerant control method based on reinforcement learning

Abstract

The application discloses an airplane fault tolerance control method based on reinforcement learning and belongs to the field of fault control. The application uses a deep Q learning network to extract features of data generated in a simulation environment, so as to identify fault information and adjust the deep Q learning network. The fault diagnosis method based on data reduces the difficulty of modeling a complex system, replaces a fault detection subsystem by extracting data features, and simplifies the deep Q learning network. The application expands the collected signals by using an enhanced algorithm SDA and then expands the training set, so as to improve the accuracy of the model and the generalization ability of the model. The application optimizes the reinforcement learning model by using the deep Q learning network, and compared with a traditional reinforcement learning method, the optimization method significantly improves the data feature extraction ability of the model.

Description

Technical Field

[0001] This invention belongs to the field of fault control, and in particular to a fault-tolerant control method for aircraft based on reinforcement learning. Background Technology

[0002] With the continuous development of aerospace technology, flight control systems are becoming increasingly complex. Simultaneously, the stability of flight control systems directly affects an aircraft's ability to complete its flight mission. Any type of malfunction can lead to control failure of the flight control system, posing a significant threat to personnel safety and equipment security. Due to the increasing complexity of flight control systems, model-based fault diagnosis and fault-tolerant control struggle to accurately model these systems, exhibiting considerable limitations. With the development of machine learning, the advantages of data-driven flight control system fault diagnosis are becoming increasingly apparent. However, research on applying reinforcement learning for aircraft fault-tolerant control remains limited. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a fault-tolerant control method for aircraft based on reinforcement learning.

[0004] To achieve the above objectives, the present invention employs the following technical solution:

[0005] A fault-tolerant control method for aircraft based on reinforcement learning includes the following steps:

[0006] Step 1: Build a simulation model of the aircraft based on open-source aircraft aerodynamic data;

[0007] Step 2: Input and define different flight states of the aircraft and the fault modes of the aircraft under fault states in the simulation environment, and collect data and fault data under different flight states in the simulation environment.

[0008] Step 3: Expand the collected data using the data augmentation algorithm SDA, and then input the expanded data into the deep Q-learning network for training to obtain the trained deep Q-learning network.

[0009] Furthermore, the training process in step 3 is as follows:

[0010] Step (301): Set the training set to empty, and collect aircraft data under different flight conditions in the simulation environment; collect the aircraft operating state S once in each sampling period. t S under different flight conditions t The data is input into a deep Q-learning network to obtain the reward value Q corresponding to the current flight state. π (s t ,a t );

[0011] Step (302), perform action a t Observe the next state s t+1 and reward R t+1 ;

[0012] Step (303): Update Q using the Bellman equation according to different flight states. new (s t ,a t This yields the maximum expected future return across all flight conditions. And update the current deep Q-learning network;

[0013] Step (304): Repeat steps (301) to (303) for the flight state prediction value of the next sampling period to obtain the maximum expected future return value under different flight states;

[0014] Step (305): The current aircraft status data and the maximum expected future return under the current flight status are... Reward R t+1 As new data samples, they are expanded into the current training dataset, and the expanded training set is used to update the current deep Q-learning network.

[0015] For each sampling period, repeat steps (301) to (305) until training is complete;

[0016] (306) Input the new aircraft state data obtained during the actual flight into the trained deep Q learning network and test whether the trained model can achieve the expected fault-tolerant control effect; if it cannot achieve the expected fault-tolerant control effect, repeat the steps (301) to (305) with the new data to retrain until the training is completed.

[0017] Furthermore, in step (301), Q π (s t ,a t The calculation formula is:

[0018] Q π (s t ,a t )=E[R t+1 +γR t+2 +γ 2 R t+3 +…|S t ,a t ]

[0019] Among them, Q π (s t ,a t ) is state S t Take action at Q value, R t γ is the reward at each time step, and γ is the discount rate; the Q function takes the state and action as input and feeds back the expected future reward.

[0020] Furthermore, the Q mentioned in step (303) new (s t ,a t The calculation formula is:

[0021]

[0022] Among them: Q new (s t ,a t )-states t and action a t The new Q value; Q(s) t ,a t ) - Current Q value; α - Learning rate; R t+1 - in state s t Take action a at the time t The reward; -in the new state t+1 And the maximum expected future return given under all possible actions.

[0023] Furthermore, the network update method described in step (303) is as follows:

[0024] Q new (s t ,a t As a new data sample, it is expanded into the current training dataset, and the current deep Q-learning network is updated using the current training dataset.

[0025] Furthermore, in step 3, the collected data is expanded using the data augmentation algorithm SD by employing a cut-and-flip method:

[0026] Five clipping and flipping methods are used: horizontal flip, vertical flip, center flip, data mirroring, and data repetition, to operate on data under the same label.

[0027] Furthermore, the expanded aircraft attitude dataset from step 3 is sorted according to time series order to form a training set.

[0028] Furthermore, the different flight states of the aircraft include normal flight state and malfunction flight state.

[0029] Furthermore, the fault modes of the aircraft under fault conditions include constant deviation, constant gain, and sensor jamming.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] This invention provides a reinforcement learning-based fault-tolerant control method for aircraft. It utilizes a deep Q-learning network to extract features from data generated in a simulation environment, thereby identifying fault information and adjusting the deep Q-learning network. Compared to traditional model-based fault-tolerant control methods, this invention's data-based fault diagnosis method reduces the difficulty of modeling complex systems. By extracting data features, it replaces the fault detection subsystem, simplifying the deep Q-learning network. This invention expands the training set by enhancing the acquired signals using the SDA algorithm, thereby improving the model's accuracy and generalization ability. This invention optimizes the reinforcement learning model using a deep Q-learning network, significantly improving the model's data feature extraction capability compared to traditional reinforcement learning methods. Specifically, a deep neural network is used to fit the Q-function to predict the reward for performing each possible operation in a given state, and an experience replay mechanism is used to increase the efficiency of training data. Furthermore, an explore-exploitation strategy is adopted to balance the trade-off between exploration and exploitation, making the deep Q-learning model more flexible and accurate, and better able to adapt to constantly changing environments. The application of this optimization method enables the deep Q-learning model to better extract data features and more accurately predict optimal actions, improving the model's real-time performance and robustness, and providing a stronger foundation for real-time system control. This invention's deep Q-learning network reinforcement learning, used in uncertain fault scenarios, overcomes the limitations of fixed policies in traditional reinforcement learning algorithms and approximates the optimal fault-tolerant policy of the current faulty system as closely as possible. Furthermore, it uses a deep Q-learning network to predict the next state of a continuous control system. Deep Q-learning is a reinforcement learning algorithm that uses deep neural networks to learn value functions, thereby predicting the reward for performing each possible action in a given state. By continuously interacting with the environment, it learns an optimal policy that maximizes cumulative rewards. The advantage of this method lies in updating the value function and policy network through an explore-and-exploit policy and using an experience replay mechanism to increase the efficiency of training data. Unlike traditional Q-learning algorithms, deep Q-learning can handle high-dimensional state spaces and discrete or continuous action spaces, and can be applied in both online and offline learning scenarios. In continuous control systems, deep Q-learning can be used to predict the next state and use these predictions to update the policy network to ensure the system's real-time performance. This method can dynamically adjust the strategy based on the current state information, enabling the system to adapt to the ever-changing environment more quickly. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating the control system of the present invention;

[0033] Figure 2This is a structural diagram of the reinforcement learning fault-tolerant control system of the present invention;

[0034] Figure 3 This is a schematic diagram illustrating the algorithm enhancement for SDA data expansion in accordance with the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] The present invention will now be described in further detail with reference to the accompanying drawings:

[0038] See Figure 1 , Figure 1 This is a flowchart of a flight fault-tolerant control method based on reinforcement learning according to the present invention. The present invention includes the following steps:

[0039] Step S1: An aircraft simulation model with a controller is built using open-source aircraft aerodynamic parameter data. Data and fault data under different flight conditions are collected from sensors in the simulation environment. The expanded dataset is then input into a deep Q-learning network for training and updates. This invention employs the SDA enhancement algorithm to expand the sensor signals, improving the model's accuracy and generalization ability. The evaluation network in this invention is optimized using a deep Q-learning network, improving the model's real-time performance and robustness.

[0040] The specific implementation steps of step 1 are as follows:

[0041] Step S101: Build an aircraft simulation model with a controller using open-source aircraft aerodynamic parameter data; input and define different flight states of the aircraft in the simulation environment, including normal flight state, actual aircraft state, and fault flight state. Define the fault modes of the aircraft under fault states, including constant deviation, constant gain, sensor jamming, and excessive noise. Input the data under different flight states into a deep Q-learning network. Collect normal and fault data of the aircraft through sensors in the simulation environment.

[0042] Step S102: Expand the data obtained from step 101 using the data augmentation algorithm SD by cutting and flipping. See [link to relevant documentation]. Figure 3 , Figure 3 To illustrate the algorithm-enhanced SDA data expansion, the specific steps for expanding the database are as follows:

[0043] There are five methods for cropping and flipping: horizontal flip, vertical flip, center flip, data mirroring, and data duplication. These are applied to data under the same label without requiring parameter selection. The number of sample points for each data point is set to N, and the sensor signal Y = [y1, y2, y3, ..., y...]. N ], where Y represents: a sensor signal consisting of N sample points, y1, y2, y3, ..., y N These represent the sample points corresponding to different sampling times. They are then divided into two equal segments, denoted as Y. L = [y1, y2, ..., y N / 2 ], where Y L Representative: The sensor signal constructed from the first N / 2 sample points of the sensor signal Y under the simulation environment, y1, y2, ..., y N / 2 They represent: the first N / 2 signal sample points of the sensor signal Y under the simulation environment and Y... R =[y N / 2+1 y N / 2+2 , ..., y N ], where Y R Representative: The sensor signal Y constructed from N / 2 sample points under simulated conditions, where y N / 2+1 y N / 2+2 , ..., y N These represent: N / 2 sensor sample points following the sensor signal Y in the simulation environment.

[0044] Horizontal flipping reverses the sensor signal horizontally along the vertical center line, which is represented as...

[0045] Y hf =[y N y N-1 y N-2 ,…,y1]

[0046] Where Y hf Representative: Constructed horizontal flip signal, y N y N-1 y N-2 y1, ..., y1 represent the sample points corresponding to different sampling times.

[0047] Vertical flipping reverses the sensor signal vertically along the horizontal center line, as shown below.

[0048] Y vf =[-y1,-y2,-y3,…,-y N ]

[0049] Where Y vf Representative: Constructed vertical flip signals, -y1, -y2, -y3, ..., -y N These represent the negative values ​​of the sample points at different sampling times.

[0050] Center flipping involves symmetrically flipping the sensor signal around its center point, i.e., simultaneously flipping it horizontally and symmetrically. This is represented as...

[0051] Y cf =[-y N , -y N-1 , -y N-2 ,…,-y1]

[0052] Where Y cf Representative: The center-flipping signal of the construction, -y N , -y N-1 , -y N-2 , ..., -y1 represent the negative values ​​of the sample points at different sampling times.

[0053] Data mirroring, symmetrical about the center point, with Y... L Or Y R Mirror copy to the other side, represented as

[0054]

[0055] Where Y Lm Representative: The sensor signal obtained by mirroring and copying the first N / 2 sensor sample points of the sensor signal Y. These represent the first N / 2 sensor sample points of the sensor signal Y and its mirror sample points, respectively.

[0056] Where Y Rm Representative: The sensor signal obtained by mirroring N / 2 sensor sample points after the sensor signal Y. These represent: N / 2 sensor sample points following sensor signal Y and their mirror sample points.

[0057] Data repetition, using the center point as the boundary, Y L Or Y R Translate and copy to the other side, represented as

[0058]

[0059] Among them, Y Ld Representative: The sensor signal obtained by translating and copying the first N / 2 sensor sample points of the sensor signal Y. These represent: the first N / 2 sensor sample points of the sensor signal Y and its translational sample points, respectively.

[0060] Among them, Y Rd Representative: The sensor signal is constructed by translating and copying N / 2 sensor sample points after the sensor signal Y. These represent: N / 2 sensor sample points after sensor signal Y and its translation sample points, respectively.

[0061] Step S2: Utilizing a deep Q-learning network, the evaluation network is updated by assessing flight and fault data under different conditions. During aircraft fault-tolerant control, the evaluation network is continuously updated, enabling the reinforcement learning fault-tolerant controller to learn and improve itself. To enhance the model's accuracy and generalization ability, this invention expands the data set. This invention incrementally optimizes the reinforcement learning method, allowing the model to progressively approach the optimal fault-tolerant control, thereby better achieving fault-tolerant control of the aircraft. Figure 2 This is a structural diagram of the reinforcement learning fault-tolerant control system of the present invention. The state data of the aircraft control system, expanded via SDA, is input into a deep Q-learning network. Through optimization and updating by the deep Q-learning network, the maximum expected future reward value is obtained. Then, the model is updated using the deep Q-learning network, and the execution network units are verified to achieve the expected fault-tolerant control effect on the aircraft. If the expected fault-tolerant effect is not achieved, the process returns to step S202 to update the training of the deep Q-learning network. Step S201: The reinforcement learning Q-learning algorithm is optimized using the Extreme Learning Machine (ELM) method. The deep Q-learning network of the reinforcement learning fault-tolerant controller is a three-layer ELM network with a single hidden layer.

[0062] Step S202: Based on the expanded data, train and update the already constructed deep Q-learning network, and optimize the model using the deep Q-learning network. The process is as follows:

[0063] Step S2021: The expanded training data samples are arranged into a training data sample set according to the time series order.

[0064] Step S2022: The agent attempts an action based on the observation of the current state using a deep Q-learning network. The Q-learning algorithm can be used to learn each value of the Q-table through the Q-function. The Q-function is defined as:

[0065] Q π (s t ,a t )=E[R t+1 +γR t+2 +γ 2 R t+3 +…|S t ,a t ]

[0066] Among them, Q π (s t ,a t ) is state S t Take action a t Q value, R t γ is the reward at each time step, and γ is the discount rate. The Q-function takes the state and action as input and feeds back the expected future reward.

[0067] Step S2023, Execute action a t And observe the next state s t+1 and reward R t+1 Record and update the rewards or penalties returned by the environment. The Q-learning algorithm updates Q(s,a) using the Bellman equation, as shown below:

[0068]

[0069] In the formula: Q new (s t ,a t )-states t and action a t The new Q value; Q(s) t ,a t ) - Current Q value; α - Learning rate; R t+1 - in state s t Take action a at the time t The reward; -in the new state t+1 And the maximum expected future return given under all possible actions.

[0070] Step S2024: Update the Q-table using the Bellman equation described above. Repeat the above steps until the training process reaches the predefined terminal conditions.

[0071] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A fault-tolerant control method for aircraft based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Build a simulation model of the aircraft based on open-source aircraft aerodynamic data; Step 2: Input and define different flight states of the aircraft and the fault modes of the aircraft under fault states in the simulation environment, and collect data and fault data under different flight states in the simulation environment. Step 3: Expand the collected data using the data augmentation algorithm SDA, and then input the expanded data into the deep Q-learning network for training to obtain the trained deep Q-learning network. The training process in step 3 is as follows: Step (301): Set the training set to empty, and collect aircraft data under different flight conditions in the simulation environment; collect aircraft operating status once per sampling period. To different flight states The data is input into a deep Q-learning network to obtain the reward value corresponding to the current flight state. ; Step (302), Perform the action Observe the next state and rewards ; Step (303): Update the Bellman equations according to different flight states. To obtain the maximum expected future return in all flight states. And update the current deep Q-learning network; Step (304): Repeat steps (301) to (303) for the flight state prediction value of the next sampling period to obtain the maximum expected future return value under different flight states; Step (305): Combine the current aircraft status data with the maximum expected future return under the current flight status. ,award As new data samples, they are expanded into the current training dataset, and the expanded training set is used to update the current deep Q-learning network. For each sampling period, repeat steps (301) to (305) until training is complete; (306) Input the new aircraft state data obtained during the actual flight into the trained deep Q learning network and test whether the deep Q learning network can achieve the expected fault-tolerant control effect; if it cannot achieve the expected fault-tolerant control effect, repeat the steps (301) to (305) with the new data to retrain until the training is completed.

2. The aircraft fault-tolerant control method based on reinforcement learning according to claim 1, characterized in that, In step (301) The calculation formula is: in, It is a state Take action of value, It is the reward for each time step. It is the discount rate; the Q function takes the state and action as input and feeds back its expected future reward.

3. The aircraft fault-tolerant control method based on reinforcement learning according to claim 1, characterized in that, The steps described in step (303) The calculation formula is: in: -state and actions The new Q value; - Current Q value; -Learning rate; - in state Take action at the time The reward; -In the new state And the maximum expected future return given under all possible actions.

4. The aircraft fault-tolerant control method based on reinforcement learning according to claim 1, characterized in that, The network update method described in step (303) is as follows: Will As a new data sample, it is expanded into the current training dataset, and the current deep Q-learning network is updated using the current training dataset.

5. The aircraft fault-tolerant control method based on reinforcement learning according to claim 1, characterized in that, In step 3, the collected data is expanded using the data augmentation algorithm SD by a cut-and-flip method: Five clipping and flipping methods are used: horizontal flip, vertical flip, center flip, data mirroring, and data repetition, to operate on data under the same label.

6. The aircraft fault-tolerant control method based on reinforcement learning according to claim 5, characterized in that, The expanded aircraft attitude dataset from step 3 is sorted according to time series order to form the training set.

7. The aircraft fault-tolerant control method based on reinforcement learning according to claim 1, characterized in that, The different flight states of the aircraft include normal flight state and faulty flight state.

8. The aircraft fault-tolerant control method based on reinforcement learning according to claim 1, characterized in that, The fault modes of the aircraft under fault conditions include constant deviation, constant gain, and sensor jamming.

Images