A general method for unmanned aerial vehicle autonomous stunt flight based on offline reinforcement learning

Abstract

The application discloses a kind of general method of unmanned plane autonomous stunt flight based on offline reinforcement learning, including collecting the historical flight data of unmanned plane, when collecting, try to cover various postures in unmanned plane flight;Filter key observation of unmanned plane, action quantity, process observation and action quantity, construct flight data that meet Markov decision process;Unmanned plane aerodynamics network model is constructed;Using flight data that meet Markov decision process, unmanned plane aerodynamics network model is trained, and the network model of unmanned plane aerodynamics that fits real data is obtained;In the network model of unmanned plane aerodynamics that fits real data, strategy model is trained using reinforcement learning algorithm;After specific stunt flight trajectory is processed, input strategy model after training, model output action, guide unmanned plane to realize specific stunt flight action, this method gives up traditional virtual environment clone based on supervised learning, using optimization method based on adversarial neural network and distribution alignment loss, meanwhile, this method uses double environment test, avoid overfitting problem in strategy learning.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) technology, specifically relating to a general method for autonomous aerobatic flight of UAVs based on offline reinforcement learning. Background Technology

[0002] The autonomous aerobatic flight of unmanned aerial vehicles (UAVs) demonstrates the country's advanced aviation technology and independent research and development capabilities. The aerobatic flight showcases the maneuverability and performance of UAVs. As part of the air show and military exercises, it demonstrates China's military achievements to the international community, enhances the outside world's understanding of its military strength, and strengthens the country's deterrent power.

[0003] The existing methods for achieving aerobatic flight can be roughly categorized as follows:

[0004] 1. The pilot directly controls the drone's flight status, enabling real-time adjustments to its attitude, speed, and direction to quickly respond to environmental changes and execute complex flight maneuvers. In aerobatic flight, pilots require advanced control skills and extensive flight experience to ensure stable drone flight under highly dynamic and high-risk conditions.

[0005] 2. An interactive autonomous aerobatic flight method, comprising: acquiring key attitude sequence pairs and initial positions sent by a user, wherein the key attitude sequence pairs include the positions of key attitudes and the corresponding thrust directions; generating a flight corridor with interconnected ends based on the initial positions specified by the user, forming a safety space to represent the topological shape of the desired trajectory; establishing a multi-objective optimization problem, wherein the objective function of the problem includes the following four categories: the smoothness of the trajectory itself and the time required to execute the entire trajectory, UAV dynamic constraints, safety and topological constraints, and aerobatic constraints; and solving the multi-objective optimization problem based on the flight corridor to obtain an optimized aerobatic flight trajectory.

[0006] 3. A method for autonomous aerobatic flight of unmanned aerial vehicles (UAVs) based on deep imitation learning, comprising: acquiring UAV flight data to construct a system model of the UAV and confirming the system model data; constructing a data buffer using the system model data; constructing a deep simulation learning network model for autonomous aerobatic flight of the UAV using specific UAV data and flight data; training the deep simulation learning network model after processing the flight data to obtain a deep simulation learning network model for the control variables of autonomous aerobatic flight of the UAV; and connecting the data buffer, the deep simulation learning network model for the control variables, and the UAV. This invention enables truly autonomous aerobatic flight of UAVs with high control accuracy and good stability.

[0007] The common drawback of these three techniques is that they are all customized techniques for aerobatic maneuvers. This means that the methods are not universal for different aerobatic maneuvers, requiring significant reinvestment of resources. Technique 1 requires pilots to repeatedly practice new aerobatic maneuvers; Techniques 2 and 3 are optimization solutions and imitations of specific aerobatic flight trajectories, respectively, and therefore lack universality. Summary of the Invention

[0008] To achieve the above objectives, the technical solution of the present invention is as follows: A general method for autonomous aerobatic flight of unmanned aerial vehicles based on offline reinforcement learning, comprising the following steps:

[0009] S1. Collect historical flight data of the drone, and try to cover all attitudes of the drone during flight when collecting data;

[0010] S2. Filter the key observations and motions of the UAV, process the observations and motions, and construct flight data that conforms to the Markov decision process.

[0011] S3. Construct an aerodynamic network model for the unmanned aerial vehicle (UAV);

[0012] S4. Using flight data that conforms to the Markov decision process, train the UAV aerodynamic network model to obtain a UAV aerodynamic network model that fits the real data.

[0013] S5. Use reinforcement learning algorithms to train the policy model in a network model of UAV aerodynamics that fits real data.

[0014] S6. Process the specific aerobatic flight trajectory and input it into the trained strategy model. The model outputs actions to guide the drone to perform specific aerobatic flight maneuvers.

[0015] The specific implementation logic of this method is as follows:

[0016] In step S1, historical flight data of the UAV is collected, and the collection should cover as many attitudes of the UAV as possible during flight.

[0017] Collecting historical flight data from drones is a crucial task, providing data support for subsequent aircraft aerodynamic network model cloning, optimization of flight control algorithms, and training of aerobatic flight strategies. First, ArduPilot is used as the flight control system, and its logging function enables data collection. Before flight, it's essential to ensure the drone's firmware is correctly installed and configured. Flight missions can be set using ground station software such as Mission Planner, ensuring the flight path covers various flight attitudes, including hovering, level flight, ascent, descent, and turning.

[0018] During flight, detailed logging can be enabled by setting parameters. For example, the 'LOG_BITMASK' parameter can be used to select the data types of sensors to record, ensuring that critical information such as IMU data, XKF data, ATT data, GPS information, battery status, motor speed commands RCOU.C1-C12, and AETR control inputs are recorded. Furthermore, the logging frequency can be adjusted to obtain sufficient data detail in different flight modes.

[0019] To comprehensively cover flight attitudes, it is recommended to conduct multiple flights, testing different altitudes, speeds, and directions. Before each flight, develop a detailed flight plan to ensure sufficient flight time for each attitude. Simultaneously, record changes in the flight environment, such as wind speed and temperature, as these factors can also affect the drone's flight performance.

[0020] After the flight, flight logs can be downloaded using ground station software and analyzed using tools such as Mission Planner. By parsing the log files, key flight data can be extracted, providing data support for subsequent aircraft aerodynamic network model cloning, optimization of flight control algorithms, and training of aerobatic flight strategies.

[0021] In the specific implementation of step S2, key observations and motions of the UAV are screened, the observations and motions are processed, and flight data that conforms to the Markov decision process is constructed.

[0022] To screen and process historical UAV flight data to construct a dataset conforming to a Markov Decision Process (MDP), a systematic analysis and reconstruction of the data is required. First, the extracted observations should include sensor data directly related to flight performance, such as attitude angle controller (ATT), inertial measurement unit (IMU), and integrated navigation extended Kalman filter (XKF) data. These data provide essential state information to help describe the UAV's flight state at each moment.

[0023] When constructing the state space, these sensor data need to be combined into a unified state representation. The basic state variables required for UAV aerodynamic environment cloning and strategy model training include: three-dimensional position (X,Y,Z), velocity (Vx,Vy,Vz), gyroscope rotation speed (GyrX,GyrY,GyrZ), acceleration (AccX,AcccY,AcccZ), and attitude (Pitch, Roll, Yaw). To train the aircraft aerodynamic network model more effectively, the aircraft attitude angles are sine-cosine normalized to a continuous [-1,1] space. In the body coordinate system, the processed continuous observations (obs) are used. t To predict the difference delta between the observations at the next time step and the current time step. t ,

[0024] Next, the selection of motion parameters should include control inputs that affect the UAV's state in the next moment. The basic motion parameters required for UAV aerodynamic environment cloning and strategy model training include: roll surface deflection, elevator surface deflection, throttle, and yaw surface deflection. These motion parameters should correspond to each state to ensure that the model can learn the appropriate decision-making strategy under different states. The motion space needs to be normalized.

[0025] Finally, the filtered dataset is organized into a state-action format to ensure that each pair (state, action) corresponds to the subsequent state transition. This structured data needs to conform to Markov properties, providing a solid foundation for subsequent aerodynamic model cloning and reinforcement learning policy model training, enabling the final policy model to control the UAV to autonomously complete various aerobatic flight maneuvers.

[0026] In step S3, an aerodynamic network model of the UAV is constructed;

[0027] The construction of UAV aerodynamic network models refers to the selection of appropriate neural networks to clone UAV aerodynamics. Commonly used network structures include multi-layer ResNet, Transform, or RNN. In model-based reinforcement learning, UAV aerodynamic network models are more commonly referred to as virtual environments or virtual worlds. This method will use the term virtual environment in its technical description.

[0028] In the specific implementation of step S4, the UAV aerodynamic network model is trained using flight data that conforms to the Markov decision process to obtain a UAV aerodynamic network model that fits the real data.

[0029] The mathematical form of a Markov decision process is:

[0030] State sequence: S = (s0, s1, s2, ..., s t ), where s t ∈S represents the state at time step t.

[0031] Action sequence: A = (a0, a1, a2, ..., a t ), where a t ∈A represents the action at time step t.

[0032] The transition probability, state transition can be expressed as: P(s) t+1 |s t ,a t ), indicating that in state s t Take action a t Then, transition to the next state s.t+1 The probability of.

[0033] The entire trajectory format is as follows:

[0034] The training process of an UAV aerodynamic network model is essentially the process of learning the state transition probabilities P in the real world. The training objective is to train the UAV in any given state and any legal action (s) performed in that state. any ,a any The probability transitions in the real world are the same as the probability transitions output by the network model.

[0035] To more accurately clone real-world drone aerodynamic models, reduce the cumulative composite error of multi-step predictions in virtual environments, and avoid overfitting problems in neural networks, this method abandons traditional supervised learning-based virtual environment cloning and uses an optimization method based on adversarial neural networks and distribution alignment loss.

[0036] Starting from the beginning of a trajectory, input the state and action variables (s0, a0) of the beginning point into the generator network, and output the next...

[0037] state quantity at time 1 Will Combined with the motion a1 in the trajectory at the next moment, we get Input the generator network and output the state variables at the next time step. And so on. Obtain the predicted state variables for the entire trajectory.

[0038] The predicted state variables of the entire trajectory With the state variables s0, s1, s2, ..., s in the historical data t Calculate the segmented distribution alignment loss. The segment length L is set, dividing the entire trajectory into N segments, with one-third of the state variables overlapping between adjacent segments. For each segment, calculate the distribution alignment loss; loss functions such as KLDivergence and Wasserstein Distance can be used.

[0039] The discriminant network is used to determine whether a piece of data is real data; the training data is historical data.

[0040] s0,a0,s1,a1,s2,a2,…,s t And the generated data of the generative network A segmented strategy was also adopted during the training process.

[0041] To enable the network to see more actions and improve its generalization ability, this method provides an optional second-stage training. In this second-stage generator, the state and action variables for the next time step are generated simultaneously, resulting in the predicted state variables for the entire trajectory. Using s0, a0, s1, a1, s2, a2, ..., s t and Calculate the segmented distribution alignment loss and optimize the discriminator.

[0042] In the specific implementation of step S5, a strategy model is trained using a reinforcement learning algorithm in a network model of UAV aerodynamics that fits real data.

[0043] Dual-environment verification: To verify the learning effect of the control strategy, in step S4, the historical flight data is divided into two parts, which are used to train the virtual environment, resulting in two different virtual environments. The same algorithm and parameters are used to train the strategy, resulting in two control strategies. The two strategies are then tested in the two virtual environments. If the test results of a single strategy in both virtual environments are excellent and the difference is small, it indicates that the strategy learning is excellent.

[0044] The trained UAV aerodynamic network model serves as a data-driven virtual simulation environment for the UAV in the real world. Any reinforcement learning algorithm can be integrated into this virtual environment to train control strategies, typically using the stable PPO algorithm.

[0045] The key to training a general control strategy based on an UAV aerodynamic network model (virtual environment) lies in setting the reward function during strategy training. This method sets the differences in position (X, Y, Z), overload (AccX, Accy, AccZ), velocity (Vx, Vy, Vz), and attitude (Pitch, Roll, Yaw) between the current state and the target state as the reward function. The smaller the difference between the UAV and the target state before the target state changes, the greater the reward. During training, the target state changes randomly every 2 seconds within a certain range. The strategy model, having seen the differences with various target points, can reach the target state within two seconds.

[0046] In the specific implementation of step S6, the specific aerobatic flight trajectory is processed and input into the trained strategy model. The model outputs actions to guide the UAV to perform specific aerobatic flight actions.

[0047] The trained strategy model is imported into the UAV control unit to calculate the target trajectory for specific aerobatic flights. Key points in the target trajectory are set as target points for the UAV. The strategy continuously controls the UAV to chase these key points, achieving autonomous aerobatic flight. In actual UAV flights, it is also necessary to separately configure emergency stops for special situations or remote control by an experienced pilot to prevent accidents and losses during strategy control.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] This invention utilizes a model-based offline reinforcement learning method to generate an aerodynamic network model of a drone. This aims to more accurately clone real-world drone aerodynamic models, reduce the cumulative composite error of multi-step predictions in a virtual environment, and avoid overfitting issues in neural networks.

[0050] This method abandons the traditional supervised learning-based virtual environment cloning and uses an optimization method based on adversarial neural networks and distribution alignment loss. Furthermore, this method employs dual-environment verification to avoid overfitting issues in policy learning.

[0051] Finally, this method uses the difference between the target state and the target state as the input for policy training. The reward function is set to make the UAV reach the target state as quickly as possible. During the training process, the target state changes randomly within a certain range. In this way, the control policy trained can reach the target state in a short time. When in use, it is only necessary to calculate different key point states according to different aerobatic flight trajectories and transmit them to the UAV to realize a variety of autonomous aerobatic flight maneuvers of the UAV. Therefore, this method is called a general method for autonomous aerobatic flight of UAVs based on offline reinforcement learning. Attached Figure Description

[0052] Figure 1 This is a flowchart of a general method for autonomous aerobatic flight of unmanned aerial vehicles based on offline reinforcement learning in this embodiment;

[0053] Figure 2 This is a logic diagram illustrating the implementation of a general method for autonomous aerobatic flight of unmanned aerial vehicles based on offline reinforcement learning in this embodiment;

[0054] Figure 3 This is a comparison chart of the test results of a single strategy in two virtual environments in this embodiment. Detailed Implementation

[0055] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0056] Example: Figure 1As shown, a specific embodiment of the technical solution for implementing the present invention is as follows: a general method for autonomous aerobatic flight of unmanned aerial vehicles based on offline reinforcement learning, comprising the following steps:

[0057] S1. Collect historical flight data of the drone, and try to cover all attitudes of the drone during flight when collecting data;

[0058] S2. Filter the key observations and motions of the UAV, process the observations and motions, and construct flight data that conforms to the Markov decision process.

[0059] S3. Construct an aerodynamic network model for the unmanned aerial vehicle (UAV);

[0060] S4. Using flight data that conforms to the Markov decision process, train the UAV aerodynamic network model to obtain a UAV aerodynamic network model that fits the real data.

[0061] S5. Use reinforcement learning algorithms to train the policy model in a network model of UAV aerodynamics that fits real data.

[0062] S6. Process the specific aerobatic flight trajectory and input it into the trained strategy model. The model outputs actions to guide the drone to perform specific aerobatic flight maneuvers.

[0063] The implementation logic of this method can be further abstracted as follows: Figure 2 .

[0064] In the specific implementation of step S1, historical flight data of the UAV is collected, and the collection should cover as many attitudes as possible during the flight of the UAV.

[0065] Collecting historical flight data from drones is a crucial task, providing data support for subsequent aircraft aerodynamic network model cloning, optimization of flight control algorithms, and training of aerobatic flight strategies. First, ArduPilot is used as the flight control system, and its logging function enables data collection. Before flight, it's essential to ensure the drone's firmware is correctly installed and configured. Flight missions can be set using ground station software such as Mission Planner, ensuring the flight path covers various flight attitudes, including hovering, level flight, ascent, descent, and turning.

[0066] During flight, detailed logging can be enabled by setting parameters. For example, the 'LOG_BITMASK' parameter can be used to select the data types of sensors to record, ensuring that critical information such as IMU data, XKF data, ATT data, GPS information, battery status, motor speed commands RCOU.C1-C12, and AETR control inputs are recorded. Furthermore, the logging frequency can be adjusted to obtain sufficient data detail in different flight modes.

[0067] To comprehensively cover flight attitudes, it is recommended to conduct multiple flights, testing different altitudes, speeds, and directions. Before each flight, develop a detailed flight plan to ensure sufficient flight time for each attitude. Simultaneously, record changes in the flight environment, such as wind speed and temperature, as these factors can also affect the drone's flight performance.

[0068] After the flight, flight logs can be downloaded using ground station software and analyzed using tools such as Mission Planner. By parsing the log files, key flight data can be extracted, providing data support for subsequent aircraft aerodynamic network model cloning, optimization of flight control algorithms, and training of aerobatic flight strategies.

[0069] In the specific implementation of step S2, key observations and motions of the UAV are screened, the observations and motions are processed, and flight data that conforms to the Markov decision process is constructed.

[0070] To screen and process historical UAV flight data to construct a dataset conforming to a Markov Decision Process (MDP), a systematic analysis and reconstruction of the data is required. First, the extracted observations should include sensor data directly related to flight performance, such as attitude angle controller (ATT), inertial measurement unit (IMU), and integrated navigation extended Kalman filter (XKF) data. These data provide essential state information to help describe the UAV's flight state at each moment.

[0071] When constructing the state space, these sensor data need to be combined into a unified state representation. The basic state variables required for UAV aerodynamic environment cloning and strategy model training include: three-dimensional position (X,Y,Z), velocity (Vx,Vy,Vz), gyroscope rotation speed (GyrX,GyrY,GyrZ), acceleration (AccX,AcccY,AcccZ), and attitude (Pitch, Roll, Yaw). To train the aircraft aerodynamic network model more effectively, the aircraft attitude angles are sine-cosine normalized to a continuous [-1,1] space. In the body coordinate system, the processed continuous observations (obs) are used. t To predict the difference delta between the observations at the next time step and the current time step. t ,

[0072] Next, the selection of motion parameters should include control inputs that affect the UAV's state in the next moment. The basic motion parameters required for UAV aerodynamic environment cloning and strategy model training include: roll surface deflection, elevator surface deflection, throttle, and yaw surface deflection. These motion parameters should correspond to each state to ensure that the model can learn the appropriate decision-making strategy under different states. The motion space needs to be normalized.

[0073] Finally, the filtered dataset is organized into a state-action format to ensure that each pair (state, action) corresponds to the subsequent state transition. This structured data needs to conform to Markov properties, providing a solid foundation for subsequent aerodynamic model cloning and reinforcement learning policy model training, enabling the final policy model to control the UAV to autonomously complete various aerobatic flight maneuvers.

[0074] In the specific implementation of step S3, an aerodynamic network model of the UAV is constructed;

[0075] The construction of UAV aerodynamic network models refers to the selection of appropriate neural networks to clone UAV aerodynamics. Commonly used network structures include multi-layer ResNet, Transform, or RNN. In model-based reinforcement learning, UAV aerodynamic network models are more commonly referred to as virtual environments or virtual worlds. This method will use the term virtual environment in its technical description.

[0076] In the specific implementation of step S4, the UAV aerodynamic network model is trained using flight data that conforms to the Markov decision process to obtain a UAV aerodynamic network model that fits the real data.

[0077] The mathematical form of a Markov decision process is:

[0078] State sequence: S = (s0, s1, s2, ..., s t ), where s t ∈S represents the state at time step t.

[0079] Action sequence: A = (a0, a1, a2, ..., a t ), where a t ∈A represents the action at time step t.

[0080] The transition probability, state transition can be expressed as: P(s) t+1 |s t ,a t ), indicating that in state s t Take action a t Then, transition to the next state s. t+1 The probability of.

[0081] The entire trajectory format is as follows:

[0082] The training process of an UAV aerodynamic network model is essentially the process of learning the state transition probabilities P in the real world. The training objective is to train the UAV in any given state and any legal action (s) performed in that state. any ,a any The probability transitions in the real world are the same as the probability transitions output by the network model.

[0083] To more accurately clone real-world drone aerodynamic models, reduce the cumulative composite error of multi-step predictions in virtual environments, and avoid overfitting problems in neural networks, this method abandons traditional supervised learning-based virtual environment cloning and uses an optimization method based on adversarial neural networks and distribution alignment loss.

[0084] Starting from the beginning of a trajectory, the state and action variables (s0, a0) of the beginning are input into the generator network, and the state variable of the next time step is output. Will Combined with the motion a1 in the trajectory at the next moment, we get Input the generator network and output the state variables at the next time step. And so on. Obtain the predicted state variables for the entire trajectory.

[0085] The predicted state variables of the entire trajectory With the state variables s0, s1, s2, ..., s in the historical data t Calculate the segmented distribution alignment loss. The segment length L is set, dividing the entire trajectory into N segments, with one-third of the state variables overlapping between adjacent segments. For each segment, calculate the distribution alignment loss; loss functions such as KLDivergence and Wasserstein Distance can be used.

[0086] The discriminant network is used to determine whether a piece of data is real data; the training data is historical data.

[0087] s0,a0,s1,a1,s2,a2,…,s t And the generated data of the generative network A segmented strategy was also adopted during the training process.

[0088] To enable the network to see more actions and improve its generalization ability, this method provides an optional second-stage training. In this second-stage generator, the state and action variables for the next time step are generated simultaneously, resulting in the predicted state variables for the entire trajectory. Using s0, a0, s1, a1, s2, a2, ..., s t and Calculate the segmented distribution alignment loss and optimize the discriminator.

[0089] In the specific implementation of step S5, a strategy model is trained using a reinforcement learning algorithm in a network model of UAV aerodynamics that fits real data.

[0090] Dual-environment verification: To verify the learning effect of the control strategy, in step S4, the historical flight data is divided into two parts, which are used to train the virtual environment, resulting in two different virtual environments. The same algorithm and parameters are used to train the strategy, resulting in two control strategies. The two strategies are then tested in the two virtual environments respectively. Figure 3 As shown, the test results of the single policy in both virtual environments are excellent and the difference is small, indicating that the policy learning is excellent.

[0091] The trained UAV aerodynamic network model serves as a data-driven virtual simulation environment for the UAV in the real world. Any reinforcement learning algorithm can be integrated into this virtual environment to train control strategies, typically using the stable PPO algorithm.

[0092] The key to training a general control strategy based on an UAV aerodynamic network model (virtual environment) lies in setting the reward function during strategy training. This method sets the differences in position (X, Y, Z), overload (AccX, Accy, AccZ), velocity (Vx, Vy, Vz), and attitude (Pitch, Roll, Yaw) between the current state and the target state as the reward function. The smaller the difference between the UAV and the target state before the target state changes, the greater the reward. During training, the target state changes randomly every 2 seconds within a certain range. The strategy model, having seen the differences with various target points, can reach the target state within two seconds.

[0093] In the specific implementation of step S6, the specific aerobatic flight trajectory is processed and input into the trained strategy model. The model outputs actions to guide the UAV to perform specific aerobatic flight actions.

[0094] The trained strategy model is imported into the UAV control unit to calculate the target trajectory for specific aerobatic flights. Key points in the target trajectory are set as target points for the UAV. The strategy continuously controls the UAV to chase these key points, achieving autonomous aerobatic flight. In actual UAV flights, it is also necessary to separately configure emergency stops for special situations or remote control by an experienced pilot to prevent accidents and losses during strategy control.

[0095] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A general method for autonomous stunt flight of UAV based on offline reinforcement learning, characterized in that, The method comprises the following steps: S1, collecting historical flight data of the unmanned aerial vehicle, and collecting various attitudes in the flight of the unmanned aerial vehicle as much as possible; S2, screening key observation quantities and action quantities of the unmanned aerial vehicle, processing the observation quantities and the action quantities, and constructing flight data conforming to a Markov decision process; S3, constructing an aerodynamics network model of the unmanned aerial vehicle; S4, training the aerodynamics network model of the unmanned aerial vehicle by using the flight data conforming to the Markov decision process, and obtaining an aerodynamics network model of the unmanned aerial vehicle conforming to real data; The specific steps of step S4 are as follows: training the aerodynamics network model of the unmanned aerial vehicle by using the flight data conforming to the Markov decision process, and obtaining an aerodynamics network model of the unmanned aerial vehicle conforming to real data; The mathematical form of the Markov decision process is as follows: State sequence: S = (s0, s1, s2, ..., s t ), where s t ∈S represents the state at time step t. Action sequence: A = (a0, a1, a2,..., an) where a t ∈ A denotes an action at time step t, t and a0is the initial state of the environment. The transition probabilities, state transitions can be represented as: P(s t+1 | s t , a t ), representing the probability of transitioning to the next state s t after taking action a t in state s t+1 , Whole trajectory form: The training process of the UAV aerodynamic network model is a process of learning the state transition probability P in the real world, and the training target is to obtain the same probability transition as the real world for any state of the UAV and any legal action (s any ,a any ) made in the state. An optimization method based on an adversarial neural network and a distribution alignment loss is used: 1) From the start point of a trajectory, input the state quantity and action quantity (s0, a0) of the start point into the generation network, and output the state quantity at the next time Combine And the action quantity a1 in the next time of the trajectory to obtain Input the generation network, and output the state quantity at the next time In this way Obtain the predicted state quantity of the whole trajectory 2) the predicted state quantity of the entire trajectory and the state quantity s0, s1, s2, …, s t The segmented distribution alignment loss is calculated, wherein the segmented length L is set, the entire trajectory is divided into N segments, 1 / 3 of the state quantity before and after the adjacent two segments is repeated, and the distribution alignment loss is calculated for each segment; 3) Discriminator network is used to determine whether a piece of data is real data, and the training data is historical data s0, a0, s1, a1, s2, a2, …, s t and the generated data s0, a0 of the generation network, Segmentation strategy is also used in the training process; 4) To generate more diverse actions that the network has seen, to improve the generalization on actions, provide an optional second stage training, the second stage generator will generate the state quantity and action quantity of the next moment at the same time, get the predicted state quantity s0,a0, Using s0,a0,s1,a1,s2,a2,…,s t And s0,a0, Calculate the segment distribution alignment loss and optimize the discriminator; S5, training a strategy model in the aerodynamics network model of the unmanned aerial vehicle conforming to real data by using a reinforcement learning algorithm; S6, inputting a specific stunt flight trajectory after processing into the trained strategy model, outputting an action of the model, and guiding the unmanned aerial vehicle to realize a specific stunt flight action.

2. The general method for autonomous stunt flight of UAV based on offline reinforcement learning according to claim 1, characterized in that, The step S1 of collecting historical flight data of the unmanned aerial vehicle includes the following steps: Firstly, ArduPilot is used as a flight control system, and data collection is realized by using a log recording function provided by ArduPilot; During the flight, detailed log recording is enabled by setting parameters; After the flight, a ground station software is used to download the flight log, and data analysis is performed by using a tool, key flight data is extracted by analyzing the log file, and data support is provided for subsequent cloning of the aerodynamics network model of the aircraft, optimization of the flight control algorithm and training of the stunt flight strategy.

3. The general method for autonomous stunt flight of UAV based on offline reinforcement learning according to claim 1, characterized in that, The step S2 comprises screening and processing historical flight data of the unmanned aerial vehicle to construct a data set conforming to the Markov decision process, and the data needs to be systematically analyzed and reconstructed: Firstly, the extracted observation quantities should include sensor data directly related to flight performance; When constructing a state space, sensor data needs to be combined into a unified state representation, and the aerodynamics environment cloning and strategy model training of the unmanned aerial vehicle; Next, the selection of action quantities should include control inputs that affect the state of the unmanned aerial vehicle at the next moment, and the aerodynamics environment cloning and strategy model training of the unmanned aerial vehicle; Finally, the data set after screening and processing is organized in a state-action format, ensuring that each pair of state and action can correspond to a subsequent state transition, and the structured data needs to conform to the Markov property, providing a solid foundation for subsequent cloning of the aerodynamics model and training of the reinforcement learning strategy model, so that the final strategy model can control the unmanned aerial vehicle to autonomously complete various stunt flight actions.

4. The general method for autonomous stunt flying of UAVs based on off-line reinforcement learning according to claim 3, wherein, The required base state quantities are: three-dimensional position (X, Y, Z), velocity (Vx, Vy, Vz), gyroscope rotation rate (GyrX, GyrY, GyrZ), acceleration (AccX, AccY, AccZ), attitude (Pitch, Roll, Yaw) information, in order to more effectively train the aircraft aerodynamic network model, the aircraft attitude angle is normalized to the continuous [-1, 1] space by sine and cosine, in the body coordinate system, the processed continuous observation quantity obs t Predict the difference delta between the next time and the current time observation t , 5. The general method for autonomous stunt flying of UAVs based on offline reinforcement learning according to claim 3, wherein, The required basic action quantities are: roll rudder deflection angle, elevator deflection angle, throttle, yaw rudder deflection angle, and these action quantities should correspond to each state, ensuring that the model can learn the corresponding decision strategy in different states, and the action space needs to be normalized.

6. The general method for autonomous stunt flying of UAVs based on offline reinforcement learning according to claim 1, wherein, In the implementation of the step S3, the step of constructing the UAV aerodynamics network model includes constructing a UAV aerodynamics network model, which is expressed as selecting an appropriate neural network for cloning UAV aerodynamics, and the network structure includes a multi-layer ResNet, Transform or RNN.

7. The general method for autonomous stunt flying of UAVs based on offline reinforcement learning according to claim 1, wherein, In the implementation of the step S5, the strategy model is trained in the network model of the UAV aerodynamics fitted to the real data by using a reinforcement learning algorithm. Double-environment test: In order to test the learning effect of the control strategy, the historical flight data is divided into two parts in the step S4, which are respectively used for training virtual environments, two different virtual environments are obtained, the same algorithm and parameters are used for training the strategy, two control strategies are obtained, and the two strategies are tested in the two virtual environments. The test results of a single strategy in the two virtual environments are excellent, and the difference is not large, which indicates that the strategy learning is excellent. The trained UAV aerodynamics network model is used as a data-driven real-world virtual simulation environment of the UAV, and any reinforcement learning algorithm can be connected to the virtual environment to train the control strategy. The key point of training a general control strategy based on the UAV aerodynamics network model is the setting of the reward function when training the strategy. The position (X, Y, Z), overload (AccX, AccY, AccZ), speed (Vx, Vy, Vz), and attitude (Pitch, Roll, Yaw) differences between the current state and the target state are set as the reward function. Before the target state changes, the smaller the difference between the UAV and the target state, the greater the reward obtained. The target state changes randomly within a certain range every 2 seconds during the training process. The strategy model has seen the difference with various target points, and reaches the target state within two seconds.

8. The general method for autonomous stunt flying of UAVs based on offline reinforcement learning according to claim 1, wherein, In the step S6, the specific stunt flight trajectory is processed and input into the trained strategy model, the model outputs the action, and guides the UAV to realize the specific stunt flight action. The trained strategy model is imported into the UAV control unit, the target trajectory of the specific stunt flight is calculated, the key points in the target trajectory are set as the target points of the UAV, the strategy continuously controls the UAV to pursue the key points of the target trajectory, and the autonomous stunt flight of the UAV is realized. In the actual flight of the UAV, special emergency stop or experienced pilot remote control of the UAV is separately set.

Images