Universal method for autonomous stunt flight of unmanned aerial vehicle based on offline reinforcement learning

Abstract

The invention discloses a universal method for unmanned aerial vehicle autonomous stunt flight based on offline reinforcement learning, and the method comprises the steps: collecting the historical flight data of an unmanned aerial vehicle, and covering various attitudes in the flight of the unmanned aerial vehicle as much as possible during the collection; screening key observed quantities and action quantities of the unmanned aerial vehicle, processing the observed quantities and the action quantities, and constructing flight data conforming to a Markov decision process; constructing an unmanned aerial vehicle aerodynamic network model; training the unmanned aerial vehicle aerodynamics network model by using the flight data conforming to the Markov decision process to obtain an unmanned aerial vehicle aerodynamics network model fitting real data; training a strategy model in the unmanned aerial vehicle aerodynamics network model fitting real data by using a reinforcement learning algorithm; a specific stunt flight path is processed and then input into the trained strategy model, the model outputs an action to guide the unmanned aerial vehicle to realize a specific stunt flight action, the traditional virtual environment cloning based on supervised learning is abandoned, an optimization method based on an adversarial neural network and distribution alignment loss is used, and the method has the advantages of high robustness and high robustness. And meanwhile, the method adopts dual-environment inspection, so that the problem of overfitting in strategy learning is avoided.

Description

Technical Field

[0001] The invention belongs to the technical field of unmanned aerial vehicles, and in particular relates to a general method for autonomous aerobatic flight of unmanned aerial vehicles based on offline reinforcement learning. Background Art

[0002] Autonomous UAV aerobatic flights demonstrate the country's advanced aviation technology and independent research and development capabilities. Stunt flights show the maneuverability and performance of UAVs. As part of air shows and military exercises, they showcase China's military achievements to the international community, enhance the outside world's understanding of military strength, and enhance the country's deterrence.

[0003] There are several existing methods for achieving aerobatics:

[0004] 1. The pilot directly controls the flight status of the drone. The pilot can adjust the attitude, speed and direction of the drone in real time to quickly respond to environmental changes and perform complex flight maneuvers. In aerobatic flying, the pilot needs to have a high level of control skills and rich flying experience to ensure the stable flight of the drone under highly dynamic and high-risk conditions.

[0005] 2. An interactive autonomous aerobatic flying method, comprising: obtaining a key posture sequence pair and an initial position sent by a user, wherein the key posture sequence pair includes the position of the key posture and the corresponding thrust direction; generating a flight corridor connected end to end according to the initial position specified by the user to form a safe space for representing the topological shape of a desired trajectory; establishing a multi-objective optimization problem, wherein the objective function of the problem includes the following four categories: the smoothness of the trajectory itself and the time required for the execution of the entire trajectory, UAV dynamics constraints, safety and topological constraints, and stunt constraints; based on the flight corridor, solving the multi-objective optimization problem to obtain an optimized aerobatic flying trajectory.

[0006] 3. A method for autonomous aerobatic flight of unmanned aerial vehicles based on deep imitation learning, comprising obtaining flight data of unmanned aerial vehicles to construct a system model of the unmanned aerial vehicle and confirming the system model data; constructing a data buffer using the system model data; constructing a deep simulation learning network model for autonomous aerobatic flight of the unmanned aerial vehicle through specific and flight data of the unmanned aerial vehicle; training the deep simulation learning network model after processing the flight data to obtain a control quantity deep simulation learning network model for autonomous aerobatic flight of the unmanned aerial vehicle; and connecting the data buffer, the control quantity deep simulation learning network model, and the unmanned aerial vehicle. The method of the present invention can realize true autonomous aerobatic flight of unmanned aerial vehicles with high control accuracy and good stability.

[0007] The common disadvantage of these three technologies is that they are all customized technologies for aerobatic maneuvers, that is, the methods are not universal for different aerobatic maneuvers and require a lot of reinvestment. For technology 1, for new aerobatic maneuvers, pilots need to practice the maneuvers again and again; technology 2 and technology 3 are respectively optimized solutions and imitations for specific aerobatic flight trajectories, which are not universal. Summary of the invention

[0008] To achieve the above object, the technical solution of the present invention is as follows: A general method for autonomous aerobatic flight of a UAV based on offline reinforcement learning, comprising the following steps:

[0009] S1. Collect the historical flight data of the drone, and try to cover all the postures of the drone during flight;

[0010] S2, screening the key observations and actions of the UAV, processing the observations and actions, and constructing flight data that conforms to the Markov decision process;

[0011] S3, build a UAV aerodynamic network model;

[0012] S4. Using the flight data that conforms to the Markov decision process, the UAV aerodynamic network model is trained to obtain a UAV aerodynamic network model that fits the real data;

[0013] S5. Use reinforcement learning algorithm to train the policy model in the network model of drone aerodynamics that fits the real data;

[0014] S6. The specific aerobatic flight trajectory is processed and input into the trained strategy model, and the model outputs actions to guide the UAV to perform specific aerobatic flight actions.

[0015] The implementation logic of this method is as follows:

[0016] In step S1, historical flight data of the UAV is collected, and various postures of the UAV in flight are covered as much as possible during the collection;

[0017] Collecting historical flight data of drones is an important task, which provides data support for subsequent aircraft aerodynamic network model cloning, optimizing flight control algorithms, and training aerobatic flight strategies. First of all, using ArduPilot as the flight control system, data collection can be achieved through the logging function it provides. Before flying, you need to ensure that the firmware of the drone has been correctly installed and configured. You can set flight missions through ground station software such as Mission Planner to ensure that the flight path covers a variety of flight attitudes, including hovering, level flight, ascent, descent, and turning.

[0018] During flight, detailed logging is enabled by setting parameters, such as using the LOG_BITMASK parameter to select the type of sensor data to be logged, ensuring that key information such as IMU data, XKF data, ATT data, GPS information, battery status, and motor speed commands RCOU.C1-C12, AETR control input, etc. are logged. In addition, the frequency of the log can be adjusted to obtain sufficient data details in different flight modes.

[0019] In order to fully cover the flight attitude, it is recommended to conduct multiple flights to test different flight altitudes, speeds and directions. Before each flight, make a detailed flight plan to ensure that each attitude has enough flight time. At the same time, record the changes in the flight environment, such as wind speed and temperature, which will also affect the flight performance of the drone.

[0020] After the flight, you can use the ground station software to download the flight log and use tools such as Mission Planner to analyze the data. By parsing the log file, you can extract key flight data, which will provide data support for subsequent aircraft aerodynamic network model cloning, flight control algorithm optimization, and aerobatic flight strategy training.

[0021] In the specific implementation of step S2, the key observation quantities and action quantities of the UAV are screened, the observation quantities and action quantities are processed, and flight data that conforms to the Markov decision process is constructed;

[0022] Screening and processing the historical flight data of UAVs to construct a data set that conforms to the Markov decision process (MDP) requires systematic analysis and reconstruction of the data. First, the extracted observations should include sensor data directly related to flight performance, such as attitude angle controller (ATT), inertial measurement unit (IMU) and integrated navigation extended Kalman filter data (XKF), which provide the necessary state information to help describe the flight state of the UAV at every moment.

[0023] When constructing the state space, it is necessary to combine these sensor data into a unified state representation. The basic state quantities required for drone aerodynamic environment cloning and strategy model training include: three-dimensional position (X, Y, Z), speed (Vx, Vy, Vz), gyroscope speed (GyrX, GyrY, GyrZ), acceleration (AccX, AccY, AccZ), attitude (Pitch, Roll, Yaw) and other information. In order to more effectively train the aircraft aerodynamic network model, the aircraft attitude angle is normalized to the continuous [-1, 1] space by sine and cosine. In the body coordinate system, the processed continuous observation obs is used. t To predict the difference delta between the next moment and the current moment t ,

[0024] Next, the action quantity screening should include the control input that affects the state of the drone at the next moment, the aerodynamic environment cloning of the drone and the training of the strategy model. The basic action quantities required are: roll control surface deflection angle, elevator control surface deflection angle, throttle, yaw control surface deflection angle, etc. These action quantities should correspond to each state to ensure that the model can learn the corresponding decision-making strategy in different states. The action space needs to be normalized.

[0025] Finally, the filtered and processed data set is organized into a state-action format to ensure that each pair of (state, action) can correspond to the subsequent state transfer. This structured data needs to conform to the Markov characteristics to provide a solid foundation for subsequent aerodynamic model cloning and reinforcement learning strategy model training, so that the final strategy model can control the UAV to autonomously complete a variety of aerobatic maneuvers.

[0026] In step S3, a UAV aerodynamic network model is constructed;

[0027] Constructing a UAV aerodynamic network model means selecting an appropriate neural network for cloning UAV aerodynamics. Generally, the network structures used are multi-layer ResNet, Transform or RNN. The more common name for the UAV aerodynamic network model in model-based reinforcement learning is virtual environment or virtual world. When this method is technically explained, the name of the virtual environment will be used.

[0028] In the specific implementation of step S4, the UAV aerodynamic network model is trained using flight data that conforms to the Markov decision process to obtain a UAV aerodynamic network model that fits the real data;

[0029] The mathematical form of the Markov decision process is:

[0030] State sequence: S = (s0, s1, s2, ..., s t ), where s t ∈S represents the state at time step t.

[0031] Action sequence: A=(a0,a1,a2,…,a t ), where a t ∈A represents the action at time step t.

[0032] Transition probability, state transition can be expressed as: P(s t+1 |s t ,a t ), indicating that in state s t Take action a t Then, transfer to the next state st+1 probability.

[0033] The whole track form:

[0034] The training process of the UAV aerodynamic network model is the process of learning the state transition probability P in the real world. The training goal is to predict any state of the UAV and any legal action (s any ,a any ), the probability transition in the real world is the same as the probability transition output by the network model.

[0035] In order to more accurately clone the aerodynamic model of drones in the real world, reduce the cumulative compound error of multi-step predictions in the virtual environment, and avoid the overfitting problem in the neural network, this method abandons the traditional virtual environment cloning based on supervised learning and uses an optimization method based on adversarial neural networks and distribution alignment loss.

[0036] Starting from the starting point of a trajectory, the state and action (s0, a0) of the starting point are input into the generative network, and the next

[0037] State quantity at the moment Will Combined with the action amount a1 at the next moment in the trajectory, we get Input the generated network and output the state quantity at the next moment And so on Get the predicted state of the entire trajectory

[0038] The predicted state quantity of the entire trajectory and the state quantities s0,s1,s2,…,s in the historical data t Calculate the segment distribution alignment loss. Set the segment length L, divide the entire trajectory into N segments, and 1 / 3 of the state quantities before and after two adjacent segments are repeated. Calculate the distribution alignment loss for each segment. The loss function can use KL Divergence, Wasserstein Distance, etc.

[0039] The discriminant network is used to determine whether a piece of data is real data, and the training data is historical data.

[0040] s0,a0,s1,a1,s2,a2,…,s t And the generated data of the generative network A segmented strategy was also adopted during the training process.

[0041] In order to generate more actions that the network has seen and improve the generalization of actions, this method provides an optional second-stage training. The generator in the second stage will simultaneously generate the state and action of the next moment to obtain the predicted state of the entire trajectory. Use s0,a0,s1,a1,s2,a2,…,s t and Compute the piecewise distribution alignment loss and optimize the discriminator.

[0042] In the specific implementation of step S5, the strategy model is trained using a reinforcement learning algorithm in a network model of drone aerodynamics that fits real data;

[0043] Dual environment test: In order to test the learning effect of the control strategy, the historical flight data will be divided into two parts in step S4, which are used to train the virtual environment respectively, and two different virtual environments are obtained. The same algorithm and parameters are used when training the strategy to obtain two control strategies, and the two strategies are tested in the two virtual environments respectively. The test results of a single strategy in the two virtual environments are excellent, and the difference is not large, indicating that the strategy learning is excellent.

[0044] The trained UAV aerodynamic network model serves as a data-driven real-world virtual simulation environment for the UAV. Any reinforcement learning algorithm can be connected to this virtual environment to train the control strategy. Generally, a stable PPO algorithm is used.

[0045] The key point of training a general control strategy based on the UAV aerodynamic network model (virtual environment) is the setting of the reward function when training the strategy. This method sets the difference between the current state and the target state in position (X, Y, Z), overload (AccX, AccY, AccZ), speed (Vx, Vy, Vz), and attitude (Pitch, Roll, Yaw) as the reward function. Before the target state changes, the smaller the difference between the UAV and the target state, the greater the reward. During the training process, the target state changes randomly every 2 seconds within a certain range. The strategy model has seen the difference with various target points and can reach the target state within two seconds.

[0046] In the specific implementation of step S6, the specific aerobatic flight trajectory is processed and input into the trained strategy model, and the model outputs an action to guide the UAV to perform the specific aerobatic flight action;

[0047] The trained strategy model is imported into the drone control unit to calculate the target trajectory of a specific aerobatic flight, and the key points in the target trajectory are set as the drone target points. The strategy continuously controls the drone to chase the key points of the target trajectory to achieve autonomous aerobatic flight of the drone. In the actual flight of the drone, it is also necessary to set up emergency stops in special situations or have experienced pilots remotely control the drone to avoid accidents and losses during the strategy control process.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] The present invention uses a model-based offline reinforcement learning method to generate a UAV aerodynamic network model. In order to more accurately clone the UAV aerodynamic model in the real world, reduce the cumulative compound error of multi-step prediction in the virtual environment, and avoid the overfitting problem in the neural network;

[0050] This method abandons the traditional virtual environment cloning based on supervised learning and uses an optimization method based on adversarial neural networks and distribution alignment loss. At the same time, this method uses dual environment testing to avoid overfitting problems in strategy learning;

[0051] Finally, this method uses the difference between the state and the target state as the input of strategy training, and the reward function sets the UAV to reach the target state as soon as possible. During the training process, the target state changes randomly within a certain range. In this way, the control strategy obtained by training can reach the target state in a shorter time. When in use, it only needs to calculate different key point states according to different stunt flight trajectories and pass them to the UAV to realize a variety of autonomous aerobatic actions of the UAV. Therefore, this method is called a general method for autonomous aerobatic flight of UAVs based on offline reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flow chart of a general method for autonomous aerobatic flight of a UAV based on offline reinforcement learning in this embodiment;

[0053] Figure 2 A logic diagram for realizing a general method for autonomous aerobatic flight of a UAV based on offline reinforcement learning in this embodiment;

[0054] Figure 3 This is a comparison chart of the test results of a single strategy in two virtual environments in this embodiment. DETAILED DESCRIPTION

[0055] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0056] Example: Figure 1As shown, a specific embodiment of the technical solution of the present invention is: a general method for autonomous aerobatic flight of a UAV based on offline reinforcement learning, comprising the following steps:

[0057] S1. Collect the historical flight data of the drone, and try to cover all the postures of the drone during flight;

[0058] S2, screening the key observations and actions of the UAV, processing the observations and actions, and constructing flight data that conforms to the Markov decision process;

[0059] S3, build a UAV aerodynamic network model;

[0060] S4. Using the flight data that conforms to the Markov decision process, the UAV aerodynamic network model is trained to obtain a UAV aerodynamic network model that fits the real data;

[0061] S5. Use reinforcement learning algorithm to train the policy model in the network model of drone aerodynamics that fits the real data;

[0062] S6. The specific aerobatic flight trajectory is processed and input into the trained strategy model, and the model outputs actions to guide the UAV to perform specific aerobatic flight actions.

[0063] The implementation logic of this method can be further abstracted as Figure 2 .

[0064] In the specific implementation of step S1, historical flight data of the UAV is collected, and various postures of the UAV in flight are covered as much as possible during the collection;

[0065] Collecting historical flight data of drones is an important task, which provides data support for subsequent aircraft aerodynamic network model cloning, optimizing flight control algorithms, and training aerobatic flight strategies. First of all, using ArduPilot as the flight control system, data collection can be achieved through the logging function it provides. Before flying, you need to ensure that the firmware of the drone has been correctly installed and configured. You can set flight missions through ground station software such as Mission Planner to ensure that the flight path covers a variety of flight attitudes, including hovering, level flight, ascent, descent, and turning.

[0066] During flight, detailed logging is enabled by setting parameters, such as using the LOG_BITMASK parameter to select the type of sensor data to be logged, ensuring that key information such as IMU data, XKF data, ATT data, GPS information, battery status, and motor speed commands RCOU.C1-C12, AETR control input, etc. are logged. In addition, the frequency of the log can be adjusted to obtain sufficient data details in different flight modes.

[0067] In order to fully cover the flight attitude, it is recommended to conduct multiple flights to test different flight altitudes, speeds and directions. Before each flight, make a detailed flight plan to ensure that each attitude has enough flight time. At the same time, record the changes in the flight environment, such as wind speed and temperature, which will also affect the flight performance of the drone.

[0068] After the flight, you can use the ground station software to download the flight log and use tools such as Mission Planner to analyze the data. By parsing the log file, you can extract key flight data, which will provide data support for subsequent aircraft aerodynamic network model cloning, flight control algorithm optimization, and aerobatic flight strategy training.

[0069] In the specific implementation of step S2, the key observation quantities and action quantities of the UAV are screened, the observation quantities and action quantities are processed, and flight data that conforms to the Markov decision process is constructed;

[0070] Screening and processing the historical flight data of UAVs to construct a data set that conforms to the Markov decision process (MDP) requires systematic analysis and reconstruction of the data. First, the extracted observations should include sensor data directly related to flight performance, such as attitude angle controller (ATT), inertial measurement unit (IMU) and integrated navigation extended Kalman filter data (XKF), which provide the necessary state information to help describe the flight state of the UAV at every moment.

[0071] When constructing the state space, it is necessary to combine these sensor data into a unified state representation. The basic state quantities required for drone aerodynamic environment cloning and strategy model training include: three-dimensional position (X, Y, Z), speed (Vx, Vy, Vz), gyroscope speed (GyrX, GyrY, GyrZ), acceleration (AccX, AccY, AccZ), attitude (Pitch, Roll, Yaw) and other information. In order to more effectively train the aircraft aerodynamic network model, the aircraft attitude angle is normalized to the continuous [-1, 1] space by sine and cosine. In the body coordinate system, the processed continuous observation obs is used. t To predict the difference delta between the next moment and the current moment t ,

[0072] Next, the action quantity screening should include the control input that affects the state of the drone at the next moment, the aerodynamic environment cloning of the drone and the training of the strategy model. The basic action quantities required are: roll control surface deflection angle, elevator control surface deflection angle, throttle, yaw control surface deflection angle, etc. These action quantities should correspond to each state to ensure that the model can learn the corresponding decision-making strategy in different states. The action space needs to be normalized.

[0073] Finally, the filtered and processed data set is organized into a state-action format to ensure that each pair of (state, action) can correspond to the subsequent state transfer. This structured data needs to conform to the Markov characteristics to provide a solid foundation for subsequent aerodynamic model cloning and reinforcement learning strategy model training, so that the final strategy model can control the UAV to autonomously complete a variety of aerobatic maneuvers.

[0074] In the specific implementation of step S3, a UAV aerodynamic network model is constructed;

[0075] Constructing a UAV aerodynamic network model means selecting an appropriate neural network for cloning UAV aerodynamics. Generally, the network structures used are multi-layer ResNet, Transform or RNN. The more common name for the UAV aerodynamic network model in model-based reinforcement learning is virtual environment or virtual world. When this method is technically explained, the name of the virtual environment will be used.

[0076] In the specific implementation of step S4, the UAV aerodynamic network model is trained using flight data that conforms to the Markov decision process to obtain a UAV aerodynamic network model that fits the real data;

[0077] The mathematical form of the Markov decision process is:

[0078] State sequence: S = (s0, s1, s2, ..., s t ), where s t ∈S represents the state at time step t.

[0079] Action sequence: A=(a0,a1,a2,…,a t ), where a t ∈A represents the action at time step t.

[0080] Transition probability, state transition can be expressed as: P(s t+1 |s t ,a t ), indicating that in state s t Take action a t Then, transfer to the next state s t+1 probability.

[0081] The whole track form:

[0082] The training process of the UAV aerodynamic network model is the process of learning the state transition probability P in the real world. The training goal is to predict any state of the UAV and any legal action (s any ,a any ), the probability transition in the real world is the same as the probability transition output by the network model.

[0083] In order to more accurately clone the aerodynamic model of drones in the real world, reduce the cumulative compound error of multi-step predictions in the virtual environment, and avoid the overfitting problem in the neural network, this method abandons the traditional virtual environment cloning based on supervised learning and uses an optimization method based on adversarial neural networks and distribution alignment loss.

[0084] Starting from the starting point of a trajectory, the state and action (s0, a0) of the starting point are input into the generating network, and the state at the next moment is output. Will Combined with the action amount a1 at the next moment in the trajectory, we get Input the generated network and output the state quantity at the next moment And so on Get the predicted state of the entire trajectory

[0085] The predicted state quantity of the entire trajectory and the state quantities s0,s1,s2,…,s in the historical data t Calculate the segment distribution alignment loss. Set the segment length L, divide the entire trajectory into N segments, and 1 / 3 of the state quantities before and after two adjacent segments are repeated. Calculate the distribution alignment loss for each segment. The loss function can use KL Divergence, Wasserstein Distance, etc.

[0086] The discriminant network is used to determine whether a piece of data is real data, and the training data is historical data.

[0087] s0,a0,s1,a1,s2,a2,…,s t And the generated data of the generative network A segmented strategy was also adopted during the training process.

[0088] In order to generate more actions that the network has seen and improve the generalization of actions, this method provides an optional second-stage training. The generator in the second stage will simultaneously generate the state and action of the next moment to obtain the predicted state of the entire trajectory. Use s0,a0,s1,a1,s2,a2,…,s t and Compute the piecewise distribution alignment loss and optimize the discriminator.

[0089] In the specific implementation of step S5, a reinforcement learning algorithm is used to train a strategy model in a network model of drone aerodynamics that fits real data;

[0090] Dual environment test: In order to test the learning effect of the control strategy, the historical flight data will be divided into two parts in step S4, which are used to train the virtual environment respectively, to obtain two different virtual environments. The same algorithm and parameters are used to train the strategy, to obtain two control strategies, and the two strategies are tested in the two virtual environments respectively. Figure 3 As shown in the figure, the test results of a single strategy in both virtual environments are excellent and the difference is not large, indicating that the strategy learning is excellent.

[0091] The trained UAV aerodynamic network model serves as a data-driven real-world virtual simulation environment for the UAV. Any reinforcement learning algorithm can be connected to this virtual environment to train the control strategy. Generally, a stable PPO algorithm is used.

[0092] The key point of training a general control strategy based on the UAV aerodynamic network model (virtual environment) is the setting of the reward function when training the strategy. This method sets the difference between the current state and the target state in position (X, Y, Z), overload (AccX, AccY, AccZ), speed (Vx, Vy, Vz), and attitude (Pitch, Roll, Yaw) as the reward function. Before the target state changes, the smaller the difference between the UAV and the target state, the greater the reward. During the training process, the target state changes randomly every 2 seconds within a certain range. The strategy model has seen the difference with various target points and can reach the target state within two seconds.

[0093] In the specific implementation of step S6, the specific aerobatic flight trajectory is processed and input into the trained strategy model, and the model outputs an action to guide the UAV to perform the specific aerobatic flight action;

[0094] The trained strategy model is imported into the drone control unit to calculate the target trajectory of a specific aerobatic flight, and the key points in the target trajectory are set as the drone target points. The strategy continuously controls the drone to chase the key points of the target trajectory to achieve autonomous aerobatic flight of the drone. In the actual flight of the drone, it is also necessary to set up emergency stops in special situations or have experienced pilots remotely control the drone to avoid accidents and losses during the strategy control process.

[0095] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A general method for autonomous aerobatic flight of unmanned aerial vehicles based on offline reinforcement learning, characterized in that: The method comprises the following steps: S1. Collect the historical flight data of the drone, and try to cover all the postures of the drone during flight; S2, screening the key observations and actions of the UAV, processing the observations and actions, and constructing flight data that conforms to the Markov decision process; S3, build a UAV aerodynamic network model; S4. Using the flight data that conforms to the Markov decision process, the UAV aerodynamic network model is trained to obtain a UAV aerodynamic network model that fits the real data; S5. Use reinforcement learning algorithm to train the policy model in the network model of drone aerodynamics that fits the real data; S6. The specific aerobatic flight trajectory is processed and input into the trained strategy model, and the model outputs actions to guide the UAV to perform specific aerobatic flight actions.

2. A general method for autonomous aerobatic flight of unmanned aerial vehicle based on offline reinforcement learning according to claim 1, characterized in that: The step S1 collects historical flight data of the drone, and tries to cover various postures of the drone during flight, including: First, use ArduPilot as the flight control system and use its logging function to collect data. During flight, enable detailed logging by setting parameters; After the flight, the ground station software is used to download the flight log, and tools are used to perform data analysis. By parsing the log file, key flight data is extracted to provide data support for subsequent aircraft aerodynamic network model cloning, optimization of flight control algorithms, and training of aerobatic flight strategies.

3. The general method for autonomous aerobatic flight of unmanned aerial vehicle based on offline reinforcement learning according to claim 1, characterized in that: The step S2 includes screening and processing the historical flight data of the UAV to construct a data set that conforms to the Markov decision process, which requires systematic analysis and reconstruction of the data: First, the extracted observations should include sensor data directly related to flight performance; When constructing the state space, it is necessary to combine the sensor data into a unified state representation, the UAV aerodynamic environment cloning and strategy model training; Next, the action volume screening should include the control input that affects the state of the drone at the next moment, the drone’s aerodynamic environment cloning, and the strategy model training; Finally, the filtered and processed data set is organized into a state-action format to ensure that each pair of state and action can correspond to the subsequent state transfer. The structured data needs to conform to the Markov characteristics to provide a solid foundation for the subsequent aerodynamic model cloning and reinforcement learning strategy model training, so that the final strategy model can control the UAV to autonomously complete a variety of aerobatic maneuvers.

4. The general method for autonomous aerobatic flight of unmanned aerial vehicle based on offline reinforcement learning according to claim 3, characterized in that: The required basic state quantities are: three-dimensional position (X, Y, Z), velocity (Vx, Vy, Vz), gyroscope speed (GyrX, GyrY, GyrZ), acceleration (AccX, AccY, AccZ), attitude (Pitch, Roll, Yaw) information. In order to more effectively train the aircraft aerodynamic network model, the aircraft attitude angle is normalized to the continuous [-1, 1] space by sine and cosine. In the body coordinate system, the processed continuous observation obs is used. t To predict the difference delta between the next moment and the current moment t , 5. The general method for autonomous aerobatic flight of unmanned aerial vehicle based on offline reinforcement learning according to claim 3, characterized in that: The required basic action quantities are: roll surface angle, elevator surface angle, throttle, and yaw surface angle. These action quantities should correspond to each state to ensure that the model can learn the corresponding decision-making strategy under different states. The action space needs to be normalized.

6. The general method for autonomous aerobatic flight of unmanned aerial vehicle based on offline reinforcement learning according to claim 1, characterized in that: In the specific implementation of step S3, the step of constructing the UAV aerodynamics network model includes constructing the UAV aerodynamics network model to express the selection of an appropriate neural network for cloning UAV aerodynamics, and the network structure includes multi-layer ResNet, Transform or RN.

7. The general method for autonomous aerobatic flight of unmanned aerial vehicle based on offline reinforcement learning according to claim 1, characterized in that: The specific steps of step S4 are to train the UAV aerodynamic network model using flight data that conforms to the Markov decision process to obtain a UAV aerodynamic network model that fits the real data; The mathematical form of the Markov decision process is: State sequence: S = (s0, s1, s2, ..., s t ), where s t ∈S represents the state at time step t, Action sequence: A=(a0,a1,a2,…,a t ), where a t ∈A represents the action at time step t, Transition probability, state transition can be expressed as: P(s t+1 |s t ,a t ), indicating that in state s t Take action a t Then, transfer to the next state s t+1 The probability of The whole track form: The training process of the UAV aerodynamic network model is the process of learning the state transition probability P in the real world. The training goal is to predict any state of the UAV and any legal action (s any ,a any ), the probability transition in the real world is the same as the probability transition output by the network model.

8. The general method for autonomous aerobatic flight of unmanned aerial vehicle based on offline reinforcement learning according to claim 1, characterized in that: In order to more accurately clone the real-world drone aerodynamic model, reduce the cumulative compound error of multi-step predictions in the virtual environment, and avoid the overfitting problem in the neural network, an optimization method based on adversarial neural networks and distribution alignment loss is used: 1) Starting from the starting point of a trajectory, the state and action (s0, a0) of the starting point are input into the generative network, and the state at the next moment is output Will Combined with the action amount a1 at the next moment in the trajectory, we get Input the generated network and output the state quantity at the next moment And so on Get the predicted state s0 of the entire trajectory, 2) The predicted state quantity s0 of the entire trajectory, and the state quantities s0,s1,s2,…,s in the historical data t Calculate the segment distribution alignment loss, where the segment length L is set, the entire trajectory is divided into N segments, 1 / 3 of the state quantities before and after two adjacent segments are repeated, and the distribution alignment loss is calculated for each segment; 3) The discriminant network is used to determine whether a piece of data is real data. The training data is the historical data s0, a0, s1, a1, s2, a2, …, s t and the generated data s0,a0 of the generated network, A segmented strategy was also used during the training process; 4) In order to generate more actions that the network has seen and improve the generalization of actions, an optional second-stage training is provided. The generator in the second stage will simultaneously generate the state and action of the next moment to obtain the predicted state s0, a0, Use s0,a0,s1,a1,s2,a2,…,s t and s0,a0, Compute the piecewise distribution alignment loss and optimize the discriminator.

9. The general method for autonomous aerobatic flight of unmanned aerial vehicle based on offline reinforcement learning according to claim 1, characterized in that: In the specific implementation of step S5, the strategy model is trained using a reinforcement learning algorithm in a network model of drone aerodynamics that fits real data: Dual environment test: In order to test the learning effect of the control strategy, in step S4, the historical flight data will be divided into two parts, which are used to train the virtual environment respectively to obtain two different virtual environments. The same algorithm and parameters are used when training the strategy to obtain two control strategies. The two strategies are tested in the two virtual environments respectively. The test results of a single strategy in the two virtual environments are excellent, and the difference is not large, indicating that the strategy learning is excellent; The trained UAV aerodynamic network model serves as a data-driven real-world virtual simulation environment for the UAV. Any reinforcement learning algorithm can be connected to this virtual environment to train the control strategy. The general control strategy based on the UAV aerodynamic network model training is key in the setting of the reward function when training the strategy. The difference in position (X, Y, Z), overload (AccX, AccY, AccZ), speed (Vx, Vy, Vz), and attitude (Pitch, Roll, Yaw) between the current state and the target state is set as the reward function. Before the target state changes, the smaller the difference between the UAV and the target state, the greater the reward. During the training process, the target state changes randomly every 2 seconds within a certain range. The strategy model has seen the difference with various target points and reaches the target state within two seconds.

10. The general method for autonomous aerobatic flight of unmanned aerial vehicle based on offline reinforcement learning according to claim 1, characterized in that: In step S6, the specific aerobatic flight trajectory is processed and input into the trained strategy model, and the model outputs an action to guide the UAV to perform the specific aerobatic flight action; The strategy model after training is imported into the UAV control unit, the target trajectory of a specific aerobatic flight is calculated, and the key points in the target trajectory are set as the target points of the UAV. The strategy continuously controls the UAV to chase the key points of the target trajectory to realize the autonomous aerobatic flight of the UAV. In the actual flight of the UAV, special emergency stops are set separately or experienced pilots remotely control the UAV.

Images