A rotor unmanned aerial vehicle flight control training and migration method and system based on reinforcement learning

Abstract

The application provides a rotor unmanned aerial vehicle flight control training and migration method based on reinforcement learning, and specifically comprises the following steps: step 100, setting multiple typical training tasks and reward functions in a high-fidelity simulation environment configured; step 210, sampling a batch of tasks from the constructed tasks according to priorities, and training a strategy by using a reinforcement learning method; step 220, adding disturbance to construct multiple environments during the training process, including state disturbance and dynamics disturbance, training the strategy with disturbance to improve the adaptability and robustness of the strategy; and step 300, verifying the performance of the strategy in a verification scene, correcting the strategy based on verification data, and finally deploying the strategy to a rotor unmanned aerial vehicle. The application adopts multiple fusion state information, realizes low, medium and high speed agile control of the rotor unmanned aerial vehicle in each flight stage, and migrates to a real machine.

Description

Technical Field

[0001] This application relates to the field of autonomous control of rotary-wing unmanned aerial vehicles (UAVs) based on reinforcement learning, and more particularly to a method and system for flight control training and transfer of rotary-wing UAVs based on reinforcement learning. Background Technology

[0002] Rotary-wing drones, due to their small size, lightweight design, ability to take off and land vertically in confined spaces, and ease of operation, are now widely used in industries such as manufacturing, agriculture, security, and defense. In some industry applications, rotary-wing drones are already capable of autonomously performing functions such as takeoff, landing, hovering, and flying along planned paths, with some more flexible operations performed by the pilot. On the control side, control-based methods typically design controllers based on known dynamic models of the rotary-wing drone, such as PID (Proportional Integral-Derivative) controllers, to control the attitude and position of the drone. On the path planning side, a path is usually calculated based on the target position and the drone's own position using a path planning algorithm before the drone begins its flight.

[0003] However, traditional control methods suffer from several drawbacks. Firstly, they require extensive domain knowledge for controller design and calibrate some controller parameters. Secondly, feedback control methods, such as PID control, are prone to overshoot and other excessive adjustments, resulting in poor control response timeliness. Path planning necessitates precise knowledge of the drone's current position, and path errors accumulate with positioning accuracy errors. Furthermore, the rotorcraft drone cannot move too fast during control, otherwise it may miss planned waypoints, leading to mission failure. For instance, in strong headwinds, increasing the rotorcraft drone's speed to counteract wind effects, such as switching to certain consumer-grade rotorcraft flight modes, presents significant control challenges. In such situations, manufacturers typically do not provide automatic return-to-home functionality, and obstacle avoidance features at low to medium speeds often fail.

[0004] Therefore, this application proposes a training method for autonomous flight control of rotary-wing UAVs based on reinforcement learning. It adopts multi-fusion situational information to achieve agile control of rotary-wing UAVs at low, medium and high speeds in various flight stages, and transfers this method to actual aircraft. Summary of the Invention

[0005] To address the aforementioned issues, this invention discloses a method and system for training and transferring flight control of rotary-wing UAVs based on reinforcement learning. By employing multi-fusion situational information, it enables agile control of rotary-wing UAVs at low, medium, and high speeds in various flight phases and transfers this control to real-world aircraft.

[0006] A method for training and transferring flight control of a rotary-wing UAV based on reinforcement learning, specifically including the following steps:

[0007] Step 100: Set up a variety of typical training tasks and full-process control training tasks in the configured high-fidelity simulation environment, including initial situation, task objective, termination condition and reward function.

[0008] Step 210: Sample a batch of tasks from the constructed tasks according to priority, and train the policy using reinforcement learning methods;

[0009] Step 220: During the training process, perturbations are introduced to construct various environments, including situational perturbations and dynamic perturbations, to train the policy with perturbations, thereby improving the policy's adaptability and robustness.

[0010] Step 300: Perform policy performance verification in the verification scenario, and migrate and correct the policy based on the verification data, and finally deploy it on the rotary-wing UAV.

[0011] Furthermore, in step 100, the flight phase of the rotary-wing UAV is divided into takeoff, flight, and return phases. Multiple different initial states and mission objectives are set for each type of mission. In the flight phase, different target points need to be set. The rotary-wing UAV should approach the target point with the required attitude and speed to complete a partial flight mission. A reward function is also designed.

[0012] Furthermore, the reward for getting closer to the target is set as... Where δ and C represent the distance to the target and the normalization constant determined according to the task, respectively. When training the reinforcement learning agent, the difference between the distance reward to the target point in the current step and the previous step is used for calculation. After the task is set up, the task set is denoted as T.

[0013] Furthermore, steps 210 and 220 are specifically performed as follows:

[0014] A100. Construct an initial policy model using a deep neural network, and set the same weights w for each task. i =1;

[0015] A200, according to probability distribution Repeated sampling yields N tasks, and individual sampling accumulates 10*N full-process flight tasks;

[0016] A300. In each task, the initial state S0 is sampled from the corresponding initial state set. Using the policy model, action a is executed according to the current situation. The first step is the initial situation. The simulation environment returns the state S' and the corresponding reward information r at the next moment. This process continues until the task is completed, and finally a batch of trajectories in the form of (S0, a0, r0, S1, a1, r1, ..., Sn) is obtained.

[0017] A400 randomly adds random noise to the state and actions on the trajectory collected in A300 to perturb them, and uses a reinforcement learning algorithm to optimize the policy.

[0018] A500, Expected cumulative reward R under the current strategy i =∑ t＝1 γ t r t , γ∈[0,1], i=1,2,…,N, and recalculate the sampling weights for each task i. This increases the probability of poorly trained tasks, i.e. tasks with low expected returns, being sampled later, where T represents the task set;

[0019] A600, repeat steps A200 to A500 until the expected return for each task no longer increases.

[0020] Furthermore, the policy model is converted into ONNX format and loaded onto the UAV. The trained policy is then validated in a validation task. If the validation passes, the model is ready to be migrated to the UAV board for deployment. Otherwise, based on the validation results, the weights of the sub-tasks that performed poorly in the validation task are added, and then A200 to A600 are executed for fine-tuning.

[0021] A flight control training and transfer system for rotary-wing UAVs based on reinforcement learning includes multiple typical training task modules: setting up multiple typical training tasks and full-process control training tasks in a high-fidelity simulation environment, including initial situation, task objective, termination condition and reward function;

[0022] The reinforcement learning-based training module samples a batch of tasks according to priority from the constructed tasks and trains the policy using reinforcement learning methods; during the training process, perturbations are added to construct various environments to train policies with perturbations.

[0023] The strategy verification and migration module verifies the strategy performance in a verification scenario, migrates and corrects the strategy based on the verification data, and finally deploys it on the rotary-wing UAV.

[0024] The beneficial effects of this invention are:

[0025] This invention employs reinforcement learning to train the autonomous flight control strategy of a rotary-wing UAV, which can better address the need for real-time planning adjustments and control under high-speed motion conditions. During the training process, multi-task segmented training combined with full-process task training is used, which can reduce the difficulty of training the strategy and improve the generalization of the strategy during deployment. The directional perturbation added during training can further reduce the dependence on high-precision sensors. The strategy can autonomously control itself under situational data with certain errors, which can improve the adaptability and robustness of the strategy when transferred to real-world scenarios. Attached Figure Description

[0026] Figure 1 A flowchart of the method of the present invention. Detailed Implementation

[0027] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, and the terms "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0028] This embodiment is based on reinforcement learning and uses a high-fidelity simulation environment to train the entire flight mission. This high-fidelity simulation environment requires a corresponding rotorcraft UAV dynamics model and provides corresponding situational awareness and control interfaces. This allows the UAV's current situation to be obtained from the simulation environment, and the strategy model can output control variables based on the current situation. The situation is typically a fusion of multiple sensors, including vision, GPS, and infrared, and usually contains some error. To enable the trained rotorcraft UAV to achieve autonomous flight control under various missions, this invention pre-sets multiple missions in the simulation and provides a perturbation training method, allowing the trained flight control strategy to be better transferred to the actual aircraft.

[0029] like Figure 1 As shown, the specific operation steps and units of the present invention are as follows:

[0030] 100. Set up various typical training tasks and full-process control training tasks in the configured high-fidelity simulation environment, including initial situation, task objective, termination condition and reward function;

[0031] 210. Sample a batch of tasks from the constructed tasks according to priority, and use reinforcement learning methods to train the policy;

[0032] 220. Incorporate perturbations into the training process to construct various environments, including situational perturbations and dynamic perturbations, and train policies with perturbations to improve the adaptability and robustness of the policies.

[0033] 300. Conduct policy performance verification in the verification scenario, and migrate and correct the policy based on the verification data, and finally deploy it on the rotary-wing UAV.

[0034] To illustrate the technical solutions related to this invention in more detail, the invention will be further described below with reference to specific embodiments and accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0035] Example 1:

[0036] Based on the flight phases of a rotary-wing UAV, the process is divided into takeoff, flight, and return phases. Multiple different initial states and mission objectives are set for each type of task. Specifically, the flight phase requires setting different target points. The rotary-wing UAV must approach the target point at the required attitude and speed to complete a partial flight mission. A reward function is designed; for example, the reward for approaching the target is set as... Where δ and C represent the distance to the target and a normalization constant determined according to the task, respectively, they are calculated using the difference between the distance reward to the target point in the current step and the previous step during reinforcement learning agent training. After completing the task settings, the task set is denoted as T, and training is performed according to the following steps:

[0037] A100. Construct the initial policy model, typically using a deep neural network, and assign the same weights w for each task. i =1;

[0038] A200, according to probability distribution Repeated sampling yields N tasks, and a certain number (e.g., 5*N) of the entire flight process tasks are sampled separately.

[0039] A300. In each task, the initial state S0 (including position, velocity, direction, etc., which is usually obtained by fusing information such as vision and GPS positioning) is sampled from the corresponding initial state set. Using the policy model, action a is executed according to the current situation (the first step is the initial situation). The simulation environment returns the state S' of the next moment and the corresponding reward information r until the task is completed. Finally, a batch of trajectories in the form of (S0, a0, r0, S1, a1, r1, ..., Sn) can be obtained.

[0040] A400: Randomly add random noise to the state and actions on the trajectory collected in [A300] to perturb them, and use reinforcement learning algorithms, such as PPO or SAC algorithms, to optimize the policy.

[0041] A500, Expected cumulative reward R under the current strategy i =∑ t＝1 γ t r t , γ∈[0,1], i=1,2,…,N, and recalculate the sampling weights for each task i. This increases the probability of poorly trained tasks (tasks with low expected returns) being sampled later, where T represents the task set;

[0042] A600, Repeat steps [A200] to [A500] until the expected reward for each task no longer increases;

[0043] A700 converts the policy model into ONNX format and loads it onto the UAV. In the verification task, it verifies the trained policy. If the verification is successful, it is ready to be migrated to the UAV board for model deployment. Otherwise, based on the verification results, it adds weights to the sub-tasks that did not perform well in the verification task, and then executes [A200] to [A600] for fine-tuning.

[0044] Preliminary experiments were conducted in Gazebo simulations. Based on our understanding, there are some differences between the flight control systems of rotorcraft and fixed-wing aircraft. For rotorcraft drones, fully autonomous flight control requires very high-precision GPS; otherwise, it is difficult to design corresponding control strategies (slight errors can lead to serious control deviations). Reinforcement learning can autonomously learn how to control the high-speed flight of rotorcraft drones at different high-speed states (20m / s or even 30-40m / s), and it has a high tolerance for GPS errors.

[0045] The autonomous flight control strategy obtained through the above training steps can adjust the flight volume in real time to better reach the expected goal or complete the task according to the set target (pre-set task or target point, or sent through communication, etc.). It can perform autonomous control in high-speed mode. After being deployed on UAVs, it can complete autonomous flight control in a variety of different flight conditions, solving the problems of current autonomous flight control methods that are difficult to adjust the plan in real time and difficult to cope with high-speed control.

[0046] In this embodiment, a multi-task and priority sampling autonomous flight control training method is adopted during the training process to reduce the training difficulty. Directional perturbations are added during training to improve the adaptability and robustness of the strategy, meaning it can tolerate errors in situational awareness and control effectiveness, thus providing a guarantee for strategy transfer.

[0047] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A method for training and transferring flight control of a rotary-wing unmanned aerial vehicle based on reinforcement learning, characterized in that: Specifically, the steps include the following: Step 100: Set up a variety of typical training tasks and full-process control training tasks in the configured high-fidelity simulation environment, including initial situation, task objective, termination condition and reward function. Step 210: Sample a batch of tasks from the constructed tasks according to priority, and train the policy using reinforcement learning methods; Step 220: During the training process, perturbations are introduced to construct various environments, including situational perturbations and dynamic perturbations, to train the policy with perturbations, thereby improving the policy's adaptability and robustness. Steps 210 and 220 are performed as follows: A100. Construct an initial policy model using a deep neural network, and assign the same weights to each task. ; A200, according to probability distribution The entire flight mission is obtained by repeated sampling to obtain N tasks, and each task is sampled individually. The sampling multiple is an integer multiple of the number of tasks, usually set to 5, 10, or 20 times, depending on the specific task and the simulator's operating efficiency. A300. In each task, sample the initial state S0 from the corresponding initial state set, use the policy model, and execute action a according to the current situation. The first step is the initial situation. The simulation environment returns the state S' and the corresponding reward information r at the next moment until the task is completed. Finally, the trajectory (S0, a0, r0, S1, a1, r1, …, Sn) is obtained. A400 randomly adds random noise to the state and actions on the trajectory collected in A300 to perturb them, and uses a reinforcement learning algorithm to optimize the policy; A500, Expected cumulative reward under the current strategy And recalculate the sampling weights for each task i. This increases the probability of poorly trained tasks, i.e. tasks with low expected returns, being sampled later, where T represents the task set. A600, repeat steps A200~A500 until the expected reward of each task no longer increases; Step 300: Perform policy performance verification in the verification scenario, and migrate and correct the policy based on the verification data, and finally deploy it on the rotary-wing UAV.

2. The method for training and transferring flight control of a rotary-wing unmanned aerial vehicle based on reinforcement learning according to claim 1, characterized in that: In step 100, the flight phase of the rotary-wing UAV is divided into take-off, flight, and return phases. Multiple different initial states and mission objectives are set for each type of mission. In the flight phase, different target points need to be set. The rotary-wing UAV should approach the target point with the required attitude and speed to complete a partial flight mission. A reward function is also designed.

3. A method for training and transferring flight control of a rotary-wing unmanned aerial vehicle based on reinforcement learning according to claim 1 or 2, characterized in that: Set the reward for getting closer to the target as ,in Let T represent the distance to the target and the normalization constant determined according to the task, respectively. When training the reinforcement learning agent, the difference between the distance reward to the target point in the current step and the previous step is used for calculation. After the task is set up, the task set is denoted as T.

4. The method for training and transferring flight control of a rotary-wing unmanned aerial vehicle based on reinforcement learning according to claim 1, characterized in that: The policy model is converted into ONNX format and loaded onto the UAV. The trained policy is then validated in the validation task. If the validation passes, the model is ready to be migrated to the UAV board for deployment. Otherwise, based on the validation results, the weights of the subtasks that did not perform well in the validation task are added, and then A200~A600 are executed for fine-tuning.

5. A reinforcement learning-based rotorcraft unmanned aerial vehicle (UAV) flight control training and transfer system, based on the reinforcement learning-based rotorcraft UAV flight control training and transfer method according to any one of claims 1-4, characterized in that: Multiple typical training task modules: Set up multiple typical training tasks and full-process control training tasks in a high-fidelity simulation environment, including initial situation, task objective, termination condition and reward function; The reinforcement learning-based training module samples a batch of tasks from the constructed tasks according to priority and uses reinforcement learning methods to train the policy. During training, perturbations are introduced to create various environments, and strategies with perturbations are trained. The strategy verification and migration module verifies the strategy performance in a verification scenario, migrates and corrects the strategy based on the verification data, and finally deploys it on the rotary-wing UAV.

Images