Unmanned aerial vehicle autonomous navigation method based on fusion of imitation learning and reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle autonomous navigation method based on imitation learning and reinforcement learning fusion, and belongs to the technical field of unmanned aerial vehicle processing, and the method comprises the following steps: S1, collecting a data set of an unmanned aerial vehicle, and carrying out normalization processing on the data set to obtain a standard data set; s2, constructing an Actor network by utilizing expert operation data of the standard data set; s3, inputting the depth image and the state information of the standard data set into an Actor network, and generating a pre-judgment action; and S4, inputting the pre-judgment action into a discriminator to complete autonomous navigation. Through introduction of the generative adversarial imitation learning module, the unmanned aerial vehicle can continuously optimize a navigation strategy, and high robustness and adaptability can be maintained even in a complex environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle processing, and in particular relates to an unmanned aerial vehicle autonomous navigation method based on the fusion of imitation learning and reinforcement learning. Background Art

[0002] In recent years, autonomous navigation technology of unmanned aerial vehicles has shown important application value in many fields such as environmental monitoring, logistics and transportation, and agricultural inspection. However, efficient navigation and obstacle avoidance in complex environments remain the core challenges facing autonomous control of unmanned aerial vehicles. During autonomous navigation, unmanned aerial vehicles need to make intelligent decisions in a variety of complex environments and make real-time adjustments based on dynamically changing information. To achieve this goal, reinforcement learning and imitation learning have become important methods for autonomous navigation technology of unmanned aerial vehicles.

[0003] Reinforcement learning has made significant progress in UAV navigation and control. Reinforcement learning uses environmental feedback rewards to continuously optimize its decision-making behavior through continuous interaction between the agent and the environment. Policy-based reinforcement learning methods, such as the soft actor-critic algorithm, have become one of the core algorithms for UAV navigation due to their stability and efficiency. SAC can achieve continuous control in the UAV navigation environment and effectively cope with the challenges of complex terrain and uncertain obstacles. However, in practical applications, reinforcement learning often requires a lot of data collection and computing resources, and the training process is also time-consuming.

[0004] In order to improve the training efficiency of intelligent agents, imitation learning provides an efficient training method that shortens the learning time of intelligent agents by learning expert demonstration data. Behavior cloning is a typical imitation learning method that enables intelligent agents to quickly learn the operation strategies of experts by minimizing the differences between the behaviors of intelligent agents and experts. In the UAV navigation task, behavior cloning trains intelligent agents through trajectory data demonstrated by experts, enabling them to quickly master basic navigation skills. However, behavior cloning may have the problem of error accumulation when facing environmental changes, causing the intelligent agent to perform unstable in scenarios outside of training.

[0005] Generative adversarial learning is another imitation learning technique that combines the idea of ​​generative adversarial networks. GAIL uses a discriminator to distinguish the behavior of experts and agents, thereby guiding the agent to imitate the expert's strategy and learn the underlying pattern of the expert's behavior. Compared with behavioral cloning, GAIL has more advantages in adapting to complex environments and dynamic changes, which helps drones to autonomously explore and navigate in unseen environments.

[0006] In the autonomous navigation of UAVs, the combination of imitation learning and reinforcement learning can give full play to the advantages of both and improve the adaptability of intelligent agents in complex environments. By introducing behavioral cloning and GAIL on the basis of reinforcement learning, the intelligent agent can first obtain the initial navigation ability through imitation learning, and then use reinforcement learning for fine-tuning to achieve the optimization of navigation performance. This strategy fusion method can effectively reduce the training time and improve the autonomous navigation ability of UAVs in complex environments. Therefore, the present invention studies the autonomous navigation system of UAVs based on the combination of imitation learning and reinforcement learning, and explores its application potential in dynamic and complex environments.

[0007] Although imitation learning and reinforcement learning have made significant progress in autonomous navigation of UAVs, their respective limitations still affect their navigation performance and robustness in complex environments. The following are the main defects in existing technologies.

[0008] Imitation learning, especially behavior cloning methods, trains the navigation capabilities of drones by learning from expert demonstration data. However, behavior cloning methods are highly dependent on the quality and diversity of expert data. When the training data is insufficient or lacks diversity in the environment, the agent often performs poorly in unseen environments, prone to error accumulation, and causes the navigation strategy to fail. In addition, imitation learning lacks direct use of reward signals, making the agent less adaptable in complex and unknown environments.

[0009] Although reinforcement learning algorithms such as SAC can help drones obtain efficient navigation strategies in complex environments, their high performance often comes at the cost of a lot of training. Drones interact with the environment in complex environments and rely on reward signals for continuous self-optimization, which requires a lot of sampling and repeated training. Due to the high real-time requirements of drones, relying solely on reinforcement learning to train autonomous navigation strategies is usually inefficient and time-consuming.

[0010] The generative adversarial imitation learning method introduces the discriminator and generator into the learning process of the UAV, enabling the intelligent agent to imitate the expert strategy. However, in high-dimensional state spaces (e.g., containing deep images and environmental state features), the optimization difficulty of the discriminator and generator of the generative adversarial imitation learning method increases significantly, which can easily lead to unstable training and may produce suboptimal solutions. Overfitting of the discriminator and insufficient diversity of the generator may make it difficult for the UAV to generate effective adversarial rewards in complex scenarios, thus affecting the navigation effect.

[0011] In summary, the application of existing imitation learning and reinforcement learning technologies in autonomous navigation still has deficiencies in data dependence, training efficiency, model stability, strategy coordination, and environmental adaptability. In view of these shortcomings, exploring new methods of integrating imitation learning and reinforcement learning to improve the navigation performance of drones in complex environments is a key direction for further improvement. Summary of the invention

[0012] In order to solve the above problems, the present invention proposes an autonomous navigation method for a UAV based on the fusion of imitation learning and reinforcement learning.

[0013] The technical solution of the present invention is: a method for autonomous navigation of a UAV based on the fusion of imitation learning and reinforcement learning comprises the following steps:

[0014] S1, collect the data set of the drone, and normalize the data set to obtain a standard data set;

[0015] S2, build an Actor network using expert operation data from standard datasets;

[0016] S3, input the depth image and state information of the standard data set into the Actor network to generate predicted actions;

[0017] S4: Input the predicted action into the discriminator to complete autonomous navigation.

[0018] Furthermore, in S1, the dataset of the UAV includes the depth image, status information and expert operation data of the UAV.

[0019] Furthermore, S2 includes the following sub-steps:

[0020] S21. Construct a reward function for the expert operation data;

[0021] S22. Based on the reward function, construct the behavior cloning learning objective;

[0022] S23. Construct the loss function of the Actor network based on the behavior cloning learning goal.

[0023] Furthermore, in S21, the expression of the reward function Reward is:

[0024] Reward=

[0025] reward_distance-λ1*punishment_pose-λ2*punishment_obs-λ3*

[0026] punishment_action-λ4*yaw_error_cost;

[0027] Wherein, reward_distance represents the reward item used to encourage the drone to fly towards the target point, λ1 represents the weight coefficient of penalty_pose, which is used to control the impact of the drone's posture deviation on the total reward, penalty_pose represents the comprehensive penalty item used to evaluate the deviation of the drone's current posture relative to the target posture, λ2 represents the weight coefficient of penalty_obs, which is used to control the penalty weight of the minimum distance between the drone and the obstacle, penalty_obs represents the penalty item used to evaluate the minimum distance between the drone and the obstacle, λ3 represents the weight coefficient of penalty_action, which is used to control the penalty weight of the stability and efficiency of the drone's action, penalty_action represents the comprehensive penalty item for evaluating the stability and efficiency of the drone when performing the action, λ4 represents the weight coefficient of yaw_error_cost, yaw_error_cost represents the penalty item used to measure the error between the current heading of the drone and the target heading.

[0028] Furthermore, in S22, the behavior cloning learning target θ * The expression is:

[0029]

[0030] In the formula, θ represents the dependency and model parameters, E (s,a)～B represents the expected value of the state and action space sampled from the data set, s represents the state, α represents the first weight coefficient, β represents the second weight coefficient, λ represents the third weight coefficient, a represents the action space, B represents the data set, π θ represents the strategy distribution, represents the loss function, which is used to measure the difference between the strategy output action and the target action, R(·) represents the regularization term, and D kL represents the Kullback-Leibler divergence, and P(·) represents the prior distribution of domain knowledge or expert behavior.

[0031] Furthermore, in S23, the loss function The expression is:

[0032]

[0033] In the formula, N represents the number of samples, θ represents the dependency and model parameters, w(·) represents the data weight term, and s i represents the i-th state, a i represents real actions provided by experts, represents the action predicted by the model, π θ represents the strategy distribution, a represents the action space, represents the gradient of the action predicted by the model to the state, α represents the first weight coefficient, λ represents the third weight coefficient, β represents the second weight coefficient, and D KL represents the Kullback-Leibler divergence, and P(·) represents the prior distribution of domain knowledge or expert behavior.

[0034] Furthermore, in S3, the discriminator loss function The expression is:

[0035]

[0036] In the formula, represents the real sample loss, W1(·) represents the weight of the real sample, s represents the state, α represents the first weight coefficient, represents the discriminator function, E ρE represents the loss of generated samples, W2(·) represents the weight of generated samples. W1(·) is used to adjust the importance of each state-action pair (s, a) in the loss. Represents the output value on the state-action pair (s, a), usually a probability, and W2(·) is used to adjust the importance of each state-action pair (s, a) in generating sample loss.

[0037] The beneficial effects of the present invention are as follows: the present invention significantly improves the autonomous navigation performance of UAVs in complex environments by combining deep reinforcement learning with imitation learning; firstly, by introducing behavioral cloning at the beginning of training, UAVs can quickly master basic navigation skills under the guidance of expert teaching data, avoiding the inefficiency of random exploration, thereby significantly shortening the training time; at the same time, by introducing a generative adversarial imitation learning module, UAVs can continuously optimize navigation strategies and maintain high robustness and adaptability even in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the UAV autonomous navigation method based on the fusion of imitation learning and reinforcement learning. DETAILED DESCRIPTION

[0039] The embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

[0040] like Figure 1 As shown, the present invention provides a method for autonomous navigation of a UAV based on the fusion of imitation learning and reinforcement learning, comprising the following steps:

[0041] S1, collect the data set of the drone, and normalize the data set to obtain a standard data set;

[0042] S2, build an Actor network using expert operation data from standard datasets;

[0043] S3, input the depth image and state information of the standard data set into the Actor network to generate predicted actions;

[0044] S4: Input the predicted action into the discriminator to complete autonomous navigation.

[0045] In the embodiment of the present invention, in S1, the data set of the drone includes a depth image, state information, and expert operation data of the drone.

[0046] Collect the expert data status information of the drone, including the depth image with an image size of 100X80 pixels, the horizontal distance to the target point, the vertical distance and the relative yaw angle. It is also necessary to collect the corresponding expert actions, including the speed in the horizontal plane, the speed in the vertical plane and the yaw rate. After collecting the expert data, it is necessary to process the expert data and normalize the vertical distance so that its value is in the range of [0,255]. Normalize the relative yaw angle to [0,255].

[0047] Initialize an all-zero state feature array to store the current state information of the drone. At this point, the system extracts key state features, including the horizontal distance, vertical distance, and yaw angle from the drone to the target point. This information is stored in the first few columns of the state feature array, providing important basis for subsequent decision-making.

[0048] Finally, the processed depth image and state feature array are merged and the necessary dimensional transformation is performed to form a comprehensive input containing multi-dimensional information. This comprehensive data not only provides a rich foundation for subsequent data processing and analysis, but also significantly enhances the decision-making and control capabilities of the drone, facilitating its autonomous navigation and operation in complex environments. Through this series of processing, the system ensures that the drone can effectively perceive and respond to challenges in a real-time environment, thereby improving its overall performance.

[0049] In this embodiment of the present invention, S2 includes the following sub-steps:

[0050] S21. Construct a reward function for the expert operation data;

[0051] S22. Based on the reward function, construct the behavior cloning learning objective;

[0052] S23. Construct the loss function of the Actor network based on the behavior cloning learning goal.

[0053] In the embodiment of the present invention, in S21, the expression of the reward function Reward is:

[0054] Reward=reward_distance-λ1*punishment_pose-λ2*

[0055] punishment_obs-λ3*punishment_action-λ4*yaw_error_cost;

[0056] Wherein, reward_distance represents the reward item used to encourage the drone to fly towards the target point, λ1 represents the weight coefficient of penalty_pose, which is used to control the impact of the drone's posture deviation on the total reward, penalty_pose represents the comprehensive penalty item used to evaluate the deviation of the drone's current posture relative to the target posture, λ2 represents the weight coefficient of penalty_obs, which is used to control the penalty weight of the minimum distance between the drone and the obstacle, penalty_obs represents the penalty item used to evaluate the minimum distance between the drone and the obstacle, λ3 represents the weight coefficient of penalty_action, which is used to control the penalty weight of the stability and efficiency of the drone's action, penalty_action represents the comprehensive penalty item for evaluating the stability and efficiency of the drone when performing the action, λ4 represents the weight coefficient of yaw_error_cost, yaw_error_cost represents the penalty item used to measure the error between the current heading of the drone and the target heading.

[0057] Combine multiple factors to promote better navigation performance. First, set rewards and penalties for target arrival, collision, and out-of-bounds. When the drone does not reach the target, the reward comes from the change in distance to the target, the position penalty, the obstacle distance penalty, and the action penalty. The distance reward is adjusted according to the current distance to the target, the position penalty takes into account the difference between the current position of the drone and the target position, and the obstacle distance penalty increases when the drone approaches an obstacle. At the same time, the cost of penalizing actions includes yaw rate, vertical speed, and altitude error. If the drone reaches the target, it will receive a target reward; if a collision occurs or the boundary is exceeded, the corresponding penalties are given. By combining these factors, the method aims to guide the drone to optimize its path and behavior.

[0058] reward_distance is a reward item used to encourage the drone to fly towards the target point. It evaluates the reward by calculating the change in distance between the drone and the target. Specifically, the reward value is based on the change in the drone's current distance to the target, that is, the difference between the previous distance and the current distance, and is scaled by a distance coefficient (distance_reward_coef) so that the reward value corresponds to the ratio of the target distance. The goal is to shorten the distance between the drone and the target as much as possible to obtain a higher reward. This mechanism helps to improve the goal orientation and efficiency of the drone during flight.

[0059] Penalty_pose is a comprehensive penalty term that evaluates the deviation of the drone's current pose (position and altitude) from the target pose. It consists of two parts: penalty_xy and penalty_z. Penalty_xy measures the distance of the drone from the target in a two-dimensional plane (x,y coordinates), normalized to a range of 0 to 1; penalty_z measures the deviation of the drone's altitude in the vertical direction, calculated and scaled by 0.5. The higher the total penalty value, the farther the drone is from the target, thus incentivizing the drone to adjust its position and altitude to fly towards the target.

[0060] punishment_obs is a penalty term used to evaluate the minimum distance between the drone and the obstacle. When the drone's minimum distance is less than 10 meters, the difference between this distance and the set collision distance is calculated and normalized. Specifically, if the drone is close to an obstacle, the penalty value will increase, thereby incentivizing the drone to maintain a safe flight distance and avoid collision. If the distance between the drone and the obstacle is within a safe range, the penalty value is zero. This mechanism is designed to ensure that the drone always maintains an appropriate safety interval during flight, improving the safety and stability of its autonomous navigation.

[0061] punishment_action is a comprehensive penalty value that is used to evaluate the stability and efficiency of the drone when performing actions. It achieves this goal through multiple factors. First, the heading rate penalty calculates the ratio of the absolute value of the heading rate to the maximum allowed rate, encouraging the drone to avoid violent turning maneuvers. Secondly, the vertical speed penalty is based on the ratio of the current vertical speed to the maximum vertical speed, prompting the drone to maintain an appropriate ascent or descent speed. Finally, the altitude error penalty evaluates the ratio of the drone's current relative altitude to the maximum vertical difference, encouraging the drone to get as close to the target altitude as possible. By combining these penalties, punishment_action helps ensure that the drone is more stable and efficient when performing tasks, thereby reducing unnecessary dynamic changes and improving overall performance.

[0062] yaw_error_cost is a penalty term that measures the error between the current heading of the drone and the target heading. It evaluates the degree of penalty by calculating the ratio between the absolute value of the heading error and a reference value (90 degrees). The purpose of this penalty is to encourage the drone to minimize the heading error to ensure that its flight path is closer to the intended target, thereby improving navigation accuracy and stability. Smaller heading errors result in lower penalty values, which have a positive impact on the drone's decision-making.

[0063] In the embodiment of the present invention, in S22, the behavior cloning learning target θ * The expression is:

[0064]

[0065] In the formula, θ represents the dependency and model parameters, E (s,a)～B represents the expected value of the state and action space sampled from the data set, s represents the state, α represents the first weight coefficient, β represents the second weight coefficient, λ represents the third weight coefficient, a represents the action space, B represents the data set, π θ represents the strategy distribution, represents the loss function, which is used to measure the difference between the strategy output action and the target action, R(·) represents the regularization term, and D kL represents the Kullback-Leibler divergence, and P(·) represents the prior distribution of domain knowledge or expert behavior.

[0066] In the embodiment of the present invention, in S23, the loss function The expression is:

[0067]

[0068] In the formula, N represents the number of samples, θ represents the dependency and model parameters, w(·) represents the data weight term, and s i represents the i-th state, a i represents real actions provided by experts, represents the action predicted by the model, π θ represents the strategy distribution, a represents the action space, represents the gradient of the action predicted by the model to the state, α represents the first weight coefficient, λ represents the third weight coefficient, β represents the second weight coefficient, and D KL represents the Kullback-Leibler divergence, and P(·) represents the prior distribution of domain knowledge or expert behavior.

[0069] In this embodiment of the present invention, in S3, the discriminator loss function The expression is:

[0070]

[0071] In the formula, represents the real sample loss, W1(·) represents the weight of the real sample, s represents the state, α represents the first weight coefficient, represents the discriminator function, E ρE represents the loss of generated samples, W2(·) represents the weight of generated samples. W1(·) is used to adjust the importance of each state-action pair (s, a) in the loss. Represents the output value on the state-action pair (s, a), usually a probability, and W2(·) is used to adjust the importance of each state-action pair (s, a) in generating sample loss.

[0072] In the embodiment of the present invention, the experimental environment of the UAV is based on the AirSim simulation platform launched by Microsoft, and the physical experimental scene is constructed using Unreal Engine 4. The experimental results are shown in Table 1.

[0073] Table 1

[0074] Model Reward Success rate Average step length SAC 21.19 0.968 158.31 BC+SAC 26.60 0.986 156.92 BC+GAIL+SAC 26.76 0.988 156.24

[0075] From Table 1, we can conclude that combining behavior cloning (BC) and SAC can significantly improve the reward (from 21.19 to 26.60) and success rate (from 96.8% to 98.6%). Further combining GAIL can slightly improve the reward (from 26.60 to 26.76) and success rate (from 98.6% to 98.8%), while slightly reducing the average number of steps required to complete the task (from 156.92 to 156.24).

[0076] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for autonomous navigation of unmanned aerial vehicles based on the fusion of imitation learning and reinforcement learning, characterized in that: The following steps are involved: S1, collect the data set of the drone, and normalize the data set to obtain a standard data set; S2, build an Actor network using expert operation data from standard datasets; S3, input the depth image and state information of the standard data set into the Actor network to generate predicted actions; S4: Input the predicted action into the discriminator to complete autonomous navigation.

2. The method for autonomous navigation of unmanned aerial vehicles based on the fusion of imitation learning and reinforcement learning according to claim 1 is characterized in that: In S1, the data set of the drone includes the depth image, status information and expert operation data of the drone.

3. The method for autonomous navigation of unmanned aerial vehicle based on the fusion of imitation learning and reinforcement learning according to claim 1 is characterized in that: The S2 comprises the following sub-steps: S21. Construct a reward function for the expert operation data; S22. Based on the reward function, construct the behavior cloning learning objective; S23. Construct the loss function of the Actor network based on the behavior cloning learning goal.

4. The method for autonomous navigation of unmanned aerial vehicle based on the fusion of imitation learning and reinforcement learning according to claim 3 is characterized in that: In S21, the expression of the reward function Reward is: Reward= reward_distance-λ1*punishment_pose-λ2*punishment_obs-λ3* punishment_action-λ4*yaw_error_cost; Wherein, reward_distance represents the reward item used to encourage the UAV to fly toward the target point, λ1 represents the weight coefficient of penalty_pose, which represents the comprehensive penalty item used to evaluate the deviation of the current pose of the UAV relative to the target pose, λ2 represents the weight coefficient of penalty_obs, which represents the penalty item used to evaluate the minimum distance between the UAV and the obstacle, λ3 represents the weight coefficient of penalty_action, which represents the comprehensive penalty item used to evaluate the stability and efficiency of the UAV when performing actions, and λ4 represents the weight coefficient of yaw_error_cost, which represents the penalty item used to measure the error between the current heading of the UAV and the target heading.

5. The method for autonomous navigation of unmanned aerial vehicle based on the fusion of imitation learning and reinforcement learning according to claim 3 is characterized in that: In S22, the behavior cloning learning target θ * The expression is: In the formula, θ represents the dependency and model parameters, E (s,a)～B represents the expected value of the state and action space sampled from the data set, s represents the state, α represents the first weight coefficient, β represents the second weight coefficient, λ represents the third weight coefficient, a represents the action space, B represents the data set, π θ represents the strategy distribution, represents the loss function, R(·) represents the regularization term, and D kL represents the Kullback-Leibler divergence, and P(·) represents the prior distribution of domain knowledge or expert behavior.

6. The method for autonomous navigation of unmanned aerial vehicle based on the fusion of imitation learning and reinforcement learning according to claim 3 is characterized in that: In S23, the loss function The expression is: In the formula, N represents the number of samples, θ represents the dependency and model parameters, w(·) represents the data weight term, and s i represents the i-th state, a i represents real actions provided by experts, represents the action predicted by the model, π θ represents the strategy distribution, a represents the action space, represents the gradient of the action predicted by the model to the state, α represents the first weight coefficient, λ represents the third weight coefficient, β represents the second weight coefficient, and D KL represents the Kullback-Leibler divergence, and P(·) represents the prior distribution of domain knowledge or expert behavior.

7. The method for autonomous navigation of unmanned aerial vehicle based on the fusion of imitation learning and reinforcement learning according to claim 1 is characterized in that: In S3, the discriminator loss function The expression is: In the formula, represents the real sample loss, W1(·) represents the weight of the real sample, s represents the state, α represents the first weight coefficient, represents the discriminator function, E ρE represents the loss of generated samples, and W2(·) represents the weight of generated samples.

Images