An unmanned aerial vehicle autonomous navigation and obstacle avoidance method based on deep reinforcement learning

Abstract

The application provides a kind of unmanned aerial vehicle autonomous navigation and obstacle avoidance method based on deep reinforcement learning, it is related to unmanned aerial vehicle autonomous navigation field, the method takes the observation of unmanned aerial vehicle as input, extracts time sequence characteristics in Actor network, enhances the perception of unmanned aerial vehicle to surrounding environment;The observation vector of the current time of unmanned aerial vehicle is input into the decision offset method module, and the weighted observation feature is output to enhance the obstacle avoidance ability;The two features obtained by processing are spliced and input into the fully connected network, and the action to be taken by the unmanned aerial vehicle at the current time is output;Finally, the output three-axis acceleration of the trained model Actor part is used to control the flight of unmanned aerial vehicle.The application solves the problem that the flexibility of unmanned aerial vehicle in the prior art is low and the obstacle avoidance ability is weak, enhances the speed control ability of unmanned aerial vehicle and the obstacle avoidance ability of unmanned aerial vehicle autonomous navigation model.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation for unmanned aerial vehicles (UAVs), and in particular to a method for autonomous navigation and obstacle avoidance of UAVs based on deep reinforcement learning. Background Technology

[0002] With the rapid development of unmanned technology, research on unmanned aerial vehicles (UAVs) has become increasingly popular. Compared to manned aircraft, UAVs are widely used in both military and civilian fields due to their smaller size, lower cost, zero casualties, and high flexibility. In the military field, UAVs are widely used in target reconnaissance, terrain mapping, precision strikes, communication jamming, communication relay, and other battlefield situational awareness scenarios. In the civilian field, with the popularization of consumer-grade UAVs, their application scenarios are very diverse, such as disaster relief, aerial photography, news reporting, agriculture, plant protection and pest and disease detection, power line inspection, and express delivery. The foundation for UAVs to perform these tasks is their ability to quickly and safely avoid obstacles and reach designated target points in complex environments. The autonomy of this process can greatly reduce costs and improve mission execution efficiency.

[0003] In existing technologies, many methods utilize traditional path planning algorithms (such as A* algorithm, RRT algorithm, and artificial potential field method) and intelligent optimization algorithms (such as particle swarm optimization, ant colony optimization, and genetic algorithm) to achieve autonomous navigation for UAVs. However, these non-learning algorithms require global information and a robust action execution mechanism to plan feasible paths within a given environment. They struggle to handle dynamically changing environments, exhibiting drawbacks such as poor real-time performance, weak online navigation, and limited obstacle avoidance capabilities. Furthermore, given the sequential decision-making characteristics of UAV navigation, some methods model this problem as a Markov Decision Process (MDP) and study it using deep reinforcement learning. Reinforcement learning, through the agent's perception of environmental changes and guided by a reward function, makes decisions without requiring global information, enabling UAVs to autonomously complete navigation and obstacle avoidance tasks while satisfying various constraints. However, these deep reinforcement learning-based autonomous navigation methods for UAVs suffer from low flexibility and weak obstacle avoidance capabilities. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) based on deep reinforcement learning, to address the problems of low flexibility and weak obstacle avoidance capabilities of existing methods. The method includes:

[0005] The observation features are extracted from various detection indicators of the UAV and the historical trajectory data of the UAV is constructed according to a preset number of steps N.

[0006] The historical trajectory data is input into the Actor network of the neural network model to obtain the temporal characteristics of the UAV. The neural network model includes an Actor network, a Critic network, and a fully connected network.

[0007] The observed features are input into the decision offset method module, and the weighted observed features are output.

[0008] The weighted observation features and the temporal features are concatenated and then input into the fully connected network, which outputs the action vector of the UAV.

[0009] The historical trajectory data, current observation features, and action vectors are concatenated and input into the Critic network, which outputs a score for the "observation feature - action vector" pair.

[0010] The Actor network uses the negative of the score as a loss function for gradient descent. After gradient descent, the Actor network outputs a triaxial acceleration, which is used to control the flight of the drone.

[0011] Furthermore, the historical trajectory data includes the coordinate information of the UAV, the three-axis velocity of the UAV, the yaw angle of the UAV, the distance between the UAV and the target point, and the return value of the distance sensor.

[0012] Furthermore, the observation features include the observation features of the previous n time steps and their corresponding action vectors, as well as the current observation feature.

[0013] Furthermore, the neural network model is trained based on the TD3 algorithm.

[0014] Furthermore, the loss function of the Actor network is the score output by the Critic network, and the loss function of the Critic network is MSE.

[0015] Furthermore, both the Actor network and the Critic network are trained using the Adam optimizer for gradient descent.

[0016] Furthermore, the step of extracting observation features from various detection indicators of the UAV and constructing the historical trajectory data of the UAV according to a preset number of steps N includes:

[0017] The organization method of the historical trajectory data is selected based on the value of the number of steps N;

[0018] When the current time step is greater than N, select the observation features and their action vectors that are N steps away from the current time step, and splice them into the historical trajectory data according to the corresponding organization method.

[0019] When the current time step is less than N, select all the observed features and their motion vectors from the start of the drone's flight to the present, and stitch them together to form the historical trajectory data according to the corresponding organization method.

[0020] Furthermore, the step of concatenating the weighted observation features and the temporal features and then inputting them into the fully connected network includes:

[0021] The temporal features are input into the Attention network to calculate the attention weights;

[0022] The attention weights are combined with the temporal features using a Hardmard product to obtain the weighted temporal features.

[0023] The weighted observation features and the weighted temporal features are concatenated and then input into the fully connected network.

[0024] Furthermore, the step of inputting the observed features into the decision offset method module and outputting the weighted observed features includes:

[0025] The current observed features are input into the decision offset method module, and the current observed features are divided into three parts: k, q, and v.

[0026] Perform a Hardmard product between the current observed features of part k and the mask matrix to mask the navigation part and obtain the masked matrix;

[0027] Add the masked matrix to the bias matrix, change the relative size of the navigation part and the obstacle avoidance part, and obtain the guide;

[0028] The currently observed features of the q part are input into the fully connected network, and noise is obtained based on the Tanh activation function;

[0029] The attention weights are obtained by adding guide and noise together and then normalizing the result using the softmax activation function.

[0030] The current observed features of part v are combined with the attention weights using the Hardmard product to obtain the weighted observed features.

[0031] Furthermore, the Actor network outputs the triaxial acceleration based on the Tanh activation function.

[0032] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: This invention provides a method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) based on deep reinforcement learning, which uses a memory mechanism to process the historical trajectory of the UAV, thereby enhancing the UAV navigation model's perception of the surrounding environment; it employs a decision offset algorithm to shift the decision center of the UAV navigation model between navigation and obstacle avoidance tasks according to changes in the surrounding environment, thereby improving the UAV's obstacle avoidance capability; and it adds a speed constraint loss function to the loss function of the Actor part of the UAV autonomous navigation model, causing the UAV's speed to approach the optimal speed, thereby improving speed control capability and thus enhancing navigation and obstacle avoidance capabilities. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of constructing historical trajectories and extracting temporal features from them, provided by an embodiment of the present invention;

[0035] Figure 2 This is a flowchart of processing the observation vector at the current moment using the decision offset algorithm, provided in an embodiment of the present invention;

[0036] Figure 3 This is an overall flowchart of an autonomous navigation and obstacle avoidance method for unmanned aerial vehicles based on deep reinforcement learning, provided by an embodiment of the present invention. Detailed Implementation

[0037] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0038] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] This invention provides a method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) based on deep reinforcement learning. It employs a memory mechanism to process the UAV's historical trajectory, enhancing the UAV navigation model's perception of its surrounding environment. A decision offset algorithm is used to shift the UAV navigation model's decision center of gravity between navigation and obstacle avoidance tasks based on changes in the surrounding environment, improving the UAV's obstacle avoidance capability. A velocity constraint loss function is added to the loss function of the Actor part of the UAV's autonomous navigation model, causing the UAV's speed to approach the optimal speed, improving speed control capability, and thus enhancing navigation and obstacle avoidance capabilities.

[0040] In embodiments of the present invention, such as Figure 3 As shown, a method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) based on deep reinforcement learning includes:

[0041] Step S100: Extract observation features from various detection indicators of the UAV and construct the historical trajectory data of the UAV according to the preset number of steps N;

[0042] Furthermore, in step S100, the historical trajectory data includes the coordinate information of the UAV, the three-axis velocity components of the UAV, the yaw angle of the UAV, the distance between the UAV and the target point, and the return value of the distance sensor; the observation features include the observation features of the previous n time steps and their corresponding motion vectors, and the current observation feature.

[0043] Specifically, the current coordinates of the drone on the map include three coordinate axes: x, y, and z; the yaw angle of the drone is the angle between the drone's flight direction and the line connecting the drone itself and the target point; the return value of the distance sensor is obtained as follows: seven distance sensors are distributed in the first half of the horizontal direction, and sampling is performed every 30 degrees; the other three are distributed in the upper half of the vertical direction, and sampling is performed every 30 degrees; the ten distance sensors are of equal and fixed length.

[0044] Furthermore, such as Figure 1 The diagram illustrates the organization of historical trajectories and the method for extracting time-series features. Step S100 further includes:

[0045] Step S110: Select the organization method of the historical trajectory data according to the value of the number of steps N;

[0046] Step S120: When the current time step is greater than N, select the observation features and their action vectors that are N steps away from the current time step, and splice them into the historical trajectory data according to the corresponding organization method;

[0047] Step S130: When the current time step is less than N, select all the observed features and their motion vectors from the start of the UAV flight to the present, and stitch them together to form the historical trajectory data according to the corresponding organization method.

[0048] Step S200: Input the historical trajectory data into the Actor network of the neural network model to obtain the temporal features of the UAV, wherein the neural network model includes an Actor network, a Critic network, and a fully connected network;

[0049] Furthermore, the neural network model is trained based on the TD3 algorithm. TD3 (Twin Delayed Deep Deterministic Policy Gradient) is a reinforcement learning algorithm used to train an agent to learn which actions to take in different environments to maximize cumulative rewards. The neural network model mainly consists of two parts: the Actor network takes the observation vector as input and outputs the drone's current action, which is the drone's three-axis acceleration, applied to the corresponding velocity components to control the drone's flight; the Critic network takes the observation vector and the action vector generated by it as input and outputs a score for the "observation feature - action vector" pair, which the Actor network uses for learning. In addition, the neural network model also includes a fully connected network: a fully connected network is a basic neural network structure, also known as a multilayer perceptron (MLP). In a fully connected network, each neuron is connected to all neurons in the previous layer, so information can be transferred between different layers of the network. Fully connected networks can be used as the basic structure for Critic and Actor networks to extract features from input data and for information transfer and processing. The model is trained using TD3 as the basic algorithm. In the Actor network, historical trajectory data is input into the LSTM and Attention networks to extract temporal features and enhance the drone's perception of the surrounding environment.

[0050] Furthermore, in the Actor-Critic algorithm, the Critic network and the Actor network typically share some parameters to achieve joint learning and mutual influence. The Critic network provides an evaluation of the current policy, helping the Actor network update its policy to improve it, while the Actor network provides the probability distribution of generated actions, helping the Critic network estimate the value function. This deep reinforcement learning method with an Actor-Critic structure can achieve efficient training and learning of agents in complex environments.

[0051] Specifically, the input to the Actor network consists of two parts. The first part is a combination of the observation vectors and corresponding action vectors from the previous n time steps, which serves as historical data. This data is first input into the LSTM (Long Short-Term Memory) and Attention (attention mechanism) networks for feature extraction. The specific process is as follows: Figure 1 The second part is the observation vector at the current moment. This observation vector is first input into the decision offset module for weighted processing to enhance obstacle avoidance capability. The specific process is as follows: Figure 2 After extracting features from the input features of the two parts mentioned above, the network concatenates the extracted features and then inputs them into a fully connected neural network to output the three-axis acceleration that the drone should take at the current moment.

[0052] The input to the Critic network consists of three parts: historical data composed of the observation vectors and action vectors from the previous n time steps, the observation vector at the current time step, and the action at the current time step output by the Actor network. The historical data uses an LSTM and Attention structure to extract features, which are then concatenated with the other two parts of features and input into a fully connected neural network to output an evaluation value.

[0053] Step S300: Input the observed features into the decision offset method module and output the weighted observed features;

[0054] Specifically, the observation vector of the UAV at the current moment is input into the decision offset method module, and the weighted observation features are output. This allows the UAV's decision center of gravity to shift between navigation and obstacle avoidance tasks according to changes in the surrounding environment, thereby enhancing the UAV's obstacle avoidance capability.

[0055] Furthermore, such as Figure 2 The flowchart shown is for the decision offset method of processing the current observation vector. Step S300 includes:

[0056] Step S310: Input the current observed features into the decision offset method module, and divide the current observed features into three parts: k, q, and v;

[0057] Step S320: Perform a Hardmard product between the currently observed features of part k and the mask matrix to mask the navigation part and obtain the masked matrix;

[0058] Step S330: Add the occluded matrix to the bias matrix, change the relative size of the navigation part and the obstacle avoidance part, and obtain the guide;

[0059] Step S340: Input the currently observed features of the q part into the fully connected network, and obtain noise based on the Tanh activation function;

[0060] Step S350: Add guide and noise, and normalize them using the softmax activation function to obtain the attention weights;

[0061] Step S360: Perform a Hardmard product between the current observation features of part v and the attention weights to obtain the weighted observation features.

[0062] Specifically, the mask matrix has a batch size of rows, with the first eight columns being 0 and the last ten columns being 1; the guide matrix is ​​the guidance matrix; the bias matrix has a batch size of rows, with the first eight columns being 0.25 and the last ten columns being 1; and the noise matrix is ​​the noise matrix.

[0063] Step S400: The weighted observation features and the temporal features are concatenated and then input into the fully connected network, and the fully connected network outputs the action vector of the UAV;

[0064] Furthermore, step S400 also includes:

[0065] Step S410: Input the temporal features into the Attention network and calculate the attention weights;

[0066] Step S420: Perform a Hardmard product between the attention weights and the temporal features to obtain the weighted temporal features;

[0067] Step S430: The weighted observation features and the weighted temporal features are concatenated and then input into the fully connected network.

[0068] Step S500: The historical trajectory data, current observation features, and action vector are concatenated and input into the Critic network, and the Critic network outputs a score;

[0069] Furthermore, the Actor network uses the rating output by the Critic network as its loss function, and the loss function of the Critic network is MSE.

[0070] Step S600: The Actor network uses the negative of the score as the loss function to perform gradient descent. After completing gradient descent, the Actor network outputs a three-axis acceleration, which is used to control the flight of the UAV.

[0071] Furthermore, both the Actor network and the Critic network are trained using the Adam optimizer for gradient descent.

[0072] Furthermore, the Actor network outputs the triaxial acceleration based on the Tanh activation function.

[0073] Specifically, the loss function of the Actor network of the neural network model is the evaluation value output by the Critic network, and the speed constraint loss function is added to the original loss to enhance the learning ability of the UAV to speed control, as shown in formula (1).

[0074]

[0075] In formula (1), v t Let v be the current speed of the drone, v be the optimal speed for the drone to perform navigation and obstacle avoidance tasks in the environment, and a be the speed of the drone. t Let x and y represent the acceleration of the drone at the current moment, and o represent the x-axis, y-axis, and o. t h represents the observed data. t Let θ represent historical trajectory data, θ represent network parameters, and μ represent the Actor network. The Critic loss function is MSE, and both network parts are trained using the Adam optimizer for gradient descent.

[0076] Furthermore, such as Figure 3 As shown, the UAV autonomous navigation and obstacle avoidance method based on deep reinforcement learning provided in this embodiment specifically includes the following implementation steps:

[0077] 1. Set the current iteration count n = 0 and the maximum iteration count;

[0078] 2. The algorithm terminates when the current iteration count is greater than the maximum iteration count; the algorithm continues when the current iteration count is less than the maximum iteration count.

[0079] 3. Based on the selected N, input the historical trajectory of N time steps and the observation characteristics of the UAV at the current moment into the algorithm;

[0080] 4. Input the historical trajectories of N time steps into the memory processing module composed of LSTM and Attention for feature extraction. See the detailed steps below. Figure 1 ;

[0081] 5. Input the current observation vector into the decision offset algorithm module for feature extraction. See details in the documentation. Figure 2 ;

[0082] 6. The features obtained from the two modules are concatenated and then input into the fully connected neural network;

[0083] 7. Pass the output of the fully connected neural network through the Tanh activation function to obtain the triaxial acceleration at the current moment;

[0084] 8. Concatenate the historical trajectory data of the above N time steps, the current observation data of the UAV, and the motion vector output by the Actor part;

[0085] 9. Input the spliced ​​data into the Critic network, which is isomorphic to the Actor network;

[0086] 10. The Critic network outputs the evaluation value at the current time step and uses MSE as the activation function for gradient descent;

[0087] 11. Use the negative of the current evaluation value output by the Critic network as the main loss function of the Actor network, and add the velocity constraint loss function together to perform gradient descent updates.

[0088] 12. Increment the iteration count by 1 and return to step 2.

[0089] In summary, the embodiments of the present invention achieve the following technical effects:

[0090] 1. This invention utilizes a memory mechanism to process the historical trajectory of the UAV, enhancing the UAV navigation model's perception of the surrounding environment; employs a decision offset algorithm to shift the decision center of gravity of the UAV navigation model between navigation and obstacle avoidance tasks based on changes in the surrounding environment, improving the UAV's obstacle avoidance capability; adds a speed constraint loss function to the loss function of the Actor part of the UAV autonomous navigation model, causing the UAV's speed to approach the optimal speed, improving speed control capability, and thus enhancing navigation and obstacle avoidance capabilities; uses an improved TD3 (deep reinforcement learning algorithm) with the above enhancements, trained in a simulated 3D environment, and saves the model with the highest task success rate; uses the Actor part of the saved model for UAV autonomous navigation action generation, inputting UAV observations and directly outputting the UAV's actions;

[0091] 2. This invention uses three-axis acceleration directly output by a neural network to control the movement of the UAV, making its flight more flexible. Based on the deep reinforcement learning model TD3, a decision shift algorithm is used to shift the UAV's decision center of gravity between navigation and obstacle avoidance tasks according to changes in the surrounding environment, thereby enhancing obstacle avoidance capabilities. A velocity constraint loss function is added to the original loss function of the Actor network to enhance the UAV's speed control capabilities, and further enhance the obstacle avoidance capabilities of the UAV's autonomous navigation model.

[0092] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) based on deep reinforcement learning, characterized in that, The method includes: The process involves extracting observation features from various detection indicators of the UAV and constructing historical trajectory data for the UAV according to a preset number of steps N. This includes: selecting an organization method for the historical trajectory data based on the value of the number of steps N; when the current time step is greater than N, selecting the observation features and their motion vectors that are N steps away from the current time step, and splicing them into the historical trajectory data according to the corresponding organization method; when the current time step is less than N, selecting all the observation features and their motion vectors from the start of the UAV's flight to the present, and splicing them into the historical trajectory data according to the corresponding organization method. The historical trajectory data is input into the Actor network of the neural network model to obtain the temporal characteristics of the UAV. The neural network model includes an Actor network, a Critic network, and a fully connected network. The observed features are input into the decision offset method module, and the weighted observed features are output. The weighted observation features and the temporal features are concatenated and then input into the fully connected network, which outputs the action vector of the UAV. This process includes: inputting the temporal features into an Attention network to calculate attention weights; performing a Hardmard product between the attention weights and the temporal features to obtain the weighted temporal features; and concatenating the weighted observation features and the weighted temporal features and then inputting them into the fully connected network. The historical trajectory data, current observation features, and action vector are concatenated and input into the Critic network, which outputs a score for the "observation features - action vector". The Actor network uses the negative of the score as a loss function for gradient descent. After gradient descent, the Actor network outputs a triaxial acceleration, which is used to control the flight of the drone.

2. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that, The historical trajectory data includes the coordinate information of the UAV, the three-axis velocity of the UAV, the yaw angle of the UAV, the distance between the UAV and the target point, and the return value of the distance sensor.

3. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that, The observation features include the current observation features, as well as the observation features of the current observation features at the previous n time steps and their corresponding action vectors.

4. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that, The neural network model is trained based on the TD3 algorithm.

5. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that, Both the Actor network and the Critic network are trained using the Adam optimizer for gradient descent.

6. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that, The step of inputting the observed features into the decision offset method module and outputting the weighted observed features includes: The current observed features are input into the decision offset method module, and the current observed features are divided into three parts: k, q, and v. Perform a Hardmard product between the current observed features of part k and the mask matrix to mask the navigation part and obtain the masked matrix; Add the masked matrix to the bias matrix, change the relative size of the navigation part and the obstacle avoidance part, and obtain the guide matrix; The current observation features of the q part are input into the fully connected network, and the noise matrix is ​​obtained based on the Tanh activation function; The attention weights are obtained by adding the guide matrix and the noise matrix, and then normalizing them using the softmax activation function. The current observed features of part v are combined with the attention weights using the Hardmard product to obtain the weighted observed features.

7. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that, The Actor network, based on the Tanh activation function, outputs the triaxial acceleration.

Images