An autonomous obstacle avoidance navigation method for UAV based on memory-enhanced learning

Abstract

The present invention discloses an autonomous obstacle avoidance navigation method for unmanned aerial vehicles based on memory-enhanced learning, which overcomes the problem that traditional deep reinforcement learning navigation methods are very likely to fall into local dilemmas in partially measurable environments. First, the autonomous obstacle avoidance navigation problem of unmanned aerial vehicles is modeled as a target-driven Markov decision process, and a target feature extraction method for dynamic relative targets is proposed to guide the unmanned aerial vehicle to learn the essential characteristics of the navigation problem; then, considering that the traditional navigation method does not save historical information, causing the unmanned aerial vehicle to easily fall into local dilemmas in partially measurable environments, the present invention designs a memory enhancement module that saves action memory and spatial information, and additionally considers historical observations and action sequences when making decisions, so that the unmanned aerial vehicle can more easily get out of the dilemma; finally, the present invention proposes a deep reinforcement learning algorithm based on Gaussian distribution exploration enhancement, so that it can maintain and improve the success rate of unmanned aerial vehicle obstacle avoidance navigation while improving the convergence speed of the algorithm.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology, and in particular to an autonomous obstacle avoidance navigation technology for unmanned aerial vehicles based on memory enhancement learning. Background Art

[0002] As one of the main research points of drone control systems, the obstacle avoidance navigation algorithm for drones based on visual information has been proposed by scholars at home and abroad for numerous theories and methods, but these methods still have some defects. In the field of traditional algorithms, the method based on perception and avoidance can only be applied to open environments with sparse obstacles. The obstacle avoidance navigation method based on visual SLAM algorithm is generally applied to closed, static, and feature-rich scenes, and it takes a lot of computational overhead to construct the map information of the scene through image information. In addition, these methods require careful adjustment of the algorithm parameters for different environments, which cannot meet the requirements of autonomy and the needs of existing drone applications. A better method should be to perform real-time avoidance based on the currently perceived information, and reinforcement learning is an effective method to achieve real-time avoidance.

[0003] The autonomous obstacle avoidance navigation system of unmanned aerial vehicles using reinforcement learning technology can learn through interaction with the environment, which is sufficient to achieve true autonomy. The application of deep learning technology enables these systems to directly process high-dimensional information such as pictures. In addition, the obstacle avoidance navigation system of unmanned aerial vehicles based on deep reinforcement learning technology does not require expert experience or labeled data, and can learn autonomously from experience, which can be well applied to various application scenarios of unmanned aerial vehicles. However, the application of reinforcement learning algorithms in autonomous obstacle avoidance navigation systems of unmanned aerial vehicles is still in its initial stage. There are problems such as the need to retrain new models for different goals and long training time. In addition, in the classic deep reinforcement learning algorithm, the intelligent agent only makes decisions according to the current observation. Since it is difficult to determine the accurate state of the intelligent agent by relying only on the current observation, the deep reinforcement learning algorithm has problems such as slow convergence speed and poor learning effect. In addition, due to the lack of memory of historical actions and historical observations, it is easy to fall into a local dilemma, which greatly hinders the application of classic deep reinforcement learning methods in partially measurable environments. Summary of the invention

[0004] To solve the above problems, the present invention proposes a method for autonomous obstacle avoidance of unmanned aerial vehicles based on memory-enhanced learning. The method first models the obstacle avoidance navigation problem of unmanned aerial vehicles as a target-driven Markov decision process, and proposes to replace the global target and the single-step position of unmanned aerial vehicles with dynamic relative distances, thereby using the common characteristics between different navigation targets to guide the essential characteristics of the unmanned aerial vehicle learning navigation problem, solving the problem that the classic deep reinforcement learning algorithm needs to retrain new models for different targets. Then, a memory-enhanced deep reinforcement learning algorithm is proposed. By designing a memory structure containing spatial memory and action memory, an attention mechanism is used to integrate and process historical information, and then the action of the unmanned aerial vehicle is determined based on these historical information and current observations, solving the problem that traditional navigation algorithms are prone to fall into local difficulties in partially measurable environments. Finally, a Gaussian distribution-enhanced deep reinforcement learning algorithm is proposed, which achieves a balance between exploration and utilization of reinforcement learning, so that it can fully utilize existing samples and enhance the diversity of samples, solves the problem of slow convergence speed of the existing technology, and can maintain and improve the success rate of obstacle avoidance navigation of unmanned aerial vehicles on the premise of improving the convergence speed of the algorithm.

[0005] To achieve the above object, the present invention provides a method for autonomous obstacle avoidance navigation of a UAV based on deep reinforcement learning, comprising the following steps:

[0006] Step 1: Define the autonomous obstacle avoidance navigation problem of the UAV as a goal-driven Markov model using the following steps: Model the Markov decision process MDP quintuple (s, A, P, R, γ), where s is the input state of the UAV, A is the output action set of the UAV, P is the state transition probability function, R is the reward function, and γ is the discount coefficient;

[0007] Step 1-1: Determine the input state s of the drone: At each time t, the drone observes the environment and obtains the observation o of the environment t , and calculate the corresponding local target g according to the global target t , and then use the deep learning method to extract its features and obtain the specific state feature representation s t ; UAV visual observation information t The observation of the drone consists of two parts: visual sensor information and drone position information, which mainly includes the deflection angle yaw of the drone's flight direction relative to the XOY plane. t And the current position of the drone (x t ,y t ); Dynamic relative target replaces the global absolute target method, using the relative value between the current position of the drone and the navigation target as the temporary target g of each decision node t, including the position of the temporary target and the deflection angle yaw of the temporary target as an auxiliary target feature gt , g t =(x gt ,y gt ,yaw gt ) as fusion information to replace the target information and location information of the drone; a convolutional neural network is used to extract features from the image information to obtain the observed feature vector Use multi-layer perceptron to extract the feature vector of target information And use the splicing method to fuse the image features with the target features to obtain the state features The obtained state features are used for subsequent fusion and processing; the input state of the drone is

[0008] Step 1-2: Determine the output action set A of the drone: The present invention mainly focuses on the problem of obstacle avoidance navigation of the drone at a fixed altitude. There are two reasons for choosing fixed altitude flight. First, because navigation is performed in an open environment, the drone is prone to learn some cheating behaviors (such as flying to a sufficient height first, crossing the obstacle and then landing) in the case of changing altitude. Secondly, because the present invention only uses the information of the front visual sensor of the drone as the main input for obstacle avoidance, this information cannot indicate whether a collision will occur in the altitude direction. In order to avoid the coupling of the reinforcement learning algorithm with the dynamic control instructions, some relatively high-level control instructions are first implemented according to some speed control and direction control instructions of the drone, mainly including moving forward at a fixed speed for a specific time, turning left at a certain angle, and turning right at a certain angle. Furthermore, we discretize the control action of the drone, which includes three actions of turning left, turning right, and going straight after discretization. The specific details of the action are as follows:

[0009] Action (1) Turn left: Turn left at a speed of 30° / s for 1 second;

[0010] Action (2) Turn right: Turn right at a speed of 30° / s for 0.7 seconds;

[0011] Action (3) Straight: Move forward in the direction of the camera at a speed of 2 m / s for 1 second. The angles of the left and right turns are different. The main motivation for this design is to create two actions that will not cancel each other out, preventing the drone from being trapped in certain environments.

[0012] Step 1-3: Define the probability that the quadrotor drone can reach the next input state s' when the current input state s is taken and the action amount is a as the state transition probability function P;

[0013] Step 1-4: The present invention designs a non-sparse reward function for the obstacle avoidance navigation problem of unmanned aerial vehicles. The reward function mainly consists of four parts, namely, transfer reward, collision penalty, target reward and step penalty: the transfer reward is used to represent the distance change of the unmanned aerial vehicle relative to the navigation target, so that the unmanned aerial vehicle can get corresponding feedback signals for each step of movement instead of getting effective rewards at the end of the plot; using (x g ,y g ) represents the navigation goal of the drone in a complete episode, and the transfer reward r trans for:

[0014]

[0015] The collision penalty is defined as r collision =c t , where c t is an indicator variable indicating whether the drone has collided. If a collision occurs, c t is 1, otherwise c t is 0; similarly, the target reward is used to indicate whether the drone successfully reaches the target, which is defined as r tgt =d t , d t It is also an indicator variable used to indicate whether the drone has reached the target. If the drone reaches the target, then d t =1, otherwise d t is 0; the step penalty is used to break the dilemma when there are obstacles in the navigation target direction of the drone; the main reason for using the step penalty is that if there is no step penalty, and the obstacle and the navigation target direction are both in front of the drone, since the turning action almost does not produce negative rewards, and moving in other directions will result in less rewards, it may cause the drone to be trapped in a turning cycle, so the step penalty is also a necessary part; the step penalty is simply defined as r step =-0.01; the final reward function is obtained by weighting each part, and the final reward function R(s t ,a t ,s t+1 )for:

[0016] R(s t ,a t ,s t+1 )=0.01*r trans -3*r collision +2*r tgt +r step

[0017] Step 1-5: Define the discount coefficient 0<γ<1, which is used to calculate the cumulative value of returns in the entire process. The larger the discount coefficient, the more emphasis is placed on long-term returns;

[0018] Step 2: Based on the Markov decision process quintuple (s, A, P, R, γ) modeled in step 1, the obstacle avoidance navigation strategy of the UAV is solved based on the DQN algorithm and the priority experience replay technology based on the temporal difference method is introduced to improve the utilization rate of rare samples;

[0019] Step 2-1: Use the Q network instead of the Q table to approximate the state-action value function. Instead of directly using the state transition sequence obtained by interacting with the environment for training the deep neural network, the experience replay mechanism is used to train the Q network. The DQN algorithm saves the state transition sequence obtained by interacting with the environment, including the current state, action, reward, and next state, into the experience replay pool D. Then, when training the strategy, a certain amount of data samples are randomly sampled and the batch gradient descent algorithm is executed with these extracted data.

[0020] Step 2-2: The experience replay unit in the DQN algorithm is a fixed-size list that stores state transition information during the interaction between the agent and the environment. It mainly includes<s,a,r,s',d> , where s represents the current state, a represents the action performed by the agent in state s, r represents the immediate reward obtained after performing action a, s' represents the state of the environment after performing the action, and d is an indicator variable used to indicate whether state s' is a terminal state. If it is a terminal state, d = 1, otherwise d = 0;

[0021] Step 2-3: There are two neural networks with the same structure in the fixed target DQN algorithm, namely the current value network and the target value network. The current value network is used to interact with the environment and calculate the Q value Q(s,a|θ) of the current state. The target value network is used to calculate the Q value Q(s′,a′|θ′) of the next state and synchronizes parameters with the current value network at regular intervals.

[0022] Step 2-4: For a state transition sequence<s,a,r,s',d> , according to the Bellman equation, the current state action value Q(s,a|θ) is estimated, and its estimated value is used as Q target (s,a) means, Q target The recursive relationship between (s,a) and the target state action value Q(s′,a′|θ′) is:

[0023]

[0024] Step 2-5: Q targetThe difference between (s,a) and Q(s,a) is called the temporal difference method. The parameter θ of the Q network is updated using the method of minimizing the mean square temporal difference method. B is used to represent a set of state transition sequence samples sampled from the experience replay pool. The corresponding loss function is:

[0025]

[0026] Step 2-6: Introduce the priority experience replay technology based on the temporal difference method to improve the utilization rate of rare samples: The temporal difference method is represented by δ, and the priority experience sampling algorithm uses the temporal difference method to calculate the priority of each state transition sequence; the temporal difference method in the DQN algorithm is:

[0027]

[0028] Step 2-7: After calculating the temporal difference method of the state transition sequence, its priority can be expressed as p = |δ| + ∈, where ∈ is a very small positive number, mainly used to prevent the situation where the temporal difference method is 0; after obtaining the priority, the priority is normalized to calculate the probability of being sampled as P(i), α is a hyperparameter, mainly used to determine the impact of the error on the priority. If α is 0, it degenerates into uniform sampling;

[0029]

[0030] Step 2-8: In order to solve the problem that priority sampling causes the data distribution in the experience replay pool to change and brings additional deviation to the loss function, the loss function is corrected by the importance sampling technique. A correction coefficient is multiplied by the loss accumulation before each state transition sequence. In the priority experience replay, the correction coefficient is ω i , N represents the number of empirical samples, β is a hyperparameter that determines the importance factor. If β is 0, it means that the importance sampling technique is not used to correct the sample bias;

[0031]

[0032] Step 3: Design a memory module for deep reinforcement learning scenarios to store action memory and spatial information, additionally consider historical observations and action sequences, and propose an additional penalty method to give the drone additional reward signals. Combined with historical memory, the agent is given the ability to escape from some difficulties.

[0033] Step 3-1: Constructing action memory: Action memory saves and extracts the features of historical action sequences, integrating the historical trajectory information of the drone into the state information; using simple linear transformation to construct action memory, assuming A t =(a t-1 ,at-2 ,…,a t-k ) represents the vector of actions made by K decisions before time t, using Represents the content of the action memory at time t, and uses a two-layer fully connected neural network to extract its features;

[0034]

[0035] Step 3-2: Construct spatial memory: Spatial memory is constructed by the historical visual observations and historical actions of the drone. The spatial layout corresponding to the observation information is restored based on the drone's observation information and the motion information after obtaining the observation information; historical visual information represents the drone's observations at a certain location, and historical actions represent the drone's movement mode before these observations are obtained; Research on deep reinforcement learning algorithm enhanced by historical memory The spatial memory of size K at time t is obtained by the observation information of the most recent K steps (o t-1 ,o t-2 ,…,o t-k ) and action sequence (a t-1 ,a t-2 ,…,a t-k ) is generated, where o i represents the observation information at time i, a i represents the action performed at time i; the spatial memory of size K at time t is defined as in Represents the content of the i-th spatial memory unit; when calculating the content of each memory unit, use Represents the feature vector extracted by the convolutional neural network from the visual observation at time ti, using The feature vector representing the execution action at time ti is calculated by combining a multi-layer perceptron with action embedding. Action embedding uses a vector to encode the action number. First, the action number is encoded using one-hot encoding to obtain the action encoding vector. Then, the encoding vector is linearly transformed using the embedding matrix E to obtain the intermediate features corresponding to the action. Finally, a two-layer fully connected neural network is used to extract features to obtain the feature vector f corresponding to the action. a , the specific calculation process is:

[0036]

[0037] After obtaining the feature vector of the action sequence and the feature vector of the observation sequence, the content of the i-th memory unit at time t can be obtained by concatenating the action feature and the observation feature, that is, Spatial memory at time t can be rewritten as

[0038] Step 3-3: Integration and extraction of spatial memory: After the spatial memory is constructed, a memory extraction and integration mechanism based on the attention mechanism is designed to extract and integrate spatial memory; assuming that the weight vector obtained by the attention mechanism is (α t-1 ,α t-2 ,…,α t-k ), then the characteristics of spatial memory are:

[0039]

[0040] The weight vector of spatial memory is calculated through the attention mechanism of the multi-layer perceptron; for each i that satisfies tK≤ti≤t-1, the weight ratio e of the i-th memory unit can be obtained i =W T tanh(W ∝ q i +U ∝ m i +b ∝ ), where W, W ∝ ,U ∝ ,b ∝ is a trainable weight, q t It is the feature vector of the current observation and the feature after the target feature is fused. After obtaining the weight ratio of each memory unit, in order to standardize the weight, the softmax function is used to normalize it; the weight vector of spatial memory is:

[0041]

[0042] Step 3-4: Memory-enhanced deep reinforcement learning algorithm: historical memory information is combined with current observations and target features in a spliced ​​manner to serve as the current environment state.

[0043]

[0044] Get the environmental state characteristics at time t Afterwards, the state information replaces the original observation information and target information, calculates the Q value and strategy, and optimizes the relevant parameters according to the optimization goal of the corresponding algorithm. The memory module is integrated into the DQN algorithm, which is called the MEDQN algorithm.

[0045] Step 3-5: Improvement of reward function based on historical action sequence: By incorporating memory module into deep reinforcement learning algorithm, the agent can learn the actions it has performed in the past; the drone will encounter local loop problem caused by the contradiction between obstacle avoidance strategy and navigation strategy. This paper proposes an additional penalty method based on historical action sequence; named action penalty r act , which is calculated as:

[0046]

[0047] In r act In the calculation formula, l lr represents the length of the action sequence in which the agent only performs left and right turns; in order to integrate the action penalty, the additional action penalty r act Add it to the reward function, and the new reward function is:

[0048] R(s t ,a t ,s t+1 )=0.01*r trans -3*r collision +2*r tgt +r step +r act

[0049] Step 4: A Gaussian distribution-based reinforcement learning exploration enhancement method is proposed and applied to the DQN algorithm, resulting in the Gaussian distribution exploration-enhanced GEDQN algorithm, thereby verifying the effectiveness of the Gaussian distribution exploration enhancement method in value-based reinforcement learning algorithms and actor-critic-based reinforcement learning algorithms;

[0050] Step 4-1: Using the value distribution information learned by the deep reinforcement learning algorithm, design a reinforcement learning exploration method based on state-action value distribution sampling: When selecting an action, the reinforcement learning algorithm based on value distribution first obtains the value corresponding to a certain action under state s according to the value distribution sampling. Then, the corresponding action is selected according to the sampling results. In this method, although the mean values ​​of different action value distributions are determined, due to the intersection between distributions, the action with the largest mean value may not be selected when selecting an action, and it still has randomness; and with the convergence of the reinforcement learning algorithm, the value distribution estimate becomes more accurate, and the variance of the distribution becomes smaller, so the exploratory nature of the algorithm will gradually decrease with convergence, and it can automatically control the balance of exploration and utilization without additional operations; after obtaining the expectation and variance of the state value distribution, the state-action value distribution is used for sampling and exploration using strategies; in order to use the state-action value distribution information, each time the state-action value function is calculated, the mean Q of the action value function of the current state is first calculated π(s,a) and variance D π (s, a), and then sample according to the distribution to obtain the final state action value function And select the action with the highest value of the state-action value function as the output of the strategy, thereby increasing the randomness of the strategy; the final strategy formula is:

[0051]

[0052] Step 4-2: Design a DQN algorithm based on Gaussian distribution exploration enhancement: The structure of the Gaussian distribution DQN algorithm (GEDQN) is similar to the classic DQN algorithm, but the GEDQN algorithm predicts the mean and variance of the value function at the same time, and uses the EM distance as the error function of the value function; Like the DQN algorithm, this algorithm also contains two Q networks, which are used to calculate the current value distribution and the target value distribution respectively; when interacting with the environment, the current value network calculates the strategy and saves the generated state transition information <s,a,r,s′> to the memory playback unit D; during training, the current value network is used to calculate the value distribution information corresponding to the current state-action pair <s,a>, including the mean Q(s,a|θ) and variance D(s,a| θ); the target value network calculates the value distribution information corresponding to all actions according to the next state information, including {Q(s′,a′|θ′)|a′∈A} and {D(s′,a′|θ′)|a′∈A}, and periodically synchronizes the parameters of the current value network to update its network parameters; the loss function of GEDQN is different from that of the traditional DQN algorithm: in the loss function of GEDQN, firstly, according to {Q(s′,a′|θ′)|a′∈A}, the action a* that maximizes the average value of the action is selected, and then Q(s′,a*|θ′) and D(s′,a*|θ′) are calculated as the state-action value distribution of the next state, and then the current value network is optimized in a way that maximizes the EM distance. The Gaussian DQN loss function is:

[0053] When the current value network interacts with the environment, in order to better balance the exploration and utilization of the strategy, the ∈-greedy algorithm in the traditional DQN algorithm is replaced by an automatic exploration method based on value distribution; specifically, when making a decision in state s, the Q(s, a|θ) and D(s, a|θ) corresponding to each action are first calculated based on the current value network, and then the value distribution corresponding to each action is sampled to obtain the value corresponding to each action. Then select the action with the largest value obtained by sampling as the decision of the strategy, that is, The GE-DQN algorithm combined with the memory enhancement method is called the GEMEDQN algorithm. In the GEMEDQN algorithm, a deep neural network is used to simultaneously predict Q(s, a) and D(s, a), and the two share the parameters of the shallow network, and a fully connected network is added at the last layer to predict the mean and variance.

[0054] Beneficial effects of the present invention:

[0055] (1) The present invention proposes an obstacle avoidance navigation method for UAVs based on a target-driven reinforcement learning algorithm. The method replaces the global target and the single-step UAV position with a dynamic relative distance, and utilizes the common features between different navigation targets to guide the UAV to learn the essential features of the navigation problem. This solves the problem that the classic deep reinforcement learning algorithm needs to retrain a new model for different targets.

[0056] (2) The present invention proposes a memory-enhanced deep reinforcement learning method, designs a memory structure that includes spatial memory and action memory, and then uses the attention mechanism to extract the temporal dependency between historical information. The action of the drone is determined based on this historical information and current observations, thereby enhancing the accuracy of the estimation of the drone and environmental state in partially measurable scenarios, balancing the contradiction between the obstacle avoidance strategy and the navigation strategy of the reinforcement learning algorithm, and combining additional action penalties to guide the drone out of the target obstacle dilemma, solving the problem that traditional navigation solutions cannot escape the dilemma in partially measurable scenarios.

[0057] (3) The present invention proposes a deep reinforcement learning algorithm enhanced by Gaussian distribution, and gives the application of the Gaussian distribution enhancement method in the value-based reinforcement learning algorithm and the actor-critic-based deep reinforcement learning algorithm. The Gaussian distribution enhancement method can enhance the exploration ability of the deep reinforcement learning strategy under the premise of making full use of existing samples, increase the diversity of samples, and better balance the contradiction between exploration and utilization of the deep reinforcement learning algorithm, so that it can spend less training time to achieve a higher navigation success rate, solving the problem that the classic deep reinforcement learning algorithm requires long-term training. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of the obstacle avoidance and navigation method for unmanned aerial vehicles based on the deep reinforcement learning algorithm proposed in the present invention.

[0059] Figure 2 This is a schematic diagram of solving the obstacle avoidance navigation strategy for UAV based on the DQN algorithm proposed in the present invention.

[0060] Figure 3 Schematic diagram of the memory-enhanced deep reinforcement learning algorithm proposed in the present invention.

[0061] Figure 4Schematic diagram of the Gaussian distribution exploration enhanced reinforcement learning method proposed in the present invention.

[0062] Figure 5 Schematic diagram of the UAV obstacle avoidance navigation framework based on memory enhanced learning proposed in the present invention.

[0063] Figure 6 The average cumulative reward experimental results of the goal-driven deep reinforcement learning obstacle avoidance navigation algorithm in different environments. (Figure (a) is the average cumulative reward of the Blocks environment, and Figure (b) is the average cumulative reward of the NeighborHood environment)

[0064] Figure 7 The experimental results of the success rate of the goal-driven deep reinforcement learning obstacle avoidance navigation algorithm in different environments. (Figure (a) shows the success rate in the Blocks environment, and Figure (b) shows the success rate in the NeighborHood environment)

[0065] Figure 8 The experimental results of the memory-enhanced DQN algorithm in different environments show the average cumulative reward. (Figure (a) shows the average cumulative reward in the Blocks environment, and Figure (b) shows the average cumulative reward in the NeighborHood environment)

[0066] Fig. 9 The experimental results of the success rate of the memory-enhanced DQN algorithm in different environments. (Figure (a) shows the success rate in the Blocks environment, and Figure (b) shows the success rate in the NeighborHood environment)

[0067] Fig.10 The following is a graph showing the performance of the Gaussian distribution exploration enhancement method in the DQN algorithm. (Figure (a) shows the average cumulative reward in the Blocks environment, and Figure (b) shows the average cumulative reward in the NeighborHood environment) DETAILED DESCRIPTION

[0068] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0069] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0070] Step 1: If Figure 1The figure shows a schematic diagram of the drone obstacle avoidance navigation method using the deep reinforcement learning algorithm proposed in the present invention. After the drone obstacle avoidance navigation problem is modeled as a Markov decision process, the model can be solved using the deep reinforcement learning algorithm. The interaction process between the drone and the environment mainly includes the following steps:

[0071] Step 1-1: At each time t, the drone observes the environment and obtains the observation o of the environment t , and calculate the corresponding local target g according to the global target t , and then use the deep learning method to extract its features and obtain the specific state feature representation s t ;

[0072] Step 1-2: Place s t Input into the decision network and get the corresponding action a t ;

[0073] Step 1-3: After the drone performs the action, the system enters the next state s t+1 , and receive certain rewards t+1 ; Reinforcement learning algorithm obtains t ,a t ,s t+1 ,r t > Update its own parameters according to a certain optimization algorithm to maximize the expected cumulative discounted return;

[0074] Step 1-4: If s t+1 If it is a termination state, the environment is reset; in the present invention, if the drone reaches the target, a collision occurs, or the number of decisions exceeds the upper limit, the drone enters the termination state;

[0075] Step 1-5: Repeat steps 1-1 to 1-4 until the strategy converges;

[0076] Step 2: If Figure 2 As shown, the present invention provides a method for solving the obstacle avoidance navigation strategy of a UAV based on the DQN algorithm, and the specific steps are as follows:

[0077] Step 2-1: Use the Q network instead of the Q table to approximate the state-action value function. Instead of directly using the state transition sequence obtained by interacting with the environment for deep neural network training, the experience replay mechanism is used to train the Q network. The DQN algorithm saves the state transition sequence obtained by interacting with the environment, including the current state, action, reward, and next state, into the experience replay pool. Then, when training the strategy, a certain amount of data samples are randomly sampled and the batch gradient descent algorithm is performed on these sampled data.

[0078] ​Step 2-2: The experience replay unit in the DQN algorithm is a fixed-size list that stores state transition information during the interaction between the agent and the environment. It mainly includes<s,a,r,s',d> , where s represents the current state, a represents the action performed by the agent in state s, r represents the immediate reward obtained after performing action a, s' represents the state of the environment after performing the action, and d is an indicator variable used to indicate whether state s' is a terminal state. If it is a terminal state, d = 1, otherwise d = 0;

[0079] Step 2-3: There are two neural networks with the same structure in the fixed target DQN algorithm, namely the current value network and the target value network. The current value network is used to interact with the environment and calculate the Q value Q(s,a|θ) of the current state. The target value network is used to calculate the Q value Q(s′,a′|θ′) of the next state and synchronizes parameters with the current value network at regular intervals.<s,a,r,s',d> , according to the Bellman equation, the current state action value Q(s,a|θ) is estimated, and its estimated value is used as Q target (s,a) means, Q target The recursive relationship between (s,a) and the target state action value Q(s′,a′|θ′) is:

[0080] Step 2-4: Q target The difference between (s,a) and Q(s,a) is called the temporal difference method. The parameter θ of the Q network is updated using the method of minimizing the mean square temporal difference method. B is used to represent a set of state transition sequence samples sampled from the experience replay pool. The corresponding loss function is: The priority experience replay technology based on the temporal difference method is introduced to improve the utilization rate of rare samples; the temporal difference method is represented by δ, and the priority experience sampling algorithm uses the temporal difference method to calculate the priority of each state transition sequence; the temporal difference method in the DQN algorithm is: After calculating the temporal difference method of the state transition sequence, its priority can be expressed as p = |δ| + ∈, where ∈ is a very small positive number, which is mainly used to prevent the situation where the temporal difference method is 0; after obtaining the priority, the priority can be normalized to calculate the probability of being sampled as P(i), α is a hyperparameter, which is mainly used to determine the impact of the error on the priority. If α is 0, it degenerates into uniform sampling.

[0081] Step 2-5: In order to solve the problem that priority sampling causes the data distribution in the experience replay pool to change and brings additional deviation to the loss function, the loss function is corrected by the importance sampling technique. A correction coefficient is multiplied by the loss accumulation before each state transition sequence. In the priority experience replay, the correction coefficient is ω i , N represents the number of empirical samples, β is a hyperparameter that determines the importance factor. If β is 0, it means that the importance sampling technique is not used to correct the sample bias.

[0082] Step 3: If Figure 3 As shown, the present invention provides a memory-enhanced deep reinforcement learning algorithm, comprising the following steps:

[0083] Step 3-1: Constructing action memory: Action memory saves and extracts the features of historical action sequences, integrating the historical trajectory information of the drone into the state information; using simple linear transformation to construct action memory, assuming A t =(a t-1 ,a t-2 ,…,a t-k ) represents the vector of actions made by K decisions before time t, using represents the content of the action memory at time t, and uses a two-layer fully connected neural network to extract its features. for

[0084] Step 3-2: Construct spatial memory: The spatial memory is constructed by the historical visual observations and historical actions of the drone. The deep reinforcement learning algorithm enhanced by historical memory studies the spatial memory of size K at time t through the observation information of the most recent K steps (o t-1 ,o t-2 ,…,o t-k ) and action sequence (a t-1 ,a t-2 ,…,a t-k ) is generated, where o i represents the observation information at time i, a i represents the action performed at time i; the spatial memory of size K at time t is defined as in Represents the content of the i-th spatial memory unit; when calculating the content of each memory unit, use Represents the feature vector extracted by the convolutional neural network from the visual observation at time ti, using The feature vector representing the execution action at time ti is calculated by combining the multi-layer perceptron with action embedding. After obtaining the feature vector of the action sequence and the feature vector of the observation sequence, the content of the i-th memory unit at time t can be obtained by concatenating the action feature and the observation feature, that is, Spatial memory at time t can be rewritten as

[0085] Step 3-3: Integration and extraction of spatial memory: After the spatial memory is constructed, a memory extraction and integration mechanism based on the attention mechanism is designed to extract and integrate spatial memory; assuming that the weight vector obtained by the attention mechanism is (α t-1 ,α t-2 ,…,α t-k ), then the characteristics of spatial memory are: The weight vector of spatial memory is calculated through the attention mechanism of the multi-layer perceptron; for each i that satisfies tK≤ti≤t-1, the weight ratio e of the i-th memory unit can be obtained i =W T tanh(W ∝ q i +U ∝ m i +b ∝ ), where W,U ∝ ,b ∝ is the trainable weight, q t It is the feature vector of the current observation and the feature after the target feature is fused. After obtaining the weight ratio of each memory unit, in order to standardize the weight, the softmax function is used to normalize it; the weight vector of spatial memory is:

[0086] Step 3-4: Memory-enhanced deep reinforcement learning algorithm: historical memory information is combined with current observations and target features in a spliced ​​manner to serve as the current environment state. The state information is used to replace the original observation information and target information, and the Q value and strategy are calculated. The relevant parameters are optimized according to the optimization target of the corresponding algorithm, and the memory module is integrated into the DQN algorithm, which is called the MEDQN algorithm.

[0087] Step 3-5: Improvement of reward function based on historical action sequence: By incorporating memory module into deep reinforcement learning algorithm, the agent can learn the actions it has performed in the past; the drone will encounter local loop problem caused by the contradiction between obstacle avoidance strategy and navigation strategy. This paper proposes an additional penalty method based on historical action sequence; named action penalty r act , which is calculated as: In r act In the calculation formula, l lr represents the length of the action sequence in which the agent only performs left and right turns; in order to integrate the action penalty, the additional action penalty r act Add to the reward function, and the new reward function is: R(s t ,a t ,s t+1 )=0.01*r trans -3*r collision +2*r tgt +r step +r act ;

[0088] Step 4: Figure 4 FIG. 1 is a schematic diagram of a Gaussian distribution exploration enhanced reinforcement learning method proposed in the present invention, which includes the following steps:

[0089] Step 4-1: The GEDQN algorithm predicts the mean and variance of the value function at the same time, and uses the EM distance as the error function of the value function; Like the DQN algorithm, this algorithm also contains two Q networks, which are used to calculate the current value distribution and the target value distribution respectively; When interacting with the environment, the current value network calculates the strategy and saves the generated state transition information <s,a,r,s′> to the memory playback unit D; During training, the current value network is used to calculate the value distribution information corresponding to the current state-action pair <s,a>, including the mean Q(s,a|θ) and variance D(s,a|θ) of the distribution; The target value network calculates the value distribution information corresponding to all actions based on the next state information, including {Q(s′,a′|θ′)|a′∈A} and {D(s′,a′|θ′)|a′∈A}, and periodically synchronizes the parameters of the current value network to update its network parameters;

[0090] Step 4-2: In the loss function of GEDQN, first select the action a* that maximizes the average value of the action according to {Q(s′,a′|θ′)|a′∈A}, then calculate Q(s′,a*|θ′) and D(s′,a*|θ′) as the state-action value distribution of the next state, and then optimize the current value network by maximizing the EM distance.

[0091] The Gaussian DQN loss function is:

[0092] Step 4-3: When the current value network interacts with the environment, in order to better balance the exploration and utilization of the strategy, the ∈-greedy algorithm in the traditional DQN algorithm is replaced by an automatic exploration method based on value distribution; specifically, when making a decision in state s, first calculate the Q(s, a|θ) and D(s, a|θ) corresponding to each action based on the current value network, and then sample the value distribution corresponding to each action to obtain the value corresponding to each action. Then select the action with the largest value obtained by sampling as the decision of the strategy, that is, In the GEMEDQN algorithm, a deep neural network is used to simultaneously predict Q(s,a) and D(s,a), and the two share the parameters of the shallow network, and a fully connected network is added in the last layer to predict the mean and variance respectively;

[0093] Example:

[0094] This example uses the high-fidelity 3D environment provided by AirSim to test and evaluate our algorithm; AirSim is a plug-in based on the Unreal Engine that can be used to simulate the flight scenes and physics engines of quadcopters in the real world;

[0095] In the target selection process of the UAV navigation task, the current position of the UAV is used as the starting point of navigation; in order to maintain the consistency of the navigation distance, the end point is randomly and uniformly selected on a circle with a radius of 100 meters centered on the starting point; in the obstacle avoidance navigation of the UAV based on reinforcement learning, the UAV repeats the process of observing the environment, making decisions, obtaining rewards and saving the state transformation sequence to the experience replay pool until it reaches the vicinity of the navigation target or a collision occurs; due to the existence of obstacles in the environment, it is assumed that the UAV has reached the vicinity of the target if the distance from the target is within 6 meters; in addition, an upper limit on the number of decision steps is set. If the UAV still has not reached the target after 200 steps of decision-making, the navigation task is also considered to have failed; unless otherwise specified, the reinforcement learning algorithm uses the same hyperparameters; for model optimization, the RMSprop optimizer is used to optimize the parameters of the deep neural network, with a learning rate of 0.001 and momentum parameters all taking the default value of 0.99.

[0096] Figure 6 and Figure 7 They are the experimental results of the average cumulative reward and success rate of the goal-driven deep reinforcement learning obstacle avoidance navigation algorithm in different environments.

[0097] Figure 8 and Fig. 9 They are the experimental results of average cumulative rewards and success rates of the memory-enhanced DQN algorithm in different environments.

[0098] Fig.10This is the performance result of the Gaussian distribution exploration enhancement method in the DQN algorithm.

[0099] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above technical features.

Claims

1. A method for autonomous obstacle avoidance navigation of unmanned aerial vehicles based on memory enhancement learning, characterized in that: The steps include: Step 1: Define the autonomous obstacle avoidance navigation problem of the UAV as a goal-driven Markov model using the following steps: Model the Markov decision process MDP quintuple s, A, P, R, γ, where s is the input state of the UAV, A is the output action set of the UAV, P is the state transition probability function, R is the reward function, and γ is the discount coefficient; Step 2: Based on the Markov decision process quintuple s, A, P, R, γ modeled in step 1, the obstacle avoidance navigation strategy of the UAV is solved based on the DQN algorithm and the priority experience replay technology based on the temporal difference method is introduced to improve the utilization rate of rare samples; Step 3: Design a memory module for deep reinforcement learning scenarios to store action memory and spatial information, additionally consider historical observations and action sequences, and propose an additional penalty method to give the drone additional reward signals. Combined with historical memory, the agent is given the ability to escape from some difficulties. Step 4: A Gaussian distribution-based reinforcement learning exploration enhancement method is proposed and applied to the DQN algorithm, and the Gaussian distribution exploration enhancement GEDQN algorithm is obtained, thereby verifying the effectiveness of the Gaussian distribution exploration enhancement method in the value-based reinforcement learning algorithm and the actor-critic based reinforcement learning algorithm; Step 4 specifically includes the following: Step 4-1: Using the value distribution information learned by the deep reinforcement learning algorithm, design a reinforcement learning exploration method based on state-action value distribution sampling: When selecting an action, the reinforcement learning algorithm based on value distribution first obtains the value corresponding to a certain action under state s according to the value distribution sampling. Then, the corresponding action is selected according to the sampling results. In this method, although the mean values ​​of different action value distributions are determined, due to the intersection between distributions, the action with the largest mean value may not be selected when selecting an action, and it still has randomness; and with the convergence of the reinforcement learning algorithm, the value distribution estimate becomes more accurate, and the variance of the distribution becomes smaller, so the exploratory nature of the algorithm will gradually decrease with convergence, and it can automatically control the balance of exploration and utilization without additional operations; after obtaining the expectation and variance of the state value distribution, the state-action value distribution is used for sampling and exploration using strategies; in order to use the state-action value distribution information, each time the state-action value function is calculated, the mean Q of the action value function of the current state is first calculated π (s,a) and variance D π (s, a), and then sample according to the distribution to obtain the final state action value function And select the action with the highest value of the state-action value function as the output of the strategy, thereby increasing the randomness of the strategy; the final strategy formula is: Step 4-2: Design a Gaussian distribution-enhanced DQN algorithm: The structure of the Gaussian distribution-enhanced DQN algorithm (GEDQN) is similar to the classic DQN algorithm, but the GEDQN algorithm predicts the mean and variance of the value function at the same time, and uses the EM distance as the error function of the value function; Like the DQN algorithm, this algorithm also contains two Q networks, which are used to calculate the current value distribution and the target value distribution respectively; when interacting with the environment, the current value network calculates the strategy and generates the state transition information <s,a,r,s ′ >Save to the memory playback unit D; during training, the current value network is used to calculate the current state action pair<s,a> The corresponding value distribution information includes the mean Q(s,a|θ) and variance D(s,a|θ) of the distribution; the target value network calculates the value distribution information corresponding to all actions based on the next state information, including {Q(s ′ ,a ′ |θ ′ )|a ′ ∈A} and {D(s ′ ,a ′ |θ ′ )|a ′ ∈A}, and periodically synchronize the parameters of the current value network to update its network parameters; the loss function of GEDQN is different from the traditional DQN algorithm: in the loss function of GE-DQN, first according to {Q(s ′ ,a ′ |θ ′ )|a ′ ∈A}, select the action a* that maximizes the average value of the action, and then calculate Q(s ′ ,a * |θ ′ ) and D(s ′ ,a * |θ ′ ) as the state-action value distribution of the next state, and then optimize the current value network in a way that maximizes the EM distance. The Gaussian DQN loss function is: When the current value network interacts with the environment, in order to better balance the exploration and utilization of the strategy, the ∈-greedy algorithm in the traditional DQN algorithm is replaced by an automatic exploration method based on value distribution; specifically, when making a decision in state s, the Q(s, a|θ) and D(s, a|θ) corresponding to each action are first calculated based on the current value network, and then the value distribution corresponding to each action is sampled to obtain the value corresponding to each action. Then select the action with the largest value obtained by sampling as the decision of the strategy, that is, The GEDQN algorithm combined with the memory enhancement method is called the GEMEDQN algorithm. In the GEMEDQN algorithm, a deep neural network is used to simultaneously predict Q(s, a) and D(s, a), and the two share the parameters of the shallow network, and a fully connected network is added at the last layer to predict the mean and variance.

2. The method for autonomous obstacle avoidance navigation of a UAV based on memory enhanced learning according to claim 1, characterized in that: The step 1 specifically includes the following: Step 1-1: Determine the input state s of the drone: At each time t, the drone observes the environment and obtains the observation o of the environment t , and calculate the corresponding local target g according to the global target t , and then use the deep learning method to extract its features and obtain the specific state feature representation s t ; UAV visual observation information t The observation of the drone consists of two parts: visual sensor information and drone position information, which mainly includes the deflection angle yaw of the drone's flight direction relative to the XOY plane. t And the current position of the drone (x t ,y t ); Dynamic relative target replaces the global absolute target method, using the relative value between the current position of the drone and the navigation target as the temporary target g of each decision node t , including the position of the temporary target and the deflection angle yaw of the temporary target as an auxiliary target feature gt , g t =(x gt ,y gt ,yaw gt ) as fusion information to replace the target information and location information of the drone; a convolutional neural network is used to extract features from the image information to obtain the observed feature vector Use multi-layer perceptron to extract the feature vector of target information And use the splicing method to fuse the image features with the target features to obtain the state features The obtained state features are used for subsequent fusion and processing; the input state of the drone is Step 1-2: Determine the output action set A of the drone: We discretize the control actions of the drone, which include three actions: turn left, turn right, and go straight. The specific details of the actions are: Action (1) Turn left: Turn left at a speed of 30° / s for 1 second; Action (2) Turn right: Turn right at a speed of 30° / s for 0.7 seconds; Action (3) Straight: Move forward in the direction of the camera at a speed of 2 m / s for 1 second. The angles of the left and right turns are different. The main motivation for this design is to create two actions that will not cancel each other out, preventing the drone from being trapped in certain environments. Step 1-3: Define that the quadcopter can reach the next input state s under the condition that the current input state s and the action amount a is taken ′ The probability of is the state transition probability function P; Step 1-4: Design a non-sparse reward function for the obstacle avoidance navigation problem of UAV. The reward function consists of four parts: transfer reward, collision penalty, target reward and step penalty. The transfer reward is used to represent the change in the distance of the UAV relative to the navigation target, so that the UAV can get corresponding feedback signals for each step of movement instead of getting effective rewards at the end of the plot. Use (x g ,y g ) represents the navigation goal of the drone in a complete episode, and the transfer reward r trans for: The collision penalty is defined as r collision =c t , where c t is an indicator variable indicating whether the drone has collided. If a collision occurs, c t is 1, otherwise c t is 0; similarly, the target reward is used to indicate whether the drone successfully reaches the target, which is defined as r tgt =d t , d t It is also an indicator variable used to indicate whether the drone has reached the target. If the drone reaches the target, then d t =1, otherwise d t is 0; the step penalty is used to break the dilemma when there are obstacles in the navigation target direction of the drone; the main reason for using the step penalty is that if there is no step penalty, and the obstacle and the navigation target direction are both in front of the drone, since the turning action almost does not produce negative rewards, and moving in other directions will result in less rewards, it may cause the drone to be trapped in a turning cycle, so the step penalty is also a necessary part; the step penalty is simply defined as r step =-0.01; the final reward function is obtained by weighting each part, and the final reward function R(s t ,a t ,s t+1 )for: R(s t ,a t ,s t+1 )=0.01*r trans -3*r collision +2*r tgt +r step Step 1-5: Define the discount coefficient 0<γ<1, which is used to calculate the cumulative value of returns in the entire process. The larger the discount coefficient, the more emphasis is placed on long-term returns.

3. The method for autonomous obstacle avoidance navigation of a UAV based on memory enhanced learning according to claim 1, characterized in that: The step 2 specifically includes the following: Step 2-1: Use the Q network instead of the Q table to approximate the state-action value function. Instead of directly using the state transition sequence obtained by interacting with the environment for training the deep neural network, the experience replay mechanism is used to train the Q network. The DQN algorithm saves the state transition sequence obtained by interacting with the environment, including the current state, action, reward, and next state, into the experience replay pool D. Then, when training the strategy, a certain amount of data samples are randomly sampled and the batch gradient descent algorithm is executed with these extracted data. Step 2-2: The experience replay unit in the DQN algorithm is a fixed-size list that stores state transition information during the interaction between the agent and the environment. It mainly includes<s,a,r,s',d> , where s represents the current state, a represents the action performed by the agent in state s, r represents the immediate reward obtained after performing action a, s' represents the state of the environment after performing the action, and d is an indicator variable used to indicate whether state s' is a terminal state. If it is a terminal state, d = 1, otherwise d = 0; Step 2-3: There are two neural networks with the same structure in the fixed target DQN algorithm, namely the current value network and the target value network; The current value network is used to interact with the environment and calculate the Q value Q(s,a|θ) of the current state. The target value network is used to calculate the Q value Q(s′,a′|θ′) of the next state and synchronizes parameters with the current value network at regular intervals. Step 2-4: For a state transition sequence<s,a,r,s',d> , according to the Bellman equation, the current state action value Q(s,a|θ) is estimated, and its estimated value is used as Q target (s,a) means, Q target The recursive relationship between (s,a) and the target state action value Q(s′,a′|θ′) is: Step 2-5: Q target The difference between (s,a) and Q(s,a) is called the temporal difference method. The parameter θ of the Q network is updated using the method of minimizing the mean square temporal difference method. B is used to represent a set of state transition sequence samples sampled from the experience replay pool. The corresponding loss function is: Step 2-6: Introduce the priority experience replay technology based on the temporal difference method to improve the utilization rate of rare samples: The temporal difference method is represented by δ, and the priority experience sampling algorithm uses the temporal difference method to calculate the priority of each state transition sequence; the temporal difference method in the DQN algorithm is: Step 2-7: After calculating the TD of the state transition sequence, its priority can be expressed as p = |δ| + ∈, where ∈ is a very small positive number, mainly used to prevent the TD from being 0; After obtaining the priority, the priority is normalized to calculate the probability of being sampled as P(i). α is a hyperparameter, which is mainly used to determine the impact of error on the priority. If α is 0, it degenerates into uniform sampling. Step 2-8: In order to solve the problem that priority sampling causes the data distribution in the experience replay pool to change and brings additional deviation to the loss function, the loss function is corrected by the importance sampling technique. A correction coefficient is multiplied by the loss accumulation before each state transition sequence. In the priority experience replay, the correction coefficient is ω i , N represents the number of empirical samples, β is a hyperparameter that determines the importance factor. If β is 0, it means that the importance sampling technique is not used to correct the sample bias; 4. The method for autonomous obstacle avoidance navigation of a UAV based on memory enhanced learning according to claim 1, characterized in that: The step 3 specifically includes the following: Step 3-1: Constructing action memory: Action memory saves and extracts the features of historical action sequences, integrating the historical trajectory information of the drone into the state information; using simple linear transformation to construct action memory, assuming A t =(a t-1 ,a t-2 ,…,a t-k ) represents the vector of actions made by K decisions before time t, using Represents the content of the action memory at time t, and uses a two-layer fully connected neural network to extract its features; Step 3-2: Construct spatial memory: Spatial memory is constructed by the historical visual observations and historical actions of the drone. The spatial layout corresponding to the observation information is restored based on the drone's observation information and the motion information after obtaining the observation information; historical visual information represents the drone's observations at a certain location, and historical actions represent the drone's movement mode before these observations are obtained; Research on deep reinforcement learning algorithm enhanced by historical memory The spatial memory of size K at time t is obtained by the observation information of the most recent K steps (o t-1 ,o t-2 ,…,o t-k ) and action sequence (a t-1 ,a t-2 ,…,a t-k ) is generated, where o i represents the observation information at time i, a i represents the action performed at time i; The spatial memory of size K at time t is defined as in Represents the content of the i-th spatial memory unit; When calculating the contents of each memory cell, use Represents the feature vector extracted by the convolutional neural network from the visual observation at time ti, using The feature vector representing the execution action at time ti is calculated by combining a multi-layer perceptron with action embedding. Action embedding uses a vector to encode the action number. First, the action number is encoded using one-hot encoding to obtain the action encoding vector. Then, the encoding vector is linearly transformed using the embedding matrix E to obtain the intermediate features corresponding to the action. Finally, a two-layer fully connected neural network is used to extract features to obtain the feature vector f corresponding to the action. a , the specific calculation process is: After obtaining the feature vector of the action sequence and the feature vector of the observation sequence, the content of the i-th memory unit at time t can be obtained by concatenating the action feature and the observation feature, that is, Spatial memory at time t can be rewritten as Step 3-3: Integration and extraction of spatial memory: After the spatial memory is constructed, a memory extraction and integration mechanism based on the attention mechanism is designed to extract and integrate spatial memory; assuming that the weight vector obtained by the attention mechanism is (α t-1 ,α t-2 ,…,α t-k ), then the characteristics of spatial memory are: The weight vector of spatial memory is calculated through the attention mechanism of the multi-layer perceptron; for each i that satisfies tK≤ti≤t-1, the weight ratio e of the i-th memory unit can be obtained i =W T tanh(W ∝ q i +U ∝ m i +b ∝ ), where W, W ∝ ,U ∝ ,b ∝ is a trainable weight, q t It is the feature vector of the current observation and the feature after the target feature is fused. After obtaining the weight ratio of each memory unit, in order to standardize the weight, the softmax function is used to normalize it; the weight vector of spatial memory is: Step 3-4: Memory-enhanced deep reinforcement learning algorithm: historical memory information is combined with current observations and target features in a spliced ​​manner to serve as the current environment state. Get the environmental state characteristics at time t Afterwards, the state information replaces the original observation information and target information, calculates the Q value and strategy, and optimizes the relevant parameters according to the optimization goal of the corresponding algorithm. The memory module is integrated into the DQN algorithm, which is called the MEDQN algorithm. Step 3-5: Improvement of reward function based on historical action sequence: By incorporating memory module into deep reinforcement learning algorithm, the agent can learn the actions it has performed in the past; the drone will encounter local loop problem caused by the contradiction between obstacle avoidance strategy and navigation strategy. This paper proposes an additional penalty method based on historical action sequence; named action penalty r act , which is calculated as: In r act In the calculation formula, l lr represents the length of the action sequence in which the agent only performs left and right turns; in order to integrate the action penalty, the additional action penalty r act Add it to the reward function, and the new reward function is: R(s t ,a t ,s t+1 )=0.01*r trans -3*r collision +2*r tgt +r step +r act 。

Images