Unmanned aerial vehicle autonomous navigation method based on improved TD3 algorithm

Abstract

The application discloses an unmanned aerial vehicle autonomous navigation method based on an improved TD3 algorithm, and comprises the following steps: designing a state space, an action space, a reward function, a policy network, a value network, a success experience pool and a failure experience pool of the improved TD3 algorithm; initializing a state and a temporary experience pool; collecting data of the policy network and adding the data to the temporary experience pool; adding overflow data of the temporary experience pool to the success experience pool when the data of the temporary experience pool overflows; judging whether an update condition is reached; when the update condition is reached, randomly sampling data from the success experience pool and the failure experience pool according to a proportion, and updating the value network and the policy network by using the sampled data; when the update condition is not reached, judging whether an unmanned aerial vehicle reaches a termination state; when a successful termination state is reached, putting data of the temporary experience pool into the success experience pool; and when a failure termination state is reached, putting data of the temporary experience pool into the failure experience pool. The application improves the quality of the sampled data, thereby improving the efficiency of the unmanned aerial vehicle autonomous navigation.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous navigation of unmanned aerial vehicles (UAVs), and in particular relates to an autonomous navigation method of UAVs based on an improved TD3 algorithm. Background Art

[0002] With the rapid development of artificial intelligence (AI), drones are increasingly being used in emergency rescue, logistics, agriculture, and forestry. Autonomous drone navigation is a key technology in drone technology. It typically refers to the process by which drones use onboard sensors (such as cameras and lidar) and high-precision map information to make autonomous flight decisions. This process enables drones to find a suitable path from their starting point to their destination without colliding with obstacles.

[0003] A wealth of research has been conducted on autonomous navigation for drones. Classic planning-based approaches, including geometric methods, graph theory methods, and probabilistic roadmaps, primarily focus on providing pre-planned paths using precise global maps, where the locations of obstacles are known. For uncertain environments, simultaneous localization and mapping (SLAM) is used to construct the unknown environment. Localization algorithms are then used to determine the drone's current position and navigate. As drone applications continue to expand, the environments they encounter are often dynamic, unknown, and complex. SLAM-based obstacle avoidance and navigation methods for drones struggle to efficiently build scene maps of these environments, hindering their real-time responsiveness and autonomous decision-making capabilities. To address these challenges, reinforcement learning algorithms are gaining increasing attention in the field of autonomous navigation for drones. Reinforcement learning, through continuous exploration and trial-and-error learning within the environment, enables drones to autonomously learn navigation strategies without the need for a priori maps, improving their adaptability in dynamic and complex environments. Deep reinforcement learning, combining the characteristics of deep learning with the use of neural networks to process high-dimensional inputs and complex environmental states, further enhances drones' perception capabilities for autonomous navigation. When a drone faces a complex and unknown environment, the TD3 algorithm in the reinforcement learning algorithm may encounter problems such as high collision rate and easy falling into local optimality. Summary of the Invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a UAV autonomous navigation method based on an improved TD3 algorithm, so as to solve the problems of underestimation and easy falling into local optimum of the TD3 algorithm in UAV autonomous navigation.

[0005] In order to achieve the above technical objectives, the present invention provides the following technical solutions:

[0006] The unmanned aerial vehicle autonomous navigation method based on the improved TD3 algorithm comprises the following steps:

[0007] S1, establishing an unmanned aerial vehicle autonomous navigation model based on an improved TD3 algorithm, describing the unmanned aerial vehicle autonomous navigation problem as a Markov decision process, and designing an improved TD3 algorithm with two strategy networks and multiple experience pools, namely a DAMEP-TD3 algorithm; defining the state space, action space, reward, strategy network, value network, success experience pool and failure experience pool of the DAMEP-TD3 algorithm;

[0008] S2, initializing the state of the unmanned aerial vehicle, and defining a temporary experience pool;

[0009] S3, collecting data obtained in the process of interaction between the unmanned aerial vehicle and the environment according to the strategy network, adding the data to the temporary experience pool, adding the overflow data to the success experience pool when the temporary experience pool data overflows, and then judging the update condition; if the temporary experience pool data does not overflow, directly judging the update condition;

[0010] S4, judging whether the update condition is reached:

[0011] When the update condition is reached, the success experience pool and the failure experience pool are randomly sampled according to a proportion respectively, and the sampled data are used to update the value network and the strategy network;

[0012] When the update condition is not reached, a termination state is judged;

[0013] S5, judging whether the unmanned aerial vehicle reaches the termination state:

[0014] When the termination state is not reached, step S3 is performed again;

[0015] When the termination state is reached, it is judged according to the reward value whether it is a successful termination state or a failed termination state; when it is a successful termination state, the temporary experience pool is emptied, and all the data thereof are put into the success experience pool; when it is a failed termination state, the temporary experience pool is emptied, and all the data thereof are put into the failure experience pool;

[0016] S6, after the temporary experience pool is emptied, returning to step S2 and repeating steps S2-S5 for multiple iterations to verify the effect of the overall model of the improved TD3 algorithm.

[0017] Further, in step S1, the state space and the action space are defined as follows:

[0018] The unmanned aerial vehicle information and the target point information are given by GPS and gyroscope, and the fusion state information of the two is denoted as s p , and the formula is expressed as:

[0019] s p =[xg -x,y g -y,z g -z,v x ,v y ,v z ,β,dis];

[0020] wherein x, y, z are the current position of the UAV, x g ,y g ,z g is the target point position, dis represents the straight-line distance between the UAV and the target point, β is the angle between the line connecting the UAV and the target point and the first visual angle direction of the UAV, v x ,v y ,v z represent the speed of the UAV in 3 dimensions, x g -x,y g -y,z g -z represents the relative position of the target point to the UAV;

[0021] The obstacle information is detected by the laser distance sensor on the horizontal plane and the vertical plane of the UAV, and the obstacle information is denoted as s e , and the formula is expressed as:

[0022] s e = [p i , l i ] ;

[0023] wherein p i is one-hot encoding, and the value is 1 when an obstacle is detected, and 0 when no obstacle is detected; l i is the length of the laser ray, and the value is the distance between the UAV and the obstacle when an obstacle is detected, and the value is the maximum detectable distance when an obstacle is detected;

[0024] Thus, the state s = [s p , s e ] is defined.

[0025] The action space of the DAMEP-TD3 algorithm stores the action a, which is represented by the force received by the UAV in each direction, and the formula is expressed as:

[0026] a = [a Forward , a Right , a Up , a Rotation ] ;

[0027] wherein a Forward , a Right , a Uprespectively represent the forces received by the UAV in the front-back, left-right, and up-down directions, and reflect the movement of the UAV in the front-back, left-right, and up-down directions; a Rotation represents the turning force received, and reflects the change in the yaw angle of the UAV.

[0028] Further, the defined reward in step 1 is specifically:

[0029] Different rewards are given to the action decisions of the UAV in different states to avoid sparse rewards, and the greater the total reward value, the higher the efficiency and safety of the action decisions of the UAV; mainly including:

[0030] Distance reward r dis ; evaluates whether the UAV reaches the vicinity of the target point, and takes the straight-line distance between the UAV and the target point as the reward standard, and the farther the distance, the smaller the reward, and the formula is expressed as:

[0031]

[0032] where dis represents the straight-line distance between the UAV and the target point, d is the set farthest reward distance, q1 is the distance reward constant, and p1 is the distance reward factor;

[0033] Obstacle reward r obs ; evaluates whether the UAV successfully avoids the obstacle, and sets a safety distance l safe , the smaller the distance between the UAV and the obstacle than the safety distance l safe , the smaller the reward, and the formula is expressed as:

[0034]

[0035] where l min represents the minimum distance between the UAV and the obstacle, p2 is the collision reward factor, and q2 is the collision reward constant;

[0036] Angle reward r angle ; evaluates whether the UAV flies in the direction of the target point, and takes the included angle β between the line connecting the UAV and the target point and the first visual angle direction of the UAV as the standard, the greater the included angle β, the smaller the reward, and the formula is expressed as:

[0037]

[0038] where p3 is the angle reward factor;

[0039] Time step reward r step ; evaluates how fast the UAV reaches the target point, and the more time steps required, the smaller the reward, and the formula is expressed as:

[0040] r step = -p4·n;

[0041] wherein p4 is a time step reward factor, and n is a number of time steps;

[0042] The four rewards are added to obtain the total reward r of the UAV, r = r dis + r obs + r angle + r step .

[0043] More specifically, the DAMEP-TD3 algorithm further includes two policy networks two value networks and corresponding target networks

[0044] On the basis of the policy network, the value network and the target network, further, step S4 specifically includes:

[0045] S41, when the update condition is reached, the sampling data (s, a, s', r, d) is obtained by randomly sampling from the success experience pool and the failure experience pool according to a proportion, wherein s represents the current state, a represents the action, s' represents the state at the next time, r represents the reward, and d represents the end value;

[0046] S42, using the sampling data (s, a, s', r, d), combining the target network updating the value network and the policy network

[0047] More specifically, step S42 includes the following sub-steps:

[0048] S421, updating the value network with the minimum loss function Loss The formula expression of the minimum loss function Loss is:

[0049]

[0050] wherein N is the batch size of the sampling data, λ is a regularization parameter, y is a time difference target value used to calculate the error of the current value estimate, and the formula expression is:

[0051] y = r + γ (1-d) V (s');

[0052] wherein γ is a discount factor representing the value decay of future rewards; and V (s') is a value function estimate value, and the formula expression is:

[0053] V (s') = v min {Q1 (s', a'), Q2 (s', a'')} + (1-v) max {Q1 (s', a'), Q2 (s', a'')}

[0054] wherein v [0, 1) is a parameter of the balancing target value network, denotes the minimum value of the values output by the two target value networks under the state s' and the action a' at the next moment, denotes the minimum value of the values output by the two target value networks under the state s' and the action a' at the next moment;

[0055] wherein a' and a" respectively denote the action at the next moment according to the target policy network and under the noise ∈, which is expressed by the formula as follows:

[0056]

[0057] S422, update the policy network by using the policy gradient which is expressed by the formula as follows:

[0058]

[0059] wherein J (φ i ) denotes a target function of the policy, used to evaluate the expected return of the policy in the environment, denotes the action output by the policy network at the state s when the action a is equal to the gradient of the action a, denotes the value under the state s and the action a, denotes the parameter gradient of the policy network at the state s, used to update the parameter φ i of the policy network.

[0060] Further, the step S5 comprises:

[0061] determining whether a termination state is reached according to the end value d obtained by the unmanned aerial vehicle interacting with the environment;

[0062] when the termination state is reached, determining whether the termination state is a successful termination state or a failed termination state according to the reward: if the reward is greater than 0, the termination state is determined to be successful, and if the reward is not greater than 0, the termination state is determined to be failed.

[0063] Based on the above technical solutions, the present application has the following beneficial effects:

[0064] 1. For the underestimation problem existing in the TD3 algorithm, the network structure of the TD3 algorithm is adjusted, two policy networks are adopted, and the uncertainty brought by the double value networks is relieved by using the regularization value network.

[0065] 2. A multi-experience pool mechanism is introduced, and the data generated by the interaction between the unmanned aerial vehicle and the environment is stored in the success experience pool and the failure experience pool respectively. Compared with the traditional single experience pool, this mechanism can improve the quality of the sampled data, thereby improving the efficiency of the unmanned aerial vehicle autonomous navigation.

[0066] 3. The DAMEP-TD3 algorithm proposed in the application can have better navigation success rate, and can avoid obstacles and shorten the navigation path during navigation. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 Flowchart of the application;

[0068] Figure 2 Unmanned aerial vehicle information and target point information schematic diagram in the experiment of the application;

[0069] Figure 3 Horizontal laser distance sensor of the unmanned aerial vehicle in the experiment of the application;

[0070] Figure 4 Vertical laser distance sensor of the unmanned aerial vehicle in the experiment of the application;

[0071] Figure 5 Action space schematic diagram of the unmanned aerial vehicle in the experiment of the application;

[0072] Figure 6 Comparison of reward values of the application and TD3 algorithm in the training process;

[0073] Figure 7 Comparison of data of the application and TD3 algorithm in the test scene. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application, and are not used to limit the application.

[0075] The steps in the application are arranged by using labels, but are not used to limit the order of the steps, unless the order of the steps is clearly stated or the execution of a step needs other steps as a basis, otherwise the relative order of the steps can be adjusted. It can be understood that the term "and / or" used herein relates to and covers any and all possible combinations of one or more of the associated listed items.

[0076] Reference Figures 1 to 7 is given to the specific implementation of the present embodiment:

[0077] A kind of unmanned aerial vehicle autonomous navigation method based on improved TD3 algorithm, specific process is as follows:Figure 1 As shown, comprising:

[0078] S1, establish a UAV road autonomous navigation model based on improved TD3 algorithm, describe the UAV autonomous navigation problem as a Markov decision process, design an improved TD3 algorithm with two strategy networks and multiple experience pools: DAMEP-TD3 algorithm; define the state space, action space, reward, strategy network, value network, success experience pool and failure experience pool of the DAMEP-TD3 algorithm;

[0079] More specifically, the state space in the embodiment includes UAV self information, target point position and obstacle information, etc. The state space is a set of variables used to describe the current environment in reinforcement learning. A reasonable state space design can reduce the complexity of the problem, speed up the learning process, and improve the performance of the agent.

[0080] As a preferred embodiment, as shown in Figure 2 The UAV self information and target point information in the embodiment are shown. The UAV self information and target point information are given by GPS and gyroscope. The fusion state information of the two is denoted as s p , which is expressed by the formula:

[0081] s p =[x g -x,y g -y,z g -z,v x ,v y ,v z ,β,dis];

[0082] Wherein, x, y, z are the current position of the UAV, x g , y g , z g is the target point position, dis represents the straight-line distance between the UAV and the target point, β is the included angle between the line connecting the UAV and the target point and the first visual direction of the UAV, v x , v y , v z represent the speed of the UAV in three dimensions, x g -x, y g -y, z g -z represents the relative position of the target point to the UAV.

[0083] As shown in Figure 3 , Figure 4 The obstacle information is detected by the laser distance sensor on the horizontal plane and the vertical plane of the UAV. The angle range scanned by the sensor is represented by π, and the angle between every two laser beams is l1 to l7 represent the ray length of the sensor in the horizontal plane, l8 to l 14 are the ray length of the vertical plane, wherein l4 and l 11 are the same laser beam. If the sensor does not detect an obstacle within the specified orientation, the length of the ray is the maximum distance that can be detected, otherwise the length is the distance between the UAV and the obstacle; the obstacle information is recorded as s e , whose formula is expressed as:

[0084] s e = [p i , l i ], i = 1...14,;

[0085] wherein p i is one-hot encoding, the value is 1 when an obstacle is detected, and 0 when no obstacle is detected; combined with the UAV itself information, target point information and obstacle information, the state of the environment to the UAV s = [s p , s e ] can be defined.

[0086] As a preferred embodiment, the action space in step S1 is further defined, and in this embodiment, the UAV used is a quadrotor UAV, which is a flying vehicle composed of four rotors. During flight, the quadrotor UAV controls the attitude and flight direction by adjusting the rotation speed of the rotors. In this experiment, the forces in each direction of the UAV are taken as executable actions to realize forward, backward, left, right, up and down movements, and the steering of the UAV is controlled by the rotation angle, as shown in Figure 5 .

[0087] 500.a Forward , 300.a Right , 500.a Up respectively represent the forces acting on the X-axis, Y-axis and Z-axis of the UAV, and in this embodiment, the numerical values of the forces are 500, 300 and 500 respectively, a new = a old - 2a Rotation represents the change of the yaw angle of the UAV around the Z-axis, and the formula of the obtained action a is expressed as:

[0088] a = [a Forward , a Right , a Up , a Rotation ];

[0089] wherein a Forward , a Right , a Up respectively represent the forces received by the UAV in the forward, backward, left, right, up and down directions, which represent the forward, backward, left, right, up and down movements of the UAV; a Rotationrepresents the steering force received, reflecting the change of the unmanned aerial vehicle yaw angle.

[0090] As a preferred embodiment, in step S1 of the embodiment, different rewards are given to the action decisions of the unmanned aerial vehicle in different states to avoid sparse rewards, and the greater the total reward value is, the higher the efficiency and safety of the action decisions of the unmanned aerial vehicle are; the rewards mainly include:

[0091] Distance reward r dis ; whether the unmanned aerial vehicle reaches the vicinity of the target point is evaluated, and the straight-line distance between the unmanned aerial vehicle and the target point is taken as the reward standard, and the farther the distance is, the smaller the reward is, and the formula is expressed as:

[0092]

[0093] wherein dis represents the straight-line distance between the unmanned aerial vehicle and the target point, d is the set farthest reward distance, q1 is the distance reward constant, and p1 is the distance reward factor;

[0094] In the embodiment, q1=5 and p1=0.1 are taken;

[0095] Obstacle reward r obs ; whether the unmanned aerial vehicle successfully avoids the obstacle is evaluated, and a safety distance l safe is set, and the smaller the distance between the unmanned aerial vehicle and the obstacle is than the safety distance l safe , the smaller the reward is, and the formula is expressed as:

[0096]

[0097] wherein l min represents the minimum distance between the unmanned aerial vehicle and the obstacle, p2 is the collision reward factor, and q2 is the collision reward constant;

[0098] In the embodiment, q2=5 and p2=0.1 are taken;

[0099] Angle reward r angle ; whether the unmanned aerial vehicle flies in the direction of the target point is evaluated, and the included angle β between the straight line connecting the unmanned aerial vehicle and the target point and the first visual angle direction of the unmanned aerial vehicle is taken as the standard, the greater the included angle β is, the smaller the reward is, and the formula is expressed as:

[0100]

[0101] wherein p3 is the angle reward factor;

[0102] In the embodiment, p3=0.05 is taken;

[0103] Time step reward r step ; whether the unmanned aerial vehicle reaches the target point quickly or slowly is evaluated, and the greater the time step required is, the smaller the reward is, and the formula is expressed as:

[0104] r step = -p4*n;

[0105] wherein p4 is a time step reward factor, and n is a number of time steps;

[0106] In this embodiment, p4 = 0.02 is taken;

[0107] The four rewards are added to obtain the total reward r of the UAV, r = r dis + r obs + r angle + r step .

[0108] In addition, the DAMEP-TD3 algorithm in this embodiment also includes two policy networks two value networks and corresponding target networks

[0109] S2, after the preparation of step S1, the training task of the overall model is started; first, the state of the UAV is initialized, and a temporary experience pool is defined, the size of the temporary experience pool is set to 200;

[0110] S3, according to the policy network, the data obtained in the process of interaction between the UAV and the environment is collected, and the data is added to the temporary experience pool; when the temporary experience pool data overflows, if the UAV still does not reach the termination state (1. successfully reaches the destination, 2. collides with the obstacle), it is considered that the past decision has a positive impact on the present, so the overflow data is added to the success experience pool, and then the update condition is judged; if the temporary experience pool data does not overflow, the update condition is directly judged;

[0111] S4, judge whether the update condition is reached, the judgment basis is whether the success experience pool is full;

[0112] When the update condition is reached, the data is randomly sampled from the success experience pool and the failure experience pool according to the proportion, the batch size of the sampled data is defined as N, in this embodiment N is set to 128, when the update condition is reached, 108 data is taken from the success experience pool, and 20 data is taken from the failure experience pool for updating, the value network and the policy network are updated using the sampled data;

[0113] If the update condition is not reached, the termination state is judged;

[0114] As a preferred embodiment, step S4 includes:

[0115] S41, when the update condition is reached, random sampling is performed from the success experience pool and the failure experience pool according to a proportion, and the obtained sampling data is defined as (s, a, s', r, d), wherein s represents a current state, a represents an action, s' represents a state at a next moment, r represents a reward, and d represents an end value;

[0116] S42, the sampling data (s, a, s', r, d) is used to update the target network update the value network and the policy network

[0117] More specifically, in the embodiment, a minimum loss function is used to update the value network, and a policy operator is used to update the policy network, and the specific process is as follows:

[0118] S421, the minimum loss function Loss is used to update the value network The formula of the minimum loss function Loss is as follows:

[0119]

[0120] Wherein, N is the batch size of the sampling data, λ is a regularization parameter, y is a time difference target value, which combines the immediate reward and the value estimation of the future state, and provides a target value for calculating the error of the current value estimation, and the formula is as follows:

[0121] y=r+γ(1-d)V(s');

[0122] Wherein, γ is a discount factor, indicating the value decay of future rewards, reflecting the importance of future rewards; V(s') is a value function estimation value, and the formula is as follows:

[0123] V(s')=ν·min{Q1(s',a'),Q2(s',a")}+(1-ν)·max{Q1(s',a'),Q2(s',a")};

[0124] Wherein, ν∈[0,1), is a parameter for balancing the target value network, Indicates the minimum value of the values output by the two target value networks under the state s' and the action a' at the next moment, Indicates the minimum value of the values output by the two target value networks under the state s' and the action a' at the next moment; the traditional TD3 algorithm has an underestimation problem, and the value function estimation value designed in the present application can alleviate the underestimation bias of the values output by the target value network.

[0125] In addition, in the above formula, a' and a" respectively represent the state s' at the next moment according to the target policy network and The next action at noise e is obtained, which is expressed by the formula:

[0126]

[0127] S422, update the policy network by using the policy gradient The formula is:

[0128]

[0129] Wherein, J (φ i ) represents the objective function of the policy, which is used to evaluate the expected return of the policy in the environment, represents the action a equal to the action output by the policy network In state s , The gradient of action a, represents the value in state s, action a, represents the parameter gradient of the policy network in state s, which is used to update the parameters of the policy network φ i .

[0130] S5, judge whether the unmanned aerial vehicle reaches the termination state:

[0131] If the termination state is not reached, step S3 is performed again;

[0132] If the termination state is reached, it is judged whether it is a successful termination state or a failed termination state according to the reward value; if it is a successful termination state, the temporary experience pool is emptied, and all its data are put into the successful experience pool; if it is a failed termination state, the temporary experience pool is emptied, and all its data are put into the failed experience pool;

[0133] More specifically, whether the termination state is reached is judged according to the end value d obtained by the unmanned aerial vehicle interacting with the environment;

[0134] When the termination state is reached, it is judged whether it is a successful termination state or a failed termination state according to the reward: if the reward is greater than 0, it is judged as successful, and if the reward is not greater than 0, it is judged as failed.

[0135] S6, after emptying the temporary experience pool, return to step S2 again, repeat steps S2-S5, and perform multiple iterations to verify the effect of the improved TD3 algorithm overall model.

[0136] Through the above specific process, the actual effect shown in Figure 6 , Figure 7 is obtained by the embodiment.

[0137] Figure 6is a comparison chart of reward values of DAMEP-TD3 algorithm and traditional TD3 algorithm in the training process. Based on Unity 3D, the training scene is built, the task space of the UAV is set as a cuboid region with a size of 400x400x100, the starting point of the UAV is randomly initialized on one side of the region, and the target point is randomly initialized on the other side. Different shaped obstacles are randomly generated in the middle region, including 5 cubes with a size of 30x30x100 and 5 cylinders with a size of 15x100. The target point is simplified as a ball with a diameter of 10, and the flight height of the UAV is limited between 20 and 100. The number of iterations in the training process is 5000, the maximum step size is set to 500, and three termination conditions are included in the training process, namely: (1) the UAV reaches the target point; (2) the UAV collides with the obstacle; (3) the maximum step size is reached. Each termination is recorded as an iteration end. The results after multiple iterations can be seen from the figure, compared with TD3, the convergence speed of DAMEP-TD3 algorithm is faster, and the convergence process is more stable.

[0138] Figure 7 The comparison results of the two algorithms in different environments of the test scene are shown in the figure, where environment 1 represents the training environment, including 10 obstacles; environment 2 changes the shape of the obstacle, 30x30x100 becomes 100x30x30; environment 3 increases the number of obstacles, a total of 12 obstacles; environment 4 changes the shape of the obstacle (30x30x100 becomes 100x30x30) and increases the number (12 obstacles). Four evaluation indexes are used to evaluate the performance of the UAV autonomous navigation, including success rate (the ratio of successfully reaching the target point in 2000 iterations), collision rate (the ratio of UAV and obstacle collision), loss rate (in the test process, the ratio of UAV neither colliding with the obstacle nor reaching the target point), and path length (the average path length of the UAV successfully navigating to the target point).

[0139] From Figure 7 it can be seen that in all environments, the effect of DAMEP-TD3 algorithm is better than TD3, and it has higher success rate, lower collision rate and loss rate, and shorter path length. When the environment changes, the change of success rate of DAMEP-TD3 algorithm is less than that of TD3. For example, environment 1 and environment 4, compared with environment 1, the shape and number of obstacles in environment 4 are changed, the success rate of TD3 algorithm decreases from 92.2% to 81.2%, a total decrease of 11%, and the success rate of DAMEP-TD3 decreases from 95.75% to 87.6%, a decrease of 8.15%. Therefore, it can be concluded that the generalization ability of DAMEP-TD3 is better than that of TD3, and it can better adapt to complex environments. Combined with Figure 6 and Figure 7 it can be concluded that the improved algorithm has obvious performance better than TD3.

[0140] In summary, the DAMEP-TD3 algorithm proposed in the present application is compared with the traditional TD3 algorithm, through two policy networks, the uncertainty brought by the double value network is alleviated, and the quality of the collected data is improved by combining multiple experience pools, so that the algorithm model proposed in the present application has faster convergence speed and more stable convergence process, and has better generalization ability when facing complex environment, the performance of the overall model has obvious advantages, has better navigation success rate, and can avoid obstacles and shorten the path length.

[0141] In the present specification, the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" and the like describe at least one embodiment or example described in connection with a specific feature, structure, material or characteristic. These described specific features, structures, materials or characteristics can be combined in one or more embodiments or examples in an appropriate manner. In addition, the skilled person can combine and integrate different embodiments or examples described in the present specification and their features without contradiction.

[0142] The logic and / or steps shown in the flowchart or otherwise described can be considered a sequence of executable instructions for implementing the logic function. These instructions can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus. These systems, devices or apparatuses include processor systems or other systems capable of receiving and executing instructions.

[0143] The above embodiments detail the principles and implementation of the present application, and use specific examples to illustrate the working principle. These examples are only used to help understand the method of the present application and its core idea. At the same time, according to the idea of the present application, the actual implementation and application range may vary. Therefore, the content of the specification should not be understood as a limitation of the present application.

Claims

1. An unmanned aerial vehicle autonomous navigation method based on an improved TD3 algorithm, characterized in that, The application relates to an unmanned aerial vehicle (UAV) autonomous navigation model based on an improved TD3 algorithm. S1, establishing an unmanned aerial vehicle (UAV) autonomous navigation model based on an improved TD3 algorithm, describing the UAV autonomous navigation problem as a Markov decision process, and designing an improved TD3 algorithm with two strategy networks and multiple experience pools, namely a DAMEP-TD3 algorithm; defining the state space, action space, reward, strategy network, value network, successful experience pool and failed experience pool of the DAMEP-TD3 algorithm; The state space of the DAMEP-TD3 algorithm stores a state s, and the state s specifically comprises UAV self information, target point information and obstacle information; The UAV self information and target point information are given through GPS and gyroscope, and the fusion state information of the two is recorded as s p , which is expressed by the following formula: s p = [x g - x, y g - y, z g - z, v x , v y , v z , β, dis] ; Wherein, x, y, z are the current position of the UAV, x g ,y g ,z g is the target point position, dis represents the straight-line distance between the UAV and the target point, β is the included angle between the line connecting the UAV and the target point and the first visual direction of the UAV, v x ,v y ,v z represent the speed of the UAV in three dimensions, x g -x, y g -y, z g -z represents the relative position of the target point to the UAV; Obstacle information is given by the laser distance sensor on the horizontal plane and the vertical plane of the UAV, and the obstacle information is denoted as s e The formula is: s e = [p i , l i ], i = 1...14; wherein, ρ i is one-hot encoding, the value is 1 when an obstacle is detected, and the value is 0 when no obstacle is detected; l i is the length of the laser ray, if the sensor does not detect an obstacle within the specified orientation, the length of the ray is the maximum distance that can be detected, otherwise the length is the distance between the UAV and the obstacle; the angular range of the sensor scan is denoted by π, and the angle between each two lasers is l1 to l7 represent the length of the ray of the sensor in the horizontal plane, and l8 to l 14 are the lengths of the ray in the vertical plane, wherein l4 and l 11 are the same laser; The action space of the DAMEP-TD3 algorithm stores an action a, and the action a is represented by forces received by the UAV in each direction, and is expressed by a formula as follows: a = [a Forward ,a Right ,a Up ,a Rotation ]; Wherein, a Forward , a Right , a Up respectively represent the force received by the unmanned aerial vehicle in the front-back, left-right, and up-down directions, and embody the movement of the unmanned aerial vehicle in the front-back, left-right, and up-down directions; a Rotation represents the turning force received, and embodies the change of the yaw angle of the unmanned aerial vehicle. The DAMEP-TD3 algorithm further includes 2 policy networks 2 value networks and a corresponding target network S2, initializing the state of the UAV, and defining a temporary experience pool; S3, collecting data obtained in the process of UAV and environment interaction according to the strategy network, adding the data to the temporary experience pool, adding the overflow data of the temporary experience pool to the successful experience pool, and then judging the update condition; if the data of the temporary experience pool does not overflow, directly judging the update condition; S4, judging whether the update condition is reached: When the update condition is reached, the successful experience pool and the failed experience pool are randomly sampled according to a proportion, and the sampling data are used to update the value network and the strategy network; When the update condition is not reached, a termination state is judged; step S4 comprises: S41, when the update condition is reached, the successful experience pool and the failed experience pool are randomly sampled according to a proportion, and the sampling data are defined as (s, a, s', r, d), wherein s represents the current state, a represents the action, s' represents the state at the next moment, r represents the reward, and d represents the end value; S42, using the sampled data (s, a, s', r, d), the target network updating the value network and the policy network Step S42 specifically comprises the following steps: S421、updating the value network with a minimum loss function Loss The formula of the minimum loss function Loss is: Wherein N is the batch size of the sampling data, lambda is a regularization parameter, y is a time difference target value used to calculate the error of the current value estimation, and is expressed by a formula as follows: y=r+gamma(1-d)V(s'); Wherein gamma is a discount factor representing the value attenuation of future rewards; V(s') is a value function estimation value, and is expressed by a formula as follows: V(s')=nu.min{Q1(s',a'),Q2(s',a")}+(1-nu).max{Q1(s',a'),Q2(s',a")}; where v e [0, 1) is a parameter of the balancing target value network, denotes the minimum of the values output by the two target value networks under the next time state s' and action a', denotes the minimum of the values output by the two target value networks under the next time state s' and action a'. where a' and a" represent the next state s' according to the target policy network and the next action a' under noise ∈, which is expressed in the formula as: S422, update the policy network using policy gradient The formula is expressed as: where J(φ i ) denotes the objective function of the policy, which evaluates the expected return of the policy in the environment, denotes the gradient of the value function with respect to the parameters of the policy network at state s, the gradient of the action a, denotes the value at state s, action a, denotes the gradient of the parameters of the policy network at state s, which is used to update the parameters of the policy network φ i ;​ S5, judging whether the UAV reaches a termination state: When the termination state is not reached, step S3 is performed again; When the termination state is reached, whether the termination state is a successful termination state or a failed termination state is judged according to the reward value; when the termination state is the successful termination state, the temporary experience pool is emptied, and all data of the temporary experience pool are put into the successful experience pool; when the termination state is the failed termination state, the temporary experience pool is emptied, and all data of the temporary experience pool are put into the failed experience pool; S6, after the temporary experience pool is emptied, step S2 is returned to, steps S2-S5 are repeated, and the effect of the improved TD3 algorithm is verified.

2. The improved TD3 algorithm-based unmanned aerial vehicle autonomous navigation method according to claim 1, characterized in that, In step S1, different rewards are given to the action decisions of the UAV in different states to avoid sparse rewards. The greater the total reward value, the higher the efficiency and safety of the UAV action decision. The rewards mainly include: Distance reward r dis ; whether the UAV reaches the target point, the straight-line distance between the UAV and the target point as the reward standard, the farther the reward is smaller, its formula is expressed as: where dis represents the straight-line distance between the UAV and the target point, d is the set farthest reward distance, q1 is the distance reward constant, and p1 is the distance reward factor. Obstacle reward r obs ; evaluate whether the UAV successfully avoids the obstacle and sets a safety distance l safe , the smaller the distance between the UAV and the obstacle is than the safety distance l safe , the smaller the reward is, which is expressed in the formula: wherein, l min represents the minimum distance between the UAV and the obstacle, p2 is a collision reward factor, and q2 is a collision reward constant. Angle reward r angle ; whether the UAV flies along the direction of the target point, the included angle β between the line connecting the UAV and the target point and the first visual angle direction of the UAV is taken as the standard, the larger the included angle β is, the smaller the reward is, and the formula is expressed as: where p3 is the angle reward factor. Time step reward r step To evaluate the speed of the UAV to reach the target point, the more time steps needed, the smaller the reward, expressed in the formula: r step = -p4-n; where p4 is the time step reward factor, and n is the number of time steps. The four rewards are added together to obtain the total reward r of the UAV, r = r dis + obs + angle + step .

3. The improved TD3 algorithm-based unmanned aerial vehicle autonomous navigation method according to claim 1, characterized in that, In step S5: According to the end value d obtained by the UAV interacting with the environment, it is judged whether the termination state is reached. When the termination state is reached, it is judged whether it is a successful termination state or a failed termination state according to the reward: if the reward is greater than 0, it is judged as successful, and if the reward is not greater than 0, it is judged as failed.

Images