Unmanned aerial vehicle autonomous navigation method based on improved TD3 algorithm
By introducing the DAMEP-TD3 algorithm in the autonomous navigation of drones, using two policy networks and multiple experience pool mechanisms, the TD3 algorithm underestimation and local optimization problems are solved, the navigation success rate and path efficiency are improved, and better obstacle avoidance ability and complex environment adaptability.
Patent Information
- Application Number
- CN202411065005.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-05
AI Technical Summary
The TD3 algorithm has problems of underestimation and easy to fall into local optimality in autonomous navigation of drones, making it difficult to effectively avoid obstacles in complex environments and ensure navigation success rate.
Using the improved TD3 algorithm, the DAMEP-TD3 algorithm is designed, and the state space, action space, reward function is defined through the mechanism of two policy networks and multiple experience pools, and the success and failure experience pool is introduced to optimize the network structure and data sampling. Strategy to improve navigation efficiency.
Improve the success rate of autonomous navigation of drones and the efficiency of navigation paths, better avoid obstacles and shorten navigation paths, alleviate the uncertainty brought by dual-value networks, and improve the adaptation of navigation in complex environments. ability.
Smart Images

Figure CN118963407A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of unmanned aerial vehicle autonomous navigation, and in particular relates to an unmanned aerial vehicle autonomous navigation method based on an improved TD3 algorithm. Background Art
[0002] With the rapid development of artificial intelligence technology, drones are increasingly being used in emergency rescue, logistics, agriculture and forestry operations, and other fields. Autonomous navigation is a key technology in drone technology, which usually refers to the process of autonomous decision-making by drones using onboard sensors (such as cameras, lidar) and high-precision map information. In this process, drones are able to find a suitable path from their starting location to their destination without colliding with obstacles.
[0003] There are a lot of relevant results on the research of autonomous navigation of UAVs. Classic planning-based methods, including geometric methods, graph theory methods, and probabilistic roadmap methods, mainly focus on providing pre-planned paths with the help of accurate global maps, where the positions of obstacles are known. For uncertain environments, Simultaneous Localization and Mapping (SLAM) is used to construct unknown environments, and then the positioning algorithm is used to determine the current position of the UAV and navigate. With the continuous expansion of UAV application scenarios, the environments they face are often dynamic, unknown, and complex. The obstacle avoidance navigation method of UAVs based on SLAM is difficult to efficiently establish scene maps of these environments, thus failing to guarantee the real-time response capability and autonomous decision-making capability of UAVs. In order to meet these challenges, the application of reinforcement learning algorithms in the field of autonomous navigation of UAVs has gradually attracted attention. Reinforcement learning can autonomously learn navigation strategies without prior maps through continuous exploration and trial-and-error learning of UAVs in the environment, thereby improving the adaptability of UAVs in dynamic and complex environments. Deep reinforcement learning combines the characteristics of deep learning and uses neural networks to process high-dimensional inputs and complex environmental states, further enhancing the perception ability of UAVs in autonomous navigation. When a drone faces a complex and unknown environment, the TD3 algorithm in the reinforcement learning algorithm may encounter problems such as high collision rate and easy falling into local optimality. Summary of the invention
[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a UAV autonomous navigation method based on an improved TD3 algorithm, so as to solve the problems of underestimation and easy falling into local optimum of the TD3 algorithm in the autonomous navigation of UAV.
[0005] In order to achieve the above technical objectives, the present invention provides the following technical solutions:
[0006] The autonomous navigation method of UAV based on the improved TD3 algorithm includes the following steps:
[0007] S1. Establish a UAV autonomous navigation model based on the improved TD3 algorithm, describe the UAV autonomous navigation problem as a Markov decision process, and design an improved TD3 algorithm with two strategy networks and multiple experience pools: DAMEP-TD3 algorithm; define the state space, action space, reward, strategy network, value network, success experience pool and failure experience pool of the DAMEP-TD3 algorithm;
[0008] S2, initialize the state of the drone and define a temporary experience pool;
[0009] S3. Collect data obtained during the interaction between the drone and the environment according to the strategy network, and add the data to the temporary experience pool. If the temporary experience pool overflows, add the overflowed data to the success experience pool, and then judge the update conditions; if the temporary experience pool does not overflow, directly judge the update conditions;
[0010] S4. Determine whether the update condition is met:
[0011] When the update conditions are met, random sampling is performed from the successful experience pool and the failed experience pool in proportion, and the sampled data is used to update the value network and the strategy network;
[0012] If the update conditions are not met, the termination status is determined;
[0013] S5. Determine whether the drone has reached the termination state:
[0014] If the termination state is not reached, step S3 is repeated;
[0015] When the termination state is reached, the reward value is used to determine whether it is a successful termination state or a failed termination state; if it is a successful termination state, the temporary experience pool is cleared and all its data is put into the successful experience pool; if it is a failed termination state, the temporary experience pool is cleared and all its data is put into the failed experience pool;
[0016] S6. After clearing the temporary experience pool, return to step S2 and repeat steps S2-S5 for multiple iterations to verify the effect of the overall model of the improved TD3 algorithm.
[0017] Furthermore, in step S1, the state space and action space defined are specifically:
[0018] The drone’s own information and target point information are given by GPS and gyroscope, and the fusion state information of the two is recorded as s p , the formula is:
[0019] s p =[xg -x,y g -y,z g -z,v x ,v y ,v z ,β,dis];
[0020] Among them, x, y, z are the current positions of the drone, x g ,y g ,z g is the position of the target point, dis is the straight-line distance between the drone and the target point, β is the angle between the line connecting the drone and the target point and the direction of the drone's first-person perspective, and v x ,v y ,v z represents the speed of the drone in three dimensions, x g -x,y g -y,z g -z indicates the relative position of the target point to the drone;
[0021] Obstacle information is detected by laser distance sensors on the horizontal and vertical surfaces of the drone. The obstacle information is recorded as s e , its formula is expressed as:
[0022] s e =[ρ i ,l i ];
[0023] Among them, ρ i It is a one-hot encoding, the value is 1 if an obstacle is detected, and 0 if no obstacle is detected; l i is the length of the laser beam. When an obstacle is detected, its value is the distance between the drone and the obstacle. When an obstacle is detected, its value is the maximum detectable distance.
[0024] Thus, the state s=[s p ,s e ].
[0025] The action space of the DAMEP-TD3 algorithm stores the action a, which is represented by the forces in all directions of the drone, and is expressed as follows:
[0026] a=[a Forward ,a Right ,a Up ,a Rotation ];
[0027] Among them, a Forward 、a Right 、a UpThey respectively represent the forces received by the drone in the front, back, left, right, and up and down directions, reflecting the front, back, left, right, and up and down movements of the drone; a Rotation Indicates the steering force and reflects the change in the yaw angle of the drone.
[0028] Furthermore, the rewards defined in step 1 are specifically:
[0029] Different rewards are given to the drone's action decisions in different states to avoid sparse rewards. The larger the total reward value, the higher the efficiency and safety of the drone's action decision-making; mainly including:
[0030] Distance Reward dis ; To evaluate whether the drone has reached the target point, the straight-line distance between the drone and the target point is used as the reward standard. The farther the distance, the smaller the reward. The formula is expressed as:
[0031]
[0032] Among them, dis represents the straight-line distance between the drone and the target point, d is the set maximum reward distance, q1 is the distance reward constant, and p1 is the distance reward factor;
[0033] Obstacle Rewards obs ; Evaluate whether the drone successfully avoids obstacles and sets a safe distance l safe The closer the distance between the drone and the obstacle is to the safety distance l safe , the smaller the reward, the formula is expressed as:
[0034]
[0035] Among them, l min represents the minimum distance between the drone and the obstacle, p2 is the collision reward factor, and q2 is the collision reward constant;
[0036] Angle Reward r angle ; To evaluate whether the drone is flying in the direction of the target point, the angle β between the line connecting the drone and the target point and the direction of the drone's first-person perspective is used as the standard. The larger the angle β, the smaller the reward. The formula is expressed as:
[0037]
[0038] Among them, p3 is the angle reward factor;
[0039] Time step reward r step ; To evaluate how fast the drone reaches the target point, the more time steps required, the smaller the reward. The formula is:
[0040] r step = -p4·n;
[0041] Among them, p4 is the time step reward factor, and n is the number of time steps;
[0042] Add up the four rewards to get the total reward r of the drone, r = r dis +r obs +r angle +r step .
[0043] More specifically, the DAMEP-TD3 algorithm also includes two strategy networks: 2 Value Networks And the corresponding target network
[0044] On the basis of the above-mentioned strategy network, value network and target network, further, step S4 specifically includes:
[0045] S41. When the update condition is met, random sampling is performed from the success experience pool and the failure experience pool in proportion, and the obtained sampling data is defined as (s, a, s′, r, d), where s represents the current state, a represents the action, s′ represents the state at the next moment, r represents the reward, and d represents the end value;
[0046] S42, using the sampled data (s, a, s′, r, d), combined with the target network Update value network and policy network
[0047] More specifically, step S42 includes the following sub-steps:
[0048] S421. Update the value network with the minimum loss function Loss The formula of the minimum loss function Loss is expressed as:
[0049]
[0050] Where N is the batch size of the sampled data, λ is the regularization parameter, and y is the time series difference target value, which is used to calculate the error of the current value estimate. The formula is expressed as:
[0051] y = r + γ (1-d) V (s');
[0052] Among them, γ is the discount factor, which indicates the value decay of future rewards; V(s′) is the estimated value of the value function, which is expressed as:
[0053] V(s′)=ν·min{Q1(s′,a′),Q2(s′,a″)}+(1-ν)·max{Q1(s′,a′),Q2(s′,a″)};
[0054] Among them, ν∈[0,1), is the parameter of the balanced target value network, It represents the minimum value of the output of the two target value networks under the state s′ and action a′ at the next moment. It represents the minimum value of the output of the two target value networks under the state s′ and action a′′ at the next moment;
[0055] Among them, a′ and a″ respectively represent the state s′ at the next moment according to the target strategy network and The action at the next moment obtained under noise ∈ is expressed as:
[0056]
[0057] S422, using policy gradient to update the policy network The formula is:
[0058]
[0059] Among them, J(φ i ) represents the objective function of the strategy, which is used to evaluate the expected return of the strategy in the environment. Indicates that action a is equal to the policy network The action output in state s hour, The gradient of action a, Indicates the value of action a in state s. Represents the parameter gradient of the policy network at state s, which is used to update the parameter φ of the policy network i .
[0060] Further, step S5 includes:
[0061] Determine whether the termination state is reached based on the end value d obtained by the interaction between the drone and the environment;
[0062] When the terminal state is reached, the reward is used to determine whether it is a successful terminal state or a failed terminal state: if the reward is greater than 0, it is considered a success; if the reward is not greater than 0, it is considered a failure.
[0063] Based on the above technical solution, the present invention has the following beneficial effects:
[0064] 1. In response to the underestimation problem in the TD3 algorithm, the network structure of the TD3 algorithm was adjusted, two strategy networks were adopted, and the uncertainty brought by the dual value network was alleviated by regularizing the value network.
[0065] 2. A multi-experience pool mechanism is introduced to store the data generated by the interaction between the drone and the environment into the success experience pool and the failure experience pool respectively. Compared with the traditional single experience pool, this mechanism can improve the quality of sampled data, thereby improving the efficiency of autonomous navigation of drones.
[0066] 3. The overall model of the DAMEP-TD3 algorithm proposed in the present invention can have a better navigation success rate, and can avoid obstacles well during the navigation process and shorten the navigation path. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 Flowchart of the present invention;
[0068] Figure 2 Schematic diagram of the drone's own information and target point information in the experiment of the present invention;
[0069] Figure 3 The horizontal laser distance sensor of the UAV in the experiment of the present invention;
[0070] Figure 4 The vertical laser distance sensor of the UAV in the experiment of the present invention;
[0071] Figure 5 Schematic diagram of the motion space of the drone in the experiment of the present invention;
[0072] Figure 6 Comparison of reward values between the present invention and the TD3 algorithm during training;
[0073] Figure 7 Data comparison between the present invention and the TD3 algorithm in test scenarios. DETAILED DESCRIPTION
[0074] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0075] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.
[0076] Please refer to Figures 1 to 7 , gives the specific implementation of this embodiment:
[0077] A UAV autonomous navigation method based on the improved TD3 algorithm. The specific process is as follows: Figure 1 As shown, including:
[0078] S1. Establish a UAV autonomous navigation model based on the improved TD3 algorithm, describe the UAV autonomous navigation problem as a Markov decision process, and design an improved TD3 algorithm with two strategy networks and multiple experience pools: DAMEP-TD3 algorithm; define the state space, action space, reward, strategy network, value network, success experience pool and failure experience pool of the DAMEP-TD3 algorithm;
[0079] More specifically, the state space in this embodiment includes the drone's own information, the location of the target point, and information about obstacles, etc. The state space is a set of variables used to describe the current environment in reinforcement learning. A reasonable state space design can reduce the complexity of the problem, accelerate the learning process, and improve the performance of the agent.
[0080] As a preferred embodiment, Figure 2 As shown in the figure, the drone's own information and target point information in this embodiment are schematically shown. The drone's own information and target point information are given by GPS and gyroscope, and the fusion state information of the two is recorded as s p , the formula is:
[0081] s p =[x g -x,y g -y,z g -z,v x ,v y ,v z ,β,dis];
[0082] Among them, x, y, z are the current positions of the drone, x g ,y g ,z g is the position of the target point, dis is the straight-line distance between the drone and the target point, β is the angle between the line connecting the drone and the target point and the direction of the drone's first-person perspective, and v x ,v y ,v z represents the speed of the drone in three dimensions, x g -x,y g -y,z g -z indicates the relative position of the target point to the drone;
[0083] like Figure 3 , Figure 4 As shown in the figure, obstacle information is detected by laser distance sensors on the horizontal and vertical surfaces of the drone. The angle range of the sensor scan is represented by π, and the angle between each two laser beams is l1 to l7 represent the ray length of the sensor in the horizontal plane, l8 to l 14 is the ray length of the vertical plane, where l4 and l 11 is the same laser beam. If the sensor does not detect an obstacle in the specified direction, the length of the ray is the maximum detectable distance, otherwise the length is the distance between the drone and the obstacle; the obstacle information is recorded as s e , its formula is expressed as:
[0084] s e =[ρ i ,l i ], i = 1...14,;
[0085] Among them, ρ i is a one-hot encoding, with a value of 1 if an obstacle is detected and 0 if no obstacle is detected. Combining the drone’s own information, target point information, and obstacle information, the state of the environment given to the drone can be defined as s = [s p ,s e ].
[0086] As a preferred implementation mode, the action space in step S1 is further defined. In this embodiment, the drone used is a quad-rotor drone, which is an aircraft composed of four rotors. During the flight, the quad-rotor drone controls the attitude and flight direction by adjusting the rotation speed of the rotors. This experiment uses the forces in all directions of the drone as executable actions to achieve forward, backward, left, right, up and down movement, and at the same time controls the steering of the drone by the rotation angle, such as Figure 5 As shown:
[0087] 500.a Forward , 300.a Right , 500.a Up They represent the forces acting on the X-axis, Y-axis, and Z-axis of the drone. In this embodiment, the values of the forces are 500, 300, and 500, respectively. new =α old -2a Rotation Represents the change of the yaw angle of the drone around the Z axis, and the formula for the obtained action a is expressed as:
[0088] a=[a Forward ,a Right ,a Up ,a Rotation ];
[0089] Among them, a Forward 、a Right 、a Up They respectively represent the forces received by the drone in the front, back, left, right, and up and down directions, reflecting the front, back, left, right, and up and down movements of the drone; a RotationIndicates the steering force and reflects the change in the yaw angle of the drone.
[0090] As a preferred implementation, in step S1 of this embodiment, different rewards are also given to the action decisions of the drone in different states to avoid sparse rewards. The larger the value of the total reward, the higher the efficiency and safety of the drone action decision. The rewards mainly include:
[0091] Distance Reward dis ; To evaluate whether the drone has reached the target point, the straight-line distance between the drone and the target point is used as the reward standard. The farther the distance, the smaller the reward. The formula is expressed as:
[0092]
[0093] Among them, dis represents the straight-line distance between the drone and the target point, d is the set maximum reward distance, q1 is the distance reward constant, and p1 is the distance reward factor;
[0094] In this embodiment, q1=5, p1=0.1;
[0095] Obstacle Rewards obs ; Evaluate whether the drone successfully avoids obstacles and sets a safe distance l safe The closer the distance between the drone and the obstacle is to the safety distance l safe , the smaller the reward, the formula is expressed as:
[0096]
[0097] Among them, l min represents the minimum distance between the drone and the obstacle, p2 is the collision reward factor, and q2 is the collision reward constant;
[0098] In this embodiment, q2=5, p2=0.1;
[0099] Angle Reward r angle ; To evaluate whether the drone is flying in the direction of the target point, the angle β between the line connecting the drone and the target point and the direction of the drone's first-person perspective is used as the standard. The larger the angle β, the smaller the reward. The formula is expressed as:
[0100]
[0101] Among them, p3 is the angle reward factor;
[0102] In this embodiment, p3=0.05;
[0103] Time step reward r step ; To evaluate how fast the drone reaches the target point, the more time steps required, the smaller the reward. The formula is:
[0104] r step = -p4·n;
[0105] Among them, p4 is the time step reward factor, and n is the number of time steps;
[0106] In this embodiment, p4=0.02;
[0107] Add up the four rewards to get the total reward r of the drone, r = r dis +r obs +r angle +r step .
[0108] In addition, the DAMEP-TD3 algorithm in this embodiment also includes two strategy networks: 2 Value Networks And the corresponding target network
[0109] S2. After the preparation in step S1 above, the training task of the overall model is started; first, the state of the drone is initialized, and a temporary experience pool is defined, and the size of the temporary experience pool is set to 200;
[0110] S3. Collect data obtained during the interaction between the drone and the environment according to the policy network, and add the data to the temporary experience pool. When the temporary experience pool data overflows, if the drone still has not reached the terminal state (1. successfully reached the destination, 2. collided with an obstacle), it is also considered that the past decision has a positive impact on the present, so the overflowed data is added to the successful experience pool, and then the update condition is judged; if the temporary experience pool data does not overflow, the update condition is directly judged;
[0111] S4. Determine whether the update condition is met, based on whether the successful experience pool is full;
[0112] When the update condition is met, random sampling is performed from the successful experience pool and the failed experience pool in proportion. The batch size of the sampled data is defined as N. In this embodiment, N is set to 128. When the update condition is met, 108 data are taken from the successful experience pool and 20 data are taken from the failed experience pool for update. The sampled data is used to update the value network and the strategy network.
[0113] If the update conditions are not met, the termination status is determined;
[0114] As a preferred embodiment, step S4 includes:
[0115] S41. When the update condition is met, random sampling is performed from the success experience pool and the failure experience pool in proportion, and the obtained sampling data is defined as (s, a, s′, r, d), where s represents the current state, a represents the action, s′ represents the state at the next moment, r represents the reward, and d represents the end value;
[0116] S42, using the sampled data (s, a, s′, r, d), combined with the target network Update value network and policy network
[0117] More specifically, in this embodiment, the minimum loss function is used to root the value network, and the strategy operator is used to update the strategy network. The specific process is:
[0118] S421. Update the value network with the minimum loss function Loss The formula of the minimum loss function Loss is expressed as:
[0119]
[0120] Where N is the batch size of the sampled data, λ is the regularization parameter, and y is the time-series difference target value, which combines the immediate reward and the value estimate of the future state to provide a target value for calculating the error of the current value estimate. The formula is expressed as:
[0121] y = r + γ (1-d) V (s');
[0122] Among them, γ is the discount factor, which indicates the value decay of future rewards and reflects the importance attached to future rewards; V(s′) is the estimated value of the value function, which is expressed as follows:
[0123] V(s′)=ν·min{Q1(s′,a′),Q2(s′,a″)}+(1-ν)·max{Q1(s′,a′),Q2(s′,a″)};
[0124] Among them, ν∈[0,1), is the parameter of the balanced target value network, It represents the minimum value of the output of the two target value networks under the state s′ and action a′ at the next moment. It represents the minimum value of the values output by the two target value networks under the state s′ and action a′′ at the next moment. There is an underestimation problem in the traditional TD3 algorithm. The present invention can alleviate the underestimation bias of the value output by the target value network by designing the above value function estimation value.
[0125] In addition, in the above formula, a′ and a″ respectively represent the state s′ at the next moment according to the target strategy network and The action at the next moment obtained under noise ∈ is expressed as:
[0126]
[0127] S422, using policy gradient to update the policy network The formula is:
[0128]
[0129] Among them, J(φ i ) represents the objective function of the strategy, which is used to evaluate the expected return of the strategy in the environment. Indicates that action a is equal to the policy network The action output in state s hour, The gradient of action a, Indicates the value of action a in state s. Represents the parameter gradient of the policy network at state s, which is used to update the parameter φ of the policy network i .
[0130] S5. Determine whether the drone has reached the termination state:
[0131] If the termination state is not reached, step S3 is repeated;
[0132] When the termination state is reached, the reward value is used to determine whether it is a successful termination state or a failed termination state; if it is a successful termination state, the temporary experience pool is cleared and all its data is put into the successful experience pool; if it is a failed termination state, the temporary experience pool is cleared and all its data is put into the failed experience pool;
[0133] More specifically, whether the termination state is reached is determined based on the end value d obtained by the interaction between the drone and the environment;
[0134] When the terminal state is reached, the reward is used to determine whether it is a successful terminal state or a failed terminal state: if the reward is greater than 0, it is considered a success; if the reward is not greater than 0, it is considered a failure.
[0135] S6. After clearing the temporary experience pool, return to step S2 and repeat steps S2-S5 for multiple iterations to verify the effect of the overall model of the improved TD3 algorithm.
[0136] Through the above specific process, this embodiment obtains the following Figure 6 , Figure 7 The actual effect is shown.
[0137] Figure 6This is a comparison chart of the reward values of the DAMEP-TD3 algorithm and the traditional TD3 algorithm during the training process. This embodiment builds a training scene based on Unity 3D, sets the drone's mission space to a rectangular area of 400×400×100, and the drone's starting point is randomly initialized on one side of the area, while the target point is randomly initialized on the other side. Obstacles of different shapes are randomly generated in the middle area, including 5 cubes of 30×30×100 and 5 cylinders of 15×100. The target point is simplified to a sphere with a diameter of 10, and the flight altitude of the drone is limited to between 20 and 100. The number of iterations during the training process is 5000, and the maximum step size is set to 500. There are three termination conditions during the training process, namely: (1) the drone reaches the target point; (2) the drone collides with the obstacle; (3) the maximum step size is reached. Each termination is recorded as the end of an iteration. The results after multiple iterations can be seen from the figure. Compared with TD3, the DAMEP-TD3 algorithm converges faster and has a smoother convergence process.
[0138] Figure 7 The comparison results of the two algorithms in different test scenarios are shown in Figure 1, where environment 1 represents the training environment, including 10 obstacles; environment 2 changes the obstacle shape, 30×30×100 becomes 100×30×30; environment 3 increases the number of obstacles, with a total of 12 obstacles; environment 4 changes the obstacle shape (30×30×100 becomes 100×30×30) and increases the number (12 obstacles). Four evaluation indicators are used to evaluate the performance of UAV autonomous navigation, including success rate (the ratio of successfully reaching the target point in 2000 iterations), collision rate (the ratio of collisions between the UAV and obstacles), loss rate (the ratio of the UAV that neither collides with obstacles nor reaches the target point during the test), and path length (the average path length of the UAV that successfully navigates to the target point).
[0139] from Figure 7 It can be seen that in all environments, the effect of the DAMEP-TD3 algorithm is better than that of TD3, with a higher success rate, lower collision rate and loss rate, and a shorter path length. And when the environment changes, the change in the success rate of the DAMEP-TD3 algorithm is less than that of TD3. For example, in environment 1 and environment 4, the shape and number of obstacles in environment 4 have changed compared with environment 1. The success rate of the TD3 algorithm has decreased by 11% from 92.2% to 81.2%, and the success rate of DAMEP-TD3 has decreased by 8.15% from 95.75% to 87.6%. It can be concluded that the generalization ability of DAMEP-TD3 is better than that of TD3, and it can better adapt to complex environments. Combined with Figure 6 and Figure 7 It can be concluded that the improved algorithm has significantly better performance than TD3.
[0140] In summary, compared with the traditional TD3 algorithm, the DAMEP-TD3 algorithm proposed in the present invention alleviates the uncertainty brought by the dual value network through two strategy networks, and combines multiple experience pools to improve the quality of collected data, so that the algorithm model proposed in the present invention has a faster convergence speed and a smoother convergence process. When facing complex environments, it also has better generalization ability. The performance of the overall model has obvious advantages, and it has a better navigation success rate while being able to avoid obstacles and shorten the path length.
[0141] In this specification, the terms "one embodiment", "some embodiments", "examples", "specific examples" or "some examples" refer to at least one embodiment or example described in combination with specific features, structures, materials or characteristics. These described specific features, structures, materials or characteristics can be combined in an appropriate manner in one or more embodiments or examples. In addition, the technician can combine and combine different embodiments or examples and their features described in this specification without contradiction.
[0142] The logic and / or steps shown in the flowchart or described in other ways can be regarded as a sequence of executable instructions for implementing logical functions. These instructions can be implemented in any computer-readable medium for use by instruction execution systems, devices or equipment. These systems, devices or equipment include processor systems or other systems capable of receiving and executing instructions.
[0143] The above embodiments have described the principles and implementations of the present invention in detail, and have used specific examples to illustrate the working principles thereof. These examples are only used to help understand the method of the present invention and its core concept. Meanwhile, according to the concept of the present invention, the actual implementation methods and application scope may vary. Therefore, the contents of this specification should not be construed as limiting the present invention.
Claims
1. The autonomous navigation method of UAV based on improved TD3 algorithm is characterized by: include: S1. Establish a UAV autonomous navigation model based on the improved TD3 algorithm, describe the UAV autonomous navigation problem as a Markov decision process, and design an improved TD3 algorithm with two strategy networks and multiple experience pools: DAMEP-TD3 algorithm; define the state space, action space, reward, strategy network, value network, success experience pool and failure experience pool of the DAMEP-TD3 algorithm; S2, initialize the state of the drone and define a temporary experience pool; S3. Collect data obtained during the interaction between the drone and the environment according to the strategy network, and add the data to the temporary experience pool. If the temporary experience pool overflows, add the overflowed data to the success experience pool, and then judge the update conditions; if the temporary experience pool does not overflow, directly judge the update conditions; S4. Determine whether the update condition is met: When the update conditions are met, random sampling is performed from the successful experience pool and the failed experience pool in proportion, and the sampled data is used to update the value network and the strategy network; If the update conditions are not met, the termination status is determined; S5. Determine whether the drone has reached the termination state: If the termination state is not reached, step S3 is repeated; When the termination state is reached, the reward value is used to determine whether it is a successful termination state or a failed termination state; if it is a successful termination state, the temporary experience pool is cleared and all its data is put into the successful experience pool; if it is a failed termination state, the temporary experience pool is cleared and all its data is put into the failed experience pool; S6. After clearing the temporary experience pool, return to step S2 and repeat steps S2-S5 for multiple iterations to verify the effect of the overall model of the improved TD3 algorithm.
2. The unmanned aerial vehicle autonomous navigation method based on the improved TD3 algorithm according to claim 1 is characterized in that: In step S1: The state space of the DAMEP-TD3 algorithm stores the state s, which specifically includes: drone information, target point information, and obstacle information; The drone’s own information and target point information are given by GPS and gyroscope, and the fusion state information of the two is recorded as s p , the formula is: s p [x g -x,y g -y,z g -z,v x ,v y ,v z ,β,dis]: Among them, x, y, z are the current positions of the drone, x g ,y g ,z g is the position of the target point, dis is the straight-line distance between the drone and the target point, β is the angle between the line connecting the drone and the target point and the direction of the drone's first-person perspective, and v x ,v y ,v z represents the speed of the drone in three dimensions, x g -x,y g -y,z g -z indicates the relative position of the target point to the drone; Obstacle information is detected by laser distance sensors on the horizontal and vertical surfaces of the drone. The obstacle information is recorded as s e , its formula is expressed as: s e =[ρ i ,l i ]; Among them, ρ i It is a one-hot encoding, the value is 1 if an obstacle is detected, and 0 if no obstacle is detected; l i is the length of the laser beam. When an obstacle is detected, its value is the distance between the drone and the obstacle. When an obstacle is detected, its value is the maximum detectable distance. Thus, the state s=[s p ,s e ].
3. The unmanned aerial vehicle autonomous navigation method based on the improved TD3 algorithm according to claim 1, characterized in that: In step S1: The action space of the DAMEP-TD3 algorithm stores the action a, which is represented by the forces in all directions of the drone, and the formula is: a=[a Forward ,a Right ,a Up ,a Rotation ]; Among them, a Forward 、a Right 、a Up They respectively represent the forces received by the drone in the front, back, left, right, and up and down directions, reflecting the front, back, left, right, and up and down movements of the drone; a Rotation Indicates the steering force and reflects the change in the yaw angle of the drone.
4. The unmanned aerial vehicle autonomous navigation method based on the improved TD3 algorithm according to claim 1 is characterized in that: In step S1, different rewards are given to the action decisions of the drone in different states to avoid sparse rewards. The larger the total reward value, the higher the efficiency and safety of the drone action decision. The rewards mainly include: Distance Reward dis ; To evaluate whether the drone has reached the target point, the straight-line distance between the drone and the target point is used as the reward standard. The farther the distance, the smaller the reward. The formula is expressed as: Among them, dis represents the straight-line distance between the drone and the target point, d is the set maximum reward distance, q1 is the distance reward constant, and p1 is the distance reward factor; Obstacle Rewards obs ; Evaluate whether the drone successfully avoids obstacles and sets a safe distance l safe The closer the distance between the drone and the obstacle is to the safety distance l safe , the smaller the reward, the formula is expressed as: Among them, l min represents the minimum distance between the drone and the obstacle, p2 is the collision reward factor, and q2 is the collision reward constant; Angle Reward r angle ; To evaluate whether the drone is flying in the direction of the target point, the angle β between the line connecting the drone and the target point and the direction of the drone's first-person perspective is used as the standard. The larger the angle β, the smaller the reward. The formula is expressed as: Among them, p3 is the angle reward factor; Time step reward r step ; To evaluate how fast the drone reaches the target point, the more time steps required, the smaller the reward. The formula is: r step =-p4·n; Among them, p4 is the time step reward factor, and n is the number of time steps; Add up the four rewards to get the total reward r of the drone, r = r dis +r obs +r angle +r step .
5. The unmanned aerial vehicle autonomous navigation method based on the improved TD3 algorithm according to claim 1 is characterized in that: The DAMEP-TD3 algorithm also includes two strategy networks 2 Value Networks And the corresponding target network 6. The unmanned aerial vehicle autonomous navigation method based on the improved TD3 algorithm according to claim 5 is characterized in that: Step S4 includes: S41. When the update condition is met, random sampling is performed from the success experience pool and the failure experience pool in proportion, and the obtained sampling data is defined as (s, a, s′, r, d), where s represents the current state, a represents the action, s′ represents the state at the next moment, r represents the reward, and d represents the end value; S42, using the sampled data (s,a,s′,r,d), the target network Update value network Q θ1 , Q θ2 and policy network 7. The unmanned aerial vehicle autonomous navigation method based on the improved TD3 algorithm according to claim 6 is characterized in that: Step S42 specifically includes the following steps: S421. Update the value network with the minimum loss function Loss The formula of the minimum loss function Loss is expressed as: Where N is the batch size of the sampled data, λ is the regularization parameter, and y is the time series difference target value, which is used to calculate the error of the current value estimate. The formula is expressed as: y = r + γ (1-d) V (s'); Among them, γ is the discount factor, which indicates the value decay of future rewards; V(s′) is the estimated value of the value function, which is expressed as: V(s′)=ν·min{Q1(s′,a′),Q2(s′,a″)}+(1-ν)·max{Q1(s′,a′),Q2(s′,a″)}; Among them, ν∈[0,1), is the parameter of the balanced target value network, It represents the minimum value of the output of the two target value networks under the state s′ and action a′ at the next moment. It represents the minimum value of the output of the two target value networks under the state s′ and action a′′ at the next moment; Among them, a′ and a″ respectively represent the state s′ at the next moment according to the target strategy network and The action at the next moment obtained under noise ∈ is expressed as: S422, using policy gradient to update the policy network The formula is: Among them, J(φ i ) represents the objective function of the strategy, which is used to evaluate the expected return of the strategy in the environment. Indicates that action a is equal to the policy network The action output in state s hour, The gradient of action a, represents the value of action a in state s. Represents the parameter gradient of the policy network at state s, which is used to update the parameter φ of the policy network i .
8. The unmanned aerial vehicle autonomous navigation method based on the improved TD3 algorithm according to claim 1, characterized in that: In step S5: Determine whether the termination state is reached based on the end value d obtained by the interaction between the drone and the environment; When the terminal state is reached, the reward is used to determine whether it is a successful terminal state or a failed terminal state: if the reward is greater than 0, it is considered a success; if the reward is not greater than 0, it is considered a failure.
Citation Information
Patent Citations
Real-time obstacle detection and positioning method based on monocular vision
CN108909624A
Obstacle avoidance method for unmanned aerial vehicle flying in indoor coal yard
CN112783194A
Urban road automatic parking method and system based on reinforcement learning
CN114454875A
Potential field and deep reinforcement learning fused unmanned aerial vehicle path planning method
CN114518770A
Improved QMIX method applied to unmanned aerial vehicle cooperative confrontation
CN116430898A
Cited By
Fighter bottom layer flight control method based on deep reinforcement learning
CN120233791A
