Unmanned aerial vehicle dynamic path planning algorithm for enhanced near-end strategy optimization

Abstract

The invention discloses an unmanned aerial vehicle dynamic path planning algorithm based on enhanced near-end strategy optimization, and belongs to the technical field of unmanned aerial vehicle dynamic path planning. The method aims at solving core challenges such as sample scarcity and insufficient algorithm convergence stability in unmanned aerial vehicle dynamic path planning, a priority experience replay strategy with correction is adopted, meanwhile, a network structure of a PPO algorithm is improved, and a double-Critic network architecture is introduced to relieve a Q value overestimation phenomenon; the method can be applied to the technical fields of unmanned aerial vehicle path planning, robot path planning and the like, and has important engineering application value.

Description

Technical Field

[0001] This invention relates to the field of UAV path planning, and more specifically to a UAV dynamic path planning algorithm with enhanced near-end strategy optimization. Background Technology

[0002] Unmanned Aerial Vehicle Dynamic Path Planning (UAV-DPP) is a complex optimization problem that, during the mission execution of unmanned aerial vehicles (UAVs) or unmanned combat aerial vehicles (UCAVs), addresses dynamically changing environmental characteristics, real-time updated mission requirements, and uncertain constraints. It utilizes intelligent decision-making and optimization algorithms to generate, adjust, or reconstruct the optimal flight path from the starting point to the target point in real time. Its core objective is to achieve global or local optimization of the path across multiple objective dimensions, including time efficiency, energy cost, and stealth, while satisfying UAV kinematic constraints, mission constraints, and safety constraints. Simultaneously, it aims to possess rapid response capabilities to sudden environmental changes and mission adjustments.

[0003] Traditional path planning algorithms include Dynamic Window Approach (DWA), Artificial Potential Field (APF), Rapid Exploration Random Number (RRT), and Neural Networks. The Dynamic Window Approach is a typical online path planning algorithm that generates feasible trajectories under dynamic constraints for obstacle avoidance. During execution, DWA dynamically calculates a feasible speed window based on the current state of the UAV, constraining the speed range the UAV can reach. Subsequently, the flight trajectory at different speeds is evaluated using an evaluation function, and the optimal solution is selected based on the score. This allows the UAV to choose an optimal path to reach the target location while avoiding obstacles. However, in complex dynamic environments, the distribution of obstacles is quite complex, and the DWA algorithm uses a fixed-weight evaluation function, which is insensitive to environmental changes, resulting in poor performance in practice. The Artificial Potential Field Method (APF) is a commonly used online path planning algorithm inspired by the potential field theory in physics. It assumes that the entire UAV and its environment are a virtual potential field, and uses the attractive and repulsive forces fed back by the potential field for obstacle avoidance and path planning. However, in the dynamic path planning scenario of UAVs, its core framework based on static potential field modeling is fundamentally contradictory to the time-varying and uncertain nature of the dynamic environment, leading to many significant shortcomings in practical applications. The Rapidly Exploring RandomTree algorithm... RRT (Probabilistic Path Finding) is an incremental path search algorithm based on probabilistic sampling. Essentially, it achieves efficient exploration of unknown or complex environments by randomly sampling environmental states and progressively constructing a tree-like search structure, ultimately finding a feasible path from the starting point (root node) to the target point (target area). It possesses the core advantages of probabilistic completeness and applicability to high-dimensional non-convex spaces. However, limited by its core framework of "random sampling - greedy expansion," it suffers from drawbacks in practical UAV path planning scenarios, such as insufficient path optimality and smoothness due to randomness, and search redundancy. The rapid development of neural networks has demonstrated their advantages in UAV path planning. Their powerful learning capabilities and efficient processing of complex data patterns enable them to learn from environmental data and flexibly adjust strategies based on real-time environmental changes. However, neural networks typically require a large amount of training data to exhibit good performance. In such cases, the challenge of obtaining a large number of high-quality training samples may arise. Furthermore, neural networks are susceptible to extreme values ​​during training, leading to performance instability.

[0004] Reinforcement learning (RL) utilizes the interaction between an agent and its environment, learning and optimizing based on the reward signals it receives, ultimately maximizing the reward value. Through continuous trial and error during the learning process, it continuously improves and refines itself, as seen in object detection and continuous Markov decision problems. Pei et al. proposed an improved Dyna-Q algorithm, incorporating heuristic search and simulated annealing methods, and introduced a novel action selection strategy to address the problem of insufficient exploration and mining, improving the efficiency of path planning. Yang et al. proposed an N-step priority dual-depth Q-network path planning algorithm, introducing a screening mechanism to accumulate experience and enhance the utilization rate of effective samples, thereby improving algorithm stability and obstacle avoidance capabilities. Chen et al. proposed an innovative path planning algorithm with a proximal optimization strategy to solve obstacle avoidance problems in dynamic environments. With the development of artificial intelligence technology, more and more researchers are applying reinforcement learning techniques to the field of UAV dynamic path planning.

[0005] In drone motion planning experiments, each takeoff scenario requires extensive pre-deployment and application work, making it difficult to generate large amounts of data. Furthermore, the repetitive nature of most experimental scenarios and the selection of high-quality data present significant challenges. Therefore, subsequent processes need to improve the algorithm's sampling efficiency and data utilization efficiency. Schaul [9] First, Prioritized Experience Replay (PER) was proposed, which allows agents to replay experiences based on importance and accelerates algorithm convergence. Various DRL algorithms combined with PER have evolved into PER-DDPG, PER-TD 3, PER-PPO, etc. However, existing prioritized experience replay mechanisms often focus on high-reward samples, easily neglecting low-reward samples with potential value, leading to one-sided sample utilization. Furthermore, the single-critic network structure used in the standard PPO algorithm suffers from overestimation of action value, easily causing the policy to converge to a suboptimal solution.

[0006] Therefore, there is an urgent need for a dynamic path planning algorithm for drones that can solve the above problems. Summary of the Invention

[0007] In view of this, this application provides an enhanced near-end strategy optimization algorithm for UAV dynamic path planning, the specific technology of which is as follows: An enhanced near-end strategy optimization algorithm for UAV dynamic path planning, comprising the following steps: Step S1: Initialize the learning rate, maximum number of rounds, maximum number of iterations, priority memory, number of resampling attempts, number of delayed policy updates, initialize the batch size, and initialize the current environment to obtain the environment state; Step S2: Select actions and action probabilities using a normal distribution; Step S3: Obtain the state at the next moment, the reward in the current state, and whether the round count is complete; Step S4: Calculate the cumulative reward discount based on the reward function; Step S5: Reset the environment and learning rate, and update the status information of the drone and obstacles; Step S6: Calculate the preferred empirical sampling range; Step S7: Segment the sample batches and calculate the sample priority; Step S8: Update value network 1 and value network 2. Update the policy network once after every m value network updates. Step S9: Resample the b groups of data samples according to the priority of the samples and calculate the importance weights; Step S10: Update value network 1 and value network 2. Update the policy network once after every m value network updates. Step S11: Output the optimal solution.

[0008] Preferably, the state space includes UAV state information, obstacle state information, and target information; therefore, the state space is represented as: ; Here, represents the drone's status information, represents the obstacle's status information, and represents the obstacle's status information. The obstacle's status information is obtained in real time through the drone's sensors and changes continuously based on the current drone status information.

[0009] Preferably, the reward function provides a corresponding reward based on the drone's obstacle avoidance performance and the situation where it gets closer to the target position without a collision. The specific formula is as follows: ; ; .

[0010] The preferred, corrected priority experience replay strategy focuses on selecting samples with high absolute values ​​in the early stages of the algorithm. However, in the later stages, the algorithm's focus shifts to its development capabilities, requiring faster convergence to maximize gains and reach the optimal solution. The loss function of the value network is corrected using importance sampling weights, as shown in the following formula: .

[0011] Preferably, two independent Critic networks are deployed, and the minimum of their respective action value estimates is taken as the final value estimate to suppress the overestimation bias of a single network; the loss functions of the two Critic networks are as follows: .

[0012] Preferably, the method is applicable to the dynamic path planning problem of UAVs in complex scenarios, and has good performance in terms of execution speed and planned path.

[0013] Compared with the prior art, the present invention has the following beneficial effects. This invention proposes a corrected priority experience replay algorithm. During priority experience replay, samples with high absolute reward values ​​are selected to improve sample utilization and enhance exploratory power. Replaying samples with high positive reward values ​​in the later stages of the algorithm improves convergence speed. The loss function of the value network is corrected using importance sampling weights. The network structure of the PPO algorithm is improved by introducing a dual-Critic network architecture to alleviate Q-value overestimation and constructing an Actor-Critic network structure with a shared underlying feature extraction layer to improve sample utilization efficiency. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the UAV path planning described in this invention; Figure 2 This is a schematic diagram of the climbing angle and turning angle of the UAV in this invention; Figure 3 This is an example of dynamic path planning for unmanned aerial vehicles in this invention. Detailed Implementation

[0016] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0017] Example The flowchart of the multimodal target detection method of the present invention is as follows: Figure 1 As shown, the specific steps include the following: Step 1: Initialize network parameters, including shared feature extraction layer, Actor network layer, dual Critic layer, value correction sub-network layer; experience pool and hyperparameter initialization.

[0018] Step 2: The UAV collects the state at time t in real time in a dynamic environment. The system acquires information including its own position coordinates, velocity, and heading angle; obtains information about dynamic obstacles, including their position, velocity, and center point coordinates; and obtains the location information of the target area and map boundary information. After normalizing the features, the data is input into the shared feature extraction layer.

[0019] Step 3: Processing the shared feature layer Output shared feature vector The Actor network layer is based on Output action distribution, and sample from the action distribution. State transition is performed based on the action. The reward function assigns a reward based on the drone's obstacle avoidance performance and the scenario where it gets closer to the target without a collision. The specific formula is shown below:

[0020]

[0021]

[0022] The transferred tuple is then stored in the experience pool of the corresponding reward segment.

[0023] Step 4: Next, the sample priority is calculated. For newly added samples, the combination priority is calculated, starting with the correction coefficient. Then, the TD error is calculated using the following formula:

[0024] The final formula for calculating the combination priority is as follows:

[0025] Step 5: Gather all batch samples Input a shared feature layer, output the corresponding shared feature vector. Subsequently, the dual-Critic network was based on Output And calculate the corrected value estimate. Finally, the GAE dominance function is calculated.

[0026] Step 6: Two independent Critic networks output action value estimates, and the minimum of these is taken as the final value estimate to suppress overestimation bias from a single network. The loss functions for the two Critic networks are as follows:

[0027] Backpropagation updates using the Adam optimizer: Critic parameters are updated independently, while shared layer parameters are updated jointly by the dual Critic loss. Step 7: Actor network update, calculation The Adam optimizer updates the top-level parameters of the Actor and the parameters of the shared feature layer using the gradients of the Actor parameters and the gradients of the parameters of the shared feature layer.

[0028] Step 8: Once the convergence condition is met or the total number of training steps is reached, stop training, save the final network parameters, and output the optimal solution.

[0029] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An enhanced near-end strategy optimization algorithm for UAV dynamic path planning, characterized in that: The method includes the following steps: Step S1: Initialize the learning rate, maximum number of rounds, maximum number of iterations, priority memory, number of resampling attempts, number of delayed policy updates, initialize the batch size, and initialize the current environment to obtain the environment state; Step S2: Select actions and action probabilities using a normal distribution; Step S3: Obtain the state at the next moment, the reward in the current state, and whether the round count is complete; Step S4: Calculate the cumulative reward discount based on the reward function; Step S5: Reset the environment and learning rate, and update the status information of the drone and obstacles; Step S6: Calculate the preferred empirical sampling range; Step S7: Segment the sample batches and calculate the sample priority; Step S8: Update value network 1 and value network 2. Update the policy network once after every m value network updates. Step S9: Resample the b groups of data samples according to the priority of the samples and calculate the importance weights; Step S10: Update value network 1 and value network 2. Update the policy network once after every m value network updates. Step S11: Output the optimal solution.

2. The enhanced near-end strategy optimization UAV dynamic path planning algorithm according to claim 1, characterized in that: The state space includes UAV state information, obstacle state information, and target information; therefore, the state space is represented as: ； Here, represents the drone's status information, represents the obstacle's status information, and represents the obstacle's status information. The obstacle's status information is obtained in real time through the drone's sensors and changes continuously based on the current drone status information.

3. The enhanced near-end strategy optimization UAV dynamic path planning algorithm according to claim 1, characterized in that: The reward function assigns a reward based on the drone's obstacle avoidance performance and the scenario where it gets closer to the target without a collision. The specific formula is shown below: ； ； 。 4. The enhanced near-end strategy optimization UAV dynamic path planning algorithm according to claim 1, characterized in that: The corrected priority experience replay strategy focuses on selecting samples with high absolute values ​​in the early stages of the algorithm. However, in the later stages, the focus shifts to improving the algorithm's performance, requiring faster convergence to obtain greater gains and reach the optimal solution. The loss function of the value network is corrected using importance sampling weights, as shown in the following formula: 。 5. The enhanced near-end strategy optimization UAV dynamic path planning algorithm according to claim 1, characterized in that: Two independent Critic networks are deployed, and the minimum of their respective action value estimates is taken as the final value estimate to suppress the overestimation bias of a single network. The loss functions of the two Critic networks are as follows: .。 6. The enhanced near-end strategy optimization UAV dynamic path planning algorithm according to claim 1, characterized in that: The proposed method is applicable to the dynamic path planning problem of UAVs in complex scenarios, and has good performance in terms of execution speed and planned path.

Images