A path planning method, system and product based on mixed experience replay

Abstract

This invention discloses a trajectory planning method, system, and product based on hybrid experience replay, relating to the field of unmanned aerial vehicle (UAV) technology. The method includes: establishing a three-dimensional UAV dynamic model; constructing a heuristic non-sparse reward function to address the difficulties in modeling complex environments and the slow convergence of sparse rewards; proposing the TD3-MER algorithm, introducing a dual buffer pool and dynamic priority mechanism, and employing a proportional hybrid sampling strategy to balance high-value sample learning with data diversity; combining target action noise smoothing and delayed update strategies to improve training stability; freezing the main Actor network and removing redundant modules after training convergence to achieve lightweight airborne deployment. This significantly improves sample utilization and convergence speed, enhancing the UAV's generalization ability and obstacle avoidance success rate in dense obstacles and dynamic scenarios.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a trajectory planning method, system, and product based on hybrid experience playback. Background Technology

[0002] With the rapid development of aerospace technology, sensor technology, and artificial intelligence, unmanned aerial vehicles (UAVs) are widely used in civilian and industrial scenarios, such as logistics delivery, disaster relief, environmental monitoring, power line inspection, and agricultural plant protection. In these applications, UAVs often need to fly at low altitudes in complex urban or wilderness environments, facing interference from buildings, trees, and dynamic obstacles (such as other aircraft or birds). However, most current UAV trajectory planning studies still restrict the movement of UAVs to a two-dimensional or quasi-two-dimensional plane, assuming a fixed flight altitude or a flat environment. While this simplified model reduces computational complexity, in the real world, obstacles vary in height and are unevenly distributed. The assumption of considering only two-dimensional plane movement greatly limits the maneuverability, autonomy, and environmental adaptability of UAVs, making it difficult to meet the flight requirements in complex three-dimensional spaces.

[0003] In existing technologies, UAV trajectory planning in complex 3D scenes mainly focuses on the following two aspects: First, the sparse reward leads to convergence difficulties. In a vast three-dimensional space, a drone needs a long time to explore in order to reach the target point and obtain a positive reward. In complex environments with dense obstacles, if traditional sparse rewards (feedback only upon arrival or collision) are used, the drone will find it difficult to capture effective learning signals in the massive state space, resulting in blind exploration in the early stages of training, extremely slow convergence, or even failure to converge.

[0004] Second, the inefficiency of the experience replay mechanism. The standard TD3 algorithm employs a uniform random sampling experience replay mechanism. However, during drone flight, most samples are "ordinary samples" of smooth flight, while "critical samples" of collisions or successful obstacle avoidance are extremely rare. Uniform sampling results in a very low probability of extracting these high-value sparse samples, making it difficult for the model to learn quickly from critical events. Although the existing Prioritized Experience Replay (PER) can address this issue, relying solely on high-priority samples can easily alter the state distribution, leading to model overfitting and reduced generalization ability in unseen scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a trajectory planning method, system, and product based on hybrid experience playback, which aims to solve or improve at least one of the above-mentioned technical problems.

[0006] To achieve the above objectives, the present invention provides the following solution: A trajectory planning method based on hybrid experience playback includes: Establish a three-dimensional dynamic model architecture for unmanned aerial vehicles; Construct a heuristic non-sparse reward function to generate the instantaneous total reward value of the drone; A TD3-MER sample management system was constructed, which introduced a dual buffer pool and a dynamic priority mechanism, and adopted a proportional mixed sampling strategy to generate mini-batches for each iteration; Introduce normally distributed noise to generate smooth target actions and calculate the target Q-value of the Critic network; A delayed update strategy is adopted to optimize network parameters, and the updated strategy is fed back to the UAV for the next round of interaction. When the maximum number of training rounds is reached or the reward curve converges, training is stopped, the parameters of the main Actor network are frozen, the Critic network, the experience replay pool, and the target network are removed, and the model is deployed to a drone for trajectory planning.

[0007] Furthermore, a three-dimensional UAV dynamics model architecture is established, including: The task of planning the flight path of the UAV is set according to three-dimensional space; The UAV trajectory planning task is modeled as a Markov decision process, with state space S and action space A set. The motion parameters for the next moment are calculated based on the UAV's own state and motion vectors, and a three-dimensional dynamic equation is constructed.

[0008] Furthermore, a heuristic non-sparse reward function is constructed to generate the instantaneous total reward value of the drone, including: The expression for the reward function is: In the formula, This represents the instantaneous total reward value. For planar driven rewards; As a reward for height difference; Penalty for obstacle collision; Penalty for time step; Plane-driven reward, expressed as: In the formula, The Euclidean distance between the current position of the drone and the target position on the horizontal plane; This is the distance normalization coefficient; The height difference reward is expressed as follows: In the formula, This represents the absolute value of the vertical height difference between the drone and the target point; The obstacle collision penalty is expressed as follows: In the formula, This is the penalty coefficient; The radius of the danger threshold; This represents the distance to the nearest detected obstacle. The time step penalty is expressed as: In the formula, This represents the consumption coefficient per unit time step.

[0009] Furthermore, a TD3-MER sample management system is constructed, introducing a dual-buffer pool and dynamic priority mechanism, and employing a proportional-mixed sampling strategy to generate mini-batches for each iteration, including: When a new transfer sample is generated through the interaction between the drone and the environment, the priority is calculated based on the instantaneous total reward and the prediction bias of the current model, expressed as: In the formula, The priority of the i-th sample; Let be the instantaneous total reward for the i-th sample; This refers to the time difference error; The target Q value; This is an estimate of the current Critic network value; These are weight parameters; and It is a smoothing constant; Let i be the spatial vector of the i-th sample; Let i be the action vector of the i-th sample; Set a priority threshold. If the priority is greater than the priority threshold, the sample will be stored in the priority experience replay pool; otherwise, it will be stored in the normal experience replay pool. During network iterative training, a proportional mixed sampling strategy is used to extract data from two buffer pools to construct mini-batches.

[0010] Furthermore, normally distributed noise is introduced to generate smooth target actions, and the target Q-value of the Critic network is calculated, including: Based on the action vector output by the target Actor network, truncated normal distribution noise is added to generate a smooth target action, expressed as: In the formula, The target action after smoothing; For the target Actor network, based on the next state Deterministic actions output; It is normally distributed noise; Noise cutoff boundary; The standard deviation of noise; This represents the upper and lower bounds of the action space; The Q-values ​​are calculated using two independent target Critic networks, and the minimum value is taken as the final target Q-value. The expression is as follows: In the formula, Let Q be the target Q value for the i-th sample; This represents the actual instantaneous reward for the i-th sample; Discount factor; Let be the estimation function for the j-th target Critic network.

[0011] Furthermore, a delayed update strategy is adopted to optimize network parameters, and the updated strategy is fed back to the UAV for the next round of interaction, including: Minimize the mean squared error between the target Q-value and the current Critic network estimate, while simultaneously updating the parameters of both Critic networks. The expression is as follows: In the formula, This represents the mean squared error loss value of the Critic network; Let Q be the target Q value for the i-th sample; This is the estimated Q-value of the current main Critic network; This represents the total number of training batches in a single session. Using a delayed update strategy, the optimization objective of the Actor network is to maximize the Q-value, and its gradient expression is: In the formula, Actor network parameters The gradient direction; Mini batch size; The gradient of the Critic network for the action; The gradient of the Actor network with respect to the parameters; It is a state vector; For action vectors; This is the current main Actor network; A minimum step tracking mechanism is used to softly update the Actor network parameters, and the updated strategy is fed back to the UAV for the next round of interaction. The expression is: In the formula, and For the network parameters of the target Critic and the target Actor; and These are the network parameters for the current main Critic and main Actor; This is the soft update coefficient.

[0012] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This invention discloses a trajectory planning method, system, and product based on hybrid experience replay. The method solves the modeling and sparse reward learning challenges of UAVs in complex 3D environments by establishing a 3D dynamic model and a non-sparse reward function. By introducing a hybrid experience replay mechanism, it combines the efficiency of priority replay with the diversity of ordinary replay, significantly improving sample utilization efficiency and algorithm convergence speed. At the same time, it enhances the model's generalization ability and obstacle avoidance success rate in unfamiliar dense obstacles, large-scale scenes, and dynamic environments. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a framework diagram of the TD3-MER algorithm in this invention; Figure 2 This is a comparison chart of the convergence performance curves of the algorithm in this embodiment; Figure 3 This is a comparison chart of the average success rates under the control of the three algorithms in this embodiment; Figure 4 This is a comparison chart of the average path length under the control of the three algorithms in this embodiment. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] The purpose of this invention is to provide a trajectory planning method, system, and product based on hybrid experience playback, which aims to solve or improve at least one of the above-mentioned technical problems.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] like Figure 1 As shown, this invention provides a trajectory planning method based on hybrid experience playback, comprising: Step 1: Establish the UAV dynamics model architecture in three-dimensional space, including: Step 11: Set the UAV's flight path planning task based on three-dimensional space; The flight path planning task is as follows: the UAV starts from the designated starting position, autonomously avoids static and dynamic obstacles in the environment (including but not limited to buildings, trees, high-voltage lines, etc.) during the flight, and finally safely arrives at the target position.

[0019] To closely resemble real-world flight scenarios, the simulation environment set in this embodiment has a starting altitude and target position altitude of more than 50 meters, ensuring that the drone can operate safely in airspace beyond visual range or above complex terrain.

[0020] Step 12, model the UAV trajectory planning task as a Markov decision process, setting the state space S and action space A, including: The state vector expression for state space S is: , , In the formula, It is a state vector; This represents the internal information vector of the UAV. and Let be the velocity and yaw angle of the UAV in the xy plane at time t, respectively; The obstacle environment information vector is generated by measuring the nearest n=9 obstacles of the UAV using an airborne LiDAR (if the actual number of detected obstacles is insufficient). Then, fill in the remaining dimensions. Let be the Euclidean distance between the drone and the i-th obstacle; n is the number of obstacles; The target state vector; Let t be the projected distance between the UAV and the target point on the horizontal plane. Let t be the vertical height difference between the UAV and the target point at time t; The horizontal angle between the drone's current heading and the target direction; The expression for the action vector in action space A is: In the formula, For action vectors; This represents the speed adjustment of the UAV on the horizontal plane, indicating the speed increment commanded at time t, used to control the increase or decrease of flight speed; The yaw angle adjustment of the UAV represents the angle increment commanded at time t, used to control the left and right yaw of the flight heading; This represents the vertical altitude adjustment of the UAV, indicating the altitude change at time t, used to control the ascent and descent of flight altitude.

[0021] Step 13: Calculate the motion parameters for the next moment based on the UAV's own state and motion vectors, and construct the three-dimensional dynamic equations, including: The equation for updating motion parameters is expressed as follows: In the formula, Let t+1 be the speed of the UAV on the horizontal plane. Let yaw angle be the yaw angle of the UAV at time t+1; Let be the change in the vertical height of the UAV from time t to t+1. , and These are the speed adjustment, yaw angle adjustment, and altitude adjustment for the drone, respectively. This is a truncation function, indicating that when the calculation result exceeds the range, the boundary value is forcibly taken to meet the safety constraints; Maximum flight speed; Maximum yaw angle; This represents the maximum vertical lifting speed; In this embodiment, considering safety constraints during drone flight, the maximum flight speed of the drone is set... Set to 20 m / s, maximum yaw angle Set it to π / 6, and the maximum vertical lifting speed is set to 5m / s.

[0022] Based on the updated motion parameters, the three-dimensional dynamic equations of the UAV are constructed using the first-order Euler integral method, and the expression is as follows: In the formula, and These are the speed and yaw angle of the UAV at time t+1, respectively; Let t+1 be the coordinates of the UAV in three-dimensional space; Let t be the coordinates of the UAV in three-dimensional space at time t; The discrete time step represents the time resolution of the simulation environment, which is 1 second in this embodiment.

[0023] Step 2: Construct a heuristic non-sparse reward function to generate the instantaneous total reward value for the drone and guide its learning, including: The expression for the reward function is: In the formula, This represents the instantaneous total reward value. The planar drive reward is used to guide the drone to approach the target on a horizontal plane; The altitude difference reward is used to guide the drone to align with the target altitude in the vertical direction; Obstacle collision penalty is used to force drones to avoid obstacles in the environment; A time step penalty is used to encourage drones to complete tasks in the shortest time and avoid lingering. Planar drive reward, which encourages the drone to approach the target in the horizontal direction, is expressed as: In the formula, The Euclidean distance between the current position of the drone and the target position on the horizontal plane; This is the distance normalization coefficient, a positive constant used to normalize the distance dimension, preventing gradient explosion or masking other reward items due to excessively large distance values.

[0024] In the embodiments of the present invention The value is set to 2000m, which corresponds to the maximum expected horizontal range in a typical mission scenario.

[0025] Altitude difference reward guides the drone to reduce the vertical distance to the target; the expression is: In the formula, The absolute value of the vertical height difference between the drone and the target point. Regardless of whether the drone is higher or lower than the target, the drone is driven to maneuver towards the target's altitude level by rewarding the height difference. The obstacle collision penalty is implemented as a piecewise function, applying a penalty only when the drone enters the obstacle's danger threshold range. The expression is: In the formula, The penalty coefficient is set to a positive real constant and is used to adjust the priority of obstacle avoidance behavior. The larger the value, the stronger the drone's repulsion force against obstacles. In this embodiment, it is 30, which ensures that the obstacle avoidance penalty is significantly greater than the benefit of approaching the target, thus prioritizing flight safety. Where is the danger threshold radius, when Timely assessment of drones entering potentially dangerous areas; This is the distance to the nearest obstacle currently detected, which is the smallest Euclidean distance between the drone and all surrounding obstacles as measured by the onboard sensors.

[0026] The aforementioned obstacle collision penalty is continuous; the closer the drone is to the obstacle, the greater the penalty value, thus smoothly guiding the drone to avoid obstacles rather than only receiving feedback at the moment of collision.

[0027] To encourage drones to reach their targets quickly and avoid ineffective wandering or oscillation in the environment, a constant time step penalty is introduced, expressed as: In the formula, The consumption coefficient per unit time step is set to a small positive real number, which is 0.01 in this embodiment.

[0028] The aforementioned time-step penalty reduces the cumulative reward for each action taken. This prompts the agent to learn the shortest path strategy. If the task times out and the goal is not reached, the accumulated huge negative rewards will further reinforce the "quick completion" behavior pattern.

[0029] In the above steps, a planar driving reward is introduced. and height difference bonus This allows the drone to receive continuous gradient guidance even in the intermediate state where it has not reached the target and has not collided, significantly accelerating the convergence speed of the reinforcement learning algorithm. (This is in contrast to obstacle collision penalty.) Introducing a danger threshold radius By imposing gradually increasing penalties as the drone approaches an obstacle but before colliding with it, the drone can perform evasive maneuvers earlier and more smoothly, thus improving flight safety.

[0030] Step 3: Construct TD3-MER sample management, introduce a dual buffer pool and dynamic priority mechanism, and adopt a proportional mixed sampling strategy to generate mini-batches for each iteration, including: Step 31: After the UAV interacts with the environment and generates a new transfer sample, the priority is calculated based on the instantaneous total reward and the prediction bias of the current model. The expression is: In the formula, The priority of the i-th sample; Let be the instantaneous total reward for the i-th sample, reflecting the immediate benefit of the action; This refers to the time difference error; The target Q value; This is the current estimate of the Critic network, reflecting the model's prediction bias for the current sample; The weight parameter has a value range of [0,1]. and This is a smoothing constant, a small positive constant used to ensure that the priority is greater than 0 and to avoid errors in probability calculation. Let i be the spatial vector of the i-th sample; Let i be the action vector of the i-th sample; The steps described above, by introducing instantaneous total reward, allow the model to focus on beneficial behaviors more quickly in the early stages of training. By introducing temporal difference error, the model can focus on difficult samples with large prediction biases in the later stages.

[0031] Step 32: Set a priority threshold. If the priority is greater than the priority threshold, the sample is stored in the priority experience replay pool; otherwise, it is stored in the ordinary experience replay pool. The priority threshold is the median priority of the current batch.

[0032] In the above steps, the priority experience replay pool is set to a sufficiently large capacity to ensure that high-value samples are reserved first and frequently used for training; the ordinary experience replay pool has a fixed capacity and implements a first-in-first-out mechanism to cover old data in order to maintain the diversity of sample state distribution and improve the generalization ability of the model.

[0033] Step 33: During network iterative training, a proportional-mixed sampling strategy is used to extract data from two buffer pools to construct mini-batches, including: The number of samples is expressed as: In the formula, The number of samples collected for the priority experience replay pool; This represents the total number of training batches in a single session. This is the sampling ratio coefficient for the ordinary pool; This represents the number of samples collected in a typical experience replay pool.

[0034] Step 4: Introduce normally distributed noise to generate smooth target motion, and calculate the target Q-value of the Critic network, including: Step 41: Based on the action vector output by the target Actor network, truncated normal distribution noise is added to generate a smooth target action, expressed as: In the formula, The target action after smoothing; For the target Actor network, based on the next state Deterministic actions output; It is normally distributed noise; Noise cutoff boundary; The standard deviation of noise; This represents the upper and lower bounds of the action space; Step 42: Calculate the Q-value using two independent target Critic networks, and take the minimum value as the final target Q-value. The expression is: In the formula, Let Q be the target Q value for the i-th sample; This represents the actual instantaneous reward for the i-th sample; This is a discount factor with a value of 0.99, used to balance current rewards with expected future rewards; Let be the estimation function for the j-th target Critic network.

[0035] Step 5: Optimize network parameters using a delayed update strategy, and feed the updated strategy back to the drone for the next round of interaction, including: Step 51: Minimize the mean squared error (MSE) between the target Q value and the current Critic network estimate, and update the parameters of both Critic networks simultaneously. The expression is: In the formula, The mean squared error loss value of the Critic network represents the average sum of squared deviations between the current network prediction and the target value. The optimization objective is to minimize the L value, thereby improving the accuracy of the target Q value estimation. Let Q be the target Q value for the i-th sample; This is the estimated Q-value of the current main Critic network; This represents the total number of training batches in a single session. Step 52: A delayed update strategy is adopted. The Actor network is updated once for every d updates to the Critic network. The optimization objective of the Actor network is to maximize the Q-value, and its gradient expression is: In the formula, Actor network parameters The gradient direction; Mini batch size; The gradient of the action in the Critic network indicates how the action can be adjusted to obtain a higher Q value; The gradient of the Actor network with respect to the parameters indicates how to adjust the parameters to output the action; Let this be the state vector, representing the current state in the current mini-batch; The action vector is the action generated by the current Actor network; For the current main Actor network, the input state vector Output action vector ; Step 53: Employ a minimum step size tracking mechanism to softly update the Actor network parameters and feed the updated strategy back to the UAV for the next round of interaction. The expression is: In the formula, and For the network parameters of the target Critic and the target Actor; and These are the network parameters for the current main Critic and main Actor; This is the soft update coefficient, which takes a minimum value, indicating that the target network parameters only track the main network with tiny steps, ensuring the smoothness and stability of the training process.

[0036] In this embodiment, the entire training process of the TD3-MER model specifically includes: 1) Initialization phase Network parameter initialization: Initialize the main Actor network parameters. and target Actor network parameters Initialize the parameters of the two main Critic networks. , and the corresponding two target Critic network parameters , Establish a priority experience replay pool and a normal experience replay pool.

[0037] Hyperparameter settings: Set the maximum number of training rounds, the maximum time step per round, the batch size, the sampling ratio of the normal pool, the priority threshold, the soft update coefficient, and the delayed update frequency.

[0038] 2) Iterative training loop For each training epoch from 1 to the maximum training epoch, perform the following operations: Environment reset: Randomly initialize the drone's starting position, obstacle distribution, and target position to obtain the initial state vector; Time-step interaction loop: For each time step from 1 to the maximum time step of each round: The current Actor network outputs an action vector based on the state vector and enters the exploration noise; the action vector is executed, the environment provides instantaneous total reward and transitions to the next state vector; transition samples are constructed and the priority of the samples is calculated; If the priority is greater than the priority threshold, the sample is stored in the priority experience replay pool; otherwise, it is stored in the ordinary experience replay pool.

[0039] Based on the sampling ratio of the ordinary pool, proportional mixing sampling is performed to obtain a mini-batch with a batch size of M.

[0040] The Critic network is updated based on mini-batch updates.

[0041] Based on the delayed update frequency d, the Actor network is soft-updated after the Critic network has been updated d times.

[0042] 3) Model saving When the maximum number of training rounds is reached or the reward curve converges, training is stopped, and the final main Actor network parameters are saved for deployment.

[0043] Step 6: When the maximum number of training rounds is reached or the reward curve converges, stop training, freeze the parameters of the main Actor network, remove the Critic network, the experience replay pool and the target network, and deploy the model to the drone for trajectory planning.

[0044] In the above steps, removing the Critic network, the experience replay pool, and the target network can effectively reduce the onboard computing load.

[0045] The drone collects the current state vector in real time and controls itself according to the optimal action vector output by the main Actor network until it reaches the target position.

[0046] To verify the effectiveness of the present invention, simulation experiments were conducted, and the present invention was compared with traditional algorithms.

[0047] like Figure 2 The figure shows a comparison of the convergence performance curves of the TD3-MER algorithm with those of the DDPG and TD3 algorithms. As can be seen from the figure, the TD3-MER algorithm (red curve) exhibits a significant advantage in average reward, consistently outperforming the other two comparison algorithms. This is because the hybrid experience replay mechanism allows the algorithm to focus more on important samples highly correlated with rewards, thereby improving learning efficiency.

[0048] like Figure 3The figure shows a comparison of the average success rates of UAV trajectory planning under three different algorithms. As can be seen from the figure, the TD3-MER algorithm proposed in this invention achieves an average success rate of 96.3%, a significant improvement of 29.8% compared to the original TD3 algorithm (74.2%), and far exceeding the DDPG algorithm (42.8%). In complex 3D obstacle environments, an extremely high success rate translates to an extremely low collision risk. This fully demonstrates the hybrid experience replay mechanism introduced in this invention, which effectively enhances the obstacle avoidance decision-making ability of the neural network by prioritizing the extraction and learning of "high-value" samples with high immediate rewards and high temporal difference errors (such as key decision points for successful obstacle avoidance), enabling the UAV to complete navigation tasks with an extremely high probability of safety.

[0049] like Figure 4 As shown, the average path length of the UAV successfully reaching the target under the control of three algorithms is compared. Statistical results show that the TD3-MER algorithm has the shortest average path length among the three algorithms. One of the core objectives of UAV trajectory planning is to reach the target as quickly as possible while ensuring safety; a shorter path length directly reflects the superiority of the strategy. This indicates that the TD3-MER algorithm not only possesses excellent obstacle avoidance capabilities but also plans a more direct and efficient flight route for the UAV. Thanks to the effective balancing of sample distribution by the hybrid experience replay pool, it effectively overcomes the hesitation and ineffective hovering phenomena that traditional reinforcement learning algorithms easily produce at complex obstacle boundaries, guiding the UAV decisively toward the target, thereby significantly optimizing the flight trajectory.

[0050] Based on the same inventive concept, embodiments of the present invention also disclose a trajectory planning system based on hybrid experience playback, comprising: The environmental modeling and state perception module establishes a three-dimensional dynamic model architecture for unmanned aerial vehicles. The heuristic reward shaping module constructs a heuristic non-sparse reward function to generate the instantaneous total reward value of the drone and guide the drone's learning. The TD3-MER sample intelligent management module constructs TD3-MER sample management, introduces a dual buffer pool and dynamic priority mechanism, and adopts a proportional mixed sampling strategy to generate mini-batches for each iteration; The robust target value calculation module introduces normally distributed noise to generate smooth target actions and calculates the target Q value of the Critic network. The strategy delay optimization and update module uses a delayed update strategy to optimize network parameters and feeds the updated strategy back to the UAV for the next round of interaction. The deployment module stops training when the maximum number of training rounds is reached or the reward curve converges, freezes the parameters of the main Actor network, removes the Critic network, the experience replay pool, and the target network, and deploys the model to the drone for trajectory planning.

[0051] This invention also discloses a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the trajectory planning method based on hybrid experience playback.

[0052] This invention also discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the trajectory planning method based on hybrid experience playback.

[0053] The program code used to implement the method of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the steps of the method of the present invention to be performed. All aspects of the present invention not described in detail are well-known to those skilled in the art.

[0054] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0055] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A trajectory planning method based on hybrid experience playback, characterized in that, include: Establish a three-dimensional dynamic model architecture for unmanned aerial vehicles; Construct a heuristic non-sparse reward function to generate the instantaneous total reward value of the drone; A TD3-MER sample management system was constructed, which introduced a dual buffer pool and a dynamic priority mechanism, and adopted a proportional mixed sampling strategy to generate mini-batches for each iteration; Introduce normally distributed noise to generate smooth target actions and calculate the target Q-value of the Critic network; A delayed update strategy is adopted to optimize network parameters, and the updated strategy is fed back to the UAV for the next round of interaction. When the maximum number of training rounds is reached or the reward curve converges, training is stopped, the parameters of the main Actor network are frozen, the Critic network, the experience replay pool, and the target network are removed, and the model is deployed to a drone for trajectory planning.

2. The trajectory planning method based on hybrid experience playback according to claim 1, characterized in that, The establishment of the three-dimensional UAV dynamics model architecture includes: The task of planning the flight path of the UAV is set according to three-dimensional space; The UAV trajectory planning task is modeled as a Markov decision process, with state space S and action space A set. The motion parameters for the next moment are calculated based on the UAV's own state and motion vectors, and a three-dimensional dynamic equation is constructed.

3. The trajectory planning method based on hybrid experience playback according to claim 1, characterized in that, The construction of the heuristic non-sparse reward function to generate the instantaneous total reward value of the UAV includes: The expression for the reward function is: In the formula, This represents the instantaneous total reward value. For planar driven rewards; As a reward for height difference; Penalty for obstacle collision; Penalty for time step; Plane-driven reward, expressed as: In the formula, The Euclidean distance between the current position of the drone and the target position on the horizontal plane; This is the distance normalization coefficient; The height difference reward is expressed as follows: In the formula, This represents the absolute value of the vertical height difference between the drone and the target point; The obstacle collision penalty is expressed as follows: In the formula, This is the penalty coefficient; The radius of the danger threshold; This represents the distance to the nearest detected obstacle. The time step penalty is expressed as: In the formula, This represents the consumption coefficient per unit time step.

4. The trajectory planning method based on hybrid experience playback according to claim 1, characterized in that, The TD3-MER sample management system introduces a dual-buffer pool and dynamic priority mechanism, and employs a proportional-mixed sampling strategy to generate mini-batches for each iteration, including: When a new transfer sample is generated through the interaction between the drone and the environment, the priority is calculated based on the instantaneous total reward and the prediction bias of the current model, expressed as: In the formula, The priority of the i-th sample; Let be the instantaneous total reward for the i-th sample; This refers to the time difference error; The target Q value; This is an estimate of the current Critic network value; These are weight parameters; and It is a smoothing constant; Let i be the spatial vector of the i-th sample; Let i be the action vector of the i-th sample; Set a priority threshold. If the priority is greater than the priority threshold, the sample will be stored in the priority experience replay pool; otherwise, it will be stored in the normal experience replay pool. During network iterative training, a proportional mixed sampling strategy is used to extract data from two buffer pools to construct mini-batches.

5. The trajectory planning method based on hybrid experience playback according to claim 1, characterized in that, The process of introducing normally distributed noise, generating smooth target actions, and calculating the target Q-value of the Critic network includes: Based on the action vector output by the target Actor network, truncated normal distribution noise is added to generate a smooth target action, expressed as: In the formula, The target action after smoothing; For the target Actor network, based on the next state Deterministic actions output; It is normally distributed noise; Noise cutoff boundary; The standard deviation of noise; This represents the upper and lower bounds of the action space; The Q-values ​​are calculated using two independent target Critic networks, and the minimum value is taken as the final target Q-value. The expression is as follows: In the formula, Let Q be the target Q value for the i-th sample; This represents the actual instantaneous reward for the i-th sample; Discount factor; Let be the estimation function for the j-th target Critic network.

6. The trajectory planning method based on hybrid experience playback according to claim 1, characterized in that, The process of optimizing network parameters using a delayed update strategy and feeding the updated strategy back to the UAV for the next round of interaction includes: Minimize the mean squared error between the target Q-value and the current Critic network estimate, while simultaneously updating the parameters of both Critic networks. The expression is as follows: In the formula, This represents the mean squared error loss value of the Critic network; Let Q be the target Q value for the i-th sample; This is the estimated Q-value of the current main Critic network; This represents the total number of training batches in a single session. Using a delayed update strategy, the optimization objective of the Actor network is to maximize the Q-value, and its gradient expression is: In the formula, Actor network parameters The gradient direction; Mini batch size; The gradient of the Critic network for the action; The gradient of the Actor network with respect to the parameters; It is a state vector; For action vectors; This is the current main Actor network; A minimum step tracking mechanism is used to softly update the Actor network parameters, and the updated strategy is fed back to the UAV for the next round of interaction. The expression is: In the formula, and For the network parameters of the target Critic and the target Actor; and These are the network parameters for the current main Critic and main Actor; This is the soft update coefficient.

7. A trajectory planning system based on hybrid experience playback, characterized in that, include: The environmental modeling and state perception module establishes a three-dimensional dynamic model architecture for unmanned aerial vehicles. The heuristic reward shaping module constructs a heuristic non-sparse reward function to generate the instantaneous total reward value of the drone and guide the drone's learning. The TD3-MER sample intelligent management module constructs TD3-MER sample management, introduces a dual buffer pool and dynamic priority mechanism, and adopts a proportional mixed sampling strategy to generate mini-batches for each iteration; The robust target value calculation module introduces normally distributed noise to generate smooth target actions and calculates the target Q value of the Critic network. The strategy delay optimization and update module uses a delayed update strategy to optimize network parameters and feeds the updated strategy back to the UAV for the next round of interaction. The deployment module stops training when the maximum number of training rounds is reached or the reward curve converges, freezes the parameters of the main Actor network, removes the Critic network, the experience replay pool, and the target network, and deploys the model to the drone for trajectory planning.

8. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the trajectory planning method based on hybrid experience playback as described in claims 1-6.

Images