Double-depth Q network dynamic path planning method fusing attention mechanism

Abstract

The invention relates to a double-depth Q network dynamic path planning method fusing an attention mechanism, and belongs to the technical field of path planning. The invention provides a path planning method suitable for a complex dynamic environment by improving a DDQN algorithm and combining an attention mechanism and a self-adaptive reward mechanism. The method comprises the following innovation points: 1) introducing a self-adaptive reward mechanism, dynamically adjusting the weight of path length, collision avoidance and energy consumption, and ensuring the safety and efficiency of path planning; (2) an attention mechanism is embedded in a network structure, the focusing capability on key environment information is enhanced, and the decision-making precision is improved; 3) adopting a priority experience playback strategy to accelerate model convergence; and 4) designing a dynamic exploration strategy, and adaptively adjusting the exploration rate according to the training progress. The method can effectively improve the efficiency and quality of path planning, and is suitable for path planning tasks in dynamic environments such as automatic driving and robot navigation.

Description

Technical Field

[0001] The present invention relates to the field of path planning, and specifically relates to a path planning method based on improved DQN. Background Technique

[0002] Since the advent of robotics, path planning has always been one of the core research topics. Its main goal is to design a collision-free path for the robot so that it can reach the target position safely and efficiently. Path planning for mobile robots usually needs to solve three key problems: First, ensure that the robot can move smoothly from the starting point to the end point; Second, enable the robot to avoid obstacles through specific algorithms and pass through specific points when necessary to complete specific tasks; Finally, on the basis of meeting the above conditions, optimize the motion trajectory of the robot as much as possible. For mobile robots performing tasks such as rescue and transportation, the accuracy of path planning is crucial.

[0003] Traditional path planning methods, such as the A* algorithm or the Dijkstra algorithm, usually rely on a pre-acquired environmental map. However, with the development of technology, dynamic path planning has gradually become a research hotspot. This method enables the robot to navigate without pre-setting static environmental information. At the same time, Deep Reinforcement Learning (DRL), as an emerging technology, is being widely used to solve dynamic path planning problems and has attracted much attention because of its ability to adapt to environmental changes and dynamic obstacles.

[0004] In the field of path planning for autonomous mobile robots, it is mainly divided into two methods: offline and online. Offline path planning relies on a prior map of the environment. By modeling the map as a maze structure and using search algorithms such as A* or Dijkstra for global path planning. Online path planning emphasizes real-time performance. The robot navigates to the target position by continuously processing sensor data. This method pays more attention to local path planning to ensure that the robot can effectively avoid obstacles. In online path planning, the artificial potential field method, convolutional neural network, and reinforcement learning are commonly used technical means. In recent years, reinforcement learning has been widely applied and deeply studied in the field of online path planning because of its strong adaptability to environmental mutations and dynamic obstacles.

[0005] DRL focuses on the performance of software objects in the environment to maximize cumulative rewards. Through trial and error, the agent interacts with the environment to learn the optimal policy for solving sequential decision-making problems in fields such as natural science, social science, and engineering. The sequential decision-making process is as follows: The agent is provided with some initial observations of the environment and is required to select some actions from a given set of possible actions. The environment responds by switching to another state and generating a reward signal (a scalar number), which is regarded as a fundamental truth estimate of the agent's performance. This process continues, and the agent makes choices based on its observations and the environment, and reacts to the next state and reward signal. The only goal of the agent is to maximize the cumulative reward.

[0006] With the evolution of technology, deep reinforcement learning methods have become an emerging force in solving path planning problems. Among them, the DDQN algorithm is an important advancement compared to traditional Q-learning algorithms. It successfully overcomes the overestimation problem that is prone to occur in Q-learning, making the learning process more stable. This is like walking on a foggy road. Without the correct guidance (without overcoming the overestimation problem), it is easy to deviate from the correct direction, while the DDQN algorithm provides relatively accurate road signs for learners. However, despite such advantages, the traditional DDQN algorithm is not perfect in path planning tasks. When faced with complex and changing actual tasks, it has relatively significant problems such as slow convergence speed and insufficient performance in path optimization. In a complex urban traffic scenario, if the convergence speed is too slow, autonomous vehicles may not be able to determine a suitable driving route for a long time; if the path optimization ability is poor, it may lead to the vehicle choosing some circuitous and non-optimal routes, resulting in adverse consequences such as energy waste or excessive travel time. Summary of the Invention

[0007] The present invention elaborately constructs a path planning method based on an improved DDQN algorithm, aiming to comprehensively improve the planning efficiency and path quality of the algorithm when performing path planning tasks in complex environments. This is achieved by innovatively optimizing multiple key elements such as the network structure, reward mechanism, experience replay, and exploration strategy.

[0008] 1. Optimization of the adaptive reward mechanism

[0009] The reward mechanism of traditional DDQN has obvious defects in the path planning process, that is, it is very difficult to balance the two key factors of path length and collision avoidance at the same time. In actual application scenarios, the path length is directly related to resource consumption, such as the energy consumption of a robot or the fuel consumption of a vehicle; collision avoidance is related to the basic requirement of safe operation. If the reward mechanism cannot balance the two well, just like the uneven distribution of weights on both ends of a balance, it will lead to unbalanced training of the intelligent agent. For example, in some path planning scenarios in narrow channels, only focusing on the path length may cause the intelligent agent to ignore the collision risk. On the contrary, if only collision avoidance is emphasized, it may lead to an overly long path.

[0010] To address the above problems, the present invention designs a unique dynamic weighted reward mechanism. This mechanism is not limited to only the two factors of path length and collision avoidance, but further takes energy consumption into consideration. In this dynamic weighted system, the weights of various factors will be flexibly adjusted according to different environmental conditions and task requirements. In the initial stage of training, when the intelligent agent has limited knowledge of the environment, the weight of the collision avoidance factor can be appropriately increased to ensure that the intelligent agent explores under the premise of safety; as the training progresses and the collision risk significantly decreases, the weights of the path length and energy consumption factors can be gradually increased to guide the intelligent agent to learn towards a shorter and more energy-efficient path. In this way, it is like equipping the intelligent agent with an experienced navigator, enabling it to quickly and accurately converge to a path that is both safe and efficient (high-quality) during the training process.

[0011] 2. Improvement of the network structure based on the attention mechanism

[0012] When the DDQN algorithm is used for path planning, the network structure has a fundamental impact on the accuracy of decision-making. The original DDQN network structure often lacks the ability to accurately focus on key information when dealing with complex environmental information. In a complex and busy urban traffic scenario or an indoor robot working environment with a complex layout, numerous environmental elements (such as a large number of buildings, moving pedestrians, vehicles, or various obstacles of different shapes and sizes) constitute a vast amount of input information. If the network structure cannot effectively filter and focus on the most critical information, it is like losing direction in the vast ocean of information, resulting in decision-making errors or unreasonable path planning.

[0013] The present invention ingeniously introduces an attention mechanism on the basis of DDQN. The attention mechanism is like an information focus, which can automatically identify and highlight the information that is most crucial for path planning in the environment. For example, in a complex traffic scenario, it can focus on information such as approaching vehicles and key road direction features. In this way, the intelligent agent can make decisions based on this filtered and enhanced key information, thereby greatly improving the accuracy of decision-making and the rationality of path planning, as if finding a clear and optimal route in a complex maze.

[0014] 3. Training Optimization Based on Prioritized Experience Replay

[0015] In the traditional DDQN training process, experience replay adopts the method of uniform random sampling. There is a serious problem with this method, that is, it does not distinguish the value levels of different experience samples. In the training process of path planning, the experience values generated by different decisions and action results vary greatly. For example, the decision-making experiences that successfully avoid dangerous obstacles or directly lead to the target location obviously have higher value than the experiences of blind exploration without results. If all experience samples are sampled for training with the same probability, it is like treating precious gems and ordinary stones equally, and the beneficial effects brought by high-value experiences cannot be fully utilized, thereby affecting the overall training efficiency.

[0016] The present invention adopts improved prioritized experience replay. This mechanism assigns corresponding priorities to each experience sample according to its potential value for learning. Experience samples with high priorities, that is, those with important guiding significance for path planning, will have a higher probability of being sampled for training. This is like preferentially selecting the most precious secrets in a knowledge treasure house for learning. In this way, the usage frequency of high-value experience samples can be effectively increased, the training efficiency can be greatly improved, the learning process of the intelligent agent in path planning can be accelerated, and the intelligent agent can master effective path planning strategies faster.

[0017] 4. Enhancement of Dynamic Exploration Strategy

[0018] Traditional DDQN uses a fixed Exploration-Exploitation strategy. In this fixed mode, regardless of the training stage, the exploration rate remains unchanged. This is like setting a mechanical action mode for the agent, which cannot flexibly adjust the strategy according to the actual learning progress. In the initial stage of path planning training, when the agent knows little about the environment, if the exploration rate is too low, the agent may fall into a local optimum and miss a better global path; while if the exploration rate is too high in the later stage when enough knowledge has been accumulated, it will lead to unnecessary repeated exploration, wasting computing resources and delaying the convergence speed.

[0019] The present invention adopts a carefully designed dynamic exploration strategy, which can adaptively adjust the exploration rate according to the training progress. In the initial stage of training, the agent is in a state of exploration and ignorance of the environment. At this time, increasing the exploration rate encourages the agent to actively explore more unknown environmental areas and action possibilities, avoiding falling into a local optimal path prematurely; as the training progresses gradually, the agent obtains more and more experience. At this time, gradually reducing the exploration rate allows the agent to make optimal decisions by making more use of the knowledge already learned. This dynamic adjustment mechanism, like an experienced explorer, flexibly adjusts the exploration pace according to different terrains and journey stages, thus effectively balancing the relationship between exploration and exploitation, and significantly improving the robustness of path planning, that is, regardless of whether the environment is simple or complex, static or dynamic, it can stably and reliably plan a reasonable path. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings used:

[0021] Figure 1 It is a schematic flowchart of the method according to an embodiment of the present invention;

[0022] Figure 2 It is a 25×25 grid network map;

[0023] Figure 3 It is a brief reinforcement learning process;

[0024] Figure 4 It is a reinforcement learning process with a prioritized experience replay buffer;

[0025] Figure 5 It is a schematic diagram of the SumTree data structure; DETAILED DESCRIPTION OF THE INVENTION

[0026] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0027] To make the above objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Embodiment 1

[0029] As Figure 1 shown, it is a schematic diagram of the method flow of this embodiment, and the steps include:

[0030] step1. Construct an environment for path planning.

[0031] As Figure 2 shown, it is a grid network environment map of 25×25. Each square has the same size and is evenly distributed. The squares are divided into four types, namely the starting square, the feasible square, the obstacle square, and the end square; the starting square and the end square are in a connected state.

[0032] step2. Use the MDP characteristics for modeling.

[0033] The framework of using reinforcement learning methods to solve the path planning problem is to learn how to maximize the benefit of the sequential decision-making problem, which can be represented by the Markov decision process (MDP).

[0034] MDP is a framework that can be composed of a five-tuple (S, A, P, R, γ), where (S, A, R) are the three sets included in MDP, representing respectively:

[0035] S: State set

[0036] A: Action set

[0037] R: Reward set

[0038] P is the probability distribution included in MDP, and there are two probability distributions, namely:

[0039] State transition probability: p(s1|s,a) represents the probability of transferring to state s1 when performing action a in state s

[0040] Reward probability: p(r|s,a) represents the probability of obtaining reward r when performing action a in state s

[0041] γ ∈ (0, 1): Discount factor, which is used to control the agent to pursue a shorter path to reach the goal in the path planning problem. If γ is smaller, the agent will be more "myopic"; on the contrary, if γ is larger, the agent will be more "far-sighted"; if γ is 1, the agent ignores the time steps and treats the rewards of all time steps equally.

[0042] Policy: The policy Π(a, s) represents the probability of choosing action a in state s.

[0043] The MDP also has an important property, namely the memoryless property, and the mathematical formula is as follows:

[0044] p(s t+1 |a t+1 ,s t ,..., a1, s0) = p(s t+1 |a t+1 ,s t )

[0045] p(r t+1 |a t+1 ,s t ,..., a1, s0) = p(r t+1 |a t+1 ,s t )

[0046] The purpose of using reinforcement learning to solve the path planning problem is to finally find an optimal action sequence in the given environment to maximize the cumulative reward of the agent. For a given action policy π, the cumulative reward is defined, and the formula is as follows:

[0047]

[0048] In the formula, Gt is called the cumulative reward, which represents the sum of discounted rewards from time step t to the end of the action sequence.

[0049] Since the action sequences of the agent in the same state may be different, the cumulative reward of a certain state is an expected value, rather than a definite value. Define a state value function V π (s) to quantify the expected value of the cumulative reward of state s under the given policy π, and the formula is as follows:

[0050]

[0051] Among them, V π (s) represents the expected cumulative reward of the agent starting from state s under the given policy π. Further generalize it to action a, and change the state value function to the state-action value function q(s t ,at ) to describe the cumulative reward, the formula is as follows:

[0052] q π (s,a) = E[G t |S t = s,A t = a]

[0053] The algorithm of reinforcement learning needs to obtain all state-action value functions for a given state, select the maximum value from them, and then obtain the optimal policy. The formula is as follows:

[0054]

[0055] The brief reinforcement learning process is as Figure 3 shown.

[0056] step3. Build a path planning model of the DDQN algorithm based on the two-dimensional grid network environment map.

[0057] The introduction of deep neural networks has made great success in both application and method of reinforcement learning algorithms, and the role of neural networks is crucial. Traditional reinforcement learning algorithms rely on constructing and updating the Q-table to optimize the policy. However, the Q-table-based method is limited by storage and computing capabilities and is difficult to handle the problems of large state space and action space. The proposed DQN effectively combines the Q-learning algorithm with deep neural networks, uses neural networks to replace the Q-value table, thus alleviating the dimensionality disaster problem encountered by Q-learning and enhancing the generalization ability of the model. By estimating Q(s,a) through a deep convolutional neural network, DQN solves the problem of limited capacity, and at the same time, the generalization ability of the deep convolutional neural network makes it unnecessary to train each state-action value function separately.

[0058] Although DQN has made breakthrough progress in reinforcement learning, there are still certain limitations. Its main problem is that the maximization operation adopted by Q-learning will lead to an overestimation bias in Q-value estimation, that is, there is an overestimation problem. To solve this problem, DDQN (Double Deep Q Network) came into being. DDQN reduces the overestimation phenomenon in the Q-learning process by splitting action selection and Q-value calculation, thus improving the stability and performance of the algorithm.

[0059] DDQN is an improved algorithm based on Q-learning, aiming to minimize the objective function, as follows:

[0060]

[0061] The specific process is as follows: In the two-dimensional grid network environment map, each grid corresponds to a state S. The agent starts from the current state S t and selects an action a according to the decaying ε-greedy strategy t , and then transfers to the new state S t+1 , while obtaining a reward R t . The state-action sequence (S t , a t , R t , R t+1 ) formed during the interaction between the agent and the environment is regarded as an experience and stored in the experience pool. There are 5 optional actions during the interaction between the agent and the environment, namely up, down, left, right, and staying still. Among them, the four actions of up, down, left, and right only move one grid distance each time. The setting of the decaying ε-greedy strategy is as follows:

[0062] The decaying ε-greedy makes a minor modification based on ε-greedy. ε represents the trade-off between exploration and exploitation. At the beginning, we hope that the value of ε is high, so that we can have a high degree of exploration. It can be understood that the agent can learn more new things. As the agent understands the environment and the future rewards, it should decay, so that the higher Q-values found by the agent can be fully utilized. That is, as time goes by, the value of ε continuously decays and becomes smaller and smaller.

[0063] The formula of the traditional ε-greedy method is as follows:

[0064]

[0065] where a represents the action, s represents the state, A(s) represents the number of actions in the action space, is the state-action value function.

[0066] The probability of choosing the greedy action is greater than or equal to the probability of choosing any other action. When ε = 0, it becomes the greedy strategy; when ε = 1, it becomes a uniform distribution, and an action is randomly selected from the action space.

[0067] For the decaying ε-greedy method, assume that there are N episodes in total during training. The algorithm initially sets ε = p init (for example, p init = 0.7), and then gradually decreases to ε = p end (for example, p init= 0.1), specifically, during the initial training process, the model is allowed to explore more freely with a high probability, and then gradually decreases at a rate of γ as the training progresses, increasing the probability of full utilization and decreasing the probability of free exploration. The formula is as follows:

[0068]

[0069] ε ← (p init - p end )γ + p end

[0070] During the training process, a batch of samples is drawn from the experience pool as training data, and then the TD-Error is calculated to update the network parameters. The calculation method of TD-Error is as follows:

[0071]

[0072] DDQN separates action selection from Q-value calculation by introducing two networks, namely the main network and the target network.

[0073] In the initial stage, the main network and the target network share the same parameters. During the training process, the main network updates the parameters θ through gradient descent optimization, while the parameters θ- of the target network are synchronized with those of the main network after a certain number of steps. Since the actions selected during the calculation of the target value come from the main network, and the Q-value evaluation comes from the target network, DDQN effectively reduces the problem of overestimation of the Q-value caused by the maximization operation in DQN, improving the learning stability and convergence speed.

[0074] Finally, through continuous iterative updates, the main network and the target network gradually converge to the optimal strategy, enabling the agent to efficiently learn the optimal behavior strategy in a complex environment.

[0075] step4. Apply the innovation points to improve the DDQN algorithm model to obtain the final path planning model:

[0076] stept4.1 For the optimization of the adaptive reward mechanism, the specific implementation is as follows:

[0077] During the training process of DDQN, use the adaptive reward function to replace the fixed reward function, calculate the new Q-value according to the adaptive reward function, and update the Q network.

[0078] The rewards are divided into five parts: distance reward, avoidance reward, arrival reward, obstacle penalty, and dwelling penalty, as shown in the following equations. Among them, target_distance represents the absolute value of the difference in the distance between the agent and the target point in the St+1 stage and the St stage. The closer the agent is to the target point, the higher the reward it obtains; conversely, the farther it deviates from the target point, the greater the penalty it receives. At the same time, when the agent avoids obstacles, it should be given a reward to encourage the agent to avoid obstacles. obstacle_distance is the absolute value of the difference in the distance between the agent and the nearest obstacle in the St+1 stage and the St stage, and b is the stall penalty, which is used to encourage the agent to find a solution more quickly.

[0079]

[0080] When the agent reaches the target, it gets a reward of +30, and when it collides with an obstacle, it gets a reward of -30. Experiments show that setting k = -0.25, μ = 0.004, and b = -0.004 can promote the rapid convergence of the model.

[0081] Step 4.2 Improvement of the network structure based on the attention mechanism is specifically implemented as follows:

[0082] The attention mechanism helps the network focus on the key parts of the input and improve its ability to process complex environments.

[0083] The network structure of DDQN includes:

[0084] Input layer: used to input the 25×25 map state (such as a two-dimensional matrix).

[0085] Convolutional layer: extracts spatial features.

[0086] Attention layer: calculates attention weights and highlights important regions.

[0087] Fully connected layer: inputs Q values.

[0088] Apply the attention mechanism after the convolutional layer of the DDQN network structure, perform fully connected processing, and then use the DDQN with the attention mechanism for training, store experiences, and update the network.

[0089] Step 4.3 Training optimization based on prioritized experience replay is specifically implemented as follows:

[0090] A brief reinforcement learning process with a prioritized experience replay buffer is shown in Figure 4Given that the enhancement effect of each experience on the model varies, to improve the utilization efficiency of data, TD-Error is used to define the priority of experiences, and training focuses on the data that can provide greater assistance to the model. TD-Error reflects the deviation between the model output and the true value. By selectively sampling data with high TD-Error, the model performance is strengthened while reducing the proportion of low-quality data.

[0091] Calculate the priority priority of the data i The formula for calculating the priority priority and the sampling rate P(i) of the data is as follows, where σ i represents the value of TD-Error:

[0092] priority i = |σ i |

[0093]

[0094] The schematic diagram of the SumTree data structure is shown in Figure 5 Using the SumTree data structure to store the sampling rate of data can efficiently save the search time. The SumTree belongs to a binary tree type of data structure. All leaf nodes are used to store the data sampling rate P(i), and the value of all parent nodes is the sum of the values of their child nodes.

[0095] Step 4.4 Dynamic exploration strategy enhancement is specifically implemented as follows:

[0096] The core idea of the dynamic exploration strategy is to dynamically adjust the exploration rate according to the training progress or environmental feedback, rather than using a fixed exploration rate (such as the fixed ε in ε-greedy). Common dynamic exploration strategies include:

[0097] Decaying ε-greedy: Gradually decrease ε as the number of training steps increases.

[0098] Exploration based on uncertainty: Adjust the exploration rate according to the uncertainty of the model prediction.

[0099] Exploration based on entropy: Use the entropy value of the policy to adjust the exploration rate.

[0100] Exploration based on priority: Adjust the exploration according to the priority of samples in the experience replay buffer.

[0101] In the present invention, decaying ε-greedy is adopted. Decaying ε-greedy makes a minor modification based on ε-greedy. ε represents the trade-off between exploration and exploitation. Initially, we hope that the value of ε is high, so that we can have a high degree of exploration. It can be understood that the agent can learn more new things. As the agent learns about the environment and the future rewards, ε should decay, so that the higher Q-values found by the agent can be fully utilized. That is, as time goes by, the value of ε continuously decays and becomes smaller and smaller.

[0102] The formula for the traditional ε-greedy method is as follows:

[0103]

[0104] where a represents the action, s represents the state, A(s) represents the number of actions in the action space, is the state-action value function.

[0105] The probability of choosing the greedy action is greater than or equal to the probability of choosing any other action. When ε = 0, it becomes the greedy strategy; when ε = 1, it becomes a uniform distribution, and actions are randomly selected from the action space.

[0106] For the decaying ε-greedy method, assume that there are N episodes in the training. The algorithm initially sets ε = p init (for example, p init = 0.7), and then gradually decreases to ε = p end (for example, p init = 0.1) as the training progresses. Specifically, in the initial process of training, let the model explore more freely with a high probability, and then gradually decrease at a rate of γ as the training progresses, increasing the probability of full exploitation and decreasing the probability of free exploration. The formula is as follows:

[0107]

[0108] ε ← (p init - p end )γ + p end

[0109] step5. Use the final model to complete path planning.

[0110] After the above steps, use the trained final model to complete path planning.

Claims

1. A dual deep Q network (DDQN) dynamic path planning method integrating attention mechanism, characterized in that: The following steps are involved: Constructing an environmental model for path planning, wherein the environmental model is a two-dimensional grid network map including a starting point, a target point, a feasible area and obstacles; Use Markov decision process (MDP) to model the path planning problem and define the state set, action set, reward set and state transition probability; Building a path planning model based on the DDQN algorithm, the model includes a main network and a target network, and reducing the overestimation problem of the Q value through a dual network structure; Introducing the attention mechanism into the DDQN model to enhance the network’s ability to focus on key environmental information and improve the decision-making accuracy of path planning; Adopt an adaptive reward mechanism to dynamically adjust the weights of path length, collision avoidance, and energy consumption to ensure the safety and efficiency of path planning; Use the priority experience replay strategy to give priority to high-value experience samples and accelerate model convergence; Design a dynamic exploration strategy to adaptively adjust the exploration rate according to the training progress and balance the relationship between exploration and utilization; Use the trained model to perform path planning and output the optimal path from the starting point to the target point.

2. The dynamic path planning method according to claim 1, characterized in that: The environmental model is a 25×25 grid network map, and each grid is divided into a starting grid, a feasible grid, an obstacle grid and a terminal grid.

3. The dynamic path planning method according to claim 1, characterized in that: The attention mechanism highlights the most critical information for path planning by calculating the attention weight of environmental information, including: Add an attention layer after the convolutional layer of the DDQN network to calculate the attention distribution of the input features; The features are weighted according to the attention weights to enhance the representation of key information.

4. The dynamic path planning method according to claim 1, characterized in that: The adaptive reward mechanism includes the following reward and penalty items: Distance reward: dynamically adjust the reward value according to the distance between the agent and the target point; Avoidance reward: dynamically adjust the reward value according to the distance between the agent and the obstacle; Arrival reward: a fixed reward is given when the agent reaches the target point; Obstacle penalty: a fixed penalty is given when the agent collides with an obstacle; Stagnation penalty: Penalty given when the agent does not move for a long time.

5. The dynamic path planning method according to claim 1, characterized in that: The priority experience playback strategy defines the priority of the experience samples by calculating TD-Error, specifically including: Use SumTree data structure to store the priority of experience samples; The experience samples are sampled non-uniformly according to the priority, and high-value samples are used for training first.

6. The dynamic path planning method according to claim 1, characterized in that: The dynamic exploration strategy adopts the decay ε-greedy method, which specifically includes: Set a higher exploration rate at the beginning of training to encourage the agent to explore the environment extensively; As training progresses, the exploration rate is gradually reduced to increase the utilization of known knowledge.

7. The dynamic path planning method according to claim 1, characterized in that: The main network and target network of the DDQN model are updated in the following way: The main network updates parameters through gradient descent; The target network synchronizes the parameters of the main network every fixed number of steps.

Images