Robot path planning method in dynamic environment

Abstract

The invention provides a robot path planning algorithm in a dynamic environment, and relates to the technical field of robot path planning, dynamic data is obtained through interaction between a robot and the environment, and the dynamic data is stored in an experience playback buffer area; the method comprises the following steps: randomly sampling batch data from a buffer area, inputting the batch data into a Dueling DQN double-flow architecture, and decomposing the batch data into a state value and a dominant function; the adaptive updating mechanism calculates a Bellman error and Q value stability based on the state value and the dominant function, and adjusts the updating frequency of the adaptive target network; and a robot path planning strategy is generated based on the updated adaptive target network, the next action of the robot is determined according to the current environment state and the robot path planning strategy until the robot arrives at the target position, path planning is completed, invalid exploration behaviors in robot path planning are effectively avoided, and the efficiency of arriving at the target point is improved.

Description

Technical Field

[0001] This invention relates to the field of robot path planning technology, specifically a robot path planning method in a dynamic environment. Background Technology

[0002] Robot path planning is a key problem in robotics, especially challenging in dynamic and complex environments. In recent years, deep reinforcement learning techniques have made significant progress in path planning, with various algorithms proposed to address the challenges in different scenarios.

[0003] Traditional path planning algorithms, such as A* and Dijkstra's algorithms, perform well in static environments. A* improves efficiency through heuristic search, while Dijkstra's algorithm guarantees finding the global optimum. However, these algorithms lack adaptability and real-time performance in dynamic environments. To overcome these limitations, researchers have introduced deep reinforcement learning into the field of path planning. Deep Q-Network (DQN) and its variants (such as DuelingDQN and Double DQN) significantly improve the efficiency and adaptability of path planning by combining neural networks and reinforcement learning. However, the performance of existing methods in dynamic environments and complex scenarios still has room for improvement.

[0004] Recent research has further improved the traditional A* algorithm, optimizing path planning in complex storage environments and effectively shortening planning time and path length. An indoor evacuation path planning model based on Dijkstra's algorithm was developed, which, by combining computer vision technology, adapts to real-time dynamic changes in crowds, improving evacuation efficiency in emergency situations. The Multi-Branch Dueling DQN (MBDDQN) algorithm, used for computational offloading and resource allocation in dynamic environments, demonstrates its advantages in handling high-dimensional state-action spaces and long-term cost optimization. Meanwhile, a dynamic path planning method combining Dueling Double Deep Q-Network (D3QN) and Priority Experience Replay (PER) shows excellent path length and travel time performance in both static and dynamic obstacle environments. Furthermore, Deguale introduced an improved deep reinforcement learning strategy, combining the Dueling architecture, Priority Experience Replay, and reward shaping techniques. This method significantly improves convergence speed, stability, and overall performance in grid worlds and Gazebo simulation environments. Similarly, the improved Dueling DQN algorithm, by combining Priority Experience Replay and artificial potential field algorithms, effectively improves the learning efficiency and convergence speed of path planning.

[0005] Although these studies have made significant progress in improving path planning performance, there is still room for improvement in their adaptability to dynamic environments and computational efficiency. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a robot path planning algorithm for dynamic environments, comprising:

[0007] Dynamic data is acquired through the interaction between the robot and the environment and stored in the experience playback buffer;

[0008] Batch data is randomly sampled from the buffer and input into the two-stream architecture of Dueling DQN, where it is decomposed into state values ​​and dominance functions;

[0009] The adaptive update mechanism calculates the Bellman error and Q-value stability based on the state value and the dominance function, and adjusts the update frequency of the adaptive target network.

[0010] The robot path planning strategy is generated based on the updated adaptive target network. The robot's next action is determined according to the current environmental state and the robot path planning strategy until the target position is reached, thus completing the path planning.

[0011] Preferably, the Q-value Q(s,a) of each action a in the current state s is calculated based on the state value V(s) and the dominance function A(s,a):

[0012]

[0013] Where |α| is the size of the action space, and a' is the next action.

[0014] Preferably, the Bellman error δ is defined as:

[0015] δ=r+γmax a' Q(s',a')-Q(s,a);

[0016] Where r is the immediate reward obtained by taking action a in the current state s, γ is the discount factor, and Q(s',a') is the Q value of action a' in the next state s'.

[0017] Preferably, by utilizing the average Bellman error mean(δ) of the current batch batch ) and historical average Bellman error mean(δ) hist The ratio w bellman To represent the sudden change in Bellman error:

[0018]

[0019] Preferably, the Q-value stability w value It is measured by the ratio of the standard deviation std(Q) to the mean (|Q|):

[0020]

[0021] Preferably, the comprehensive index w is calculated:

[0022] w = 0.7 × w bellman +0.3×w value ;

[0023] The update interval t is calculated using the following formula:

[0024] t = t base ×(1+tanh(β×(a0-a));

[0025] Among them, t base β is the basic update interval, a0 is the sensitivity parameter, and a is the target stability threshold.

[0026] Preferably, the path planning strategy evaluates the Q-values ​​of different actions based on the updated adaptive target network, selects the action with the largest Q-value as the optimal action for the robot's next step, and determines the robot's movement direction or operation in the environment; in a dynamic environment, the Q-values ​​of each action are repeatedly evaluated, and the robot gradually moves from the starting position to the target position until it reaches the target position, forming a complete path.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The proposed Adaptive DQN algorithm demonstrates significant advantages in robot path planning tasks. By introducing an adaptive update mechanism and the dual-stream architecture of DuelingDQN, this algorithm outperforms the traditional FixedDQN and DuelingDQN on several key performance indicators. Experiment 1 shows that Adaptive DQN improves average reward and average steps by 88.11% and 95.72%, respectively, while reducing average Bellman error and average Q-value stability by 84.25% and 74.71%, respectively. Experiment 2 shows significant improvements in route selection success rate and path length. These results demonstrate the effectiveness of Adaptive DQN in improving the efficiency and stability of path planning.

[0029] The adaptive update mechanism achieves a good balance between stability and response speed by dynamically adjusting the update frequency. DuelingDQN's two-stream architecture significantly improves the accuracy of Q-value estimation by separating the state value and the dominance function. These innovations enable adaptive DQN to quickly adapt to dynamic environments and make better decisions. Future research will further optimize the adaptive update mechanism, explore its application potential in more complex environments, and continue to improve the algorithm's stability and computational efficiency to meet the needs of practical robotic systems. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the adaptive DQN algorithm workflow;

[0031] Figure 2 This is a schematic diagram of the dual-stream architecture of DuelingDQN;

[0032] Figure 3 A comparison of the performance parameters of three DQN algorithms;

[0033] Figure 4 A visual path comparison of three DQN algorithms. Detailed Implementation

[0034] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0035] like Figure 1 As shown, the adaptive DQN algorithm combines the DuelingDQN two-stream architecture with an adaptive update mechanism, improving the efficiency and stability of path planning. Initially, interactive data from the agent environment's dynamics is stored in an experience replay buffer. Subsequently, a batch of data is randomly sampled from this buffer to update the Q-network and the adaptive target network, ensuring training stability. The DuelingDQN two-stream architecture improves estimation accuracy by decomposing Q-value estimation into state values ​​and a dominant stream. The losses of the Q-network and the DuelingDQN two-stream architecture are calculated, and the adaptive target network is initialized with the same weights as the Q-network, remaining unchanged until the adaptive update mechanism triggers an update. Crucially, the adaptive update mechanism adjusts the update frequency of the adaptive target network based on the Bellman error and Q-value stability. If instability is detected in the adaptive target network, the update frequency is increased to improve responsiveness; conversely, a stable adaptive target network can have its frequency reduced to decrease computational cost. When predetermined conditions are met, the weights of the Q-network are propagated to the adaptive target network, ensuring that the adaptive target network incorporates the latest learning results from the Q-network while maintaining overall stability. This design allows the adaptive DQN algorithm to effectively balance exploration and development in dynamic environments, improving the success rate and efficiency of path planning.

[0036] Quantitative calculation of Bellman error and Q-value stability

[0037] In reinforcement learning, Bellman error is a key metric that measures the difference between the current Q-value estimate and the target Q-value.

[0038] Bellman error δ is defined as:

[0039] δ=r+γmax a' -Q(s,a);

[0040] Where r is the immediate reward obtained by taking action a in the current state s, γ is a discount factor used to balance the weights of immediate reward and future reward, and Q(s',a') is the maximum expected future reward in the next state s' under action a'.

[0041] By utilizing the average Bellman error mean(δ) of the current batch batch ) and historical average Bellman error mean(δ) hist The ratio w bellman To represent the sudden change in Bellman error:

[0042]

[0043] Q-value stability w value Q-value Stability is measured by the ratio of the standard deviation (std(Q)) to the mean (|Q|). The specific formula is:

[0044]

[0045] This metric reflects the volatility of the Q-value estimate. A lower stability value implies a more stable Q-value estimate. Combining these two metrics, a composite metric w is calculated to dynamically adjust the update interval of the target network:

[0046] w = 0.7 × w bellman +0.3×w value ;

[0047] In this way, the adaptive DQN algorithm can dynamically adjust the update frequency of the target network according to the current training state, thereby improving the efficiency and stability of path planning.

[0048] Adaptive update mechanism

[0049] The adaptive update mechanism proposed in this invention aims to improve the training stability and convergence speed of DQN by dynamically adjusting the update frequency of the target network. This mechanism comprehensively considers Bellman error and Q-value stability, and calculates the update interval using the following formula:

[0050] t = t base ×(1+tanh(β×(a0-a));

[0051] Where t is the calculated new target network update interval, t baseHere, w is the base update interval, β is a sensitivity parameter controlling response intensity, a0 is the target stability threshold (the ideal value of the overall index), and a is the current overall stability index. In this way, Adaptive DQN can dynamically adjust the update frequency of the target network during training to adapt to the current learning state. When the overall index w is high (i.e., the model is unstable), the update interval is shortened; when the overall index w is low (i.e., the model is stable), the update interval is lengthened. This dynamic adjustment mechanism helps to achieve a balance between stability and response speed, thereby improving training efficiency and path planning performance.

[0052] like Figure 2 The adaptive DQN algorithm shown adopts the two-stream architecture of DuelingDQN, which decomposes the Q value into the state value V(s) and the advantage function A(s,a), thereby improving the accuracy and stability of Q value estimation.

[0053] DuelingDQN's two-stream architecture consists of the following two main parts:

[0054] State value streams are used to estimate the value of the current state s, regardless of the specific action. Its output, V(s), reflects the expected reward of taking any action in state s.

[0055] The advantage function stream is used to estimate the advantage of taking a specific action a in the current state s. Its output is represented as A(s,a), which reflects the advantage of action a over other actions in state s.

[0056] The final Q value is calculated by combining the state value and the dominance function, as shown in the following formula:

[0057]

[0058] Here, |α| represents the size of the action space. In this way, Dueling DQN can estimate the Q-value more accurately, thereby improving path planning performance.

[0059] In robot path planning tasks, generating a path planning strategy using an updated target network is a crucial step in achieving effective path planning. Specifically, this strategy evaluates the Q-values ​​of different actions based on the updated target network. For the current environmental state, the target network outputs the Q-values ​​for each possible action. The action with the highest Q-value is selected as the robot's optimal next action, thus determining the robot's direction of movement or operation within the environment. This process is repeated, allowing the robot to gradually move from its starting position towards the target position until it reaches the target, forming a complete path and achieving path planning.

[0060] In dynamic environments, the environmental state may change at any time, such as the appearance of new obstacles or changes in the target position. In this case, the path planning strategy generated by the robot based on the updated target network can quickly adapt to environmental changes, re-evaluate the Q value of each action, select the optimal action to adjust the path, and ensure that the robot successfully reaches the target position.

[0061] When the robot detects a new obstacle, it interacts with the environment to acquire new dynamic data, which is stored in an experience replay buffer. Then, batch data is randomly sampled from the buffer and input into a two-stream architecture using an adaptive update mechanism and Dueling DQN. The target network update frequency is dynamically adjusted based on Bellman error and Q-value stability, adaptively updating the target network. When update conditions are met, the Q-network weights are synchronized to the target network. At this point, a new path planning strategy is generated using the updated target network, and the robot's next action is determined based on the current environmental state (including the location of new obstacles) and the strategy. The robot may choose to turn, detour, or take other suitable actions to avoid obstacles and continue moving towards the target location. By continuously repeating this process, the robot can adapt to dynamic environmental changes and eventually reach the target location, completing the path planning task.

[0062] Table 1 shows the configuration parameters of the experimental hardware and software platform.

[0063] Table 1. Configuration parameters of the experimental hardware and software platform

[0064]

[0065]

[0066] This invention utilizes the CartPole-v1 and FrozenLake-v1 environments in OpenAI Gym for experiments.

[0067] Experiment 1 evaluates the proposed algorithm in the CartPole-v1 environment of OpenAI Gym, where the agent must balance levers to obtain the highest possible reward. Comparison algorithms include the original DQN, Dueling DQN with a fixed update interval, and the proposed Adaptive DQN. Each algorithm is trained for 200 epochs, and evaluation metrics include average reward, Bellman error, and Q-value stability. To ensure fair comparison, all algorithms use the same network architecture and hyperparameter settings in the experiment. The network architecture is a three-layer fully connected neural network with two hidden layers (128 neurons each) and one output layer. The number of neurons in the output layer is the same as the action space size. Training uses the Adam optimizer with an initial learning rate of 0.001. A discount factor of 0.99 is used to balance the weights of immediate and future rewards. Each training update uses 64 samples. The target network update strategies are as follows: Fixed DQN and Dueling DQN update the target network every 100 steps, while the update interval of Adaptive DQN is dynamically adjusted by an adaptive mechanism, with an initial interval of 100 steps. The experience replay buffer has a capacity of 10,000 and is used to store the agent's experience interacting with the environment. The exploration strategy employs an ε-greedy approach, with an initial ε value of 0.5, which gradually decays to 0.01 during training to balance exploration and exploitation. Each algorithm is trained for 200 epochs, with a maximum of 500 steps per epoch. These uniform parameter settings ensure effective comparison of experimental results.

[0068] Experiment 2 was conducted in the FrozenLake-v1 environment within OpenAI Gym. This environment is an 8×8 grid where the agent needs to move across the grid and obtain the highest possible reward to reach a predetermined endpoint. The agent's state represents its position within the grid, and the action space includes four possible movement directions (up, down, left, and right). The environment is randomized, and the agent may slide into adjacent cells when performing actions. The same experimental parameters as in Experiment 1 were used. The training duration for each algorithm was changed from 200 epochs to 1000 epochs. These parameters and settings ensured the reproducibility of the experiments and the validity of the results. By comparing the performance of different algorithms, the superiority and stability of Adaptive DQN in path planning tasks were verified.

[0069] The results of Experiment 1 are as follows Figure 3 As shown in Figure 1 and Table 2, Figure 3This section presents the results of Experiment 1 in the CartPole-v1 environment of OpenAI Gym. The experiment compares the performance of three algorithms: the original DQN, Dueling DQN with a fixed update interval, and the proposed Adaptive DQN. The horizontal axis represents the number of training epochs, and the vertical axis represents three key metrics: average reward, average Bellman error, and average Q-value stability. As shown in the figure, Adaptive DQN significantly outperforms the other two algorithms in average reward, indicating that it can achieve higher cumulative rewards in path planning tasks, i.e., complete the task more efficiently. Adaptive DQN achieves significant improvements in the key metrics of average reward, average Bellman error, and average Q-value stability. Its average reward reaches 23.595, a substantial increase of 88.11% compared to 12.545 for Fixed DQN, and also 36.58% higher than 17.275 for Dueling DQN. Adaptive DQN reduced the average Bellman error to 0.00714346, a significant decrease of 84.25% compared to Fixed DQN's 0.0453862, and also a reduction of 82.91% compared to Dueling DQN's 0.0418010. The average Q-value stability also decreased to 0.0198403, a substantial reduction of 74.71% compared to Fixed DQN's 0.0784973, and a reduction of 58.42% compared to Dueling DQN's 0.0477154. Although Adaptive DQN experienced more training crashes (50) than Fixed DQN (23) and Dueling DQN (46), this is likely due to its higher exploration rate and more complex adaptive mechanism. Overall, the superior performance of Adaptive DQN in terms of average reward, average Bellman error, and average Q-value stability demonstrates its enhanced efficiency and stability in robotic path planning tasks.

[0070] The results of Experiment 2 are as follows Figure 4 As shown, Figure 4 This paper presents a comparison of the results of Experiment 2 in the OpenAI Gym's FrozenLake-v1 environment, focusing on the performance of different algorithms in terms of success rate and path length. FrozenLake-v1 is an 8×8 grid environment where the agent needs to move on the grid and reach a set destination. Random ice slippage and other unexpected situations may occur, increasing the difficulty and dynamism of the task. In terms of success rate, Adaptive DQN achieved the best result with 0.014. Regarding path length, Adaptive DQN had the shortest path length at 29, followed by Fixed DQN at 33, and Dueling DQN had the longest at 49. In summary, the Adaptive DQN algorithm has a significant advantage in path planning experiments, as shown in Table 2, which compares the performance of different algorithms.

[0071] Table 2 Performance Comparison of Different Algorithms

[0072]

[0073] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A robot path planning algorithm in a dynamic environment, characterized in that, include: Dynamic data is acquired through the interaction between the robot and the environment and stored in the experience playback buffer; Batch data is randomly sampled from the buffer and input into the two-stream architecture of Dueling DQN, where it is decomposed into state values ​​and dominance functions; The adaptive update mechanism calculates the Bellman error and Q-value stability based on the state value and the dominance function, and adjusts the update frequency of the adaptive target network. The robot path planning strategy is generated based on the updated adaptive target network. The robot's next action is determined according to the current environmental state and the robot path planning strategy until the target position is reached, thus completing the path planning.

2. The robot path planning algorithm in a dynamic environment according to claim 1, characterized in that, Calculate the Q-value Q(s,a) of each action a in the current state s based on the state value V(s) and the dominance function A(s,a): Where |α| is the size of the action space, and a' is the next action.

3. The robot path planning algorithm in a dynamic environment according to claim 2, characterized in that, Bellman error δ is defined as: δ=r+γmax a' Q(s',a')-Q(s,a); Where r is the immediate reward obtained by taking action a in the current state s, γ is the discount factor, and Q(s',a') is the Q value of action a' in the next state s'.

4. The robot path planning algorithm in a dynamic environment according to claim 3, characterized in that, By utilizing the average Bellman error mean(δ) of the current batch batch ) and historical average Bellman error mean(δ) hist The ratio w bellman To represent the sudden change in Bellman error:

5. The robot path planning algorithm in a dynamic environment according to claim 4, characterized in that, Q-value stability w value It is measured by the ratio of the standard deviation std(Q) to the mean (|Q|):

6. The robot path planning algorithm in a dynamic environment according to claim 5, characterized in that, Calculate the comprehensive index w: w=0.7×w bellman +0.3×w value ; The update interval t is calculated using the following formula: t=t base ×(1+tanh(β×(a0-a)); Among them, t base β is the basic update interval, a0 is the sensitivity parameter, and a is the target stability threshold.

7. The robot path planning algorithm in a dynamic environment according to claim 6, characterized in that, The path planning strategy evaluates the Q-values ​​of different actions based on the updated adaptive target network, selects the action with the largest Q-value as the optimal action for the robot's next step, and determines the robot's movement direction or operation in the environment. In a dynamic environment, the Q-values ​​of each action are evaluated repeatedly, and the robot gradually moves from the starting position to the target position until it reaches the target position, forming a complete path.

Images