A robot path planning method and system based on improved deep Q-network

Abstract

This invention relates to the fields of artificial intelligence and robotics, and discloses a robot path planning method and system based on an improved deep Q-network. First, a state space, an action space, and a reward function containing an exploration reward related to the robot's starting distance are constructed. A deep Q-network architecture containing a normal experience replay pool and a high-reward experience replay pool is initialized. Based on a local map update strategy, a local grid map of a preset size is generated centered on the robot's current position as the current local map. During training, exploration data is recorded using a temporary experience pool. When the total reward for the current round refreshes the historical high, the data for that round is transferred to the high-reward experience replay pool for priority learning. This invention reduces the input dimensionality by using the current local map, improves sample utilization by utilizing the high-reward experience replay pool, and effectively solves the sparse reward problem by combining the improved reward function, significantly improving the robot's path planning efficiency and success rate.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and robotics, and in particular to a robot path planning method and system based on an improved deep Q-network. Background Technology

[0002] Currently, path planning is one of the core technologies for autonomous navigation of mobile robots. Its goal is to enable robots to quickly and safely find a collision-free path from the starting point to the destination in complex environments. Traditional path planning algorithms, such as the A* algorithm and the artificial potential field method, mostly rely on pre-built, complete environmental models. However, in practical applications, environmental information is often unknown or only partially known, making it difficult to directly apply traditional algorithms.

[0003] In recent years, reinforcement learning (RL), especially deep reinforcement learning (such as deep Q-networks, DQN), has shown great potential in the field of robot path planning due to its ability to learn autonomously and make decisions without requiring an environment model. However, the classic DQN algorithm still faces the following technical problems when applied to path planning: One drawback is low training efficiency and slow convergence: traditional methods typically use the entire environment map as input to the deep neural network. When the map is large or the environment is complex, the network input dimension is high, resulting in a huge amount of computation and a very long training time.

[0004] Secondly, the learning stability is poor: although existing local path planning methods can reduce the input dimension, the map changes frequently as the robot moves, resulting in unstable environmental information, which increases the learning difficulty of the neural network and affects the convergence stability.

[0005] Thirdly, the sample utilization rate is low: DQN's experience replay pool performs uniform random sampling of all historical experience samples, which cannot distinguish between high-value experiences (such as the experience of successfully reaching the endpoint) and low-value experiences, resulting in insufficient learning of key samples, reducing learning efficiency and the quality of the final strategy.

[0006] Fourthly, the reward signal is sparse: Traditional reward functions are designed simply, giving positive rewards only when the goal is reached and negative rewards when colliding with obstacles. This results in a sparse reward signal in the early stages of exploration, making it difficult for the agent to obtain effective directional guidance. The agent tends to linger near the starting point and learns slowly. Summary of the Invention To address the technical problems of low training efficiency, unstable convergence, low sample utilization, and slow learning in sparse reward environments in the prior art, this invention provides a robot path planning method and system based on an improved deep Q-network.

[0007] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: A robot path planning method based on an improved deep Q-network includes the following steps: S1. Construct the state space, action space, and reward function for robot path planning; the reward function includes an exploration reward item related to the robot's starting distance. S2. Initialize the improved deep Q-network, which includes an evaluation network, a target network, a normal experience replay pool, a high-reward experience replay pool, and a temporary experience pool. S3. Train the evaluation network, with each training round including: S3.1 Initialize the robot's position, determine the current local map based on the robot's current position according to the local map update strategy, and generate the current state based on the state space according to the current local map and the robot's position information in the current local map; S3.2. Input the current state into the evaluation network, and the evaluation network outputs Q values ​​corresponding to each action in the action space; based on the Q values ​​output by the evaluation network, use an exploration strategy to select a target action to be executed from the action space, control the robot to execute the target action and move to a new coordinate; calculate the immediate reward after the execution of this action according to the reward function, and obtain the next state and termination flag based on the state space according to the new coordinates; store the current experience sample in both the ordinary experience replay pool and the temporary experience pool, and the current experience sample includes the current state, target action, immediate reward, next state and termination flag; S3.3 During the training process, a batch of historical experience samples are randomly sampled from the ordinary experience replay pool and the high-reward experience replay pool according to a preset ratio. The target Q value corresponding to each historical experience sample is calculated based on the target network, and the current Q value corresponding to each historical experience sample is calculated using the evaluation network. The loss function is calculated based on the target Q value and the current Q value, and the network parameters of the evaluation network are updated based on the loss function. S3.4. Every preset number of steps, copy the network parameters of the evaluation network to the target network; S3.5 Determine whether the current training round has ended. If the robot meets the preset termination condition, the current round is determined to be over. Calculate the total reward value for the current round. If the total reward value for the current round is greater than the historical highest total reward value, transfer all experience samples stored in the temporary experience pool for the current round to the high reward experience replay pool, update the historical highest total reward value, and clear the temporary experience pool; otherwise, take the next state of the current round as the new current state and return to step S3.2. S4. Iterate through step S3 until the iteration termination condition is met, complete the training of the improved deep Q network, obtain the trained evaluation network, and use the trained evaluation network to perform path planning for the robot.

[0008] A robot path planning system based on an improved deep Q-network includes the following modules: The path planning construction module is used to construct the state space, action space, and reward function for robot path planning; the reward function includes an exploration reward item related to the distance from the robot's starting point. An initialization module is used to initialize an improved deep Q-network, which includes an evaluation network, a target network, a normal experience replay pool, a high-reward experience replay pool, and a temporary experience pool. The training module is used to train the evaluation network, and each training round includes: S3.1 Initialize the robot's position, determine the current local map based on the robot's current position according to the local map update strategy, and generate the current state based on the state space according to the current local map and the robot's position information in the current local map; S3.2. Input the current state into the evaluation network, and the evaluation network outputs Q values ​​corresponding to each action in the action space; based on the Q values ​​output by the evaluation network, use an exploration strategy to select a target action to be executed from the action space, control the robot to execute the target action and move to a new coordinate; calculate the immediate reward after the execution of this action according to the reward function, and obtain the next state and termination flag based on the state space according to the new coordinates; store the current experience sample in both the ordinary experience replay pool and the temporary experience pool, and the current experience sample includes the current state, target action, immediate reward, next state and termination flag; S3.3 During the training process, a batch of historical experience samples are randomly sampled from the ordinary experience replay pool and the high-reward experience replay pool according to a preset ratio. The target Q value corresponding to each historical experience sample is calculated based on the target network, and the current Q value corresponding to each historical experience sample is calculated using the evaluation network. The loss function is calculated based on the target Q value and the current Q value, and the network parameters of the evaluation network are updated based on the loss function. S3.4. Every preset number of steps, copy the network parameters of the evaluation network to the target network; S3.5 Determine whether the current training round has ended. If the robot meets the preset termination condition, the current round is determined to be over. Calculate the total reward value for the current round. If the total reward value for the current round is greater than the historical highest total reward value, transfer all experience samples stored in the temporary experience pool for the current round to the high reward experience replay pool, update the historical highest total reward value, and clear the temporary experience pool; otherwise, take the next state of the current round as the new current state and return to step S3.2. The path planning module is used to plan the path for the robot using a trained evaluation network.

[0009] Compared with existing technologies, its advantages are as follows: This invention employs an adaptive sliding map mechanism, which generates a local grid map of a preset size centered on the robot's current global position. This mechanism reduces the dimensionality of the high-dimensional global map input to a low-dimensional local map input, significantly reducing the computational load of the neural network and greatly improving training efficiency. Furthermore, this invention designs a high-reward experience replay pool, ensuring that high-value experiences are prioritized and repeatedly learned, improving sample utilization efficiency, guiding the robot to learn effective strategies more quickly, and enhancing sample utilization and decision quality. Finally, this invention introduces an exploration reward term related to the robot's starting distance into the reward function, providing continuous and clear directional guidance for the robot in the early stages of exploration, preventing the robot from lingering at the starting point, encouraging it to explore unknown areas, effectively avoiding local optima and starting point lingering problems, accelerating the learning process, and alleviating the sparse reward problem.

[0010] Furthermore, the adaptive sliding map mechanism of this invention also includes a boundary-triggered update mechanism. That is, a new local map is generated only when the robot moves to the boundary of the current local map and is about to leave that local area, centered on the robot's current global position. This provides the robot with a relatively static learning environment, avoiding the interference caused by frequent environmental changes in traditional local planning, making the training process more stable and the convergence speed faster. The adaptive sliding map mechanism and the boundary-triggered update mechanism together constitute the local map update strategy of this invention. Attached Figure Description

[0011] Figure 1 This is a flowchart of a robot path planning method based on an improved deep Q-network provided by the present invention; Figure 2 This is a structural design diagram of the improved deep Q-network provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the traditional local path planning map transformation mechanism provided in Embodiment 1 of the present invention; Figure 4This is a schematic diagram of the adaptive sliding map transformation mechanism provided in Embodiment 1 of the present invention; Figure 5 This is a comparison chart of the training reward curves of the method of the present invention provided in Embodiment 2 of the present invention and other algorithms on five sets of random maps; Figure 6 This is a comparison chart of the training step count curves of the method of the present invention provided in Embodiment 2 and other algorithms on five sets of random maps; Figure 7 This is a bar chart comparing the training time of the method of the present invention provided in Embodiment 2 with other algorithms on five sets of random maps; Figure 8 This is a comparison chart of the final path planning results of the method of the present invention provided in Embodiment 2 of the present invention and other algorithms on five sets of random maps. Detailed Implementation

[0012] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] Example 1 like Figure 1 As shown, this embodiment provides a robot path planning method based on an improved deep Q-network, including the following steps: S1. Construct the state space, action space, and reward function for robot path planning; the reward function includes an exploration reward related to the distance from the robot's starting point. S2. Initialize the improved deep Q-network, which includes an evaluation network, a target network, a normal experience replay pool, a high-reward experience replay pool, and a temporary experience pool. S3. Train the evaluation network. Each training round includes: S3.1 Initialize the robot's position. Based on the local map update strategy, determine the current local map according to the robot's current position. Based on the current local map and the robot's position information in the current local map, generate the current state based on the state space. S3.2 Input the current state into the evaluation network, and the evaluation network outputs the Q-values ​​corresponding to each action in the action space; based on the Q-values ​​output by the evaluation network, select the target action to be executed from the action space using an exploration strategy, control the robot to execute the target action and move to the new coordinates; calculate the immediate reward after the execution of this action according to the reward function, and obtain the next state and termination flag based on the state space according to the new coordinates; store the current experience sample in both the ordinary experience replay pool and the temporary experience pool. The current experience sample includes the current state, target action, immediate reward, next state, and termination flag. S3.3 During training, a batch of historical experience samples are randomly sampled from the ordinary experience replay pool and the high reward experience replay pool according to a preset ratio. The target Q value corresponding to each historical experience sample is calculated based on the target network, and the current Q value corresponding to each historical experience sample is calculated using the evaluation network. The loss function is calculated based on the target Q value and the current Q value, and the network parameters of the evaluation network are updated based on the loss function. S3.4. Every preset number of steps, copy the network parameters of the evaluation network to the target network; S3.5 Determine whether the current training round has ended. If the robot meets the preset termination condition, the current round is determined to be over. Calculate the total reward value of the current round. If the total reward value of the current round is greater than the historical highest total reward value, transfer all experience samples of the current round stored in the temporary experience pool to the high reward experience replay pool, update the historical highest total reward value, and clear the temporary experience pool; otherwise, take the next state of the current round as the new current state and return to step S3.2. S4. Iterate through step S3 until the iteration termination condition is met, complete the training of the improved deep Q network, obtain the trained evaluation network, and use the trained evaluation network to plan the path for the robot.

[0014] Step S1: Construct the state space, action space, and reward function for robot path planning; To address the reward sparsity problem, this invention designs a novel reward function. Building upon the existing positive reward (+100) for the endpoint and negative reward (-10) for obstacles / boundaries, it introduces an exploration reward term related to the robot's distance from the starting point. This exploration reward term is constructed based on the Manhattan distance between the robot's current position and the starting point. At each step, the robot receives a penalty reward based on its Manhattan distance from the starting point. The exploration reward range is [-0.95, -0.01], encouraging the robot to quickly move away from the starting point and explore further, avoiding ineffective wandering near the starting point and providing effective gradient guidance for early exploration.

[0015] The reward function also includes a positive reward for reaching the target point and a negative reward for colliding with obstacles or map boundaries. The expression for the reward function is:

[0016] in, r For the reward function; x 1. y 1 represents the coordinates of the robot's starting position; x 2. y 2 represents the coordinates of the robot's current position; , where is the Manhattan distance between the robot's current position and the starting point.

[0017] Step S2 initializes the improved deep Q-network, which includes an evaluation network, a target network, a normal experience replay pool, a high-reward experience replay pool, and a temporary experience pool.

[0018] In this embodiment, the structural design diagram of the improved deep Q-network is as follows: Figure 2 As shown, the input layer has two channels, receiving the current local map information and the robot's position information within the current local map, respectively. The hidden layer consists of three convolutional layers (kernel sizes of 8x8, 4x4, and 3x3, with 32, 64, and 64 filters respectively) and two fully connected layers (dimensions of 64 and 32 respectively). The output layer has four neurons, corresponding to the Q-values ​​of the action space (up, down, left, and right).

[0019] The regular experience replay pool and the high-reward experience replay pool share the total storage space and are implemented through logical partitioning.

[0020] Step S3 trains the evaluation network, with each training round including: S3.1 Initialize the robot's position. Based on the local map update strategy, determine the current local map according to the robot's current position. Based on the current local map and the robot's position information in the current local map, generate the current state based on the state space. This invention employs an adaptive sliding map mechanism, which generates a local grid map of a preset size centered on the robot's current global position, serving as the current local map. Furthermore, this adaptive sliding map mechanism uses boundary-triggered updates, rather than time-triggered updates. The boundary-triggered update means that when the robot moves to the boundary of the current local map, a new local map is regenerated centered on the robot's current global map position. Further, if the robot is at the edge of the global map, preventing the generation of a complete current local map centered on itself, the center of the current local map is adjusted inwards to ensure that the current local map always remains within the global map's range. The adaptive sliding map mechanism and the boundary-triggered strategy together constitute the local map update strategy of this invention.

[0021] like Figure 3 and Figure 4 As shown, Figure 3 This is a schematic diagram of a traditional local path planning map transformation mechanism. The diagram shows that as the robot moves, the local map is updated continuously in real time with the robot as the center, and the environment changes frequently. Figure 4 This is a schematic diagram of the adaptive sliding map transformation mechanism used in this invention. The diagram shows that the map remains unchanged when the robot moves within a local map; map updates are only triggered when the robot moves to the boundary (dashed box). Figure 4 The principle of boundary-triggered updates is clearly explained.

[0022] The adaptive sliding map mechanism combines local path planning concepts with global path planning objectives. Instead of processing the entire global map each time, it extracts a fixed-size local grid map around the robot's current position as the state input to the neural network. This significantly reduces the input dimensionality of the neural network and greatly improves the stability of learning by providing a relatively static view of the environment.

[0023] S3.2 Input the current state into the evaluation network, and the evaluation network outputs the Q-values ​​corresponding to each action in the action space; based on the Q-values ​​output by the evaluation network, select the target action to be executed from the action space using an exploration strategy, control the robot to execute the target action and move to the new coordinates; calculate the immediate reward after the execution of this action according to the reward function, and obtain the next state and termination flag based on the state space according to the new coordinates; store the current experience sample in both the ordinary experience replay pool and the temporary experience pool. The current experience sample includes the current state, target action, immediate reward, next state, and termination flag. S3.3 During training, a batch of historical experience samples are randomly sampled from the ordinary experience replay pool and the high reward experience replay pool according to a preset ratio. The target Q value corresponding to each historical experience sample is calculated based on the target network, and the current Q value corresponding to each historical experience sample is calculated using the evaluation network. The loss function is calculated based on the target Q value and the current Q value, and the network parameters of the evaluation network are updated based on the loss function. S3.4. Every preset number of steps, copy the network parameters of the evaluation network to the target network; S3.5 Determine whether the current training round has ended. If the robot meets the preset termination condition, the current round is determined to be over. Calculate the total reward value of the current round. If the total reward value of the current round is greater than the historical highest total reward value, transfer all experience samples of the current round stored in the temporary experience pool to the high reward experience replay pool, update the historical highest total reward value, and clear the temporary experience pool; otherwise, take the next state of the current round as the new current state and return to step S3.2.

[0024] In this embodiment, the preset termination conditions include the robot reaching the target point, the robot colliding with another robot, the robot crossing the boundary, and the number of steps in this training round reaching the preset maximum number of steps.

[0025] This invention adds a high-reward experience replay pool to the traditional single experience replay pool. After each episode of exploration, the robot calculates the total reward value obtained in that episode using a reward function. If the total reward value of the current episode exceeds the highest total reward value in the historical record, all experience samples (states, actions, rewards, and next states) generated in that episode are additionally stored in the high-reward experience replay pool. During training sampling, a small batch of samples is randomly sampled simultaneously from both the ordinary experience pool and the high-reward experience pool according to a preset ratio for network updates. Preferably, the sampling proportion of the ordinary experience replay pool is greater than the sampling proportion of the high-reward experience replay pool.

[0026] This mechanism effectively prioritizes high-value experiences, ensuring that these crucial learning samples can be repeatedly studied, thereby improving sample utilization and learning speed.

[0027] Based on the target Q-value and the current Q-value, the loss function expression is as follows:

[0028]

[0029] in, The loss function is... N The number of samples collected; The current Q value is determined by the evaluation network based on the current state. and actions Calculations show that The parameters of the evaluation network; The target value; r j For instant rewards; γ Discount factor; The target Q value is calculated by the target network based on the next state and the target action; s' j The next state; a' For the target action; These are the parameters of the target network; d j This is a termination flag; when a sample is in a terminated state, d j =1, at which point the target value degenerates into y j = r j .

[0030] Step S4 iteratively executes step S3 until the iteration termination condition is met, completing the training of the improved deep Q network and obtaining the trained evaluation network. The trained evaluation network is then used to plan the path for the robot.

[0031] Example 2 Based on the robot path planning method based on an improved deep Q-network described in Example 1, this example provides a simulation experiment of the method, including the following steps: S1. Construct the state space, action space, and reward function for robot path planning; the reward function includes an exploration reward related to the distance from the robot's starting point. S2. Initialize the improved deep Q-network, which includes an evaluation network, a target network, a normal experience replay pool, a high-reward experience replay pool, and a temporary experience pool. S3. Train the evaluation network. Each training round includes: S3.1 Initialize the robot's position. Based on the local map update strategy, determine the current local map according to the robot's current position. Based on the current local map and the robot's position information in the current local map, generate the current state based on the state space. S3.2 Input the current state into the evaluation network, and the evaluation network outputs the Q-values ​​corresponding to each action in the action space; based on the Q-values ​​output by the evaluation network, select the target action to be executed from the action space using an exploration strategy, control the robot to execute the target action and move to the new coordinates; calculate the immediate reward after the execution of this action according to the reward function, and obtain the next state and termination flag based on the state space according to the new coordinates; store the current experience sample in both the ordinary experience replay pool and the temporary experience pool. The current experience sample includes the current state, target action, immediate reward, next state, and termination flag. S3.3 During training, a batch of historical experience samples are randomly sampled from the ordinary experience replay pool and the high reward experience replay pool according to a preset ratio. The target Q value corresponding to each historical experience sample is calculated based on the target network, and the current Q value corresponding to each historical experience sample is calculated using the evaluation network. The loss function is calculated based on the target Q value and the current Q value, and the network parameters of the evaluation network are updated based on the loss function. S3.4. Every preset number of steps, copy the network parameters of the evaluation network to the target network; S3.5 Determine whether the current training round has ended. If the robot meets the preset termination condition, the current round is determined to be over. Calculate the total reward value of the current round. If the total reward value of the current round is greater than the historical highest total reward value, transfer all experience samples of the current round stored in the temporary experience pool to the high reward experience replay pool, update the historical highest total reward value, and clear the temporary experience pool; otherwise, take the next state of the current round as the new current state and return to step S3.2. S4. Iterate through step S3 until the iteration termination condition is met, complete the training of the improved deep Q network, obtain the trained evaluation network, and use the trained evaluation network to plan the path for the robot.

[0032] This embodiment conducts a simulation experiment on a 20x20 two-dimensional grid map with an obstacle coverage rate of 20%. The start and end points are randomly generated within a four-grid distance from the edge of the map and are symmetrically distributed (e.g., the start point is on the left edge and the end point is on the right edge) to ensure the planarability of the path.

[0033] In this embodiment, the experimental parameters are set as follows: Discount factor γ: 0.9 Learning rate: 0.001 Initial exploration value ε: 1 Minimum exploration value ε_min: 0.1 Exploration value decay coefficient ε_decay: 0.995 Batch size BATCH_SIZE: 32 Total capacity of the experience replay pool: 10,000 (the high-reward experience pool does not have a separate limit and shares this total capacity, but is logically separate). Optimizer: Adam Maximum number of training rounds: 1000 Maximum number of steps per round: 200 Local map size: For example, set it to 10x10 (can be adjusted according to the actual environment).

[0034] On the MATLAB platform, the algorithm first determines the robot's current global position ( x , y Determine the center point of the local map. If (x, y) is at least half the side length of the local map from the map boundary, then use (x, y) as the center point. x , y A local map is cropped around the center. If it's close to the boundary, an offset is calculated to ensure the local map is cropped inwards, hugging the boundary tightly, thus ensuring the map is valid. Current state. s_t The data consists of extracted local map raster data (a two-dimensional matrix) and the robot's normalized coordinates within that local map. x_local , y_local These components together form the input to the neural network.

[0035] Initialization: Create a normal experience pool buffer_normal and temporary experience pool buffer_temp .

[0036] Data collection: Experience from each step of the robot's interaction with the environment during each training round. s,a,r,s ') are stored simultaneously buffer_normal and buffer_temp .

[0037] in, s The current state is composed of the current local map and the robot's position information within the current local map. a The action currently being performed by the robot; r The immediate reward obtained after performing this action; s The next state after performing this action is composed of the updated current local map and the robot's position.

[0038] Round End Determination: Calculate the total reward after the current round ends. total_reward .if total_reward > best_ total_reward Then buffer_temp All experience is transferred to the high-reward experience pool. buffer_high In, and updated best_total_reward Then clear buffer_temp In preparation for the next round of use.

[0039] Sampling training: Each time sampling training is needed, from buffer_normal Random sampling batch_size* (1 - α) One sample, from buffer_high Random sampling batch_size * α Each sample is merged into a single batch for network updates. Here, α is the sampling ratio, for example, set to 0.3.

[0040] Suppose the robot's current position in the global grid map is (5,5), the starting point is (5,5), and the target point is (18,18).

[0041] If the robot's next action is to move "right" to (6,5), and that position is an empty area: Calculate ds = |6-5| + |5-5| = 1, then the exploration reward is... r = -1 / 1 = -1, but due to the limiting function, the actual reward is limited to [-0.95, -0.01], therefore take r = -0.95.

[0042] If the robot moves to the obstacle (7,5) in its next step: but r = -10.

[0043] If the robot moves to the target point (18,18) in the next step: but r = +100.

[0044] If the robot moves continuously from (5,5) and stays near the starting point for a long time, it will receive a penalty of nearly -0.95 for each step, which forces it to move further away in order to seek a higher cumulative reward.

[0045] Other implementation methods: Action space: In this embodiment, the action space is a discrete four-direction space (up, down, left, right). Those skilled in the art can expand it to an eight-direction or continuous action space as needed.

[0046] Network structure: The number of convolutional and fully connected layers, as well as the size and quantity of convolutional kernels, can be adjusted according to the size of the local map and computational resources.

[0047] Application Platform: This embodiment is not limited to MATLAB simulation. The trained network model can be deployed on an actual mobile robot platform equipped with necessary sensors (such as LiDAR and depth cameras) to plan paths by processing real-time sensor data.

[0048] like Figure 5 As shown, Figure 5 The document provides a comparison of the training reward curves of the method of this invention with other algorithms on five sets of random maps. Each of the five sub-maps corresponds to a different 20x20 grid map. Each map plots the reward curve of the algorithm of this invention (RE curve) against four comparative algorithms: DQN, DoubleDQN, DuelingDQN, and PER-DQN. The figures clearly show that the curve of this invention converges faster and achieves a higher and more stable reward value.

[0049] like Figure 6 As shown, Figure 6 A comparison graph of the training step count curves of the method of this invention and other algorithms on five sets of random maps is provided. The horizontal axis represents the training episode, and the vertical axis represents the number of steps required for the robot to reach the target point. As training progresses, the number of steps required for the robot to complete the path planning task gradually decreases and tends to stabilize, indicating that the method proposed in this invention can effectively improve path planning efficiency and accelerate training convergence speed.

[0050] like Figure 7 As shown, Figure 7 A bar chart comparing the training time of the method of this invention with other algorithms on five sets of random maps is provided. In the figure, the horizontal axis represents the five different maps, and the vertical axis represents the training time. By comparing the height of the bars of different algorithms, it can be intuitively seen that the training time of the method of this invention is significantly shorter than that of the other comparison algorithms on each map, and the time fluctuation is minimal, proving its efficiency.

[0051] like Figure 8 As shown, Figure 8The following graphs provide a comparison of the final path planning results of the method of this invention with other algorithms on five sets of random maps. Figure 8 The path learned by each algorithm on each map is shown in matrix form. The method of this invention successfully and efficiently plans paths on all maps.

[0052] Example 3 This embodiment provides a robot path planning system based on an improved deep Q-network, including the following modules: The path planning module is used to construct the state space, action space, and reward function for robot path planning; the reward function includes an exploration reward related to the distance from the robot's starting point. The initialization module is used to initialize the improved deep Q-network, which includes an evaluation network, a target network, a normal experience replay pool, a high-reward experience replay pool, and a temporary experience pool. The training module is used to train the evaluation network. Each training round includes: S3.1 Initialize the robot's position. Based on the local map update strategy, determine the current local map according to the robot's current position. Based on the current local map and the robot's position information in the current local map, generate the current state based on the state space. S3.2 Input the current state into the evaluation network, and the evaluation network outputs the Q-values ​​corresponding to each action in the action space; based on the Q-values ​​output by the evaluation network, select the target action to be executed from the action space using an exploration strategy, control the robot to execute the target action and move to the new coordinates; calculate the immediate reward after the execution of this action according to the reward function, and obtain the next state and termination flag based on the state space according to the new coordinates; store the current experience sample in both the ordinary experience replay pool and the temporary experience pool. The current experience sample includes the current state, target action, immediate reward, next state, and termination flag. S3.3 During training, a batch of historical experience samples are randomly sampled from the ordinary experience replay pool and the high reward experience replay pool according to a preset ratio. The target Q value corresponding to each historical experience sample is calculated based on the target network, and the current Q value corresponding to each historical experience sample is calculated using the evaluation network. The loss function is calculated based on the target Q value and the current Q value, and the network parameters of the evaluation network are updated based on the loss function. S3.4. Every preset number of steps, copy the network parameters of the evaluation network to the target network; S3.5 Determine whether the current training round has ended. If the robot meets the preset termination condition, the current round is determined to be over. Calculate the total reward value of the current round. If the total reward value of the current round is greater than the historical highest total reward value, transfer all experience samples of the current round stored in the temporary experience pool to the high reward experience replay pool, update the historical highest total reward value, and clear the temporary experience pool; otherwise, take the next state of the current round as the new current state and return to step S3.2. The path planning module is used to plan the path for the robot using a trained evaluation network.

[0053] In summary, this invention provides a robot path planning method and system based on an improved deep Q-network. Compared with existing technologies, this invention significantly improves training efficiency through an adaptive sliding map mechanism, reducing the high-dimensional global map input to a low-dimensional local map input, thus significantly reducing the computational load of the neural network. Experiments show that, under the same environment, the training time of this invention is reduced by an average of about half compared to traditional DQN and its variants. Through the boundary-triggered update mechanism of the adaptive sliding map, this invention can improve the learning stability and convergence speed of the algorithm, providing a relatively static learning environment for the agent and avoiding the interference caused by frequent environmental changes in traditional local planning, making the training process more stable and the convergence speed faster. The high-reward experience replay pool designed in this invention ensures that high-value experiences are prioritized and repeatedly learned, improving sample utilization efficiency and guiding the agent to learn effective strategies more quickly. Experimental results show that this invention can successfully plan paths on all test maps, with a success rate superior to the compared algorithms. The exploration reward introduced in the reward function of this invention provides the agent with continuous and clear directional guidance in the early stage of exploration, encourages it to explore unknown areas, effectively avoids the problems of local optima and starting point lingering, alleviates the sparse reward problem, and accelerates the learning process.

[0054] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A robot path planning method based on an improved deep Q-network, characterized in that, Includes the following steps: S1. Construct the state space, action space, and reward function for robot path planning; the reward function includes an exploration reward item related to the robot's starting distance. S2. Initialize the improved deep Q-network, which includes an evaluation network, a target network, a normal experience replay pool, a high-reward experience replay pool, and a temporary experience pool. S3. Train the evaluation network, with each training round including: S3.1 Initialize the robot's position, determine the current local map based on the robot's current position according to the local map update strategy, and generate the current state based on the state space according to the current local map and the robot's position information in the current local map; S3.

2. Input the current state into the evaluation network, and the evaluation network outputs Q values ​​corresponding to each action in the action space; based on the Q values ​​output by the evaluation network, use an exploration strategy to select a target action to be executed from the action space, control the robot to execute the target action and move to a new coordinate; calculate the immediate reward after the execution of this action according to the reward function, and obtain the next state and termination flag based on the state space according to the new coordinates; store the current experience sample in both the ordinary experience replay pool and the temporary experience pool, and the current experience sample includes the current state, target action, immediate reward, next state and termination flag; S3.3 During the training process, a batch of historical experience samples are randomly sampled from the ordinary experience replay pool and the high-reward experience replay pool according to a preset ratio. The target Q value corresponding to each historical experience sample is calculated based on the target network, and the current Q value corresponding to each historical experience sample is calculated using the evaluation network. The loss function is calculated based on the target Q value and the current Q value, and the network parameters of the evaluation network are updated based on the loss function. S3.

4. Every preset number of steps, copy the network parameters of the evaluation network to the target network; S3.5 Determine whether the current training round has ended. If the robot meets the preset termination condition, the current round is determined to be over. Calculate the total reward value for the current round. If the total reward value for the current round is greater than the historical highest total reward value, transfer all experience samples stored in the temporary experience pool for the current round to the high reward experience replay pool, update the historical highest total reward value, and clear the temporary experience pool; otherwise, take the next state of the current round as the new current state and return to step S3.

2. S4. Iterate through step S3 until the iteration termination condition is met, complete the training of the improved deep Q network, obtain the trained evaluation network, and use the trained evaluation network to perform path planning for the robot.

2. The robot path planning method according to claim 1, characterized in that, In step S1, the exploration reward is constructed based on the Manhattan distance between the robot's current position and the starting point.

3. The robot path planning method according to claim 2, characterized in that, In step S1, The reward function also includes a positive reward for reaching the target point and a negative reward for colliding with obstacles or map boundaries. The expression for the reward function is: in, r The reward function; x 1. y 1 represents the coordinates of the robot's starting position; x 2. y 2 represents the coordinates of the robot's current position; , where is the Manhattan distance between the robot's current position and the starting point.

4. The robot path planning method according to claim 2, characterized in that, In step S1, the range of the exploration reward item is [-0.95, -0.01].

5. The robot path planning method according to claim 1, characterized in that, In step S3.1, the local map update strategy includes: generating a local grid map of a preset size centered on the robot's current position as the current local map; when the robot moves to the boundary of the current local map, regenerating a new local map centered on the robot's current global map position; if the robot is at the edge of the global map, making it impossible to generate a complete current local map centered on itself, then adjusting the center of the current local map to shift inward to ensure that the current local map is always within the range of the global map.

6. The robot path planning method according to claim 1, characterized in that, In step S3.2, the exploration strategy includes ε-greedy Exploration strategy.

7. The robot path planning method according to claim 1, characterized in that, In step S3.3, in the preset ratio, the sampling proportion of the ordinary experience replay pool is greater than the sampling proportion of the high reward experience replay pool.

8. The robot path planning method according to claim 1, characterized in that, In step S3.3, the loss function expression is: in, The loss function is... N The number of samples collected; The current Q value is determined by the evaluation network based on the current state. and actions Calculations show that The parameters of the evaluation network; The target value; r j For instant rewards; γ Discount factor; The target Q value is calculated by the target network based on the next state and the target action; s' j The next state; a' For the target action; These are the parameters of the target network; d j This is a termination flag; when a sample is in a terminated state, d j =1, at which point the target value degenerates into y j = r j .

9. The robot path planning method according to claim 1, characterized in that, In step S3.5, the preset termination conditions include the robot reaching the target point, the robot colliding, the robot crossing the boundary, and the number of steps in this round of training reaching the preset maximum number of steps.

10. A robot path planning system based on an improved deep Q-network, characterized in that, Includes the following modules: The path planning construction module is used to construct the state space, action space, and reward function for robot path planning; the reward function includes an exploration reward item related to the distance from the robot's starting point. An initialization module is used to initialize an improved deep Q-network, which includes an evaluation network, a target network, a normal experience replay pool, a high-reward experience replay pool, and a temporary experience pool. The training module is used to train the evaluation network, and each training round includes: S3.1 Initialize the robot's position, determine the current local map based on the robot's current position according to the local map update strategy, and generate the current state based on the state space according to the current local map and the robot's position information in the current local map; S3.

2. Input the current state into the evaluation network, and the evaluation network outputs Q values ​​corresponding to each action in the action space; based on the Q values ​​output by the evaluation network, use an exploration strategy to select a target action to be executed from the action space, control the robot to execute the target action and move to a new coordinate; calculate the immediate reward after the execution of this action according to the reward function, and obtain the next state and termination flag based on the state space according to the new coordinates; store the current experience sample in both the ordinary experience replay pool and the temporary experience pool, and the current experience sample includes the current state, target action, immediate reward, next state and termination flag; S3.3 During the training process, a batch of historical experience samples are randomly sampled from the ordinary experience replay pool and the high-reward experience replay pool according to a preset ratio. The target Q value corresponding to each historical experience sample is calculated based on the target network, and the current Q value corresponding to each historical experience sample is calculated using the evaluation network. The loss function is calculated based on the target Q value and the current Q value, and the network parameters of the evaluation network are updated based on the loss function. S3.

4. Every preset number of steps, copy the network parameters of the evaluation network to the target network; S3.5 Determine whether the current training round has ended. If the robot meets the preset termination condition, the current round is determined to be over. Calculate the total reward value for the current round. If the total reward value for the current round is greater than the historical highest total reward value, transfer all experience samples stored in the temporary experience pool for the current round to the high reward experience replay pool, update the historical highest total reward value, and clear the temporary experience pool; otherwise, take the next state of the current round as the new current state and return to step S3.

2. The path planning module is used to plan the path for the robot using a trained evaluation network.

Images