A Robot Obstacle Avoidance Path Planning Method Based on Deep Reinforcement Learning
The robot path planning model constructed by deep reinforcement learning network solves the problem of inaccurate environmental information acquisition in the existing technology, realizes fast and accurate path planning for robots in unknown environments, and improves obstacle avoidance success rate and path efficiency.
Patent Information
- Application Number
- CN202311740805.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-18
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-12-18
AI Technical Summary
Existing path planning algorithms rely on global environmental information, which makes it difficult to accurately obtain environmental information in practical applications, thus affecting the quality of path planning.
A robot obstacle avoidance path planning method based on deep reinforcement learning is adopted. The robot path planning model is constructed through a deep reinforcement learning network. By using a gridded environment map, combined with an obstacle learning record table and a playback memory unit, the reward function and exploration factor are dynamically adjusted to achieve autonomous environmental learning and decision-making.
It improves the success rate of robot obstacle avoidance and the efficiency of path planning, shortens the path length, and ensures that the robot can quickly and accurately find a collision-free path in an unknown environment.
Smart Images

Figure CN117707168B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot path planning technology, specifically relating to a robot obstacle avoidance path planning method based on deep reinforcement learning. Background Technology
[0002] With the continuous advancement of science and technology, all kinds of mobile robots have gradually entered people's daily lives and played a significant role in fields such as military, industry, exploration, and medicine.
[0003] Autonomous navigation of mobile robots can be divided into three subsystems: information perception, behavior decision-making, and manipulation control. Path planning is the foundation of navigation and control, specifically referring to the process of finding a collision-free path from the initial point to the target point.
[0004] Currently, commonly used algorithms include Dijkstra's algorithm, Rapidly-exploring Random Trees (RRT), A* algorithm, and ant colony optimization. However, most of these algorithms rely on global environmental information, so the completeness and accuracy of this information directly affect the quality of path planning. In practical applications, robots often struggle to acquire all environmental information, making it crucial to enable robots to learn autonomously, perceive their environment, and make decisions. Summary of the Invention
[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a robot obstacle avoidance path planning method based on deep reinforcement learning, which solves the problem that existing path planning methods struggle to obtain complete and accurate environmental information, thus affecting the quality of path planning.
[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a robot obstacle avoidance path planning method based on deep reinforcement learning, comprising the following steps:
[0007] S1. Rasterize the environment map;
[0008] S2. Construct a deep reinforcement learning model for robot path planning using Deep Q-Network as the backbone network;
[0009] S3. Initialize the state values, playback memory unit D, and obstacle learning record table of the robot path planning deep reinforcement model; where each cell in the grid environment map corresponds to a state, and the playback memory unit D is (s t ,a t ,r t ,s t+1 ), s t Let s be the current state, a t For the current action a, rt For instant rewards; t+1 The next state is indicated by the obstacle learning log, which records the state and actions when colliding with obstacles.
[0010] S4. During the process of the robot exploring the grid environment map based on the robot path planning deep reinforcement learning model, determine whether the current state is the target point state.
[0011] If so, then obstacle avoidance path planning is complete;
[0012] If not, proceed to step S5;
[0013] S5. Enter the next state, remove the actions recorded in the obstacle learning record table from the action set, select an action in the action set and execute it;
[0014] S6. Based on the currently executed action, obtain the immediate reward and enter the next state, and set the current (s) t ,a t ,r t ,s t+1 Stored into the playback memory unit;
[0015] S7. Determine whether the amount of data stored in the playback memory unit has reached the preset threshold.
[0016] If so, proceed to step S8;
[0017] If not, proceed to step S9:
[0018] S8. Randomly extract data from the playback memory unit and update the robot path planning deep reinforcement learning model based on it, and then proceed to step S9.
[0019] S9. Determine whether the action is an obstacle collision based on the reward obtained;
[0020] If so, record it in the obstacle learning record table and return to step S4;
[0021] If not, return directly to step S4.
[0022] Furthermore, in step S1, in the grid environment map, each cell is the same size and evenly distributed, including the cell where the robot is currently located as the starting point, the cell where the target point is to be reached is located, the obstacle area, and the feasible area.
[0023] Furthermore, in step S2, the robot path planning deep reinforcement learning model includes an estimation network and a target network;
[0024] The estimation network is used to extract features from the unknown environment state of the input using the perception function of the neural network, so as to fit the environment state to the action value function; the target network is used to make decisions based on the output of the deep learning network and the exploration strategy, so as to realize the mapping from state to action.
[0025] Furthermore, in step S3, r t Instant rewards include:
[0026]
[0027] Where C1 is 10, C2 is -10, C3 is -3, and C4 is the heuristic reward function r. e (s t ,a t ,s t+1 When D(s) t+1 ,s g )≤D(s t ,s g When C5 is 0.1, it is 0 otherwise, and C6 is a constant -1.
[0028] Furthermore, the heuristic reward function r e (s t ,a t ,s t+1 The expression for ) is:
[0029]
[0030] In the formula, N(s) t+1 ) represents the state s reached. t+1 The number of times, N(s) t ) represents the state s reached. t The number of times;
[0031] In the exploration reward, the Euclidean distance from the current state point to the target point is D(s). t ,s g The Euclidean distance from the next state point to the target point is D(s). t+1 ,s g The expressions for ) are as follows:
[0032]
[0033]
[0034] In the formula, (x t ,y t (x) represents the robot's current coordinate position. t+1 ,y t+1 (x) represents the coordinates of the robot's next state. g ,yg () represents the target coordinates.
[0035] Furthermore, in step S5, an ε-greedy greedy strategy is used to select an action a through exploration or an estimated network is used to select an action a.
[0036] Furthermore, the ε-greedy greedy strategy refers to generating a random number in [0,1) and determining whether it is less than EPSILON; the value of EPSILON is generated by the dynamic function f(x) = 0.5 × ln(6.389x + 1), where x refers to the number of rounds;
[0037] If so, then the action a with the largest Q value is selected using the estimation network; where Q value refers to the value function of the state-action pair.
[0038] If not, then randomly select an action 'a' from the action set;
[0039] The target Q-value is determined using the target state action function of the estimated network, and its expression is as follows:
[0040] y'=r+γmaxQ(s t+1 ,a t+1 ,θ)
[0041] In the formula, y' is the target Q value, γ is the discount factor, r is the reward value obtained immediately after taking action a in state s, and θ is the weight parameter trained in the neural network structure model.
[0042] Furthermore, in step S8, when updating the robot path planning deep reinforcement learning model, the loss function is the mean squared error loss function, the expression of which is:
[0043] L(θ) = E[(y'-Q(s,a,θ)) 2 ].
[0044] The beneficial effects of this invention are as follows:
[0045] (1) This invention improves obstacle learning rules, redesigns reward functions and introduces dynamic exploration factors, which allows the robot to converge faster, achieve a higher success rate and a shorter path length while avoiding obstacles.
[0046] (2) In the deep reinforcement learning model for robot path planning constructed in this invention, deep reinforcement learning (DRL) is an algorithm that combines deep learning (DL) and reinforcement learning (RL). DL is mainly responsible for extracting features from the unknown environment state of the input using the perception function of the neural network, and fitting the environment state to the state action value function; while RL is responsible for making decisions based on the output of the deep neural network and a certain exploration strategy, thereby realizing the mapping from state to action, and thus better meeting the robot's movement needs. Attached Figure Description
[0047] Figure 1 The flowchart of the robot obstacle avoidance path planning method based on deep reinforcement learning provided by the present invention is shown.
[0048] Figure 2 This is a schematic diagram illustrating the training round speed of the dynamic function of the present invention.
[0049] Figure 3 This is the neural network model structure of the present invention.
[0050] Figure 4 This is a 3D 25×25 input information diagram of the present invention.
[0051] Figure 5 These are schematic diagrams of two grid diagrams of the present invention.
[0052] Figure 6 For the present invention Figure 1 A robot obstacle avoidance path planning graph based on deep reinforcement learning.
[0053] Figure 7 For the present invention Figure 2 A robot obstacle avoidance path planning graph based on deep reinforcement learning. Detailed Implementation
[0054] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0055] This invention provides a robot obstacle avoidance path planning method based on deep reinforcement learning, such as... Figure 1 As shown, it includes the following steps:
[0056] S1. Rasterize the environment map;
[0057] S2. Construct a deep reinforcement learning model for robot path planning using Deep Q-Network as the backbone network;
[0058] S3. Initialize the state values, playback memory unit D, and obstacle learning record table of the robot path planning deep reinforcement model; where each cell in the grid environment map corresponds to a state, and the playback memory unit D is (s t ,a t ,r t ,s t+1 ), s t Let s be the current state, and a be the result of this process. t For the current action a, r t For instant rewards; t+1 The next state is indicated by the obstacle learning log, which records the state and actions when colliding with obstacles.
[0059] S4. During the process of the robot exploring the grid environment map based on the robot path planning deep reinforcement learning model, determine whether the current state is the target point state.
[0060] If so, then obstacle avoidance path planning is complete;
[0061] If not, proceed to step S5;
[0062] S5. Enter the next state, remove the actions recorded in the obstacle learning record table from the action set, select an action in the action set and execute it;
[0063] S6. Based on the currently executed action, obtain the immediate reward and enter the next state, and set the current (s) t ,a t ,r t ,s t+1 Stored into the playback memory unit;
[0064] S7. Determine whether the amount of data stored in the playback memory unit has reached the preset threshold.
[0065] If so, proceed to step S8;
[0066] If not, proceed to step S9:
[0067] S8. Randomly extract data from the playback memory unit and update the robot path planning deep reinforcement learning model based on it, and then proceed to step S9.
[0068] S9. Determine whether the action is an obstacle collision based on the reward obtained;
[0069] If so, record it in the obstacle learning record table and return to step S4;
[0070] If not, return directly to step S4.
[0071] In step S1 of this embodiment of the invention, in the grid environment map, each cell is the same size and evenly distributed, including the cell where the robot is currently located as the starting point, the cell where the target point to be reached is located, the obstacle area and the feasible area; wherein, the starting point and the target point to be reached correspond to one cell, and the obstacle area corresponds to two or more cells.
[0072] In step S2 of this embodiment of the invention, the robot path planning deep reinforcement learning model includes an estimation network and a target network;
[0073] The estimation network is used to extract features from the unknown environment state of the input using the perception function of the neural network, and to fit the environment state to the action value function; the target network is used to make decisions based on the output of the deep learning network and the exploration strategy, and to map the state to the action; the actions include up, down, left, right, upper left, upper right, lower left and lower right, thus better meeting the robot's movement needs.
[0074] In step S3 of this embodiment of the invention, the state values, playback memory unit D and obstacle learning record table are initialized, including initializing the state values s of each position, and the playback memory unit D and obstacle learning record table are 0; wherein, there are a total of 625 states s, which are 25×25 cells in a grid environment, wherein each cell is the same size and evenly distributed.
[0075] In this embodiment, the obstacle learning record table refers to the record table when the mobile robot encounters obstacles while exploring the environment. There is a penalty of -10 for colliding with obstacles. When the immediate reward r is -10, the state s and action a are recorded in the record table.
[0076] Specifically, in step S3 of this embodiment, r t Instant rewards include:
[0077]
[0078] Where C1 is 10, C2 is -10, C3 is -3, and C4 is the heuristic reward function r. e (s t ,a t ,s t+1 When D(s) t+1 ,s g )≤D(s t ,s g When C5 is 0.1, it is 0 otherwise, and C6 is a constant -1.
[0079] In this embodiment, heuristic reward refers to the immediate reward *r* generated by the mobile robot when it encounters obstacles and target points while exploring the environment. Introducing heuristic reward influences the robot's behavioral choices, making it more inclined to choose paths closer to the target point and eliminating penalties for paths farther from the target point, thus achieving adaptive adjustment and better reward effects. Based on this, the heuristic reward function *r*... e (s t ,a t ,s t+1 The expression for ) is:
[0080]
[0081] In the formula, N(s) t+1 ) represents the state s reached. t+1 The number of times, N(s) t ) represents the state s reached. t The number of times; as can be seen from its calculation method, in state s t The more times a state is visited, the smaller its reciprocal value becomes. Therefore, subtracting the reciprocal value of a state with fewer visits from the reciprocal value of a state with more visits will yield a positive reward, thus encouraging the robot to explore more unexplored states. Conversely, subtracting the reciprocal value of a state with more visits from the reciprocal value of a state with fewer visits will not yield a reward, and will result in a reward of 0.
[0082] In this embodiment, for exploration rewards, the mobile robot generates an immediate reward *r* when encountering obstacles and target points while exploring the environment. Exploration rewards are introduced by using the difference in the number of visits between different states to provide rewards, balancing the number of visits to different state points, avoiding excessive repeated visits, and thus improving learning effectiveness. Based on this, in the exploration reward, the Euclidean distance from the current state point to the target point is D(s). t ,s g The Euclidean distance from the next state point to the target point is D(s). t+1 ,s g The expressions for ) are as follows:
[0083]
[0084]
[0085] In the formula, (x t ,y t (x) represents the robot's current coordinate position. t+1 ,y t+1 (x) represents the coordinates of the robot's next state. g ,y g () represents the target coordinates.
[0086] In step S5 of this embodiment of the invention, removing the recorded action from the action set specifically includes removing the action that has been recorded in the record table from the action set according to the state s and action a recorded in the obstacle record table, so as to prevent it from repeatedly selecting the action to collide with the obstacle in the next time, and avoid a large number of repeated learnings of facing the same obstacle in the subsequent learning process.
[0087] In step S5 of this embodiment of the invention, the ε-greedy greedy strategy is used to select action a through exploration or the estimated network is used to select action a.
[0088] Specifically, in this embodiment, the ε-greedy strategy refers to generating a random number in [0,1) and determining whether it is less than EPSILON; EPSILON is generated by the dynamic function f(x) = 0.5 × ln(6.389x + 1), where x refers to the number of rounds;
[0089] If so, the estimation network is used to select the action 'a' with the largest Q value. Here, Q value refers to the value function of the state-action pair. It represents the estimated cumulative reward value obtained by taking a certain action in a given state. Q value can be used to evaluate the merits of different actions and help the robot make decisions to select the optimal action. In general, Q value represents the value of an action in the DQN algorithm. By learning and optimizing the Q value function, the robot can find the optimal decision strategy.
[0090] If not, then randomly select an action 'a' from the action set;
[0091] The target Q-value is determined using the target state action function of the estimated network, and its expression is as follows:
[0092] y'=r+γmaxQ(s t+1 ,a t+1 ,θ)
[0093] In the formula, y' is the target Q value, γ is the discount factor, r is the reward value obtained immediately after taking action a in state s, and θ is the weight parameter trained in the neural network structure model.
[0094] In this embodiment, the value of EPSILON is generated using the dynamic function f(x) = 0.5 × ln(6.389x + 1). EPSILON is a parameter that controls the balance between exploration and exploitation. It determines whether to select a random action or the action currently considered the best during training. When the random number is greater than or equal to EPSILON, DQN randomly selects an action to continue exploring the environment and discover potentially better actions. This ensures that the algorithm does not get stuck in local optima but can explore more states and actions. When the random number is less than EPSILON, DQN selects the currently estimated best action, i.e., it makes a decision using the learned value function. This ensures that the algorithm reduces exploration during training and tends to select better actions. By adjusting the value of EPSILON, the trade-off between exploration and exploitation can be controlled.
[0095] In this embodiment, Figure 2 This is a schematic diagram of the dynamic exploration factor generated by a dynamic function in this embodiment. The exploration factor ε in this embodiment determines whether the robot makes a random selection or uses an estimation network when selecting an action. By using a dynamic exploration factor, the randomness of the exploration is gradually reduced, and the neural network is made more fully utilized for action selection and learning in the later training, thereby improving the algorithm performance and learning efficiency.
[0096] In step S8 of this embodiment of the invention, when the data storage in the playback memory unit D is greater than a preset threshold, the old data is overwritten with new data. When the robot's steps are greater than the preset threshold and every 5 steps, a number of (s) are randomly selected from the memory unit D. t ,a t ,r t ,s t+1 The data is used to update the estimation network and the target network, where the loss function is the mean squared error loss function:
[0097] L(θ) = E[(y'-Q(s,a,θ)) 2 ].
[0098] In the formula, θ represents the weight parameters trained in the neural network structure model.
[0099] In step S9 of this embodiment of the invention, based on the immediate reward r t Determine whether this action should be recorded in the obstacle learning record table. If r t If the value is C2, meaning a collision with an obstacle, then the action a in the current state will be changed. t Record the obstacle in the obstacle record learning table, return to step S4, and continue until the current state s is satisfied. t The target point state s g .
[0100] In this embodiment of the invention, the model structure of the robot path planning deep reinforcement learning model is as follows: Figure 3 As shown, its neural network consists of a three-layer neuron structure: an input layer, a hidden layer, and an output layer; as... Figure 4 The diagram illustrates the input information used in this invention to facilitate the reading of information into a convolutional neural network. Figure 4 In this network, the input information is a three-dimensional 25×25 array. Each dimension represents the positional state of an object. For example, the first dimension represents the robot's positional state, with position [0,0] in the first layer array set to 1 and other regions set to 0. The second dimension represents the obstacle's positional state, with its position in the second dimension set to 1 and its movable area set to 0. The third dimension represents the target point's positional state, with position [24 24] in the third layer array set to 1 and other regions set to 0. The activation function is ReLU. The hidden layer consists of a fully connected layer with 625 input neurons and 224 output neurons, using ReLU activation. The output layer consists of a fully connected layer with 224 input neurons and 8 output neurons, corresponding to 8 actions. The optimizer used is Adam, and the loss function is MSELOSS.
[0101] Figure 5 Figure 6 shows two 25×25 grid environments for this example. In both environments, the red dots represent the robot's starting point, the blue stars represent the robot's target point, and the black squares represent obstacles. The obstacle areas and feasible areas are different in the two environments. Figure 6 shows the effect after path planning in environment 1. The green line is the optimal obstacle avoidance path from the red dot starting point to the large blue star target point. Figure 7 The green line represents the optimal obstacle avoidance path from the red starting point to the target point on the large blue star, as shown in Environment 2 after path planning.
[0102] In the description of this invention, it should be understood that the terms "center," "thickness," "upper," "lower," "horizontal," "top," "bottom," "inner," "outer," and "radial," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, a feature defined by "first," "second," and "third" may explicitly or implicitly include one or more of that feature.
[0103] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.
[0104] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.
Claims
1. A robot obstacle avoidance path planning method based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Rasterize the environment map; S2. Construct a deep reinforcement learning model for robot path planning using Deep Q-Network as the backbone network; S3. Initialize the state values, playback memory unit D, and obstacle learning record table of the robot path planning deep reinforcement model; where each cell in the grid environment map corresponds to a state, and the playback memory unit D is... , This is the current state. For the current action, For instant rewards; The next state is indicated by the obstacle learning log, which records the state and actions when colliding with obstacles. S4. During the process of the robot exploring the grid environment map based on the robot path planning deep reinforcement learning model, determine whether the current state is the target point state. If so, then obstacle avoidance path planning is complete; If not, proceed to step S5; S5. Enter the next state, remove the actions recorded in the obstacle learning record table from the action set, select an action in the action set and execute it; S6. Based on the current action, obtain the immediate reward and proceed to the next state, and set the current state... Stored into the playback memory unit; S7. Determine whether the amount of data stored in the playback memory unit has reached the preset threshold. If so, proceed to step S8; If not, proceed to step S9: S8. Randomly extract data from the playback memory unit and update the robot path planning deep reinforcement learning model based on it, and then proceed to step S9. S9. Determine whether the action is an obstacle collision based on the reward obtained; If so, record it in the obstacle learning record table and return to step S4; If not, return directly to step S4; In step S3 Instant rewards include: in, It is 10. -10, -3, For heuristic reward functions ,when hour, C1 is 0.1, and C2 is 0.0; C3 is a constant -1. Heuristic reward function The expression is: In the formula, To reach the state Number of times, To reach the state The number of times; In the exploration reward, the Euclidean distance from the current state point to the target point is: The Euclidean distance from the next state point to the target point is The expressions are as follows: In the formula, This represents the robot's current coordinates. This is the coordinate position of the robot in the next state. The target coordinates are the location point; In step S5, the ε-greedy greedy strategy is used to select the exploration random selection action. Or use the estimation network to select actions ; The ε-greedy strategy refers to generating a random number within the range [0, 1) and determining whether it is less than EPSILON; the value of EPSILON is determined by a dynamic function. generate, This refers to the number of rounds; If so, then use the estimated network to select the maximum. Value action ;in, The value refers to the value function of the state-action pair; If not, then randomly select an action from the action set. ; Among them, the target is determined by estimating the target state action function of the network. The value, whose expression is: In the formula, For the goal value, As a discount factor, The reward value obtained immediately after taking action a in state s. These are the weight parameters trained in the neural network structure model.
2. The robot obstacle avoidance path planning method based on deep reinforcement learning according to claim 1, characterized in that, In step S1, in the grid environment map, each cell is the same size and evenly distributed, including the cell where the robot is currently located as the starting point, the cell where the target point is to be reached is located, the obstacle area, and the feasible area.
3. The robot obstacle avoidance path planning method based on deep reinforcement learning according to claim 1, characterized in that, In step S2, the robot path planning deep reinforcement learning model includes an estimation network and a target network; The estimation network is used to extract features from the unknown environment state of the input using the perception function of the neural network, so as to fit the environment state to the action value function; the target network is used to make decisions based on the output of the deep learning network and the exploration strategy, so as to realize the mapping from state to action.
4. The robot obstacle avoidance path planning method based on deep reinforcement learning according to claim 1, characterized in that, In step S8, when updating the robot path planning deep reinforcement learning model, the loss function is the mean squared error loss function, the expression of which is: 。
