An Online Path Planning Method for Mobile Robots Based on Deep Reinforcement Learning

Abstract

The present invention discloses an online path planning method for a mobile robot based on deep reinforcement learning. The specific steps are as follows: Obtain obstacle information through a lidar, obtain the robot position information and target point information through ROS built-in plugins, and process and integrate the two into the robot state S. Input the state S into the DDPG network to obtain the optimal execution action, enabling the robot to quickly and safely reach the target point during navigation; Design a cascaded structure of LSTM and Actor networks; The structure of the improved experience replay pool adopts a segment tree, which can efficiently update the weight changes caused by the continuous increase in the number of episodes; For the optimization of the Critic network, a fractional gradient descent algorithm with a momentum term is adopted, which can improve both the accuracy and the convergence speed compared to the integer order; The present invention is used to solve problems such as low accuracy and slow convergence speed when conventional DDPG is used for robot path planning.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and relates to an online path planning method for a mobile robot based on deep reinforcement learning. Background Art

[0002] With the development of artificial intelligence technology, the types of robots have been gradually refined, the structures of robots have become more complex, and the level of intelligence has become higher. Various intelligent robots with special functions have emerged, such as mobile robots, industrial robots, micro-robots, underwater robots, service robots, etc. Among them, mobile robots have become one of the current research hotspots due to advantages such as wide application scenarios and strong reliability.

[0003] One of the important reasons for the wide application of mobile robots is path planning. Path planning refers to starting from a starting point in a specific scenario, reaching a predetermined location while avoiding all obstacles, and optimizing this path according to certain criteria. With the complexity of application scenarios and working environments, this requires the mobile robot itself to have strong environmental adaptability. In an unknown environment, the mobile robot lacks an understanding of the environment, and the environment may change at any time. Therefore, the mobile robot needs to have perception and learning abilities in an unknown environment. The mobile robot needs to continuously sense information, make decisions, and execute actions in a dynamic environment, and continuously repeat learning to finally obtain a safe path that adapts to dynamic changes.

[0004] Reinforcement learning is a decision control method different from traditional machine learning algorithms. Through reinforcement learning, an intelligent agent can adapt to the environment in an unknown environment without any guidance signals through online learning and continuous trial and error. However, there are still unsolved problems in cases with a large number of dimensions such as path planning. Deep learning has strong perception capabilities and can handle high-dimensional information well. Deep reinforcement learning that combines it with reinforcement learning can not only process high-dimensional environmental information but also make corresponding decisions. In 2015, the DQN (Deep Q-Network) algorithm emerged, but its algorithm can only solve problems with a low-dimensional and discrete action space. In 2016, the DDPG algorithm (Deep Deterministic Policy Gradient algorithm) changed the network structure on the basis of the DQN algorithm and at the same time changed the discrete actions into continuous actions. However, when applying the DDPG algorithm to solve the path planning problem in a dynamic environment, there are still problems such as low success rate, slow convergence speed, and difficult parameter debugging. Summary of the Invention

[0005] In order to solve the above existing problems, the present invention provides an improved DDPG path planning algorithm to solve the problems of low success rate and slow convergence speed when the conventional DDPG algorithm is applied to path planning.

[0006] The technical solution of the present invention:

[0007] An online path planning method for a mobile robot based on deep reinforcement learning, the steps are as follows:

[0008] Step 1: The mobile robot obtains obstacle information through the lidar carried by itself, and obtains and processes the robot's position information and target point information in real time through the built-in plug-in of ROS (Robot Operating System), which together with the obstacle information obtained by the lidar constitutes the state information S of the robot.

[0009] Step 2: Design a neural network structure model for improving the DDPG algorithm. On the basis of the dual-network structure (main network and target network) of the DDPG algorithm, the Actor networks of the main network and the target network are set as the structure cascaded by a fully connected layer and an LSTM network.

[0010] Step 3: Design the state space, action space and reward function of the robot in the improved DDPG algorithm. The reward function consists of four parts, the reward for reaching the target point, the penalty for hitting an obstacle, the reward and penalty for the straight-line distance to the target point, and the penalty for continuous identical actions.

[0011] Step 4: Place the mobile robot in the set environment and train it using the improved DDPG algorithm. Use the trained neural network to complete the collision-free real-time path planning of the robot. Each piece of experience generated by the interaction between the robot and the environment is placed in a specific experience replay pool according to the setting for training the neural network parameters; and an improved experience replay method is used to select experiences for training the neural network. After training, the output of the neural network is the optimal executable action of the robot, which can enable the robot to reach the set location quickly and safely.

[0012] Furthermore, the obstacle information obtained in Step 1 includes the distances between the robot and static obstacles and dynamic obstacles. The relationship between the robot and the obstacles is divided into two types, as shown in the following formula:

[0013]

[0014] where d i (t) is the measurement information of the i-th dimensional lidar of the robot at time t, and thr is the distance threshold between the robot and the obstacle. When the distance d i (t) between the robot and the obstacle ≤ thr, the robot hits the obstacle and is in a stagnant state C. When the distance d i (t) between the robot and the obstacle > thr, the robot is in a normal navigation state N.

[0015] Further, the robot state information and target point information in step 1 include the absolute position of the target point relative to the base coordinate system, the difference angles between the robot and the target point relative to the base coordinate system, the relative angle between the robot and the target point, the yaw angle of the robot, and the execution action of the previous step.

[0016] Further, the reward and punishment for the straight-line distance to the target point in step 3 include whether the straight-line distance from the current position of the robot to the target point is closer than the previous moment. If it is closer, a reward is given; otherwise, a punishment is given.

[0017] Further, the punishment for consecutive identical actions in step 3 includes retaining the consecutive h actions generated by the robot in a list, which is continuously updated during the robot's movement, and feeding the reward and punishment back to the total reward according to the setting; if there are more than l consecutive left turns or right turns in the actions, it indicates that the robot is in a stagnant state at this time, which will slow down the speed of reaching the target point and a punishment needs to be given.

[0018] Further, the experience generated during the robot's movement in step 4 consists of the current state information, actions, rewards, and the next state information of the robot.

[0019] Advantages of the present invention: By introducing the LSTM network to consider the influence of previous states and actions on the current state, the present invention solves the long-term dependence problem. The improved experience replay method reduces the replay of useless experiences, improves the replay efficiency, and reduces the computational amount of repeatedly calculating the weights of each episode in the experience pool by brute-force modification. Using the momentum fraction gradient descent method, the convergence speed and stability are improved compared with the conventional SGD algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of the method of the present invention.

[0021] Figure 2 is the DDPG-LSTM neural network structure model of the present invention.

[0022] Figures 3(a) and 3(b) are the architecture diagrams of the Actor network and the Critic network respectively.

[0023] Figures 4(a) and 4(b) are the initial state and the training environment during training for the robot built using Gazebo respectively. DETAILED DESCRIPTION OF THE INVENTION

[0024] The following further describes the specific implementation manners of the present invention in combination with the drawings and technical solutions.

[0025] The present invention improves the problems of low success rate and slow convergence existing in the DDPG algorithm by introducing the LSTM network, an improved experience replay method, a reward mechanism, and the momentum fraction gradient descent method.

[0026] As Figure 1 shown, it is the flowchart of the method of the present invention, and the specific steps are as follows:

[0027] Step 1: The robot obtains obstacle information through a lidar and obtains the current state information and target point information of the robot through the built-in plug-in of ROS;

[0028] The robot obtains the distance between itself and surrounding obstacles through a lidar. The survey range of the lidar is from -90° to 90° (the front of the robot is 0°), and a total of 10 beams of laser are generated. In the range of 180°, the angle between two adjacent lasers is 18°. The survey distance of the lidar is 0.2m - 5m. As shown in Figures 4(a) and 4(b), the points reached by the straight lines emitted from the front end of the robot model are the lidar detection areas.

[0029] Step 2: Process and integrate the obtained information and input it into the DDPG-LSTM network model to obtain the action information that the robot will execute;

[0030] As Figure 2 shown is the specific model of the DDPG-LSTM neural network structure.

[0031] The DDPG algorithm consists of 4 network modules, including two Actor networks and two Critic networks. The structures of the two Actor networks are exactly the same, and the structures of the two Critic networks are also exactly the same.

[0032] The input end of DDPG is respectively connected to two Actor networks and two Critic networks. One Actor network and one Critic network form a main network of DDPG, and the remaining two networks form the target network of DDPG. The network parameters are initialized randomly, the parameters of the main network are updated in real time, and the parameters of the target network are updated by sliding according to the parameters of the main network.

[0033] The input of the main Actor network is the state information of the robot, and the output is the action that the robot will execute next. The input of the target Actor network is the state information of the robot in the next step, and the output is the best action that is predicted to be executed after reaching this state, rather than the action that will actually be executed in the next step.

[0034] The input of the main Critic network is the current state and the action executed in the current state, and the output is the evaluation value of executing the corresponding action in this state. The input of the target Critic network is the state information of the next step and the predicted execution action, and the output is the evaluation value of executing the predicted action in this state.

[0035] In the present invention, the Actor networks of the main network and the target network are combined with the LSTM network, that is, a layer of LSTM network is cascaded in front of the Actor networks of the main network and the target network. LSTM has a strong memory ability and can process time series data. When solving decision-making problems, the influence of previous states and actions on the current state can be considered, enabling the neural network to have long-term prediction ability and output more reasonable linear velocity and angular velocity, which is in line with practical applications.

[0036] In order to enable the robot to accumulate exploration effects, that is, to move forward continuously in one direction along inertia, OU noise is added to the actions output by the Actor network of the main network.

[0037] In DDPG, the Critic network updates the network parameters by using the method of Stochastic Gradient Descent (SGD). In order to improve the convergence speed of the neural network and avoid parameter oscillation, the present invention uses the fractional gradient descent method algorithm with momentum term addition to update the parameters of the Critic network, and the training method of the Actor network is not modified. The specific formula of the momentum fractional gradient descent algorithm is as follows:

[0038]

[0039]

[0040] In the formula, M(t) is the weight momentum term at time t, and N(t) is the bias momentum term at time t. represents the q-th corresponding weight value of the i-th neuron in the p-th layer of the fully connected layer. represents the q-th corresponding bias value of the i-th neuron in the p-th layer of the fully connected layer. represents the current weight value. represents the current bias value. LOSS is the mean square error loss function of the estimated value output by the target network and the evaluation value output by the main network. γ is the fractional order and 0 < γ < 2, α is the learning rate and 0 < α ≤ 1, δ is the momentum term coefficient and 0 < δ < 1, and δ is used to determine the influence degree of the gradient update direction at the previous moment on the current moment. ε is a very small positive number used to avoid singular values when updating parameters.

[0041] The order of the fractional gradient descent algorithm can be set to a fraction, which is more flexible than the integer-order parameter setting and has a faster convergence speed. The momentum term can reduce the oscillation phenomenon in the optimization process and accelerate the convergence speed. Combining the advantages of both can improve both the convergence speed and the accuracy.

[0042] As shown in FIGS. 3(a) and 3(b), they are respectively composed of an Actor network and a Critic network.

[0043] Specifically, the Actor network consists of 3 fully connected layers, 2 batch normalization layers, and 2 layers of activation functions. It receives the state information of the robot and outputs action information, that is, the linear velocity and angular velocity. The Critic network consists of 5 fully connected layers and 4 layers of activation functions. It receives the state and executed actions of the robot and outputs the evaluation value of executing the action in this state.

[0044] The action information executed by the robot described in step 2 includes the linear velocity and angular velocity of the robot. In order to make the action trajectory of the robot smoother and the output linear velocity and angular velocity more in line with the motion characteristics in reality, in this embodiment, the maximum angular velocity is set to 0.5 rad / s, and the maximum linear velocity is set to 0.25 m / s. Finally, the linear velocity range output by the action network is (0, 0.25 m / s), and the angular velocity range is (-0.5 rad / s - 0.5 rad / s).

[0045] Step 3: Design the state space, action space, and reward function of the robot in the improved DDPG algorithm. The reward function consists of four parts: the reward for reaching the target point, the penalty for hitting an obstacle, the reward and penalty for the straight-line distance from the target point, and the penalty for consecutive identical actions.

[0046] Furthermore, the reward and penalty for the straight-line distance from the target point in the reward function described in step 2 includes whether the straight-line distance between the state of the robot at this time and the target point is closer than the previous moment. If it is closer, a reward is given; otherwise, a penalty is given.

[0047] The reward function in step 3 is generally summarized as the following formula:

[0048]

[0049] where r is the return value for each step, d t is the distance between different set points of the robot at time t, d target is the distance threshold between the robot and the target point (set to 0.2 m in this embodiment). When the distance d t of the robot from the target point ≤ 0.2 m, it is determined that the robot reaches the target point, and r a is the reward for reaching the target point. d collisionis the distance threshold between the robot and the obstacle (set to 0.2 m in this embodiment). When the distance d of the robot from the obstacle t ≤ 0.2 m, it is determined that the robot hits the obstacle, and r c is the penalty for hitting the obstacle. c p is a set constant, d t-1 is the straight-line distance from the target point at the previous moment, and d t is the straight-line distance from the target point at this moment, and c p (d t-1 - d t ) is the reward value continuously obtained by the robot during the navigation process. Approaching the target point is positive, and moving away from the target point is negative. This setting is to enable the robot to generate timely feedback when performing any action, encourage the robot to continuously move towards the target point, and at the same time make the overall reward function continuous and accelerate the convergence speed of the algorithm. r r is the penalty for consecutive identical actions, which is a fixed constant and is set to -50 in this embodiment.

[0050] As shown in Figures 4(a) and 4(b), the initial state and the training environment during training for the robot built using Gazebo are presented. A square wall with a length and width of 4.5 m is established. The starting point of the robot is set to (0, 1), and the target point is a red square with a length and width of 0.3 m. Eight rectangular static obstacles with a length of 1 m, a width of 0.2 m, and a height of 1 m are fixedly placed within the square wall. There is 1 cylindrical dynamic obstacle with a diameter of 0.2 m and a height of 0.8 m at both the upper left and lower right corners, moving uniformly along the pre-programmed trajectory at a speed of 0.1 m / s.

[0051] Step 4: Since the robot has insufficient experience and the environmental state is unknown at the initial stage of training, situations such as spinning in place or frequently hitting the wall may occur. After these large amounts of ineffective experiences are collected in the experience replay pool, using these experiences by the conventional DDPG algorithm to update the neural network parameters will cause problems such as low success rate of path planning and long time taken for the successfully planned path.

[0052] The present invention proposes a new experience replay method. Specifically, during the process of the robot moving from the starting point, hitting an obstacle or reaching the target point is recorded as a round. All the experiences generated during the intermediate process are stored in this round, and this round is placed in the experience replay pool and given an initial weight (before experience replay, the weights of each round are the same). If the number of experiences in this round < length (set to 16 in this embodiment), it is not placed in the experience replay pool. When the number of rounds in the experience replay pool meets the experience replay condition, experience replay starts, and m rounds are extracted each time for training the neural network parameters.

[0053] To adapt to the training of the sequential LSTM network, randomly select consecutive n experiences from each episode (a total of m * n experiences). While updating the network parameters, calculate the total time difference error TD_ERROR of the n experiences in each episode, which is denoted as the TD_ERROR of this episode. After the update, update the weights of the corresponding episodes in the experience replay pool according to the magnitude of TD_ERROR. The larger the TD_ERROR, the greater the room for improvement in prediction accuracy, and it will better improve the parameter accuracy. This not only ensures that all experiences have the possibility of being drawn, but also the larger the weights corresponding to different episodes, the greater the probability of being drawn. After each experience extraction, it is necessary to wait for the robot to run UPDATE episodes (set to 4 in this embodiment) before performing experience extraction again.

[0054] Specifically, when extracting experiences, divide the episodes in the experience pool into m layers. At the same time, use the random function to generate a random number in each layer (the range of the random number is 0 - 1), and find the i-th item that satisfies the sum of the weights of the first i items being less than this unknown number. In this way, the index can be effectively sampled according to the discrete weights, as shown below:

[0055] batchsize = p total / m

[0056] T target = random() * batchsize + (0 ~ m) * batchsize

[0057] i = sum(arr[0] + arr[1] +... + arr[i - 1]) ≤ T target

[0058] In the formula, p total is the total number of episodes in the experience replay pool, T target is the finally generated random number, and arr is the experience replay pool.

[0059] The experience replay pool is composed of a segment tree. A segment tree is a data structure based on a tree structure, often used in cases where quick query of one-dimensional interval information is required. The leaf nodes of the segment tree correspond to the weights of each episode. The segment tree can efficiently update the weight changes caused by the continuous increase in the number of episodes, reducing the heavy computational workload of repeatedly calculating the weights of each episode in the experience pool by brute-force modification. Compared with brute-force modification, it is also more flexible and can store different types of information according to different requirements, such as interval sum, interval maximum value, and interval minimum value, etc.

[0060] Step 5: The mobile robot applies the improved DDPG algorithm to continuously train and learn in the environments shown in Figures 4(a) and 4(b). After the training is completed, the mobile robot can effectively avoid obstacles in the environment and reach the specified location quickly and safely.

Claims

1. An online path planning method for a mobile robot based on deep reinforcement learning, characterized in that, The steps are as follows: Step 1: The mobile robot obtains obstacle information through the lidar carried by itself, and obtains and processes the robot's position information and target point information in real time through the built-in plug-in of the robot operating system, which together with the obstacle information obtained by the lidar constitutes the state information S of the robot; Step 2: Design a neural network structure model for improving the DDPG algorithm: On the basis of the dual-network structure of the DDPG algorithm, set the Actor networks of the main network and the target network as the cascaded structure of the fully connected layer and the LSTM network; Step 3: Design the state space, action space and reward function of the robot in the improved DDPG algorithm; the reward function consists of four parts, the reward for reaching the target point, the penalty for hitting an obstacle, the reward and penalty for the straight-line distance to the target point, and the penalty for continuous identical actions; Step 4: Place the mobile robot in the set environment and train it using the improved DDPG algorithm, and use the trained neural network to complete the collision-free real-time path planning of the robot; each piece of experience generated by the interaction between the robot and the environment is placed in the experience replay pool according to the setting for training the neural network parameters; and use the improved experience replay method to select experiences for training the neural network; after training, the output of the neural network is the optimal executable action of the robot, which can enable the robot to reach the set location quickly and safely; In the step 2, the neural network structure model of the improved DDPG algorithm is specifically as follows: The DDPG algorithm consists of 4 network modules, including two Actor networks and two Critic networks, where the structures of the two Actor networks are exactly the same, and the structures of the two Critic networks are also exactly the same; The input ends of the DDPG are respectively connected to the two Actor networks and the two Critic networks; one Actor network and one Critic network form a main network of the DDPG, and the remaining two networks form the target network of the DDPG; the network parameters are initialized randomly, the parameters of the main network are updated in real time, and the parameters of the target network are updated by sliding according to the parameters of the main network; The input of the main Actor network is the state information of the robot, and the output is the next execution action of the robot; The input of the target Actor network is the next state information of the robot, and the output is the best action predicted to be executed after reaching this state, rather than the actual next action to be executed; The input of the main Critic network is the current state and the action executed in the current state, and the output is the evaluation value of executing the corresponding action in this state; the input of the target Critic network is the next state information and the predicted execution action, and the output is the evaluation value of executing the predicted action in this state; Combine the Actor networks of the main network and the target network with the LSTM network, that is, cascade a layer of LSTM network in front of the Actor networks of the main network and the target network; Use the fractional gradient descent method algorithm with added momentum term to update the parameters of the Critic network; the specific formula of the momentum fractional gradient descent algorithm is as follows: where \(M(t)\) is the weight momentum term at time \(t\), and \(N(t)\) is the bias momentum term at time \(t\). represents the \(q\)-th corresponding weight value of the \(i\)-th neuron in the \(p\)-th layer of the fully connected layer. represents the \(q\)-th corresponding bias value of the \(i\)-th neuron in the \(p\)-th layer of the fully connected layer; represents the current weight value, represents the current bias value; LOSS is the mean square error loss function of the estimated value output by the target network and the evaluated value output by the main network; \(\gamma\) is the fractional order and \(0 < \gamma < 2\), \(\alpha\) is the learning rate and \(0 < \alpha \leq 1\), \(\delta\) is the momentum term coefficient and \(0 < \delta < 1\), \(\delta\) is used to determine the influence degree of the gradient update direction at the previous moment on the current moment; \(\varepsilon\) is a very small positive number, used to avoid singular values when updating parameters; In step 4, the improved experience replay method is as follows: During the process of the robot moving from the starting point, hitting an obstacle or reaching the target point is recorded as one episode. All the experiences generated during the intermediate process are stored in this episode, and this episode is placed in the experience replay pool and given an initial weight. If the number of experiences in this episode < length, it is not put into the experience replay pool; when the number of episodes in the experience replay pool meets the experience replay condition, experience replay starts, and m episodes are randomly selected each time for training the neural network parameters; To adapt to the training of the sequential LSTM network, n consecutive experiences are randomly selected from each episode, so there are a total of m * n experiences. While updating the network parameters, the total sum of the time difference errors TD_ERROR of the n experiences in each episode is calculated and recorded as the TD_ERROR of this episode; after the update is completed, the weight of the corresponding episode in the experience replay pool is updated according to the size of TD_ERROR; after each experience extraction is completed, the robot needs to run UPDATE episodes before performing experience extraction again; Specifically, when extracting experiences, the episodes in the experience pool are evenly divided into m layers. At the same time, a random number between 0 and 1 is randomly generated using the random function in each layer, and the i-th item is found such that the sum of the weights of the first i items is less than this random number, and the index is effectively sampled according to the discrete weights, as shown below: batchsize = p total / m T target = random() * batchsize + (0 to m) * batchsize i = sum(arr[0]+arr[1]+...+arr[i - 1]) ≤ T target where p total is the total number of rounds in the experience replay pool, T target is the finally generated random number, and arr is the experience replay pool.

2. The online path planning method for a mobile robot based on deep reinforcement learning according to claim 1, wherein The obstacle information obtained in step 1 includes the distances between the robot and static obstacles and dynamic obstacles. The relationship between the robot and the obstacles is divided into two types, as shown in the following formula: where d i (t) is the measurement information of the i-th dimensional lidar of the robot at time t, and thr is the distance threshold between the robot and the obstacle; when the distance d i (t) ≤ thr between the robot and the obstacle, the robot hits the obstacle and is in the stagnant state C; when the distance d i (t) > thr between the robot and the obstacle, the robot is in the normal navigation state N.

3. An online path planning method for a mobile robot based on deep reinforcement learning according to claim 1 or 2, characterized in that, The robot state information and target point information in step 1 include the absolute position of the target point relative to the base coordinate system, the difference angle between the robot and the target point relative to the base coordinate system, the relative angle between the robot and the target point, the yaw angle of the robot, and the execution action of the previous step.

4. A mobile robot online path planning method based on deep reinforcement learning according to claim 1 or 2, characterized in that, The reward and punishment for the straight-line distance to the target point in step 3 include whether the straight-line distance from the current position of the robot to the target point is closer than the previous moment. If it is closer, a reward is given, otherwise a punishment is given; the punishment for the occurrence of consecutive same actions includes using a list to retain the consecutive h actions generated by the robot, and this list is continuously updated during the robot's movement, and the reward and punishment are fed back to the total reward according to the setting; if there are more than l consecutive left turns or right turns in the actions, it indicates that the robot is in a stagnant state at this time, which will slow down the speed of reaching the target point and a punishment needs to be given.

5. A method for online path planning of a mobile robot based on deep reinforcement learning according to claim 1 or 2, characterized in that, The experiences generated during the robot's movement in step 4 consist of the robot's current state information, actions, rewards, and the next state information.

6. The online path planning method for a mobile robot based on deep reinforcement learning according to claim 1 or 2, characterized in that Add OU noise to the actions output by the Actor network of the main network.

7. According to the method for online path planning of a mobile robot based on deep reinforcement learning described in claim 1 or 2, the experience replay pool is composed of a segment tree.

Images