Robot path planning method and device based on deep reinforcement learning
By using the path planning algorithm and motion control function to work in the underwater robot motion control, the problem of out-of-synchronization between the simulation model and the deep reinforcement learning control algorithm is solved, and the synchronous operation of the model and the algorithm is realized, which improves the convergence speed and stability of the training results.
Patent Information
- Application Number
- CN202411725339.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-03-25
AI Technical Summary
In underwater robot motion control, there is a problem of out-of-synchronization between the simulation model and the deep reinforcement learning control algorithm, which leads to the serious inconsistent effect of the trained control algorithm with the expected goals.
During the operation of the robot simulation model, the path planning algorithm obtains the simulation model status and calculates the next action, uses the motion control function to control the model to execute the action, and updates the network parameters of the path planning algorithm after the model stops running to ensure the synchronous operation of the model and the algorithm.
The synchronous operation of the model and algorithm is realized, the algorithm run time is shortened, the convergence speed and stability of the training results are greatly improved, and the control effect of the path planning algorithm is improved.
Smart Images
Figure CN119689889A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of robot control technology, and specifically relates to a robot path planning method and device based on deep reinforcement learning. Background Art
[0002] At present, the exploration, development and utilization of deep-sea resources are important issues facing mankind in the new century, and they are of great significance to human survival and development. Underwater operation robots, as robots that can operate for a long time in extreme underwater environments, have become an important tool for human exploration of the ocean and have been applied in many fields. At present, the motion control of underwater operation robots has become the key to ensure their normal operation underwater. However, since it is difficult to test directly in a real environment, in order to perform high-precision motion control of underwater robots, control algorithm training is often performed through simulation model simulation to obtain control methods for underwater robots to achieve various control functions.
[0003] Among all simulation models, the simulation model based on the ROS platform has been used in many fields, including the field of underwater robots. In addition, among the many control methods, the underwater robot motion control method based on deep reinforcement learning can be trained in the simulation data model, which has the characteristics of fast response speed and high accuracy.
[0004] In order to achieve the combined training of ROS simulation model and deep reinforcement learning control algorithm, the communication method of publishing topic / subscribing topic based on ROS platform has become the preferred method for information exchange between the two. However, due to the asynchrony between the online operation of simulation model and the online operation of deep reinforcement algorithm, especially when the deep reinforcement algorithm performs real-time update of network functions, the model information that should have been collected by the algorithm within the same running time cannot be received on time because the actual running time of ROS simulation model is too long or too short. There is a lack of timeliness between the algorithm and the model, resulting in the effect of the trained control algorithm being seriously inconsistent with the expected goal. Summary of the invention
[0005] The purpose of the present invention is to provide a robot path planning method and device based on deep reinforcement learning, so as to solve the problems of asynchrony between simulation model and deep optimization algorithm and uncertainty of running time of deep optimization algorithm.
[0006] In order to solve the above technical problems, the present invention provides a robot path planning method based on deep reinforcement learning, which inputs the current state information of the robot into the trained path planning algorithm, and executes actions according to the action information output by the algorithm, so that the robot runs according to the planned path; it is characterized in that the path planning algorithm is trained using a robot simulation model, and the following training steps are repeated during the training process: During the operation of the robot simulation model, the path planning algorithm obtains the state of the robot simulation model and calculates the action that the robot simulation model needs to perform under the state, and the motion control function of the robot simulation model controls the robot simulation model to perform the action within a set time; After the robot simulation model stops running, the network parameters in the path planning algorithm are updated according to the running data.
[0007] Furthermore, the motion control function includes a time-sleep function, which is used to achieve: the robot simulation model executes the action calculated by the path planning algorithm within a set time, and the path planning algorithm is suspended at this time; after exceeding the set time, the robot simulation model stops the action and the path planning algorithm continues to run.
[0008] Furthermore, the path planning algorithm is a DDPG algorithm, which includes four neural networks, and each neural network includes two Actor networks and two Critic networks.
[0009] Furthermore, the network structures of the two Actor networks are the same, and the network structures of the two Critic networks are also the same.
[0010] Furthermore, the Actor network includes four modules connected in sequence, each of which includes a fully connected layer and an activation function.
[0011] Furthermore, the Critic network includes four modules connected in sequence, wherein the first three modules each include a fully connected layer and an activation function, and the last module includes a fully connected layer.
[0012] Furthermore, the state of the robot is represented by a state space tuple, which includes three-dimensional coordinate information and a four-element direction angle; the action of the robot is represented by an action space tuple, which includes a three-dimensional linear velocity and a three-dimensional angular velocity.
[0013] Furthermore, the reward function in the path planning algorithm includes a distance reward function from the target point, a posture reward function and a speed reward function. The distance reward function is used to calculate the corresponding reward value according to the distance between the robot and the target position, and the closer the distance, the greater the reward value; the posture reward function is used to calculate the corresponding reward value according to the robot posture, and the more stable the posture, the greater the reward value; the speed reward function is used to calculate the corresponding reward value according to the robot's forward linear speed, and the greater the forward linear speed, the greater the reward value.
[0014] The present invention is an improved invention, and its beneficial effects are as follows: the robot path planning method based on deep reinforcement learning of the present invention uses a robot simulation model to train the path planning algorithm. During training, the path planning algorithm calculates the next action of the model according to the state of the model, and then the motion control function controls the model to execute the action, and the time for each step of the action is the same. In order to ensure that the path planning algorithm can complete the calculation in the same time, the present invention also places the parameter update step of the path planning algorithm after the model stops running, preventing the model running time from being too long or too short due to the uncertainty of the parameter update time during the running process, ensuring the synchronous operation of the model and the algorithm, and shortening the algorithm running time, greatly improving the convergence speed and stability of the training results, and improving the control effect of the path planning algorithm.
[0015] To solve the above technical problems, the present invention also provides a robot path planning device based on deep reinforcement learning, characterized in that it includes a processor and a memory, and the processor is used to execute a program stored in the memory to implement a path planning method as described in a robot path planning method based on deep reinforcement learning.
[0016] Its beneficial effects are: the robot path planning device based on deep reinforcement learning of the present invention includes a processor and a memory to realize the training of the path planning algorithm using a robot simulation model. During training, the path planning algorithm calculates the next action of the model according to the state of the model, and then the motion control function controls the model to execute the action, and the robot path planning method with the same execution time for each step of the action. In order to ensure that the path planning algorithm can complete the calculation in the same time, the present invention also places the parameter update step of the path planning algorithm after the model stops running, preventing the model running time from being too long or too short due to the uncertainty of the parameter update time during the running process, ensuring the synchronous operation of the model and the algorithm, and shortening the algorithm running time, greatly improving the convergence speed and stability of the training results, and improving the control effect of the path planning algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 is a flowchart of an algorithm establishment of a method embodiment of the present invention; Figure 2 This is a schematic diagram of the Actor network structure of the method embodiment of the present invention; Figure 3 is a schematic diagram of a Critic network structure of an embodiment of the method of the present invention; Figure 4 is an operation diagram of an underwater robot ROS simulation model of a method embodiment of the present invention; Figure 5 is an algorithm operation flow chart of a method embodiment of the present invention; Figure 6 It is a screenshot of the training operation of the underwater robot ROS simulation model using the improved DDPG algorithm in the method embodiment of the present invention. DETAILED DESCRIPTION
[0018] The robot path planning method based on deep reinforcement learning of the present invention uses a robot simulation model to train the path planning algorithm. During training, the path planning algorithm calculates the next action of the model according to the state of the model, and then the motion control function controls the model to execute the action, and the time for each step of the action is the same. In order to ensure that the path planning algorithm can complete the calculation in the same time, the present invention also places the parameter update step of the path planning algorithm after the model stops running, preventing the model from running too long or too short due to the uncertainty of the parameter update time during the running process, ensuring the synchronous operation of the model and the algorithm, and improving the control effect of the path planning algorithm.
[0019] Method Example: This embodiment takes the path planning of an underwater robot as an example, takes the underwater robot simulation model based on the ROS platform as the controlled robot, trains the robot path planning algorithm based on deep reinforcement learning of this application, and the trained algorithm can be used for the path planning of the underwater robot. The ROS version used by the underwater robot ROS model is ROS melodic. Considering the version compatibility of ROS, the entire process needs to be run on the Ubuntu 18.04 operating system.
[0020] like Figure 1 The route planning method of the present invention is constructed by following the steps below: Step 1: Build an improved DDPG neural network structure model as needed, determine the number of network layers of the DDPG algorithm, and construct a noise function and experience replay pool.
[0021] DDPG (Deep Deterministic Policy Gradient Algorithm) is an improved algorithm of DQN (Deep Reinforcement Learning) algorithm. It has 4 neural networks, including two Actor networks and two Critic networks. The network structure of the two Actor networks is the same, and the network structure of the two Critic networks is also the same. Figure 2 and Figure 3 These are the network structures of Actor and Critic respectively. It can be seen that the Actor network consists of four sequentially connected modules, of which the first three modules include fully connected layers and ReLu activation functions, and the last module includes a fully connected layer and a tanh activation function. The Critic network consists of four sequentially connected modules, of which the first three modules include fully connected layers and ReLu activation functions, and the last module only includes a fully connected layer.
[0022] At the same time, one pair of Actor networks and Critic networks form the target network, and the other pair forms the current network. The current Actor network inputs the underwater robot state information and outputs the action that the underwater robot should make in this state; the target Actor network inputs the next state information after executing the action, and outputs the action predicted in the next state. The current Critic network inputs the underwater robot state information and action information, and outputs the evaluation value of the underwater robot under this state and action; the target Critic network inputs the underwater robot's next state information and the action information predicted using the next state, and outputs a predicted evaluation value.
[0023] The initial parameters of all networks are random, and the network of the current network will adopt the adaptive moment estimation (Adam) method to update the parameters online synchronously with the real-time operation of the ROS model, while the network parameters of the target Actor network and the target Critic network will be updated by the network parameters of the corresponding two current networks.
[0024] In addition, the DDPG algorithm also has a noise function and an experience replay pool. The noise function adds noise to the output action of the current Actor network. This embodiment uses OU noise to improve the randomness of the action strategy. The experience replay pool stores the correct simulation training data, which is convenient for calling training data when the network parameters of the current network are updated.
[0025] Step 2: According to the ROS-related communication topics, determine the state space and action space of the underwater robot in the algorithm, and design the information interaction function. The information interaction function includes the topic subscription function and the topic publishing function, which can achieve information interaction with the ROS model. The operation of the underwater robot ROS model is shown in the figure. Figure 4 shown.
[0026] Before designing the state space and action space of the underwater robot ROS model, it is necessary to investigate the topics that the underwater robot ROS can provide state information and action information, and determine the number of elements in the state space and action space according to the content format and actual operation of the topics used by the model to transmit information. On the Ubuntu 18.04 system, after running the ROS simulation model, you can query the topics published and subscribed by the ROS model through ROS commands. After querying the message types of each topic, it is confirmed that nav_msgs / Odometry is an odometer message topic, which contains three-dimensional coordinate data describing the position of the underwater robot and quaternions describing the posture, while the geometry_msgs / Twist topic can send speed information, including linear velocity and angular velocity in three-dimensional coordinates.
[0027] Therefore, in the algorithm of the present invention, the seven-element state space tuple is determined ( is the three-dimensional coordinate position, is the quaternion direction angle, where For time, , and are the angles between the line connecting the underwater robot and the origin of the coordinate system and the three coordinate axes) and the six-element action space tuple ( is the linear speed, is the angular velocity).
[0028] Python-based algorithms need to use information interaction functions to subscribe to and publish topic information from the ROS model, so the rospy module needs to be introduced into the algorithm. rospy is the Python version of the ROS client library, which provides the interface required for Python programming. After clarifying the topic names that the information interaction functions in the algorithm need to publish and subscribe, the topic information can be subscribed and published through the Subscriber function and Publisher function in the rospy module respectively.
[0029] Step 3: Design the underwater robot training reward function, including the distance reward function between the underwater robot and the target point, the posture reward function for the underwater robot to run smoothly, and the speed reward function.
[0030] The reward function is an important function for delivering goals to the DDPG algorithm model, which encourages the Actor network to output results with higher and higher reward values after updating parameters. The reward functions used in this embodiment include the distance reward function of the underwater robot from the target point, the posture reward function for the underwater robot to run smoothly, and the speed reward function, and the formula is roughly as follows: Among them, r is the reward value of each step of the algorithm, d is the distance between the current position of the underwater robot and the target point, is the roll angle calculated by the position quaternion, is the angular velocity, is the forward linear speed, is the reward coefficient (not less than 0).
[0031] It can be seen from the reward function that the algorithm strategy expected by the present invention is not only to reach the target point, but also to reach it with the smallest possible roll angle and angular velocity, while encouraging the underwater robot to move in a straight line.
[0032] Step 4: Design the motion control function for the underwater robot ROS model to achieve the movement of the model at fixed time intervals.
[0033] During the training of the underwater robot ROS model, the underwater robot needs to perform a certain action in a certain state to obtain the next state. For the accuracy of the training, the time for executing each step must be consistent. Therefore, this embodiment adds a time.sleep function in the motion control function to suspend the execution of the algorithm, so that the ROS model executes the action within a fixed time period, and then starts the algorithm to stop the movement of the model.
[0034] Step 5: Open the ROS underwater robot simulation model and run the improved DDPG path planning algorithm. Both are trained synchronously.
[0035] In order to maintain the synchronization of the underwater robot ROS model and the DDPG algorithm running online at the same time, it is far from enough to use only motion control functions. After actual training, it was found that in the training of the later algorithm, the execution interval of the action in the model was longer than that in the early stage. After investigation, it was found that the main reason for this situation was that the time for data update of the experience replay pool and parameter update of the DDPG network in the later stage of the algorithm was longer than that in the early stage, and both functions were not required to be run at every step. Therefore, in order to ensure the stability of the algorithm results, the two processes of data update of the experience replay pool and parameter update of the DDPG network need to be extracted from each step of the algorithm cycle and run uniformly at a fixed time point.
[0036] Figure 5 This is the operation flow chart of the entire algorithm. It can be seen that when the entire algorithm is running, the algorithm obtains the state of the underwater robot ROS simulation model by subscribing to the topic, and then calculates the action performed in this state through the neural network. By publishing the topic, the ROS model uses the motion control function to move in a fixed time period according to the action. At the same time, the experience gained from the interaction between the ROS model and the training algorithm is temporarily stored in an array.
[0037] After the training has been run a certain number of times, the motion control function is used to stop the ROS model movement, discard some of the temporarily stored experience that may have deviations obtained during the stopping process, fill the remaining experience into the experience replay pool, and update the DDPG network at the same time.
[0038] Step 6: Therefore, the underwater robot simulation model is continuously trained using the improved DDPG algorithm (such as Figure 6 ), and even save network parameters for repeated training. After the training is completed, a path planning method that allows the underwater robot to quickly reach the target can be obtained.
[0039] Device Example: A robot path planning device based on deep reinforcement learning of the present invention includes a processor and a memory, wherein the processor is used to execute a program stored in the memory to implement a method as in the method embodiment. The memory includes at least one software function module stored in the memory, and the processor executes various functional applications and data processing by running the software program and module stored in the memory.
[0040] The processor may be a processing device such as a microprocessor MCU, a programmable logic device FPGA, etc. The memory may be any memory that uses electrical energy to store information, such as RAM, ROM, etc.
Claims
1. A robot path planning method based on deep reinforcement learning, which inputs the current state information of the robot into the trained path planning algorithm, and performs actions according to the action information output by the algorithm, so that the robot runs according to the planned path; characterized in that: The path planning algorithm is trained using a robot simulation model, and the following training steps are repeated during the training process: During the operation of the robot simulation model, the path planning algorithm obtains the state of the robot simulation model and calculates the action that the robot simulation model needs to perform under the state, and the motion control function of the robot simulation model controls the robot simulation model to perform the action within a set time; After the robot simulation model stops running, the network parameters in the path planning algorithm are updated according to the running data.
2. The robot path planning method based on deep reinforcement learning according to claim 1, characterized in that: The motion control function includes a time-sleep function, which is used to achieve: the robot simulation model executes the action calculated by the path planning algorithm within a set time, and the path planning algorithm is suspended at this time; after exceeding the set time, the robot simulation model stops the action and the path planning algorithm continues to run.
3. The robot path planning method based on deep reinforcement learning according to claim 1, characterized in that: The path planning algorithm is a DDPG algorithm, which includes four neural networks, each of which includes two Actor networks and two Critic networks.
4. The robot path planning method based on deep reinforcement learning according to claim 3 is characterized in that: The network structures of the two Actor networks are the same, and the network structures of the two Critic networks are also the same.
5. The robot path planning method based on deep reinforcement learning according to claim 4 is characterized in that: The Actor network includes four modules connected in sequence, each of which includes a fully connected layer and an activation function.
6. The robot path planning method based on deep reinforcement learning according to claim 4 is characterized in that: The Critic network includes four modules connected in sequence, wherein the first three modules each include a fully connected layer and an activation function, and the last module includes a fully connected layer.
7. The robot path planning method based on deep reinforcement learning according to claim 1, characterized in that: The state of the robot is represented by a state space tuple, which includes three-dimensional coordinate information and a four-element direction angle; the action of the robot is represented by an action space tuple, which includes three-dimensional linear velocity and three-dimensional angular velocity.
8. The robot path planning method based on deep reinforcement learning according to claim 1, characterized in that: The reward functions in the path planning algorithm include a distance reward function from the target point, a posture reward function and a speed reward function. The distance reward function is used to calculate the corresponding reward value according to the distance between the robot and the target position. The closer the distance, the greater the reward value; the posture reward function is used to calculate the corresponding reward value according to the robot's posture. The more stable the posture, the greater the reward value; the speed reward function is used to calculate the corresponding reward value according to the robot's forward linear speed. The greater the forward linear speed, the greater the reward value.
9. A robot path planning device based on deep reinforcement learning, characterized in that: It comprises a processor and a memory, wherein the processor is used to execute a program stored in the memory to implement the path planning method as described in any one of claims 1 to 8.