A robot path planning and control method combining reinforcement learning with recurrent networks
Through reinforcement learning combined with recurrent network methods, robot paths are generated, which solves the problem of difficulty in finding feasible paths in complex scenarios, and realizes efficient path planning under local information constraints.
Patent Information
- Application Number
- CN202210442298.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-25
- Publication Date
- 2025-04-04
- Estimated Expiration
- 2042-04-25
AI Technical Summary
In the prior art, it is difficult to find the feasible path for a robot from the starting point to the target point through an end-to-end method in complex scenarios, especially in static corner walls or obstacle environments.
Using reinforcement learning combined with recurrent network methods, a recurrent network that generates robot paths is constructed, and the network is trained using reinforcement learning to generate robot paths. The method includes using radar information, robot target point information and previous path point information as input, output path points, and gradually generating robot paths.
It can generate relatively optimized local paths under limited information conditions, avoiding dependence on global information, no prediction paths are required, and there is no need to optimize existing paths, and directly obtain feasible paths within the observable range of the robot, improving the path planning efficiency in complex scenarios.
Smart Images
Figure CN114815828B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of robot path planning and control, and in particular to a robot path planning and control method combining reinforcement learning with a recurrent network. Background Art
[0002] At present, many studies are devoted to applying reinforcement learning to robots. In terms of robot movement control, reinforcement learning is also widely used. Among them, there are many applications of robots using end-to-end methods for motion control. The end-to-end method can achieve good results in dense obstacle and dynamic obstacle environments based on local information. However, when encountering some complex static corner walls or obstacles, the end-to-end method needs to spend more time to explore feasible paths. In many cases, the end-to-end method can only find the local optimal action, which makes it impossible to find the target point in complex scenes. Summary of the invention
[0003] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a robot path planning and control method combining reinforcement learning with a recurrent network.
[0004] The purpose of the present invention can be achieved by the following technical solutions:
[0005] A robot path planning method combining reinforcement learning with a recurrent network, the method comprising:
[0006] Constructing a cyclic network for generating a robot path, wherein the cyclic network sequentially generates path points in the robot path;
[0007] Training the recurrent network using a reinforcement learning method;
[0008] Perform robotic path planning using trained recurrent networks.
[0009] Preferably, the cyclic network includes several cascaded path cyclic network models for outputting path points, and the input of the path network model includes radar information, robot target point information, and previous path point information.
[0010] Preferably, the specific method of generating a robot path using a cyclic network includes:
[0011] Establish the robot's local polar coordinate system. The robot's own coordinates in the robot's local polar coordinate system are expressed as Q o (0,0), the path point set composed of the path points in the robot path is expressed as Among them, Q i represents the i-th path point, ρ i , α iCorresponding to the i-th path point Q i Relative to the i-1th path point Q i-1 The displacement distance and rotation angle of the robot are n, and n is the total number of path points that constitute the robot path;
[0012] Get current radar information s , before the path generation is completed s constant;
[0013] Determine the coordinates T(ρ) of the robot target point T in the robot's local polar coordinate system t ,θ t ), before the path generation is completed, T remains unchanged;
[0014] For the kth path point to be generated, the radar information O s 、Robot target point T(ρ t ,θ t ) and the k-1th path point location information Q k-1 (ρ k-1 ,α k-1 ) is input into the path cycle network model, and the path cycle network model outputs the kth path point location information Q k (ρ k ,α k ), k=1, 2,…, n.
[0015] Preferably, the reinforcement learning training recurrent network includes:
[0016] Start multiple processes, train multiple robots simultaneously in the simulation map, and generate robot paths based on the recurrent network;
[0017] A reward function is constructed for each robot path, and a reinforcement learning algorithm is used to update and optimize the recurrent network that generates the path.
[0018] Preferably, the reward function of the robot path is expressed as r:
[0019] r=r c +r n +r s
[0020] Among them, r c is the collision feedback between the generated robot path and obstacles, r n To provide feedback close to the target point, r s To generate path smoothness feedback.
[0021] Preferably, the reward function of the robot path is specifically determined by:
[0022] To establish the robot's local polar coordinate system, the robot's own coordinates in the robot's local polar coordinate system are expressed as Q o (0,0), the path point set composed of the path points in the robot path is expressed as Among them, Q i represents the i-th path point, ρ i , α i Corresponding to the i-th path point Q i Relative to the i-1th path point Q i-1 The displacement distance and rotation angle of the robot are n, and n is the total number of path points that constitute the robot path;
[0023] Calculate collision feedback r c :
[0024]
[0025] Where a is a constant;
[0026] Calculate the feedback r of the approaching target point n :
[0027]
[0028] Among them, d is the distance from the robot's current position to the target point, s i is the distance from the generated i-th path point to the target point;
[0029] Calculate the path smoothness feedback r s :
[0030]
[0031] Where b is a constant;
[0032] Calculate r = r c +r n +r s .
[0033] Preferably, the reinforcement learning algorithm includes a PPO algorithm.
[0034] Preferably, the method of performing robot path planning using a trained recurrent network comprises:
[0035] Deploy the trained path cycle network model to the mobile robot;
[0036] Determine the current position of the robot, select the target point of the robot, turn on the robot's laser radar, and obtain radar information;
[0037] The robot generates path points in sequence according to the robot's current position, radar information, target point information and the path cycle network model, and the generated path points are arranged in sequence to form a robot path point set.
[0038] A robot control method combining reinforcement learning with a recurrent network, the method comprising:
[0039] The path planning method is used to plan the path of the robot, and a robot path is determined in which the robot moves from a current position to a target point, wherein the robot path includes a plurality of path points;
[0040] Control the robot to move sequentially along the planned path points.
[0041] Preferably, after the robot moves along the planned path points for a period of time, the path planning method is used again to plan the path of the robot, and the robot is controlled to move according to the new robot path, and the above process is repeated until the robot reaches the target point.
[0042] Compared with the prior art, the present invention has the following advantages:
[0043] (1) Traditional path planning methods require global information in order to plan the global path. The path planning method provided by the present invention combines reinforcement learning and recurrent networks to generate a more optimized local path through limited information. Therefore, the present invention does not require global information like traditional methods, nor does it need to predict the path, nor does it optimize the already generated path. The method proposed by the present invention can directly obtain a feasible path within the observable range of the robot.
[0044] (2) Since path generation is a relatively rare task in reinforcement learning, the present invention has discovered a recurrent network model for this purpose. In most cases, the previous path will affect the path generated later. The causal relationship between the previous result and the reasoning in the recurrent network can perfectly match the process of sequential generation of path points. The present invention uses the proximal optimization strategy (PPO) method in reinforcement learning to train the recurrent network model to plan the path of the robot. The path planned by this method can generate corresponding optimization results according to the real-time environment. At the same time, such a reasoning process is less time-consuming and requires fewer reasoning times. While local information is limited, it can reason about the unknown environment to a great extent, saving resources and improving efficiency.
[0045] (3) The control method of the robot of the present invention first shifts the application focus of reinforcement learning to the path planning of the robot. Based on the limited information that the robot can obtain, the present invention uses reinforcement learning to generate a local feasible path for the robot, and then controls the movement of the robot based on the feasible path. Different from the end-to-end output method of robot movement control, the present invention needs to add a controller to enable the robot to move along the generated local feasible path, so that it can find the target point in complex scenes and realize the movement control of the robot. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is the radar scanning space O in the embodiment of the present invention s Schematic diagram;
[0047] Figure 2 Schematic diagram of target point T and path in an embodiment of the present invention;
[0048] Figure 3 A schematic diagram of a process of using a recurrent network to generate a robot path in an embodiment of the present invention;
[0049] Figure 4 A schematic diagram of the structure of a path cycle network model in an embodiment of the present invention;
[0050] Figure 5 Schematic diagram of the process of robot path planning and control method combining reinforcement learning with recurrent network in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the following embodiments are merely illustrative in nature, and the present invention is not intended to limit its applicable objects or uses, and the present invention is not limited to the following embodiments.
[0052] Example
[0053] This embodiment provides a robot path planning method combining reinforcement learning with a recurrent network, the method comprising:
[0054] Construct a recurrent network for generating the robot path, and the recurrent network sequentially generates path points in the robot path;
[0055] Reinforcement learning methods are used to train recurrent networks;
[0056] Perform robotic path planning using trained recurrent networks.
[0057] The recurrent network includes several cascaded path recurrent network models for outputting path points, and the input of the path network model includes radar information, robot target point information, and previous path point information.
[0058] The present invention combines reinforcement learning with a recurrent network to generate an optimal path that a wheeled robot can safely follow through environmental information, and uses limited sensor information to explore the surrounding environment to ensure that the robot can complete the mobile task through the path.
[0059] It specifically includes the following parts:
[0060] S1. Definition of path:
[0061] S1-1. The path is represented by a point set, which is called a "path point set". All points in a path point set constitute a path. The path point set is: Lp = {Q1, Q2, Q3, ..., Q n}, where n is the maximum number of path points, Q1, Q2, Q3, ..., Q n They are the 1st, 2nd, 3rd, …, nth path points respectively.
[0062] S1-2. Take the robot's own position as the origin of the robot's local coordinate system and the robot's facing direction as the x-axis to establish the robot's local polar coordinate system. The robot's own coordinates in the robot's local polar coordinate system are expressed as Q o (0,0).
[0063] S1-3, Generate the i-th point Q in order from 1 to n (n is the maximum number of path points, also the length of the path point set) i Relative to the i-1th point Q i-1 The displacement distance and rotation angle (ρ i ,α i ),i=1,2,…,n.
[0064] S1-4, the path point set can be expressed as: Among them, Q i represents the i-th path point, ρ i , α i Corresponding to the i-th path point Q i Relative to the i-1th path point Q i-1 The displacement distance and rotation angle are measured, and n is the total number of path points that constitute the robot path.
[0065] S2. Construction of recurrent network:
[0066] S2-1, such as Figure 1 , Figure 2 As shown, in this embodiment, the robot radar information O s Obtained through the following methods:
[0067] The robot is facing in the direction of 0°, and the left and right ranges of 90° are the effective areas of the robot radar scanning. The area is divided into 180 parts, and a distance data is recorded for an average of 1°. Take the three frames of sensor data scanned by the robot radar sensor, namely:
[0068]
[0069] S2-2. Get the robot's target point information T:
[0070] T((ρ t ,θ t )
[0071] S2-3, path information Lp, the path consists of n points, namely:
[0072] Lp=[Q1,Q2,Q3,…,Q n ]
[0073] S2-4, construct a path loop network model, the input of the path network model includes radar information, robot target point information, and previous path point information. The loop network includes several cascades for outputting path points. The n path loop network models are cascaded to form Figure 3 A recurrent network for generating robot paths.
[0074] S3, cyclic network generates path Lp:
[0075] As attached Figure 3 As shown, all path points are generated sequentially through the cyclic network, and all path points constitute a path point set to generate a path Lp.
[0076] S3-1. The input data required by the loop network is as follows:
[0077] S3-1-1. Radar information at the current moment O s . With the robot facing direction as 0°, the range of 90° on the left and right (i.e. (-90°, 90°)) totals 180°, and 180° is divided into 180 dimensions, each of which represents the distance from the obstacle to the radar sensor within 1°. The closer the obstacle is to the sensor, the smaller the value of this angle. Every time the radar sensor rotates one circle, the radar data updates the 180-dimensional data once. After the sensor rotates three circles, the radar data is updated three times, and 3×180 data (3 frames×180 dimensions) are obtained. Then O s That is 540-dimensional data. Before the path generation is completed, O s constant.
[0078] S3-1-2, the coordinates of the robot target point T in the robot local polar coordinate system T (ρ t ,θ t). T remains unchanged until the path generation is completed.
[0079] S3-1-3, previous point Q k-1 (ρ k-1 ,α k-1 ). Before the path generation is completed, when the i-th point Q is generated i When , the input becomes the previous point Q of the i-th point i-1 .
[0080] S3-2, Path Cycle Network Model:
[0081] The structure of the path cycle network model is as follows Figure 4 As shown, the O generated in step S3-1-1 s After two layers of convolution, the convolution is 256-dimensional data. The 256-dimensional data after convolution is combined with the two-dimensional data T(ρ t ,θ t ) and two-dimensional data Q k-1 (ρ k-1 ,α k-1 ) are merged into 260-dimensional data, and then input into the fully connected layer for processing, and the output point Q k (ρ k ,α k ).
[0082] S3-3, the output of the loop network is as follows:
[0083] S3-3-1. At step k, the output point Q k (ρ k ,α k After generating a path point k, the robot will update its own state, update its own hypothetical predicted position and orientation (state information) to the newly generated path point k, and then use the hypothetical predicted position and orientation, combined with the unchanged radar information O during the path generation process. s And the target point information T to infer the next point Q k+1 (ρ k+1 ,α k+1 ).
[0084] S3-3-2. When all n points in the path point set are generated, the path point set and path Lp generation process is completed, and the radar data O s Update, target point data T is updated.
[0085] S4, training phase:
[0086] Reinforcement learning training recurrent networks involves:
[0087] Start multiple processes, train multiple robots simultaneously in the simulation map, and generate robot paths based on the recurrent network;
[0088] A reward function is constructed for each robot path, and a reinforcement learning algorithm is used to update and optimize the recurrent network that generates the path.
[0089] The specific steps include:
[0090] S4-1. After the path is generated, the reward function is used to judge the path. The reward function of the robot path is expressed as r:
[0091] r=r c +r n +r s
[0092] Among them, r c is the collision feedback between the generated robot path and obstacles, r n To provide feedback close to the target point, r s To generate path smoothness feedback.
[0093] For collision feedback c Determined by:
[0094]
[0095] Wherein, a is a constant, and in this embodiment, the value of a is 10.
[0096] Here, the collision refers to the collision between the generated path and the obstacle, not the collision between the robot and the obstacle during driving. If the generated path collides with the scanned obstacle in the robot's local polar coordinate system, then r c = -10, if the generated path can avoid obstacles and eventually reach the target point, then r c = +10. In other cases r c =0.
[0097] Feedback on whether the target point is close to the target point n :
[0098]
[0099] Among them, d is the distance from the robot's current position to the target point, s i The distance from the generated i-th path point to the target point is expressed as ds i Indicates the length by which the generated i-th path point is closer to the target point than the current robot position. If ds i > 0, it means that the robot is farther away from the target point, and the generated path point is closer to the target point, so the feedback is positive. iIf i is larger, the generated path point may be closer to the target point, (ds i ) / i can balance the feedback of all points to better calculate the impact of each point on the entire path.
[0100] Feedback r on the smoothness of the generated path s By alpha i The deviation from the current direction can reflect the smoothness of the robot's path, expressed as:
[0101]
[0102] Wherein, b is a constant, and in this embodiment, the value of b is 0.0005.
[0103] Calculate r = r c +r n +r s .
[0104] S4-2. Start multiple processes and train multiple robots simultaneously in the simulation map. Use the paths generated by multiple robots and the reward function r defined in S4-1 to update and optimize the recurrent network of the generated paths using the reinforcement learning proximal policy optimization (PPO) algorithm.
[0105] S5. Execution phase
[0106] S5-1. Deploy the trained path cycle network model to the mobile robot.
[0107] S5-2. Select the target point of the robot, turn on the robot's laser radar, and obtain radar information.
[0108] S5-3, the robot based on radar information O s , target point information T and path cycle network model M, generate the followable path Lp at the current moment.
[0109] Based on the above path planning method, such as Figure 5 As shown, this embodiment also provides a robot control method combining reinforcement learning with a recurrent network, the method comprising:
[0110] The path planning method is used to plan the path of the robot, and a robot path is determined in which the robot moves from a current position to a target point, and the robot path includes a plurality of path points;
[0111] The robot's own controller controls its own movements, including the release of information such as linear velocity and angular velocity, and controls the robot to move sequentially along the planned path points.
[0112] After the robot moves along the planned path for a period of time Δt, it will s′, target point data T′, starting position Q0′, and network model M, generate the path Lp′ of the robot at the new position, and the mobile controller issues corresponding instructions again according to the new path until the robot reaches the target point.
[0113] Traditional path planning methods need to know global information in order to plan the global path. The path planning method provided by the present invention combines reinforcement learning with a recurrent network to generate a more optimized local path through limited information. Therefore, the present invention does not need global information like traditional methods, nor does it need to predict the path, let alone optimize the already generated path. The method proposed by the present invention can directly obtain a feasible path within the observable range of the robot. Since the generation of a path is a relatively rare task in reinforcement learning, the present invention has discovered a recurrent network model for this purpose. In most cases, the previous path will affect the path generated later. The causal relationship between the pre-result and the reasoning in the recurrent network can perfectly match the process of sequential generation of path points. The present invention uses the proximal optimization strategy (PPO) method in reinforcement learning to train the recurrent network model to plan the path of the robot. The path planned by such a method can produce corresponding optimization results according to the real-time environment. At the same time, such a reasoning process is less time-consuming and requires fewer reasoning times. While the local information is limited, it can greatly reason about the unknown environment, save resources, and improve efficiency.
[0114] The robot is controlled on the basis of the above-mentioned robot path planning. Therefore, the robot control method of the present invention shifts the application focus of reinforcement learning to the path planning of the robot. The present invention uses reinforcement learning to generate a local feasible path for the robot based on the limited information that the robot can obtain, and then controls the movement of the robot based on the feasible path. Different from the end-to-end output robot movement control method, the present invention needs to add a controller to make the robot move along the generated local feasible path, so that it can find the target point in complex scenes and realize the movement control of the robot.
[0115] The above embodiments are merely examples and do not limit the scope of the present invention. These embodiments can also be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the technical concept of the present invention.
Claims
1. A robot path planning method combining reinforcement learning with recurrent networks, characterized in that: The method includes: Constructing a cyclic network for generating a robot path, wherein the cyclic network sequentially generates path points in the robot path; Training the recurrent network using a reinforcement learning method; Perform robot path planning using trained recurrent networks; Reinforcement learning training recurrent networks involves: Start multiple processes, train multiple robots simultaneously in the simulation map, and generate robot paths based on the recurrent network; Construct a reward function for each robot path and use a reinforcement learning algorithm to update and optimize the recurrent network that generates the path; The reward function of the robot path is expressed as r: r＝r c +r n +r s Among them, r c is the collision feedback between the generated robot path and obstacles, r n To provide feedback close to the target point, r s To generate path smoothness feedback; The specific method of determining the reward function of the robot path includes: To establish the robot's local polar coordinate system, the robot's own coordinates in the robot's local polar coordinate system are expressed as Q o (0,0), the path point set composed of the path points in the robot path is expressed as Among them, Q i represents the i-th path point, ρ i , α i Corresponding to the i-th path point Q i Relative to the i-1th path point Q i-1 The displacement distance and rotation angle of the robot are n, and n is the total number of path points that constitute the robot path; Calculate collision feedback r c : Where a is a constant; Calculate the feedback r of the approaching target point n : Among them, d is the distance from the robot's current position to the target point, s i is the distance from the generated i-th path point to the target point; Calculate the path smoothness feedback r s : Where b is a constant; Calculate r = r c +r n +r s .
2. According to claim 1, a robot path planning method combining reinforcement learning with recurrent networks is characterized in that: The cyclic network includes several cascaded path cyclic network models for outputting path points, and the input of the path network model includes radar information, robot target point information, and previous path point information.
3. The robot path planning method combining reinforcement learning and recurrent network according to claim 2 is characterized in that: The specific method of generating the robot path by the cyclic network includes: Get current radar information s , before the path generation is completed s constant; Determine the coordinates T(ρ) of the robot target point T in the robot's local polar coordinate system t ,θ t ), before the path generation is completed, T remains unchanged; For the kth path point to be generated, the radar information O s 、Robot target point T(ρ t ,θ t ) and the k-1th path point location information Q k-1 (ρ k-1 ,α k-1 ) is input into the path cycle network model, and the path cycle network model outputs the kth path point location information Q k (ρ k ,α k ), k=1, 2,…, n.
4. The robot path planning method combining reinforcement learning and recurrent network according to claim 1 is characterized in that: The reinforcement learning algorithm includes the PPO algorithm.
5. The robot path planning method combining reinforcement learning with recurrent networks according to claim 1, characterized in that: The method of using the trained recurrent network to perform robot path planning includes: Deploy the trained path cycle network model to the mobile robot; Determine the current position of the robot, select the target point of the robot, turn on the robot's laser radar, and obtain radar information; The robot generates path points in sequence according to the robot's current position, radar information, target point information and the path cycle network model, and the generated path points are arranged in sequence to form a robot path point set.
6. A robot control method combining reinforcement learning with a recurrent network, characterized in that: The method includes: The method according to any one of claims 1 to 5 is used to perform path planning for a robot, and a robot path for the robot to move from a current position to a target point is determined, wherein the robot path includes a plurality of path points; Control the robot to move sequentially along the planned path points.
7. The robot control method combining reinforcement learning and recurrent network according to claim 6, characterized in that: After the robot moves along the planned path points for a period of time, the method described in any one of claims 1 to 5 is used again to plan the robot's path, and the robot is controlled to move according to the new robot path, and the above process is repeated until the robot reaches the target point.