Distributed multi-robot system movement control method based on reinforcement learning
By using a path prediction recurrent network combined with reinforcement learning and radar sensors in a multi-robot system, the problems of inaccurate path planning and slow speed in dynamic environments are solved, and the robot can quickly and accurately avoid obstacles in a multi-robot system are realized.
Patent Information
- Application Number
- CN202310691961.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-12
- Publication Date
- 2024-12-17
AI Technical Summary
In multi-robot collaborative systems, existing path planning methods are difficult to achieve fast and accurate obstacle avoidance movement in dynamic environments, especially due to the reduction of interactive information between robots and inaccurate prediction paths.
Using a distributed multi-robot system movement control method based on reinforcement learning, the surrounding environment information of each robot is obtained in real time, and 360-degree data is collected using radar sensors. The path prediction loop network is constructed in combination with PPO algorithm and Markov model, and the reward function is designed to Judge the accuracy and security of the path and generate a continuous and derivable complete prediction path.
It realizes that the robot can quickly and accurately avoid obstacles and move to the set target point in a dynamic environment, improves the accuracy and speed of path planning, and ensures the effective obstacle avoidance ability of multi-robot systems.
Smart Images
Figure CN119148692A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multi-robot mobile control, and in particular to a distributed multi-robot system mobile control method based on reinforcement learning. Background Art
[0002] Robot path planning is an important task in multi-robot collaborative systems. Most current path planning methods are aimed at robot path planning in static environments, that is, they mainly control movement based on manually set planning points or paths. However, in dynamic environments, such as when there are people or other moving objects, these static methods will be greatly limited.
[0003] In addition, in a multi-robot system, if it is a distributed robot system, the interactive information between robots will be drastically reduced. Therefore, it is necessary for the robot to recognize other robots as obstacles and obtain the position and direction of the dynamic obstacle movement based on the sensor information. When the obstacle is within a certain speed range, the robot needs to be able to effectively avoid dynamic and static obstacles.
[0004] In recent years, reinforcement learning technology has been widely used in robot path planning, but in practice there are still defects such as inaccurate predicted paths and low prediction speed, which makes it impossible for robots to quickly and accurately achieve obstacle avoidance and continuous movement. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a distributed multi-robot system mobile control method based on reinforcement learning, which can cyclically and accurately generate predicted paths and control multiple robots accordingly to achieve precise obstacle avoidance movement.
[0006] The object of the present invention can be achieved by the following technical solution: A distributed multi-robot system mobile control method based on reinforcement learning, comprising the following steps:
[0007] S1, respectively obtaining the surrounding environment information of each robot in the multi-robot system in real time;
[0008] S2, inputting the robot's surrounding environment information and the set target point information corresponding to the robot into a pre-built path prediction loop network to obtain multiple predicted path points corresponding to each robot;
[0009] S3, determining a complete predicted path corresponding to each robot according to multiple predicted path points corresponding to each robot;
[0010] S4. Based on the complete predicted path corresponding to each robot, each robot is controlled to move according to the corresponding complete predicted path.
[0011] Furthermore, each robot in the multi-robot system is equipped with a radar sensor for collecting surrounding environment information, specifically distance information between surrounding obstacles and the robot.
[0012] Furthermore, the step S1 specifically includes the following steps:
[0013] S11. Robot M in the multi-robot system collects information through its onboard radar sensor:
[0014]
[0015] in, is the i-th frame information collected by the radar sensor of robot M, and k is the number of consecutive frames of the radar sensor;
[0016] S12: For each frame of sensor information According to the characteristics of the radar, it scans the surrounding environment for a circle and obtains a data at each angle, for a total of 360 data. The data records the distance information of the obstacle from the radar sensor. The closer the obstacle is to the radar sensor, the smaller the data value.
[0017] The farther the obstacle is from the radar sensor, the larger the data value;
[0018] That is: di represents the distance information of the i-th frame collected by the radar sensor;
[0019] S13, summarizing the distance information of all frames of the radar sensor to obtain the surrounding environment information of the robot M:
[0020] Furthermore, the path prediction loop network in step S2 is specifically constructed by combining the PPO (Proximal Policy Optimization) algorithm and the Markov model, and the construction process includes:
[0021] Build a neural network structure to map the surrounding environment information to the input layer, and output the robot's action information after multiple layers of nonlinear changes;
[0022] According to the feedback information of the reward function, the neural network is iteratively trained to obtain the neural network corresponding to the optimal reward value, which is the path prediction recurrent network.
[0023] Furthermore, the reward function includes a first reward and a second reward, wherein the first reward is used to judge whether the predicted path brings the robot closer to the corresponding set target point, i.e., the robot can move toward the corresponding set target point;
[0024] The second reward is used to judge whether the predicted path collides with an obstacle.
[0025] Furthermore, the calculation formula of the first reward is specifically:
[0026]
[0027] Among them, d s The last path point Q for the complete predicted path s The distance between the robot and the target point T, d t Set the distance between the target point T and the current position of the robot. The condition not reach means that the target point has not been reached, and the condition reach goal means that the target point has been reached.
[0028] Furthermore, the calculation formula of the second reward is specifically:
[0029]
[0030] Among them, the condition safe means that all path points on the complete predicted path have no intersection with the safety range of other obstacles, and the condition else represents other situations except the condition safe, that is, the predicted path collides with the obstacle.
[0031] Furthermore, the specific process of obtaining the multiple predicted path points corresponding to the robot in step S2 is:
[0032] The current position of the robot, the surrounding environment information of the robot and the set target point information corresponding to the robot are input into the path prediction recurrent network, and the first predicted path point is output;
[0033] Then the first predicted path point, the robot's surrounding environment information and the robot's corresponding set target point information are input into the path prediction recurrent network, and the second predicted path point is output, and the operation is repeated in sequence to obtain multiple predicted path points.
[0034] Furthermore, the information of the predicted path point includes displacement and rotation angle, and the displacement specifically refers to the displacement between the previous position point of the robot and the current predicted path point;
[0035] The rotation angle specifically refers to the angle that the robot needs to rotate to move from the previous position point to the current predicted path point.
[0036] Furthermore, in step S3, a plurality of predicted path points are optimized in sequence through a curve to obtain a continuous and differentiable arc path, which is a complete predicted path.
[0037] Compared with the prior art, the present invention has the following advantages:
[0038] 1. The present invention obtains the surrounding environment information of each robot in the multi-robot system in real time, and then combines it with the constructed path prediction loop network to obtain multiple predicted path points corresponding to each robot, based on which the complete predicted path corresponding to each robot is determined, and then each robot is controlled to move according to the corresponding complete predicted path. In this way, the complete predicted path corresponding to a single robot can be generated in a real-time dynamic loop, so that each robot can move to the corresponding set target point according to the corresponding predicted path and avoid all surrounding obstacles.
[0039] 2. The present invention combines the PPO algorithm and the Markov model, and designs a reward function, and constructs a path prediction recurrent network through iterative training. The reward function uses the distance between the predicted path and the target point and the degree of obstacle avoidance as evaluation indicators. This can speed up the generation of the predicted path on the one hand, and improve the accuracy of the generated predicted path on the other hand, so that the robot can accurately avoid all surrounding obstacles and move towards the corresponding set target point.
[0040] 3. The present invention uses a radar sensor to collect information about the robot's surrounding environment, and can collect information from 360 angles around the robot, which is conducive to accurately avoiding all surrounding obstacles when subsequently generating a predicted path.
[0041] 4. The present invention utilizes a path prediction recurrent network to cyclically generate multiple predicted path points, wherein the information of each predicted path point includes the displacement between the robot's previous position point and the current predicted path point, and the angle of rotation required for the robot to move from the previous position point to the current predicted path point. Then, through curve optimization, a complete predicted path consisting of an arc path is obtained to ensure that each robot in the multi-robot system can dynamically avoid obstacles and reach the set target point according to its own complete predicted path. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a schematic diagram of the method flow of the present invention;
[0043] Figure 2 A schematic diagram of the robot's predicted path and the distance between the obstacle and the predicted path;
[0044] Figure 3 Schematic diagram of path point evaluation for multi-robot prediction;
[0045] Figure 4 A diagram showing a predicted path generation process of a single robot M in an embodiment;
[0046] Figure 5 4 is an operation flow chart of the multi-robot system in the embodiment. DETAILED DESCRIPTION
[0047] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0048] Example
[0049] like Figure 1 As shown, a distributed multi-robot system mobile control method based on reinforcement learning includes the following steps:
[0050] S1, respectively obtaining the surrounding environment information of each robot in the multi-robot system in real time;
[0051] S2, inputting the robot's surrounding environment information and the set target point information corresponding to the robot into a pre-built path prediction loop network to obtain multiple predicted path points corresponding to each robot;
[0052] S3, determining a complete predicted path corresponding to each robot according to multiple predicted path points corresponding to each robot;
[0053] S4. Based on the complete predicted path corresponding to each robot, each robot is controlled to move according to the corresponding complete predicted path.
[0054] This embodiment applies the above technical solution, and the main contents are:
[0055] 1. A single robot obtains environmental information, including obstacle location, speed, etc., by carrying sensors such as radar.
[0056] 1-1. In the multi-robot system of this embodiment, the total number of robots is N, and robot M is one of the robots.
[0057] 1-2. Robot M obtains obstacle information through radar sensors. To obtain dynamic obstacle information, robot M needs to obtain sensor information of consecutive frames within a continuous time. In this embodiment, the number of consecutive sensor frames is 5. The total information of the five frames of the sensor is recorded as M represents robot M, and the sensor information of each frame is recorded as
[0058] 1-3. For each frame of sensor information According to the characteristics of the laser radar, it scans the surrounding environment for a circle and obtains a data at each angle, for a total of 360 data. The data records the distance information of the obstacle from the sensor. The closer the obstacle is to the sensor, the smaller the data value; the farther the obstacle is from the sensor, the larger the data value. Therefore, di represents the distance information of each frame of the sensor.
[0059] 1-4. Get the total information based on the five frames of sensor data:
[0060] 2. For each of the N robots, the corresponding target point information is pre-set. Taking robot M as an example, the target point of robot M is T M .
[0061] 3. Use the PPO algorithm to build a reinforcement learning network. For each robot, the environmental information O obtained by its own radar is d , the target point T is taken as input.
[0062] 4. Generate predicted path points in a loop. The path is a local path, which is generated by the reinforcement learning network loop based on the obstacle information scanned by the current radar, not a global path.
[0063] 4-1. The starting position of the path is the robot itself. With the robot's own position as the origin and the facing direction as the x-axis, a rectangular coordinate system is established. The first point of the path is itself (0, 0), denoted as Q 0 .
[0064] 4-2. The current five frames of environmental information O d , target point T, first position point Q 0 As the neural network input, generate the first predicted path point Q 1 (ρ 1 , α 1 );
[0065] Predicted path point Q 1 (ρ 1 , α 1 ) 1 For robots from Q 0 Move to Q 1 The displacement, α 1 For robots from Q 0 Move to Q 1 The angle of rotation required.
[0066] 4-3. After optimization, Figure 2 As shown in the figure, in the robot's local rectangular coordinate system, the robot is facing horizontally to the right, the robot's current position is the point at the top of the dotted line, the generated path point is the point on the arc, and the red solid arc is the optimized arc path. The arc radius is The coordinates of the center of the arc are
[0067] 4-4. By analogy, the robot passes the previous predicted path point Q i-1 Generate the current predicted path point Q i (ρ i , α i ), ρ i That is, the robot from Q i-1Move to Q i The displacement, α i That is, the robot from Q i-1 Move to Q i The angle of rotation required.
[0068] 4-5. Each time, the robot generates a total of s path points based on the current information, and then forms a complete path L using path arcs.
[0069] 4-6. Each robot then avoids dynamic obstacles and reaches the corresponding set target point based on its own complete predicted path.
[0070] 5. Based on the output predicted path, the robot moves and reaches a new environment, regenerating the predicted path to cope with the new environment and new dynamic obstacles. By calculating the reward function, the back propagation algorithm trains the neural network and optimizes the neural network to make the robot's actions more accurate.
[0071] The reward function is designed as follows.
[0072] 5-1. Judge whether the path generated by the robot is closer to the target point, so that the robot can move towards the target point. For the last path point Q of the complete path s , Q s The distance from the robot target point T is d s , the distance between the robot target point T and the current robot position is d t Then r d =d t -d s If the distance between the path point generated by the robot and the target point is smaller than the distance between the robot and the target point, it means that the path generated by the robot points to the target point, and a positive reward value is obtained. Otherwise, the reward value is negative, and the larger the difference, the closer the path is to the target point, the better. If the target point is reached, then r d The value of is 15, that is:
[0073]
[0074] 5-2. Determine whether the path generated by the robot collides with obstacles. Figure 3 As shown. Taking robot M as an example, the robot M target point T M , there is a circular safety range for other robots and obstacles. The safety range here means the safe distance that will not collide if the robot moves at its maximum speed and the opponent also moves at this maximum speed. Therefore, as long as the generated predicted path and path points do not intersect with the safety range of other obstacles, no collision will occur. Therefore, if Figure 1As shown, the distance between the obstacle (x, y) and the arc segment path is calculated. If the distance is greater than the safe range, the system determines that this path is a safe path. If a collision occurs, the reward value is -15, that is:
[0075]
[0076] 5-3. For each robot, its reward function is: r = r d +r c .
[0077] 6. In this solution, the working principle of the reinforcement learning network is: by mapping environmental information to the input layer of the neural network, the neural network will output the robot's actions after multiple layers of nonlinear changes, and then update the neural network according to the feedback information of the reward function to improve the output effect. The working principle of the reward function is: since reinforcement learning is essentially a process of learning the optimal strategy, it is necessary to evaluate the robot's actions to judge the pros and cons of the current strategy. Therefore, the reward function is a very important component of reinforcement learning, which plays a role in evaluating the robot's actions and promoting robot learning. The overall prediction path generation process is as follows: Figure 4 shown.
[0078] Through multiple iterations of training the reinforcement learning network, a prediction path generation network with high reward value is obtained. In a distributed multi-robot system, each robot uses the same robot network. Therefore, multiple robots can move in the corresponding environment, share iterative information, collect path evaluations of multiple robots, and jointly complete the iterative process of the robot prediction path generation network, reducing iteration time and improving training efficiency.
[0079] 7. In the multi-robot system of this scheme, the control movement is as follows Figure 5 As shown in the figure, the system gives corresponding target points to different robots. The robots generate their own paths according to the predicted path generation network trained by reinforcement learning. Through their own corresponding paths, the controller generates corresponding movement instructions for the robots, thereby completing the movement and obstacle avoidance tasks.
[0080] In summary, in this solution, multiple robots use radar sensors to obtain information about the surrounding environment, and the PPO network uses a multi-layer perceptron structure. Through the Markov model, a recurrent network of its own is constructed to generate a predicted path. The reward function uses the distance between the predicted path and the target point and the degree of obstacle avoidance as evaluation indicators. Among them, the PPO algorithm is a commonly used deep reinforcement learning algorithm with the characteristics of fast convergence speed, stable training, and strong scalability. Through the constructed path prediction recurrent network, multiple predicted path points in the predicted path are generated in sequence, and a continuous and derivable complete predicted path is obtained by curve optimization of the predicted path points. Applying this solution in practice can effectively solve the problems of inaccurate robot path planning and slow convergence speed in dynamic environments, and ensure that the robot can quickly and accurately achieve obstacle avoidance and continuous movement.
Claims
1. A distributed multi-robot system mobile control method based on reinforcement learning, characterized in that: The following steps are involved: S1, respectively obtaining the surrounding environment information of each robot in the multi-robot system in real time; S2, inputting the robot's surrounding environment information and the set target point information corresponding to the robot into a pre-built path prediction loop network to obtain multiple predicted path points corresponding to each robot; S3, determining a complete predicted path corresponding to each robot according to multiple predicted path points corresponding to each robot; S4. Based on the complete predicted path corresponding to each robot, each robot is controlled to move according to the corresponding complete predicted path.
2. The distributed multi-robot system mobile control method based on reinforcement learning according to claim 1 is characterized in that: Each robot in the multi-robot system is equipped with a radar sensor for collecting surrounding environment information, specifically the distance information between surrounding obstacles and the robot.
3. The distributed multi-robot system mobile control method based on reinforcement learning according to claim 2 is characterized in that: The step S1 specifically includes the following steps: S11. Robot M in the multi-robot system collects information through its onboard radar sensor: in, is the i-th frame information collected by the radar sensor of robot M, and k is the number of consecutive frames of the radar sensor; S12: For each frame of sensor information According to the characteristics of the radar, it scans the surrounding environment for a circle and obtains a data at each angle, for a total of 360 data. The data records the distance information of the obstacle from the radar sensor. The closer the obstacle is to the radar sensor, the smaller the data value. The farther the obstacle is from the radar sensor, the larger the data value; That is: di represents the distance information of the i-th frame collected by the radar sensor; S13, summarizing the distance information of all frames of the radar sensor to obtain the surrounding environment information of the robot M:
4. The distributed multi-robot system mobile control method based on reinforcement learning according to claim 1 is characterized in that: The path prediction recurrent network in step S2 is specifically constructed by combining the PPO algorithm and the Markov model, and the construction process includes: Build a neural network structure to map the surrounding environment information to the input layer, and output the robot's action information after multiple layers of nonlinear changes; According to the feedback information of the reward function, the neural network is iteratively trained to obtain the neural network corresponding to the optimal reward value, which is the path prediction recurrent network.
5. The distributed multi-robot system mobile control method based on reinforcement learning according to claim 4 is characterized in that: The reward function includes a first reward and a second reward, wherein the first reward is used to judge whether the predicted path brings the robot closer to the corresponding set target point, i.e., the robot can move toward the corresponding set target point; The second reward is used to judge whether the predicted path collides with an obstacle.
6. The distributed multi-robot system mobile control method based on reinforcement learning according to claim 5 is characterized in that: The calculation formula of the first reward is specifically: Among them, d s The last path point Q for the complete predicted path s The distance from the robot to the target point T, d t Set the distance between the target point T and the current position of the robot. The condition notreach indicates that the target point has not been reached, and the condition reachgoal indicates that the target point has been reached.
7. The distributed multi-robot system mobile control method based on reinforcement learning according to claim 5 is characterized in that: The calculation formula of the second reward is specifically: Among them, the condition indicates that all path points on the complete predicted path have no intersection with the safety range of other obstacles, and the condition else indicates other situations except the condition, that is, the predicted path collides with the obstacle.
8. The distributed multi-robot system mobile control method based on reinforcement learning according to claim 4 is characterized in that: The specific process of obtaining the multiple predicted path points corresponding to the robot in step S2 is: The current position of the robot, the surrounding environment information of the robot and the set target point information corresponding to the robot are input into the path prediction recurrent network, and the first predicted path point is output; Then the first predicted path point, the robot's surrounding environment information and the robot's corresponding set target point information are input into the path prediction recurrent network, and the second predicted path point is output, and the operation is repeated in sequence to obtain multiple predicted path points.
9. The distributed multi-robot system mobile control method based on reinforcement learning according to claim 8, characterized in that: The information of the predicted path point includes displacement and rotation angle, and the displacement specifically refers to the displacement between the previous position point of the robot and the current predicted path point; The rotation angle specifically refers to the angle that the robot needs to rotate to move from the previous position point to the current predicted path point.
10. A distributed multi-robot system mobile control method based on reinforcement learning according to any one of claims 1 to 9, characterized in that: In the step S3, a plurality of predicted path points are optimized in sequence through a curve to obtain a continuous and differentiable arc path, which is a complete predicted path.