Layered reinforcement learning motion planning method and system for multiple mobile robots

Abstract

The invention provides a multi-mobile-robot layered reinforcement learning motion planning method and system, and belongs to the technical field of robots and swarm intelligence. According to the method, a double-layer planner is constructed; the upper planner and the lower planner perform action planning by adopting a neural network model, and are trained through reinforcement learning. The upper layer planner generates the speed of the robot in a continuous time space according to the local map information and the physical information of the robot, and the speed is used as the global guiding speed and transmitted to the lower layer planner. And the lower layer planner takes optimal reciprocal obstacle avoidance ORCA observation and robot self observation as lower layer observation, and generates a robot action planning result by taking the global guidance speed provided by the upper layer planner as a target speed. By using the method, the problem of multi-robot motion planning in a real indoor unstructured environment can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robots and swarm intelligence, and relates to a multi-robot motion planning scheme, in particular to a multi-mobile robot hierarchical reinforcement learning motion planning method and system. BACKGROUND

[0002] Multi-mobile robot coordination systems have been widely used in industrial automation, warehouse logistics, intelligent services, medical care and military fields in recent years. The motion planning technology, which includes global path planning and local obstacle avoidance trajectory planning, is one of the core technologies that affect the coordination efficiency and intelligence performance of the system. Due to the exponential growth of the solution time of the multi-robot motion planning problem with the number of robots and the size of the planning area, existing methods generally decompose the motion planning architecture into two levels of global path shape planning and local trajectory planning in order to balance real-time performance and solution quality. The global layer has global information and only plans a global path shape that reaches the target without collision. The local layer uses less local information to further solve the dynamic obstacle avoidance between robots and output an optimized local path trajectory while tracking the global layer path shape as much as possible. However, this system architecture is prone to triggering local dynamic obstacle avoidance when robots are close to each other, which requires higher requirements for robot maneuverability and real-time solution capability of the planner. Some advanced methods realize this problem by adding a global multi-robot conflict resolution layer after the single-robot global path planning layer to plan the path in advance. However, due to the limitation of computational cost, these methods simplify the kinematic model used to calculate the collision condition to a uniform speed motion model in the ideal grid map, and ignore the local multi-robot obstacle avoidance time cost in the global path planning layer. Essentially, this is still only outputting a path shape, and the occurrence and time of collision are not considered, so the quality of the planning scheme still has room for improvement. A more reasonable multi-robot motion planning architecture should be that the global layer outputs a spatiotemporal trajectory, and the local layer tracks the spatiotemporal trajectory output by the global layer and further processes unknown dynamic obstacle constraints, which is beneficial to further improve the motion planning quality of the multi-mobile robot system. However, under this more reasonable architecture, the solution speed of conventional methods is very slow, so it is limited by applications.

[0003] In recent years, deep reinforcement learning has shown far superior solution efficiency to conventional methods in various complex planning problems, providing technical support for real-time solution of this more reasonable architecture. However, most current research work is limited to simulation environment and discrete action space, and the strategy transfer effect in the real environment is poor, which is difficult to achieve universal application. Moreover, the current deep reinforcement learning method mostly adopts an end-to-end planning strategy when solving the motion planning problem of mobile robots, without strategy decomposition, which leads to low training convergence efficiency and insufficient model flexibility, making it difficult to transfer. Hierarchical reinforcement learning is a reinforcement learning method that simplifies the learning process through a hierarchical structure. By modularizing complex tasks, different modules have more flexibility, so it meets the needs of our new multi-robot motion planning architecture structure design and real-time solution, and is conducive to training convergence and model transfer, so as to achieve more intelligent and efficient application effect in the real multi-robot environment.

[0004] Therefore, the research on the multi-mobile robot hierarchical reinforcement learning motion planning method and system is not only conducive to promoting the progress of multi-robot motion planning technology, but also conducive to providing important technical support for solving the intelligent and automatic needs of multi-robot cooperation in complex scenes. SUMMARY

[0005] Therefore, the research on the multi-mobile robot hierarchical reinforcement learning motion planning method and system is not only conducive to promoting the progress of multi-robot motion planning technology, but also conducive to providing important technical support for solving the intelligent and automatic needs of multi-robot cooperation in complex scenes.

[0006] In order to solve the above technical problems, the present application is implemented as follows.

[0007] A multi-mobile robot hierarchical reinforcement learning motion planning method, comprising: A double-layer planner is constructed; the upper-layer planner and the lower-layer planner both adopt a neural network model for action planning, and are trained through reinforcement learning; The upper-layer planner generates the velocity of the robot in continuous space-time as a global guide velocity according to local map information and physical information of the robot itself, and transmits the global guide velocity to the lower-layer planner; The lower-layer planner takes the optimal reciprocal collision avoidance (ORCA) observation and the self-observation of the robot as lower-layer observations, and takes the global guide velocity provided by the upper-layer planner as the target velocity in the lower-layer observation, to generate the action planning result of the robot.

[0008] Preferably, the observation space and the action space of the upper-layer planner and the lower-layer planner are set as: Upper-layer observation Including the occupancy map and the cost map in the local field of view, the current position, the historical position and the target position of the robot; Upper-layer action Defined as the velocity of the robot in continuous space , i.e. the global guidance speed; Lower level observation Including the robot's own observation information and two-way obstacle avoidance observation information ;Robot's own observation information Including the robot's current speed Target speed , posture angle and its own radius ; Two-way obstacle avoidance observation information ,in It means that the robot's current position is the center of the circle and the j The observation information corresponding to the neighbor robots, m is the number of neighbor robots; define ,in Indicates the j The ORCA velocity half-plane of the neighbor robot relative to itself, Used to determine whether the robot's current speed is within the ORCA speed half plane If in, The value is 1, otherwise it is 0. Expressing oneself and j The distance between neighbor robots; Lower-level actions Defined as the robot acceleration in continuous space; Among them, the lower observation Target speed in Use the global guidance speed provided by the upper-level planner.

[0009] Preferably, the upper-level planner includes an upper-level planner strategy network model and an upper-level planner value network model, both of which have the same structure and are composed of a convolutional neural network, a fully connected layer and a long short-term memory recurrent neural network; The map information in the upper layer observation is input to the convolutional neural network, the robot position information in the upper layer observation is input to the fully connected layer, the output of the convolutional neural network and the fully connected layer are connected and input to the long short-term memory recurrent neural network; the upper layer planner strategy network model outputs the upper layer action , which is the speed of the robot ; The upper-level planner value network model outputs the upper-level action 's assessed value.

[0010] Preferably, the lower-level planner includes a lower-level planner strategy network model and a lower-level planner value network model, both of which have the same structure and are composed of a recurrent neural network and a fully connected layer in series; The input of the lower-level planner policy network model is a lower-level observation , and the output is a lower-level action ; the input of the lower-level planner value network model is a lower-level observation , and the output is an evaluation value of the current lower-level action ; wherein the target speed in the lower-level observation adopts the speed of the robot provided by the upper-level planner ; during training, the target speed is replaced by the unit vector from the current position of the robot to the global target position multiplied by the maximum speed constraint.

[0011] Preferably, the reinforcement learning training of the upper-level planner and the lower-level planner is as follows: First, train the lower-level planner, including steps a1-a4: Step a1: input the lower-level observation into the lower-level planner policy network model, wherein the target speed is replaced by the unit vector from the current position of the robot to the target position multiplied by the maximum speed constraint, and the lower-level planner policy network model outputs the lower-level action of the robot, and interacts with the environment to obtain the lower-level reward and a new lower-level observation ; Step a2: input the lower-level observation into the lower-level planner value network model to obtain the evaluation value of the current lower-level action ; Step a3: obtain a set of experience samples stored in the respective experience replay buffer of the robot according to the results of steps a1 and a2; wherein, is an identifier of whether the robot has reached the target point; Step a4: randomly extract multiple sets of experience samples from the experience replay buffer, calculate the loss function based on the proximal policy optimization (PPO) algorithm, and update the network parameters according to the loss function; Then train the upper-level planner, including steps b1-b4: Step b1: input the upper-level observation into the upper-level planner policy network model, and the upper-level planner policy network model outputs the upper-level action , i.e., the speed of the robot; the speed of the robot is passed as the target speed to the lower-level planner, which continues to perform the planning task and interacts with the environment to obtain the upper-level reward and a new upper-level observation ; Step b2: Observe the upper layer Input the value network model of the upper layer planner with the current state and action to get the evaluation value of the current upper layer action ; Step b3: Obtain a set of experience samples according to the results of steps b1 and b2 and store them in the respective experience replay buffer of the robot; Step b4: Randomly draw multiple sets of experience samples from the experience replay buffer, calculate the loss function based on the PPO algorithm, and update the network parameters according to the loss function.

[0012] Preferably, the reward function of the upper layer planner is: Upper reward is the weighted sum of three rewards, which are the reward when the robot reaches the end point , the reward when the robot collides with the obstacle and the tracking reward used to accelerate convergence at the beginning of training .

[0013] The reward function of the lower layer planner is: Lower reward is the weighted sum of two rewards, which are the reward for tracking the target speed and the collision risk reward ; Preferably, the reward function of the upper layer planner is: wherein is gradually reduced as the training process progresses;

[0014]

[0015]

[0016] wherein, and represent the current position and target position of the robot, represents the current speed of the robot, represents the expected speed generated according to the cost map; The reward function of the lower layer planner is: ,

[0017]

[0018] wherein, a , b , c are adjustable hyperparameters.

[0019] The present invention also provides a multi-mobile robot hierarchical reinforcement learning motion planning system, which includes a two-layer planner and a training module; The two-layer planner consists of an upper-layer planner and a lower-layer planner, both of which are implemented using a neural network model and are used for action planning. The training module uses reinforcement learning to train the two-layer planner. The upper-level planner generates the robot's velocity in continuous spacetime as the global guidance velocity based on the local map information and the robot's own physical information, and passes it to the lower-level planner. The lower-level planner uses ORCA observations and the robot's own observations as the lower-level observations, and uses the global guidance speed provided by the upper-level planner as the target speed in the lower-level observations to generate the robot motion planning results.

[0020] Preferably, the upper-level planner includes an upper-level planner strategy network model and an upper-level planner value network model; The input of the upper-level planner strategy network model is the upper-level observation , including the occupancy map and cost map within the local field of view, the robot's current position, historical position, and target position; the output of the upper-level planner strategy network model is the upper-level action , upper-level action Defined as the robot velocity in continuous space , i.e. the global guidance speed; The input of the upper-level planner value network model is the upper-level observation , the output is the current upper layer action the assessed value; The lower-level planner includes a lower-level planner strategy network model and a lower-level planner value network model; The input of the lower-level planner strategy network model is the lower-level observation , including the robot's own observation information and two-way obstacle avoidance observation information ;Robot's own observation information Including the robot's current speed Target speed , posture angle and its own radius ; Two-way obstacle avoidance observation information ,in It means that the robot's current position is the center of the circle and the j The observation information corresponding to the neighbor robots, m is the number of neighbor robots; define ,in Indicates the j The ORCA velocity half-plane of the neighbor robot relative to itself, to determine whether the current speed of the robot is in the ORCA speed half-plane if, is 1, otherwise 0; denotes the distance between the robot and the j th neighbor robot; the output of the lower planner policy network model is the lower action , the lower action is defined as the acceleration of the robot in continuous space; The input of the lower planner value network model is the lower observation , and the output is the evaluation value of the current lower action ; During training, the target speed of the lower planner is calculated by multiplying the unit vector of the robot's current position to the target position by the maximum speed constraint; during motion planning, the target speed is the global guide speed provided by the upper planner.

[0021] Preferably, during training, the lower planner is first trained separately using the PPO algorithm; then the upper planner and the lower planner are combined, and the upper planner is trained using the PPO algorithm.

[0022] Advantages: (1) The present application provides a general multi-robot hierarchical motion planning architecture to solve the problem of lack of foresight in existing multi-robot cluster motion planning architecture. In a conventional hierarchical planner, the upper layer usually only plans the global path form, and the lower layer further optimizes the local motion trajectory. The hierarchical planner designed in the present application not only plans the global path form, but also plans the space-time trajectory, which is characterized by robot speed. The lower layer generates a smoother and collision-free motion trajectory based on this, thereby realizing the advance planning of congestion and obstacle avoidance factors by the upper planner, avoiding the situation that the robot starts local obstacle avoidance when it is very close to the obstacle in the conventional method. The latter has a larger cost and requires stronger maneuverability for obstacle avoidance and conflict resolution. This planning architecture is suitable for different mobile robot models, and has obvious advantages over the conventional hierarchical architecture in a robot cluster system.

[0023] (2) The application provides a multi-machine layered reinforcement learning motion planning scheme, which adopts a hierarchical neural network to solve the problem of insufficient real-time performance of non-learning methods and the problem that learning methods are difficult to migrate in real indoor scenes. The multi-machine planning system strategy is fitted by using a deep neural network, so that millisecond-level continuous action generation can be realized, which is beneficial to improving the real-time performance of the multi-machine system planning system and realizing more flexible maneuverability response. The hierarchical strategy structure is adopted to decompose the overall strategy network into two layers of global planning and local obstacle avoidance. Compared with conventional reinforcement learning methods, the layered strategy is more likely to obtain a denser reward, which is beneficial to accelerating the learning convergence process. More importantly, when optimizing the global planning strategy, the conventional method usually simplifies the local planning strategy into uniform straight line motion in order to solve the problem within a tolerable limited time, which is inconsistent with the actual local planning strategy interacting with the environment. The global strategy of the present application optimizes its strategy through the feedback of the cost of the unsimplified real local planner interacting with the environment in the training stage, so that the strategy is more conducive to the migration and application in the real environment, and a more optimal solution is obtained.

[0024] (3) The application provides a multi-machine layered motion planning system to solve the problem of multi-machine intelligent smooth obstacle avoidance for indoor unstructured dynamic obstacles, so as to further improve the versatility and practicality of the system in different indoor environments. Firstly, the double-layer planning system provided by the application can guarantee obstacle avoidance for unstructured obstacles, because the upper layer of the planning system of the application not only provides position planning information, but also can provide an ideal speed to avoid unstructured obstacles, and the speed of the upper layer system based on learning is millisecond-level, which can solve the problem of dynamic unstructured obstacle avoidance in real time. Secondly, the lower local planning system further optimizes the mutual avoidance ability between robots through a reciprocal mechanism, and improves the intelligent performance of multi-machine cooperative planning. Since the lower layer system provides continuous acceleration, the smoothness of the output trajectory of the planning system can be improved, and multi-robot smooth obstacle avoidance in various irregular and dynamic obstacles in real indoor environments can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 It is a schematic diagram of the layered training architecture of the application.

[0026] Figure 2 It is a simulation training environment of the application.

[0027] Figure 3 It is the network structure of the upper layer planner and the lower layer planner of the application. DETAILED DESCRIPTION

[0028] The application provides a multi-mobile robot hierarchical reinforcement learning motion planning scheme, and the core idea is to construct a double-layer planner which is different from the conventional upper layer to generate a global path and the lower layer to perform trajectory optimization, but the upper layer performs global space-time path planning to avoid congested road sections in advance, and the lower layer generates a smoother collision-free motion trajectory on this basis. The upper layer planner is responsible for global planning, which takes the map and robot position as observation and outputs the planned action, i.e., the speed of the robot, which is used as global guide speed information and transmitted to the lower layer planner. The lower layer planner is responsible for local planning, which takes the optimal reciprocal collision avoidance ORCA observation and the robot's own observation as lower layer observation, and at the same time, the global guide speed provided by the upper layer planner as the target speed to generate the robot action planning result, thereby completing the hierarchical planning process. As can be seen, the global planner of the upper layer of the application can not only perform space-time trajectory planning to avoid congested road sections in advance, but also provide an ideal speed for the local planner to avoid static obstacles, which makes up for the defect of the conventional optimal reciprocal collision avoidance ORCA method that cannot solve unstructured obstacles, and better adapts to the real indoor environment with irregular obstacles.

[0029] Moreover, the double-layer planner of the application adopts a neural network model and is trained through reinforcement learning. In a densely congested scene, as the number of robots increases, the application adopts reinforcement learning to learn the mechanism of a fast multi-robot obstacle avoidance method in a distributed manner, which can reduce the communication cost while improving the training speed, planning speed and deployment quality.

[0030] The application will be described in detail below with reference to the accompanying drawings and examples.

[0031] As shown in Figure 1 , the application provides a multi-mobile robot hierarchical reinforcement learning motion planning method in a real indoor scene, which specifically includes the following steps: Step 1: Build a simulation environment.

[0032] In this step, a two-dimensional simulation environment based on Unity is built, as shown in Figure 2 , which includes a two-dimensional map, random static obstacles, mobile robots and random target points, wherein the initial position of the robot in each round is random, and the simulation time can be accelerated. The coordinate axis and origin of the two-dimensional map are defined.

[0033] Step 2: Use the simulation environment to build a multi-mobile robot hierarchical learning motion planning system.

[0034] In this embodiment, the multi-mobile robot hierarchical learning motion planning system comprises an upper planner and a lower planner. A series of initial information is obtained from a simulation environment, including the current pose, velocity, acceleration of the robot, and the rasterized map information. The observation space, action space and reward function of the robot in each layer planner are defined using the above initial information.

[0035] (1) The observation space, action space and reward function of the upper planner, specifically: Observation space: upper observation of a single robot including map information and position information; the map information specifically refers to a local occupancy map and a local cost map, wherein the local occupancy map represents a raster map within a certain range around the robot as the center, and is represented by a two-dimensional matrix with values of 0 or 1, 1 indicating that there is an obstacle in the grid, and 0 indicating that there is no obstacle in the grid; the range size of the local cost map is the same as that of the local occupancy map, and the value of the matrix represents the cost of moving from a certain grid to the grid where the robot target point is located, and the closer to the target point, the smaller the cost of the grid; the position information specifically refers to the current position, historical position and target position of the robot. Figure 1

[0036] Action space: upper action defined as the velocity of the robot , wherein , the maximum speed here is given according to the actual situation; Reward function: upper reward includes three rewards, which are the reward when the robot reaches the end point , the reward when the robot collides with the obstacle and the tracking reward used to accelerate convergence at the beginning of training , and the immediate reward obtained by the single robot is represented by the weighted sum of the above three rewards, that is, , wherein decreases gradually with the training process.

[0037] In a preferred embodiment, the specific calculation formulas of the above three rewards are as follows:

[0038]

[0039]

[0040] , wherein and represent the current position and target position of the robot, respectively ​represents the current speed of the robot; select the grid with the minimum value in the local cost map of the robot, define as the vector from the current position of the robot to the center of the grid with the minimum value multiplied by a certain coefficient.

[0041] (2) The observation space, action space and reward function of the lower-level planner, specifically: Observation space: the lower-level observation of a single robot is divided into self-observation information and bidirectional obstacle avoidance observation information , wherein the self-observation information includes the current speed of the robot , the target speed , the pose angle and the self-radius , that is , wherein the target speed is provided by the upper-level planner.

[0042] Bidirectional obstacle avoidance observation information , wherein represents the observation information of the j th neighbor robot within a given radius range with the current position of the robot as the center, m is the number of neighbor robots. Define , wherein represents the ORCA speed half-plane of the j th neighbor robot relative to itself, is used to determine whether the current speed of the robot is within the ORCA speed half-plane, if it is within the half-plane, the value is 1, otherwise it is 0, represents the distance between the j th neighbor robot and itself.

[0043] Among them, the optimal reciprocal collision avoidance (ORCA) is a multi-robot obstacle avoidance method with very small communication cost, which can train a multi-robot motion planning strategy with efficient communication, better solution and good real-time performance by combining it with hierarchical reinforcement learning. The basic principle of calculating the ORCA speed half-plane is to regard the agent as a circle, according to its position and speed, calculate the speed obstacle area of the agent relative to other agents, and on this basis, calculate a minimum adjustment speed linear boundary, i.e. ORCA speed half-plane, to ensure that the agent selects a speed within the half-plane to avoid collision.

[0044] Suppose there are moving agents and , which will cause the speed of the agent to collide with within the future time It is expressed as follows:

[0045] in and Represents the intelligent agent and location, and Represents the intelligent agent and The radius, , q represents any point on the plane; this means that if the agent Relative to The speed is not within this range, that is , then the agent and exist No collision will occur during this time. Assume and A collision is imminent, definition is the relative speed The vector closest to the speed barrier boundary, Speed ​​barrier Boundary at point Considering the actual situation, and Each agent takes half of the responsibility for avoiding obstacles. Speed ​​adjustment ,assumed Adjusting the other half, the ORCA velocity half-plane is defined as follows:

[0046] When the agent When you choose a velocity that falls within the ORCA velocity half plane, you will not A collision occurs. Consider the robot itself as an intelligent agent , around A robot is considered an intelligent agent , then the surrounding The ORCA velocity half-plane of the robot relative to itself ; Action Space: Lower-Level Actions Defined as the acceleration of the robot ,in , here is the maximum acceleration given according to actual conditions.

[0047] Reward function: lower layer reward Two kinds of rewards are included, which are the reward of tracking target speed and the reward of collision risk , and finally get , the specific formula is as follows:

[0048]

[0049] wherein, a , b , c are adjustable hyperparameters.

[0050] Figure 3 The specific design of the double-layer planner is shown, as shown in Figure 3 : For the lower-layer planner, a lower-layer planner strategy network model and a lower-layer planner value network model are constructed, wherein the lower-layer planner strategy network model is composed of a recurrent neural network and a full connection layer in series. The structure of the lower-layer planner value network model is consistent with that of the lower-layer planner strategy network model.

[0051] The input of the lower-layer planner strategy network model is the lower-layer observation , and the output is the lower-layer action , i.e. the acceleration of the robot .

[0052] The input of the lower-layer planner value network model is the lower-layer observation , and the output is the evaluation value of the current lower-layer action .

[0053] wherein the target speed in the lower-layer observation adopts the speed of the robot provided by the upper-layer planner ; during training, the target speed is replaced by the calculation of the unit vector from the current position of the robot to the global target position multiplied by the maximum speed constraint.

[0054] For the upper-layer planner, an upper-layer planner strategy network model and an upper-layer planner value network model are constructed. The upper-layer planner strategy network model is composed of a convolutional neural network, a full connection layer and a long short-term memory recurrent neural network. The map information in the upper-layer observation is input to the convolutional neural network, and the robot position information in the upper-layer observation is input to the full connection layer. The outputs of the convolutional neural network and the full connection layer are connected and input to the long short-term memory recurrent neural network. The structure of the upper-layer planner value network model is consistent with that of the upper-layer planner strategy network model.

[0055] The input of the upper-layer planner strategy network model is the upper-layer observation, and the output is the upper-layer action the speed of the robot .

[0056] The input of the upper planner value network model is the upper observation, and the output is the upper action .

[0057] Step 3: The training module trains the lower planner by the PPO algorithm.

[0058] In this step, the lower planner is first trained separately by the PPO algorithm; then the upper planner and the lower planner are combined, and the upper planner is trained by the PPO algorithm.

[0059] This step 3 specifically includes the following sub-steps: Step 3.1: input the lower observation defined in step 2 to the lower planner policy network model, wherein the target speed is calculated by multiplying the unit vector of the current position of the robot to the target position by the maximum speed constraint, and the policy network model outputs the lower action of the robot , and interacts with the environment to obtain the lower reward and the new lower observation .

[0060] Step 3.2: input the lower observation defined in step 2 to the lower planner value network model to obtain the evaluation value of the current lower action , and store a set of experience samples containing training data in the experience replay buffer of the robot, wherein is used to determine whether the robot has reached the target point.

[0061] Step 3.3: randomly extract S a set of experience samples from the experience replay buffer to calculate the loss function of the neural network, wherein the loss function of the lower planner policy network model is:

[0062] wherein, is the ratio between the new and old lower policy, is a hyperparameter that limits the update amplitude of the lower policy, is the advantage function, which is used to calculate the advantage of a certain lower action relative to the average value in a given state. clip means limiting the value of to and . represents expectation.

[0063] The loss function of the lower planner value network model is:

[0064] wherein, is the lower layer cumulative discounted reward value.

[0065] The network parameters are updated according to the loss function using the Adam algorithm.

[0066] Step 3.4: Repeat steps 3.1 to 3.3 until the maximum number of training rounds is reached.

[0067] Step 4: The training module trains the upper layer planner using the PPO algorithm, and finally obtains the trained hierarchical learning motion planning model.

[0068] Step 4 includes the following sub-steps: Step 4.1: The upper layer observation received by the upper layer planner is input to the upper layer planner policy network model, and the model outputs the global guidance velocity, i.e. the upper layer action , which is passed to the lower layer planner as the target velocity , and the lower layer planner continues to perform planning tasks and interacts with the environment to obtain the upper layer reward and new upper layer observation ; Step 4.2: The upper layer observation received by the upper layer planner is input to the upper layer planner value network model to obtain the evaluation value of the current upper layer action A set of experience samples containing training data is stored in the respective experience replay buffer of the robot; Step 4.3: A set of experience samples is randomly extracted from the experience replay buffer S to calculate the loss function of the neural network, wherein the loss function of the upper layer planner policy network model is

[0069] wherein, is the ratio between the new and old upper layer policies, is a hyperparameter that limits the update amplitude of the upper layer policy, is the advantage function, which is used to calculate the advantage of a certain upper layer action relative to the average value in a given state. clip represents limiting the value of to be between and .

[0070] The loss function of the upper layer planner value network model is

[0071] wherein, is the upper layer cumulative discounted reward value.

[0072] The network parameters are updated according to the loss function by using an Adam algorithm.

[0073] Step 4.4: Repeat steps 4.1 to 4.3 until a maximum number of training rounds is reached.

[0074] Step 5: Motion planning.

[0075] In actual motion planning, the upper planner generates a continuous spatiotemporal robot velocity as a global guide velocity according to local map information and robot physical information, and transmits the global guide velocity to the lower planner; the lower planner takes the optimal reciprocal collision avoidance (ORCA) observation and the robot self-observation as lower-level observations, and takes the global guide velocity provided by the upper planner as a target velocity to generate a robot motion planning result.

[0076] To sum up, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-robot hierarchical reinforcement learning motion planning method, characterized in that: include: Build a two-layer planner: Both the upper and lower planners use neural network models for action planning and are trained through reinforcement learning; The upper-level planner generates the robot's velocity in continuous spacetime as the global guidance velocity based on the local map information and the robot's own physical information, and passes it to the lower-level planner. The lower-level planner uses the optimal reciprocal obstacle avoidance ORCA observation and the robot's own observation as the lower-level observation, and uses the global guidance speed provided by the upper-level planner as the target speed in the lower-level observation to generate the robot motion planning result.

2. The multi-mobile robot hierarchical reinforcement learning motion planning method according to claim 1, characterized in that: The observation space and action space of the upper and lower planners are set as: Upper observation Includes the occupancy map and cost map within the local field of view, as well as the robot's current position, historical position, and target position; Upper level action Defined as the robot velocity in continuous space, i.e. the global guidance velocity; Lower level observation Including the robot's own observation information and two-way obstacle avoidance observation information ;Robot's own observation information Including the robot's current speed Target speed , posture angle and its own radius ; Two-way obstacle avoidance observation information ,in It means that the robot's current position is the center of the circle and the j The observation information corresponding to the neighbor robots, m is the number of neighbor robots; define ,in Indicates the j The ORCA velocity half-plane of the neighbor robot relative to itself, Used to determine whether the robot's current speed is within the ORCA speed half plane If in, The value is 1, otherwise it is 0; Expressing oneself and j The distance between neighbor robots; Lower-level actions Defined as the robot acceleration in continuous space; Among them, the lower observation Target speed in Use the global guidance speed provided by the upper-level planner.

3. The multi-mobile robot hierarchical reinforcement learning motion planning method according to claim 1, characterized in that: The upper-level planner includes an upper-level planner strategy network model and an upper-level planner value network model, both of which have the same structure and are composed of a convolutional neural network, a fully connected layer, and a long short-term memory recurrent neural network; The map information in the upper layer observation is input to the convolutional neural network, the robot position information in the upper layer observation is input to the fully connected layer, the output of the convolutional neural network and the fully connected layer are connected and input to the long short-term memory recurrent neural network; the upper layer planner strategy network model outputs the upper layer action , which is the speed of the robot; the upper-level planner value network model outputs the upper-level action 's assessed value.

4. The multi-mobile robot hierarchical reinforcement learning motion planning method according to claim 1, characterized in that: The lower-level planner includes a lower-level planner strategy network model and a lower-level planner value network model, both of which have the same structure and are composed of a recurrent neural network and a fully connected layer in series; The input of the lower-level planner policy network model is the lower-level observation , the output is the lower level action ; The input of the lower-level planner value network model is the lower-level observation , the output is the current lower layer action The evaluation value of Target speed in Use the robot speed provided by the upper planner; during training, the target speed The unit vector from the robot's current position to the global target position is multiplied by the maximum velocity constraint.

5. The multi-mobile robot hierarchical reinforcement learning motion planning method according to claim 1, characterized in that: The reinforcement learning training for the upper and lower planners is: First train the lower-level planner, including steps a1 to a4: Step a1: Observe the lower layer Input to the lower-level planner policy network model, where the target speed The unit vector from the robot's current position to the target position is multiplied by the maximum speed constraint calculation, and the lower-level planner strategy network model outputs the robot's lower-level action , and interact with the environment to get the lower-level rewards and new lower-level observations ; Step a2: Input the lower-level observation into the lower-level planner value network model to obtain the evaluation value of the current lower-level action ; Step a3: Obtain a set of experience samples Stored in the robot's respective experience replay buffer; among them, It is an indicator of whether the robot has reached the target point; Step a4: Randomly extract multiple sets of experience samples from the experience replay buffer, calculate the loss function based on the proximal strategy optimization PPO algorithm, and update the network parameters according to the loss function; Then train the upper planner, including steps b1~b4: Step b1: Upper level observation Input to the upper-level planner policy network model, the upper-level planner policy network model outputs the upper-level action , that is, the speed of the robot; the speed of the robot is taken as the target speed Pass it to the lower-level planner, which continues to perform the planning task and interacts with the environment to obtain the upper-level reward and new upper-level observations ; Step b2: Observe the upper layer Input to the upper-level planner value network model to obtain the evaluation value of the current upper-level action ; Step b3: Obtain a set of experience samples Store the experience replay buffer of each robot; Step b4: Randomly extract multiple groups of experience samples from the experience replay buffer, calculate the loss function based on the PPO algorithm, and update the network parameters according to the loss function.

6. The multi-mobile robot hierarchical reinforcement learning motion planning method according to claim 2, characterized in that: The reward function of the upper planner is constructed as follows: It is the weighted sum of three rewards, which are the rewards when the robot reaches the end point. , rewards when the robot collides with an obstacle and tracking rewards used to accelerate convergence in the early stages of training ; The reward function of the lower-level planner is constructed as follows: It is the weighted sum of two rewards, the two rewards are the rewards for tracking the target speed and collision risk reward .

7. The multi-mobile robot hierarchical reinforcement learning motion planning method according to claim 6, characterized in that: The reward function of the upper planner is: ,in Gradually decreases as the training progresses; in, and Represent the robot’s current position and target position respectively, Indicates the current speed of the robot, Indicates the expected speed generated based on the cost map; The reward function of the lower-level planner is: , in, a 、 b 、 c is an adjustable hyperparameter.

8. A multi-mobile robot hierarchical reinforcement learning motion planning system, characterized by: The system includes a two-layer planner and training module; The two-layer planner consists of an upper-layer planner and a lower-layer planner, which are implemented using a neural network model and are both used for action planning. The training module uses reinforcement learning to train the two-layer planner; The upper-level planner generates the robot's velocity in continuous spacetime as the global guidance velocity based on the local map information and the robot's own physical information, and passes it to the lower-level planner. The lower-level planner uses ORCA observations and the robot's own observations as the lower-level observations, and uses the global guidance speed provided by the upper-level planner as the target speed in the lower-level observations to generate the robot motion planning results.

9. The multi-mobile robot hierarchical reinforcement learning motion planning system according to claim 8, characterized in that: The upper-level planner includes an upper-level planner strategy network model and an upper-level planner value network model; The input of the upper-level planner strategy network model is the upper-level observation , including the occupancy map and cost map within the local field of view, as well as the robot's current position, historical position, and target position; the output of the upper-level planner strategy network model is the upper-level action , upper-level action Defined as the robot velocity in continuous space, i.e. the global guidance velocity; The input of the upper-level planner value network model is the upper-level observation , the output is the current upper layer action the assessed value; The lower-level planner includes a lower-level planner strategy network model and a lower-level planner value network model; The input of the lower-level planner strategy network model is the lower-level observation , including the robot's own observation information and two-way obstacle avoidance observation information ;Robot's own observation information Including the robot's current speed Target speed , posture angle and its own radius ; Two-way obstacle avoidance observation information ,in It means that the robot's current position is the center of the circle and the j The observation information corresponding to the neighbor robots, m is the number of neighbor robots; define ,in Indicates the j The ORCA velocity half-plane of the neighbor robot relative to itself, Used to determine whether the robot's current speed is within the ORCA speed half plane If in, The value is 1, otherwise it is 0; Expressing oneself and j The distance between neighbor robots; the output of the lower-level planner strategy network model is the lower-level action , lower-level actions Defined as the robot acceleration in continuous space; The input of the lower-level planner value network model is the lower-level observation , the output is the current lower layer action the assessed value; During training, the target speed of the lower planner is The unit vector from the robot's current position to the target position is multiplied by the maximum speed constraint; during motion planning, the target speed The global guidance speed provided by the upper-level planner is adopted.

10. The multi-mobile robot hierarchical reinforcement learning motion planning system according to claim 8, characterized in that: During training, the PPO algorithm is first used to train the lower-level planner separately; then the upper-level planner and the lower-level planner are combined, and the PPO algorithm is used to train the upper-level planner.

Images