A multi-mobile robot hierarchical reinforcement learning motion planning method and system

Abstract

The present disclosure provides a multi-mobile robot hierarchical reinforcement learning motion planning method and system, belonging to the field of robot and swarm intelligence technology. The method constructs a double-layer planner; the upper-layer planner and the lower-layer planner both adopt a neural network model for action planning and are trained through reinforcement learning. The upper-layer planner generates the velocity of the robot in continuous space-time as a global guide velocity according to local map information and physical information of the robot, and delivers the global guide velocity to the lower-layer planner. The lower-layer planner takes the optimal reciprocal collision avoidance (ORCA) observation and the self-observation of the robot as lower-layer observations, and takes the global guide velocity provided by the upper-layer planner as a target velocity to generate the action planning result of the robot. The present application can solve the multi-robot motion planning problem in a real indoor unstructured environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robots and swarm intelligence, and relates to a multi-robot motion planning scheme, in particular to a multi-mobile robot hierarchical reinforcement learning motion planning method and system. BACKGROUND

[0002] Multi-mobile robot coordination systems have been widely used in industrial automation, warehouse logistics, intelligent services, medical care and military fields in recent years. The motion planning technology, which includes global path planning and local obstacle avoidance trajectory planning, is one of the core technologies that affect the coordination efficiency and intelligence performance of the system. Due to the exponential growth of the solution time of the multi-robot motion planning problem with the number of robots and the size of the planning area, existing methods generally decompose the motion planning architecture into two levels of global path shape planning and local trajectory planning in order to balance real-time performance and solution quality. The global layer has global information and only plans the global path shape that reaches the target without collision. The local layer uses less local information to further solve the dynamic obstacle avoidance between robots and output the optimized local path trajectory while tracking the global layer path shape as much as possible. However, this system architecture is prone to the problem that robots are close to each other before triggering local dynamic obstacle avoidance, which puts higher requirements on the robot maneuverability and the real-time solving ability of the planner. Some advanced methods are aware of this problem and add a global multi-robot conflict resolution layer after the single-robot global path planning layer to plan the path in advance. However, due to the limitation of computational cost, these methods simplify the kinematic model that depends on the collision condition to the uniform speed motion model in the ideal grid map, and ignore the local multi-robot obstacle avoidance time cost in the global path planning layer. Essentially, this is still only outputting the path shape, and the occurrence and time of collision are not considered, so there is still room for improvement in the quality of the planning scheme. A more reasonable multi-robot motion planning architecture should be that the global layer outputs the space-time trajectory, and the local layer tracks the space-time trajectory output by the global layer and further processes unknown dynamic obstacle constraints, which is beneficial to further improve the motion planning quality of the multi-mobile robot system. However, under this more reasonable architecture, the solution speed of conventional methods is very slow, so it is limited by applications.

[0003] In recent years, deep reinforcement learning has demonstrated significantly higher solution efficiency than conventional methods in various complex planning problems, providing technical support for real-time solutions to this more rational architecture. However, much current research is limited to simulation environments and discrete action spaces, resulting in poor policy transfer performance in real-world environments and hindering widespread application. Furthermore, current deep reinforcement learning methods for solving mobile robot motion planning problems mostly employ end-to-end planning strategies without policy decomposition, leading to low training convergence efficiency, insufficient model flexibility, and difficulty in transfer learning. Hierarchical reinforcement learning, a method that simplifies the learning process through a hierarchical structure, modularizes complex tasks, increasing flexibility between modules. Therefore, while meeting the design requirements of our new multi-machine motion planning architecture and real-time solution needs, it also facilitates training convergence and model transfer, enabling more intelligent and efficient applications in real-world multi-machine environments.

[0004] Therefore, researching hierarchical reinforcement learning motion planning methods and systems for multi-mobile robots is not only beneficial to promoting the advancement of multi-robot motion planning technology, but also to providing important technical support for solving the needs of multi-robot collaborative intelligence and automation in complex scenarios. Summary of the Invention

[0005] In view of this, the present invention provides a multi-robot hierarchical reinforcement learning motion planning method and system for solving the multi-robot motion planning problem in real indoor unstructured environments.

[0006] To solve the above-mentioned technical problems, the present invention is implemented as follows.

[0007] A hierarchical reinforcement learning motion planning method for multi-mobile robots includes:

[0008] A two-layer planner is constructed; both the upper and lower planners use neural network models for action planning and are trained through reinforcement learning.

[0009] The upper-level planner generates the robot's continuous temporal and spatial velocity as a global guiding velocity based on local map information and the robot's own physical information, and then passes it to the lower-level planner.

[0010] The lower-level planner uses the optimal reciprocal obstacle avoidance (ORCA) observation and the robot's own observation as the lower-level observation, and uses the global guiding velocity provided by the upper-level planner as the target velocity in the lower-level observation to generate robot motion planning results.

[0011] Preferably, the observation space and action space of the upper-level planner and the lower-level planner are set as follows:

[0012] Upper-level observation The upper layer observation includes the occupancy map and the cost map within the local field of view, the current position, the historical position and the target position of the robot;

[0013] Upper layer action The upper layer action is defined as the robot velocity in continuous space , i.e. the global guidance velocity

[0014] Lower layer observation The lower layer observation includes the robot self-observation information and the bidirectional obstacle avoidance observation information ; the robot self-observation information includes the current velocity , the target velocity , the pose angle and the self-radius ; the bidirectional obstacle avoidance observation information , wherein represents the observation information of the i-th neighbor robot within a given radius range with the current position of the robot as the center, j is the number of neighbor robots; the definition m , wherein represents the ORCA velocity half-plane of the i-th neighbor robot relative to itself, is used to determine whether the current velocity of the robot is within the ORCA velocity half-plane j , if yes, the value is 1, otherwise 0, represents the distance between the self and the i-th neighbor robot Lower layer action j The lower layer action is defined as the robot acceleration in continuous space

[0015] Lower layer action The lower layer action is defined as the robot acceleration in continuous space

[0016] , wherein the target velocity in the lower layer observation adopts the global guidance velocity provided by the upper layer planner. Preferably, the upper layer planner includes an upper layer planner policy network model and an upper layer planner value network model, both of which have the same structure and are composed of a convolutional neural network, a fully connected layer and a long short-term memory recurrent neural network;

[0017] The map information in the upper layer observation is input to the convolutional neural network, and the robot position information in the upper layer observation is input to the fully connected layer. The outputs of the convolutional neural network and the fully connected layer are connected and input to the long short-term memory recurrent neural network. The upper layer planner policy network model outputs the upper layer action

[0018] , i.e. the velocity of the robot ​​; the upper-layer planner value network model outputs an evaluation value of the upper-layer action .

[0019] Preferably, the lower-layer planner comprises a lower-layer planner policy network model and a lower-layer planner value network model, both of which have the same structure and are composed of a recurrent neural network and a fully connected layer in series;

[0020] The input of the lower-layer planner policy network model is the lower-layer observation , and the output is the lower-layer action ; the input of the lower-layer planner value network model is the lower-layer observation , and the output is an evaluation value of the current lower-layer action ; wherein the target speed in the lower-layer observation adopts the speed of the robot provided by the upper-layer planner ; during training, the target speed is replaced by the unit vector from the current position of the robot to the global target position multiplied by the maximum speed constraint.

[0021] Preferably, the reinforcement learning training of the upper-layer planner and the lower-layer planner is as follows:

[0022] First, the lower-layer planner is trained, including steps a1-a4:

[0023] Step a1: input the lower-layer observation into the lower-layer planner policy network model, wherein the target speed is replaced by the unit vector from the current position of the robot to the target position multiplied by the maximum speed constraint, and the lower-layer planner policy network model outputs the lower-layer action of the robot, and interacts with the environment to obtain the lower-layer reward and the new lower-layer observation ;

[0024] Step a2: input the lower-layer observation into the lower-layer planner value network model to obtain the evaluation value of the current lower-layer action ;

[0025] Step a3: obtain a set of experience samples from steps a1 and a2 and store them in the respective experience replay buffer of the robot; wherein, is an identifier of whether the robot has reached the target point;

[0026] Step a4: randomly extract multiple sets of experience samples from the experience replay buffer, calculate the loss function based on the proximal policy optimization (PPO) algorithm, and update the network parameters according to the loss function;

[0027] Then, the upper-layer planner is trained, including steps b1-b4:

[0028] Step b1: upper layer observation is input into the upper layer planner policy network model, and the upper layer planner policy network model outputs an upper layer action , i.e. the speed of the robot ; the speed of the robot is taken as the target speed and passed to the lower layer planner, which continues to perform the planning task and interacts with the environment to obtain an upper layer reward and a new upper layer observation ;

[0029] Step b2: the upper layer observation is input into the upper layer planner value network model to obtain the evaluation value of the current upper layer action ;

[0030] Step b3: a set of experience samples is obtained according to the results of steps b1 and b2 and stored in the respective experience replay buffer of the robot;

[0031] Step b4: multiple sets of experience samples are randomly extracted from the experience replay buffer, a loss function is calculated based on the PPO algorithm, and the network parameters are updated according to the loss function.

[0032] Preferably, the reward function of the upper layer planner is: upper layer reward is the weighted sum of three rewards, and the three rewards are respectively the reward when the robot reaches the end point , the reward when the robot collides with the obstacle and the tracking reward used to accelerate convergence in the early stage of training .

[0033] The reward function of the lower layer planner is: lower layer reward is the weighted sum of two rewards, and the two rewards are respectively the reward for tracking the target speed and the collision risk reward ;

[0034] Preferably, the reward function of the upper layer planner is: , wherein decreases gradually with the training process;

[0035]

[0036]

[0037]

[0038] wherein, and These represent the robot's current position and the target position, respectively. This indicates the robot's current speed. This represents the expected velocity generated based on the cost map;

[0039] The reward function for the lower-level planner is: ,

[0040]

[0041]

[0042] in, a , b , c These are adjustable hyperparameters.

[0043] The present invention also provides a hierarchical reinforcement learning motion planning system for multiple mobile robots, the system comprising a two-layer planner and a training module;

[0044] The two-layer planner consists of an upper-layer planner and a lower-layer planner, both implemented using a neural network model and used for action planning; the training module uses reinforcement learning to train the two-layer planner.

[0045] The upper-level planner generates the robot's continuous temporal and spatial velocity as a global guiding velocity based on local map information and the robot's own physical information, and then passes it to the lower-level planner.

[0046] The lower-level planner uses ORCA observations and the robot's own observations as lower-level observations, and uses the global guiding velocity provided by the upper-level planner as the target velocity in the lower-level observations to generate robot motion planning results.

[0047] Preferably, the upper-level planner includes an upper-level planner policy network model and an upper-level planner value network model;

[0048] The input to the upper-level planner policy network model is the upper-level observation. This includes the occupancy map and cost map within the local field of view, the robot's current position, historical position, and target position; the output of the upper-level planner policy network model is the upper-level action. Upper-level actions Defined as robot velocity in continuous space That is, the global guidance speed;

[0049] The input to the upper-level planner value network model is the upper-level observation. The output is the current parent action. The evaluation value;

[0050] The lower-level planner includes the lower-level planner policy network model and the lower-level planner value network model;

[0051] The input to the lower-level planner policy network model is the lower-level observation. Including the robot's own observation information and two-way obstacle avoidance observation information Robot's own observation information Including the robot's current speed Target speed Posture angle and its own radius Two-way obstacle avoidance observation information ,in This represents the first circle with the robot's current position as the center and a given radius as the boundary. j Observation information corresponding to each neighboring robot m The number of neighboring robots; defined. ,in Indicates the first j The ORCA velocity half-plane of a neighboring robot relative to itself Used to determine whether the robot's current velocity is within the ORCA velocity half-plane Inside, if in, The value is 1 if it is not 1, otherwise it is 0. Indicates oneself and the first j The distance between neighboring robots; the output of the lower-level planner policy network model is the lower-level action. Lower-level actions Defined as robot acceleration in continuous space;

[0052] The input to the lower-level planner value network model is the lower-level observation. The output is the current lower-level action. The evaluation value;

[0053] During training, the target speed of the lower-level planner The calculation is replaced by multiplying the unit vector from the robot's current position to the target position by the maximum velocity constraint; during motion planning, the target velocity... The global guidance speed is adopted using the upper-level planner.

[0054] Preferably, during training, the lower-level planner is first trained separately using the PPO algorithm; then the upper-level planner and the lower-level planner are combined, and the upper-level planner is trained using the PPO algorithm.

[0055] Beneficial effects:

[0056] (1) The application provides a general multi-robot hierarchical motion planning architecture to solve the problem of lack of foresight in existing multi-robot cluster motion planning architecture. In a conventional hierarchical planner, the upper layer usually only plans the global path form, and the lower layer further optimizes the local motion trajectory. The hierarchical planner designed in the application plans not only the global path form, but also the space-time trajectory. The space-time trajectory is characterized by the speed of the robot. The lower layer generates a smoother collision-free motion trajectory on this basis, thereby realizing the early planning of congestion and obstacle avoidance factors by the upper layer planner, avoiding the situation that the robot starts local obstacle avoidance when it is very close to the obstacle in the conventional method. The cost of obstacle avoidance and conflict resolution in the latter will be greater and requires stronger maneuverability. This planning architecture is suitable for different mobile robot models, and has obvious advantages over the conventional hierarchical architecture in a robot cluster system.

[0057] (2) The application provides a multi-robot hierarchical reinforcement learning motion planning scheme. The planning scheme adopts a hierarchical neural network to solve the problem of insufficient real-time of non-learning methods and the difficulty of learning methods to migrate in real indoor scenes. The multi-robot planning system strategy is fitted by a deep neural network, which can generate millisecond-level continuous actions, and is beneficial to improve the real-time of the multi-robot system planning system and realize more flexible maneuverability response. The hierarchical strategy structure decomposes the overall strategy network into two layers of global planning and local obstacle avoidance. Compared with conventional reinforcement learning methods, the hierarchical strategy is more likely to obtain a denser reward, which is beneficial to accelerate the learning convergence process. More importantly, in order to solve the problem within a tolerable limited time, the conventional method usually simplifies the local planning strategy to uniform straight line motion when optimizing the global planning strategy, which is inconsistent with the actual local planning strategy interacting with the environment. The global strategy of our application optimizes its strategy through the feedback of the cost of the unsimplified real local planner interacting with the environment in the training stage, so it is more conducive to the migration and application of the strategy in the real environment, and obtains a more optimal solution.

[0058] (3) The application provides a multi-robot hierarchical motion planning system to solve the multi-robot intelligent smooth obstacle avoidance problem facing indoor unstructured dynamic obstacles, thereby further improving the versatility and practicality of the system in different indoor environments. First, the double-layer planning system provided by the application can guarantee obstacle avoidance for unstructured obstacles because the upper layer of the planning system not only provides position planning information but also provides an ideal speed to avoid unstructured obstacles, and the learning-based upper layer system planning speed is millisecond-level, which can solve the obstacle avoidance problem of dynamic unstructured obstacles in real time. Secondly, the lower layer local planning system further optimizes the mutual avoidance ability between robots through the reciprocal mechanism, and improves the intelligence performance of multi-robot cooperative planning. Since the lower layer system provides continuous acceleration, the smoothness of the planning system output trajectory can be improved, and multi-robot smooth obstacle avoidance in various irregular and dynamic obstacles in real indoor environments can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 It is a hierarchical training architecture of the application.

[0060] Figure 2 It is a simulation training environment of the application.

[0061] Figure 3 It is the network structure of the upper layer planner and the lower layer planner of the application. DETAILED DESCRIPTION

[0062] The application provides a multi-mobile robot hierarchical reinforcement learning motion planning scheme, the core idea of which is to construct a double-layer planner, which is not a conventional upper layer to generate a global path, but a lower layer to optimize the trajectory. The double-layer planner plans a global space-time path in the upper layer to avoid congested sections in advance, and generates a smoother and collision-free motion trajectory on this basis in the lower layer. The upper layer planner is responsible for global planning, which takes the map and robot position as observation and outputs the planned action, i.e. the speed of the robot, which is used as global guide speed information and transmitted to the lower layer planner. The lower layer planner is responsible for local planning, which takes the optimal reciprocal collision avoidance ORCA observation and the robot's own observation as lower layer observation, and takes the global guide speed provided by the upper layer planner as target speed to generate robot action planning results, thereby completing the hierarchical planning process. As can be seen, the global planner of the upper layer of the application not only plans a space-time trajectory to avoid congested sections in advance, but also provides an ideal speed to avoid static obstacles for the local planner, which makes up for the defect of the traditional optimal reciprocal collision avoidance ORCA method that cannot solve unstructured obstacles, and better adapts to the real indoor environment with irregular obstacles.

[0063] Moreover, the double-layer planner of the application adopts a neural network model and is trained through reinforcement learning. In a dense congestion scenario, as the number of robots increases, the application adopts reinforcement learning to learn the mechanism of a fast multi-robot obstacle avoidance method in a distributed manner, thereby reducing the communication cost and improving the training speed, planning speed and deployment quality.

[0064] The application will be described in detail below with reference to the drawings and examples.

[0065] As shown in Figure 1 , the application provides a multi-mobile robot hierarchical reinforcement learning motion planning method in a real indoor scene, specifically including the following steps:

[0066] Step 1: Build a simulation environment.

[0067] In this step, a two-dimensional simulation environment based on Unity is built, as shown in Figure 2 , including a two-dimensional map, random static obstacles, mobile robots, and random target points, wherein the initial position of the robot in each round is random, and the simulation time can be accelerated. The coordinate axis and origin of the two-dimensional map are defined.

[0068] Step 2: Use the simulation environment to build a multi-mobile robot hierarchical learning motion planning system.

[0069] In this embodiment, the multi-mobile robot hierarchical learning motion planning system includes an upper-layer planner and a lower-layer planner. A series of initial information is obtained from the simulation environment, including the current pose, velocity, acceleration of the robot, and the grid information of the map. Using the above initial information, the observation space, action space and reward function of the robot in each layer planner are defined.

[0070] (1) The observation space, action space and reward function of the upper-layer planner are specifically:

[0071] The observation space of a single robot in the upper layer includes map information and position information; the map information is specifically a local occupancy map and a local cost map, wherein the local occupancy map represents a grid map within a certain range around the robot, and is represented by a two-dimensional matrix with values of 0 or 1, 1 indicating that there is an obstacle in the grid, and 0 indicating that there is no obstacle in the grid; the range of the local cost map is the same as that of the local occupancy map, and the value of the matrix represents the cost of moving from a certain grid to the grid where the robot target point is located, and the closer to the target point, the smaller the cost of the grid; the position information is specifically the current position, historical position and target position of the robot. Figure 1 The action space of the upper layer is defined as the velocity of the robot

[0072] ​​​where here is the maximum speed given according to the actual situation;

[0073] Reward function: upper reward includes three kinds of rewards, which are the reward when the robot reaches the end , the reward when the robot collides with the obstacle and the tracking reward used to accelerate convergence at the beginning of training The final immediate reward obtained by a single robot is represented by the weighted sum of the above three rewards, that is where is gradually reduced as the training process progresses.

[0074] In a preferred embodiment, the specific calculation formula of the above three rewards is as follows:

[0075]

[0076]

[0077]

[0078] where, and represent the current position and target position of the robot respectively, represents the current speed of the robot; select the grid with the minimum value in the local cost map of the robot, define as the vector from the current position of the robot to the center of the grid with the minimum value multiplied by a certain coefficient.

[0079] (2) The observation space, action space and reward function of the lower planner, specifically:

[0080] Observation space: the lower observation of a single robot is divided into self observation information and bidirectional obstacle avoidance observation information , wherein the self observation includes the current speed of the robot , the target speed , the pose angle and the self radius , that is , wherein the target speed is provided by the upper planner.

[0081] Bidirectional obstacle avoidance observation information , wherein represents the observation information of the j th neighbor robot in the given radius range with the current position of the robot as the center.m The number of neighboring robots. Definition ,in Indicates the first j The ORCA velocity half-plane of a neighboring robot relative to itself This value is used to determine whether the robot's current velocity is within the ORCA velocity half-plane. If it is, its value is 1; otherwise, it is 0. Indicates the first j The distance between a neighboring robot and itself.

[0082] Among them, Optimal Reciprocal Obstacle Avoidance (ORCA) is a multi-machine obstacle avoidance method with low communication costs. Combining it with hierarchical reinforcement learning can train a multi-machine motion planning strategy with efficient communication, better solutions, and good real-time performance. The basic principle of calculating the ORCA velocity half-plane is to treat the agent as a circle, calculate its velocity obstacle region relative to other agents based on its position and velocity, and then calculate a minimum-adjustment linear velocity boundary, i.e., the ORCA velocity half-plane, to ensure that the agent chooses its velocity within this half-plane to avoid collisions.

[0083] Suppose there exists a moving agent. and This will lead to the future Within a time and The intelligent agents that collide speed It is expressed as follows:

[0084]

[0085] in and Representing intelligent agents respectively and Location, and Representing intelligent agents respectively and radius, q represents any point on the plane; this means that if the agent Compared to The speed is not within this range, that is So, intelligent agents and exist No collision will occur within the specified time. (Assume...) and An impending collision is defined. relative velocity The vector closest to the velocity barrier boundary Speed ​​obstacle Boundary at point The external normal vector at that location. Considering the practical situation, and Each party assumes half of the responsibility for obstacle avoidance; the intelligent agent Speed ​​adjustment ,assumed Adjusting the other half, we obtain the following definition of the ORCA velocity half-plane:

[0086]

[0087] When the intelligent agent When choosing a velocity that falls within the ORCA velocity half-plane, it will not be related to... A collision occurs. The robot itself is considered an intelligent agent. The surrounding area The robot is considered an intelligent agent. Then the surrounding number The robot's ORCA velocity half-plane relative to itself ;

[0088] Action Space: Lower-level Actions Defined as the acceleration of a robot ,in , here This is the maximum acceleration given based on the actual situation.

[0089] Reward function: Lower-level reward It includes two types of rewards, one for the speed at which the target is tracked. and collision risk reward Finally obtained The specific formula is as follows:

[0090]

[0091]

[0092] in, a , b , c These are adjustable hyperparameters.

[0093] Figure 3 The specific design of the two-level planner is shown, such as Figure 3 As shown:

[0094] For the lower-level planner, a policy network model and a value network model are constructed. The policy network model consists of a recurrent neural network and a fully connected layer connected in series. The structure of the value network model is consistent with that of the policy network model.

[0095] The input of the lower layer planner strategy network model is the lower layer observation , and the output is the lower layer action , that is, the acceleration of the robot .

[0096] The input of the lower layer planner value network model is the lower layer observation , and the output is the evaluation value of the current lower layer action .

[0097] Wherein the target speed in the lower layer observation adopts the speed of the robot provided by the upper layer planner ; during training, the target speed is replaced by the unit vector from the current position of the robot to the global target position multiplied by the maximum speed constraint.

[0098] For the upper layer planner, an upper layer planner strategy network model and an upper layer planner value network model are constructed. The upper layer planner strategy network model is composed of a convolutional neural network, a fully connected layer and a long short-term memory recurrent neural network. The map information in the upper layer observation is input to the convolutional neural network, and the robot position information in the upper layer observation is input to the fully connected layer. The outputs of the convolutional neural network and the fully connected layer are connected and input to the long short-term memory recurrent neural network. The structure of the upper layer planner value network model is consistent with that of the upper layer planner strategy network model.

[0099] The input of the upper layer planner strategy network model is the upper layer observation, and the output is the upper layer action , that is, the speed of the robot .

[0100] The input of the upper layer planner value network model is the upper layer observation, and the output is the evaluation value of the upper layer action .

[0101] Step 3: The training module trains the lower layer planner by using the PPO algorithm.

[0102] In this step, the lower layer planner is first trained separately by using the PPO algorithm; then the upper layer planner and the lower layer planner are combined, and the upper layer planner is trained by using the PPO algorithm.

[0103] This step 3 specifically includes the following sub-steps:

[0104] Step 3.1: input the lower layer observation defined in step 2 into the lower layer planner strategy network model, wherein the target speed is replaced by the unit vector from the current position of the robot to the target position multiplied by the maximum speed constraint, and the strategy network model outputs the lower layer action of the robotand interacts with the environment to get the lower-level reward and new lower-level observation .

[0105] Step 3.2: The lower-level observation defined in Step 2 is input into the lower-level planner value network model to get the evaluation value of the current lower-level action Step 3.3: A set of experience samples containing training data is randomly extracted from the experience replay buffer of the robot Step 3.4: The loss function of the neural network is calculated based on the set of experience samples, where the loss function of the lower-level planner policy network model is: Step 3.5: The loss function of the neural network is calculated based on the set of experience samples, where the loss function of the lower-level planner policy network model is: Step 3.6: The loss function of the neural network is calculated based on the set of experience samples, where the loss function of the lower-level planner policy network model is:

[0106] Step 3.3: A set of experience samples containing training data is randomly extracted from the experience replay buffer of the robot S Step 3.4: The loss function of the neural network is calculated based on the set of experience samples, where the loss function of the lower-level planner policy network model is:

[0107]

[0108] where, is the ratio between the new and old lower-level policies, is a hyperparameter that limits the update amplitude of the lower-level policy, is the advantage function used to calculate the advantage of a certain lower-level action relative to the average value in a given state. clip represents limiting the value of to and . represents expectation.

[0109] The loss function of the lower-level planner value network model is:

[0110]

[0111] where, is the lower-level cumulative discounted reward value.

[0112] The network parameters are updated using the Adam algorithm based on the loss function.

[0113] Step 3.4: Repeat Steps 3.1 to 3.3 until the maximum number of training rounds is reached.

[0114] Step 4: The training module trains the upper-level planner using the PPO algorithm, and finally obtains the trained hierarchical learning motion planning model.

[0115] Step 4 includes the following sub-steps:

[0116] Step 4.1: The upper-level observation received by the upper-level planner is input into the upper-level planner policy network model, and the model outputs the global guidance velocity, i.e., the upper-level action the speed as the target speed pass to the lower planner, continue to perform planning tasks by the lower planner, and interact with the environment to get the upper reward and new upper observations ;

[0117] Step 4.2: input the upper observations received by the upper planner into the upper planner value network model to obtain the evaluation value of the current upper action a set of experience samples containing training data are stored in the respective experience replay buffer of the robot;

[0118] Step 4.3: randomly draw S a set of experience samples from the experience replay buffer to calculate the loss function of the neural network, wherein the loss function of the upper planner policy network model is

[0119]

[0120] wherein, is the ratio between the new and old upper policies, is a hyperparameter that limits the update amplitude of the upper policy, is an advantage function used to calculate the advantage of a certain upper action relative to the average value in a given state. clip means limiting the value of to and .

[0121] The loss function of the upper planner value network model is

[0122]

[0123] wherein, is the upper cumulative discounted reward value.

[0124] The network parameters are updated according to the loss function using the Adam algorithm.

[0125] Step 4.4: repeat steps 4.1 to 4.3 until the maximum number of training rounds is reached.

[0126] Step 5: motion planning.

[0127] When actually performing motion planning, the upper planner generates the speed of the robot in continuous space-time as the global guide speed according to the local map information and the physical information of the robot itself, and passes it to the lower planner; the lower planner takes the optimal reciprocal collision avoidance ORCA observation and the robot's own observation as the lower observation, and at the same time takes the global guide speed provided by the upper planner as the target speed to generate the robot motion planning result.

[0128] To sum up, the above is only the preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-mobile robot hierarchical reinforcement learning motion planning method, characterized in that, The system comprises a double-layer planner and a training module. The double-layer planner comprises an upper-layer planner and a lower-layer planner, both of which are implemented by using neural network models and are used for action planning. The upper-layer planner generates a global guiding velocity of the robot in continuous space-time according to local map information and physical information of the robot, and delivers the global guiding velocity to the lower-layer planner. The lower-layer planner generates an action planning result of the robot by using ORCA observation and self-observation of the robot as lower-layer observation, and using the global guiding velocity provided by the upper-layer planner as a target velocity in the lower-layer observation. The observation space and the action space of the upper-layer planner and the lower-layer planner are set as follows: Upper level observations including an occupancy map and a cost map within a local field of view, also including the current position, historical positions, and target position of the robot; Upper layer action defined as the robot velocity in continuous space, i.e. the global guide velocity; Lower layer observation Including the robot's own observation information and two-way obstacle avoidance observation information Robot's own observation information Including the robot's current speed Target speed Posture angle and its own radius Two-way obstacle avoidance observation information ,in This represents the first circle with the robot's current position as the center and a given radius as the boundary. j Observation information corresponding to each neighboring robot m The number of neighboring robots; defined. ,in Indicates the first j The ORCA velocity half-plane of a neighboring robot relative to itself Used to determine whether the robot's current velocity is within the ORCA velocity half-plane Inside, if in, The value is 1 if it is not 1, otherwise it is 0. Indicates oneself and the first j The distance between neighboring robots; Lower layer actions defined as the robot acceleration in continuous space; wherein the lower layer observation the target speed Adopt the global guide speed provided by the upper planner.

2. The multi-robot hierarchical reinforcement learning motion planning method of claim 1, wherein, The upper-layer planner comprises an upper-layer planner strategy network model and an upper-layer planner value network model, both of which have the same structure and are composed of a convolutional neural network, a fully connected layer and a long short-term memory recurrent neural network. The map type information in the upper layer observation is input to a convolutional neural network, and the robot position type information in the upper layer observation is input to a fully connected layer. The outputs of the convolutional neural network and the fully connected layer are connected and input to a long short-term memory recurrent neural network. The upper layer planner strategy network model outputs an upper layer action , i.e., the speed of the robot. The upper layer planner value network model outputs an evaluation value of the upper layer action .

3. The multi-robot hierarchical reinforcement learning motion planning method of claim 1, wherein, The lower-layer planner comprises a lower-layer planner strategy network model and a lower-layer planner value network model, both of which have the same structure and are composed of a recurrent neural network and a fully connected layer. The input of the lower planner policy network model is a lower observation , and the output is a lower action ; The input of the lower layer planner value network model is a lower layer observation , and the output is an evaluation value of a current lower layer action ; wherein the target speed in the lower layer observation adopts the speed of the robot provided by the upper layer planner; during training, the target speed is replaced by the unit vector from the current position of the robot to the global target position multiplied by the maximum speed constraint.

4. The multi-robot hierarchical reinforcement learning motion planning method of claim 1, wherein, The reinforcement learning training of the upper-layer planner and the lower-layer planner is as follows: First, the lower-layer planner is trained, including steps a1-a4: Step a1: the lower layer observation Input the lower layer planner policy network model, where the target speed Instead of multiplying the maximum speed constraint by the unit vector from the current position of the robot to the target position, the lower layer planner policy network model outputs the lower layer action of the robot , and interacts with the environment to get the lower layer reward And the new lower layer observation ; Step a2: input the lower layer observation into the lower layer planner value network model to obtain the evaluation value of the current lower layer action ; Step a3: obtaining a set of experience samples storing in respective experience replay buffers of the robots; wherein, an indication of whether the robot reached the goal point. Step a4: a plurality of experience samples are randomly extracted from an experience replay buffer, a loss function is calculated based on a proximal policy optimization (PPO) algorithm, and network parameters are updated according to the loss function. Then, the upper-layer planner is trained, including steps b1-b4: Step b1: upper layer observation The input is given to the upper planner policy network model, and the upper planner policy network model outputs an upper layer action , i.e. the speed of the robot; the speed of the robot is taken as the target speed is passed to the lower planner, and the lower planner continues to perform the planning task and interacts with the environment to obtain an upper reward and a new upper layer observation ; Step b2: Observe the upper layer Input the value network model of the upper layer planner to obtain the evaluation value of the current upper layer action ; Step b3: Obtain a set of experience samples Storing in the respective experience replay buffer of the robot; Step b4: a plurality of experience samples are randomly extracted from an experience replay buffer, a loss function is calculated based on a PPO algorithm, and network parameters are updated according to the loss function.

5. The multi-robot hierarchical reinforcement learning motion planning method of claim 1, wherein, The reward function of the upper planner is constructed as: upper reward is a weighted sum of three rewards, which are respectively the reward when the robot reaches the goal , the reward when the robot collides with the obstacle , and the tracking reward used to accelerate convergence at the beginning of training ; The reward function of the lower planner is constructed as: lower reward is a weighted sum of two rewards, the reward of tracking the target speed and the collision risk reward .

6. The multi-robot hierarchical reinforcement learning motion planning method of claim 5, wherein, The reward function of the upper planner is: where decreases gradually as the training process progresses; wherein, and respectively represent the current position and the target position of the robot, represents the current velocity of the robot, represents the desired velocity generated according to the cost map; The reward function of the lower-level planner is: , wherein, a , b , c are adjustable hyperparameters.

7. A multi-mobile robot hierarchical reinforcement learning motion planning system, comprising: The system comprises a double-layer planner and a training module. The double-layer planner comprises an upper-layer planner and a lower-layer planner, both of which are implemented by using neural network models and are used for action planning. The training module trains the double-layer planner by using reinforcement learning. The upper-layer planner generates a global guiding velocity of the robot in continuous space-time according to local map information and physical information of the robot, and delivers the global guiding velocity to the lower-layer planner. The lower-layer planner generates an action planning result of the robot by using ORCA observation and self-observation of the robot as lower-layer observation, and using the global guiding velocity provided by the upper-layer planner as a target velocity in the lower-layer observation. The upper-layer planner comprises an upper-layer planner strategy network model and an upper-layer planner value network model. The input of the upper-layer planner policy network model is an upper-layer observation , including an occupancy map and a cost map in a local field of view, and further including a current position, a historical position and a target position of the robot; the output of the upper-layer planner policy network model is an upper-layer action , the upper-layer action is defined as a robot velocity in a continuous space, i.e. the global guide velocity; An input of the upper planner value network model is the upper observation An output is an evaluation value of a current upper action ​ The lower-layer planner comprises a lower-layer planner strategy network model and a lower-layer planner value network model. The input to the lower-level planner policy network model is the lower-level observation. Including the robot's own observation information and two-way obstacle avoidance observation information Robot's own observation information Including the robot's current speed Target speed Posture angle and its own radius Two-way obstacle avoidance observation information ,in This represents the first circle with the robot's current position as the center and a given radius as the boundary. j Observation information corresponding to each neighboring robot m The number of neighboring robots; defined. ,in Indicates the first j The ORCA velocity half-plane of a neighboring robot relative to itself Used to determine whether the robot's current velocity is within the ORCA velocity half-plane Inside, if in, The value is 1 if it is not 1, otherwise it is 0. Indicates oneself and the first j The distance between neighboring robots; the output of the lower-level planner policy network model is the lower-level action. Lower-level actions Defined as robot acceleration in continuous space; The input of the lower-level planner value network model is the lower-level observation , and the output is the evaluation value of the current lower-level action . At training time, the target velocity of the lower planner The unit vector from the current position of the robot to the target position is multiplied by the maximum velocity constraint to calculate a replacement; at motion planning time, the target velocity The global guide velocity provided by the upper planner is adopted.

8. The multi-robot hierarchical reinforcement learning motion planning system of claim 7, wherein, During training, the lower-layer planner is first trained by using a PPO algorithm, and then the upper-layer planner and the lower-layer planner are combined to train the upper-layer planner by using the PPO algorithm.

Images