Multi-robot path planning method based on course deep reinforcement learning

Abstract

The invention discloses a multi-robot path planning method based on course deep reinforcement learning, and the method comprises the steps: designing each stage of course learning through the thought of course learning, and changing the number of robots in each stage, the size of a map, and the number of obstacles, thereby achieving the task from easiness to difficulty; building a reinforcement learning environment, defining an observation space and an action space of the robot in the environment and a reward function of the environment, and formulating an operation rule of the robot in the environment; a network structure is designed to adapt to the number of robots changing in the environment, a network model is trained through the Qatten algorithm, the multi-robot path planning problem in the continuous space can be solved, and the method has the advantages that compared with a traditional algorithm, path planning can be achieved faster and better, and the path planning efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and multi-robot path planning, and in particular to a multi-robot path planning method based on curriculum deep reinforcement learning. Background Art

[0002] Multi-agent path planning (MAPF) is an important problem in multi-robot systems. The problem is described as a graph paper with the starting and ending positions of n agents and the positions of obstacles. It is required to plan a path from the starting point to the end point for each agent in discrete time steps without collision with each other. Each agent can move to an adjacent cell or wait in place within one time step. MAPF is often used in warehousing logistics, autonomous driving, video games and other fields.

[0003] Traditional multi-agent path planning algorithms are usually based on grid map modeling. Among them, the conflict-based search algorithm (CBS) is one of the most classic and mature algorithms in traditional multi-agent path planning algorithms. The algorithm is divided into two layers. The upper layer uses a binary constraint tree to handle conflicts between robot paths and selects a conflict-free path. The lower layer independently plans a path that satisfies the constraints for each robot. At present, the conflict-based search algorithm and its various improved algorithms have improved the accuracy of the search, but they ignore the continuity of the real environment and the actual movement mode of the robot. As the number of robots increases, the computational complexity of the traditional algorithm will become very large, and the efficiency of path search will decrease.

[0004] Reinforcement learning is a research hotspot in the field of machine learning. It is a method that explores the environment and learns the optimal strategy based on environmental feedback. With the development of artificial neural networks, deep reinforcement learning, which is a combination of reinforcement learning and deep learning, can solve decision-making problems in more complex environments. Deep neural networks are used to extract feature information from the environment, and then reinforcement learning algorithms are used to achieve direct control from input to output through end-to-end learning. According to the reward information obtained through continuous exploration in the environment, the optimization strategy is continuously updated, and finally the optimal strategy with the maximum cumulative reward is obtained. In recent years, deep reinforcement learning methods have begun to be used to solve multi-agent path planning problems. However, since the state of the agent in the continuous space can be very complex and the model training is difficult, most of the multi-agent path planning algorithms based on deep reinforcement learning still model the scene as a grid map, and lack the ability to search in continuous space. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a multi-robot path planning method based on curriculum deep reinforcement learning, which can realize multi-robot path planning in free space and improve path search efficiency.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A multi-robot path planning method based on curriculum deep reinforcement learning, comprising the following steps:

[0008] Step S1, introduce the idea of ​​course learning and design the tasks for each stage of course learning;

[0009] Step S2: Build a reinforcement learning training environment, define the robot's observation space, action space and reward function for the robot's path planning problem in continuous space, and prepare for subsequent reinforcement learning;

[0010] Step S3, training the network model, first designing a network structure to adapt to the changing number of robots in the environment, and then using the Qatten algorithm, a multi-agent reinforcement learning algorithm based on value decomposition, to train the network model.

[0011] Furthermore, in step S1, the specific tasks of each course stage are:

[0012] S11, the first stage, single robot path planning, the goal is to enable the robot to have basic path search and obstacle avoidance capabilities;

[0013] S12, the second stage, multiple robots plan paths in a simple environment. The number of robots can be less than the number of robots required by the target task, so as to achieve the mutual cooperation ability of multiple robots in a simple environment and execute waiting actions at the right time;

[0014] S13, the third stage, keeping the number of robots the same as the previous stage, continuously increasing the size of the environment and the number of obstacles to achieve the transition of the training environment from simple to complex;

[0015] S14, the fourth stage, completes the path planning of the target task.

[0016] Preferably, in step S1, a course stage is added between the third stage and the fourth stage for different target tasks to ensure a smooth transition of the environment.

[0017] Furthermore, in step S2, the observation space of the robot is divided into two parts, namely, observation information related to itself and observation information related to other robots, so as to adapt to the change in the number of robots in different course learning stages;

[0018] The robot's action space is a discrete action space, which is divided into two categories: moving action and waiting action. The robot's path trajectory conforms to the RS curve, and reversing is not allowed;

[0019] Since the rewards of the environment itself are sparse, an artificial potential field method is used to construct a heuristic as the reward function. At the same time, two additional rewards are introduced. One is the fast search reward, which is used to encourage the machine to reach the end point quickly and reduce the stay in the environment. The other is the waiting mechanism reward, which is used to ensure that the agent performs the waiting action at the right time.

[0020] Furthermore, in step S2, the following definition is made for how to judge whether a round is over in the training phase: a round is considered over when all robots on the field have reached the end point or reached the upper limit of the number of steps per round. When a robot is within a certain range of the end point, it is considered to have reached the end point. A robot that has reached the end point will stop where it is and will not disappear. When a robot collides with an obstacle or collides with other robots, or walks out of the boundary, the environment will activate the collision backoff mechanism, and all robots on the field will back off to the state of the previous time step and replan the path.

[0021] Furthermore, in step S3, network models with different structures are used in different stages: in the first stage of the course, there is only one robot in the environment, so a network structure consisting of a layer of fully connected neural network, a layer of GRU recurrent neural network, and a layer of fully connected neural network connected in series is designed. The network only inputs observation information related to each robot and outputs the action Q value; from the second stage of the course to all subsequent stages, the number of robots in the environment is greater than 1. At this time, the network structure retains the network structure of the first stage, and adds a super network structure to process observation information related to other robots; in addition, all robots share parameters to facilitate training in environments with different numbers of robots.

[0022] In step S3, the Qatten algorithm is used to implement the training of the network model. The algorithm is a multi-agent reinforcement learning algorithm based on value decomposition. It uses a multi-head attention mechanism to model the contribution of each agent to the whole, and calculates the overall Q value Qtot through a trainable super network. The loss function calculation and network parameter update method are the same as the DQN algorithm; therefore, when in the first stage of the course, there is only one robot in the environment, and the algorithm degenerates into the DQN algorithm. After the second stage of the course, the number of robots in the environment is greater than 1, and the algorithm is the Qatten algorithm.

[0023] The beneficial effects of the present invention are as follows: by designing the course learning stages from easy to difficult, the training can be made smoother, the scalability of the model can be increased, and the efficiency of model training can be improved. Especially when the target task involves a large number of robots, the problem of difficult training in complex environments can be effectively solved through course learning. The multi-agent reinforcement learning algorithm based on value function decomposition is more effective in solving the problem of fully cooperative scenarios in discrete action space environments. Compared with the Actor-Critic framework of fully distributed learning, this algorithm can better achieve cooperation between robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the overall framework of the present invention.

[0025] Figure 2 Reference flow chart for each stage of course learning in the present invention.

[0026] Figure 3 Schematic diagram of the observation space of the robot in the present invention.

[0027] Figure 4 It is a flow chart of the environment operation mechanism in the present invention.

[0028] Figure 5 This is a neural network structure diagram for the first stage of course learning in the present invention.

[0029] Figure 6 This is a neural network structure diagram for the course learning in other stages except the first stage in the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings.

[0031] Reference Figure 1 to Figure 6 , a multi-robot path planning method based on curriculum deep reinforcement learning, improves the performance of the algorithm in complex environment training by introducing the idea of ​​curriculum learning; defines the observation space and action space of the robot in the environment, enables the algorithm to plan the path in continuous space, designs the reward function, and improves the efficiency of algorithm training; designs the network structure to adapt to different stages of the curriculum.

[0032] like Figure 1 As shown, the present invention proposes a multi-robot path planning method based on curriculum deep reinforcement learning, comprising the following steps:

[0033] Step S1: Introduce the idea of ​​course learning and design the course learning stages from easy to difficult. The process is as follows:

[0034] Figure 2The reference process of the course stages is shown. Each stage corresponds to a subtask, and the target task is finally completed through a series of subtasks from simple to complex.

[0035] S11. Implement single robot path planning. In multi-robot path planning, each robot needs to complete two tasks: the first is to bypass obstacles and find its own destination, and the second is to avoid other robots to prevent conflicts. The goal of this stage is to complete the first task and realize the robot's most basic path search and obstacle avoidance capabilities. Based on this capability, the second task can be realized, which can increase the frequency of encounters between robots to a certain extent, thus making it more conducive to learning the second task.

[0036] S12. Implement path planning in a simple environment for multiple robots. A simple environment generally means that the number of robots is less than the number of robots required by the task, the size of the environment map is smaller than the size of the target task map, and the number of obstacles is also less than the number of obstacles in the target task. The goal of this stage is to achieve cooperation between robots, mainly to let the robots learn when to wait to avoid other vehicles. The waiting action can only be performed under relatively strict conditions. First, the robot can only wait when it encounters other robots. When the robot encounters an obstacle, it cannot wait. Secondly, only one of the two robots that meet can wait, and the other cannot wait, otherwise a deadlock will occur. On the other hand, if in a sparse scene, the correct waiting method is difficult for the robot to explore. An environment with a small number of robots and a small map not only ensures the simplicity of the environment, but also increases the frequency of encounters between robots to promote robots to explore the correct waiting method.

[0037] S13, keep the number of robots the same as the previous stage, and continuously increase the map size and the number of obstacles in the environment to achieve a transition from easy to difficult environment. The goal of this stage is to enable the robot to initially have the ability to plan paths in complex environments. If the environment gap between stages is too large, it will increase the difficulty of training. Therefore, in this stage, the map size and the number of obstacles are gradually increased during the training process to achieve a smooth transition of the environment. The specific implementation method is that when a certain episode is trained, the map size is increased by a and the number of obstacles is increased by a when the environment is initialized, so as to achieve a gradual change from simple to complex environment. It should be noted that the initial environment of this stage should be the same as the environment of the previous stage. The environment at the end of this stage can be slightly more complex than the environment of the target task, that is, the map size and the number of obstacles are slightly larger than the number specified by the target task. The gap between the number of robots in the current stage and the number of robots in the target task can be made up by the number of obstacles, which helps to narrow the gap between the current environment and the target environment and is conducive to the training of the target task.

[0038] S14, complete the path planning of the target task. After completing the training of the first three stages, the robot has initially acquired the ability to plan paths in multi-robot scenarios. The purpose of this stage is to increase the number of robots in the environment to the number required by the target environment, and ultimately enable the robot to have the ability to plan paths in the target environment.

[0039] When the number of robots to be planned is large, an additional transition phase may be added between step S13 and step S14 so that the number of robots in the environment increases smoothly to ensure the stability of the training.

[0040] Step S2: Build a reinforcement learning environment, define the observation space, action space and reward function of the environment, and determine the operating rules of the robot in the environment.

[0041] The multi-robot path planning method based on curriculum deep reinforcement learning proposed in this invention follows the partially observable Markov property and adopts a centralized training and distributed execution training framework. Therefore, each robot can only observe local environmental information.

[0042] The present invention divides the observation space of the intelligent agent into two parts, namely, observation information related to itself and observation information related to other robots. Figure 2 A schematic diagram showing the robot's observation information.

[0043] Observation information about oneself i include:

[0044] (1) Robot a i Current position to its end point g i The distance d g ;

[0045] (2) Robot a i The end point g i The deviation angle θ relative to the robot's forward direction g ;

[0046] (3) Robot a i The distance d from the current position to the nearest obstacle o o ;

[0047] (4) With robot a i The closest obstacle o relative to the robot a i The deviation angle θ in the forward direction o ;

[0048] (5) Robot a i The distance d from the current position to the nearest boundary e ;

[0049] Observations about oneselfi It can be represented by a one-dimensional vector, o i =[d g ,θ g ,d o ,θ o ,d e ].

[0050] Observations related to other robots include:

[0051] (1) Robot a i The distance d from the current position to the nearest other robot aj a ;

[0052] (2) Distance between robots a i Other recent robots j Relative robot a i The deviation angle θ in the forward direction a ;

[0053] (3) Distance between robots a i Other recent robots j To its end point g j The distance g a .

[0054] Observations related to other robots j Represented by a one-dimensional vector, o j =[d a ,θ a ,g a ].

[0055] When conducting centralized training, the global information s of the environment is required. The global information consists of the local observation information of all robots. It can be represented by a one-dimensional vector s = [o 1 ,o 2 ,…,o n ], where o n Represents the observation information of the nth robot [o n i ,o n j ].

[0056] The present invention aims at the path planning problem in continuous space, and needs to consider the path that conforms to the kinematics of the robot. Since the continuous action space will make it difficult for the algorithm to converge, it will increase the complexity of training. Therefore, the present invention discretizes the continuous action space and uses the RS curve to plan a path trajectory that meets the requirements. The robot's actions are divided into two categories: moving and waiting. Among them, the moving action refers to the robot moving forward, left or right one step, and it is stipulated that the robot is prohibited from reversing, which can promote the robot to move towards the end point. Knowing the current position information of the robot (x, y, θ), the RS curve is used to calculate the next position information. The specific calculation formula is as follows:

[0057]

[0058] Where r represents the turning radius of the robot, δ represents the angle of rotation of the robot, s represents the step length of the robot moving forward, and s=r·δ.

[0059] In order to make the robot's movements more flexible, the present invention sets six moving movements, that is, two turning radii are set for each movement direction, so that the robot has a freer movement space, which can facilitate the robot to search for paths and avoid obstacles or other robots.

[0060] In a single-robot path planning environment, the robot needs to reach the end point to be considered successful and to obtain a positive reward value, while in a multi-robot path planning environment, all robots need to reach the end point to obtain a positive reward value. The rewards of the environment are sparse, and since the intermediate states are continuous, it is almost impossible for the robot to reach the end point when training in such an environment. Therefore, it is necessary to evaluate the process of the robot's intermediate movement and densify the sparse rewards to speed up training. The present invention first uses the method of artificial potential field to design a set of heuristics as a reward function to calculate the basic reward of each robot at the current position. The basic idea of ​​the artificial potential field is that obstacles and other robots in the environment provide repulsion to the current robot. Therefore, the closer the distance between the two, the greater the repulsion and the smaller the heuristic value; on the contrary, the end point provides gravity to the current robot. At this time, the closer the distance between the two, the greater the gravity and the greater the heuristic value. The specific calculation formula is as follows:

[0061]

[0062]

[0063]

[0064] r=a·h1+b·h2+c·h3

[0065] Where d1 represents the distance d from the robot’s current position to the nearest obstacle. oand the distance d to the nearest boundary e , the minimum value of the two, that is, d1 = min(d o ,d e );d2 represents the distance d from the current position of the robot to the nearest other robot a , that is, d2=d a When the training is in the first stage of the course, the robot has no observation information related to other robots, and h2 = 0; d3 represents the distance d from the current position of the robot to the end point. g , that is, d3=d g λ represents the range of attraction or repulsion in the potential field.

[0066] The above heuristic reward function cannot take into account all situations of the robot during execution. Therefore, the present invention adds additional rewards on the basis of the heuristic reward function to promote training.

[0067] In order to make the robot reach the end point as quickly as possible and reduce the stay in the environment, the search reward is increased to encourage the robot to move towards the end point. The specific formula is as follows:

[0068]

[0069] Among them, d g Indicates the distance from the robot's current position to the end point, d g 'Indicates the distance from the robot's last position to the end point.

[0070] When the training enters the second stage, the robot needs to cooperate with other robots, that is, to perform waiting actions at the right time to avoid other robots. Therefore, the waiting reward is increased to train the robot to use the waiting action. The specific formula is:

[0071]

[0072] The setting of the waiting condition is the key to whether the robot can correctly learn to use the waiting action. The definition of the waiting condition is as follows: (1) The distance d from the current position of the robot to the nearest other robot a is smaller than x, x should be smaller than the robot’s field of view radius; (2) the angle θ of the nearest other robot relative to the robot’s direction a ∈(-π / 4,π / 4); (3) The distance d from the current robot to the end point g Less than the distance g from the nearest other robot to its destination a , that is, d g <g a When the above three conditions are met at the same time, the current state of the robot is considered to meet the waiting condition.

[0073] In addition, when the robot's current position collides with an obstacle or other robots in the environment or moves outside the boundary, the robot's reward value is directly set to a negative number with a large absolute value; when the robot reaches the end point, the robot's reward value is the robot's remaining steps. The robot's initial remaining steps are the maximum step limit of the environment. The robot's remaining steps decrease by one for each step it takes in the environment. A collision will further reduce the remaining steps, and a minimum positive reward value r is specified. min , so that the reward value obtained by the robot when it reaches the end point is greater than or equal to r min In addition, since the time when the robots reach the finish line is not necessarily synchronized, each robot can only receive the reward for reaching the finish line once per round. While waiting for other robots to reach the finish line, the reward value is all 0.

[0074] At this point, the complete reward obtained by a robot in the current state has been calculated. By adding up the reward values ​​of all robots, we can get the global reward R in the current state.

[0075]

[0076] Where N represents the number of robots.

[0077] The present invention uses a collision backoff mechanism to solve the problem of the robot's state after the robot's current position collides with an obstacle or other robot in the environment or moves outside the boundary. The conventional method is that the robot is considered to have failed after a collision, and then the environment is reset and planning is restarted. Since the rewards in the environment are all negative, when facing a complex environment, the algorithm is easy to fall into a local optimum, and it is believed that the robot directly collides or goes out of bounds to reduce the stay in the environment is the best solution. The collision backoff mechanism means that at the nth time step, when a robot collides or goes out of bounds on the field, all robots return to the state before m time steps, that is, the global state returns to the state of the nmth time step, and then the path is replanned based on the state at the time step nm. This can effectively avoid the problem of the algorithm falling into a local optimum. In addition, the first and second stages of course learning can be successful without using this mechanism because the environment is relatively simple, so this mechanism is only used in the third stage and subsequent training. Using the collision backoff mechanism with a certain path search foundation can more effectively play the role of the mechanism and improve the success rate of the algorithm.

[0078] The complete operating mechanism of the environment, such as Figure 4 Shown

[0079] Step S3, training the network model, designing a network structure to adapt to the changing number of robots in the environment, and using the multi-agent reinforcement learning algorithm Qatten algorithm based on value decomposition to train the network model.

[0080] At different stages of course learning, the number of robots may change, so a network structure needs to be designed to meet the different numbers of robots.

[0081] First, in the first stage of course learning, there is only one robot in the environment, and the neural network only needs to input observation messages related to itself. i , so design a simple series structure, such as Figure 5 First, the robot and its own observation information o i Input a layer of fully connected network f, and get the output f(o i ) and the hidden state h at the previous moment t-1 , and pass them into a layer of GRU recurrent neural network g, and pass the output into the last layer of fully connected network fx, and finally output the Q value Q of each action i In the first stage of course learning, the network can be expressed as:

[0082] Q i (o i )=fx(g(f(o i ),h t-1 ))

[0083] In the second and subsequent stages of course learning, the number of robots in the environment is greater than 1. At this time, the neural network needs to input not only its own observation information, but also i , and also input observation information about other robots j , the network structure is as follows Figure 6 At this time, the observation information o related to itself is input i The fully connected network f, the GRU recurrent neural network layer g, and the output layer fx of the final action Q value all directly overload the parameters of the first stage network structure. An additional super network structure is added to process other observation information related to other robots. The structure of the super network is to define two two-layer fully connected networks fw and fb, and the two networks respectively input observation information related to themselves o i , we get W = fw(o i ), b=fb(o i ), then use W as the weight and b as the bias to perform the following calculation:

[0084] h(o j )=W·o j +b

[0085] Concatenate f(oi) and h(oj), pass them into a new fully connected network fh, then enter the GRU recurrent neural network layer g, and finally pass them into the fully connected layer fx to output the action Q value Q of each robot iThe network expression is:

[0086]

[0087] After the second stage of the course, each stage reloads the network parameters of the previous stage. At the same time, in order to ensure that the parameter quantity is not affected by the change in the number of robots, all robot network parameters are shared.

[0088] Use the ε-greedy method to select actions. That is, set a continuously decaying parameter ε. Each time an action is selected, a random number r is generated and compared with ε. When r < ε, a random action is selected from the action space. Otherwise, the greedy method is used to select the action with the largest Q value.

[0089]

[0090] The robot runs in the environment according to the actions selected by the neural network and outputs a set of data, including the global state s, local observation o, action a, reward R, and whether it is completed. These inputs are stored in the experience replay pool for subsequent training of the network model.

[0091] When the amount of data in the experience replay pool reaches a minibatch, the network model training begins.

[0092] The neural network is trained using the multi-agent reinforcement learning algorithm Qatten based on value decomposition. The algorithm is based on a centralized training and decentralized execution framework. Each agent calculates the action Q value Q based on its own local observations. i The centralized trainer then calculates the global Q value Q based on the action Q value of each agent tot The Qatten algorithm introduces a multi-head attention mechanism to analyze the contribution of each robot's state to the global state, calculates the weight of each attention head, and sums them up to get Q tot , the specific calculation method is:

[0093]

[0094]

[0095] Among them, w h and c(s) are obtained by inputting the global state s into a two-layer fully connected neural network. H is the number of attention heads and N is the number of robots.

[0096] Finally, the DQN algorithm is used to calculate the loss function and train the network parameters:

[0097]

[0098] Among them, y tot=R+γmax u′ Q tot (τ′,u′,s′;θ - ).

[0099] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A multi-robot path planning method based on curriculum deep reinforcement learning, characterized in that: The method comprises the following steps: Step S1: Introduce the idea of ​​course learning and design tasks for each stage of course learning; Step S2: Build a reinforcement learning training environment, define the robot's observation space, action space and reward function for the robot's path planning problem in continuous space, and prepare for subsequent reinforcement learning; Step S3, training the network model, first designing a network structure to adapt to the changing number of robots in the environment, and then using the Qatten algorithm, a multi-agent reinforcement learning algorithm based on value decomposition, to train the network model.

2. A multi-robot path planning method based on curriculum deep reinforcement learning as claimed in claim 1, characterized in that: In step S1, the specific tasks of each course stage are: S11, the first stage, single robot path planning, the goal is to enable the robot to have basic path search and obstacle avoidance capabilities; S12, the second stage, multiple robots plan paths in a simple environment. The number of robots can be less than the number of robots required by the target task, so as to achieve the mutual cooperation ability of multiple robots in a simple environment and execute waiting actions at the right time; S13, the third stage, keeping the number of robots the same as the previous stage, continuously increasing the size of the environment and the number of obstacles to achieve the transition of the training environment from simple to complex; S14, the fourth stage, completes the path planning of the target task.

3. A multi-robot path planning method based on curriculum deep reinforcement learning as claimed in claim 2, characterized in that: In step S1, a course stage is added between the third stage and the fourth stage for different target tasks to ensure a smooth transition of the environment.

4. A multi-robot path planning method based on curriculum deep reinforcement learning as described in any one of claims 1 to 3, characterized in that: In step S2, the observation space of the robot is divided into two parts, namely, observation information related to itself and observation information related to other robots, so as to adapt to the change in the number of robots in different course learning stages; The robot's action space is a discrete action space, which is divided into two categories: moving action and waiting action. The robot's path trajectory conforms to the RS curve, and reversing is not allowed; Since the rewards of the environment itself are sparse, an artificial potential field method is used to construct a heuristic as the reward function. At the same time, two additional rewards are introduced. One is the fast search reward, which is used to encourage the machine to reach the end point quickly and reduce the stay in the environment. The other is the waiting mechanism reward, which is used to ensure that the agent performs the waiting action at the right time.

5. A multi-robot path planning method based on curriculum deep reinforcement learning as claimed in claim 4, characterized in that: In step S2, the following definition is made on how to judge whether a round is over in the training phase: a round is considered over when all robots on the field have reached the finish line or the upper limit of the number of steps per round is reached; a robot is considered to have reached the finish line when it is within a certain range from the finish line, and a robot that has reached the finish line will stop where it is and will not disappear; when a robot collides with an obstacle or collides with other robots, or walks outside the boundary, the environment will activate a collision backoff mechanism, and all robots on the field will back off to the state of the previous time step and replan their paths.

6. A multi-robot path planning method based on curriculum deep reinforcement learning as claimed in claim 2, characterized in that: In step S2, the global information s of the environment is required for centralized training. The global information consists of the local observation information of all robots and is represented by a one-dimensional vector s=[o 1 ,o 2 ,…,o n ], where o n Represents the observation information of the nth robot For the path planning problem in continuous space, consider the path that conforms to the robot's kinematics; discretize the continuous action space, use the RS curve to plan the path trajectory that meets the requirements, and divide the robot's actions into two categories: moving and waiting. Among them, the moving action refers to the robot moving forward, left or right one step. It is stipulated that the robot is prohibited from reversing, which can promote the robot to move forward to the end point. The current position information of the robot (x, y, θ) is known, and the next position information is calculated using the RS curve. The calculation formula is as follows: Where r represents the turning radius of the robot, δ represents the angle of rotation of the robot, s represents the step length of the robot forward, and s = r·δ; Six moving actions are set, that is, two turning radii are set for each movement direction, so that the robot has a freer movement space; In a single-robot path planning environment, the robot needs to reach the end point to be considered successful and to obtain a positive reward value, while in a multi-robot path planning environment, all robots need to reach the end point to obtain a positive reward value; the rewards in the environment are sparse, and since the state space is continuous, it is almost impossible for the robot to reach the end point when training in such an environment, so it is necessary to evaluate the process of the robot's intermediate movement and densify the sparse rewards to speed up training; first, a set of heuristics is designed using the artificial potential field method as a reward function to calculate the basic reward of each robot at the current position; the basic idea of ​​the artificial potential field is that obstacles and other robots in the environment provide repulsion to the current robot, so the closer the distance between the two, the greater the repulsion and the smaller the heuristic value; on the contrary, the end point provides gravity to the current robot, and at this time, the closer the distance between the two, the greater the gravity and the greater the heuristic value, and the calculation formula is as follows: r=a·h1+b·h2+c·h3 Where d1 represents the distance d from the robot’s current position to the nearest obstacle. o and the distance d to the nearest boundary e , the minimum value of the two, that is, d1 = min(d o ,d e );d2 represents the distance d from the current position of the robot to the nearest other robot a , that is, d2=d a When the training is in the first stage of the course, the robot has no observation information related to other robots, and h2 = 0; d3 represents the distance d from the current position of the robot to the end point. g , that is, d3=d g , λ represents the range of attraction or repulsion in the potential field; The above heuristic reward function cannot take into account all situations of the robot during execution. Therefore, additional rewards are added on the basis of the heuristic reward function to promote training. In order to make the robot reach the end point as quickly as possible and reduce the stay in the environment, the search reward is increased to encourage the robot to move towards the end point. The formula is as follows: Among them, d g Indicates the distance from the robot's current position to the end point, d g 'Indicates the distance from the robot's last position to the end point; When the training enters the second stage, the robot needs to cooperate with other robots, that is, to perform waiting actions at the right time to avoid other robots. Therefore, the waiting reward is increased to train the robot to use the waiting action. The formula is: The setting of the waiting condition is the key to whether the robot can correctly learn to use the waiting action. The definition of the waiting condition is as follows: (1) The distance d from the current position of the robot to the nearest other robot a is smaller than x, x should be smaller than the robot’s field of view radius; (2) the angle θ of the nearest other robot relative to the robot’s direction a ∈(-π / 4,π / 4); (3) The distance d from the current robot to the end point g Less than the distance g from the nearest other robot to its destination a , that is, d g <g a ,When the above three conditions are met at the same time, it is considered that the current state of the robot meets the waiting condition; In addition, when the robot's current position collides with an obstacle or other robots in the environment or moves outside the boundary, the robot's reward value is directly set to a negative number with a large absolute value; when the robot reaches the end point, the robot's reward value is the robot's remaining steps. The robot's initial remaining steps are the maximum step limit of the environment. The robot's remaining steps decrease by one for each step it takes in the environment. A collision will further reduce the remaining steps, and a minimum positive reward value r is specified. min , so that the reward value obtained by the robot when it reaches the end point is greater than or equal to r min ; In addition, since the time when the robots reach the finish line is not necessarily synchronized, each robot can only receive the reward for reaching the finish line once per round. While waiting for other robots to reach the finish line, the reward value is all 0; At this point, the complete reward a robot gets in the current state is calculated, and the reward values ​​of all robots are added together to get the global reward R in the current state. Where N represents the number of robots; The collision-backoff mechanism is used to solve the problem of the robot's state after its current position collides with obstacles or other robots in the environment or moves out of the boundary. With the collision-backoff mechanism, at the nth time step, when a robot collides or goes out of bounds, all robots return to the state before m time steps, that is, the global state returns to the state at the nmth time step, and then the path is replanned based on the state at time step nm. The first and second stages of course learning can be successful without using this mechanism because the environment is relatively simple, so this mechanism is only used in the third stage and subsequent training.

7. A multi-robot path planning method based on curriculum deep reinforcement learning as described in claim 2 or 3, characterized in that: In step S3, network models with different structures are used in different stages: in the first stage of the course, there is only one robot in the environment, so a network structure consisting of a layer of fully connected neural network, a layer of GRU recurrent neural network, and a layer of fully connected neural network in series is designed. The network only inputs observation information related to each robot and outputs the action Q value; from the second stage of the course to all subsequent stages, the number of robots in the environment is greater than 1. At this time, the network structure retains the network structure of the first stage and adds a super network structure to process observation information related to other robots; in addition, all robots share parameters to facilitate training in environments with different numbers of robots.

8. A multi-robot path planning method based on curriculum deep reinforcement learning as described in claim 2 or 3, characterized in that: In step S3, the Qatten algorithm is used to implement the training of the network model. The algorithm is a multi-agent reinforcement learning algorithm based on value decomposition. It uses a multi-head attention mechanism to model the contribution of each agent to the whole, and calculates the overall Q value Qtot through a trainable super network. The loss function calculation and network parameter update method are the same as the DQN algorithm; therefore, when in the first stage of the course, there is only one robot in the environment, and the algorithm degenerates into the DQN algorithm. After the second stage of the course, the number of robots in the environment is greater than 1, and the algorithm is the Qatten algorithm.

9. A multi-robot path planning method based on curriculum deep reinforcement learning as claimed in claim 8, characterized in that: In S3, first, in the first stage of course learning, there is only one robot in the environment, and the neural network only needs to input observation messages related to itself. i Therefore, a simple serial structure is designed. First, the robot and its own observation information o i Input a layer of fully connected network f, and get the output f(o i ) and the hidden state h at the previous moment t-1 , and pass them into a layer of GRU recurrent neural network g, and pass the output into the last layer of fully connected network fx, and finally output the Q value Q of each action i ,In the first stage of course learning, the network is represented as: Q i (o i )=fx(g(f(o i ),h t-1 )) In the second and subsequent stages of course learning, the number of robots in the environment is greater than 1. At this time, the neural network needs to input not only its own observation information, but also i , and also input observation information about other robots j At this time, the observation information o related to itself is passed in i The fully connected network f, the GRU recurrent neural network layer g, and the output layer fx of the final action Q value directly overload the parameters of the first stage network structure, and an additional super network structure is added to process other observation information related to other robots. The structure of the super network is to define two two-layer fully connected networks fw and fb, and the two networks respectively input observation information related to themselves o i , we get W = fw(o i ), b=fb(o i ), then use W as the weight and b as the bias to perform the following calculation: h(o j )=W·o j +b Concatenate f(oi) and h(oj), pass them into a new fully connected network fh, then enter the GRU recurrent neural network layer g, and finally pass them into the fully connected layer fx to output the action Q value Q of each robot i , the network expression is: Q i (o i ,o j )=fx(g(fh(cat(f(o i ),h(o j ))),h t-1 )) After the second stage of the course, each stage reloads the network parameters of the previous stage. At the same time, in order to ensure that the parameter quantity is not affected by the change in the number of robots, all robot network parameters are shared; Use the ε-greedy method to select actions, that is, set a continuously decaying parameter ε. Each time an action is selected, a random number r is generated and compared with ε. When r < ε, a random action is selected from the action space. Otherwise, the greedy method is used to select the action with the largest Q value. The robot runs in the environment according to the actions selected by the neural network and outputs a set of data, including the global state s, local observation o, action a, reward R, and whether it is completed. These inputs are stored in the experience replay pool for subsequent training of the network model; When the amount of data in the experience replay pool reaches a minibatch, start training the network model; The neural network is trained using the multi-agent reinforcement learning algorithm Qatten based on value decomposition. The algorithm is based on a centralized training and decentralized execution framework. Each agent calculates the action Q value Q based on its own local observations. i The centralized trainer then calculates the global Q value Q based on the action Q value of each agent tot The Qatten algorithm introduces a multi-head attention mechanism to analyze the contribution of each robot's state to the global state, calculates the weight of each attention head, and sums them up to get Q tot , calculated as: Among them, w h and c(s) are obtained by inputting the global state s into a two-layer fully connected neural network, H is the number of attention heads, and N is the number of robots; Finally, the DQN algorithm is used to calculate the loss function and train the network parameters: Among them, y tot =R+γmax u′ Q tot (τ′,u′,s′;θ - )。

Images