A multi-robot path planning method based on curriculum deep reinforcement learning

Abstract

A kind of multi-robot path planning method based on course depth reinforcement learning, utilize the thought of course learning, design each stage of course learning, by changing the number of robots in each stage, and map size and the number of obstacles, realize the task from easy to difficult;Build a reinforcement learning environment, define the observation space of robot in the environment, action space and the reward function of environment, formulate the operation rules of robot in the environment;Design a network structure to adapt to the changing number of robots in the environment, use Qatten algorithm to train network model, the present application can solve the problem of multi-robot path planning in continuous space, its advantage lies in that it can realize path planning faster and better than traditional algorithm, improve the efficiency of path planning.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multi-robot path planning, specifically to a multi-robot path planning method based on curriculum-based deep reinforcement learning. Background Technology

[0002] Multi-agent path planning (MAPF) is an important problem in multi-robot systems. The problem is described as follows: Given n agents on a grid, their starting and ending positions, as well as the locations of obstacles, the task is to plan a path from the starting point to the ending point for each agent in discrete time steps, ensuring no collisions between them. Each agent can move to an adjacent cell or remain in place within a time step. MAPF is commonly used in warehousing and logistics, autonomous driving, and video games.

[0003] Traditional multi-agent path planning algorithms are typically based on grid map modeling. Among them, the Conflict-Based Search (CBS) algorithm is one of the most classic and mature algorithms in traditional multi-agent path planning. This algorithm consists of two layers: the upper layer uses a binary constraint tree to handle conflicts between robot paths and selects a conflict-free path; the lower layer independently plans a path that satisfies the constraints for each robot. Currently, the CBS algorithm and its various improved algorithms have improved the accuracy of the search, but they ignore the continuity of the real environment and the actual movement patterns of the robots. As the number of robots increases, the computational cost of traditional algorithms becomes very large, and the efficiency of path search decreases.

[0004] Reinforcement learning is a hot research topic in machine learning. It's a method that learns optimal policies by exploring the environment and learning from its feedback. With the development of artificial neural networks, deep reinforcement learning, which combines reinforcement learning and deep learning, can solve decision-making problems in more complex environments. It utilizes deep neural networks to extract feature information from the environment, and then uses reinforcement learning algorithms to achieve direct control from input to output through end-to-end learning. Based on the reward information obtained from continuous exploration in the environment, the policy is continuously updated and optimized, ultimately obtaining the optimal policy that maximizes cumulative rewards. In recent years, deep reinforcement learning methods have begun to be used to solve multi-agent path planning problems. However, due to the high complexity of agent states in continuous space and the difficulty of model training, most current deep reinforcement learning-based multi-agent path planning algorithms still model the scene as a grid map, lacking the ability to search in continuous space. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a multi-robot path planning method based on curriculum deep reinforcement learning, which can realize multi-robot path planning in free space and improve path search efficiency.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A multi-robot path planning method based on curriculum-based deep reinforcement learning includes the following steps:

[0008] Step S1: Introduce the concept of course learning and design tasks for each stage of course learning;

[0009] Step S2: Build a reinforcement learning training environment. For the path planning problem of the robot in continuous space, define the robot's observation space, action space and reward function to prepare for subsequent reinforcement learning.

[0010] Step S3: Train the network model. First, design a network structure to adapt to the changing number of robots in the environment. Then, use the Qatten algorithm, a multi-agent reinforcement learning algorithm based on value decomposition, to train the network model.

[0011] Furthermore, in step S1, the specific tasks for each course stage are as follows:

[0012] S11, First stage: Single robot path planning, the goal is to enable the robot to have basic path search and obstacle avoidance capabilities;

[0013] S12. Second stage: Path planning by multiple robots in a simple environment. The number of robots can be less than the number required by the target task. This enables multiple robots to cooperate with each other in a simple environment and to perform waiting actions at the right time.

[0014] S13, the third stage, keeping the number of robots the same as the previous stage, continuously increases the size of the environment and the number of obstacles to achieve the transition of the training environment from simple to complex.

[0015] S14, the fourth stage, path planning to complete the target task.

[0016] Preferably, in step S1, a course phase is added between the third and fourth phases for different target tasks to ensure a smooth transition of the environment.

[0017] Furthermore, in step S2, the robot's observation space is divided into two parts: observation information related to itself and observation information related to other robots, in order to adapt to the change in the number of robots at different learning stages.

[0018] The robot's motion space is a discrete motion space, divided into two categories: moving motion and waiting motion. The robot's path trajectory conforms to the RS curve and reversing is not allowed.

[0019] Since the rewards from the environment itself are sparse, an artificial potential field method is used to construct a heuristic as the reward function. At the same time, two additional rewards are introduced: one is a fast search reward, which is used to encourage the machine to reach the destination quickly and reduce the time spent in the environment; the other is a waiting mechanism reward, which is used to ensure that the agent performs the waiting action at the right time.

[0020] Furthermore, in step S2, the determination of whether a game has ended during the training phase is defined as follows: a game is considered to end when all robots on the field reach the finish line or reach the maximum number of moves per game; a robot is considered to have reached the finish line when it is within a certain range of the finish line, and the robot that has reached the finish line will remain in place and will not disappear. When a robot collides with an obstacle or another robot, or goes outside the boundary, the environment will activate a collision rollback mechanism, and all robots on the field will roll back to the state of the previous time step and replan their paths.

[0021] Furthermore, in step S3, different network models with different structures are used in different stages: In the first stage of the course, there is only one robot in the environment, so a network structure consisting of a fully connected neural network layer, a GRU recurrent neural network layer, and a fully connected neural network layer connected in series is designed. This network only takes in the observation information related to each robot itself and outputs the action Q value; In the second stage of the course and all subsequent stages, the number of robots in the environment is greater than 1. At this time, the network structure retains the network structure of the first stage while adding a super network structure to process the observation information related to other robots; In addition, all robots share parameters to facilitate training in environments with different numbers of robots.

[0022] In step S3, the Qatten algorithm is used to train the network model. This algorithm is a multi-agent reinforcement learning algorithm based on value decomposition. It uses a multi-head attention mechanism to model the contribution of each agent to the whole, and calculates the overall Q value Qtot through a trainable supernetwork. The loss function calculation and network parameter update methods are the same as those of the DQN algorithm. Therefore, when in the first stage of the course, there is only one robot in the environment, and the algorithm degenerates into the DQN algorithm. When in the second stage of the course, the number of robots in the environment is greater than 1, and the algorithm becomes the Qatten algorithm.

[0023] The beneficial effects of this invention are as follows: by designing learning stages from easy to difficult, training can be made more stable, the scalability of the model can be increased, and the efficiency of model training can be improved. Especially when the target task involves a large number of robots, learning stages can effectively solve the problem of training difficulties in complex environments. The multi-agent reinforcement learning algorithm based on value function decomposition performs well in solving the problem of fully cooperative scenarios in discrete action space environments. Compared with the fully distributed learning Actor-Critic framework, this algorithm can better achieve cooperation between robots. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall framework of the present invention.

[0025] Figure 2 This is a reference flowchart for each stage of course learning in this invention.

[0026] Figure 3 This is a schematic diagram of the observation space of the robot in this invention.

[0027] Figure 4 This is a flowchart of the environmental operation mechanism in this invention.

[0028] Figure 5 This is a neural network structure diagram for the first stage of course learning in this invention.

[0029] Figure 6 This is a diagram of the neural network structure for the learning stages other than the first stage in this invention. Detailed Implementation

[0030] The present invention will now be further described with reference to the accompanying drawings.

[0031] Reference Figures 1-6 A multi-robot path planning method based on curriculum-based deep reinforcement learning is proposed. By introducing the concept of curriculum learning, the algorithm's training performance in complex environments is improved. The observation space and action space of the robot in the environment are defined to enable the algorithm to perform path planning in continuous space. A reward function is designed to improve the efficiency of algorithm training. A network structure is designed to adapt to different stages of the curriculum.

[0032] like Figure 1 As shown, this invention proposes a multi-robot path planning method based on curriculum-based deep reinforcement learning, comprising the following steps:

[0033] Step S1: Introduce the concept of course learning and design course learning stages from easy to difficult. The process is as follows:

[0034] Figure 2The course demonstrates a reference flow for each stage. Each stage corresponds to a sub-task, and the goal is achieved through a series of sub-tasks that progress from simple to complex.

[0035] S11. Implement single-robot path planning. In multi-robot path planning, each robot needs to complete two tasks: first, to avoid obstacles and find its destination; and second, to avoid other robots to prevent conflicts. The goal of this stage is to complete the first task, realizing the robot's most basic path search and obstacle avoidance capabilities. Building on this capability, implementing the second task can increase the frequency of encounters between robots to some extent, thus making it more conducive to learning the second task.

[0036] S12. Implement path planning for multiple robots in a simple environment. A simple environment generally refers to an environment where the number of robots is less than the required number for the task, the size of the environment map is smaller than the target task map size, and the number of obstacles is also less than the target task obstacle number. The goal of this stage is to achieve cooperation between robots, mainly by teaching them when to wait to avoid other vehicles. The waiting action needs to be executed under relatively strict conditions. First, a robot can only wait when it encounters another robot; it cannot wait when it encounters an obstacle. Second, only one of the two robots that meet can wait; the other cannot wait, otherwise a deadlock will occur. On the other hand, in a sparse scene, it is difficult for robots to discover the correct waiting method. An environment with a small number of robots and a small map ensures both environmental simplicity and increases the frequency of robot encounters, thus promoting the discovery of the correct waiting method.

[0037] S13. Maintain the same number of robots as in the previous stage, and continuously increase the map size and the number of obstacles in the environment to achieve a transition from easy to difficult. The goal of this stage is to enable the robots to initially possess the ability to plan paths in complex environments. Large differences in environment between stages will increase the difficulty of training. Therefore, this stage achieves a smooth transition in environment by gradually increasing the map size and the number of obstacles during training. Specifically, after training a certain number of episodes, the map size and the number of obstacles are increased by 'a' during environment initialization, achieving a gradual change in environment from simple to complex. It is important to note that the initial environment of this stage should be the same as the environment of the previous stage. The environment at the end of this stage can be slightly more complex than the target task environment, i.e., the map size and the number of obstacles are slightly larger than the number specified in the target task. The difference in the number of robots in the current stage compared to the number of robots in the target task can be compensated for by the increase in the number of obstacles. This helps to narrow the gap between the current environment and the target environment, which is beneficial for training the target task.

[0038] S14. Path planning for the target task. After completing the first three stages of training, the robot has initially acquired the ability to plan paths in multi-robot scenarios. The purpose of this stage is to increase the number of robots in the environment to the number required by the target environment, and ultimately enable the robot to perform path planning in the target environment.

[0039] When there are a large number of robots to be planned, an additional transition phase can be added between steps S13 and S14 to allow the number of robots in the environment to increase smoothly, thus ensuring the stability of training.

[0040] Step S2: Build a reinforcement learning environment, define the observation space, action space and reward function of the environment, and determine the operating rules of the robot in the environment.

[0041] The multi-robot path planning method based on curriculum-based deep reinforcement learning proposed in this invention follows the partially observable Markov property and employs a training framework of centralized training and distributed execution. Therefore, each robot can only observe local environmental information.

[0042] This invention divides the observation space of an intelligent agent into two parts: observation information related to itself and observation information related to other robots. Figure 2 A schematic diagram illustrating the information observed by the robot is shown.

[0043] Observational information related to oneself i include:

[0044] (1) Robot a i Current position to its endpoint g i distance d g ;

[0045] (2) Robot a i The endpoint g i The deflection angle θ relative to the robot's direction of travel g ;

[0046] (3) Robot a i The distance d from the current position to the nearest obstacle o o ;

[0047] (4) with robot a i The nearest obstacle o relative to robot a i The angle θ of the direction of travel o ;

[0048] (5) Robot a i The distance d from the current position to the nearest boundary e ;

[0049] Observations related to oneselfi It can be represented by a one-dimensional vector, o i =[d g ,θ g ,d o ,θ o ,d e ].

[0050] Observations related to other robots include:

[0051] (1) Robot a i The distance d from the current position to the nearest other robot aj a ;

[0052] (2) Distance robot a i Other recent robots a j Relative robot a i The angle θ of the direction of travel a ;

[0053] (3) Distance robot a i Other recent robots a j to its endpoint g j distance g a .

[0054] Observations related to other robots j Represented by a one-dimensional vector, o j =[d a ,θ a ,g a ].

[0055] During intensive training, global environmental information 's' is required. This global information is composed of the local observation information of all robots. It can be represented by a one-dimensional vector as s = [o...]. 1 ,o 2 ,…,o n ], where o n Represents the observation information of the nth robot [o n i ,o n j ].

[0056] This invention addresses the path planning problem in continuous space, requiring consideration of paths conforming to the robot's kinematics. Since a continuous action space leads to difficulties in algorithm convergence and increases training complexity, this invention discretizes the continuous action space and uses RS curves to plan a path trajectory that meets the requirements. Robot actions are divided into two categories: movement and waiting. Movement refers to the robot moving one step forward, left, or right; reversing is prohibited to encourage the robot to move towards the destination. Given the robot's current position information (x, y, θ), the next position information is calculated using RS curves, with the specific calculation formula as follows:

[0057]

[0058] Where r represents the robot's turning radius, δ represents the robot's rotation angle, and s represents the robot's step length, and s = r·δ.

[0059] To make the robot's movements more flexible, this invention sets six movement actions, that is, two turning radii are set for each direction of movement, giving the robot more freedom of movement space, which makes it easier for the robot to search for paths and avoid obstacles or other robots.

[0060] In a single-robot path planning environment, a robot needs to reach the destination to achieve success and receive a positive reward. In a multi-robot path planning environment, all robots need to reach the destination to receive a positive reward. Environmental rewards are sparse, and because the intermediate states are continuous, it's almost impossible for a robot to reach the destination during training in such an environment. Therefore, it's necessary to evaluate the robot's intermediate movements to densify the sparse rewards and accelerate training. This invention first uses an artificial potential field method to design a set of heuristics as a reward function to calculate the basic reward for each robot at its current position. The basic idea of ​​the artificial potential field is that obstacles and other robots in the environment provide repulsive forces to the current robot; therefore, the closer they are, the greater the repulsive force and the smaller the heuristic value. Conversely, the destination provides attractive forces to the current robot; in this case, the closer they are, the greater the attractive force and the larger the heuristic value. The specific calculation formula is as follows:

[0061]

[0062]

[0063]

[0064] r = a·h1 + b·h2 + c·h3

[0065] Where d1 represents the distance d from the robot's current position to the nearest obstacle. oand the distance d to the nearest boundary e The minimum value of the two, i.e., d1 = min(d o ,d e ); d2 represents the distance d from the robot's current position to the nearest other robot. a That is, d2 = d a When training is in the first stage of the course, the robot's observations do not contain any information related to other robots, so h2 = 0; d3 represents the distance d from the robot's current position to the endpoint. g , that is, d3=d g λ represents the range of influence of the attractive or repulsive force in the potential field.

[0066] The heuristic reward function described above cannot take into account all situations that the robot may encounter during execution. Therefore, this invention adds an additional reward on top of the heuristic reward function to promote training.

[0067] To help the robot reach the finish line as quickly as possible and reduce its time spent in the environment, we increase the search reward to encourage the robot to move towards the finish line. The specific formula is as follows:

[0068]

[0069] Where, d g d represents the distance from the robot's current position to the destination. g 'Indicates the distance from the robot's previous position to the destination.

[0070] Once training enters the second phase, the robot needs to cooperate with other robots, that is, to perform waiting actions at the correct time to avoid other robots. Therefore, waiting rewards are increased to train the robot to use waiting actions. The specific formula is as follows:

[0071]

[0072] Among them, the setting of the waiting conditions is the key to whether the robot can correctly learn to use the waiting action. The waiting conditions are defined as follows: (1) The distance d from the robot's current position to the nearest other robot. a (1) Less than x, where x should be less than the robot's field of vision radius; (2) The angle θ between the nearest other robot and the robot's direction. a ∈(-π / 4, π / 4); (3) The distance d from the current robot to the destination. g Less than the distance g from the nearest other robot to its destination a , that is, d g <g a When all three conditions above are met simultaneously, the robot's current state is considered to satisfy the waiting condition.

[0073] Furthermore, when the robot collides with obstacles or other robots in the environment or moves outside the boundary, the robot's reward value is directly set to a negative number with a large absolute value. When the robot reaches the destination, its reward value is the number of steps remaining. The robot's initial number of remaining steps is the maximum number of steps allowed in the environment. The remaining steps decrease by one for each step the robot takes in the environment. Collisions further reduce the remaining steps. A minimum positive reward value r is defined. min This ensures that the reward value obtained by the robot upon reaching the finish line is greater than or equal to r. min Additionally, since the robots may not reach the finish line at the same time, each robot can only receive the reward once per game for reaching the finish line. While waiting for other robots to reach the finish line, the reward value is always 0.

[0074] At this point, the complete reward a robot receives in the current state has been calculated. By adding up the reward values ​​of all robots, the global reward R in the current state is obtained.

[0075]

[0076] Where N represents the number of robots.

[0077] This invention utilizes a collision backoff mechanism to address the robot's state after a collision with obstacles or other robots in the environment, or after it moves outside the boundary. The conventional method considers a collision a failure, resets the environment, and restarts planning. Since rewards in the environment are negative, the algorithm easily gets trapped in local optima in complex environments, assuming that direct collisions or going out of bounds to minimize time spent in the environment are optimal. The collision backoff mechanism, however, means that at time step n, when a robot collides or goes out of bounds, all robots revert to their state from m time steps ago; that is, the global state reverts to the state at time step nm, and then the path is replanned based on the state at time step nm. This effectively avoids the algorithm getting trapped in local optima. Furthermore, since the first and second stages of the course can succeed without this mechanism due to the relatively simple environment, it is only used in the third stage and subsequent training. Using the collision backoff mechanism with a certain level of pathfinding experience can more effectively leverage its power and improve the algorithm's success rate.

[0078] The complete operating mechanism of the environment, such as Figure 4 As shown

[0079] Step S3: Train the network model. Design a network structure to adapt to the changing number of robots in the environment. Train the network model using the Qatten algorithm, a multi-agent reinforcement learning algorithm based on value decomposition.

[0080] The number of robots may change at different stages of the course, so a network structure needs to be designed to accommodate different numbers of robots.

[0081] In the first phase of the course, there is only one robot in the environment. At this time, the neural network only needs to be inputted with observational information related to itself. i Therefore, a simple series structure is designed, such as Figure 5 As shown. First, the observation information related to the robot itself is... i Input a fully connected layer f, and output the obtained f(o) i ) and the hidden state h from the previous time step t-1 The inputs are fed into a single GRU recurrent neural network g, and the outputs are fed into the final fully connected network fx, which ultimately outputs the Q-value Q for each action. i The first stage of the course can be represented online as follows:

[0082] Q i (o i )=fx(g(f(o i ),h t-1 ))

[0083] In the second and subsequent stages of the course, the number of robots in the environment is greater than one. At this point, the neural network needs to input not only its own observational information, but also... i It also requires inputting observation information related to other robots. j Network structure such as Figure 6 As shown. At this time, observation information related to itself is input. i The fully connected network f, the GRU recurrent neural network layer g, and the final action Q-value output layer fx all directly overload the parameters of the first-stage network structure. An additional supernetwork structure is added to handle other observation information related to the robot. The supernetwork structure defines two two-layer fully connected networks fw and fb, with each network receiving its own observation information o. i We get W = fw(o i ), b = fb(o i Then, using W as the weight and b as the bias, the following calculation is performed:

[0084] h(o j ) = W·o j +b

[0085] Concatenate f(oi) and h(oj), feed them into a new fully connected network fh, then into a GRU recurrent neural network layer g, and finally into a fully connected layer fx to output the Q-value of each robot's action Q. iThe network expression is:

[0086]

[0087] After the second phase of the course, each phase reloads the network parameters from the previous phase. Meanwhile, to ensure that the number of parameters is not affected by changes in the number of robots, all robots share the same network parameters.

[0088] The ε-greedy method is used to select actions. This involves setting a continuously decaying parameter ε. Each time an action is selected, a random number r is generated and compared to ε. If r < ε, an action is randomly selected from the action space; otherwise, the action with the highest Q-value is selected using the greedy method.

[0089]

[0090] The robot operates in the environment according to actions selected by the neural network and outputs a set of data, including the global state s, local observation o, action a, reward R, and whether the task was completed. These inputs are stored in an experience replay pool for subsequent training of the network model.

[0091] When the amount of data in the experience replay pool reaches the minibatch size, the network model training begins.

[0092] The neural network was trained using the Qatten algorithm, a multi-agent reinforcement learning algorithm based on value factorization. This algorithm employs a centralized training and distributed execution framework, where each agent computes its action Q-value Q based on its own local observations. i The centralized trainer then calculates the global Q-value Q based on the action Q-value of each agent. tot The Qatten algorithm introduces a multi-head attention mechanism to analyze the contribution of each robot's state to the global state, calculates the weight of each attention head, and sums them to obtain Q. tot The specific calculation method is as follows:

[0093]

[0094]

[0095] Among them, w h c(s) and c(s) are obtained by inputting the global state s into a two-layer fully connected neural network. H is the number of attention heads, and N is the number of robots.

[0096] Finally, the loss function is calculated using the DQN algorithm, and the network parameters are trained.

[0097]

[0098] Among them, y tot=R+γmax u′ Q tot (τ′,u′,s′;θ - ).

[0099] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A multi-robot path planning method based on curriculum-based deep reinforcement learning, characterized in that, The method includes the following steps: Step S1: Introduce the concept of course learning and design tasks for each stage of course learning; Step S2: Build a reinforcement learning training environment. For the path planning problem of the robot in continuous space, define the robot's observation space, action space and reward function to prepare for subsequent reinforcement learning. Step S3: Train the network model. First, design a network structure to adapt to the changing number of robots in the environment. Then, use the Qatten algorithm, a multi-agent reinforcement learning algorithm based on value decomposition, to train the network model. In step S3, the Qatten algorithm is used to train the network model. This algorithm is a multi-agent reinforcement learning algorithm based on value decomposition. It uses a multi-head attention mechanism to model the contribution of each agent to the whole, and calculates the overall Q value Qtot through a trainable supernetwork. The loss function calculation and network parameter update methods are the same as those of the DQN algorithm. Therefore, when in the first stage of the course, there is only one robot in the environment, and the algorithm degenerates into the DQN algorithm. When in the second stage of the course, the number of robots in the environment is greater than 1, and the algorithm becomes the Qatten algorithm. In step S3, during the first stage of the learning process, there is only one robot in the environment. At this time, the neural network only needs to input observational information relevant to itself. i Therefore, a simple serial structure is designed. First, the observation information related to the robot and itself is stored in the serial structure. i Input a fully connected layer f, and output the obtained f(o) i ) and the hidden state h from the previous time step t-1 The inputs are fed into a single GRU recurrent neural network g, and the outputs are fed into the final fully connected network fx, which ultimately outputs the Q-value Q for each action. i The first stage of the course is represented online as follows: Q i (o i )=fx(g(f(o i ),h t-1 )) In the second and subsequent stages of the course, the number of robots in the environment is greater than one. At this point, the neural network needs to input not only its own observational information, but also... i It also requires inputting observation information related to other robots. j At this point, observational information related to itself is transmitted. i The fully connected network f, the GRU recurrent neural network layer g, and the final action Q-value output layer fx all directly overload the parameters of the first-stage network structure. An additional supernetwork structure is added to process other observation information related to other robots. The structure of the supernetwork is as follows: two two-layer fully connected networks fw and fb are defined, and the two networks are respectively fed with observation information o related to themselves. i We get W = fw(o i ), b = fb(o i Then, using W as the weight and b as the bias, the following calculation is performed: h(o j )=W·o j +b Concatenate f(oi) and h(oj), feed them into a new fully connected network fh, then into a GRU recurrent neural network layer g, and finally into a fully connected layer fx to output the Q-value of each robot's action Q. i The network expression is: Q i (o i ,o j )=fx(g(fh(cat(f(o i ),h(o j ))),h t-1 )) After the second stage of the course, each stage reloads the network parameters of the previous stage. At the same time, in order to ensure that the number of parameters is not affected by changes in the number of robots, the network parameters of all robots are shared. The ε-greedy method is used to select actions. Specifically, a continuously decaying parameter ε is set. Each time an action is selected, a random number d is generated and compared with ε. If d < ε, an action is randomly selected from the action space. Otherwise, the action with the largest Q value is selected using the greedy method. The robot operates in the environment according to the actions selected by the neural network and outputs a set of data, including global state s, local observation o, action a, global reward R, and whether it has been completed. These inputs are stored in the experience replay pool for subsequent training of the network model. When the amount of data in the experience replay pool reaches the minibatch size, training of the network model begins. The neural network is trained using the Qatten algorithm, a multi-agent reinforcement learning algorithm based on value factorization. This algorithm is based on a framework of centralized training and distributed execution, where each agent calculates its action Q-value Q based on its own local observations. i The centralized trainer then calculates the global Q-value Q based on the action Q-value of each agent. tot The Qatten algorithm introduces a multi-head attention mechanism to analyze the contribution of each robot's state to the global state, calculates the weight of each attention head, and sums them to obtain Q. tot The calculation method is as follows: Among them, w h c(s) and c(s) are obtained by inputting the global state s into a two-layer fully connected neural network, where H is the number of heads for attention and N is the number of robots. Finally, the loss function is calculated using the DQN algorithm, and the network parameters are trained. y tot =R+γmax a′ Q tot (o′,a′,s′;θ - )。 2. The multi-robot path planning method based on curriculum deep reinforcement learning as described in claim 1, characterized in that, In step S1, the specific tasks for each course stage are as follows: S11, First stage: Single robot path planning, the goal is to enable the robot to have basic path search and obstacle avoidance capabilities; S12. Second stage: Multiple robots plan paths in a simple environment. The number of robots is less than the number required by the target task. This enables multiple robots to cooperate with each other in a simple environment and to perform waiting actions at the right time. S13, the third stage, keeping the number of robots the same as the previous stage, continuously increases the size of the environment and the number of obstacles to achieve the transition of the training environment from simple to complex. S14, the fourth stage, path planning to complete the target task.

3. The multi-robot path planning method based on curriculum-based deep reinforcement learning as described in claim 2, characterized in that, In step S1, a course phase is added between the third and fourth phases to ensure a smooth transition of the environment, depending on the different target tasks.

4. A multi-robot path planning method based on curriculum-based deep reinforcement learning as described in any one of claims 1 to 3, characterized in that, In step S2, the robot's observation space is divided into two parts: observation information related to itself and observation information related to other robots, in order to adapt to the change in the number of robots at different learning stages. The robot's motion space is a discrete motion space, divided into two categories: moving motion and waiting motion. The robot's path trajectory conforms to the RS curve and reversing is not allowed. Since the rewards from the environment itself are sparse, an artificial potential field method is used to construct a heuristic as the reward function. At the same time, two additional rewards are introduced: one is a fast search reward, which is used to encourage the machine to reach the destination quickly and reduce the time spent in the environment; the other is a waiting mechanism reward, which is used to ensure that the agent performs the waiting action at the right time.

5. The multi-robot path planning method based on curriculum deep reinforcement learning as described in claim 4, characterized in that, In step S2, the determination of whether a game has ended during the training phase is defined as follows: a game is considered to end when all robots on the field reach the finish line or reach the maximum number of steps per game; when a robot is a certain distance from the finish line, it is considered to have reached the finish line, and the robot that has reached the finish line will stay in place and will not disappear; when a robot collides with an obstacle or another robot, or goes outside the boundary, the environment will activate the collision rollback mechanism, and all robots on the field will roll back to the state of the previous time step and replan their paths.

6. The multi-robot path planning method based on curriculum-based deep reinforcement learning as described in claim 2, characterized in that, In step S2, the global state s of the environment is needed during centralized training. The global state consists of the local observation information of all robots and is represented by a one-dimensional vector as s = [o 1 ,o 2 ,…,o n ], where o n This represents the observation information of the nth robot. For path planning problems in continuous space, we consider paths that conform to the robot's kinematics. The continuous motion space is discretized, and RS curves are used to plan the required path trajectory. Robot actions are divided into two categories: movement and waiting. Movement refers to the robot moving one step forward, left, or right. Reversing is prohibited, and the robot is encouraged to move towards the destination. Given the robot's current position information (x, y, θ), the next position information is calculated using RS curves. The calculation formula is as follows: Where τ represents the robot's turning radius, δ represents the robot's rotation angle, and l represents the robot's step length, and l = τ·δ; Six movement actions were set, that is, two turning radii were set for each direction of movement, giving the robot more freedom of movement. In a single-robot path planning environment, a robot needs to reach the destination to achieve success and receive a positive reward. However, in a multi-robot path planning environment, all robots need to reach the destination to receive a positive reward. Environmental rewards are sparse, and because the state space is continuous, it's almost impossible for a robot to reach the destination during training in such an environment. Therefore, it's necessary to evaluate the robot's intermediate movements to densify the sparse rewards and accelerate training. First, a set of heuristic reward functions is designed using an artificial potential field method to calculate the basic reward for each robot at its current position. The basic idea of ​​the artificial potential field is that obstacles and other robots in the environment provide repulsive forces to the current robot; therefore, the closer they are, the greater the repulsive force and the smaller the heuristic value. Conversely, the destination provides attractive forces to the current robot; in this case, the closer they are, the greater the attractive force and the larger the heuristic value. The calculation formula is as follows: r i =W a ·h1+W b ·h2+W c ·h3 Where d1 represents the distance d from the robot's current position to the nearest obstacle. o and the distance d to the nearest boundary e The minimum value of the two, i.e., d1 = min(d o ,d e ); d2 represents the distance d from the robot's current position to the nearest other robot. a That is, d2 = d a When training is in the first stage of the course, the robot's observations do not contain any information related to other robots, so h2 = 0; d3 represents the distance d from the robot's current position to the endpoint. g , that is, d3=d g λ represents the range of the attractive or repulsive force in the potential field; The heuristic reward function described above cannot take into account all situations that the robot may encounter during execution. Therefore, additional rewards are added to the heuristic reward function to promote training. To encourage the robot to reach the finish line as quickly as possible and reduce its time spent in the environment, the search reward is increased to incentivize the robot to move towards the finish line, as shown in the following formula: Where, d g d represents the distance from the robot's current position to the destination. g 'Indicates the distance from the robot's previous position to the destination; In the second phase of training, the robot needs to cooperate with other robots, that is, to perform waiting actions at the correct time to avoid other robots. Therefore, waiting rewards are increased to train the robot to use waiting actions, as shown in the formula: Among them, the setting of the waiting condition is the key to whether the robot can correctly learn to use the waiting action. The waiting condition is defined as follows: (1) The distance d from the robot's current position to the nearest other robot. a (1) Less than x, where x should be less than the robot's field of vision radius; (2) The angle θ between the nearest other robot and the robot's direction. a ∈(-π / 4, π / 4); (3) The distance d from the current robot to the destination. g Less than the distance g from the nearest other robot to its destination a , that is, d g <g a When all three conditions above are met, the robot's current state is considered to meet the waiting condition. Furthermore, when the robot collides with obstacles or other robots in the environment or moves outside the boundary, the robot's reward value is directly set to a negative number with a large absolute value. When the robot reaches the destination, its reward value is the number of steps remaining. The robot's initial number of remaining steps is the maximum number of steps allowed in the environment. The remaining steps decrease by one for each step the robot takes in the environment. Collisions further reduce the remaining steps. A minimum positive reward value r is defined. min This ensures that the reward value obtained by the robot upon reaching the finish line is greater than or equal to r. min Additionally, since the robots may not reach the finish line at the same time, each robot can only receive the reward once per game for reaching the finish line. While waiting for other robots to reach the finish line, the reward value is always 0. This calculates the total reward a robot receives in the current state. Adding the rewards of all robots together gives the global reward R for the current state. Where N represents the number of robots; A collision rollback mechanism is used to address the robot's state after it collides with obstacles or other robots in the environment or moves outside the boundary. This mechanism works as follows: at time step n, if a robot collides or goes out of bounds, all robots revert to their state from m time steps ago; that is, the global state reverts to the state at time step nm. Then, the path is replanned based on the state at time step nm. Since the first and second phases of the course can be completed successfully without this mechanism due to the relatively simple environment, it is only used in the third phase and subsequent training.

7. A multi-robot path planning method based on curriculum-based deep reinforcement learning as described in claim 2 or 3, characterized in that, In step S3, different network models are used in different stages: In the first stage of the course, there is only one robot in the environment, so a network structure consisting of a fully connected neural network layer, a GRU recurrent neural network layer, and a fully connected neural network layer connected in series is designed. This network only takes in the observation information related to each robot itself and outputs the action Q value. In the second stage of the course and all subsequent stages, the number of robots in the environment is greater than 1. At this time, the network structure retains the network structure of the first stage and adds a super network structure to process the observation information related to other robots. In addition, all robots share parameters to facilitate training in environments with different numbers of robots.

Images