Global path coordination method and system for multiple mobile robots and storage medium

Abstract

The invention relates to the technical field of mobile robot intelligent control, in particular to a global path coordination method and system for multiple mobile robots and a storage medium, and the method comprises the steps: constructing a simulation environment and a kinematics model of the mobile robots, and designing a path planning model of the multiple mobile robots; designing a reward function based on an obstacle avoidance planning layer and a relative position coordination layer in the double-layer neural network, wherein the reward function is used for training a path planning model of the multiple mobile robots; and performing global path planning of the multiple mobile robots based on the trained path planning model of the multiple mobile robots. According to the method, the problems of dimension explosion and difficulty in convergence of deep reinforcement learning in multi-robot navigation are solved, the requirement of autonomous navigation of the multiple mobile robots in a complex and irregular environment can be effectively met, and the efficiency and accuracy of global path planning of the multiple mobile robots are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile robot intelligent control, and in particular to a global path coordination method, system and storage medium for multiple mobile robots. Background Art

[0002] At present, mobile robots have been widely used in various industries, such as factories, hospitals, homes, hotels, exhibition halls, restaurants, etc., mainly responsible for logistics, handling and distribution tasks. In these application scenarios, multiple mobile robots need to achieve autonomous navigation, have accurate positioning, path planning, and the ability to complete tasks efficiently and safely.

[0003] In multi-robot operation scenarios, a global map of the surrounding environment is usually built in advance, and safe travel from the starting point to the target point is achieved according to the task requirements. However, only relying on the global path planning algorithm suitable for single-robot operation scenarios, although it can find the shortest or most time-saving path, it cannot take multiple robots into account, resulting in poor driving of the mobile robot in some narrow passages or complex environments such as intersections, or even deadlock. When deep reinforcement learning models are applied to multi-robot navigation, if too many multi-robot state parameters are input, problems such as dimensionality explosion and difficulty in convergence will be encountered. Summary of the invention

[0004] To this end, the technical problem to be solved by the present invention is to overcome the problems of dimensionality explosion and difficulty in convergence encountered when applying deep reinforcement learning models in the prior art to multi-robot navigation.

[0005] In order to solve the above technical problems, the present invention provides a global path coordination method for multiple mobile robots, comprising:

[0006] Constructing a simulation environment and a kinematic model of a mobile robot, and designing a path planning model for multiple mobile robots; the path planning model for multiple mobile robots includes a double-layer neural network and a reinforcement learning network; the double-layer neural network includes an obstacle avoidance planning layer and a relative position coordination layer;

[0007] Reward functions are designed based on the obstacle avoidance planning layer and the relative position coordination layer in the two-layer neural network to train the path planning model of multiple mobile robots.

[0008] Based on the trained multi-mobile robot path planning model, global path planning of multiple mobile robots is performed, including:

[0009] Obtain the observation space and action space of each mobile robot; input the observation space of each mobile robot into the two-layer neural network corresponding to the current mobile robot; the obstacle avoidance planning layer takes the observation space as input and outputs the obstacle avoidance eigenvalue; the relative position coordination layer obtains the relative position information of each mobile robot and other robots as input according to the observation space and action space, and outputs the relative coordination eigenvalue; the obstacle avoidance eigenvalue and the relative coordination eigenvalue are spliced ​​to obtain the state space of the mobile robot;

[0010] The state space of each mobile robot is input into the reinforcement learning network corresponding to the current mobile robot, and the action of each mobile robot is output.

[0011] Preferably, the observation space of the mobile robot includes the current position of the robot, the local point that the current robot needs to reach, and the surrounding environment information obtained by the current robot through sensors; the action space includes linear velocity and angular velocity.

[0012] Preferably, the structures of the obstacle avoidance planning layer and the relative position coordination layer both include: three fully connected layers, a long short-term memory layer and an encoding layer connected in sequence.

[0013] Preferably, the reward function designed based on the obstacle avoidance planning layer is:

[0014]

[0015] Among them, R OA represents the reward function of the obstacle avoidance planning layer, V represents the linear velocity of the mobile robot, and W represents the angular velocity of the mobile robot.

[0016] Preferably, the reward function designed based on the relative position coordination layer is:

[0017]

[0018] Among them, R NAV represents the reward function of the relative position coordination layer, K represents the collision coefficient, and d curr Indicates the distance between the current mobile robot and the local point to be reached, d step Indicates the time step of the current robot execution.

[0019] Preferably, the reinforcement learning network is a MATD3 algorithm network; the MATD3 algorithm network includes an experience replay pool for storing all information on the interaction between the mobile robot and the environment as a sample for training the network; the information on the interaction between the mobile robot and the environment stored in the experience replay pool includes: the observation space and action space of the mobile robot, the action, the state space at the next moment and the reward value after executing the action.

[0020] Preferably, the MATD3 algorithm network includes:

[0021] Each mobile robot corresponds to a set of independent policy-evaluation networks, including a policy network, a target policy network, two evaluation networks, and two target evaluation networks;

[0022] The policy network of each mobile robot selects multiple actions based on the current state space of the mobile robot; the GAE algorithm is used to search all actions, and the value of the action is calculated based on the state space of all mobile robots through the evaluation network, and the action with the highest value is output;

[0023] After each mobile robot performs an action and interacts with the environment, it obtains the state space at the next moment and generates a reward value after executing the action.

[0024] Preferably, the parameter updating method of the MATD3 algorithm network includes:

[0025] Select several samples from the experience replay pool to train the MATD3 algorithm network corresponding to each mobile robot;

[0026] The target strategy network of each mobile robot outputs the action at the next moment according to the state space at the next moment, and transmits it to the two target evaluation networks respectively; the target evaluation network calculates the value of the action at the next moment according to the state space of all mobile machines at the next moment; selects the smallest value as the target action value; calculates the loss function according to the target action value, and updates the parameters of the evaluation network;

[0027] The network parameters of the target evaluation network are soft-updated according to the evaluation network with updated parameters.

[0028] The present invention also provides a global path coordination system for multiple mobile robots, comprising:

[0029] A model building module is used to build a simulation environment and a kinematic model of a mobile robot, and to design a path planning model for multiple mobile robots; the path planning model for multiple mobile robots includes a double-layer neural network and a reinforcement learning network; the double-layer neural network includes an obstacle avoidance planning layer and a relative position coordination layer;

[0030] The reward function design module is used to design the reward functions of the obstacle avoidance planning layer and the relative position coordination layer in the two-layer neural network, respectively, for training the path planning model of multiple mobile robots;

[0031] The path planning module is used to perform global path planning of multiple mobile robots based on the trained path planning model of multiple mobile robots, including:

[0032] Obtain the observation space and action space of each mobile robot; input the observation space of each mobile robot into the two-layer neural network corresponding to the current mobile robot; the obstacle avoidance planning layer takes the observation space as input and outputs the obstacle avoidance eigenvalue; the relative position coordination layer obtains the relative position information of each mobile robot and other robots as input according to the observation space and action space, and outputs the relative coordination eigenvalue; the obstacle avoidance eigenvalue and the relative coordination eigenvalue are spliced ​​to obtain the state space of the mobile robot;

[0033] The state space of each mobile robot is input into the reinforcement learning network corresponding to the current mobile robot, and the action of each mobile robot is output.

[0034] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the global path coordination method of multiple mobile robots are implemented.

[0035] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0036] The global path coordination method for multiple mobile robots described in the present invention extracts features from the observation space of the mobile robots by constructing a two-layer neural network, generates the state space of the mobile robots, and reduces the input dimension of the reinforcement learning network; and for the two-layer neural network, independent reward functions are designed for the obstacle avoidance planning layer and the relative position coordination layer, respectively, to assist in the path planning model training of the multiple mobile robots and improve the convergence of the model training. The present invention solves the problems of dimensional explosion and difficulty in convergence in deep reinforcement learning in multi-robot navigation, can effectively adapt to the needs of autonomous navigation of multiple mobile robots in complex and irregular environments, has high security, good orderliness and strong real-time performance, reduces the collision and blockage problems that may occur in the actual working process of multiple mobile robots, and improves the efficiency and accuracy of global path planning of multiple mobile robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0038] Figure 1 is a flow chart of a global path coordination method for multiple mobile robots of the present invention;

[0039] Figure 2 This is a diagram of the double-layer neural network structure of a single robot;

[0040] Figure 3 This is the training structure diagram of the MATD3 algorithm network. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0042] Reference Figure 1 As shown, the present invention provides a global path coordination method for multiple mobile robots, comprising:

[0043] S1. Construct a simulation environment and a kinematic model of a mobile robot, and design a path planning model for multiple mobile robots; the path planning model for multiple mobile robots includes a double-layer neural network and a reinforcement learning network; the double-layer neural network includes an obstacle avoidance planning layer and a relative position coordination layer;

[0044] S2. Design reward functions for the obstacle avoidance planning layer and the relative position coordination layer in the two-layer neural network respectively, which are used to train the path planning model of multiple mobile robots.

[0045] S3. Performing global path planning of multiple mobile robots based on the trained path planning model of multiple mobile robots, including:

[0046] Obtain the observation space of each mobile robot; input the observation space of each mobile robot into the two-layer neural network corresponding to the current mobile robot; the obstacle avoidance planning layer takes the observation space as input and outputs the obstacle avoidance eigenvalue; the relative position coordination layer obtains the relative position information of each mobile robot and other robots according to the observation space as input and outputs the relative coordination eigenvalue; the obstacle avoidance eigenvalue and the relative coordination eigenvalue are spliced ​​to obtain the state space of the mobile robot;

[0047] The state space of each mobile robot is input into the reinforcement learning network corresponding to the current mobile robot, and the action of each mobile robot is output.

[0048] The technical solutions in the embodiments of the present invention are described clearly and completely below.

[0049] When multiple mobile robots navigate in a complex environment, each mobile robot senses the surrounding environment in real time through the 2D laser sensor it carries, and each mobile robot can directly obtain the relative position information with other mobile robots. Each mobile robot needs to know its own position and the local point it needs to reach.

[0050] Construct the kinematic model of the simulation environment and mobile robots, and design the path planning model of multiple mobile robots based on the kinematic model of the mobile robots. Collect point cloud data of the simulation or real environment based on the laser radar equipment, perceive the obstacle avoidance environment of multiple robots in real time, set up the reinforcement learning framework, and define the observation space and action space of each mobile robot.

[0051] The observation space of the mobile robot includes the current position of the robot, the local point information that the current robot needs to reach, and the surrounding environment information obtained by the current robot through sensors. The action space of the mobile robot includes linear velocity and angular velocity.

[0052] In indoor navigation, in order to avoid the problem of model non-convergence caused by the increase in the dimension of the network model input, the surrounding environment information obtained by each mobile robot through the sensor is the radar point cloud data of the current mobile robot from -90° to 90°.

[0053] The path planning model of the multi-mobile robots includes a double-layer neural network and a reinforcement learning network. The double-layer neural network includes an obstacle avoidance planning layer and a relative position coordination layer.

[0054] The obstacle avoidance planning layer and the relative position coordination layer have the same neural network architecture, both of which include three fully connected layers, a long short-term memory layer and an encoding layer connected in sequence, and output corresponding eigenvalues ​​for subsequent splicing.

[0055] The observation space of each mobile robot is input into the two-layer neural network corresponding to the current mobile robot. The obstacle avoidance planning layer takes the observation space as input and outputs the obstacle avoidance eigenvalue. The relative position coordination layer introduces the signal strength method and obtains the relative position information of each mobile robot and other robots as input according to the observation space. That is, with each robot as the center, the distance and direction of all other robots relative to itself are input from near to far, and the relative coordination eigenvalue is output. The obstacle avoidance eigenvalue and the relative coordination eigenvalue are spliced ​​to obtain the state space of the mobile robot. The two-layer neural network structure of a single robot is referenced Figure 2 shown.

[0056] The present invention designs reward functions of an obstacle avoidance planning layer and a relative position coordination layer in a double-layer neural network respectively, which are used for training the double-layer neural network.

[0057] The obstacle avoidance planning layer is divided into the driving process and the collision process. When the mobile robot is driving, the absolute value of the angular velocity is calculated by subtracting the linear velocity to ensure the stability of the robot during driving. When the mobile robot collides, a higher penalty of -100 is given to distinguish. The reward function of the obstacle avoidance planning layer is:

[0058]

[0059] Among them, R OA represents the reward function of the obstacle avoidance planning layer, V represents the linear velocity of the mobile robot, and W represents the angular velocity of the mobile robot.

[0060] The relative position coordination layer is divided into reaching the target point and others. When the mobile robot reaches the target point, a higher reward of 100 is given. In other cases, when more coordination steps are used to solve the coordination between multiple mobile robots, more penalty values ​​will be given. The reward function of the relative position coordination layer is:

[0061]

[0062] Among them, R NAV represents the reward function of the relative position coordination layer, K represents the collision coefficient used to distinguish between arrival and collision, and the value in this embodiment is 0.1; d curr Indicates the distance between the current mobile robot and the local point to be reached, d step Indicates the time step of the current robot execution.

[0063] Therefore, the reward function R of the two-layer neural network is:

[0064] R=R OA +R NAV

[0065] The state space of each mobile robot is input into the reinforcement learning network corresponding to the current mobile robot, and the action of each mobile robot is output.

[0066] The present invention extracts features of observation space and action space through a double-layer neural network, splices them into a set of state spaces that can adapt to multi-agent training of the MATD3 algorithm network, reduces the input dimension, and independently designs corresponding reward functions for these two layers of neural networks to assist model training and improve the convergence of model training. The above-mentioned double-layer neural network design can make the model effectively adapt to the needs of multi-robot autonomous navigation in complex and irregular environments.

[0067] The reinforcement learning network adopted in the present invention is the MATD3 algorithm network.

[0068] After each robot interacts with the environment, all information about the interaction between the mobile robots and the environment is stored in the experience replay pool, including the observation space and action space of the mobile robot, the action, the state space at the next moment, and the reward value after executing the action, as samples for training the reinforcement learning network.

[0069] In this embodiment, the experience replay pool is a list of fixed size. A historical training data storage H is designed to store the experience sets of h buffers as a minibatch data set, which is expressed as <O set ,A set ,R set ,S set >, where O setRepresents the observation space and action space of the mobile robot, including the original data set consisting of the point cloud data, linear velocity, angular velocity and posture of the latest three frames of the 2D laser radar of all mobile robots; S set Represents the state space of all mobile robots at the next moment; A set Indicates that all mobile robots are in S set The action performed in the state space, that is, the action output by the reinforcement learning network; R set Indicates that all mobile robots are performing action A set Get the reward value collection.

[0070] Reference to the training structure diagram of the MATD3 algorithm network Figure 3 As shown. The MATD3 algorithm network is a multi-agent reinforcement learning algorithm based on the Actor-Critic architecture, in which each agent has an independent set of independent strategy-evaluation networks. The MATD3 algorithm network adopts a centralized training and distributed execution structure. During the training process, the evaluation network of each agent can access the states and actions of all agents to maximize the cumulative return of each agent. In the execution stage, the agent selects actions only based on its own observations.

[0071] In this embodiment, the MATD3 algorithm network includes:

[0072] Each mobile robot corresponds to a set of independent policy-evaluation networks, including a policy network, a target policy network, two evaluation networks, and two target evaluation networks;

[0073] The policy network of each mobile robot selects multiple actions based on the current state space of the mobile robot; the GAE algorithm is used to search all actions, and the value of the action is calculated based on the state space of all mobile robots through the evaluation network, and the action with the highest value is output;

[0074] After each mobile robot performs an action and interacts with the environment, it obtains the state space at the next moment and generates a reward value after executing the action.

[0075] The GAE algorithm is used to search all actions, and the formula of the advantage function calculated is:

[0076]

[0077] in, represents the action with the highest value of the ith mobile robot at time t, T i represents the time point of the i-th mobile robot, and its value range is from 0 to T; l represents the l-th moment, and λ and γ both represent proportional coefficients. represents the proportional coefficient of the i-th mobile robot at time t, and denote the state space of the i-th mobile robot at time t+1 and time t respectively, and They represent the value of the i-th mobile robot action at time t and time t+1 respectively.

[0078] T in the above formula i It represents the time span of an experience, which is learned during the training process. l is the time span T i The accumulation in the sequence is to achieve the action at time t that can maximize the final benefit in a sequence process. t is the time t in the operation of the mobile robot, which is equivalent to the real time. The action at time t depends on the optimal action in the previous training process, that is, from T i Select in.

[0079] The advantage function can be used to calculate the action with the greatest value based on the state space of all current mobile robots.

[0080] The parameter updating method of the MATD3 algorithm network includes:

[0081] Select several samples from the experience replay pool to train the MATD3 algorithm network corresponding to each mobile robot;

[0082] The target strategy network of each mobile robot outputs the action at the next moment according to the state space at the next moment, and transmits it to the two target evaluation networks respectively; the target evaluation network calculates the value of the action at the next moment according to the state space of all mobile machines at the next moment; selects the smallest value as the target action value; calculates the loss function according to the target action value, and updates the parameters of the evaluation network;

[0083] Soft update the network parameters of the target evaluation network according to the evaluation network with updated parameters;

[0084] Finally, repeat the above steps until the optimal strategy is learned.

[0085] Based on the MATD3 algorithm network, the present invention designs an experience pool playback mechanism to store historical data, thereby improving training efficiency; finally, through the comprehensive judgment of the number of iterations, average reward value and loss function, it is determined whether the model training is completed and the final result is output.

[0086] In summary, the global path coordination method of multiple mobile robots described in the present invention extracts features from the observation space of the mobile robots by constructing a two-layer neural network, generates the state space of the mobile robots, and reduces the input dimension of the reinforcement learning network; and for the two-layer neural network, independent reward functions are designed for the obstacle avoidance planning layer and the relative position coordination layer, respectively, to assist in the path planning model training of multiple mobile robots and improve the convergence of model training. The present invention solves the problems of dimensional explosion and difficulty in convergence of deep reinforcement learning in multi-robot navigation, can effectively adapt to the needs of autonomous navigation of multiple mobile robots in complex and irregular environments, has high security, good orderliness and strong real-time performance, reduces the collision and blockage problems that may occur in the actual working process of multiple mobile robots, and improves the efficiency and accuracy of global path planning of multiple mobile robots.

[0087] Based on the above-mentioned global path coordination method for multiple mobile robots, this embodiment further provides a global path coordination system for multiple mobile robots, including:

[0088] A model building module is used to build a simulation environment and a kinematic model of a mobile robot, and to design a path planning model for multiple mobile robots; the path planning model for multiple mobile robots includes a double-layer neural network and a reinforcement learning network; the double-layer neural network includes an obstacle avoidance planning layer and a relative position coordination layer;

[0089] The reward function design module is used to design the reward functions of the obstacle avoidance planning layer and the relative position coordination layer in the two-layer neural network, respectively, for training the path planning model of multiple mobile robots;

[0090] The path planning module is used to perform global path planning of multiple mobile robots based on the trained path planning model of multiple mobile robots, including:

[0091] Obtain the observation space and action space of each mobile robot; input the observation space of each mobile robot into the two-layer neural network corresponding to the current mobile robot; the obstacle avoidance planning layer takes the observation space as input and outputs the obstacle avoidance eigenvalue; the relative position coordination layer obtains the relative position information of each mobile robot and other robots as input according to the observation space and action space, and outputs the relative coordination eigenvalue; the obstacle avoidance eigenvalue and the relative coordination eigenvalue are spliced ​​to obtain the state space of the mobile robot;

[0092] The state space of each mobile robot is input into the reinforcement learning network corresponding to the current mobile robot, and the action of each mobile robot is output.

[0093] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the global path coordination method for multiple mobile robots as described above are implemented.

[0094] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0095] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0096] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0098] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A global path coordination method for multiple mobile robots, characterized in that: include: Constructing a simulation environment and a kinematic model of a mobile robot, and designing a path planning model for multiple mobile robots; the path planning model for multiple mobile robots includes a double-layer neural network and a reinforcement learning network; the double-layer neural network includes an obstacle avoidance planning layer and a relative position coordination layer; Reward functions are designed based on the obstacle avoidance planning layer and the relative position coordination layer in the two-layer neural network to train the path planning model of multiple mobile robots. Based on the trained multi-mobile robot path planning model, global path planning of multiple mobile robots is performed, including: Obtain the observation space and action space of each mobile robot; input the observation space of each mobile robot into the two-layer neural network corresponding to the current mobile robot; the obstacle avoidance planning layer takes the observation space as input and outputs the obstacle avoidance eigenvalue; the relative position coordination layer obtains the relative position information of each mobile robot and other robots as input according to the observation space and action space, and outputs the relative coordination eigenvalue; the obstacle avoidance eigenvalue and the relative coordination eigenvalue are spliced ​​to obtain the state space of the mobile robot; The state space of each mobile robot is input into the reinforcement learning network corresponding to the current mobile robot, and the action of each mobile robot is output.

2. A global path coordination method for multiple mobile robots according to claim 1, characterized in that: The observation space of the mobile robot includes the current position of the robot, the local point that the current robot needs to reach, and the surrounding environment information obtained by the current robot through sensors; the action space includes linear velocity and angular velocity.

3. The global path coordination method of multiple mobile robots according to claim 1, characterized in that: The structures of the obstacle avoidance planning layer and the relative position coordination layer both include: three fully connected layers connected in sequence, a long short-term memory layer and an encoding layer.

4. A global path coordination method for multiple mobile robots according to claim 1, characterized in that: The reward function designed based on the obstacle avoidance planning layer is: Among them, R OA represents the reward function of the obstacle avoidance planning layer, V represents the linear velocity of the mobile robot, and W represents the angular velocity of the mobile robot.

5. The global path coordination method of multiple mobile robots according to claim 1, characterized in that: The reward function designed based on the relative position coordination layer is: Among them, R NAV represents the reward function of the relative position coordination layer, K represents the collision coefficient, and d curr Indicates the distance between the current mobile robot and the local point to be reached, d step Indicates the time step of the current robot execution.

6. A global path coordination method for multiple mobile robots according to claim 1, characterized in that: The reinforcement learning network is a MATD3 algorithm network; the MATD3 algorithm network includes an experience replay pool for storing all information about the interaction between the mobile robot and the environment as samples for training the network; The information about the interaction between the mobile robot and the environment stored in the experience replay pool includes: the observation space and action space of the mobile robot, the action, the state space at the next moment, and the reward value after executing the action.

7. A global path coordination method for multiple mobile robots according to claim 6, characterized in that: The MATD3 algorithm network includes: Each mobile robot corresponds to a set of independent policy-evaluation networks, including a policy network, a target policy network, two evaluation networks, and two target evaluation networks; The policy network of each mobile robot selects multiple actions based on the current state space of the mobile robot; the GAE algorithm is used to search all actions, and the value of the action is calculated based on the state space of all mobile robots through the evaluation network, and the action with the highest value is output; After each mobile robot performs an action and interacts with the environment, it obtains the state space at the next moment and generates a reward value after executing the action.

8. A global path coordination method for multiple mobile robots according to claim 7, characterized in that: The parameter updating method of the MATD3 algorithm network includes: Select several samples from the experience replay pool to train the MATD3 algorithm network corresponding to each mobile robot; The target strategy network of each mobile robot outputs the action at the next moment according to the state space at the next moment, and transmits it to the two target evaluation networks respectively; the target evaluation network calculates the value of the action at the next moment according to the state space of all mobile machines at the next moment; selects the smallest value as the target action value; calculates the loss function according to the target action value, and updates the parameters of the evaluation network; The network parameters of the target evaluation network are soft-updated according to the evaluation network with updated parameters.

9. A global path coordination system for multiple mobile robots, characterized in that: include: A model building module is used to build a simulation environment and a kinematic model of a mobile robot, and to design a path planning model for multiple mobile robots; the path planning model for multiple mobile robots includes a double-layer neural network and a reinforcement learning network; the double-layer neural network includes an obstacle avoidance planning layer and a relative position coordination layer; The reward function design module is used to design the reward functions of the obstacle avoidance planning layer and the relative position coordination layer in the two-layer neural network, respectively, for training the path planning model of multiple mobile robots; The path planning module is used to perform global path planning of multiple mobile robots based on the trained path planning model of multiple mobile robots, including: Obtain the observation space and action space of each mobile robot; input the observation space of each mobile robot into the two-layer neural network corresponding to the current mobile robot; the obstacle avoidance planning layer takes the observation space as input and outputs the obstacle avoidance eigenvalue; the relative position coordination layer obtains the relative position information of each mobile robot and other robots as input according to the observation space and action space, and outputs the relative coordination eigenvalue; the obstacle avoidance eigenvalue and the relative coordination eigenvalue are spliced ​​to obtain the state space of the mobile robot; The state space of each mobile robot is input into the reinforcement learning network corresponding to the current mobile robot, and the action of each mobile robot is output.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a global path coordination method for multiple mobile robots as described in any one of claims 1 to 8.

Images