Multi-robot path planning method and electronic device based on deep reinforcement learning
Through deep reinforcement learning methods, multi-robot path planning is optimized, local motion space and action information are processed using neural networks, combined with loss characteristics and reward and punishment parameters, the problem of inaccurate path planning in the existing technology is solved, and the success of multi-robot path planning is improved rate and accuracy.
Patent Information
- Application Number
- CN202210233137.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-09
- Publication Date
- 2024-12-24
- Estimated Expiration
- 2042-03-09
AI Technical Summary
Existing path planning algorithms are difficult to achieve accurate path planning in high-density obstacles and multi-robot environments, especially when robots are transported in warehousing and logistics centers, which are prone to congestion and collision.
The multi-robot path planning method based on deep reinforcement learning is adopted to optimize the robot's local motion space information and action information through neural networks, and combine loss feature parameters and reward and punishment parameters to train the neural network to improve the accuracy and success rate of path planning.
It realizes more accurate path planning in complex environments, improves the success rate and accuracy of multi-robot path planning, avoids collision between robots and obstacles, and optimizes the performance of path planning algorithms.
Smart Images
Figure CN114742270B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a multi-robot path planning method based on deep reinforcement learning, an electronic device and a computer-readable storage medium. Background Art
[0002] Path planning is a hot topic in the field of artificial intelligence. It involves many aspects of life, including mobile robot navigation, aircraft trajectory planning, game automatic navigation, and logistics navigation. Especially in today's popular online shopping, more and more consumers choose to shop online, which also increases the difficulty of logistics transportation in warehouses. When the goods surge, the storage and logistics center needs a large number of robots to transport goods, which will cause unavoidable congestion on the way to the destination. In addition, it is necessary to deal with external unstable factors such as the fall of goods or sudden intrusion of third parties.
[0003] There are many traditional path planning algorithms at home and abroad, such as Breadth First Search (BFS), Dijkstra algorithm or A* algorithm. However, the above algorithms lack communication between agents in various map sizes and high-density obstacles, which leads to messy planning information and inaccurate path planning. Summary of the invention
[0004] The present invention aims to solve at least one of the technical problems existing in the prior art, and provide a multi-robot path planning method, an electronic device and a computer-readable storage medium based on deep reinforcement learning to improve the success rate of multi-robot path planning.
[0005] In a first aspect, the present invention provides a multi-robot path planning method based on deep reinforcement learning, wherein there are multiple robots, including:
[0006] For each robot, a first reference path of the robot is obtained, and a training optimization process is performed based on the first reference path of the robot until a target training condition is met, wherein the training optimization process includes:
[0007] Determining local motion space information of the robot according to a first reference path of the robot;
[0008] Inputting the local motion space information into a first neural network, so that the first neural network performs feature processing on the local motion space information and outputs local state features of the robot;
[0009] Inputting the local state features of the robot into a second neural network, so that the second neural network outputs the action information of the robot according to the local state features;
[0010] Based on the motion information and the first reference path, obtaining a second reference path of the robot;
[0011] Inputting the encoding information of several robots closest to the robot into a third neural network, so that the third neural network outputs loss feature parameters according to the plurality of encoding information, and calculating a loss function based on the loss feature parameters and the determined reward and punishment parameters, wherein the encoding information is obtained by encoding the local state features and the action information;
[0012] Training the second neural network and the third neural network by the loss function;
[0013] The target training condition is that a target arrival ratio parameter is not less than a preset ratio parameter, wherein the target arrival ratio parameter is a ratio of the robot that has arrived at the target point of the second reference path to all the robots;
[0014] The second reference path is determined as the target planning path of the robot.
[0015] The multi-robot path planning method based on deep reinforcement learning provided by the embodiment of the present invention has at least the following beneficial effects: first, a first reference path of the robot is obtained, wherein the first reference path is obtained for each robot using a traditional path planning algorithm, and then the first reference path is optimized to facilitate planning of a more accurate path, the local motion space information of the robot is determined according to the first reference path, and then the local motion space information is input into the first neural network to obtain the local state characteristics of the robot, so that the condition of the robot's environment can be determined according to the local state characteristics, and the robot is prevented from colliding with obstacles during movement, secondly, the local state characteristics of the robot are input into the second neural network to obtain the robot's motion information, which is convenient for guiding the robot's moving path, and finally, the coding information of several robots closest to the robot is input into the third neural network, so that the third neural network outputs a loss feature parameter, and the first neural network, the second neural network and the third neural network are trained through the loss function to achieve the purpose of continuously optimizing the second neural network and the third neural network, thereby improving the accuracy of the path planning of the second neural network and the third neural network, and when the target training condition is that the target arrival ratio parameter is not less than the preset ratio parameter, the second reference path is determined as the target planning path of the robot, thereby realizing the path planning of multiple robots and improving the success rate and accuracy.
[0016] According to some embodiments of the present invention, the local state features are input into the second neural network, so that the middle layer of the second neural network transmits the local state features based on a time series, plans the local state features to obtain the action information, outputs the action information, and determines the action to be performed by the robot based on the action information, thereby achieving control of the robot's moving path.
[0017] According to some embodiments of the present invention, when it is determined that the robot moves according to the first reference path, reward and punishment parameters are calculated based on the action information to facilitate the calculation of subsequent loss characteristic parameters.
[0018] According to some embodiments of the present invention, the reward and punishment parameters include an action reward and punishment value for characterizing changes in action during the movement of the robot, a collision reward and punishment value for characterizing a collision between the robot and an obstacle, an oscillation reward and punishment value for characterizing an oscillating motion of the robot, a deviation reward and punishment value for characterizing the degree of deviation of the robot from the first reference path, and an arrival reward and punishment value for characterizing the arrival of the robot at the end point. The robot obtains feedback from the environment after executing the action during the execution of the action information, which facilitates recording the reward and punishment status during the movement of the robot.
[0019] According to some embodiments of the present invention, the coding information of several robots that are closest to the robot is input into the attention network, so that the attention network calculates the coding information based on a preset selective attention algorithm and outputs a weight matrix, wherein the selective attention algorithm is used to characterize the influence between the robots; the coding information and the weight matrix are input into the coding network for calculation, and the loss characteristic parameters are output, so as to facilitate algorithm optimization based on the loss characteristic parameters, thereby determining the final target path.
[0020] According to some embodiments of the present invention, the encoded information is calculated according to the selective attention algorithm to obtain a query matrix, a key-value matrix and a value item matrix; the weight matrix is calculated based on the query matrix, the key-value matrix and the value item matrix to facilitate the calculation of subsequent loss feature parameters and to realize the selection of several adjacent robots.
[0021] According to some embodiments of the present invention, the coding information of several robots that are closest to each robot is determined; the coding information is input into the third neural network so that the third neural network obtains a set of loss features, and enables all robots to select several surrounding robots, thereby calculating a set of loss values based on the coding information of each robot.
[0022] According to some embodiments of the present invention, a first calculation is performed on the loss characteristic parameters and the reward and punishment parameters to obtain the third loss function for optimizing the third neural network; a second calculation is performed based on the reward and punishment parameters to obtain the second loss function for optimizing the first neural network and the second neural network, thereby achieving optimization of the first neural network, the second neural network and the third neural network.
[0023] In a second aspect, the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a multi-robot path planning method based on deep reinforcement learning as described in the first aspect.
[0024] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores a program executable by a processor, and when the program executable by the processor is executed by the processor, it is used to implement the multi-robot path planning method based on deep reinforcement learning as described in the first aspect above.
[0025] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.
[0027] Figure 1 is a flow chart of a multi-robot path planning method based on deep reinforcement learning provided by an embodiment of the present invention;
[0028] Figure 2 is a flowchart of a multi-robot path planning method based on deep reinforcement learning provided by another embodiment of the present invention;
[0029] Figure 3 is a flowchart of a multi-robot path planning method based on deep reinforcement learning provided by another embodiment of the present invention;
[0030] Figure 4 is a flowchart of a multi-robot path planning method based on deep reinforcement learning provided by another embodiment of the present invention;
[0031] Figure 5 yes Figure 4 Specific method flow chart of step S620 in FIG.
[0032] Figure 6 is a flowchart of a multi-robot path planning method based on deep reinforcement learning provided by another embodiment of the present invention;
[0033] Figure 7 is a flowchart of a multi-robot path planning method based on deep reinforcement learning provided by another embodiment of the present invention;
[0034] Figure 8 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0036] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0037] The present invention provides a multi-robot path planning method, an electronic device and a computer-readable storage medium based on deep reinforcement learning. First, a first reference path of the robot is obtained, wherein the first reference path is obtained for each robot using a traditional path planning algorithm. Then, the first reference path is optimized to facilitate planning a more accurate path. The local motion space information of the robot is determined according to the first reference path, and then the local motion space information is input into a trained first neural network to obtain the local state characteristics of the robot, so that the state of the robot's environment can be determined according to the local state characteristics to avoid the robot from colliding with obstacles during movement. Secondly, the local state characteristics of the robot are input into a second neural network to obtain the robot's motion information to facilitate guiding the robot's moving path. Finally, the coding information of several robots closest to the robot is input into a third neural network so that the third neural network outputs a loss feature parameter. The first neural network, the second neural network and the third neural network are trained through a loss function to achieve the purpose of continuously optimizing the second neural network and the third neural network, thereby improving the success rate of the path planning of the second neural network and the third neural network. When the target training condition is that the target arrival ratio parameter is not less than the preset ratio parameter, the second reference path is determined as the target planning path of the robot, thereby realizing the path planning of multiple robots and improving the success rate.
[0038] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.
[0039] Reference Figure 1 , Figure 1 A multi-robot path planning method based on deep reinforcement learning is provided by an embodiment of the present invention. The multi-robot path planning method includes but is not limited to steps S100 to S300.
[0040] Step S100: For each robot, a first reference path of the robot is obtained, and training optimization processing is performed based on the first reference path of the robot until a target training condition is met;
[0041] It should be noted that to obtain the first reference path of the robot, the existence of dynamic obstacles must be ignored first, and the first reference path of the robot is obtained for each robot using a traditional path planning algorithm, where the traditional path planning algorithm includes but is not limited to a breadth-first search algorithm, a Dijkstra algorithm, or an A* algorithm, and this embodiment does not make any specific limitations.
[0042] In one embodiment, the training optimization process includes: first, determining the local motion space information of the robot according to the first reference path of the robot, inputting the local motion space information into the trained first neural network so that the first neural network performs feature processing on the local motion space information and outputs the local state features of the robot, and inputting the local state features of the robot into the second neural network so that the second neural network outputs the action information of the robot according to the local state features, and then, inputting the coding information of several robots closest to the robot into the third neural network so that the third neural network outputs the loss function feature parameters according to the multiple coding information, and calculating the loss function based on the loss feature parameters and the determined reward and punishment parameters, and finally training the first neural network, the second neural network and the third neural network through the loss function.
[0043] It should be noted that the encoded information is obtained by encoding local state features and action information.
[0044] In one embodiment, the local motion space information is input into the first neural network so that the first neural network describes the spatial state of the local motion space information, thereby outputting the local state characteristics of the robot to facilitate subsequent calculation of the robot's motion information.
[0045] It should be noted that the first neural network is a convolutional neural network.
[0046] In one embodiment, the local state features of the robot are input into the second neural network so that the second neural network transmits information to the robot according to the local state features, thereby outputting action information to facilitate planning of the robot's actions in the first reference path.
[0047] It should be noted that the second neural network is an actor (or Actor) neural network with a bidirectional recurrent neural network (or BicNet) as its core.
[0048] Step S200: obtaining a second reference path of the robot based on the motion information and the first reference path;
[0049] It should be noted that the second reference path is a target planning path to be optimized, and the target point of the first reference path and the second reference path is the same.
[0050] In one embodiment, the coding information of several robots that are closest to the robot is input into the third neural network, so that the third neural network outputs a loss characteristic parameter (or Q), and a loss function is calculated based on the loss characteristic parameter and the determined reward and punishment parameters, so as to facilitate the subsequent optimization of the second neural network and the third neural network through the loss function.
[0051] It should be noted that the third neural network is a judging (or Critic) neural network.
[0052] In one embodiment, the first neural network, the second neural network, and the third neural network are optimized and trained by using a loss function and a gradient descent algorithm to facilitate subsequent determination of a target planning path.
[0053] In one embodiment, the target training condition is that the target arrival ratio parameter is not less than the preset ratio parameter. When the target training condition is met, the training is terminated and the first neural network, the second neural network and the third neural network are saved. When the target training condition is not met, the training is continued until the target training condition is met, thereby improving the accuracy of multi-robot path planning.
[0054] It should be noted that the target arrival ratio parameter is the ratio of the robot that has arrived at the target point of the second reference path to all robots.
[0055] It should be noted that the preset proportion parameter may be 90%, 93% or 95%, etc., and is 95% in the present embodiment.
[0056] Step S300: Determine the second reference path as the target planning path of the robot.
[0057] In one embodiment, when it is determined that the target arrival ratio parameter is not less than the preset proportion parameter, the training is terminated and the second reference path is determined as the target planning path of the robot, thereby realizing path planning for multiple robots and improving accuracy.
[0058] refer to Figure 2 , Figure 2It is a flowchart of a multi-robot path planning method based on deep reinforcement learning provided by an embodiment of the present invention, which includes but is not limited to step S410.
[0059] Step S410: Input the local state features into the second neural network, so that the middle layer of the second neural network transmits the local state features based on the time series, plans the local state features to obtain action information, and outputs the action information.
[0060] In one embodiment, the local state features are input into the second neural network so that the middle layer of the second neural network transmits the local state features based on a time series. Finally, the second neural network plans the local state features to obtain action information, outputs the action information, and realizes the planning of the local state features of the robot.
[0061] It should be noted that the second neural network also includes an input layer and an output layer, etc., which are not specifically limited in this embodiment.
[0062] refer to Figure 3 , Figure 3 is a flowchart of a multi-robot path planning method based on deep reinforcement learning provided by an embodiment of the present invention, which includes but is not limited to step S610.
[0063] Step S610: When it is determined that the robot moves according to the first reference path, the reward and punishment parameters are calculated according to the action information.
[0064] In one embodiment, the reward and punishment parameters (or r) include an action reward and punishment value (or Rs) for characterizing the action changes occurring during the movement of the robot, a collision reward and punishment value (or Rc) for characterizing the collision between the robot and an obstacle, an oscillation reward and punishment value (or Ro) for characterizing the oscillating motion of the robot, a deviation reward and punishment value (or Rf) for characterizing the degree of deviation of the robot from the first reference path, and an arrival reward and punishment value (or Rg) for characterizing the arrival of the robot at the end point.
[0065] It should be noted that the reward and penalty parameters are obtained by adding the action reward and penalty value, collision reward and penalty value, shock reward and penalty value, deviation reward and penalty value, and arrival reward and penalty value, that is, formula (1):
[0066] r=Rs+Rc+Ro+γRf+Rg
[0067] It should be noted that γ can be 0.3, 0.4, or 0.5, etc., and this embodiment does not impose any specific limitation.
[0068] refer to Figure 4 , Figure 4 It is a flowchart of a multi-robot path planning method based on deep reinforcement learning provided by an embodiment of the present invention, which includes but is not limited to steps S620-S630.
[0069] Step S620: inputting the coded information of several robots closest to the robot into the attention network, so that the attention network calculates the coded information based on a preset selective attention algorithm and outputs a weight matrix;
[0070] It should be noted that the attention algorithm is selected to characterize the influence between robots.
[0071] In one embodiment, the third neural network includes an encoding network and an attention network centered on an attention mechanism, and the encoding information of several robots closest to the robot is input into the third neural network, so that the attention network in the third neural network outputs a weight matrix, which facilitates the calculation of subsequent loss feature parameters.
[0072] Step S630: Input the coding information and weight matrix into the coding network calculation, and output the loss feature parameters.
[0073] It can be understood that the loss characteristic parameters are characteristic parameters required by the loss function and are not limited here.
[0074] It should be noted that the coding network (or g j ) consists of a fully connected layer with the input of the state dimension plus the action dimension and a LeakyReLU activation function, which is used to encode the local state features and action information of the robot.
[0075] In one embodiment, the encoding information and the weight matrix are input into the encoding network for calculation, and the formula (2) for outputting the loss characteristic function parameters is as follows:
[0076] Q j (o j ,a j ) = f j (g j (o j ,a j ),y j )
[0077] Among them, f j are two fully connected layers in the encoding network, o j is the local state feature of robot j, α j is the action information of robot j, y j represents the contribution of the encoded information of the remaining robots to robot j.
[0078] refer to Figure 5 , Figure 5 yes Figure 4 Specific method flow chart of step S620 in the embodiment, wherein step S620 includes but is not limited to steps S621-S622.
[0079] Step: S621: Calculate the encoded information according to the selective attention algorithm to obtain the query matrix, key value matrix and value item matrix;
[0080] In one embodiment, the encoding information is calculated according to the selective attention algorithm to obtain the query matrix (or q j ), key-value matrix (or k i ) and the value matrix (or V i ), which is convenient for subsequent calculation of the weight matrix.
[0081] Step S622: Calculate a weight matrix based on the query matrix, the key-value matrix and the value item matrix.
[0082] It should be noted that the formula (3) for calculating the weight matrix based on the query matrix, key value matrix and value item matrix is as follows:
[0083]
[0084] Among them, d k It means k i Dimension information.
[0085] refer to Figure 6 , Figure 6 Another embodiment of the present invention provides a multi-robot path planning method based on deep reinforcement learning. The multi-robot path planning method includes but is not limited to steps S110 to S120.
[0086] Step S110: determining the coding information of several robots that are closest to each robot;
[0087] Step S120: Input the encoded information into the third neural network so that the third neural network obtains a loss feature set.
[0088] It should be noted that the loss feature parameter is a loss feature set.
[0089] In one embodiment, the encoding information of several robots closest to each robot is determined, including the action information a' and the local state feature information o', and the loss feature set Q(o',a') is calculated by formula (2).
[0090] It can be understood that the loss feature set is expanded to Q(o',a') = {Q1(o1',a1'),Q2(o2',a2'),...,Q n (o n ',a n ')}。
[0091] refer to Figure 7 , Figure 7 Another embodiment of the present invention provides a multi-robot path planning method based on deep reinforcement learning. Step S600 includes but is not limited to steps S640-S650.
[0092] In one embodiment, the loss function includes a second loss function and a third loss function, wherein the second loss function is used for the first neural network and the second neural network, and the third loss function is used to optimize the third neural network.
[0093] Step S640: performing a first calculation on the loss characteristic parameter and the reward and punishment parameter to obtain a third loss function for optimizing the third neural network;
[0094] In one embodiment, a first calculation is performed on the loss characteristic parameter and the reward and punishment parameter to obtain a third loss function. The specific formulas (4) and (5) are as follows:
[0095] y j =r j +γQ j (o' j ,a' j )
[0096]
[0097] Step S650: Perform a second calculation based on the reward and punishment parameters to obtain a second loss function for optimizing the first neural network and the second neural network.
[0098] In one embodiment, a second calculation is performed based on the reward and punishment parameters to obtain a second loss function. The specific formula and (6) are as follows:
[0099]
[0100] In addition, another embodiment of the present invention also provides a robot path planning device based on deep reinforcement learning, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor.
[0101] In addition, refer to Figure 8 As shown, an embodiment of the present invention further provides an electronic device, the electronic device 200 includes a memory 210, a processor 220, and a computer program stored in the memory 210 and executable on the processor 220.
[0102] The processor 220 and the memory 210 may be connected via a bus or in other ways.
[0103] The memory 210, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 210 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 210 may optionally include a memory remotely arranged relative to the processor 220, and these remote memories may be connected to the processor 220 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0104] The non-transient software program and instructions required to implement the multi-robot path planning method based on deep reinforcement learning of the above embodiment are stored in the memory. When executed by the processor, the multi-robot path planning method based on deep reinforcement learning of the above embodiment is executed, for example, the above-described Figure 1 The method steps S100 to S300, Figure 2 In the method step S410, Figure 3 In the method step S610, Figure 4 Steps S620 to S630 of the method, Figure 5 Steps S621 to S622 of the method, Figure 6 Steps S110 to S120 of the method, Figure 7 Method steps S640 to S650 in.
[0105] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor or a controller, for example, by a processor in the above system embodiment, so that the above processor can execute the convergence engine determination method in the above embodiment, for example, execute the above described Figure 1 The method steps S100 to S300, Figure 2 In the method step S410, Figure 3 Step S610 of the method, Figure 4 Steps S620 to S630 of the method, Figure 5 Steps S621 to S622 of the method, Figure 6 Steps S110 to S120 of the method, Figure 7 Method steps S640 to S650 in.
[0106] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0107] The above is a specific description of the preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.
Claims
1. A multi-robot path planning method based on deep reinforcement learning, characterized in that: include: For each robot, a first reference path of the robot is obtained, and a training optimization process is performed based on the first reference path of the robot until a target training condition is met, wherein the training optimization process includes: Determining local motion space information of the robot according to a first reference path of the robot; Inputting the local motion space information into a first neural network, so that the first neural network performs feature processing on the local motion space information and outputs local state features of the robot; Inputting the local state features of the robot into a second neural network, so that the second neural network outputs the action information of the robot according to the local state features; Based on the motion information and the first reference path, obtaining a second reference path of the robot; Inputting the encoding information of several robots closest to the robot into a third neural network, so that the third neural network outputs loss feature parameters according to the plurality of encoding information, and calculating a loss function based on the loss feature parameters and the determined reward and punishment parameters, wherein the encoding information is obtained by encoding the local state features and the action information; Training the first neural network, the second neural network, and the third neural network by the loss function; The target training condition is that a target arrival ratio parameter is not less than a preset ratio parameter, wherein the target arrival ratio parameter is a ratio of the robot that has arrived at the target point of the second reference path to all the robots; Determining the second reference path as the target planning path of the robot; The third neural network includes a coding network and an attention network; the coding information of a plurality of robots closest to the robot is input into the third neural network so that the third neural network outputs loss characteristic parameters according to the plurality of coding information, including: Inputting the coded information of several robots closest to the robot into the attention network, so that the attention network calculates the coded information based on a preset selective attention algorithm and outputs a weight matrix, wherein the selective attention algorithm is used to characterize the influence between the robots; Input the encoding information and the weight matrix into the encoding network for calculation, and output the loss characteristic parameter; The weight matrix is determined by the attention network based on the following steps: Calculating the encoded information according to the selective attention algorithm to obtain a query matrix, a key-value matrix, and a value item matrix; The weight matrix is calculated based on the query matrix, the key-value matrix and the value item matrix.
2. A multi-robot path planning method based on deep reinforcement learning according to claim 1, characterized in that: Inputting the local state feature of the robot into the second neural network so that the second neural network outputs the action information of the robot according to the local state feature includes: The local state feature is input into the second neural network, so that the middle layer of the second neural network transmits the local state feature based on the time series, plans the local state feature to obtain the action information, and outputs the action information.
3. The multi-robot path planning method based on deep reinforcement learning according to claim 1, characterized in that: The reward and punishment parameters are determined based on the following steps: When it is determined that the robot moves according to the first reference path, reward and punishment parameters are calculated according to the action information.
4. A multi-robot path planning method based on deep reinforcement learning according to claim 3, characterized in that: The reward and punishment parameters include an action reward and punishment value for characterizing the action changes occurring during the movement of the robot, a collision reward and punishment value for characterizing the collision between the robot and an obstacle, an oscillation reward and punishment value for characterizing the oscillating motion of the robot, a deviation reward and punishment value for characterizing the degree of deviation of the robot from the first reference path, and an arrival reward and punishment value for characterizing the arrival of the robot at the end point.
5. The multi-robot path planning method based on deep reinforcement learning according to claim 1, characterized in that: The loss feature parameter is a loss feature set; the coding information of a plurality of robots closest to the robot is input into the third neural network so that the third neural network outputs the loss feature parameter according to the plurality of coding information, including: Determine the coding information of a number of the robots that are closest to each of the robots; The encoded information is input into the third neural network so that the third neural network obtains the loss feature parameter feature set.
6. The multi-robot path planning method based on deep reinforcement learning according to claim 1, characterized in that: The loss function includes a second loss function and a third loss function; the loss function is calculated based on the loss characteristic parameter and the determined reward and punishment parameter, including: Performing a first calculation on the loss characteristic parameter and the reward and punishment parameter to obtain the third loss function for optimizing the third neural network; A second calculation is performed according to the reward and punishment parameters to obtain the second loss function for optimizing the first neural network and the second neural network.
7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a multi-robot path planning method based on deep reinforcement learning as described in any one of claims 1 to 6 is implemented.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a multi-robot path planning method based on deep reinforcement learning as described in any one of claims 1 to 6.