Multi-agent path planning method, device and equipment and storage medium

Abstract

The invention provides a multi-agent path planning method and device, equipment and a storage medium, and relates to the technical field of path planning. The method comprises the following steps: respectively performing path planning on a plurality of agents in a preset application environment to obtain a plurality of current paths, performing conflict detection on the plurality of current paths, and if at least two current paths have conflicts, establishing an observation space of the preset application environment according to the current paths of the plurality of agents in the preset application environment, and according to the observation space and the identifiers of the target intelligent agents corresponding to the at least two current paths, processing by adopting a target neural network model to obtain a target action strategy of the target intelligent agents at the target position, re-performing path planning on the target intelligent agents to obtain a target path of the target intelligent agents, and performing path planning on the target intelligent agents. And carrying out conflict detection on the current paths of the other agents and the target path of the target agent again until the paths of all the agents do not conflict.

Description

Technical Field

[0001] The present invention relates to the field of path planning, and in particular to a multi-agent path planning method, device, equipment and storage medium. Background Art

[0002] Currently, path planning for multiple agents can ensure that multiple agents can move efficiently and safely when performing tasks, while avoiding collisions and conflicts, so as to achieve the goal of completing tasks together.

[0003] A commonly used method in path planning for multiple agents is to first plan the optimal path for a single agent, and then artificially add constraints to resolve path conflicts between multiple agents. Although this method can resolve path conflicts, it usually sacrifices path length or increases the running time of the agent, making it difficult to guarantee optimal path planning. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-agent path planning method, device, equipment and storage medium to address the deficiencies in the above-mentioned prior art, so as to intelligently output the target action strategy of the target agent at the target position through reinforcement learning of the target neural network model, so as to re-plan the path of the target agent.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a multi-agent path planning method, comprising:

[0007] Performing path planning for multiple agents in a preset application environment to obtain multiple current paths, where the multiple current paths correspond to the multiple agents;

[0008] Performing conflict detection on the multiple current paths;

[0009] If there are at least two current paths that conflict, establishing an observation space of the preset application environment according to the current paths of the multiple agents in the preset application environment;

[0010] According to the identification of each target agent corresponding to the observation space and the at least two current paths, a target neural network model is used for processing to obtain a target action strategy of the target agent at a target position;

[0011] Using the target action strategy at the target position, re-planning the path of the target intelligent agent to obtain the target path of the target intelligent agent;

[0012] Re-check the conflicts between the current paths of other agents and the target path of the target agent until there are no conflicts between the paths of all agents.

[0013] In an optional implementation manner, establishing the observation space of the preset application environment according to the current paths of the multiple agents in the preset application environment includes:

[0014] An observation space of the preset application environment is established according to the static obstacle information in the preset application environment and the current paths of the multiple agents.

[0015] In an optional implementation, establishing the observation space of the preset application environment according to the static obstacle information in the preset application environment and the current paths of the multiple agents includes:

[0016] Obtaining a grid map of the preset application environment;

[0017] According to the grid map and the static obstacle information, a first observation channel is constructed, wherein the first observation channel records obstacle indication information of each grid area in the grid map; the obstacle indication information is used to indicate whether each grid area has a static obstacle;

[0018] According to the grid map and the current paths of the multiple agents, a plurality of second observation channels are respectively constructed, each of which records path indication information of each target grid area in the grid map, and the path indication information of each target grid area is used to indicate the time length for the corresponding agent to reach the target grid area;

[0019] The observation space is constructed according to the first observation channel and the plurality of second observation channels.

[0020] In an optional implementation, adopting the target action strategy at the target position to re-plan the path of the target agent to obtain the target path of the target agent includes:

[0021] Obtaining a preset action constraint corresponding to the target action strategy, wherein the preset action constraint is used to indicate that the target agent is prohibited from occupying a position within a preset time step;

[0022] The preset action constraints are adopted to re-plan the path of the target intelligent agent to obtain the target path of the target intelligent agent.

[0023] In an optional embodiment, before the target action strategy of the target agent at the target position is obtained by processing the target neural network model according to the identification of each target agent corresponding to the observation space and the at least two current paths, the method further includes:

[0024] Using an initial neural network model, a training sample set is constructed according to the observation space and the identifiers of each target intelligent agent corresponding to the at least two current paths; wherein the training sample set includes: multiple groups of training samples;

[0025] The initial neural network model is trained according to the training sample set to obtain the target neural network model.

[0026] In an optional implementation, constructing a training sample set according to the observation space and the identifiers of the target agents corresponding to the at least two current paths includes:

[0027] Using the initial neural network model, according to the observation space and the identifier of the target intelligent agent, determine a first action strategy and a first state of the target intelligent agent;

[0028] Determining a second state of the target agent according to the first action strategy and the observation space;

[0029] According to the first state and the second state, a preset reward function is used to determine a first reward value;

[0030] Constructing a set of first training samples according to the first reward value, the first state, the second state and the first action strategy;

[0031] The initial neural network model is used to determine the second action strategy of the target agent according to the first training sample, and a set of second training samples is constructed until a preset number of training sample construction times is reached to obtain the training sample set.

[0032] In an optional implementation, training the initial neural network model according to the training sample set to obtain the target neural network model includes:

[0033] According to the training sample set, the initial neural network model is trained to calculate the loss function value;

[0034] If the change in the loss function value does not satisfy the preset stop iteration condition, the initial neural network model continues to be trained until the change in the loss function value satisfies the preset stop iteration condition, or the number of training times for the initial neural network model reaches the preset number of iterations, the training of the initial neural network model is stopped, and the neural network model at the time of stopping training is determined to be the target neural network model.

[0035] In a second aspect, the embodiment of the present application further provides a multi-agent path planning device, comprising:

[0036] A planning module, used to perform path planning for multiple agents in a preset application environment to obtain multiple current paths, where the multiple current paths correspond to the multiple agents;

[0037] A detection module, used for performing conflict detection on the multiple current paths;

[0038] An establishing module, configured to establish an observation space of the preset application environment according to the current paths of the plurality of agents in the preset application environment if at least two current paths conflict with each other;

[0039] A processing module, configured to obtain a target action strategy of the target agent at a target position by using a target neural network model to process the target agent according to the identifications of the target agents corresponding to the observation space and the at least two current paths;

[0040] The planning module is further used to adopt the target action strategy at the target position to re-plan the path of the target intelligent agent to obtain the target path of the target intelligent agent;

[0041] The detection module is also used to re-detect the conflict between the current paths of other intelligent agents and the target path of the target intelligent agent until there is no conflict between the paths of all intelligent agents.

[0042] In the third aspect, an embodiment of the present application also provides a computer device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the computer device is running, the processor communicates with the storage medium through the bus, and the processor executes the program instructions to perform the steps of the multi-agent path planning method as described in any one of the first aspects.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the multi-agent path planning method as described in any one of the first aspects are executed.

[0044] The beneficial effects of this application are:

[0045] The embodiments of the present application provide a multi-agent path planning method, apparatus, device and storage medium, including: performing path planning for multiple agents in a preset application environment to obtain multiple current paths, the multiple current paths corresponding to multiple agents, performing conflict detection on the multiple current paths, if there are conflicts between at least two current paths, establishing an observation space of the preset application environment according to the current paths of the multiple agents in the preset application environment, using a target neural network model for processing according to the identification of each target agent corresponding to the observation space and at least two current paths to obtain a target action strategy of the target agent at a target position, using the target action strategy at the target position to re-plan the path of the target agent to obtain the target path of the target agent, and re-detecting conflicts between the current paths of other agents and the target path of the target agent until there are no conflicts in the paths of all agents. The method of the present application, when it is detected that there is a path conflict between at least two current paths, establishes an observation space of a preset application environment, and uses a target neural network model to process the observation space and the identification of each target intelligent agent corresponding to at least two current paths. The target neural network model can be used for reinforcement learning to output the target action strategy of the target intelligent agent at the target position, which is used to re-plan the path of the target intelligent agent to obtain the target path of the target intelligent agent, and then judge whether there is a conflict between the paths of multiple agents. If there is no conflict, the final path of the multiple agents in the preset application environment is obtained, thereby improving the efficiency of multi-agent path planning. In addition, the method of the present application takes into account the information of the preset application environment and can be applied to various application environments. For different application environments, the final path of the multiple agents in the application environment can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 One of the flow charts of a multi-agent path planning method provided in an embodiment of the present application;

[0048] Figure 2 A second flowchart of a multi-agent path planning method provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of an observation space provided in an embodiment of the present application;

[0050] Figure 4 A third flowchart of a multi-agent path planning method provided in an embodiment of the present application;

[0051] Figure 5 A fourth flowchart of a multi-agent path planning method provided in an embodiment of the present application;

[0052] Figure 6 A fifth flowchart of a multi-agent path planning method provided in an embodiment of the present application;

[0053] Figure 7 A sixth flowchart of a multi-agent path planning method provided in an embodiment of the present application;

[0054] Figure 8 A schematic diagram of the functional modules of a multi-agent path planning device provided in an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0057] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0058] In the description of the present application, it should be noted that if the terms "upper", "lower", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship in which the product of the application is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0059] In addition, the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0060] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0061] In order to solve the path conflict problem during multi-agent path planning, an embodiment of the present application provides a multi-agent path planning method. When there is a conflict in the current path of the target agent, the target action strategy of the target agent at the target position is automatically determined by processing the observation space and the identifiers of each target agent corresponding to at least two current paths using a target neural network model, so as to re-plan the path of the target agent according to the target action strategy to obtain the target path of the target agent, and then judge whether there is a conflict in the paths of multiple agents. If there is no conflict, the final path of the multi-agent in the preset application environment is obtained, which improves the efficiency of multi-agent path planning and is suitable for various application environments.

[0062] The following is a detailed explanation of a multi-agent path planning method provided by an embodiment of the present application through specific examples in conjunction with the accompanying drawings. A multi-agent path planning method provided by an embodiment of the present application can be implemented by a computer device pre-installed with: a detection algorithm or detection software for a preset multi-agent path planning method, by running the algorithm or software. The computer device can be, for example, a server or a terminal, and the terminal can be a user computer. Figure 1 This is one of the flow charts of a multi-agent path planning method provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0063] S101. Perform path planning for multiple agents in a preset application environment to obtain multiple current paths.

[0064] In this embodiment, multiple current paths correspond to multiple agents.

[0065] Among them, an intelligent agent is a software, hardware or other entity with autonomous and adaptive capabilities that can recognize or simulate human intelligent behavior. An intelligent agent can be regarded as a computing entity that can continuously and autonomously interact with the environment. Intelligent agents are used in self-driving cars, smart homes, robots and other fields. If the computing entity of the intelligent agent is a car, the preset application environment can be a warehouse environment, a park environment, etc., and path planning is performed for multiple intelligent agents in the preset application environment.

[0066] For example, the preset application environment is considered as a discrete grid world of size n×H×W, where n represents the information in the preset application environment is divided into n channels, H×W represents the size of the channel, and the set of multiple agents is represented as: A={a 1 ,…,a k Specifically, the starting position and target position of multiple agents in the preset application environment are obtained. i There is a starting position s i ∈V and the target position g i ∈V, where V represents the set of grid positions in the discrete grid world.

[0067] Using the path planning algorithm, according to the starting positions and target positions of multiple agents in the preset application environment, path planning is performed for multiple agents to obtain multiple current paths of multiple agents, and time is discretized into time steps. i The path p i ={(t 1 ,l 1 ),(t 2 ,l 2 )…(t m ,l m )} is a sequence consisting of a series of moving or waiting actions, l i Represented as agent a i The grid position, t i Indicates arrival i The timestamp of 1 =s i , l m =g i .

[0068] The path planning algorithm can use the A-star Algorithm (A*) algorithm in the Conflict-Based Search (CBS) algorithm to find each agent a i From the starting position s i To the target position g i The optimal path p i,A* algorithm is an algorithm based on graph search, which uses heuristic functions to evaluate each node, thereby accelerating the search process while ensuring that the optimal path is found, and selecting the optimal path, that is, the shortest path, the current path, by estimating the total cost of each node.

[0069] S102: Perform conflict detection on multiple current paths.

[0070] Specifically, by preselecting and setting the path conflict condition, it is determined whether there is a conflict among the multiple current paths according to the multiple current paths and the path conflict condition.

[0071] Among them, the path conflict conditions include: (1) vertex conflict condition Con = (a i ,a j ,l,t): Agent a i and a j At the same time step t at the same position l; (2) edge conflict condition Con = (a i ,a j ,u,v,t):agent a i and a j Traverse the same edge (u,v) in the opposite direction from time step t-1 to time step t.

[0072] Conflict detection is performed on multiple current paths according to the path conflict conditions, starting from the current path of the first agent of multiple agents, to determine whether the current paths of other agents have vertex conflicts or edge conflicts with the current path of the first agent. If any conflict exists, the two conflicting paths are recorded until conflict detection is performed on the current paths of all agents, and all conflicting paths are recorded.

[0073] S103: If there are conflicts between at least two current paths, an observation space of the preset application environment is established according to the current paths of multiple agents in the preset application environment.

[0074] Optionally, an observation space of the preset application environment is established according to static obstacle information in the preset application environment and current paths of multiple agents.

[0075] Among them, the static obstacle information in the preset application environment is the location information of the static obstacles in the preset application environment. For example, the preset application environment is a warehouse environment, and the static obstacle information can be the location where goods are stacked, the location where passage is prohibited, etc. Then, according to the static obstacle information in the preset application environment and the current paths of multiple intelligent agents, the observation space of the preset application environment is established.

[0076] S104, according to the identification of each target agent corresponding to the observation space and at least two current paths, a target neural network model is used for processing to obtain a target action strategy of the target agent at the target position.

[0077] Specifically, after conflict detection is performed on multiple current paths, all current paths with conflicts are recorded. Since three or four current paths among the multiple current paths may have path conflicts within a time period, the paths with conflicts within the time period are recorded as a group of current paths, and then the first group of current paths recorded is selected from all the current paths with conflicts as at least two current paths.

[0078] According to the identification of each target agent corresponding to the observation space and at least two current paths, a target neural network model is used for processing to determine the target action strategy of the target agent at the target position, which is used for path planning for the target agent.

[0079] S105: adopt the target action strategy at the target position, re-plan the path of the target intelligent agent, and obtain the target path of the target intelligent agent.

[0080] If the identities of the target agents corresponding to at least two current paths include the target agent a i and target agent a j , the target action strategy is the target agent a i and target agent a j If the position l is forbidden to be occupied at the time step t, then according to the target action strategy, the path planning algorithm is used again to respectively i and target agent a j Perform path planning and obtain the target agent a i and target agent a j The target path.

[0081] S106, re-perform conflict detection on the current paths of other intelligent agents and the target path of the target intelligent agent until there is no conflict in the paths of all intelligent agents.

[0082] According to the preset path conflict conditions, the current paths of other intelligent agents and the target paths of the target intelligent agent are detected for conflicts to determine whether there are path conflicts. If there are no conflicts in the paths of all intelligent agents, the final paths of multiple intelligent agents in the preset application environment are obtained, and the path planning for multiple intelligent agents is completed.

[0083] If it is determined that there is still a path conflict, the above steps S103-S106 are used to continue to determine the target action strategy of the intelligent agent with path conflict, and to perform path planning for the intelligent agent with path conflict, and to perform conflict detection on the re-planned path and other current paths until there is no conflict in the paths of all intelligent agents, and the final paths of multiple intelligent agents in the preset application environment are obtained, completing the path planning for multiple intelligent agents.

[0084] In summary, an embodiment of the present application provides a multi-agent path planning method, including: performing path planning for multiple agents in a preset application environment to obtain multiple current paths, the multiple current paths corresponding to multiple agents, performing conflict detection on the multiple current paths, if there are conflicts between at least two current paths, establishing an observation space of the preset application environment according to the current paths of the multiple agents in the preset application environment, using a target neural network model for processing according to the identification of each target agent corresponding to the observation space and at least two current paths to obtain a target action strategy of the target agent at the target position, using the target action strategy at the target position to re-plan the path of the target agent to obtain the target path of the target agent, and re-detecting conflicts between the current paths of other agents and the target path of the target agent until there are no conflicts in the paths of all agents. The method of the present application, when it is detected that there is a path conflict between at least two current paths, establishes an observation space of a preset application environment, and uses a target neural network model to process the observation space and the identification of each target intelligent agent corresponding to at least two current paths. The target neural network model can be used for reinforcement learning to output the target action strategy of the target intelligent agent at the target position, which is used to re-plan the path of the target intelligent agent to obtain the target path of the target intelligent agent, and then judge whether there is a conflict between the paths of multiple agents. If there is no conflict, the final path of the multiple agents in the preset application environment is obtained, thereby improving the efficiency of multi-agent path planning. In addition, the method of the present application takes into account the information of the preset application environment and can be applied to various application environments. For different application environments, the final path of the multiple agents in the application environment can be obtained.

[0085] The present application embodiment also provides another possible implementation of the multi-agent path planning method. Figure 2 The second flowchart of a multi-agent path planning method provided in an embodiment of the present application. Figure 3 A schematic diagram of an observation space provided in an embodiment of the present application. Figure 2 As shown, according to the static obstacle information in the preset application environment and the current paths of multiple agents, the observation space of the preset application environment is established, including:

[0086] S201: Obtain a grid map of a preset application environment.

[0087] In this embodiment, the grid map of the preset application environment is also a discrete grid world of the preset application environment, and the size of the grid map is a discrete grid world of n×H×W, wherein n represents dividing the information in the preset application environment into n channels, and H×W represents the size of the channel.

[0088] S202: Construct a first observation channel according to the grid map and static obstacle information.

[0089] The first observation channel records obstacle indication information of each grid area in the grid map; the obstacle indication information is used to indicate whether each grid area has a static obstacle.

[0090] Specifically, the position of the static obstacle in the preset application scenario is obtained, and it is determined whether there is a static obstacle in each grid area in the grid map, such as Figure 3 As shown, the observation values ​​in the first observation channel are composed of a binary matrix, which is used to indicate the position of the static obstacle. If there is a static obstacle in the grid area in the grid map, the obstacle indication information is represented by -1 in the binary matrix. If there is no static obstacle in the grid area in the grid map, it is determined to be a free cell, and the obstacle indication information is represented by 0 in the binary matrix. The free cell is indicated as a cell that allows the intelligent agent path to pass through.

[0091] S203: construct a plurality of second observation channels respectively according to the grid diagram and the current paths of the plurality of intelligent agents.

[0092] Each second observation channel records the path indication information of each target grid area in the grid map, and the path indication information of each target grid area is used to indicate the time length for the corresponding intelligent agent to reach each target grid area.

[0093] Specifically, each second observation channel is used to represent the current path of each agent. Figure 3 , the current path of the agent is represented by the arrival position l in the binary matrix i t i Value to represent.

[0094] S204: construct an observation space according to the first observation channel and the plurality of second observation channels.

[0095] The static obstacle information in the preset application environment and the current paths of multiple agents are divided into K+1 observation channels to obtain the observation space to simplify the learning task of the agent, where K represents the number of agents, each agent has a corresponding current path, and each agent has the information of all static obstacles in the preset application environment. During the movement of the agent, the agent can obtain its global position in the preset application environment, and can obtain the movement trajectory and current position of dynamic obstacles. Dynamic obstacles refer to other agents. When the agent approaches the edge of the discrete grid world, obstacles will be added to all positions outside the world boundary, that is, the agent cannot pass through the edge of the discrete grid world.

[0096] In the method provided in the embodiment of the present application, a grid map of a preset application environment is obtained, and a first observation channel is constructed according to the grid map and static obstacle information. The first observation channel records obstacle indication information of each grid area in the grid map; the obstacle indication information is used to indicate whether each grid area has a static obstacle. According to the grid map and the current paths of multiple intelligent agents, multiple second observation channels are respectively constructed, and each second observation channel records path indication information of each target grid area in the grid map. The path indication information of each target grid area is used to indicate the time for the corresponding intelligent agent to reach each target grid area. According to the first observation channel and the multiple second observation channels, an observation space is constructed. By dividing the static obstacle information in the preset application environment and the current paths of multiple intelligent agents into K+1 observation channels, an observation space is obtained, which can simplify the learning task of multiple intelligent agents.

[0097] The present application embodiment also provides another possible implementation of the multi-agent path planning method. Figure 4 The third flowchart of a multi-agent path planning method provided in the embodiment of the present application. Figure 4 As shown, the target action strategy at the target position is adopted to re-plan the path of the target agent to obtain the target path of the target agent, including:

[0098] S301: Obtain preset action constraints corresponding to the target action strategy.

[0099] The preset action constraint is used to indicate that the target agent is prohibited from occupying a position within a preset time step.

[0100] S302: Re-plan the path of the target intelligent agent using the preset action constraints to obtain the target path of the target intelligent agent.

[0101] In this embodiment, if the identifiers of the target agents corresponding to at least two current paths include target agent a i and target agent a j , the target action strategy is the target agent ai and target agent a j If both are prohibited from occupying position l at time step t, then according to the target action strategy, the target agent a is determined i The default action constraint is Cons i =(a i ,l,t), indicating the target agent a i The target agent a is prohibited from occupying position l within the preset time step t. j The default action constraint is Cons j =(a j ,l,t), indicating the target agent a j It is forbidden to occupy position l within the preset time step t.

[0102] Then according to the target agent a i The preset action constraints and target agent a j The preset action constraints are used to re-adopt the path planning algorithm to target the intelligent agent a i and target agent a j Perform path planning and obtain the target agent a i and target agent a j The target path.

[0103] In the method provided in the embodiment of the present application, a preset action constraint corresponding to the target action strategy is obtained, the preset action constraint is used to indicate that the target intelligent agent is prohibited from occupying a position within a preset time step, the preset action constraint is used to re-plan the path of the target intelligent agent to obtain the target path of the target intelligent agent, the preset action constraint of the target intelligent agent is determined by the target action strategy, and the path is re-planned according to the preset action constraint to obtain the target path of the target intelligent agent. There is no path conflict between the target paths, but there may be conflicts with the current paths of other intelligent agents.

[0104] The present application embodiment also provides another possible implementation of the multi-agent path planning method. Figure 5 The fourth flowchart of a multi-agent path planning method provided in an embodiment of the present application. Figure 5 As shown, according to the identification of each target agent corresponding to the observation space and at least two current paths, the target neural network model is used for processing to obtain the target action strategy of the target agent at the target position, the method also includes:

[0105] S401, using an initial neural network model, constructing a training sample set according to the observation space and the identifiers of each target intelligent agent corresponding to at least two current paths.

[0106] The training sample set includes: multiple groups of training samples.

[0107] S402: Train the initial neural network model according to the training sample set to obtain a target neural network model.

[0108] In this embodiment, the initial neural network model can be a deep reinforcement learning network, which includes: a Q-learning network, a reinforcement learning algorithm based on temporal difference (State-Action-Reward-State-Action, SARSA), and the Q-Learning network updates the Q value after path conflict detection and selects the optimal constraint based on the maximum Q value. The SARSA algorithm generates constraints based on the state-action pair update strategy.

[0109] An initial neural network is used to process the identification of each target intelligent agent corresponding to the observation space and at least two current paths, to construct multiple sets of training samples, and then the initial neural network model is trained to obtain a target neural network model.

[0110] In the method provided in the embodiment of the present application, an initial neural network model is used to construct a training sample set based on the observation space and the identifiers of each target intelligent agent corresponding to at least two current paths; wherein the training sample set includes: multiple groups of training samples; based on the training sample set, the initial neural network model is trained to obtain a target neural network model, so that the target neural network model can output a target action strategy for planning the path of the intelligent agent.

[0111] The present application embodiment also provides another possible implementation of the multi-agent path planning method. Figure 6 The fifth flow chart of a multi-agent path planning method provided in the embodiment of the present application. Figure 6 As shown, the initial neural network model is used to construct a training sample set according to the observation space and the identification of each target agent corresponding to at least two current paths, including:

[0112] S501. Using an initial neural network model, according to the observation space and the identifier of the target intelligent agent, determine a first action strategy and a first state of the target intelligent agent.

[0113] In this embodiment, the initial neural network model uses a 6-layer convolutional network with several 3×3 small kernels between each maximum pooling layer. Specifically, the input of the initial neural network model represents the channel matrix of the observation space ((K+1)×H×W tensor) through three convolution (conv2D) and maxpool stages, followed by the last convolution layer, and then the preprocessed input passes through two fully connected layers and is finally fed into a long short-term memory (LSTM) unit with an output size of 500. The residual shortcut connects the output of the connection layer to the input layer of the LSTM. The output layer consists of a policy neuron with softmax activation, a value output, and a feature layer for training each agent to know whether the added constraints can effectively resolve path conflicts.

[0114] Then, according to the observation space and the identification of the target agent, the first action strategy of the target agent is output. The first state includes: the current path of all agents, the duration of each path, etc.

[0115] S502: Determine a second state of the target agent according to the first action strategy and the observation space.

[0116] S503: Determine a first reward value using a preset reward function according to the first state and the second state.

[0117] Specifically, according to the first action strategy, the path of the target agent is replanned to obtain the second path of the target agent. The second state of the target agent includes: the second path of the target agent and the current paths of other agents, the duration of each path, etc.

[0118] Compare the first state and the second state to determine whether there is still a conflict between the paths, and determine whether the paths of the first state and the second state have changed, and then determine whether the first action strategy is invalid. For example, the constraints corresponding to the first action strategy are too weak, so that the path predicted by the target agent has not changed, or the constraints corresponding to the first action strategy are too strong, so that the target agent cannot find a path that meets the constraints. During the initial neural network model training process, actions are only sampled from valid actions. In order to counter the convergence of the oscillating strategy, a preset reward function is used during training to prevent the target agent from returning to the last planned path.

[0119] Specifically, in a discrete grid world, the reward function will follow the target agent and be penalized for conflicts between the paths planned, resulting in a strategy to find a collision-free path as quickly as possible. The penalty for agents whose path conflicts are not resolved but whose paths are changed after adding the constraints corresponding to the first action strategy is slightly lower than that for agents whose path conflicts are not resolved and whose paths are not changed, to encourage continued exploration. For situations where the current conflict is resolved but a new conflict arises immediately afterwards, appropriate rewards are given, and removing this aspect of the reward function may lead to poor convergence. When the added constraints enable the agent to find a collision-free path, that is, there is no conflict between the paths, a larger reward is given to the agent. The specific reward results are shown in Table 1:

[0120] Table 1 Reward structure

[0121] Action Reward Paths conflict and have not changed -2.0 Paths conflict but changed -1.5 The current conflict has been resolved but a new conflict has arisen +0.3 Conflict Resolution +20.0

[0122] Then, according to the first state and the second state, a preset reward function is used to determine the first reward value.

[0123] S504: Construct a set of first training samples according to the first reward value, the first state, the second state and the first action strategy.

[0124] S505: Using the initial neural network model, according to the first training sample, determine the second action strategy of the target agent, construct a set of second training samples, until reaching a preset number of training sample construction times, and obtain a training sample set.

[0125] According to the first reward value, the first state, the second state and the first action strategy, a group of first training samples are constructed, and the initial neural network model is continued to be used. According to the second state, the second action strategy of the target intelligent agent is determined. According to the second action strategy and the observation space, the third state of the target intelligent agent is determined. According to the second state and the third state, a preset reward function is used to determine the second reward value. According to the second reward value, the second state, the third state and the second action strategy, a group of second training samples are constructed until the preset number of training sample construction times is reached to obtain a training sample set.

[0126] If the preset number of training sample construction times is set to 256 times, 256 groups of training samples can be obtained to obtain a training sample set.

[0127] In the method provided in the embodiment of the present application, an initial neural network model is used to determine the first action strategy and the first state of the target intelligent agent according to the observation space and the identifier of the target intelligent agent, and the second state of the target intelligent agent is determined according to the first action strategy and the observation space. According to the first state and the second state, a preset reward function is used to determine the first reward value, and a group of first training samples are constructed according to the first reward value, the first state, the second state and the first action strategy. Then, the initial neural network model is continued to be used to determine the second action strategy of the target intelligent agent according to the first training sample, and a group of second training samples is constructed until the preset number of training sample constructions is reached to obtain a training sample set, each group of training samples has a reward value corresponding to the action strategy, and the reward value can be used to judge whether the action strategy is effective and whether path conflicts can be avoided, so that the training sample set obtained can be used to train the initial neural network model, so that the trained neural network model can learn and output a better action strategy.

[0128] The present application embodiment also provides another possible implementation of the multi-agent path planning method. Figure 7 The sixth flow chart of a multi-agent path planning method provided in the embodiment of the present application. Figure 7 As shown, according to the training sample set, the initial neural network model is trained to obtain the target neural network model, including:

[0129] S601, training the initial neural network model according to the training sample set, and calculating the loss function value;

[0130] S602. If the change in the loss function value does not satisfy the preset stop iteration condition, continue to train the initial neural network model until the change in the loss function value satisfies the preset stop iteration condition, or the number of training times for the initial neural network model reaches the preset number of iterations, stop training the initial neural network model, and determine that the neural network model at the time of stopping training is the target neural network model.

[0131] In this example, during the training of the initial neural network model, the policy, value, and "blocking" outputs are updated in batches every 256 steps or at the end of an episode. Typically, the loss can be updated to match the total gain by minimizing the value function:

[0132]

[0133] in, To update the policy, the advantage function is usually used An entropy term H(π(o)) is added to the policy update loss, which has been shown to encourage exploration and prevent premature convergence by penalizing policies that always choose the same action. The loss function is expressed as follows:

[0134]

[0135] Among them, the entropy weight σ H = 0.01, and rely on two additional loss functions to help guide and stabilize training. First, by minimizing L blocking To update the blocked prediction output, secondly, define the loss function L valid , to minimize the log-likelihood of choosing an invalid constraint.

[0136] Specifically, the preset stopping iteration condition may be that the amount of change in the loss function value for multiple consecutive times tends to be stable, and it is judged whether the number of training times for the initial neural network model reaches the preset number of iterations. If the amount of change in the loss function value for multiple consecutive times does not tend to be stable, then reinforcement learning is continued, the initial neural network model is trained, the loss function value is calculated, and it is judged again whether the amount of change in the loss function value tends to be stable.

[0137] If it is judged that the change in the loss function value has not yet tended to be flat, but the number of training times for the initial neural network model has reached the preset number of iterations, the training of the initial neural network model is stopped; if it is judged that the change in the loss function value tends to be flat, the training of the initial neural network model is also stopped, and the neural network model at the time of stopping training is determined to be the target neural network model.

[0138] In the method provided in the embodiment of the present application, the initial neural network model is trained according to the training sample set, and the loss function value is calculated; if the change in the loss function value does not meet the preset stop iteration condition, the initial neural network model continues to be trained until the change in the loss function value meets the preset stop iteration condition, or the number of training times for the initial neural network model reaches the preset number of iterations, the training of the initial neural network model is stopped, and the neural network model at the time of stopping the training is determined to be the target neural network model. The trained target neural network model is used to train the intelligent agent to intelligently generate action strategies after conflict detection, replacing the method of generating constraints based on predefined rules in the traditional CBS algorithm. It can dynamically adjust the generated action strategies according to the preset application environment and path conflict conditions, thereby improving the efficiency of path planning. At the same time, the target neural network model has continuous learning capabilities, that is, it can continuously accumulate experience in practical applications to improve the efficiency of action strategy generation.

[0139] The following is a corresponding explanation of the multi-agent path planning device and computer equipment provided by any of the above embodiments of the present application. Its specific implementation process and the technical effects produced are the same as those of the corresponding method embodiments mentioned above. For the sake of brief description, the parts not mentioned in this embodiment can refer to the corresponding contents in the method embodiments.

[0140] Figure 8 This is a functional module diagram of a multi-agent path planning device provided in an embodiment of the present application. Figure 8 As shown, the multi-agent path planning device 100 includes:

[0141] The planning module 110 is used to perform path planning for multiple agents in a preset application environment to obtain multiple current paths, where the multiple current paths correspond to multiple agents;

[0142] A detection module 120, configured to perform conflict detection on multiple current paths;

[0143] An establishing module 130, for establishing an observation space of a preset application environment according to current paths of multiple agents in the preset application environment if there are at least two current paths that conflict;

[0144] A processing module 140 is used to process the target intelligent agent at the target position using a target neural network model according to the identification of each target intelligent agent corresponding to the observation space and at least two current paths;

[0145] The planning module 110 is further used to adopt the target action strategy at the target position to re-plan the path of the target intelligent agent to obtain the target path of the target intelligent agent;

[0146] The detection module 120 is further used to re-detect the conflicts between the current paths of other agents and the target path of the target agent until there are no conflicts between the paths of all agents.

[0147] Optionally, the establishing module 130 is further used to establish an observation space of the preset application environment according to the static obstacle information in the preset application environment and the current paths of the multiple agents.

[0148] Optionally, module 130 is established and is also used to obtain a grid map of a preset application environment; based on the grid map and static obstacle information, a first observation channel is constructed, and the first observation channel records obstacle indication information of each grid area in the grid map; the obstacle indication information is used to indicate whether each grid area has a static obstacle; based on the grid map and the current paths of multiple intelligent agents, multiple second observation channels are respectively constructed, and each second observation channel records path indication information of each target grid area in the grid map, and the path indication information of each target grid area is used to indicate the time length for the corresponding intelligent agent to reach each target grid area; based on the first observation channel and the multiple second observation channels, an observation space is constructed.

[0149] Optionally, the planning module 110 is also used to obtain preset action constraints corresponding to the target action strategy, and the preset action constraints are used to indicate that the target intelligent agent is prohibited from occupying a position within a preset time step; the preset action constraints are used to re-plan the path of the target intelligent agent to obtain the target path of the target intelligent agent.

[0150] Optionally, the multi-agent path planning device 100 further includes:

[0151] The training module is used to use the initial neural network model to construct a training sample set according to the identification of each target intelligent agent corresponding to the observation space and at least two current paths; wherein the training sample set includes: multiple groups of training samples; according to the training sample set, the initial neural network model is trained to obtain the target neural network model.

[0152] Optionally, the training module is also used to use an initial neural network model to determine a first action strategy and a first state of the target intelligent agent according to the observation space and the identifier of the target intelligent agent; determine a second state of the target intelligent agent according to the first action strategy and the observation space; determine a first reward value according to the first state and the second state using a preset reward function; construct a set of first training samples according to the first reward value, the first state, the second state and the first action strategy; use the initial neural network model to determine the second action strategy of the target intelligent agent according to the first training samples, and construct a set of second training samples until a preset number of training sample construction times is reached to obtain a training sample set.

[0153] Optionally, the training module is also used to train the initial neural network model according to the training sample set and calculate the loss function value; if the change in the loss function value does not meet the preset stop iteration condition, the initial neural network model continues to be trained until the change in the loss function value meets the preset stop iteration condition, or the number of training times for the initial neural network model reaches a preset number of iterations, the training of the initial neural network model is stopped, and the neural network model at the time of stopping training is determined to be the target neural network model.

[0154] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0155] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0156] Fig. 9 Schematic diagram of a computer device provided in an embodiment of the present application, which can be used for multi-agent path planning. Fig. 9 As shown, the computer device 200 includes: a processor 210 , a storage medium 220 , and a bus 230 .

[0157] The storage medium 220 stores machine-readable instructions executable by the processor 210. When the computer device is running, the processor 210 communicates with the storage medium 220 via the bus 230, and the processor 210 executes the machine-readable instructions to perform the steps of the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0158] Optionally, the present application further provides a storage medium 220, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method embodiment are executed. The specific implementation method and technical effect are similar and will not be repeated here.

[0159] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0160] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0162] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0163] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multi-agent path planning method, characterized in that: include: Performing path planning for multiple agents in a preset application environment to obtain multiple current paths, where the multiple current paths correspond to the multiple agents; Performing conflict detection on the multiple current paths; If there are at least two current paths that conflict, establishing an observation space of the preset application environment according to the current paths of the multiple agents in the preset application environment; According to the identification of each target agent corresponding to the observation space and the at least two current paths, a target neural network model is used for processing to obtain a target action strategy of the target agent at a target position; Using the target action strategy at the target position, re-planning the path of the target intelligent agent to obtain the target path of the target intelligent agent; Re-check the conflicts between the current paths of other agents and the target path of the target agent until there are no conflicts between the paths of all agents.

2. The method according to claim 1, characterized in that: The step of establishing the observation space of the preset application environment according to the current paths of the plurality of agents in the preset application environment comprises: An observation space of the preset application environment is established according to the static obstacle information in the preset application environment and the current paths of the multiple agents.

3. The method according to claim 1, characterized in that: The step of establishing the observation space of the preset application environment according to the static obstacle information in the preset application environment and the current paths of the multiple agents includes: Obtaining a grid map of the preset application environment; According to the grid map and the static obstacle information, a first observation channel is constructed, wherein the first observation channel records obstacle indication information of each grid area in the grid map; the obstacle indication information is used to indicate whether each grid area has a static obstacle; According to the grid map and the current paths of the multiple agents, a plurality of second observation channels are respectively constructed, each of which records path indication information of each target grid area in the grid map, and the path indication information of each target grid area is used to indicate the time length for the corresponding agent to reach the target grid area; The observation space is constructed according to the first observation channel and the plurality of second observation channels.

4. The method according to claim 1, characterized in that: The adopting the target action strategy at the target position to re-plan the path of the target intelligent agent to obtain the target path of the target intelligent agent includes: Obtaining a preset action constraint corresponding to the target action strategy, wherein the preset action constraint is used to indicate that the target agent is prohibited from occupying a position within a preset time step; The preset action constraints are adopted to re-plan the path of the target intelligent agent to obtain the target path of the target intelligent agent.

5. The method according to claim 1, characterized in that Before the target action strategy of the target agent at the target position is obtained by processing the target neural network model according to the identification of each target agent corresponding to the observation space and the at least two current paths, the method further includes: Using an initial neural network model, a training sample set is constructed according to the observation space and the identifiers of each target intelligent agent corresponding to the at least two current paths; wherein the training sample set includes: multiple groups of training samples; The initial neural network model is trained according to the training sample set to obtain the target neural network model.

6. The method according to claim 5, characterized in that The initial neural network model is used to construct a training sample set according to the observation space and the identifiers of each target agent corresponding to the at least two current paths, including: Using the initial neural network model, according to the observation space and the identifier of the target intelligent agent, determine a first action strategy and a first state of the target intelligent agent; Determining a second state of the target agent according to the first action strategy and the observation space; According to the first state and the second state, a preset reward function is used to determine a first reward value; Constructing a set of first training samples according to the first reward value, the first state, the second state and the first action strategy; The initial neural network model is used to determine the second action strategy of the target agent according to the first training sample, and a set of second training samples is constructed until a preset number of training sample construction times is reached to obtain the training sample set.

7. The method according to claim 6, characterized in that The step of training the initial neural network model according to the training sample set to obtain the target neural network model includes: According to the training sample set, the initial neural network model is trained to calculate the loss function value; If the change in the loss function value does not satisfy the preset stop iteration condition, the initial neural network model continues to be trained until the change in the loss function value satisfies the preset stop iteration condition, or the number of training times for the initial neural network model reaches the preset number of iterations, the training of the initial neural network model is stopped, and the neural network model at the time of stopping training is determined to be the target neural network model.

8. A multi-agent path planning device, characterized in that: include: A planning module, used to perform path planning for multiple agents in a preset application environment to obtain multiple current paths, where the multiple current paths correspond to the multiple agents; A detection module, used for performing conflict detection on the multiple current paths; An establishing module, configured to establish an observation space of the preset application environment according to the current paths of the plurality of agents in the preset application environment if at least two current paths conflict with each other; A processing module, configured to obtain a target action strategy of the target agent at a target position by using a target neural network model to process the target agent according to the identifications of the target agents corresponding to the observation space and the at least two current paths; The planning module is further used to adopt the target action strategy at the target position to re-plan the path of the target intelligent agent to obtain the target path of the target intelligent agent; The detection module is also used to re-detect the conflict between the current paths of other intelligent agents and the target path of the target intelligent agent until there is no conflict between the paths of all intelligent agents.

9. A computer device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the computer device is running, the processor and the storage medium communicate through the bus, and the processor executes the program instructions to perform the steps of the multi-agent path planning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes the steps of the multi-agent path planning method as described in any one of claims 1 to 7.

Images