Dynamic multi-agent path planning method and system
Through the combination of multi-head dot-generating attention mechanism and greedy algorithm, the problems of poor scalability and frequent collisions in dynamic multi-agent path planning are solved, efficient path planning and coordination are achieved, and computing resource consumption and agent collision are reduced. frequency.
Patent Information
- Application Number
- CN202410889500.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-22
- Filing Date
- 2024-07-04
- Publication Date
- 2024-12-03
AI Technical Summary
The existing multi-agent path planning method has poor scalability in dynamic scenarios. It requires re-planning the routes of all agents to consume a lot of computing resources and time. In the dynamic environment, the model performance decreases, the success rate decreases, and the collision of agents increases.
The multi-head dot product attention mechanism is used to obtain the communication information of the agent, and the pre-trained path planning model is solved with the greedy algorithm to obtain the optimal planning path, and optimize the path planning through anti-collision auxiliary information to reduce the path planning between the agent and obstacles. collision.
It improves the coordination between agents and the efficiency of path planning in dynamic scenarios, reduces the number of collisions between agents and obstacles, maintains efficient path planning performance, and can operate stably even if the number of agents increases.
Smart Images

Figure CN119063723A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multi-agent path planning, and in particular to a dynamic multi-agent path planning method and system. Background Art
[0002] At present, the multi-agent path planning problem (MAPF) aims to find a joint path from the starting position to the target position for multiple agents in the map, and all agents make an action at the same time in each time step. At present, the classic MAPF problem is not enough to cover many real-world applications, such as warehouse management, autonomous driving, etc., which increasingly involve dynamic scenes. Therefore, it is necessary to study the MAPF (Dynamic MAPF, DMAPF) problem for dynamic scenes.
[0003] Traditional methods can solve the MAPF problem well, but there are problems such as poor system scalability and inability to find a solution. What's more serious is that when any scene changes, traditional methods need to re-plan the routes of all agents, which consumes a lot of computing resources and time. Therefore, it is not suitable to use traditional methods to solve the DMAPF problem. The multi-agent path planning problem in dynamic scenes brings huge challenges. Compared with static environments, dynamic environments are more difficult to predict, so the performance of the model in dynamic scenes will decline. In particular, as the number of agents increases, the success rate of the model in dynamic scenes decreases significantly, the required time step increases significantly, and the collision of agents increases significantly. Summary of the invention
[0004] In order to solve the problem that in the existing multi-agent path planning solutions, when the scene changes, the routes of all agents need to be replanned, which results in a large amount of computing resources and time consumption, the present invention proposes a dynamic multi-agent path planning method, including:
[0005] Obtain environmental observation data for each agent in a dynamic scene;
[0006] Based on the environmental observation data of each of the intelligent agents, the communication information of each of the intelligent agents is obtained through a multi-head dot product attention mechanism;
[0007] Based on the communication information of each of the intelligent agents, a pre-trained path planning model is solved using a greedy algorithm to obtain the optimal planning path of each of the intelligent agents;
[0008] Based on the optimal planned path of each of the intelligent agents, path planning is performed for the dynamic scene.
[0009] Optionally, obtaining the communication information of each of the intelligent agents through a multi-head dot product attention mechanism based on the environmental observation data of each of the intelligent agents includes:
[0010] Calculating collision avoidance assistance information of each of the intelligent agents based on the environmental observation data of each of the intelligent agents;
[0011] According to the environmental observation data of each of the intelligent agents, encoding is performed through a preset encoding network to obtain encoding auxiliary information of each of the intelligent agents;
[0012] According to the environmental observation information of each of the intelligent agents, an intelligent agent set consisting of neighboring intelligent agents that communicate with each intelligent agent is obtained;
[0013] Based on the collision avoidance auxiliary information, coding auxiliary information and the agent set of each of the intelligent agents, a multi-head dot product attention mechanism is used to aggregate information to obtain the communication information of each of the intelligent agents.
[0014] Optionally, performing path planning for the dynamic scene based on the optimal planned path of each of the intelligent agents includes:
[0015] Path planning is performed for the dynamic scene according to the optimal planned path of each intelligent agent and the collision avoidance auxiliary information.
[0016] Optionally, the expression of the agent set consisting of neighboring agents that communicate with each agent is as follows:
[0017] C i ={j|I ij >Thor},j∈N i
[0018] In the formula,
[0019]
[0020] Among them, C i I represents the set of agents consisting of neighboring agents that communicate with agent i; ij represents the importance of agent i and neighbor agent j; Thor represents the communication threshold; N i represents the set of agents consisting of all neighboring agents of agent i; diff ij represents the root mean square difference information between agent i and its neighbor agent j; CKR ij represents the overlap of common knowledge between agent i and its neighbor agent j; TSCKR ij represents the second-order common knowledge richness between agent i and its neighbor agent j; CKCI i Represents the common knowledge cooperation index of agent i.
[0021] Optionally, the information aggregation is performed based on the collision avoidance auxiliary information, the coding auxiliary information and the set of intelligent agents using a multi-head dot product attention mechanism to obtain the communication information of each intelligent agent, including:
[0022] Obtaining coding information of each of the intelligent agents according to the collision avoidance auxiliary information and coding auxiliary information of each of the intelligent agents;
[0023] According to the coding information of each of the intelligent agents and the intelligent agent set consisting of the neighboring intelligent agents that communicate with each of the intelligent agents, the coding information of the neighboring intelligent agents of each of the intelligent agents is obtained;
[0024] Based on the encoding information of each of the agents and the encoding information of the neighboring agents of the corresponding agent, a first round of information aggregation is performed through a multi-head dot product attention mechanism to obtain the first round of aggregated information of each of the agents;
[0025] Based on the first round of aggregation information of each of the intelligent agents and the first round of aggregation information of the neighboring intelligent agents of the corresponding intelligent agent, a second round of information aggregation is performed through a multi-head dot product attention mechanism to obtain the communication information of each of the intelligent agents.
[0026] Optionally, based on the communication information of each of the intelligent agents, solving the pre-trained path planning model using a greedy algorithm to obtain the optimal planning path of each of the intelligent agents includes:
[0027] For each of the intelligent agents, input the communication information of the intelligent agent into the value network of the pre-trained path planning model to obtain the action value function value of the intelligent agent; based on the action value function value of the intelligent agent, through a greedy algorithm, select the action value with the largest action value function value of the intelligent agent as the optimal action of the intelligent agent;
[0028] The optimal action of each of the intelligent agents is used as the optimal planning path of the corresponding intelligent agent.
[0029] Optionally, the path planning model includes the following training process:
[0030] Based on the environmental observation information of each intelligent agent at a historical moment in the dynamic scene, the action value function value of each intelligent agent at the historical moment is obtained through a value network;
[0031] Based on the action value function value of each of the intelligent agents at the historical moment, using a greedy algorithm, obtaining the action value of each of the intelligent agents at the historical moment;
[0032] Obtaining a reward value at the historical moment according to the action value of each of the intelligent agents at the historical moment;
[0033] According to the reward value at the historical moment, the time difference error at the historical moment of each of the intelligent agents is calculated, and the loss function value is determined according to the difference between the time difference error at the historical moment of each of the intelligent agents and the action value function value;
[0034] According to the loss function value, taking the maximization of the expected discounted total return at the historical moment as the optimization goal, the parameters in each agent are updated by the gradient descent method until the training times are reached, and a path planning model is obtained.
[0035] Optionally, the expression corresponding to the value network is as follows:
[0036]
[0037] Among them, Q i (·) represents the value network function of agent i; V(·) represents the state value function; A0(·) represents the advantage function; o represents the environmental observation information at the current moment; a represents the action value at the current moment; α represents the first parameter value of the agent decision network; β represents the second parameter value of the agent decision network; represents the communication information of agent i at time t; A1 represents the action set of agent i; a′ represents the action value at the next moment.
[0038] Optionally, the expression corresponding to the loss function is as follows:
[0039] L(α, β) = E o,a,r,o′ [(y DQN -Q i (o, a; α, β)) 2 ]
[0040] In the formula,
[0041]
[0042] Where L(α, β) represents the loss function value with respect to parameters α and β; E o,a,r,o′ represents the expected value of the Markov decision sequence; o represents the environmental observation information at the current moment; o′ represents the environmental observation information at the next moment; a represents the action value at the current moment; r represents the reward value at the current moment; y DQN represents the time difference error; Q i (·) represents the value network function of agent i; represents the discount coefficient under the time series n0; r t+1 Represents the reward value at time t+1; t=0..., n0-1; Represents the environmental observation information at time t+n0; Represents the action value at time t+n0;
[0043] The expression corresponding to the optimization objective is as follows:
[0044]
[0045] Where Y represents the agent’s expected discounted total return; γ t represents the discount coefficient at time t; R t+1 Represents the reward value at time t+1.
[0046] Based on the same inventive concept, the present invention also provides a dynamic multi-agent path planning system, comprising:
[0047] Data acquisition module: used to obtain environmental observation data of each agent in a dynamic scene;
[0048] Information aggregation module: used to obtain the communication information of each intelligent agent through a multi-head dot product attention mechanism based on the environmental observation data of each intelligent agent;
[0049] Model solving module: used to solve the pre-trained path planning model based on the communication information of each intelligent agent using a greedy algorithm to obtain the optimal planning path of each intelligent agent;
[0050] Path planning module: used to plan the path of the dynamic scene based on the optimal planning path of each intelligent agent.
[0051] Optionally, the information aggregation module is specifically used to:
[0052] Calculating collision avoidance assistance information of each of the intelligent agents based on the environmental observation data of each of the intelligent agents;
[0053] According to the environmental observation data of each of the intelligent agents, encoding is performed through a preset encoding network to obtain encoding auxiliary information of each of the intelligent agents;
[0054] According to the environmental observation information of each of the intelligent agents, an intelligent agent set consisting of neighboring intelligent agents that communicate with each intelligent agent is obtained;
[0055] Based on the collision avoidance auxiliary information, coding auxiliary information and the agent set of each of the intelligent agents, a multi-head dot product attention mechanism is used to aggregate information to obtain the communication information of each of the intelligent agents.
[0056] Optionally, the path planning module is specifically used to:
[0057] Path planning is performed for the dynamic scene according to the optimal planned path of each intelligent agent and the collision avoidance auxiliary information.
[0058] Optionally, the expression of the agent set consisting of neighboring agents that exchange information with each agent in the information aggregation module is as follows:
[0059] C i ={j|I ij >Thor},j∈N i
[0060] In the formula,
[0061]
[0062] Among them, C i I represents the set of agents consisting of neighboring agents that communicate with agent i; ij represents the importance of agent i and neighbor agent j; Thor represents the communication threshold; N i represents the set of agents consisting of all neighboring agents of agent i; diff ij represents the root mean square difference information between agent i and its neighbor agent j; CKR ij represents the overlap of common knowledge between agent i and its neighbor agent j; TSCKR ij represents the second-order common knowledge richness between agent i and its neighbor agent j; CKCI i Represents the common knowledge cooperation index of agent i.
[0063] Optionally, the information aggregation module uses a multi-head dot product attention mechanism to aggregate information based on the collision avoidance auxiliary information, the coding auxiliary information and the set of intelligent agents to obtain the communication information of each intelligent agent, including:
[0064] Obtaining coding information of each of the intelligent agents according to the collision avoidance auxiliary information and coding auxiliary information of each of the intelligent agents;
[0065] According to the coding information of each of the intelligent agents and the intelligent agent set consisting of the neighboring intelligent agents that communicate with each of the intelligent agents, the coding information of the neighboring intelligent agents of each of the intelligent agents is obtained;
[0066] Based on the encoding information of each of the agents and the encoding information of the neighboring agents of the corresponding agent, a first round of information aggregation is performed through a multi-head dot product attention mechanism to obtain the first round of aggregated information of each of the agents;
[0067] Based on the first round of aggregation information of each of the intelligent agents and the first round of aggregation information of the neighboring intelligent agents of the corresponding intelligent agent, a second round of information aggregation is performed through a multi-head dot product attention mechanism to obtain the communication information of each of the intelligent agents.
[0068] Optionally, the model solving module is specifically used for:
[0069] For each of the intelligent agents, input the communication information of the intelligent agent into the value network of the pre-trained path planning model to obtain the action value function value of the intelligent agent; based on the action value function value of the intelligent agent, through a greedy algorithm, select the action value with the largest action value function value of the intelligent agent as the optimal action of the intelligent agent;
[0070] The optimal action of each of the intelligent agents is used as the optimal planning path of the corresponding intelligent agent.
[0071] Optionally, the path planning model in the model solving module includes the following training process:
[0072] Based on the environmental observation information of each intelligent agent at a historical moment in the dynamic scene, the action value function value of each intelligent agent at the historical moment is obtained through a value network;
[0073] Based on the action value function value of each of the intelligent agents at the historical moment, using a greedy algorithm, obtaining the action value of each of the intelligent agents at the historical moment;
[0074] Obtaining a reward value at the historical moment according to the action value of each of the intelligent agents at the historical moment;
[0075] According to the reward value at the historical moment, the time difference error at the historical moment of each of the intelligent agents is calculated, and the loss function value is determined according to the difference between the time difference error at the historical moment of each of the intelligent agents and the action value function value;
[0076] According to the loss function value, taking the maximization of the expected discounted total return at the historical moment as the optimization goal, the parameters in each agent are updated by the gradient descent method until the training times are reached, and a path planning model is obtained.
[0077] Optionally, the expression corresponding to the value network in the model solving module is as follows:
[0078]
[0079] Among them, Q i (·) represents the value network function of agent i; V(·) represents the state value function; A0(·) represents the advantage function; o represents the environmental observation information at the current moment; a represents the action value at the current moment; α represents the first parameter value of the agent decision network; β represents the second parameter value of the agent decision network; represents the communication information of agent i at time t; A1 represents the action set of agent i; a′ represents the action value at the next moment.
[0080] Optionally, the expression corresponding to the loss function in the model solving module is as follows:
[0081] L(α, β) = E o,a,r,o′ [(y DQN -Q i (o, a; α, β)) 2 ]
[0082] In the formula,
[0083]
[0084] Where, L(α, β) represents the loss function value with respect to parameters α and β; E o,a,r,o′ represents the expected value of the Markov decision sequence; o represents the environmental observation information at the current moment; o′ represents the environmental observation information at the next moment; a represents the action value at the current moment; r represents the reward value at the current moment; y DQN represents the time difference error; Q i (·) represents the value network function of agent i; represents the discount coefficient under the time series n0; r t+1 Represents the reward value at time t+1; t=0..., n0-1; Represents the environmental observation information at time t+n0; Represents the action value at time t+n0;
[0085] The expression corresponding to the optimization objective is as follows:
[0086]
[0087] Where Y represents the agent’s expected discounted total return; γ t represents the discount coefficient at time t; R t+1 Represents the reward value at time t+1.
[0088] Compared with the prior art, the present invention has the following beneficial effects:
[0089] The present invention provides a dynamic multi-agent path planning method and system, including: acquiring environmental observation data of each agent in a dynamic scene; based on the environmental observation data of each agent, obtaining communication information of each agent through a multi-head dot product attention mechanism; based on the communication information of each agent, solving a pre-trained path planning model using a greedy algorithm to obtain the optimal planning path of each agent; based on the optimal planning path of each agent, performing path planning for the dynamic scene; the present invention can achieve efficient communication between agents and improve the coordination between agents by aggregating the communication information of each agent through the environmental observation data of each agent based on a multi-head dot product attention mechanism; based on the communication information of each agent, solving the path planning model using a greedy algorithm to obtain the optimal path of each agent, and as the number of agents increases, the complexity of the path planning model in the dynamic scene, the required time step and the model success rate will not be affected, which is conducive to showing the optimal path planning effect in the dynamic scene. BRIEF DESCRIPTION OF THE DRAWINGS
[0090] Figure 1 A schematic diagram of a flow chart of a dynamic multi-agent path planning method provided by the present invention;
[0091] Figure 2 The overall framework process of a dynamic multi-agent path planning method provided by the specific embodiment of the present invention
[0092] Figure 3 An example diagram of an initial map in a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0093] Figure 4 An example diagram of a dynamic scene corresponding to the appearance of obstacles in a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0094] Figure 5 An example diagram of a dynamic scene corresponding to the disappearance of obstacles in a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0095] Figure 6 An example diagram of a dynamic scene corresponding to obstacle movement in a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0096] Figure 7 An example diagram of a dynamic scene corresponding to an agent and a task in a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0097] Figure 8An example diagram of a dynamic scene corresponding to deletion of agents and tasks in a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0098] Fig. 9 A schematic diagram of channel display of an agent in a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0099] Fig.10 An agent observation information diagram of a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0100] Fig.11 A critical area pseudo-collision design diagram of a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0101] Fig.12 A real collision design diagram of a critical area of a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0102] Fig.13 A flow chart of a course study of a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0103] Fig.14 A line graph showing the success rate results of an ablation experiment of a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0104] Fig.15 A line graph showing average step length results of an ablation experiment of a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0105] Fig.16 A line graph showing the number of collisions in an ablation experiment of a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0106] Fig.17 A line graph showing the number of collisions between agents in an ablation experiment of a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0107] Fig.18 A line graph showing the number of collisions between agents and obstacles in an ablation experiment of a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0108] Fig.19 An example visualization diagram corresponding to a moment with a step length of 7 in a dynamic multi-agent path planning method provided in a specific embodiment of the present invention;
[0109] Fig. 20An example visualization diagram corresponding to the moment of step length 7-9 in a dynamic multi-agent path planning method provided by a specific embodiment of the present invention;
[0110] Fig.21 An example visualization diagram corresponding to a moment when the step length is 9 in a dynamic multi-agent path planning method provided in a specific embodiment of the present invention;
[0111] Fig. 22 An example visualization diagram corresponding to a moment when the step length is 11 in a dynamic multi-agent path planning method provided in a specific embodiment of the present invention;
[0112] Fig.23 An example visualization diagram corresponding to a moment when the step length is 12 in a dynamic multi-agent path planning method provided in a specific embodiment of the present invention;
[0113] Fig.24 An example visualization diagram corresponding to a moment when the step length is 13 in a dynamic multi-agent path planning method provided in a specific embodiment of the present invention;
[0114] Fig.25 A schematic diagram of the structural composition of a dynamic multi-agent path planning system provided by the present invention. DETAILED DESCRIPTION
[0115] The present invention proposes a dynamic multi-agent path planning method and system, and the specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings.
[0116] Embodiment 1:
[0117] A dynamic multi-agent path planning method, the flow chart is as follows Figure 1 As shown, including:
[0118] Step 1: Obtain environmental observation data of each agent in a dynamic scene;
[0119] Step 2: Based on the environmental observation data of each agent, the communication information of each agent is obtained through the multi-head dot product attention mechanism;
[0120] Step 3: Based on the communication information of each agent, the pre-trained path planning model is solved using a greedy algorithm to obtain the optimal planning path for each agent;
[0121] Step 4: Based on the optimal planning path of each agent, path planning is performed for the dynamic scene.
[0122] In general, the above dynamic scenarios are based on diverse and rich scenarios in real life. For example, the above environmental observation information may include the limited observation values and heuristic channels of the agent about neighboring agents and obstacles. For example, in the scenario of autonomous driving, each agent may include vehicles on the road, traffic signals, emergency response facilities, etc.; in the scenario of warehouse management, each agent may include automatic guided vehicles, autonomous mobile robots, intelligent scheduling systems, etc.; in the scenario of drone delivery, each agent may include drone carriers, task scheduling facilities, human-computer interaction systems, etc.
[0123] In one implementation, the process of obtaining the communication information of each agent based on the environmental observation data of each agent through a multi-head dot product attention mechanism in step 2 may include:
[0124] Calculate the collision avoidance assistance information of each intelligent agent based on the environmental observation data of each intelligent agent;
[0125] According to the environmental observation data of each intelligent agent, the encoding is performed through a preset encoding network to obtain the encoding auxiliary information of each intelligent agent;
[0126] According to the environmental observation information of each agent, an agent set consisting of neighboring agents that communicate with each agent is obtained;
[0127] Based on the collision avoidance auxiliary information, coding auxiliary information and agent collection of each agent, the multi-head dot product attention mechanism is used to aggregate information and obtain the communication information of each agent. In this implementation, the multi-head dot product attention mechanism is used to facilitate efficient information exchange between multiple agents.
[0128] For example, the expression of the agent set consisting of neighboring agents that communicate with each agent is as follows:
[0129] C i ={j|I ij >Thor},j∈N i
[0130] In the formula,
[0131]
[0132] Among them, C i I represents the set of agents consisting of neighboring agents that communicate with agent i; ij Indicates the importance of agent i and neighbor agent j, which is composed of difference information and common knowledge feature information; Thor represents the communication threshold, preferably, Thor = 0.6; N i represents the set of agents consisting of all neighboring agents of agent i; diffij represents the root mean square difference information between agent i and its neighbor agent j; CKR ij represents the overlap of common knowledge between agent i and its neighbor agent j; TSCKR ij represents the second-order common knowledge richness between agent i and its neighbor agent j; CKCI i represents the common knowledge cooperation index of agent i; the above CKR ij TSCKR ij and CKCI i They are all common knowledge feature information, which are calculated through the agent adjacency matrix. They can be used to measure the closeness between agents. Specifically, the Common Knowledge Richness (CKR) reflects the overlap of common knowledge between agents. For agent i, its CKR with neighbor agent j is ij The higher the value, the higher the overlap of their observation information, and the higher the importance of neighbor agent j. The calculation formula can be expressed as The second-stage common knowledge richness (TSCKR) further distinguishes the richness of different neighbors. For agent i, the TSCKR of neighbor agent j is ij The calculation formula can be expressed as The Common Knowledge Cooperation Index (CKCI) globally reflects the degree of common knowledge between the agent itself and its neighboring agents. The calculation formula of the common knowledge cooperation index of agent i can be expressed as CKCI i The larger the value, the more the agent needs to cooperate with nearby agents. On the contrary, it should pay more attention to its own state. j represents the agent set consisting of all neighbor agents of neighbor agent j; k represents the common agent between the agent set consisting of all neighbor agents of agent i and the agent set consisting of all neighbor agents of neighbor agent j; CKR ik Represents the common knowledge richness between agent i and agent k.
[0133] For example, the root mean square difference information diff between the above agent i and its neighbor agent j ij The calculation process may include the following:
[0134] Agent i removes neighbor agent j (j∈N i) can be expressed as o i-j , the environment observation value o of agent i i And the environment observation information o after agent i removes neighbor agent j i-j The information is sent to the preset encoding network and value network in sequence, and the obtained information can be expressed as Finally, the root mean square difference information calculated can be expressed as: Among them, a represents the action value at the current moment; A 0 Represents an action set; Encoder(o i ) represents the environmental observation information o of agent i i Corresponding coding auxiliary information; Encoder(o i-j ) represents the environmental observation information o after agent i removes neighbor agent j i-j corresponding coding auxiliary information; represents the action value a, and the environment observation information o of agent i i The corresponding encoding auxiliary information is Encoder(o i ) when the action value function is Indicates that the action value is a, and the environment observation information o after agent i removes neighbor agent j i-j The corresponding encoding auxiliary information is Encoder(o i-j )’s action value function value; MSE(·) represents the root mean square function;
[0135] In this implementation, the process of aggregating information based on the collision avoidance auxiliary information, the coding auxiliary information and the agent set of each agent using the multi-head dot product attention mechanism to obtain the communication information of each agent may include:
[0136] According to the collision avoidance auxiliary information and the coding auxiliary information of each intelligent agent, the coding information of each intelligent agent is obtained;
[0137] According to the coding information of each agent and the agent set composed of neighboring agents that communicate with each agent, the coding information of the neighboring agents of each agent is obtained;
[0138] Based on the encoding information of each agent and the encoding information of the corresponding agent's neighboring agents, the first round of information aggregation is performed through the multi-head dot product attention mechanism to obtain the first round of aggregated information of each agent;
[0139] Based on the first round of aggregated information of each agent and the first round of aggregated information of the corresponding agent's neighboring agents, the second round of information aggregation is performed through the multi-head dot product attention mechanism to obtain the communication information of each agent. The two rounds of information aggregation through the multi-head dot product attention mechanism is conducive to efficient information exchange between multiple agents;
[0140] In this implementation, in a multi-agent path planning environment, each agent observes the current environment and obtains environmental observation information;
[0141] The environmental observation information is used to calculate the collision avoidance auxiliary information on the one hand, and the coding auxiliary information is obtained through coding network coding on the other hand. The collision avoidance auxiliary information and the coding auxiliary information are used as the input of the information aggregation port together;
[0142] According to the neighboring agent set (for example, the agent set composed of the neighboring agents of agent i) i denoted by (for example, using C i Represents), where agent i is all agents in the multi-agent path planning environment;
[0143] Use the multi-head dot product attention mechanism to realize the interaction between agent i and the selected neighbor agent C i Aggregate the information of agent i to obtain the communication information of agent i (for example, using m i The present invention realizes efficient communication between intelligent agents by improving the communication protocol, thereby improving the coordination between intelligent agents;
[0144] In this implementation, when information is aggregated through two rounds of multi-head dot product attention mechanism, each independent attention head h∈H will be aggregated through the network parameters The fully connected layer maps the input information into query vector (Query), key vector (Key) and value vector (Value), and then outputs the information learned by different attention heads in series. Finally, the output information is linearly converted into the expected dimension through the neural network σ; where H represents the set of attention heads; Represents the three network parameters of the fully connected layer when the attention head is h;
[0145] In the first round of information aggregation, the input is the coded information of agent i (for example, using e i ) and the encoded information of neighboring agents (e.g., using e j Indicates that j∈C i ), the output of agent i is the first round of aggregate information (for example, using The calculation formula is as follows:
[0146]
[0147] In the second round of information aggregation, the input is the first round of aggregated information of agent i The first round of aggregation information with neighboring agents Output is communication information (for example, using m i The calculation formula is as follows:
[0148]
[0149] Where σ(·) represents the neural network; C +i ={C i , i}, is used to represent the set of neighboring agents with which agent i exchanges information; It is used to represent the correlation coefficient between agent i and neighbor agent j about attention head h; τ represents the scaling factor; concatenate[·] represents the connection information; represents the three network parameters corresponding to the attention head h of the fully connected layer; H represents the set of attention heads; e i represents the encoding information of agent i; e j represents the encoded information of neighbor agent j; e k Represents the encoded information of agent k; k represents the common agent between the agent set consisting of all neighbor agents of agent i and the agent set consisting of all neighbor agents of neighbor agent j; T represents transposition.
[0150] In this implementation, the above collision avoidance assistance information is obtained using the agent's partial observations of obstacles, and the size of this information is consistent with the size of the action space. When the agent performs an action, the collision avoidance assistance information will include a prediction of whether the action will be within the current obstacle observation value (for example, the obstacle observation value of agent i is If there is no collision, the value is set to 1, otherwise it is 0;
[0151] In one implementation, in step 3 above, based on the communication information of each agent, the process of solving the pre-trained path planning model using a greedy algorithm to obtain the optimal planning path of each agent may include:
[0152] For each agent, the communication information of the agent is input into the value network of the pre-trained path planning model to obtain the action value function value of the agent; based on the action value function value of the agent, the action value with the largest action value function value of the agent is selected as the optimal action of the agent through a greedy algorithm;
[0153] The optimal action of each agent is used as the optimal planning path of the corresponding agent;
[0154] In this implementation, the above-mentioned path planning model includes the following training process:
[0155] Based on the historical environmental observation information of each agent in the dynamic scene (for example, the environmental observation information at time t is o t ), and through the forward propagation of the value network, the action value function value of each agent at the historical moment is obtained (for example, the action value function value of agent i at time t is express);
[0156] Based on the action value function value of each agent at the historical moment, the greedy algorithm is used to obtain the action value of each agent at the historical moment (for example, the action value at time t is a t express);
[0157] According to the action value of each agent at the historical moment, the reward value at the historical moment is obtained (for example, the reward value at time t is r t By updating the state of the environment, the agent obtains the environmental observation information o at time t+1 t+1 ;
[0158] According to the reward value at the historical moment, the time difference error of each agent at the historical moment is calculated, and the loss function value is determined according to the difference between the time difference error at the historical moment of each agent and the action value function value;
[0159] According to the loss function value, the optimization goal is to maximize the expected discounted total return at the historical moment, update the parameters in each agent by the gradient descent method, and repeat the above training process until the number of training times is reached to obtain a path planning model; the present invention uses a deep reinforcement learning method to train the agent network based on feedback information from the interaction with the environment, thereby defining and modeling the multi-agent path planning problem in a dynamic scene, and uses the trained path planning model to perform path planning in the application environment;
[0160] For example, the above reward value is assigned as follows:
[0161] If the agent moves or stays at a non-target point, a reward value of -0.075 is given;
[0162] If the agent reaches the target point, a reward value of +3.0 is given;
[0163] If the agent stays at the target point, a reward value of 0 is given;
[0164] If the agent collides with other agents or obstacles, a reward value of -0.7 is given;
[0165] If the agent has a pseudo-collision with an obstacle, a reward value of -0.1 is given;
[0166] Among them, the pseudo collision between the intelligent agent and the obstacle means that the critical area of the intelligent agent and the obstacle overlaps.
[0167] In this implementation, a standard multi-agent path planning (MAPF) problem is defined, and the scenario is a two-dimensional four-neighborhood grid map modeled according to the application environment;
[0168] Corresponding to the dynamic scenes existing in reality, the dynamic MAPF (DMAPF) problem is further expanded and formally defined based on the standard MAPF problem, where the dynamic changes in the scene include the dynamic changes of obstacles and the dynamic changes of the agent's tasks.
[0169] The problem is modeled as a partially observable Markov decision process and the DMAPF problem is solved using deep reinforcement learning methods:
[0170] The above formal definition of the DMAPF problem is expressed as a tuple Where A represents the agent set A = {1, ..., n}, n is the number of agents, G = (V, E) is an undirected graph, V is the set of nodes in the graph G, v∈V represents the position that the agent can occupy, edge E is the set of traversable paths, e = (v i , v j )∈E means that the agent can i and v j Each agent i has a unique starting position s i ∈S∈V and the target position g i ∈T0∈V, S represents the starting position set, T0 represents the target position set, O represents the obstacle position, represents the maximum step length, the non-negative integer t represents the current time, and satisfies Non-empty subset represents the agents that leave the environment at time t, the non-empty subset A↓ (A↓≠A) represents the agents that join the environment at time t (the remaining A\A↑), the set S↓ represents the starting position of the new agent A↓, the set T↓ represents the target position of the new agent A↓, the set O↑ represents the obstacles that join the environment at time t, and the set O↓ represents the obstacles that leave the environment at time t;
[0171] For example, the above problem is modeled as a partially observable Markov decision process represented as a tuple Where A is a group of agents, represented by {1, ..., n}, n is the number of agents, S 0 is a finite set of states, is the finite set of actions available to agent i, is a joint action set, Ob i is the finite observation set of agent i, Ob = Ob1 × ... × Ob n is the joint observation set, P is the state transition function of the environment, satisfying s represents the state value at the current moment, a represents the action value at the current moment, and s′ represents the state value at the next moment. represents the state set at time t, represents the state set at time t-1, represents the action set at time t-1, R i represents the reward value of agent i, R i Satisfaction: S 0 ×A 0 →R, R represents the reward value at the current moment; γ is the discount factor. Multiple agents simultaneously select an action and obtain reward values and observation values from the environment. The optimization goal is to maximize the expected discounted total return of the multi-agent system during the game. in, represents the maximum step size, γ t represents the discount factor at time t, R t+1 Represents the reward value at time t+1;
[0172] For example, the expression corresponding to the above value network is as follows:
[0173]
[0174] Among them, Q i (·) represents the action value function of agent i; V(·) represents the state value function; A0(·) represents the advantage function; o represents the environmental observation information at the current moment; a represents the action value at the current moment; α represents the first parameter value of the agent decision network; β represents the second parameter value of the agent decision network; represents the communication information of agent i at time t; A1 represents the action set of agent i; a′ represents the action value at the next moment.
[0175] For example, the expression corresponding to the above loss function is as follows:
[0176] L(α, β) = E o,a,r,o′ [(y DQN -Q i (o, a; α, β)) 2 ]
[0177] In the formula,
[0178]
[0179] Where, L(α, β) represents the loss function value with respect to parameters α and β; E o,a,r,o′ represents the expected value of the Markov decision sequence; o represents the environmental observation information at the current moment; o′ represents the environmental observation information at the next moment; a represents the action value at the current moment; r represents the reward value at the current moment; y DQN represents the time difference error; Q i (·) represents the value network function of agent i; represents the discount coefficient under the time series n0; r t+1 Represents the reward value at time t+1; t=0..., n0-1; Represents the environmental observation information at time t+n0; Represents the action value at time t+n0;
[0180] The expression corresponding to the optimization objective is as follows:
[0181]
[0182] Where Y represents the agent’s expected discounted total return; γ t represents the discount coefficient at time t; R t+1 Represents the reward value at time t+1;
[0183] When training the path planning model, the present invention uses the deep reinforcement learning method to train through the feedback information of the interaction between the intelligent agent and the environment, which is conducive to realizing the self-iterative optimization of the intelligent agent strategy and can find the optimal or approximately optimal path under various complex and uncertain conditions, greatly improving the efficiency and performance of path planning.
[0184] In one implementation, the process of performing path planning for a dynamic scene based on the optimal planned path of each agent in step 4 may include:
[0185] Path planning is performed for dynamic scenes based on the optimal planning path and collision avoidance assistance information of each agent.
[0186] In this implementation, the optimal planned path is obtained based on the information after the intelligent agents communicate, and the action value function value is obtained through the value network of the path planning model, and the final action decision is made in combination with the anti-collision auxiliary decision rule based on the anti-collision auxiliary information to realize the path planning for dynamic scenes. For example, the above-mentioned anti-collision auxiliary decision rule can be: according to the optimal planned path selected by the action value function value of the intelligent agent, when it is very likely to collide with an obstacle through the analysis of the anti-collision auxiliary information updated in real time, the intelligent agent is allowed to remain stationary, and when the anti-collision auxiliary information shows that continuing to execute the optimal planned path will not collide with the obstacle, the optimal path is continued to be executed. The above method can realize the optimal planning path of all intelligent agents in dynamic scenes. When the dynamic scene changes, there is no need to re-plan the routes of all intelligent agents, which is conducive to saving computing resources and time. In addition, by combining collision avoidance auxiliary information, it is conducive to improving the vigilance of intelligent agents to nearby obstacles. By using the critical area, the intelligent agent is given a pseudo-collision warning to let it beware of obstacles that are slightly farther away, thereby effectively reducing the number of collisions between the intelligent agent and the obstacles, thereby alleviating the problem of a large number of collisions in complex and changeable dynamic scenes, which is conducive to improving the system cooperation performance, reducing collisions between intelligent agents, and achieving the optimal path planning effect in dynamic scenes.
[0187] In view of the problems that existing multi-agent path planning methods have poor system scalability and cannot find solutions in dynamic scenarios, and when the scenario changes, the routes of all agents need to be re-planned, which results in a large amount of computing resources and time consumption, the present invention provides a dynamic multi-agent path planning method, which realizes efficient communication between multiple agents based on a multi-attention mechanism and improves the coordination between agents. In the process of path planning, the path planning model constructed based on a deep reinforcement learning method and anti-collision auxiliary information are combined, which is beneficial for each agent to reduce the number of collisions between the agent and obstacles while executing the optimal path, thereby alleviating the problem of a large number of collisions in complex and changeable dynamic scenarios, and demonstrating excellent path planning effects in dynamic scenarios.
[0188] Embodiment 2:
[0189] The overall process of a dynamic multi-agent path planning method provided by the present invention is described through a specific embodiment. The flow chart is as follows: Figure 2 As shown, including:
[0190] Step S101: defining and modeling a dynamic multi-agent path planning problem;
[0191] Step S102: In a multi-agent path planning environment, each agent in the multi-agent system observes the current environment, including observation channels of neighboring agents and observation channels of obstacles, and simultaneously calculates a heuristic channel;
[0192] Step S103: In a multi-agent path planning environment, the agents have their own neighboring agents, and each agent achieves information transmission and aggregation between the agent and its neighboring agents through selective communication;
[0193] Step S104: In a multi-agent path planning environment, each agent has an independent agent decision network, and the agent value network uses the Dueling DQN algorithm based on the value function to train the path planning model. During training, the distributed priority experience replay method Ape-X is used, and the training is carried out step by step through the course learning method;
[0194] Step S105: Perform path planning for the agent using the trained path planning model in the application environment.
[0195] Specifically, in the above step S101, the dynamic multi-agent path planning problem is defined as a tuple Where A represents the agent set A = {1, ..., n}, n is the number of agents, G = (V, E) is an undirected graph, V is the set of nodes in the graph G, v∈V represents the position that the agent can occupy, edge E is the set of traversable paths, e = (v i , v j )∈E means that the agent can i and v j Each agent i has a unique starting position s i ∈S∈V and the target position g i ∈T0∈V, S represents the starting position set, T0 represents the target position set, O represents the obstacle position, represents the maximum step length, the non-negative integer t represents the current time, and satisfies Non-empty subset represents the agent leaving the environment at time t, the non-empty subset A↓(A↓≠A) represents the agent joining the environment at time t (the remaining A\A↑), the set S↓ represents the starting position of the new agent A↓, the set T↓ represents the target position of the new agent A↓, the set O↑ represents the obstacles joining the environment at time t, and the set O↓ represents the obstacles leaving the environment at time t.
[0196] The dynamic changes in the above-mentioned dynamic multi-agent path planning problem include dynamic changes of obstacles and dynamic changes of agent obstacles, which fully considers the situations that may occur in real applications. Figure 3 An initial map is shown. Figure 4-Figure 8 There are five dynamic examples. In warehouse management, goods may suddenly fall to the ground during transportation, forming sudden obstacles, such as Figure 4As shown in ; afterwards, there is a situation where the staff will pick up the goods and clear the obstacles, which corresponds to the situation where the obstacles suddenly disappear, such as Figure 5 As shown in ; all workers in the work area can be regarded as moving obstacles, such as Figure 6 As shown in ; at the same time, a new agent and target task may be added in the process of multi-agent task completion, such as Figure 7 As shown in , it is also possible that one or some agents and their tasks are cancelled by the administrator, such as Figure 8 as shown in .
[0197] Specifically, in step S101, the solution process of the dynamic multi-agent path planning problem can be modeled as a partially observable Markov decision process, using the tuple Where A is a group of agents, represented by {1, ..., n}, n is the number of agents, S 0 is a finite set of states, is the finite set of actions available to agent i, is a joint action set, Ob i is the finite observation set of agent i, Ob = Ob1 × ... × Ob n is the joint observation set, P is the state transition function of the environment, satisfying s represents the state value at the current moment, a represents the action value at the current moment, and s′ represents the state value at the next moment. represents the state set at time t, represents the state set at time t-1, represents the action set at time t-1, R i represents the reward value of agent i, R i Satisfaction: S 0 ×A 0 →R, R represents the reward value at the current moment; γ is the discount factor. Multiple agents simultaneously select an action and obtain reward values and observation values from the environment. The optimization goal is to maximize the expected discounted total return of the multi-agent system during the game. in, represents the maximum step size, γ t represents the discount factor at time t, R t+1 Represents the reward value at time t+1.
[0198] In the above step S102, in the multi-agent path planning environment, each agent in the multi-agent system observes the current environment, including the observation channel of the neighboring agent and the observation channel of the obstacle, and calculates the heuristic channel. After that, on the one hand, the observation channel of the obstacle is added to the collision avoidance auxiliary information, and on the other hand, all the channels are input into the agent value network together;
[0199] Specifically, the multi-agent system is a fully cooperative relationship under a fully decentralized multi-agent architecture.
[0200] Specifically, the anti-collision auxiliary information is obtained using the agent's partial observations of obstacles, and the size of this information is consistent with the size of the action space. When the agent performs an action, the anti-collision auxiliary information will predict in advance whether the action will collide with the obstacle under the current obstacle observation value. If there is no collision, the value is set to 1, and if there is a collision, it is 0.
[0201] Specifically, the input of the agent network includes 6 channels, such as Fig. 9 As shown in the figure, the observation channel of the agent about the neighboring agent (for example, using ) and the observation channel of the obstacle (for example, using ) are binary matrices of the size of the agent's field of view (FoV), which are used to identify neighboring agents and obstacles respectively. A value of 1 in the matrix indicates that the object exists at this location; the heuristic channel It includes four channels: up, down, left, and right. Each channel is a binary matrix of the size of FoV. When a corresponding action (including up, down, left, and right) is made at a certain position in the matrix to get closer to the target point without colliding with obstacles, the position is marked as 1.
[0202] In the above step S103, in the multi-agent path planning environment, the agents have their own neighbors, and each agent realizes information transmission and aggregation between the agent and its neighbors through selective communication;
[0203] Specifically, agents will transmit and aggregate information, and efficient information exchange can enable the multi-agent system to have better collaboration capabilities. Fig.10 As shown in the figure, at time t, the environmental observation information of agent i and neighbor agent j are After passing through the coding network and connecting with the collision avoidance auxiliary information, the coded information is After that, the agent sends a request to the other agents it wants to communicate with, and the receiving agents reply with a message, which contains two rounds of information aggregation. Each round of information aggregation consists of a multi-head dot-product attention network and a gated recurrent unit GRU to encode information. Information is transmitted and aggregated through the attention mechanism to obtain the information after output communication Among them, they choose neighbor agents to communicate with Provided during the information aggregation process.
[0204] Furthermore, the preset encoding network includes 4 convolutional layers and GRU, wherein the convolution kernel size of the first convolutional neural network is 3, the number of channels is 64, and the step size is 1; the convolution kernel size of the second convolutional neural network is 3, the number of channels is 128, and the step size is 1; the convolution kernel size of the third convolutional neural network is 3, the number of channels is 192, and the step size is 1; the convolution kernel size of the fourth convolutional neural network is 3, the number of channels is 252, and the step size is 1, and each layer of the convolutional neural network is followed by a LeakyReLU activation function.
[0205] Furthermore, the collision avoidance auxiliary information will be connected with the coding auxiliary information and then sent to the communication module and DuelingDQN module, which can alleviate the problem of the neural network ignoring obstacle observation information.
[0206] Furthermore, when information is aggregated, taking the perspective of agent i at time t as an example, it includes: when the first round of information aggregation is performed, the input is the encoded information ei, and the output is the first round of aggregated information
[0207]
[0208] Similarly, in the second round of information aggregation, the input is the first round of aggregated information of agent i The first round of aggregation information with neighboring agents The output is the communication information m i , the calculation formula is:
[0209]
[0210] Where σ(·) represents the neural network; C +i ={C i , i}, is used to represent the set of neighboring agents with which agent i exchanges information; It is used to represent the correlation coefficient between agent i and neighbor agent j about attention head h; τ represents the scaling factor; concatenate[·] represents the connection information; represents the three network parameters corresponding to the attention head h of the fully connected layer; H represents the set of attention heads; e i represents the encoding information of agent i; e j represents the encoded information of neighbor agent j; e k Represents the encoded information of agent k; k represents the common agent between the agent set consisting of all neighbor agents of agent i and the agent set consisting of all neighbor agents of neighbor agent j; T represents transposition.
[0211] Furthermore, selective communication is achieved through the communication selection module, which aims to provide a set of neighboring agents that need to communicate. The communication selection module will select neighboring agents that have an impact on the agent and have close relationships with it through calculation, which will improve the agent's ability to obtain effective neighboring agent information while discarding invalid communication.
[0212] The set of neighbor agents C selected by the communication selection module i It can be expressed as:
[0213] C i ={j|I ij >Thor},j∈N i
[0214] In the formula,
[0215]
[0216] Among them, C i represents the set of agents consisting of neighboring agents that communicate with agent i; Iij represents the importance of agent i and neighboring agent j, which is composed of difference information and common knowledge feature information; Thor represents the communication threshold, preferably, Thor = 0.6; Ni represents the set of agents consisting of all neighboring agents of agent i; diff ij represents the root mean square difference information between agent i and its neighbor agent j; CKR ij represents the overlap of common knowledge between agent i and its neighbor agent j; TSCKR ij represents the second-order common knowledge richness between agent i and its neighbor agent j; CKCI i represents the common knowledge cooperation index of agent i; the above CKR ij TSCKR ij and CKCI i They are all common knowledge feature information, which are calculated through the agent adjacency matrix. They can be used to measure the closeness between agents. Specifically, the Common Knowledge Richness (CKR) reflects the overlap of common knowledge between agents. For agent i, its CKR with neighbor agent j is i The higher j is, the higher the overlap of their observation information is, and the more important the neighbor agent j is. The calculation formula can be expressed as The second-stage common knowledge richness (TSCKR) further distinguishes the richness of different neighbors. For agent i, the TSCKR of neighbor agent j is ij The calculation formula can be expressed as The Common Knowledge Cooperation Index (CKCI) globally reflects the degree of common knowledge between the agent itself and its neighboring agents. The calculation formula of the common knowledge cooperation index of agent i can be expressed as CKCI i The larger the value, the more the agent needs to cooperate with nearby agents. On the contrary, it should pay more attention to its own state. j represents the agent set consisting of all neighbor agents of neighbor agent j; k represents the common agent between the agent set consisting of all neighbor agents of agent i and the agent set consisting of all neighbor agents of neighbor agent j; CKR ik Indicates the common knowledge richness between agent i and agent k In the above step S104, in the multi-agent path planning environment, each agent has an independent agent decision network, and the agent action value network uses the Dueling DQN algorithm based on the value function to train the agent network. During training, the distributed priority experience replay method Ape-X is used, and the training is carried out step by step through the course learning method.
[0217] Specifically, in step S104, taking agent i and neighbor agent j at time t as an example, the agent action value network Dueling DQN module inputs the communication information as respectively expressed as The output action value function values are expressed as Finally, the agent makes an action decision based on the action value function value and the collision avoidance auxiliary information. The action decision is based on the action selected by the greedy algorithm (i.e., the action with the largest Q value). The collision avoidance auxiliary information is added to the action decision. The rule is that if the action selected according to the action value function Q is very likely to hit an obstacle in the collision avoidance auxiliary module analysis, the agent is kept still. This focuses on nearby obstacles with a unit distance of 1, which can increase the agent's vigilance against obstacles. At the same time, this links the obstacle observation value with the action, which can promote the neural network to learn to link the punishment with the obstacle information when it is punished for colliding with the obstacle.
[0218] Specifically, in step S104, the process of Dueling DQN algorithm training the agent value network is as follows:
[0219] The agent observes the environment information o at time t t , forward propagation obtains communication information m through the encoding network and the communication module t , and take action a based on the action value function value through the ∈-greedy algorithmt ;
[0220] Get the feedback reward r from the environment t , the environment updates the state, and the agent obtains the environmental observation information o at time t+1 t+1 ;
[0221] The loss function value of the agent network is obtained based on the action value function and the temporal difference method, and the network parameters are updated by the gradient descent method;
[0222] Cycle the above training process;
[0223] Furthermore, the expression corresponding to the action value function of agent i is as follows:
[0224]
[0225] Among them, Q i (·) represents the action value function of agent i; V(·) represents the state value function; A0(·) represents the advantage function; o represents the environmental observation information at the current moment; a represents the action value at the current moment; α represents the first parameter value of the agent decision network; β represents the second parameter value of the agent decision network; represents the communication information of agent i at time t; A1 represents the action set of agent i; a′ represents the action value at the next moment;
[0226] Furthermore, the loss function is expressed as:
[0227] L(α, β) = E o,a,r,o′ [(y DQN -Q i (o, a; α, β)) 2 ]
[0228] In the formula,
[0229]
[0230] Where L(α, β) represents the loss function value with respect to parameters α and β; E o,a,r,o′ represents the expected value of the Markov decision sequence; o represents the environmental observation information at the current moment; o′ represents the environmental observation information at the next moment; a represents the action value at the current moment; r represents the reward value at the current moment; y DQN represents the time difference error; Q i (·) represents the value network function of agent i; represents the discount coefficient under the time series n0; r t+1 Represents the reward value at time t+1; t=0..., n0-1; Represents the environmental observation information at time t+n0; Represents the action value at time t+n0.
[0231] Among them, in order to prevent waste of experience and high correlation, a distributed priority experience replay method is used for training, which includes learners, actors and global buffer pools. Several Actors will continuously interact with the environment to generate training sequence data and put them into the Globalbuffer. Each training sequence data has a priority. Learner takes the training sequence data from the Globalbuffer for learning. In order to prevent the overestimation problem, a target network Q(s, a; ω-) is added to the neural network Q(s, a; ω), where s represents the state value at the current moment, a represents the action value at the current moment, ω represents the parameters of the neural network Q(·) at the current moment, and ω represents the parameters of the neural network Q(·) at the previous moment. When calculating the time difference error y DQN When Q(s, a; ω-) is used to replace Q(s, a; ω), the parameters of the target network are updated regularly ω-←ω.
[0232] Furthermore, the reward value at the current moment is assigned in the following way:
[0233] If the agent moves or stays at a non-target point, a reward value of -0.075 is given;
[0234] If the agent reaches the target point, a reward value of +3.0 is given;
[0235] If the agent stays at the target point, a reward value of 0 is given;
[0236] If the agent collides with other agents or obstacles, a reward value of -0.7 is given;
[0237] If the agent has a pseudo-collision with an obstacle, a reward value of -0.1 is given;
[0238] Among them, the pseudo collision between the intelligent agent and the obstacle means that the critical area of the intelligent agent and the obstacle overlap. Figure 11-Figure 12 As shown, the critical region is centered on the obstacle and has a radius of r ca , using it to provide early warning of impending collisions can help avoid collisions between the agent and obstacles. Specifically, when the agent enters the critical area of an obstacle, it is said that a pseudo-collision has occurred with the obstacle, and a small penalty will be fed back to the agent to serve as a warning.
[0239] Specifically, the above-mentioned course learning methods, such as Fig.13As shown in the figure, the goal generator initially allows the model to learn in an environment with a map size of 10 and 1 agent. When the goal discriminator detects that the model learning success rate reaches 85%, the task generator will increase the learning difficulty, that is, the map size increases by 5 and the number of agents increases by 1. The above process is then repeated until the map size is finally 40 and the number of agents is finally 20.
[0240] In the above step S105, the trained agent decision network is used to perform path planning for the agent in the application environment.
[0241] This embodiment conducts experiments in a standard two-dimensional four-neighborhood grid multi-agent path planning scenario, using the previous deep reinforcement learning-based multi-agent path planning methods DHC and DCC as comparison benchmarks.
[0242] The model is trained in a static scene, and the training parameter settings are shown in Table 1:
[0243] Table 1 Training parameter settings
[0244]
[0245] During testing, all models were tested in the same scene and map, and the random number seed was 0. When designing dynamic scenes, these dynamic situations were set to be random and sudden, with the dynamic probability of obstacles being p o , obstacles will randomly increase, delete, and move, and the dynamic probability of the agent task is p a =0.05, the agent tasks will randomly increase or delete changes, where the probability p o =0.2,p a =0.05. There are 11 initial test environments, including 5 environment settings with a map size of 40 and the number of agents of 4, 8, 16, 32, and 64, and 6 environment settings with a map size of 80 and the number of agents of 4, 8, 16, 32, 64, and 128. The obstacle density is 0.15 when generating maps, and 200 maps are generated for each environment. The test results are averaged. The evaluation indicators of this embodiment are shown in Table 2:
[0246] Table 2 Evaluation indicators
[0247] Evaluation indicators definition Success rate (SR) The ratio of successes to total times Average step length (MS) The average time step required Number of collisions (CTPS) Average number of collisions per step Number of collisions between agents (CA) Average number of collisions between agents Number of collisions between the agent and obstacles (CO) Average number of collisions between the agent and obstacles
[0248] In this example, the trained model is named CO3 and tested in the same environment as the multi-agent path planning method based on deep reinforcement learning. The obstacle density when generating the map in the experiment is 0.15. The experimental results are shown in Table 3, where (m0, n0) is used to represent the map size and the initial number of agents. ↑ indicates that the larger the evaluation index value, the better, and ↓ indicates that the smaller the evaluation index value, the better.
[0249] Table 3 Model comparison experimental results
[0250]
[0251]
[0252] As can be seen from Table 3, the proposed model CO3 is superior to other multi-agent path planning methods based on deep reinforcement learning in most scenarios on SR and MS, especially better than DHC. At the same time, CO3 is obviously better than DHC and DCC in CTPS and CO. The more complex the situation, the more outstanding the performance of the proposed model in CTPS and CO. On CA, CO3 after adding common knowledge to selective communication is worse than DCC with single-mode selective communication, but not as good as DHC with broadcast communication in some scenarios, but the inefficient communication of DHC leads to poor performance in other aspects.
[0253] Furthermore, in order to verify the role of the consensus-based communication selection module and the collision avoidance module, this embodiment sets up an ablation experiment. CO3-CK means removing the consensus-based communication selection part based on the CO3 model of the present invention, and CO3-OCA means removing the obstacle collision avoidance (OCA) module based on the CO3 model of the present invention. They are trained under exactly the same parameters and tested in the same scenario. The experimental results are as follows: Figure 14-18 As shown. It can be seen that after removing these two modules, the success rate SR will decrease, while MS will increase. At the same time, the more agents there are in CO3-CK, the greater the decrease in the success rate. Since the number of collisions between agents and obstacles in dynamic scenes is much greater than the number of collisions between agents, the model has obvious changes in the evaluation indicators CTPS and CO after ablation, especially the number of collisions between agents and obstacles in the CO3-OCA model has increased significantly, which also verifies the effectiveness of the anti-collision module. Due to the large change in the CA index value (it will increase exponentially when the number of agents increases), it does not seem to change much in the figure, but it can also be seen that the CA of CO3-CK has increased to a certain extent. From the success rate and collision indicators, it can be seen that CK can improve the synergy of the system to a certain extent.
[0254] Furthermore, this embodiment visualizes the rules of the communication selection module, such as Figure 19-Figure 24 As shown in the figure, the numbers represent the agent numbers, the circles represent the agent locations, the diamonds represent the agent target locations, the composite graphs represent the agent reaching the target locations, and the values outside the graphs represent the importance values I of the neighbor agent j relative to the central agent i. ij , the arrows represent the "request-reply" communication mode between agents. In this example, the map size is 20, the initial number of agents is 10, and Fig.19 It can be seen from the figure that at the time step 7, with agent 6 as the central agent, the important value I6 = {I 63 =1.213,I6=0.334}, Agent 6 will select neighbor agents with importance values greater than the threshold Thor=0.6 for communication, that is, Agent 6 will select Agent 3 for communication. Agent 6 will send a request message to Agent 3, and Agent 3 will respond with a reply message. The two communicate in this way and aggregate information based on the multi-head dot product attention mechanism. During time steps 7 to 9, Agent 6 and Agent 3 are likely to collide. After the above information exchange and aggregation, they have good cooperation ability, that is, Agent 6 chooses not to move, and Agent 3 also knows to turn around and let Agent 6 pass after taking a step forward. At every moment thereafter, the mechanism runs as described above until the end of the curtain. From the whole process, we can see that the calculated importance value can measure which neighbor agents the agent needs to pay more attention to, so it is very reasonable and effective to use this value to select neighbor agents that need to communicate.
[0255] The above-mentioned specific embodiments illustrate that the dynamic multi-agent path planning method proposed in the present invention combines a path planning model constructed based on a deep reinforcement learning method with a greedy algorithm, which is beneficial to strengthening information exchange between agents, and by combining action decisions made with collision avoidance auxiliary information, it can improve the agents' vigilance against obstacles, reduce the incidence of collisions between agents, and help alleviate the problem of a large number of collisions in complex and changeable dynamic scenes, thereby demonstrating the optimal path planning effect in dynamic scenes.
[0256] Embodiment 3:
[0257] The present invention based on the same inventive concept also provides a dynamic multi-agent path planning system, the structural composition diagram is as follows Fig.25 As shown, including:
[0258] Data acquisition module: used to obtain environmental observation data of each agent in a dynamic scene;
[0259] Information aggregation module: used to obtain the communication information of each agent based on the environmental observation data of each agent through the multi-head dot product attention mechanism;
[0260] Model solving module: used to solve the pre-trained path planning model based on the communication information of each intelligent agent using the greedy algorithm to obtain the optimal planning path for each intelligent agent;
[0261] Path planning module: used to plan paths for dynamic scenes based on the optimal planning path of each agent.
[0262] The above-mentioned information aggregation module is specifically used for:
[0263] Calculate the collision avoidance assistance information of each intelligent agent based on the environmental observation data of each intelligent agent;
[0264] According to the environmental observation data of each intelligent agent, the encoding is performed through a preset encoding network to obtain the encoding auxiliary information of each intelligent agent;
[0265] According to the environmental observation information of each agent, an agent set consisting of neighboring agents that communicate with each agent is obtained;
[0266] Based on the collision avoidance auxiliary information, coding auxiliary information and agent collection of each agent, the multi-head dot product attention mechanism is used to aggregate information and obtain the communication information of each agent.
[0267] The expression of the agent set composed of neighboring agents that communicate with each agent in the above information aggregation module is as follows:
[0268] C i ={j|I ij >Thor},j∈N i
[0269] In the formula,
[0270]
[0271] Among them, C i I represents the set of agents consisting of neighboring agents that communicate with agent i; ij represents the importance of agent i and neighbor agent j; Thor represents the communication threshold; N i represents the set of agents consisting of all neighboring agents of agent i; diff ij represents the root mean square difference information between agent i and its neighbor agent j; CKR ij represents the overlap of common knowledge between agent i and its neighbor agent j; TSCKR ij represents the second-order common knowledge richness between agent i and its neighbor agent j; CKCI i Represents the common knowledge cooperation index of agent i.
[0272] In the above information aggregation module, based on the collision avoidance auxiliary information, coding auxiliary information and agent set of each agent, the multi-head dot product attention mechanism is used to aggregate information to obtain the communication information of each agent, including:
[0273] According to the collision avoidance auxiliary information and the coding auxiliary information of each intelligent agent, the coding information of each intelligent agent is obtained;
[0274] According to the coding information of each agent and the agent set composed of neighboring agents that communicate with each agent, the coding information of the neighboring agents of each agent is obtained;
[0275] Based on the encoding information of each agent and the encoding information of the corresponding agent's neighboring agents, the first round of information aggregation is performed through the multi-head dot product attention mechanism to obtain the first round of aggregated information of each agent;
[0276] Based on the first round of aggregated information of each agent and the first round of aggregated information of the corresponding agent’s neighbor agents, a second round of information aggregation is performed through the multi-head dot product attention mechanism to obtain the communication information of each agent.
[0277] The above-mentioned model solving module is specifically used for:
[0278] For each agent, the communication information of the agent is input into the value network of the pre-trained path planning model to obtain the action value function value of the agent; based on the action value function value of the agent, the action value with the largest action value function value of the agent is selected as the optimal action of the agent through a greedy algorithm;
[0279] The optimal action of each agent is taken as the optimal planning path of the corresponding agent.
[0280] The path planning model in the above model solving module includes the following training process:
[0281] Based on the environmental observation information of each agent at the historical moment in the dynamic scene, the action value function value of each agent at the historical moment is obtained through the value network;
[0282] Based on the action value function value of each agent at the historical moment, the greedy algorithm is used to obtain the action value of each agent at the historical moment;
[0283] According to the action value of each agent at the historical moment, the reward value at the historical moment is obtained;
[0284] According to the reward value at the historical moment, the time difference error of each agent at the historical moment is calculated, and the loss function value is determined according to the difference between the time difference error at the historical moment of each agent and the action value function value;
[0285] According to the loss function value, the optimization goal is to maximize the expected discounted total return at the historical moment. The parameters in each agent are updated by the gradient descent method until the training number is reached to obtain the path planning model.
[0286] The expression corresponding to the value network in the above model solving module is as follows:
[0287]
[0288] Among them, Q i (·) represents the action value function of agent i; V(·) represents the state value function; A0(·) represents the advantage function; o represents the environmental observation information at the current moment; a represents the action value at the current moment; α represents the first parameter value of the agent decision network; β represents the second parameter value of the agent decision network; represents the communication information of agent i at time t; A1 represents the action set of agent i; a′ represents the action value at the next moment.
[0289] The expression corresponding to the loss function in the above model solving module is as follows:
[0290] L(α, β) = E o,a,r,o′ [(y DQN -Q i (o, a; α, β)) 2 ]
[0291] In the formula,
[0292]
[0293] Where L(α, β) represents the loss function value with respect to parameters α and β; E o,a,r,o′ represents the expected value of the Markov decision sequence; o represents the environmental observation information at the current moment; o′ represents the environmental observation information at the next moment; a represents the action value at the current moment; r represents the reward value at the current moment; y DQN represents the time difference error; Q i (·) represents the value network function of agent i; represents the discount coefficient under the time series n0; r t+1 Represents the reward value at time t+1; t=0..., n0-1; Represents the environmental observation information at time t+n0; Represents the action value at time t+n0;
[0294] For example, the expression corresponding to the above optimization goal is as follows:
[0295]
[0296] Where Y represents the agent’s expected discounted total return; γ t represents the discount coefficient at time t; t=0...T′; T′ represents the total number of time; R t+1 Represents the reward value at time t+1.
[0297] The above-mentioned path planning module is specifically used for:
[0298] Path planning is performed for dynamic scenes based on the optimal planning path and collision avoidance assistance information of each agent.
[0299] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0300] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0301] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0302] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0303] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the protection scope of the claims to be approved.
Claims
1. A dynamic multi-agent path planning method, characterized in that: include: Obtain environmental observation data for each agent in a dynamic scene; Based on the environmental observation data of each of the intelligent agents, the communication information of each of the intelligent agents is obtained through a multi-head dot product attention mechanism; Based on the communication information of each of the intelligent agents, a pre-trained path planning model is solved using a greedy algorithm to obtain the optimal planning path of each of the intelligent agents; Based on the optimal planned path of each of the intelligent agents, path planning is performed for the dynamic scene.
2. The method according to claim 1, characterized in that The communication information of each of the intelligent agents is obtained based on the environmental observation data of each of the intelligent agents through a multi-head dot product attention mechanism, including: Calculating collision avoidance assistance information of each of the intelligent agents based on the environmental observation data of each of the intelligent agents; According to the environmental observation data of each of the intelligent agents, encoding is performed through a preset encoding network to obtain encoding auxiliary information of each of the intelligent agents; According to the environmental observation information of each of the intelligent agents, an intelligent agent set consisting of neighboring intelligent agents that communicate with each intelligent agent is obtained; Based on the collision avoidance auxiliary information, coding auxiliary information and the agent set of each of the intelligent agents, a multi-head dot product attention mechanism is used to aggregate information to obtain the communication information of each of the intelligent agents.
3. The method according to claim 2, characterized in that The performing path planning for the dynamic scene based on the optimal planning path of each of the intelligent agents includes: Path planning is performed for the dynamic scene according to the optimal planned path of each intelligent agent and the collision avoidance auxiliary information.
4. The method according to claim 2, characterized in that The expression of the agent set composed of neighboring agents that communicate with each agent is as follows: C i ={j|I ij >Thor},j∈N i In the formula, Among them, C i I represents the set of agents consisting of neighboring agents that communicate with agent i; ij represents the importance of agent i and neighbor agent j; Thor represents the communication threshold; N i represents the set of agents consisting of all neighboring agents of agent i; diff ij represents the root mean square difference information between agent i and its neighbor agent j; CKR ij represents the overlap of common knowledge between agent i and its neighbor agent j; TSCKR ij represents the second-order common knowledge richness between agent i and its neighbor agent j; CKCI i Represents the common knowledge cooperation index of agent i.
5. The method according to claim 2, characterized in that The information aggregation is performed based on the collision avoidance auxiliary information, the coding auxiliary information and the set of intelligent agents by using a multi-head dot product attention mechanism to obtain the communication information of each intelligent agent, including: Obtaining coding information of each of the intelligent agents according to the collision avoidance auxiliary information and coding auxiliary information of each of the intelligent agents; According to the coding information of each of the intelligent agents and the intelligent agent set consisting of the neighboring intelligent agents that communicate with each of the intelligent agents, the coding information of the neighboring intelligent agents of each of the intelligent agents is obtained; Based on the encoding information of each of the agents and the encoding information of the neighboring agents of the corresponding agent, a first round of information aggregation is performed through a multi-head dot product attention mechanism to obtain the first round of aggregated information of each of the agents; Based on the first round of aggregation information of each of the intelligent agents and the first round of aggregation information of the neighboring intelligent agents of the corresponding intelligent agent, a second round of information aggregation is performed through a multi-head dot product attention mechanism to obtain the communication information of each of the intelligent agents.
6. The method according to claim 5, characterized in that The method of solving the pre-trained path planning model based on the communication information of each intelligent agent using a greedy algorithm to obtain the optimal planning path of each intelligent agent includes: For each of the intelligent agents, input the communication information of the intelligent agent into the value network of the pre-trained path planning model to obtain the action value function value of the intelligent agent; based on the action value function value of the intelligent agent, through a greedy algorithm, select the action value with the largest action value function value of the intelligent agent as the optimal action of the intelligent agent; The optimal action of each of the intelligent agents is used as the optimal planning path of the corresponding intelligent agent.
7. The method according to claim 6, characterized in that The path planning model includes the following training process: Based on the environmental observation information of each intelligent agent at a historical moment in the dynamic scene, the action value function value of each intelligent agent at the historical moment is obtained through a value network; Based on the action value function value of each of the intelligent agents at the historical moment, using a greedy algorithm, obtaining the action value of each of the intelligent agents at the historical moment; Obtaining a reward value at the historical moment according to the action value of each of the intelligent agents at the historical moment; According to the reward value at the historical moment, the time difference error at the historical moment of each of the intelligent agents is calculated, and the loss function value is determined according to the difference between the time difference error at the historical moment of each of the intelligent agents and the action value function value; According to the loss function value, taking the maximization of the expected discounted total return at the historical moment as the optimization goal, the parameters in each agent are updated by the gradient descent method until the training times are reached, and a path planning model is obtained.
8. The method according to claim 6 or 7, characterized in that The expression corresponding to the value network is as follows: Among them, Q i (·) represents the value network function of agent i; V(·) represents the state value function; A0(·) represents the advantage function; o represents the environmental observation information at the current moment; a represents the action value at the current moment; α represents the first parameter value of the agent decision network; β represents the second parameter value of the agent decision network; represents the communication information of agent i at time t; A1 represents the action set of agent i; a′ represents the action value at the next moment.
9. The method according to claim 7, characterized in that The expression corresponding to the loss function is as follows: L(α,β)=E o,a,r,o′ [(y DQN -Q i (o,a;a,b)) 2 ] In the formula, Where L(α,β) represents the loss function value with respect to parameters α and β; E o,a,r,o′ represents the expected value of the Markov decision sequence; o represents the environmental observation information at the current moment; o′ represents the environmental observation information at the next moment; a represents the action value at the current moment; r represents the reward value at the current moment; y DQN represents the time difference error; Q i (·) represents the value network function of agent i; represents the discount coefficient under the time series n0; r t+1 Represents the reward value at time t+1; t=0…,n0-1; Represents the environmental observation information at time t+n0; Represents the action value at time t+n0; The expression corresponding to the optimization objective is as follows: Where Y represents the agent’s expected discounted total return; γ t represents the discount coefficient at time t; R t+1 Represents the reward value at time t+1.
10. A dynamic multi-agent path planning system, characterized in that: include: Data acquisition module: used to obtain environmental observation data of each agent in a dynamic scene; Information aggregation module: used to obtain the communication information of each intelligent agent through a multi-head dot product attention mechanism based on the environmental observation data of each intelligent agent; Model solving module: used to solve the pre-trained path planning model based on the communication information of each intelligent agent using a greedy algorithm to obtain the optimal planning path of each intelligent agent; Path planning module: used to plan the path of the dynamic scene based on the optimal planning path of each intelligent agent.
Citation Information
Cited By
Decision-making method and device based on intelligent agent, medium and equipment
CN120106621A
Self-adaptive spraying optimization system and method for coating line, computing equipment and storage medium
CN121103562A
