A multi-agent path planning method based on a distributed cooperative deep reinforcement learning model

Abstract

The application discloses a multi-agent path planning method based on a distributed cooperative deep reinforcement learning model, belongs to the communication field and the technical field of reinforcement learning, and comprises the following steps: training the distributed cooperative deep reinforcement learning model designed in the application to obtain a trained distributed cooperative deep reinforcement learning model and randomly generating a map; and based on the trained distributed cooperative deep reinforcement learning model, realizing online path planning of an agent in a multi-agent environment. The distributed cooperative deep reinforcement learning method is different from a centralized deep reinforcement learning method, each agent only makes a decision according to its own observation, meanwhile, communication and sequential decision making among the agents are introduced, and coordination among the agents is promoted. The method is designed from the perspective of a single agent, without considering a group, so that the non-scalability problem of the centralized deep reinforcement learning is avoided.

Description

Technical Field

[0001] This invention belongs to the fields of communication and reinforcement learning technology, and relates to a multi-agent path planning method based on a distributed collaborative deep reinforcement learning model. Background Technology

[0002] With the rapid development of low-cost sensors and computing devices, parallel control of a large number of intelligent vehicles can be supported. Furthermore, modern warfare increasingly emphasizes unmanned warfare, which can improve combat efficiency and reduce battlefield casualties. This invention focuses on multi-agent cooperative path planning in complex environments. Specifically, in the military field, after detecting an enemy target, friendly intelligent vehicles are dispatched from various locations. In complex terrain with high obstacle density, each intelligent vehicle can make online decisions to ensure that there are no conflicts between them and that they successfully reach their respective destinations. This project uses a grid map, assuming that all intelligent vehicles are randomly distributed on the map and proceed to different destinations in an environment with obstacles.

[0003] Regarding the above research findings, a centralized decision-making scheme requires prior knowledge of all agents and the entire terrain conditions in the environment, which is difficult to implement in practice. Furthermore, centralized decision-making schemes require all agents to make decisions simultaneously, placing high demands on the computing power of the equipment. Centralized decision-making schemes also lack scalability; the trained model is only applicable to a specific number of vehicles, and becomes inapplicable once the number of vehicles changes. Distributed decision-making schemes can solve the problems of high computing power requirements and limited model scalability inherent in centralized schemes. However, in distributed decision-making schemes, each intelligent vehicle has a limited observation range and can only make decisions based on its own observations. Insufficient coordination between intelligent vehicles leads to a decrease in task success rate. Summary of the Invention

[0004] To address the issues of insufficient coordination capabilities and low success rates in complex environments in distributed multi-agent path planning schemes, this invention provides a multi-agent path planning method based on a distributed collaborative deep reinforcement learning model, comprising the following steps:

[0005] The distributed collaborative deep reinforcement learning model designed in this invention is trained to obtain a trained distributed collaborative deep reinforcement learning model, and a map is randomly generated.

[0006] Based on a pre-trained distributed collaborative deep reinforcement learning model, online path planning for agents in a multi-agent environment is realized.

[0007] Furthermore, the distributed collaborative deep reinforcement learning model is trained to obtain a trained distributed collaborative deep reinforcement learning model; this includes the following steps:

[0008] S1: Model the multi-agent path planning problem as a Markov process;

[0009] S2: Calculate the neighboring agents of agent i using the location information channels of neighboring agents in each agent's observation space.

[0010] S3: If the neighboring agent If empty, proceed to step S4; otherwise, proceed to step S5.

[0011] S4: Encode the observation space of agent i separately, extract the features of the surrounding environment of agent i, and proceed to S8;

[0012] S5: Agent i calculates its own priority using dynamic priority rules and compares it with that of its neighboring agents. The priority of neighboring agents is obtained through mid-neighbor communication;

[0013] S6: Encode the observation space of agent i and its neighboring agents, extract feature information, encode the observation space of the high-priority agent based on its own observation and make a decision based on the deep reinforcement learning algorithm. After making the decision, the agent concatenates its own action into the encoded information to form interactive information. The low-priority agent that has not made a decision first encodes its own observation space. Assuming that the low-priority agent takes a static decision and concatenates the decision into its own observation space to form interactive information.

[0014] S7: Agent i interacts with neighboring agents through a multi-head attention mechanism, integrating neighbor information;

[0015] S8: Based on the information obtained by agent i, learn from the deep reinforcement learning algorithm and update the neural network parameters.

[0016] Furthermore, the encoding method for the observation space of the neighboring intelligent agents of agent i is the same as the method for encoding the observation space of agent i separately.

[0017] Furthermore, the multi-agent path planning problem is modeled as a Markov process as follows:

[0018] S1-1: Mapping the multi-agent path planning problem from the perspective of a single agent to a Markov process:

[0019]

[0020] in, Representing the state space, Represents the action space. Represents the reward function, γ represents the transition probability, and γ∈[0,1] represents the discount factor;

[0021] In a Markov process, agent i obtains its cumulative reward over multiple steps through interaction with the environment.

[0022]

[0023] Where, r t i This represents the reward value obtained by agent i at time t. For the parameters of the target network, Let π be the Q-value obtained by agent i when it takes action a in state s; the goal of agent i is to find the optimal policy π at each time step. * To maximize the cumulative rewards across multiple steps;

[0024] S1-2: Observation space: Based on a partially observable discrete grid world of size m×m, each agent can only observe the environment within its field of view l×l (l<m), where l is an odd number;

[0025] The observation space is divided into six channels. The first channel represents the binary matrix of obstacles within its field of view, the second channel represents the binary matrix of other agents within its field of view, and the last four channels are the heuristic channels described below; assuming l = 9;

[0026] By assuming that each agent has a fixed field of vision, the multi-agent path planning problem can be generalized to an environment map of arbitrary size. Agent i uses four heuristic channels to obtain information about the target.

[0027] The specific steps are as follows: the four channels correspond to the four actions of up, down, left, and right respectively;

[0028] The size of each channel is the same as the size of the agent's field of vision. The position in the field of vision is marked as 1 if and only if the agent takes an action related to this channel to approach the target.

[0029] At each time step, agent i obtains a 9×9×6 observation space channel; where 9×9 represents the agent's field of view, 6 represents the 6 channels of the agent's observation space, the first channel represents the position of obstacles within the agent's field of view, the second channel represents the position of other agents within the agent's field of view, and the last four channels are the heuristic channels designed earlier. The main function of the heuristic channels is to help the agent obtain information about its own goals; the observation space encoding consists of eight convolutional layers and one gated recurrent unit; the convolutional layers consist of three residual modules and two independent convolutional layers, and each residual module includes two convolutional layers;

[0030] For the observation space encoding of agent i, firstly, the observation space... The input is encoded through a convolutional layer. exist Then, a one-dimensional representation of the agent's decision is constructed. Agents that have not yet made a decision use 0 as their current decision, resulting in... Compare this information with its previous information. The input-gated loop unit obtains its own information at this moment.

[0031] S1-3: Define the action space:

[0032] Agent i performs discrete actions in a discrete grid map. At each time step, agent i chooses to move to an adjacent grid or remain stationary; therefore, each agent has 5 actions; each agent i has an action space. The action at time t is defined as a = [0, 1, 2, 3, 4], which respectively represent moving one unit to the left, up, down, left, and right while remaining stationary;

[0033] S1-4: Define the reward function

[0034]

[0035] in: Let R represent the reward of the i-th agent at time t. m R represents the reward value obtained by agent i when making a move decision at time step t. r R represents the reward value corresponding to agent i making a static decision at time step t. c R represents the reward received by agent i at time step t when a collision occurs, and R represents the reward value obtained by agent i when the task is completed at time step t (when all agents have reached their respective destinations).

[0036] Furthermore, the neighboring agents of agent i are calculated through the observation space of each agent. The process is as follows:

[0037] Given the current environmental state of agent i, calculate the communication mask of agent i. The size of the communication mask is the same as the size of the agent's field of vision. If there are other agents at this location, mark it as 1, otherwise mark it as 0. If there are no other agents in the field of vision of agent i, agent i makes a decision independently based on its own observation information.

[0038] Furthermore, the agent i calculates its own priority using dynamic priority rules and compares it with that of its neighboring agents. The process of obtaining the priority of neighboring agents through mid-neighbor communication is as follows:

[0039] The priority p of agent i at each time step i The calculation method is as follows:

[0040] p i =α1d i +α2(n io +n ia (1)

[0041]

[0042] Where α1 and α2 represent the weighting factors of the agent priority index, d i Let n represent the distance from agent i to the target. io and n ia These represent the number of obstacles and other agents within the agent's field of vision, respectively. If multiple agents have the same priority, the priority is determined by the agent's own ID.

[0043] Furthermore, the observation spaces of agent i and its neighboring agents are encoded, and feature information is extracted. High-priority agents make decisions first. Each high-priority agent encodes its own observation space and makes a decision based on a deep reinforcement learning algorithm. After making a decision, it appends its own action to the encoded information to form interaction information. Low-priority agents that have not yet made a decision first encode their own observation space. Assuming that a low-priority agent makes a static decision and appends the decision to its own observation space to form interaction information, the process of fusing neighbor information for each agent i after forming interaction information is as follows:

[0044] For each agent i, at each time step, there is its own information e. t Among them, high-priority agents generate information e t The high-priority agent uses its own actual decisions, while the low-priority agent is assumed to be static, with each independent attention head acting through a matrix. Information e t Projecting onto the query, key and value, for any... The relationship between agents i and j at the m-th attention head is calculated as follows:

[0045]

[0046] Where, d k Represents the dimension of the key, and It is a scaling factor used for stable training;

[0047] The output of each attention head of agent i is The weighted sum of the values ​​above is used to concatenate the outputs of all M attention heads, and then input into a neural network σ to produce the network's final output:

[0048]

[0049] Finally, output the message. and input message Aggregation is achieved through a gated loop unit.

[0050] Furthermore, based on the information finally obtained by agent i, the neural network parameters are updated using a deep reinforcement learning algorithm:

[0051] S8-1: Agent i will transmit information The state value V is obtained through the Deuling Q network. s (·) and action advantage A(·) are separated;

[0052] S8-2: The Q-value for predicting agent i's action at this time step is as follows:

[0053]

[0054] S6-3: After obtaining the Q-value of agent i, train the loss function of the entire distributed deep reinforcement learning model, and use the mean squared error to calculate the multi-step TD error as follows:

[0055]

[0056]

[0057] Where r t i It is the reward received by agent i at time step t. It represents the multi-step expected return, where θ represents the estimated network parameters. The parameters are for the target network. The two networks have the same structure and their parameters can be interchanged. The parameters of the estimated network are periodically copied to the target network.

[0058] Course learning and multi-agent distributed priority experience replay are introduced to train the distributed collaborative deep reinforcement learning model, thereby realizing the training of the distributed collaborative deep reinforcement learning model.

[0059] This invention provides a collaborative path planning method based on a distributed collaborative deep reinforcement learning model. Building upon distributed deep reinforcement learning for multiple intelligent vehicles, this invention introduces a multi-head attention mechanism to handle communication information between intelligent vehicles. Furthermore, to further enhance coordination among intelligent vehicles, this invention introduces a dynamic priority rule: when intelligent vehicles meet, each vehicle first calculates its own priority according to the priority module. When making decisions, higher-priority vehicles make decisions first. After making their decisions, the higher-priority vehicles broadcast their observation information and decisions to lower-priority agents. Upon receiving the messages, the lower-priority agents use the multi-head attention mechanism to fuse the messages. After fusion, they use the decision module to make a decision, thereby solving the coordination problem between intelligent vehicles.

[0060] This invention has the following advantages: Based on distributed decision-making, it introduces communication between intelligent vehicles and sequential decision-making among them. If other intelligent vehicles are within a vehicle's field of vision, the vehicle communicates with them, integrates their observation information, and then makes its own decision. If no other intelligent vehicles are present, the vehicle makes its own decision based on its own observations. Furthermore, when intelligent vehicles meet, they make decisions according to priority, with higher-priority vehicles making their decisions first and then passing them on to lower-priority vehicles, further enhancing coordination among the intelligent vehicles.

[0061] This invention proposes a multi-agent path planning method based on a distributed cooperative deep reinforcement learning model. It maps the multi-agent cooperative path planning problem to a Markov decision process, introducing a multi-head attention mechanism to handle the communication problem of intelligent vehicles on top of traditional independent reinforcement learning. Simultaneously, it introduces sequential decision-making among agents, enabling agents to communicate and observe spatial information while coordinating with agents within their field of vision to make the next decision, thus greatly enhancing coordination among agents. Furthermore, unlike centralized deep reinforcement learning methods, this distributed cooperative deep reinforcement learning method allows each agent to make decisions based solely on its own observations and calculate interactions with other agents through the multi-head attention mechanism, without considering the collective, thereby avoiding the scalability problem of centralized deep reinforcement learning.

[0062] Based on the above reasons, this invention can be widely applied in the fields of communication and reinforcement learning. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart of the training process for the distributed collaborative deep reinforcement learning model of this invention.

[0065] Figure 2 This is a schematic diagram of the distributed deep reinforcement learning model for the intelligent agent in the case of having neighbors according to the present invention.

[0066] Figure 3 This is a schematic diagram of the intelligent agent's observation space according to the present invention;

[0067] Figure 4 This is a schematic diagram of the multi-agent distributed priority experience replay of the present invention;

[0068] Figure 5 Experiment comparing the method described in this invention with existing methods Figure 1 ;

[0069] Figure 6 Experiment comparing the method described in this invention with existing methods Figure 2 ;

[0070] Figure 7 This is a comparative experimental diagram showing the presence and absence of a heuristic dynamic priority module in the model described in this invention;

[0071] Figure 8 This is a comparative experimental diagram showing the presence and absence of a communication module in the model described in this invention. Detailed Implementation

[0072] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0073] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0075] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0076] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention. The directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0077] For ease of description, spatial relative terms such as "above," "over," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation besides the orientation of the device as described in the figures. For example, if the device in the figures is inverted, a device described as "above" or "above" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0078] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0079] Agent i observes whether there are other agents within its field of vision. If not, it encodes its own observation space and then makes a decision based on its own observation information. If there are other agents, it needs to calculate and communicate the priority with other agents, and then make decisions according to the priority order. After making a decision, the high-priority agent passes its own decision and observation space to agent i, while the low-priority agent only passes its own observation space to agent i. Agent i integrates the information passed by neighboring agents through a multi-head attention mechanism and makes a decision through the decision module.

[0080] A multi-agent path planning method based on a distributed collaborative deep reinforcement learning model includes the following steps:

[0081] Step 1: Train the distributed collaborative deep reinforcement learning model to obtain the trained distributed collaborative deep reinforcement learning model, and randomly generate maps;

[0082] Step 2: Based on the trained distributed collaborative deep reinforcement learning model, realize online path planning for agents in a multi-agent environment.

[0083] The intelligent agents include intelligent entities with autonomous decision-making and mobility functions, such as intelligent vehicles, robots, drones, and unmanned surface vessels.

[0084] Step one and step two are executed sequentially;

[0085] Figure 1This is a flowchart of the training process for the distributed deep reinforcement learning model of this invention.

[0086] Training a distributed deep reinforcement learning model to obtain a trained distributed deep reinforcement learning model includes the following steps:

[0087] S1: Model the multi-agent path planning problem as a Markov process;

[0088] S2: Calculate the neighboring agents of agent i using the location information channels of neighboring agents in each agent's observation space.

[0089] S3: If the neighboring agent If empty, proceed to step S4; otherwise, proceed to step S5.

[0090] S4: Encode the observation space of agent i separately, extract the features of the surrounding environment of agent i, and proceed to S8;

[0091] S5: Agent i calculates its own priority using dynamic priority rules and compares it with that of its neighboring agents. The priority of neighboring agents is obtained through mid-neighbor communication;

[0092] S6: Encode the observation space of agent i and its neighboring agents, extract feature information, encode the observation space of the high-priority agent based on its own observation and make a decision based on the deep reinforcement learning algorithm. After making the decision, the agent concatenates its own action into the encoded information to form interactive information. The low-priority agent that has not made a decision first encodes its own observation space. Assuming that the low-priority agent takes a static decision and concatenates the decision into its own observation space to form interactive information.

[0093] S7: Agent i interacts with neighboring agents through a multi-head attention mechanism, integrating neighbor information;

[0094] S8: Based on the information obtained by agent i, learn from the deep reinforcement learning algorithm and update the neural network parameters.

[0095] During the communication process, all agents make decisions according to priority order. The entire communication process takes a short time. In fact, all agents execute decisions simultaneously, and the entire decision-making process is distributed.

[0096] The method of encoding the observation space of agent i and its neighboring agents is the same as the method of encoding the observation space of agent i separately.

[0097] Figure 2This is a schematic diagram of the distributed collaborative deep reinforcement learning model for the existence of neighbors in the present invention; in the example, the observation space of agent i is 9×9, black represents obstacles, and orange represents neighboring agents; the model consists of four parts: dynamic priority module, encoder, communication module and decision module.

[0098] The dynamic priority module is used by agent i to calculate its own priority according to dynamic priority rules and to compare it with that of neighboring agents. The priority of neighboring agents is obtained through mid-neighbor communication;

[0099] The encoder is used to encode the observation space of agent i and its neighboring agents.

[0100] The main approach involves encoding the agent's observation space through convolutional operations. This encoding consists of eight convolutional layers and one gated recurrent unit. Each convolutional layer comprises three residual modules and two independent convolutional layers, with each residual module containing two convolutional layers. The gated recurrent unit is used to fuse information from the agent's previous time step.

[0101] The communication module mainly helps agent i fuse information transmitted from neighboring agents. This module consists of a multi-head attention convolutional kernel. After agent i receives information from neighboring agents, it fuses the information through the multi-head attention mechanism.

[0102] The decision-making module is used to learn based on the information obtained by agent i using a deep reinforcement learning algorithm, update the neural network parameters, make decisions, interact with the environment, and move to the next time step.

[0103] Figure 3 This is a schematic diagram of the observation space of agent i.

[0104] Furthermore, the process of modeling the multi-agent path planning problem as a Markov process is as follows:

[0105] S1-1: This invention is based on instantiation of independent reinforcement learning. Each agent learns its own action value function independently, and regards other agents as part of the environment. At the same time, it introduces communication between agents and order decision-making between agents to enhance coordination between agents.

[0106] Therefore, this will be described from the perspective of a single intelligent agent;

[0107] Mapping the multi-agent path planning problem from the perspective of a single agent into a Markov process:

[0108]

[0109] in, Representing the state space, Represents the action space. Represents the reward function, The transition probability is represented by γ, where γ∈[0,1] represents the discount factor. In the Markov process, the cumulative reward of agent i is obtained through the interaction between agent i and the environment. Where, r t i This represents the reward value obtained by agent i at time t. For the parameters of the target network, Let π be the Q-value obtained by agent i when it takes action a in state s; the goal of agent i is to find the optimal policy π at each time step. * To maximize the cumulative rewards across multiple steps;

[0110] S1-2: Observation Space: This application considers a partially observable discrete grid world of size m×m. Each agent can only observe the environment within its field of view of l×l (l<m), where l is an odd number to ensure that the agent is located at the center of its field of view. The observation space is divided into six channels. The first channel represents the binary matrix of obstacles within its field of view, the second channel represents the binary matrix of other agents within its field of view, and the remaining four channels are heuristic channels described below. In this invention, it is assumed that l = 9.

[0111] Considering the highly scalable nature of a partially observable world, and assuming that the fixed field of view of each agent can be generalized to an environment map of arbitrary size, the method designed in this invention differs from traditional centralized methods. This method is scalable. Furthermore, the agent needs access to information about its target, which is typically outside its field of view. Therefore, this invention employs four heuristic channels to provide the agent with information about the target. Specifically, the four channels correspond to four actions: up, down, left, and right. The size of each channel is the same as the size of the agent's field of view. A position in the field of view is marked as 1 if and only if the agent takes an action associated with that channel to approach the target.

[0112] At each time step, agent i will obtain a 9×9×6 observation space channel, where 9×9 represents the agent's field of vision, 6 represents the 6 channels of the agent's observation space, the first channel represents the position of obstacles within the agent's field of vision, the second channel represents the position of other agents within the agent's field of vision, and the last four channels are the heuristic channels designed earlier, whose main function is to help the agent obtain information about its own goals.

[0113] The observation space encoding consists of eight convolutional layers and one gated recurrent unit. Each convolutional layer comprises three residual modules (each containing two convolutional layers) and two independent convolutional layers. For the observation space of agent i, the observation space o is first... t i The input is encoded through a convolutional layer. exist Then, a one-dimensional representation of the agent's decision is constructed. Agents that have not yet made a decision use 0 (static decision) as their current decision, resulting in... Compare this information with its previous information. The input-gated loop unit obtains its own information at this moment.

[0114] S1-3: Define the action space Agent i performs discrete actions in a discrete grid map. At each time step, agent i can choose to move to an adjacent grid or remain stationary. Therefore, in this invention, each agent has 5 actions. Each agent has an action space. The action at time t is defined as a = [0, 1, 2, 3, 4], which respectively represent moving one unit to the left, up, down, left, and right while remaining stationary;

[0115] S1-4: Define the reward function

[0116]

[0117] in: Let R represent the reward of the i-th agent at time t. m R represents the reward value obtained by agent i when making a move decision at time step t. r R represents the reward value corresponding to agent i making a static decision at time step t. c R represents the reward received by agent i at time step t when a collision occurs, and R represents the reward value obtained by agent i when the task is completed at time step t (when all agents have reached their respective destinations).

[0118] Furthermore, the neighboring agents of agent i are calculated through the observation space of each agent. The process is as follows:

[0119] Given the current environmental state of agent i, a communication mask for agent i is calculated. The size of the communication mask is the same as the size of the agent's field of vision. If there are other agents at this location, it is marked as 1; otherwise, it is marked as 0. If there are no other agents within agent i's field of vision, agent i makes its own decision based on its observation information.

[0120] Furthermore, if other agents exist within the field of vision of agent i, the process by which agent i calculates its own priority using dynamic priority rules and communicates with its neighbors to obtain the priorities of its neighbors is as follows:

[0121] At each time step, if If the value is not empty, agent i calculates its own priority through the dynamic priority module and obtains it through communication. Priority of other intelligent agents;

[0122] The priority p of agent i at each time step i The calculation method is as follows:

[0123] p i =α1d i +α2(n io +n ia (1)

[0124]

[0125] Where α1 and α2 represent the weighting factors of the agent priority index, d i n represents the distance from agent i to the target. io and n ia These represent the number of obstacles and other agents within the agent's field of vision, respectively. If multiple agents have the same priority, their priority is determined by their own ID.

[0126] Furthermore, the agent i interacts with other agents through a multi-head attention mechanism. High-priority agents make decisions first, and after making their decisions, they splice their actions into their own observation space to form interactive information, which is then transmitted to low-priority agents. Low-priority agents that have not made decisions will assume that they will take a static decision and splice their decisions into their own observation space to form interactive information. After the interactive information is formed, the process of fusing neighbor information for each agent i is as follows:

[0127] For each agent i, at each time step, there is its own information e. t Among them, high-priority agents generate information e t The high-priority agent uses its own actual decisions, while the low-priority agent is assumed to be static, with each independent attention head acting through a matrix. Information e tProjecting to the query (the three matrices are self-learning matrices; the essence of the attention mechanism is that the closer the query is to a certain key, the closer the output of the query is to the value corresponding to the key. By projecting the information of each agent and calculating the similarity between the information of agent i and agent j according to the formula, the weight of agent i when fusing the information of agent j is obtained), key, and value, for any... The relationship between agents i and j at the m-th attention head is calculated as follows: M represents the attention head. Since Q and K may have more than one interaction relationship, a multi-head attention mechanism is adopted.

[0128]

[0129] Where, d k Represents the dimension of the key, and It is a scaling factor used for stable training.

[0130] The output of each attention head of agent i is The weighted sum of the values ​​above is used to concatenate the outputs of all M attention heads, and then input into a neural network σ to produce the network's final output:

[0131]

[0132] Finally, output the message. and input message Aggregation is achieved through a gated loop unit.

[0133] Furthermore, the process of fusing the information of neighboring agents with the information of agent i, and then learning it based on a deep reinforcement learning algorithm, with the neural network updating parameters, is as follows:

[0134] S6-1: Agent i will transmit information The state value V is represented by the Deuling Q network. s (·) and action advantage A(·) are separated;

[0135] S6-2: Predict the Q value of agent i, as follows:

[0136]

[0137] S6-3: After obtaining the Q-value of agent i, train the loss function (TD error) of the entire distributed deep reinforcement learning model. To stabilize training, mean squared error is used to calculate the multi-step TD error:

[0138]

[0139]

[0140] Where: r t i It is the reward that agent i receives at time step t. It represents the multi-step expected return, where θ represents the estimated network parameters. The target network parameters are the same for both networks (meaning the two neural networks are identical, but their parameters are different). The parameters can be interchanged, and the parameters of the estimated network are periodically copied to the target network. In addition, to accelerate the entire training process, this invention introduces course learning and multi-agent distributed priority experience replay during model training.

[0141] Learning directly from a complex environment with a large number of agents is difficult. Therefore, this invention employs a course-based learning approach, gradually introducing more complex tasks to the agents. The course-based learning process is as follows:

[0142] The specific operation method is as follows: Start with a simple task on a 10×10 map with one agent. If the success rate of the current task exceeds 90%, then create two new, more complex tasks for agent i by either increasing the number of agents by 1 or increasing the map size by 5. The two new, more complex tasks refer to randomly generating one environment by increasing the number of agents by 1 and randomly generating two environments by increasing the map size by 5, respectively, based on the original environment.

[0143] The process of multi-agent distributed priority experience replay is as follows:

[0144] To accelerate the training process, this invention employs the Ape-X architecture in the training of distributed deep reinforcement learning models, in which multiple executors generate experiences and a single learner updates the network. In the experiment, each executor has a copy of the environment with the current Q network, continuously generates new experiences, and initializes priorities for them. Experiences from all executors are sent to a shared priority replay experience pool.

[0145] Then, the learner extracts the most useful experience from the experience pool and updates the experience priority and network parameters. Although the model is trained for a single agent, the experience of all agents needs to be stored for the purpose of communication and the experience is initialized and the priority is updated. Through the sharing of priorities, the good experience generated by all agents will improve the learner's learning effect.

[0146] Figure 4 This is a schematic diagram of the distributed priority experience replay in the multi-agent system of this invention. Multiple executors generate experiences and store them in an experience pool. A single learner extracts experiences from the experience pool, updates the experience weights and network parameters, and periodically copies the network parameters to the executor's network.

[0147] Figure 5 Experiment comparing the method described in this invention with existing methods Figure 1 ;

[0148] Figure 6 Experiment comparing the method described in this invention with existing methods Figure 2 ;

[0149] PICO: This method uses priority communication, but it is an implicit priority communication. It learns the priorities of each agent by imitating a centralized planner ODRM*, and then uses the priorities of each agent as the weights of the communication links to obtain the decisions of each agent.

[0150] PRIMAL: This method uses an implicit coordination approach that combines imitation learning and reinforcement learning, and the agents do not communicate with each other.

[0151] Experimental Setup: The experiment used two agents closest to each other for communication. Obstacle density in the environment was sampled from 0 to 0.5. The agent's field of view was 9×9. The maximum length of each training episode was 256, the batch size was 192, the sequence length was 20, and a dynamic learning rate was used, initially set to 2×10⁻⁶. -4 Furthermore, the number of training steps decreased by 50% at 40,000 and 80,000 steps, with a maximum training step count of 200,000. During model testing, this invention randomly generated 100 test cases each time, that is, 100 test cases were randomly generated each time with a fixed number of vehicles, environment size, and obstacle density, and the number of test cases that finally completed the task was counted.

[0152] As can be seen from the figure, this method is significantly superior to the other two schemes. Furthermore, as the density of obstacles in the environment increases, the success rate of the algorithm's agents in completing the task decreases only slightly, indicating that the explicit coordination between agents in this method is highly effective. This method places greater emphasis on coordination between agents, and its success rate remains high even as the number of agents and the density of obstacles increase.

[0153] Figure 7The diagram shows a comparison experiment of the model described in this invention with and without a dynamic priority module. Random priority: At each decision step, all agents make decisions in a random order. In a 40×40 map environment with a fixed obstacle density of 0.3, the diagram compares the dynamic heuristic priority scheme and the random priority scheme. It can be seen from the diagram that although the random priority scheme can achieve similar results to this scheme when the task is relatively simple, the success rate of the random priority scheme decreases significantly as the environment becomes more complex. The dynamic heuristic priority scheme of this invention, however, maintains a high level of performance. This is mainly because the dynamic heuristic priority design of this scheme considers three factors that have a high impact on the agent's decision at each time step, promoting coordination among agents.

[0154] Figure 8 The diagram shows a comparative experiment of the model method of this invention with and without a communication module. The communication module determines whether the agent communicates with other agents at each decision step. In a 40×40 map environment with a fixed obstacle density of 0.3, the diagram compares two schemes: one with a communication module and one without. It can be seen from the diagram that in relatively simple tasks, the schemes without and with communication modules achieve similar results. However, as the environment becomes more complex, the success rate of the scheme without a communication module drops sharply. This is because in simpler environments, agents do not need to communicate; each agent only needs to make decisions based on its own observations. As the environment becomes more complex, agents need to coordinate through communication.

[0155] from Figure 5-8 It can be seen that the model in this invention has good scalability and generalization ability, and the trained model can meet the needs of different task environments.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent path planning method based on a distributed collaborative deep reinforcement learning model, characterized in that: Includes the following steps: Train the distributed collaborative deep reinforcement learning model to obtain the trained distributed collaborative deep reinforcement learning model, and randomly generate maps. Based on a pre-trained distributed collaborative deep reinforcement learning model, online path planning for agents in a multi-agent environment is realized. Training a distributed collaborative deep reinforcement learning model to obtain a trained distributed collaborative deep reinforcement learning model includes the following steps: S1: Model the multi-agent path planning problem as a Markov process; S2: Calculate the neighboring agents of agent i using the location information channels of neighboring agents in each agent's observation space. ; S3: If the neighboring agent If empty, proceed to step S4; otherwise, proceed to step S5. S4: Encode the observation space of agent i separately, extract the features of the surrounding environment of agent i, and proceed to S8; S5: Agent i calculates its own priority using dynamic priority rules and compares it with that of its neighboring agents. The priority of neighboring agents is obtained through mid-neighbor communication; S6: Encode the observation space of agent i and its neighboring agents, extract feature information, encode the observation space of the high-priority agent based on its own observation and make a decision based on the deep reinforcement learning algorithm. After making the decision, the agent concatenates its own action into the encoded information to form interactive information. The low-priority agent that has not made a decision first encodes its own observation space. Assuming that the low-priority agent takes a static decision and concatenates the decision into its own observation space to form interactive information. S7: Agent i interacts with neighboring agents through a multi-head attention mechanism, fusing neighbor information; the specific process is as follows: Each agent i has its own information at each time step. Among them, high-priority agents generate information The high-priority agent uses its own actual decisions, while the low-priority agent is assumed to be static, with each independent attention head acting through a matrix. , , Information Projecting onto the query, key and value, for any... The relationship between agents i and j at the m-th attention head is calculated as follows: （3） in, Represents the dimension of the key, and It is a scaling factor used for stable training; The output of each attention head of agent i is above The weighted sum of the values ​​is used to concatenate the outputs of all M attention heads, which are then fed into a neural network. This produces the final output of the network: （4） Finally, output the message. and input message Aggregation is achieved through a gated loop unit. ; S8: Based on the information obtained by agent i, learn using a deep reinforcement learning algorithm to update the neural network parameters. The specific process is as follows: Based on the information obtained by agent i, the neural network parameters are updated using a deep reinforcement learning algorithm: S8-1: Agent i will transmit information The state values ​​are processed through the Deuling Q network. and action advantages separate; S8-2: The Q-value for predicting agent i's action at this time step is as follows: （5） S8-3: After obtaining the Q-value of agent i, the loss function of the distributed deep reinforcement learning model is trained, and the multi-step TD error is calculated using the mean squared error as follows: （6） （7） in: It is the reward that agent i receives at time step t. It is a multi-step expected return, This indicates the estimation of network parameters. The parameters of the target network are the same as those of the two networks, and the parameters can be interchanged. The parameters of the estimated network are periodically copied to the target network. Course learning and multi-agent distributed priority experience replay are introduced to train the distributed collaborative deep reinforcement learning model, thereby realizing the training of the distributed collaborative deep reinforcement learning model.

2. The multi-agent path planning method based on a distributed cooperative deep reinforcement learning model according to claim 1, characterized in that: The encoding method for the observation space of agent i's neighboring agents is the same as the encoding method for agent i's observation space separately.

3. The multi-agent path planning method based on a distributed cooperative deep reinforcement learning model according to claim 1, characterized in that: The process of modeling the multi-agent path planning problem as a Markov process is as follows: S1-1: Mapping the multi-agent path planning problem from the perspective of a single agent to a Markov process: in, Representing the state space, Represents the action space. Represents the reward function, Represents the transition probability. Indicates the discount factor; In a Markov process, agent i obtains its cumulative reward over multiple steps through interaction with the environment. in, Indicates that agent i is in The reward value obtained at any time, For the parameters of the target network, For agent i in state Take action below The obtained Q-value; the goal of agent i is to find the optimal policy at each time step. To maximize the cumulative rewards across multiple steps; S1-2: Observation Space: Based on a A partially observable discrete grid world, where each agent can only observe its field of view. The internal environment It is an odd number; The observation space is divided into six channels. The first channel represents the binary matrix of obstacles within its field of view, the second channel represents the binary matrix of other agents within its field of view, and the last four channels are heuristic channels; assuming ; By assuming that each agent has a fixed field of vision, the multi-agent path planning problem is extended to an environment map of arbitrary size. Agent i uses four heuristic channels to obtain information about the target. The specific steps are as follows: the four channels correspond to the four actions of up, down, left, and right respectively; The size of each channel is the same as the size of the agent's field of vision. The position in the field of vision is marked as 1 if and only if the agent takes an action related to this channel to approach the target. At each time step, agent i will receive a The observation space channel; among which, The field of view of the agent is represented by 6, which represents the six channels of the agent's observation space. The first channel represents the position of obstacles within the agent's field of view, the second channel represents the position of other agents within the agent's field of view, and the last four channels are heuristic channels designed earlier. The role of heuristic channels is to help the agent obtain information about its own goals. The observation space encoding consists of eight convolutional layers and one gated recurrent unit. The convolutional layer consists of three residual modules and two independent convolutional layers. Each residual module includes two convolutional layers. For the observation space encoding of agent i, firstly, the observation space... The input is encoded through a convolutional layer. ,exist Then, a one-dimensional representation of the agent's decision is constructed. Agents that have not yet made a decision use 0 as their current decision, resulting in... Compare this information with its information from the previous moment. The input-gated loop unit obtains its own information at this moment. ; S1-3: Define the action space: Agent i performs discrete actions in a discrete grid map. At each time step, agent i chooses to move to an adjacent grid or remain stationary; therefore, each agent has 5 actions; each agent i has an action space. Inside The action at a given moment is defined as follows: , respectively representing moving one space to remain stationary, up, down, left, and right; S1-4: Define the reward function : in: Let represent the reward of the i-th agent at time t. This represents the reward value obtained by agent i when making a move decision at time step t. This represents the reward value obtained by agent i at time step t by making a static decision. Let represent the reward received by agent i at time step t when a collision occurs. This indicates the reward value that agent i receives when the task is completed at time step t.

4. The multi-agent path planning method based on a distributed cooperative deep reinforcement learning model according to claim 1, characterized in that: The neighboring agents of agent i are calculated through the location information channels of neighboring agents in the observation space of each agent. The process is as follows: Given the current environmental state of agent i, calculate the communication mask of agent i. The size of the communication mask is the same as the size of the agent's field of vision. If there are other agents at this location, mark it as 1, otherwise mark it as 0. If there are no other agents in the field of vision of agent i, agent i makes a decision independently based on its own observation information.

5. The multi-agent path planning method based on a distributed cooperative deep reinforcement learning model according to claim 1, characterized in that: The agent i calculates its own priority using dynamic priority rules and compares it with that of its neighboring agents. The process of obtaining the priority of neighboring agents through mid-neighbor communication is as follows: Priority of agent i at each time step The calculation method is as follows: （1） （2） in, and The weighting factor represents the priority index of the agent. This represents the distance from agent i to the target. and These represent the number of obstacles and other agents within the agent's field of vision, respectively. If multiple agents have the same priority, the priority is determined by the agent's own ID.

Images