Multi-agent collaborative decision-making method based on deep reinforcement learning under communication resource constraint

Abstract

The application discloses a kind of communication resource is limited based on multi-agent collaborative decision-making method of deep reinforcement learning, agent learns the evaluation of observation importance weight locally, is allocated channel by local search algorithm in centralized manner, and after communication is completed, agent carries out message integration and feature extraction based on multi-head self-attention mechanism, and message integration module is added in evaluation network to generate a global observation specific to agent for evaluating current value, guiding the training of policy network parameter, finally form a communication-based multi-agent collaborative decision-making method.The method effectively deals with the time-varying topology problem caused by the change of agent coordinates and communication agent, the agent integrates the input information with changing dimensions, and the method has a wide range of applications, maximizes the use of limited channel resources on the basis of ensuring no communication conflict, has good economic and social benefits, and is suitable for popularization and use.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent cooperative decision-making, and mainly designs a multi-agent cooperative decision-making method based on deep reinforcement learning under the condition of limited communication resources. Background Technology

[0002] In recent years, deep reinforcement learning has achieved remarkable success in a range of challenging tasks. Inspired by its powerful perception and learning capabilities, researchers have been continuously exploring its application in multi-agent domains to promote cooperative behavior among agents. Many cooperative problems exist in human society, such as autonomous driving, traffic light control, navigation guidance, and smart grid control. These naturally form cooperative multi-agent systems with the goal of maximizing rewards from a team perspective.

[0003] Taking the most common pursuit and escape scenario as an example, most current pursuit and escape problem scenarios assume that the performance of the pursuing agent is better than or equal to that of the escape target. However, for scenarios where the performance of the pursuing agent is inferior to that of the escape target, it is necessary to study more complex and accurate models to make full use of the advantages of the cluster to handle the pursuit problem under such conditions of unequal motion parameters. Through the cooperation of multiple agents, the collaborative pursuit of high-performance escape targets can be completed.

[0004] Traditional methods for addressing these issues rely on mathematical models to design agent control strategies. These models often impose assumptions about enemy movement or require knowledge of the enemy's control strategy. However, in a battlefield environment, it's difficult for friendly forces to ascertain the enemy's control strategy. Furthermore, controller parameters based on mathematical models are often manually set or require optimization using other algorithms. If the environment changes, the old controller parameters may no longer be optimal, presenting limitations. Most existing multi-agent reinforcement learning methods employ a centralized training and distributed execution model. In fully cooperative scenarios like pursuit environments, during distributed execution, each agent makes decisions based solely on local observations without communicating with other friendly agents. This partial observability and randomness can easily disrupt the learned cooperative strategy, leading to deviations in the cooperative strategy learned during centralized training and hindering collaborative cooperation between agents. Current research largely assumes that all agents can communicate freely with each other. However, in reality, limited bandwidth and communication range mean agents may not receive information from distant agents. Moreover, in extreme environments, limited bandwidth cannot meet the communication needs of all agents, allowing only a subset of agents to communicate within each cycle. Summary of the Invention

[0005] To address the aforementioned issues, this invention discloses a multi-agent collaborative decision-making method based on deep reinforcement learning under limited communication resources. Agents locally learn to evaluate the importance weights of observations and centrally allocate channels using a local search algorithm. After communication is completed, agents perform message integration and feature extraction based on a multi-head self-attention mechanism. A Transformer module is added to the evaluation network to generate an agent-specific global observation for evaluating the current value, guiding the training of policy network parameters, and ultimately forming a communication-based multi-agent collaborative pursuit strategy.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A multi-agent cooperative decision-making method based on deep reinforcement learning is proposed for communication-restricted environments with locally observable multi-agent cooperative mechanisms. The method employs the PPO algorithm to learn the importance of local observations in the channel allocation part, constructs a communication conflict graph based on the network topology, and uses a local search algorithm for channel allocation. Agents integrate received data based on a multi-head self-attention mechanism and learn agent control strategies through the MAPPO algorithm to complete the cooperative pursuit task. The specific steps are as follows:

[0008] Step 1: Initialize the environment parameters, perform orthogonal initialization of all network parameters in the Actor and Critic agents, and set the simplified dynamic equations of the agents;

[0009] Step 2: Randomly assign coordinates to friendly agents and targets. The initial velocity is 0;

[0010] Step 3: Set the reward function for cooperative pursuit This includes personal rewards related to the target distance. and team rewards And collision penalties between intelligent agents And target capture reward , , , , This is the proportionality coefficient;

[0011] Step 4: Each agent obtains local observations. The input is fed into the Encoder layer and GRU layer for feature extraction, and its output is used as the content of communication. , will observe Input to the weight generator policy network The importance weighting strategy for local observations is obtained, and the output is two parameters of a Beta distribution. and Importance weight parameters are obtained by sampling this distribution. ;

[0012] Step 5: Each agent sends its observation importance weight to the team leader to request a communication channel. The leader collects the weights of all agents, constructs a conflict graph, and generates a conflict-free channel allocation strategy through a local search algorithm. This maximizes the use of limited channels without causing communication conflicts, and the allocation results are broadcast to the other agents. All agents communicate based on the channel allocation results at that moment.

[0013] Step 6: After this round of communication is completed, each agent inputs all the data collected in this cycle into the multi-head self-attention layer for information integration and feature extraction, generating... And splice the original local information Policy network input to the agent The output is sampled to obtain the final action, which is then applied to the agent to obtain the environmental feedback data at that moment. ,in, For the next observation time, The reward for the current moment. Used to indicate whether the round has ended at the current time. This indicates the communication topology at the next moment;

[0014] Step 7: Steps 4, 5, and 6 above complete a trajectory sample collection process. The trajectory sample is saved to the playback buffer. The trajectory sample content is as follows: ,in, and These are the observations at the current time and the next time. For the action at the current moment, The reward for the current moment. Used to indicate whether the round has ended at the current time. This indicates the communication topology at the next moment. According to... Determine if the round has ended. If it has ended, jump back to step 2 to continue collecting data until the replay buffer pool data reaches the required level.

[0015] Step 8: Extract trajectory sample data, and calculate the advantage function of all agents for each trajectory sample using the generalized advantage function estimation method. ,in, This is the discount factor in GAE. As a discount factor for rewards, express Time difference error at any given moment The reward for the current moment. and The state value function for the current time step and the next time step;

[0016] Step 9: For the agent control policy network Using the policy given by the current policy network as a benchmark, the parameters of the policy network are iteratively updated to obtain a policy with better performance. The use of the dual policy network loss function can reduce the training variance while ensuring model convergence. The loss function for policy parameter updates is:

[0017]

[0018] in The ratio of the old to the new strategies. For the dominant function, The function is used to... Cut and limit Within the interval, For the hyperparameters of clipping, For policy entropy, This represents the corresponding strategy entropy coefficient.

[0019] For the agent control evaluation network, the parameters of the evaluation network are updated iteratively, and the loss function is:

[0020]

[0021] in, The state value function corresponding to the control policy network. For the actual value function

[0022] Its overall weighted loss function is:

[0023]

[0024] in, This is the Critic coefficient.

[0025] Step 10: For the communication weight generation strategy network Similarly, using the policy given by the current policy network as a benchmark, the parameters of the policy network are iteratively updated to obtain a policy with better performance. The loss function used is:

[0026]

[0027] in The ratio of the old to the new strategies. For the dominant function, The function is used to... Cut and limit Within the interval, For the hyperparameters of clipping, For policy entropy, This represents the corresponding strategy entropy coefficient.

[0028] For the evaluation network of the communication weight generation module, the loss function used is:

[0029]

[0030] in, Generate the state value function corresponding to the policy network for communication weights. This is the actual value function.

[0031] Its overall weighted loss function is:

[0032]

[0033] in, This is the Critic coefficient.

[0034] Step 11: Repeat steps 2 through 10 until the parameters of the policy network and the evaluation network finally converge. At this point, the algorithm has converged.

[0035] Step 12: In the online distributed execution phase, the model trained in a centralized manner is used as a benchmark. The trained parameters are loaded to verify the effectiveness of the algorithm. The multi-agent system quickly captures all escaping targets and communicates with each other to expand the perception domain and achieve better cooperation.

[0036] Compared with existing technologies, this invention has the following advantages: 1) This method promotes agent collaboration by increasing explicit communication between agents and considers scenarios where the number of communication channels is limited in actual communication processes, making it more realistic. 2) This method considers the time-varying topology problem caused by the continuous changes in agent coordinates and the changes in communicating agents in high-speed agent motion environments. Agents can integrate input information with constantly changing dimensions, rather than fixing the input dimensions of the agents. 3) This method maximizes the use of limited channel resources while ensuring no communication conflicts occur. It exhibits better collaborative performance in environments with limited communication resources and reduces the solution time based on heuristic algorithms, optimizing the solution time while maintaining certain performance. 4) This method considers communication-limited situations, has a wide range of applications, can be freely extended to any communication resource environment, has good economic and social benefits, and is suitable for widespread use. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of a multi-agent collaborative pursuit scenario involving the method of the present invention;

[0038] Figure 2This is a block diagram of the intelligent agent network structure of the method of the present invention;

[0039] Figure 3 A comparison chart of average reward curves in a 4vs2 chase scenario;

[0040] Figure 4 A comparison chart of average round end times in a 4vs2 chase scenario;

[0041] Figure 5 A comparison chart of average reward curves in a 7vs3 chase scenario;

[0042] Figure 6 A comparison chart of average round end times in a 7vs3 chase scenario. Detailed Implementation

[0043] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0044] Example 1: This invention addresses the problem in existing technologies that they do not adequately consider communication limitations and do not effectively utilize communication data. For a typical multi-agent cooperative pursuit scenario, as shown in the attached diagram... Figure 1 As shown, a multi-agent cooperative decision-making method based on deep reinforcement learning is provided under conditions of limited communication resources, including the following steps:

[0045] Step 1: Initialize the environment parameters, perform orthogonal initialization of all network parameters in the Actor and Critic agents, and set the simplified dynamic equations of the agents;

[0046] Step 2: Randomly assign coordinates to friendly agents and targets. initial velocity =0;

[0047] Step 3: Set the reward function for cooperative pursuit This includes personal rewards related to the target distance. and team rewards And collision penalties between intelligent agents And target capture reward , , , , This is the proportionality coefficient;

[0048] Step 4: Each agent obtains local observations. The input is fed into the Encoder layer and GRU layer for feature extraction, and its output is used as the content of communication. , will observe Input to the weight generator policy network The importance weighting strategy for local observations is obtained, and the output is two parameters of a Beta distribution. and Importance weight parameters are obtained by sampling this distribution. ;

[0049] Step 5: Each agent sends its observation importance weight to the team leader to request a communication channel. The leader collects the weights of all agents, constructs a conflict graph, and generates a conflict-free channel allocation strategy through a local search algorithm. This maximizes the use of limited channels without causing communication conflicts, and the allocation results are broadcast to the other agents. All agents communicate based on the channel allocation results at that moment.

[0050] Step 6: After this round of communication is completed, each agent inputs all the data collected in this cycle into the multi-head self-attention layer for information integration and feature extraction, generating... And splice the original local information Policy network input to the agent The output is sampled to obtain the final action, which is then applied to the agent to obtain the environmental feedback data at that moment. ,in, For the next observation time, The reward for the current moment. Used to indicate whether the round has ended at the current time. This indicates the communication topology at the next moment;

[0051] Step 7: Steps 4, 5, and 6 above complete a trajectory sample collection process. The trajectory sample is saved to the playback buffer. The trajectory sample content is as follows: ,in, and These are the observations at the current time and the next time. For the action at the current moment, The reward for the current moment. Used to indicate whether the round has ended at the current time. This indicates the communication topology at the next moment. According to... Determine if the round has ended. If it has ended, jump back to step 2 to continue collecting data until the replay buffer pool data reaches the required level.

[0052] Step 8: Extract trajectory sample data, and calculate the advantage function of all agents for each trajectory sample using the generalized advantage function estimation method. ,in, This is the discount factor in GAE. As a discount factor for rewards, express Time difference error at any given moment The reward for the current moment. and The state value function for the current time step and the next time step;

[0053] Step 9: For the agent control policy network Using the policy given by the current policy network as a benchmark, the parameters of the policy network are iteratively updated to obtain a policy with better performance. The use of the dual policy network loss function can reduce the training variance while ensuring model convergence. The loss function for policy parameter updates is:

[0054]

[0055] in The ratio of the old to the new strategies. For the dominant function, The function is used to... Cut and limit Within the interval, For the hyperparameters of clipping, For policy entropy, This represents the corresponding strategy entropy coefficient.

[0056] For the agent control evaluation network, the parameters of the evaluation network are updated iteratively, and the loss function is:

[0057]

[0058] in, The state value function corresponding to the control policy network. For the actual value function

[0059] Its overall weighted loss function is:

[0060]

[0061] in, This is the Critic coefficient.

[0062] Step 10: For the communication weight generation strategy network Similarly, using the policy given by the current policy network as a benchmark, the parameters of the policy network are iteratively updated to obtain a policy with better performance. The loss function used is:

[0063]

[0064] in The ratio of the old to the new strategies. For the dominant function, The function is used to... Cut and limit Within the interval, For the hyperparameters of clipping, For policy entropy, This represents the corresponding strategy entropy coefficient.

[0065] For the evaluation network of the communication weight generation module, the loss function used is:

[0066]

[0067] in, Generate the state value function corresponding to the policy network for communication weights. This is the actual value function.

[0068] Its overall weighted loss function is:

[0069]

[0070] in, This is the Critic coefficient.

[0071] Step 11: Repeat steps 2 through 10 until the parameters of the policy network and the evaluation network finally converge. At this point, the algorithm has converged.

[0072] Step 12: In the online distributed execution phase, the model trained in a centralized manner is used as a benchmark. The trained parameters are loaded to verify the effectiveness of the algorithm. The multi-agent system quickly captures all escaping targets and communicates with each other to expand the perception domain and achieve better cooperation.

[0073] Test comparison

[0074] To verify the actual beneficial effects of the proposed method, we ran the algorithm under different scales and environments to compare its performance with that of other algorithms, as shown in the attached figures. Figures 3-6 As shown, where, Figure 3 and Figure 4 Comparison charts of average reward curves and average step length required to end a round in a 4vs2 chase / escape scenario. Figure 5 and Figure 6 A comparison chart of the average reward curve and the average step length required to end a round in a 7vs3 chase environment.

[0075] from Figure 3 and Figure 5 It can be seen that the algorithm proposed in this invention converges to the highest reward value under different pursuit scale environments compared to other algorithms. An important criterion for evaluating the performance of a pursuit strategy is the time required to complete the capture. Figure 4 and Figure 6 As can be seen, the algorithm proposed in this invention can complete the capture task in the shortest time. It should be noted that the steps and rewards are not strictly correlated. The number of steps most directly reflects the time required for capture, while the reward needs to comprehensively consider factors such as capture time, whether collisions occur between agents, and the agents' strategies. Results show that the algorithm proposed in this invention performs better in multi-agent pursuit environments with limited communication resources.

[0076] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-agent cooperative decision-making method based on deep reinforcement learning under limited communication resources, characterized in that, The method includes the following steps: Step 1: Initialize the environment parameters, perform orthogonal initialization of all network parameters in the Actor and Critic agents, and set the simplified dynamic equations of the agents; Step 2: Randomly assign coordinates to friendly agents and targets. initial velocity =0; Step 3: Set the reward function for cooperative pursuit This includes personal rewards related to the target distance. Team Rewards And collision penalties between intelligent agents And target capture reward , , , , This is the proportionality coefficient; Step 4: Each agent obtains local observations. The input is fed into the Encoder layer and GRU layer for feature extraction, and its output is used as the content of communication. , will observe Input to the weight generator policy network The importance weighting strategy for local observations is obtained, and the output is two parameters of a Beta distribution. and Importance weight parameters are obtained by sampling this distribution. ; Step 5: Each agent sends its observation importance weight to the team leader to request a communication channel. The leader collects the weights of all agents, constructs a conflict graph, and generates a conflict-free channel allocation strategy through a local search algorithm. This maximizes the use of limited channels without causing communication conflicts, and the allocation results are broadcast to the other agents. All agents communicate based on the channel allocation results at that moment. Step 6: After this round of communication is completed, each agent inputs all the data collected in this cycle into the multi-head self-attention layer for information integration and feature extraction, generating... And splice the original local information Input to the agent control policy network The output is sampled to obtain the final action, which is then applied to the agent to obtain the environmental feedback data at that moment. ,in, For the next observation time, The reward for the current moment. Used to indicate whether the round has ended at the current time. This indicates the communication topology at the next moment; Step 7: Steps 4, 5, and 6 above complete a trajectory sample collection process. The trajectory sample is saved to the playback buffer. The trajectory sample content is as follows: ,in, and These are the observations at the current time and the next time. For the action at the current moment, The reward for the current moment. Used to indicate whether the round has ended at the current time. Indicates the communication topology at the next moment, based on Determine if the round has ended. If it has ended, jump back to step 2 to continue collecting data until the replay buffer pool data reaches the required level. Step 8: Extract trajectory sample data, and calculate the advantage function of all agents for each trajectory sample using the generalized advantage function estimation method. ,in, This is the discount factor in GAE. As a discount factor for rewards, express Time difference error at any given moment The reward for the current moment. and The state value function for the current time step and the next time step; Step 9: For the agent control policy network Using the policy given by the current policy network as a benchmark, the parameters of the policy network are iteratively updated to obtain a policy with better performance. The use of the dual policy network loss function can reduce the training variance while ensuring model convergence. The loss function for policy parameter updates is: in The ratio of the old to the new strategies. For the dominant function, The function is used to... Cut and limit Within the interval, For the hyperparameters of clipping, For policy entropy, For the corresponding strategy entropy coefficient, For the agent control evaluation network, the parameters of the evaluation network are updated iteratively, and the loss function is: in, The state value function corresponding to the control policy network. For the actual value function Its overall weighted loss function is: in, The Critic coefficient. Step 10: For the communication weight generation strategy network Similarly, using the policy given by the current policy network as a benchmark, the parameters of the policy network are iteratively updated to obtain a policy with better performance. The loss function used is: in The ratio of the old to the new strategies. For the dominant function, The function is used to... Cut and limit Within the interval, For the hyperparameters of clipping, For policy entropy, For the corresponding strategy entropy coefficient, For the evaluation network of the communication weight generation module, the loss function used is: in, Generate the state value function corresponding to the policy network for communication weights. For the actual value function, Its overall weighted loss function is: in, The Critic coefficient. Step 11: Repeat steps 2 through 10 until the parameters of the policy network and the evaluation network finally converge. At this point, the algorithm has converged. Step 12: In the online distributed execution phase, the model trained in a centralized manner is used as a benchmark. The trained parameters are loaded to verify the effectiveness of the algorithm. The multi-agent system quickly captures all escaping targets and communicates with each other to expand the perception domain and achieve better cooperation.

2. The multi-agent cooperative decision-making method based on deep reinforcement learning under communication resource constraints as described in claim 1, characterized in that, The simplified dynamic equations for step 1 are as follows: in, For intelligent agents The first derivative of position, For intelligent agents The first derivative of velocity, For intelligent agents The control quantity, The noise term exists in the dynamic equation. For intelligent agents quality.

3. The multi-agent cooperative decision-making method based on deep reinforcement learning under communication resource constraints as described in claim 1, characterized in that, The reward function for step 3 is set as follows: in, Set of surviving targets Let be the distance function between the two.

4. The multi-agent cooperative decision-making method based on deep reinforcement learning under communication resource constraints as described in claim 1, characterized in that, The multi-channel allocation algorithm in step 5 is solved using a local search algorithm as follows: 5-1) Determine the channel number to be allocated. 5-2) Generate a conflict graph based on the adjacency matrix , 5-3) Obtain the conflict diagram Find the complement graph and solve for the maximum weighted clique of the complement graph. The weights of the points in the complement graph are the importance weight parameters obtained in step 4. , 5-4) Initialize the selectable node array An initial solution is generated, starting from a random point. A greedy algorithm is then used to find cliques containing that point to generate different initial solutions. 5-5) Update the collection that can be added. Deletable sets and commutative sets And record the optimal weight change for each operation. 5-6) Update the current solution based on the benefits of different operations, and apply the rules accordingly. Update 5-7) Calculate the weight value of the current solution. If the current solution is better than the local optimum, then use it as the local optimum. 5-8) After reaching the specified number of iterations, exit the current local search, compare the local optimum with the global optimum, and if the local optimum is better than the global optimum, adopt it as the global optimum and jump to 5-4) to start the search again. 5-9) When the longest search time is reached, output the global optimal solution as the final result, update the channel allocation result, and if there are still unallocated channels, jump to step 5-1).

5. The multi-agent cooperative decision-making method based on deep reinforcement learning under communication resource constraints as described in claim 1, wherein the calculation formula for the multi-head self-attention mechanism in step 6 is as follows: in It is the attention coefficient between agents i and j in the attention head m. It is the corresponding proportionality coefficient. For the query matrix, key matrix, and value matrix of the attention head m, Let i be the set of agents corresponding to the messages received by agent i. and These are the encoded hidden layer states of agents i and j, respectively.

Images