A multi-agent collaborative decision-making method based on teammate representation and incentive communication

Abstract

This invention discloses a multi-agent collaborative decision-making method based on teammate representation and incentive communication, which relates to the field of multi-agent reinforcement learning. The method includes executing for each agent: generating a historical trajectory representation based on the current observation value and the previous action through a multi-layer perceptron and a gated recurrent unit; generating a local action value function through a local action value function branch; generating a message to the teammate agent through a teammate representation module and a message generation module; summing the messages sent by all teammate agents and adding them to the local action value function as an incentive; and fusing the revised local action value functions of all agents through a hybrid neural network to obtain the team's overall value function and generate collaborative decision actions. Through teammate behavior prediction and a dynamic attention communication mechanism, the method improves the efficiency and decision-making accuracy of multi-agent collaboration, achieves low-redundancy sparse communication, and ensures system training stability and large-scale scalability.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent reinforcement learning, and more specifically to a multi-agent collaborative decision-making method based on teammate representation and incentive communication. Background Art

[0002] This invention focuses on the field of multi-agent collaborative decision-making, and its core application is in practical scenarios that require efficient distributed collaboration, such as drone cluster collaborative reconnaissance and formation flying.

[0003] In a locally observable environment, existing methods struggle to overcome information asymmetry between agents. While exchanging raw or shallowly processed observations (such as sensor data) through communication can expand individual perception, this approach increases the local observation dimension and state space of each agent, increasing the complexity and training difficulty of the policy network. Teammates must independently infer the intentions and behavior patterns of other agents from the vast amount of information they receive, which not only prolongs training convergence time but also delays decision-making, making it difficult to achieve fast and accurate goal-oriented collaboration.

[0004] Existing communication mechanisms generally lack specificity and selectivity. Common methods, such as broadcast or fixed-topology communication, result in agents sending information to a large number of, or even all, teammates, often containing information that is of little or no value to a specific teammate's decision-making. This extensive information transfer results in significant communication redundancy, leading to unnecessary bandwidth usage, energy consumption, and communication latency. This high-cost, inefficient communication model is particularly problematic in practical applications with limited bandwidth resources or a large number of agents (such as large-scale drone fleets and distributed sensor networks), becoming a major bottleneck restricting system scalability, real-time performance, and deployment feasibility.

[0005] To improve synergy, some existing technologies employ centralized decision-making or tightly coupled communication architectures, sacrificing the system's decentralization and scalability, making them difficult to adapt to large-scale groups of agents. Furthermore, at the training level, effective coordination of individual optimization with the team's overall goals, and maintaining training stability after the introduction of a communication module, remain significant challenges. The lack of a robust and scalable value function decomposition and optimization mechanism makes the algorithm prone to training oscillations, convergence difficulties, or poor final performance when faced with complex tasks and high-dimensional agent spaces, limiting its practical application.

[0006] Therefore, how to design a multi-agent collaborative decision-making method based on teammate representation and incentive communication, which can achieve high-precision, low-latency collaborative decision-making under local observable conditions, and at the same time have high communication efficiency, strong scalability and training stability is a problem that technical personnel in this field urgently need to solve. Summary of the Invention

[0007] In view of this, the present invention provides a multi-agent collaborative decision-making method based on teammate representation and incentive communication. It directly optimizes individual decision-making efficiency by accurately predicting teammate behavior and generating customized incentive messages; combines a dynamic sparse communication mechanism to reduce redundant overhead; and relies on an extensible collaborative framework to ensure system stability and scalability, thereby achieving efficient team collaboration.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A multi-agent collaborative decision-making method based on teammate representation and incentive communication includes performing the following steps for each agent i in the multi-agent system:

[0010] S1, based on current observations and the previous moment's action , through multi-layer perceptron and gated recurrent unit processing, generate historical trajectory representation ;

[0011] S2. Representing the historical trajectory Input to the local action value function branch and generate the local action value function through the multi-layer perceptron ; and characterize the historical trajectory Input to the teammate representation module and message generation module for processing, generating messages to be sent to teammate agents ;

[0012] S3. Sum the messages sent by all teammate agents and add the summation result as an incentive term to the local action value function In the above example, we obtain the modified local action value function ;

[0013] S4. The local action value functions of all agents are fused through a hybrid neural network to obtain the overall value function of the team. and generate collaborative decision-making actions.

[0014] Preferably, the S1 includes:

[0015] The current observation and the previous moment's action Concatenate into input vector , and then process the multi-layer perceptron and gated recurrent unit to obtain the hidden state of the gated recurrent unit, which is the historical trajectory representation .

[0016] Preferably, in S2, the teammate characterization module processing includes:

[0017] Characterized by historical trajectory Teammate number , calculate the multidimensional Gaussian distribution through the encoder The dimensions of the multidimensional Gaussian distribution are the same as the dimensions of the agent's action space, representing the probability of the teammate's action selection in each dimension. represents the action mean vector, represents the action variance vector;

[0018] Based on the multidimensional Gaussian distribution , sample to obtain teammate representation vector .

[0019] Preferably, the teammate characterization module is optimized by adopting a strategy of maximizing mutual information, including:

[0020] Define the mutual information objective, ;

[0021] Introducing variational autoencoders to calculate variational distributions , using KL divergence as the loss function:

[0022]

[0023] in, Including encoder parameters and variational parameters , Express expectations, represents the KL divergence, and p represents the conditional distribution.

[0024] Preferably, in S2, the message generation module processing includes:

[0025] Characterized by historical trajectory and teammate representation vector , through the fully connected layer processing and combined with the attention mechanism, the communication weight is generated ;

[0026] Representing the historical trajectory and teammate representation vector Perform multi-layer perceptron processing to generate message vectors , calculate the message sent to the teammate agent .

[0027] Preferably, in the message generation module, a communication weight is generated include:

[0028] Calculating query vectors and key vector ,in, 、 is the weight coefficient;

[0029] Calculate the normalized weights:

[0030]

[0031] in, represents the scaling factor, Represents the query vector The transpose of Represents the bond vector of the ith agent to the mth agent.

[0032] Preferably, the communication weight is generated , and also defines the entropy regularization loss:

[0033]

[0034] in, Indicates the message generation module parameters.

[0035] Preferably, the communication weight is generated , also includes setting the global communication threshold β

[0036] ;

[0037] when When , the communication link from agent i to agent j is cut off, and the weights of the remaining communication links are renormalized to ensure that the total communication weight Established.

[0038] Preferably, in S3, the modified local action value function Expressed as:

[0039]

[0040] in, Represents the message sent by agent j to agent i.

[0041] Preferably, in S4, the hybrid neural network adopts a QMIX framework, including:

[0042] Hypernetwork, used to receive global state and generate weight parameters of fully connected neural network;

[0043] A fully connected network that inputs the corrected local action value functions of all agents and outputs the team's overall value function , satisfying the monotonicity constraint .

[0044] It can be seen from the above technical solution that compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0045] 1. By building a teammate behavior prediction model and generating customized incentive messages, this method enables the intelligent agent to accurately understand the intentions of teammates and directly integrate incentive information into the local decision-making function. This mechanism can effectively guide team behavior coordination while avoiding increasing the complexity of individual strategies, significantly improving the accuracy and response timeliness of multi-agent collaborative decision-making in a local observation environment.

[0046] 2. A dynamic attention mechanism is used to generate differentiated communication weights for different teammates. Combined with entropy optimization and threshold control strategies, the intelligent agent only sends targeted incentive messages to key collaborative partners and automatically cuts off inefficient communication links. This design significantly reduces communication redundancy and bandwidth consumption while ensuring the effectiveness of team collaboration, and is suitable for distributed scenarios with limited resources.

[0047] 3. Based on the value function decomposition framework, individual decisions and team goals are integrated, and global optimization and local strategies are coordinated through a centralized training and decentralized execution mechanism. The hybrid network structure strictly maintains the monotonic correlation between individual values ​​and team values, ensuring that the algorithm has stable convergence characteristics and large-scale scalability in complex multi-agent systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 A flowchart of a multi-agent collaborative decision-making method based on teammate representation and incentive communication provided by an embodiment of the present invention;

[0050] Figure 2 A flow chart of a multi-agent collaborative decision-making method for drone swarms provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] Example 1;

[0053] like Figure 1As shown, this embodiment provides a multi-agent collaborative decision-making method based on teammate representation and incentive communication, including performing the following steps for each agent i in the multi-agent system:

[0054] S1, based on current observations and the previous moment's action , through multi-layer perceptron and gated recurrent unit processing, generate historical trajectory representation ;

[0055] S2. Representing the historical trajectory Input to the local action value function branch and generate the local action value function through the multi-layer perceptron ; and characterize the historical trajectory Input to the teammate representation module and message generation module for processing, generating messages to be sent to teammate agents ;

[0056] S3. Sum the messages sent by all teammate agents and add the summation result as an incentive term to the local action value function In the above example, we obtain the modified local action value function ;

[0057] S4. The local action value functions of all agents are fused through a hybrid neural network to obtain the overall value function of the team. and generate collaborative decision-making actions.

[0058] This method integrates key information into the individual decision-making process in an efficient manner by accurately predicting teammates' behavioral intentions and generating customized incentive messages, significantly improving the accuracy and real-time performance of team collaboration. At the same time, it uses a dynamic attention mechanism to achieve targeted screening and sparse control of communication links, significantly reducing communication overhead while ensuring collaborative effectiveness. Finally, relying on a scalable value function decomposition framework and a centralized training and decentralized execution mechanism, it ensures that the system has stable and efficient collaborative decision-making capabilities in large-scale distributed scenarios.

[0059] The following further describes each step in the above embodiment in detail;

[0060] In this embodiment S1, based on the current observation value and the previous moment's action , through multi-layer perceptron and gated recurrent unit processing, generate historical trajectory representation ; Specifically include:

[0061] The current observation and the previous moment's action Concatenate into input vector , and then process the multi-layer perceptron and gated recurrent unit to obtain the hidden state of the gated recurrent unit, which is the historical trajectory representation .

[0062] In this embodiment S2, the historical trajectory is represented Input to the local action value function branch and generate the local action value function through the multi-layer perceptron ; and characterize the historical trajectory Input to the teammate representation module and message generation module for processing, generating messages to be sent to teammate agents ;

[0063] The teammate characterization module processes:

[0064] Characterized by historical trajectory Teammate number , calculate the multidimensional Gaussian distribution through the encoder The dimensions of the multidimensional Gaussian distribution are the same as the dimensions of the agent's action space, representing the probability of the teammate's action selection in each dimension. represents the action mean vector, represents the action variance vector;

[0065] Based on the multidimensional Gaussian distribution , sample to get the teammate representation variable of agent i to agent j , as a prediction of teammates’ behavior, and is used to guide the generation of information that agent i transmits to agent j. By modeling teammates to predict their behavior, agents can clearly understand each other’s intentions.

[0066] Furthermore, in order to make the teammate behaviors predicted by the established teammate model more consistent with the actual actions taken by teammates, that is, to make the predictions more accurate, this embodiment updates and optimizes the teammate model by maximizing mutual information. Mutual information can calculate the correlation between two random variables, which can be understood as the amount of information contained in one random variable X about another random variable Y. The mutual information between random variables X and Y is defined as follows:

[0067]

[0068] The larger its value means the higher the degree of correlation between the two variables. When two random variables have no relationship, their mutual information is 0. Therefore, it can be used to measure the quality of one agent's prediction of another agent's behavior.

[0069] This embodiment samples the predicted value of teammate behavior, i.e., the distribution of the teammate modeling model As a random variable, the observed teammate behavior Treat it as another random variable, and then calculate the mutual information between them to represent the correlation between the two, and update and optimize the teammate model by maximizing the mutual information.

[0070] The mutual information between the two can be expressed as follows:

[0071]

[0072] Among them, entropy Indicates that only the historical trajectory of agent i is known , the uncertainty of the teammate representation variable when the teammate number is j, entropy For the known The uncertainty of the representation variable when the teammate number and the teammate's real action are known. The difference between the two is the correlation between the predicted representation variable and the real action. The larger the difference, the more accurate the prediction. Comparison It cannot provide any additional information, that is, when the two are equal, the value of mutual information will be 0. Maximizing mutual information is equivalent to reducing the deviation between the predicted value of the teammate representation model and the actual action value under the condition of knowing the local information of agent i. q enables the teammate representation model of the agent to more accurately understand and predict the behavior of teammates, thereby achieving better cooperation between teams.

[0073] When calculating mutual information, the conditional distribution Calculation is performed, but in reality it is difficult to calculate the conditional distribution Calculate. Maximizing mutual information can be achieved by maximizing the lower bound of mutual information. In this embodiment, variational inference is used to calculate the lower bound of mutual information: a parameter is introduced as The variational autoencoder is used to calculate the variational distribution , and use this variational distribution to approximate the conditional distribution , the variational encoder is essentially a recurrent neural network. Then the KL divergence is used to measure the similarity between the two distributions, and the expected negative value of the KL divergence is used as the lower bound of the mutual information, as shown in the following formula:

[0074]

[0075] As can be seen from the above formula, maximizing mutual information is equivalent to minimizing KL divergence. Therefore, this embodiment uses KL divergence as the loss function of the teammate characterization module for updating and optimizing the characterization module:

[0076]

[0077] in, Including encoder parameters and variational parameters , Express expectations, represents the KL divergence, and p represents the conditional distribution. Through continuous training and updating, the teammate representation model will be continuously optimized, and the predicted value will approach the real action.

[0078] Furthermore, the message generation module processing includes:

[0079] Characterized by historical trajectory and teammate representation vector , through the fully connected layer processing and combined with the attention mechanism, the communication weight is generated ;

[0080] Representing the historical trajectory and teammate representation vector Perform multi-layer perceptron processing to generate message vectors , calculate the message sent to the teammate agent .

[0081] Further, generate communication weight include:

[0082] Calculating query vectors and key vector ,in, 、 is the weight coefficient;

[0083] Calculate the normalized weights:

[0084]

[0085] in, represents the scaling factor, Represents the query vector The transpose of Represents the bond vector of the ith agent to the mth agent.

[0086] After introducing the communication weights, in order to make the agents more centralized during communication, that is, to send information only to other agents that are necessary, discard most invalid communications, reduce the communication overhead to achieve efficient and sparse communication, this embodiment further introduces a sparsity regularization term. By optimizing the entropy of the classification distribution composed of the communication weights, efficient communication is achieved. The entropy loss function is as follows:

[0087]

[0088] In the above formula, are the parameters of the message generator. By minimizing the entropy loss, the message generation module can generate communication weights with lower uncertainty.

[0089] In addition, in order to further cut off the useless redundant communication links between agents and reduce the cost of communication, this embodiment proposes the concept of threshold for communication weight, that is, for a single agent, the sum of its communication weight as an information source is 1, that is, , set the communication threshold ,when When , the communication link from i to j will be cut off. By setting the communication threshold, the communication overhead of multi-agents is further reduced, the efficiency of information interaction is improved, and the collaborative decision-making level of multi-agents is enhanced.

[0090] In this embodiment S3, the messages sent by all teammate agents are summed up, and the summation result is added as an incentive item to the local action value function In the above example, we obtain the modified local action value function ; Modified local action value function Expressed as:

[0091]

[0092] in, Represents the message sent by agent j to agent i.

[0093] In this embodiment S4, the local action value functions of all agents are fused through a hybrid neural network to obtain the team's overall value function And generate collaborative decision-making actions; specifically, the hybrid neural network adopts the QMIX framework, including:

[0094] Hypernetwork, used to receive global state and generate weight parameters of fully connected neural network;

[0095] A fully connected network that inputs the corrected local action value functions of all agents and outputs the team's overall value function , satisfying the monotonicity constraint .

[0096] The collaborative decision-making method provided in this embodiment generates accurate teammate representation vectors through a teammate behavior prediction model, and dynamically generates customized incentive messages and sparse communication weights based on the attention mechanism; the incentive information is directly integrated into the individual value function to optimize local decisions, and entropy regularization and threshold control are combined to achieve efficient communication link pruning; finally, relying on the QMIX framework, individual decisions are integrated into team goals, significantly improving the collaborative efficiency and scalability of multi-agent systems in local observable environments while ensuring training stability.

[0097] Example 2;

[0098] To address the problems faced by drone swarms in forest fire reconnaissance, such as limited dynamic environment perception, tight communication bandwidth, and inefficient collaborative decision-making, this embodiment applies a multi-agent collaborative decision-making method based on teammate representation and incentive communication to a swarm of four drones. This method implements a closed-loop framework of multi-source perception data fusion → probabilistic representation of teammate behavior → attention-weighted communication → global monotonic optimization decision-making, effectively addressing the key challenges of collaborative path planning and obstacle avoidance in the complex environment of a fire scene.

[0099] In this embodiment, four UAVs are required to coordinate and perform forest fire reconnaissance missions. The goal is to dynamically cover the fire area, locate the fire point in real time and avoid the thick smoke area. For own location / GPS signal strength / visible fire point coordinates / surrounding smoke density, action For flight direction (8 azimuths), speed gear (3 gears), the bandwidth is limited, and low-value messages need to be dynamically filtered.

[0100] like Figure 2 As shown, the specific implementation steps include:

[0101] 1) Historical trajectory representation generation;

[0102] Each drone concatenates the current observation value (such as the coordinates of the fire point and the smoke density) with the action at the previous moment into a vector. Through two layers of MLP (128-dimensional hidden layer) and GRU (64-dimensional hidden state), the historical trajectory representation is output. When the UAV detects high smoke concentration multiple times on one side of the fire scene, the historical trajectory representation It will encode the high smoke risk pattern on the east side, and the GRU's temporal memory will enable the drone to automatically avoid the eastward path in subsequent decision-making, thereby improving its survival rate.

[0103] 2) Teammate intention modeling and communication;

[0104] Based on historical trajectory representation Teammate number Generate Gaussian distribution and sample to obtain teammate representation vectors ; and by communication weight Screen key teammates, cut off the communication link when the weight is lower than the threshold β, and generate a message When the drone finds a new fire point, it will only send high-weight messages to teammates near the area ( >0.7), ignoring teammates far away from the fire scene and reducing some redundant communications.

[0105] 3) Local decision revision;

[0106] Local action-value function Evaluate the action benefit, receive the teammate's message and sum it to get the incentive item, which is corrected to ; The drone originally planned to fly to the northeast received a smoke warning ( =-0.8), the correction value dropped from +1.5 to +0.7, triggering a turn to the northwest safe path.

[0107] 4) Global collaborative decision-making;

[0108] The QMIX hybrid network inputs all correction value functions and the global fire scene status and outputs the team value function And generate collaborative actions to meet the monotonicity constraint The decision results dynamically allocate drone roles: drone 1 covers the unsurveilled area, drone 2 supports the fire point, and drones 3 / 4 block the diffusion path, reducing mission completion time by 35%.

[0109] This implementation demonstrates the engineering effectiveness of the technical solution. Teammate intention modeling improves collaborative coverage by 40%, attention communication reduces bandwidth consumption by 60%, and QMIX global optimization ensures consistent firefighting targets. This system provides a patented distributed decision-making paradigm for disaster relief scenarios, combining real-time and robustness.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. References to the same or similar parts between the various embodiments are sufficient. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For relevant parts, refer to the method description.

[0111] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-agent collaborative decision-making method based on teammate representation and incentive communication, characterized in that: It involves performing the following steps for each agent i in the multi-agent system: S1, based on current observations and the previous moment's action , through multi-layer perceptron and gated recurrent unit processing, generate historical trajectory representation ; S2. Representing the historical trajectory Input to the local action value function branch and generate the local action value function through the multi-layer perceptron ; And represent the historical trajectory Input to the teammate representation module and message generation module for processing, generating messages to be sent to teammate agents ; Teammate characterization module processing includes: characterization based on historical trajectory Teammate number , calculate the multidimensional Gaussian distribution through the encoder The dimensions of the multidimensional Gaussian distribution are the same as the dimensions of the agent's action space, representing the probability of the teammate's action selection in each dimension. represents the action mean vector, Represents the action variance vector; based on the multidimensional Gaussian distribution , sample to obtain teammate representation vector In the teammate representation module, the maximization mutual information strategy is used for optimization, including: Define the mutual information objective, ; Introduce variational autoencoder to calculate variational distribution , using KL divergence as the loss function: in, Including encoder parameters and variational parameters ; S3. Sum the messages sent by all teammate agents and add the summation result as an incentive term to the local action value function In the above example, we obtain the modified local action value function : in, represents the message sent by agent j to agent i; S4. The local action value functions of all agents are fused through a hybrid neural network to obtain the overall value function of the team. and generate collaborative decision-making actions.

2. A multi-agent collaborative decision-making method based on teammate representation and incentive communication according to claim 1, characterized in that: Said S1 comprises: The current observation and the previous moment's action Concatenate into input vector , and then process the multi-layer perceptron and gated recurrent unit to obtain the hidden state of the gated recurrent unit, which is the historical trajectory representation .

3. The multi-agent collaborative decision-making method based on teammate representation and incentive communication according to claim 1 is characterized in that: In S2, the message generation module processing includes: Characterized by historical trajectory and teammate representation vector , through the fully connected layer processing and combined with the attention mechanism, the communication weight is generated ; Representing the historical trajectory and teammate representation vector Perform multi-layer perceptron processing to generate message vectors , calculate the message sent to the teammate agent .

4. The multi-agent collaborative decision-making method based on teammate representation and incentive communication according to claim 3 is characterized in that: In the message generation module, a communication weight is generated include: Calculating query vectors and key vector ,in, 、 is the weight coefficient; Calculate the normalized weights: in, represents the scaling factor, Represents the query vector The transpose of Represents the bond vector of the ith agent to the mth agent.

5. The multi-agent collaborative decision-making method based on teammate representation and incentive communication according to claim 4 is characterized in that: Generating communication weight , and also defines the entropy regularization loss: in, Indicates the message generation module parameters.

6. The multi-agent collaborative decision-making method based on teammate representation and incentive communication according to claim 4 is characterized in that: Generating communication weight , also includes setting the global communication threshold β ； when When , the communication link from agent i to agent j is cut off, and the weights of the remaining communication links are renormalized to ensure that the total communication weight Established.

7. The multi-agent collaborative decision-making method based on teammate representation and incentive communication according to claim 1 is characterized in that: In S4, the hybrid neural network adopts the QMIX framework, including: Hypernetwork, used to receive global state and generate weight parameters of fully connected neural network; A fully connected network that inputs the corrected local action value functions of all agents and outputs the team's overall value function , satisfying the monotonicity constraint .

Images