Heterogeneous multi-agent cooperation method and system based on role-driven communication and heterogeneous perception enhancement

Abstract

The application provides a heterogeneous multi-agent cooperation method and system based on role-driven communication and heterogeneous perception enhancement. By constructing a potential role space of heterogeneous perception, adaptive division of labor of strategies is realized; by using a role information driven attention communication mechanism, information interaction efficiency is improved; and by combining type enhanced value decomposition and multi-head intrinsic exploration mechanism, the problems of communication redundancy, credit allocation deviation and sparse reward in a strong heterogeneous environment are alleviated. The method is suitable for cooperation scenes where heterogeneous action spaces and heterogeneous observation features coexist, and can improve the convergence speed and overall performance in complex cooperation tasks.

Description

Technical Field

[0001] This invention belongs to the technical fields of multi-agent reinforcement learning, dynamic role modeling, and communication control, specifically involving a heterogeneous multi-agent collaboration method and system based on role-driven communication and heterogeneous perception enhancement. Background Technology

[0002] With the rapid development of deep reinforcement learning technology, multi-agent systems (MAS) have been widely applied in complex scenarios such as StarCraft, robot collaboration, drone swarming, and intelligent transportation. In real-world collaborative tasks, the agents participating in the collaboration often have different attribute types, action capabilities, and perception ranges, forming typical heterogeneous multi-agent systems. Compared with homogeneous systems, agents in heterogeneous systems exhibit significant differences in observation dimensions, action spaces, and task division, placing higher demands on collaborative learning algorithms.

[0003] In existing multi-agent collaborative learning methods, using a single shared policy network makes it difficult to simultaneously fit multiple behavioral patterns, easily leading to policy homogenization and rigid role division. Even with the introduction of one-hot IDs for differentiation, the lack of explicit role modeling often results in rigid role division and an inability to dynamically adjust responsibilities based on the battlefield situation in long-term collaborative tasks. Furthermore, in environments with limited communication bandwidth and complex interactions, traditional broadcast communication mechanisms typically ignore the "specificity of information value for a particular role," causing the channel to be filled with a large amount of redundant information. This "full broadcast" mode not only increases the computational and transmission burden of the system but also easily introduces noise interference, reducing the efficiency of transmitting key collaborative information and making it difficult for agents to reach consensus in partially observable environments.

[0004] In recent years, some research has attempted to address these issues by introducing the concept of roles. For example, ROMA (ICML 2020) introduces a random role embedding space for each agent, ensuring role identifiability through regularization, enabling agents to emerge with dynamic division of labor. However, ROMA lacks an explicit communication mechanism, limiting its collaborative efficiency in partially observable environments. ROCO (Artificial Intelligence Journal, 2025) proposed a role-oriented two-layer communication architecture, utilizing mutual information for message prediction and filtering, thus improving communication efficiency. However, ROCO's role allocation relies on mutual information computation, failing to utilize variational inference for role modeling or incorporate role information into value decomposition and exploration incentive stages. The overall framework remains focused on communication optimization, failing to achieve seamless integration of role information across the entire communication chain.

[0005] Furthermore, in the joint value assessment and credit allocation process of collaborative tasks, traditional value decomposition methods (such as QMIX) typically generate hybrid network weights based solely on the global state, failing to explicitly consider the impact of agent type differences, dynamic collaborative roles, and local contextual information on the joint value. Therefore, inaccurate credit allocation is prone to occur in highly heterogeneous scenarios. Especially in complex collaborative tasks with sparse reward signals, basic exploration mechanisms often struggle to drive agents to discover key collaborative behaviors in a timely manner, thus affecting the algorithm's convergence speed and final performance.

[0006] Therefore, for highly heterogeneous multi-agent collaborative tasks, a collaborative learning method is needed that can explicitly model heterogeneous priors, support dynamic role evolution, achieve efficient directional communication, and enhance exploration capabilities under sparse rewards, so as to improve the collaborative efficiency and adaptability of the system in complex dynamic environments. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a heterogeneous multi-agent collaboration method and system based on role-driven communication and heterogeneous perception enhancement. By constructing a collaborative architecture that includes role inference, structure modulation, directional communication and type-enhanced value decomposition, the invention improves the adaptability of division of labor, the targeting of communication and the accuracy of value assessment in heterogeneous collaboration.

[0008] In a first aspect, this invention proposes a heterogeneous multi-agent cooperation method based on role-driven communication and heterogeneous perception enhancement, which includes the following steps: S1. Using a variational autoencoder architecture, continuous latent role variables are inferred from the local observations of each agent, and the individual policy network is linearly modulated based on the latent role variables to generate role-specific individual policies. S2. Using the potential role variables as query vectors, teammate information is aggregated by introducing a type-pairing biased attention mechanism, and the aggregation message is redundancy suppressed by using role-aware gating to obtain the filtered communication representation. S3. During the intensive training phase, basic mixing parameters and incremental parameters are generated by the basic supernetwork and the type modulation supernetwork, respectively. The final mixing network parameters are obtained by global gating scalar fusion. The individual action values ​​are monotonically mixed to achieve credit allocation. S4. Construct a multi-head random network distillation module, set independent prediction heads for different types of agents, generate intrinsic rewards and fuse them with extrinsic rewards, and jointly optimize the parameters of each module.

[0009] Preferably, step S1 specifically includes: S11, the intelligent agent i Local observation at the current moment and type identifier Input the role prior encoder to generate the prior distribution. ; Represents intelligent agents Static type identifiers, such as military unit; Represents intelligent agents At any moment t The dynamic potential role vector; S12, the intelligent agent i Local movements - observing historical trajectories Introducing a role posterior encoder to generate the posterior distribution during the training phase. ; S13. Using the KL divergence constraint, the prior distribution and posterior distribution are kept consistent. The calculation formula is as follows: And extract continuous latent role vectors through reparameterized sampling. ; Indicating in the historical trajectory τ Mathematical expectation on Indicates the Kullback-Leibler divergence. Describe the prior distribution, This represents an approximate posterior distribution; S14, Transfer the potential role vector Input the character decoder to generate feature linear modulation parameters—scaling coefficients. Peaceful transfer Hidden features of the intermediate layer of the individual value network Perform modulation: This yields intermediate features after character specialization.

[0010] Preferably, step S2 specifically includes: S21. Construct a role-driven query-key-value model with intelligent agents. Potential role vectors as query vector With other intelligent agents Observation coding As a key Sum ,in , , To share the projection matrix; S22. Introduce learnable type-pairing bias terms. A type-related correction term is added to the attention weight calculation to aggregate the communication messages. : ; Represent other intelligent agents j The transpose of the key vector This represents the hidden dimension of the key-value vector in the multi-head attention mechanism.

[0011] S23. For intelligent agents At any moment t Gated networks are based on local embedding and potential role vectors Generate dynamic gating coefficients : where σ is the sigmoid function; S24. Apply the gating coefficient to the aggregated message to obtain the final communication representation. And input the action value assessment branch.

[0012] Further preferred, step S3 specifically includes: S31, the basic hypernetwork is based on the global state s Generate basic mixing parameters and And monotonicity is guaranteed through nonnegative mapping; S32, The type modulation supernetwork generates incremental parameters Δ based on the type context of heterogeneous teams. W and Δ b ; S33. Introduce globally learnable gating scalars. g , The final hybrid network parameters are synthesized according to the following formula: .

[0013] Preferably, step S4 specifically includes: S41. Construct the multi-head random network distillation module for... K Each intelligent agent is equipped with an independent prediction head to calculate the prediction error. : ; Indicates the first k The output feature vector of each prediction head Indicates the first k The output feature vector of each target head s Indicates the global environment state; S42. Integrate intrinsic rewards and extrinsic environmental rewards to adjust the total training reward. ; Indicates the intrinsic reward coefficient; This refers to external environmental rewards; Represents intrinsic reward; Clip(·) and Normalize(·) represent function operations that numerically truncate and normalize scalar data, respectively; S43. Joint optimization includes temporal difference loss. L TD KL divergence lossL KL and RND predicted loss L RND The total loss function is: .

[0014] A further preferred RND loss function is: ;in, Indicates the first in the current scene k The weighting of the number of intelligent agents. This indicates the stop gradient operator, used to ensure that the parameters of the target network remain fixed and are not updated during backpropagation.

[0015] Further preferably, the global gated scalar g The annealing strategy employs linear or exponential growth, making early training... g Approaching 0, the model prioritizes learning basic collaborative patterns in the later stages of training. g Approaching 1, the model uses type context to fine-tune credit allocation.

[0016] Secondly, embodiments of the present invention provide a heterogeneous multi-agent collaborative system based on role-driven communication and heterogeneous perception enhancement, comprising: The role inference module is used to infer continuous latent role variables from the local observations of each agent using a variational autoencoder architecture, and to perform feature linear modulation on the individual policy network based on the latent role variables to generate role-specific individual policies. The directional communication module is used to aggregate teammate information by introducing a type-pairing bias attention mechanism, using the potential role variables as query vectors, and to suppress redundancy in the aggregated messages by using role-aware gating to obtain a filtered communication representation. The value assessment module is used to generate basic mixing parameters and incremental parameters through the basic supernetwork and type modulation supernetwork respectively during the centralized training phase, and obtain the final mixing network parameters by global gating scalar fusion, so as to monotonically mix the individual action values ​​and realize credit allocation. The intrinsic exploration module is used to construct a multi-head random network distillation module, which sets independent prediction heads for different types of agents, generates intrinsic rewards and merges them with extrinsic rewards, and jointly optimizes the parameters of each module.

[0017] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0019] Fifthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Dynamic division of labor and adaptability: By constructing a heterogeneous perception role space, the intelligent agent can form a dynamic division of labor mode according to the real-time collaboration goal, which breaks through the limitations of rigid division of labor and homogeneous strategy caused by the traditional parameter sharing mechanism in highly heterogeneous scenarios.

[0021] (2) Communication efficiency and targeting: By leveraging potential roles to drive attention communication and combining type matching bias and gating mechanisms, information is distributed "on demand", significantly reducing invalid broadcasts and redundant information, and improving the collaborative efficiency of heterogeneous teams in partially observable environments.

[0022] (3) Accuracy of value assessment: The type-enhanced gating hybrid mechanism allows the system to perceive heterogeneous contexts while maintaining monotonicity constraints, solving the problem of credit allocation bias in complex heterogeneous formations.

[0023] (4) Better convergence performance: Under the guidance of dense intrinsic rewards provided by multi-head random network distillation (RND), the system exhibits faster convergence speed and higher task win rate in strongly heterogeneous and sparse reward scenarios. Experimental data show that in MMM2 strongly heterogeneous maps, the average win rate of this invention is improved by about 10.3 percentage points compared with the baseline QMIX algorithm, and the training fluctuation standard deviation is the smallest. Attached Figure Description

[0024] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0025] Figure 1 This is an exemplary device architecture diagram in which an embodiment of the present invention can be applied; Figure 2 This is a flowchart illustrating an embodiment of the heterogeneous multi-agent collaboration method based on role-driven communication and heterogeneous perception enhancement according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the overall process of a specific embodiment of the present invention; Figure 4 This is a schematic diagram of the overall module structure of the RH-QMIX of the present invention; Figure 5 This is a structural diagram of the Heterogeneous Perception Role Inference and Hyperstrategy Generation Module (HRIG) of the present invention; Figure 6 This is a structural diagram of the role-driven attention communication module (RDAC) of the present invention; Figure 7 A diagram of the Type Enhanced Value Assessment and Multi-Headed Intrinsic Exploration (TEVE) module for this invention; Figure 8 This is a schematic diagram of the architecture of a heterogeneous multi-agent collaborative system based on role-driven communication and heterogeneous perception enhancement, according to an embodiment of the present invention. Figure 9 This is a schematic diagram of the experimental scenario of the present invention (MMM2 map); Figures 10(a), 10(b), 10(c), and 10(d) are curves showing the comparison and evaluation results of key hyperparameters in the MMM2 scenario of the present invention. Figures 11(a), 11(b), and 11(c) are comparison charts of the win rate curves of the ablation experiments of each module of the present invention; Figure 12 This is a curve showing the win rate comparison between the present invention and the baseline algorithm; Figure 13 This is a heatmap of attention communication (type-level and individual-level attention weight distribution) of the present invention. Figure 14 This is a schematic diagram of the structure of a computer device suitable for implementing electronic devices according to embodiments of the present invention. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0027] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] Figure 1An exemplary system architecture 100 is shown, which can be applied to the heterogeneous multi-agent collaboration method based on role-driven communication and heterogeneous perception enhancement, or the heterogeneous multi-agent collaboration system based on role-driven communication and heterogeneous perception enhancement, according to embodiments of the present invention.

[0029] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0030] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0031] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0032] Server 105 can be a server that provides various services, such as a background information processing server that processes verification request information sent by terminal devices 101, 102, and 103. The background information processing server can analyze and process the received verification request information and obtain the processing results.

[0033] It should be noted that the heterogeneous multi-agent collaboration method based on role-driven communication and heterogeneous perception enhancement provided in this embodiment of the invention is generally executed by server 105, and correspondingly, the heterogeneous multi-agent collaboration system based on role-driven communication and heterogeneous perception enhancement is generally set in server 105. Furthermore, the heterogeneous multi-agent collaboration method based on role-driven communication and heterogeneous perception enhancement provided in this embodiment of the invention is generally executed by terminal devices 101, 102, and 103, and correspondingly, the heterogeneous multi-agent collaboration system based on role-driven communication and heterogeneous perception enhancement is generally set in terminal devices 101, 102, and 103.

[0034] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (for example, to provide distributed services), or as a single software program or multiple software modules; no specific limitations are made here.

[0035] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. If the data being processed does not need to be retrieved remotely, the above architecture may not include a network, requiring only servers or terminal devices.

[0036] This invention is mainly applied to solve the core technical problems of rigid division of labor, communication redundancy, and inaccurate credit allocation caused by the significant heterogeneity among agents when distributed autonomous systems perform complex collaborative tasks.

[0037] In a first aspect, embodiments of the present invention disclose a heterogeneous multi-agent collaboration method based on role-driven communication and heterogeneous perception enhancement, such as... Figure 2 As shown, the method includes the following steps: S1. Using a variational autoencoder architecture, continuous latent role variables are inferred from the local observations of each agent, and the individual policy network is linearly modulated based on the latent role variables to generate role-specific individual policies. Specifically, step S1 includes: S11, the intelligent agent i Local observation at the current moment and type identifier Input the role prior encoder to generate the prior distribution. ; Represents intelligent agents Static type identifiers, such as military unit; Represents intelligent agents At any moment t The dynamic potential role vector; S12, the intelligent agent i Local movements - observing historical trajectories Introducing a role posterior encoder to generate the posterior distribution during the training phase. ; S13. Using the KL divergence constraint, the prior distribution and posterior distribution are kept consistent. The calculation formula is as follows: And extract continuous latent role vectors through reparameterized sampling. ; Indicates the historical trajectory τ Mathematical expectation on Indicates the Kullback-Leibler divergence. Describe the prior distribution, This represents an approximate posterior distribution; S14, Transfer the potential role vector Input the character decoder to generate feature linear modulation parameters—scaling coefficients. Peaceful transfer Hidden features of the intermediate layer of the individual value network Perform modulation: This yields the intermediate features after character specialization.

[0038] S2. Using the potential role variables as query vectors, teammate information is aggregated by introducing a type-pairing biased attention mechanism, and the aggregation message is redundancy suppressed by using role-aware gating to obtain the filtered communication representation. Specifically, step S2 includes: S21. Construct a role-driven query-key-value model with intelligent agents. Potential role vectors as query vector With other intelligent agents Observation coding As a key Sum ,in , , To share the projection matrix; S22. Introduce learnable type-pairing bias terms. A type-related correction term is added to the attention weight calculation to aggregate the communication messages. : ; Other intelligent agents j The transpose of the key vector This represents the hidden dimension of the key-value vector in the multi-head attention mechanism.

[0039] S23, Regarding intelligent agents At any moment t Gated networks are based on local embedding and potential role vectors Generate dynamic gating coefficients : , where σ is the sigmoid function; S24. Apply the gating coefficient to the aggregated message to obtain the final communication representation. And input the action value assessment branch.

[0040] S3. During the intensive training phase, basic mixing parameters and incremental parameters are generated by the basic supernetwork and the type modulation supernetwork, respectively. The final mixing network parameters are obtained by global gating scalar fusion. The individual action values ​​are monotonically mixed to achieve credit allocation. Specifically, step S3 includes: S31, the basic hypernetwork is based on the global state s Generate basic mixing parameters and And monotonicity is guaranteed through nonnegative mapping; S32, The type modulation supernetwork generates incremental parameters Δ based on the type context of heterogeneous teams. W and Δ b ; S33. Introduce globally learnable gating scalars. g , The final hybrid network parameters are synthesized according to the following formula: .

[0041] In this embodiment, the global gated scalar g The annealing strategy employs linear or exponential growth, making early training... g Approaching 0, the model prioritizes learning basic collaborative patterns in the later stages of training. g Approaching 1, the model uses type context to fine-tune credit allocation.

[0042] S4. Construct a multi-head random network distillation module, set independent prediction heads for different types of agents, generate intrinsic rewards and fuse them with extrinsic rewards, and jointly optimize the parameters of each module.

[0043] Specifically, step S4 includes: S41. Construct the multi-head random network distillation module for... K Each intelligent agent is equipped with an independent prediction head to calculate the prediction error. : ; Indicates the first k The output feature vector of each prediction head Indicates the first k The output feature vector of each target head s Indicates the global environment state; S42. Integrate intrinsic rewards and extrinsic environmental rewards to adjust the total training reward. ; Indicates the intrinsic reward coefficient; This refers to external environmental rewards; Represents intrinsic reward; Clip(·) and Normalize(·) represent function operations that numerically truncate and normalize scalar data, respectively; S43, Joint optimization includes temporal difference loss. L TD KL divergence loss L KL and RND predicted loss L RND The total loss function is: .

[0044] In this embodiment, the RND prediction loss function is: ;in, Indicates the first in the current scene k The weighting of the number of intelligent agents. This indicates the stop gradient operator, used to ensure that the parameters of the target network remain fixed and are not updated during backpropagation.

[0045] In a specific embodiment, such as Figure 3 As shown, this invention discloses a heterogeneous multi-agent cooperation method based on role-driven communication and heterogeneous perception enhancement, comprising: I. Role Inference and Policy Structure Modulation in Heterogeneous Perception: Continuous latent role variables are inferred from the local observations of each agent using a variational autoencoder architecture. Prior and posterior distributions are constrained by KL divergence. The feature linear modulation mechanism is used to scale and translate the intermediate layer features of the individual policy network according to the latent role variables to generate role-specific individual policies. II. Role-driven attention communication and message filtering: Using the latent role variables as query vectors, a query-key-value attention model is constructed to aggregate the observation information of other agents. A learnable type pairing bias matrix is ​​introduced into the attention weights to encode the cooperative priors between heterogeneous agent types. A gating network is used to dynamically generate gating coefficients based on local embedding and the latent role variables to suppress redundancy in the aggregated communication messages and obtain the filtered communication representation. III. Value Decomposition and Credit Allocation with Type Enhancement: During the intensive training phase, a hybrid network is constructed to monotonically mix the values ​​of individual actions. The parameters of the hybrid network are jointly generated by the basic supernetwork and the type modulation supernetwork. The basic supernetwork generates basic weights and biases based on the global state, while the type modulation supernetwork generates incremental weights and incremental biases based on the type context of heterogeneous teams. The basic parameters and incremental parameters are then fused through a globally learnable gating scalar to obtain the final hybrid network parameters, thereby achieving credit allocation while maintaining monotonicity constraints. IV. Multi-head Intrinsic Exploration and Joint Optimization: Construct a multi-head random network distillation module, set independent prediction heads for different types of agents, calculate the prediction error of each prediction head for state features and use it as intrinsic reward, and weight and fuse intrinsic reward with environmental extrinsic reward to form total reward. Jointly optimize temporal difference loss, KL divergence loss and random network distillation prediction loss, and update the parameters of each module.

[0046] It should be noted that in heterogeneous multi-agent collaborative tasks, different agents in the system have different attribute identifiers, action spaces, and observation capabilities. Taking StarCraft II collaborative tasks as an example, melee units (Zealots), ranged units (Stalkers), and medical support units (Medical Ships) undertake different responsibilities. Without explicitly modeling their type differences, it is difficult to achieve effective division of labor and collaborative decision-making.

[0047] In heterogeneous multi-agent collaborative tasks, the set of agents I ={1,…, n The agents are categorized into K different functional types. This invention explicitly introduces a set of agent types into the standard Dec-POMDP framework. ={ φ 1, φ 2, .., φ K}, and potential roles ρ i As an intermediate variable, it enables the agent to complete adaptive collaboration in each stage of "role inference - message interaction - action execution".

[0048] The joint optimization objective of the system is to maximize the cumulative return of the discount, as shown in the formula. As shown, where γ As a discount factor, r t for t A joint reward signal at any moment.

[0049] To enable the system to handle dynamic division of labor, this embodiment constructs the RH-QMIX core architecture, which consists of a heterogeneous perception role inference and super-policy generation module (HRIG), a role-driven attention communication module (RDAC), and a type-enhanced value decomposition and multi-head intrinsic exploration module (TEVE).

[0050] Figure 4 This is a diagram showing the overall module structure of the RH-QMIX system of the present invention. The following detailed description of each module is provided in conjunction with specific implementation steps.

[0051] Figure 5 This is a structural diagram of the Heterogeneous Perception Role Inference and Hyperstrategy Generation Module (HRIG). The following section combines... Figure 5This section details how the HRIG module enables dynamic specialization of strategies.

[0052] First, obtain the intelligent agent. i Input information at the current moment, including local observations Type identifier And the actions from the previous moment. This information is input into the HRIG module. This module employs a variational autoencoder structure, whose core logic lies in modeling the unobservable "cooperative intention" as random variables in a continuous latent space. Through KL divergence constraints, This ensures that the prior distribution on which the execution end relies is aligned with the posterior knowledge that the training end can obtain.

[0053] Subsequently, a FiLM modulation layer is introduced to act on individual Q-networks. Compared to direct splicing, FiLM modulation performs element-wise affine transformations on the intermediate features h, allowing the same set of backbone parameters to exhibit offensive behavior in the "assault role" state and defensive behavior in the "cover role" state, thus achieving deep specialization of the strategy at the physical meaning level. This mechanism enables the same backbone network to generate different control strategies based on role differences, thereby alleviating the problems of strategy homogenization and rigid division of labor in highly heterogeneous scenarios. .

[0054] Specifically, local observation and type identifier The input is a character prior encoder (a neural network composed of multilayer perceptrons), which outputs the mean and variance of the prior distribution, thus generating the prior distribution. ; Represents intelligent agents i Static type identifiers, such as military unit; Represents intelligent agents i The dynamic potential role vector at time t. The prior distribution characterizes the role tendency formed by the agent at the current moment based on immediate local information. For example, a fanatic with multiple enemy units in sight may tend to take the "assault" role.

[0055] At the same time, during the training phase, the intelligent agent i Complete local action - observation of historical trajectory (Including observations, actions, and rewards from the past several steps) A role posterior encoder is introduced. The posterior encoder also employs a multilayer perceptron structure, outputting the mean and variance of the posterior distribution to generate the posterior distribution. Posterior distributions utilize richer historical information, enabling more stable inferences about the true role an agent plays in collaboration.

[0056] To ensure reasonable role sampling during the execution phase, relying solely on the prior distribution, this invention utilizes KL divergence to maintain consistency between the prior and posterior distributions. The loss function is... During training, the KL divergence is minimized so that the prior distribution gradually approximates the posterior distribution. Then, a reparameterization technique is used (i.e., sampling ε from the standard normal distribution and then calculating...). Extracting continuous latent role vectors The role vector is a low-dimensional representation (e.g., 3-dimensional) in a continuous space, with different dimensions encoding different behavioral tendencies.

[0057] Extracted potential role vectors Input character decoder. The character decoder is a small neural network that outputs Feature Linear Modulation (FiLM) parameters—scaling factors. Peaceful transfer Its dimensions and hidden features in the middle layer of the individual value network The dimensions are the same. Then, an element-wise affine transformation is performed on the hidden features: , where ⊙ denotes element-wise multiplication. After FiLM modulation, the same set of backbone network parameters can generate different behavioral strategies based on different role vectors. For example, when the role vector is biased towards "assault," the modulation parameters will amplify the feature channels related to attack; when the role vector is biased towards "cover," the modulation parameters will suppress the attack channel and enhance the feature channels related to defense. This achieves the effect of dynamic specialization of the policy network according to the role.

[0058] Figure 6 This is a diagram of the Role-Driven Attention Communication Module (RDAC). The following section combines... Figure 6 This paper further explains the operational mechanism of the RDAC module in actual communication. In complex adversarial tasks such as StarCraft, due to the problem of local observability of information, agents must compensate for the limitations of their field of vision through communication. This invention abandons the traditional global communication network and instead uses a role-driven attention model. Its calculation process is as follows: intelligent agent i Generate corresponding role query vectors Q i This is used to search for states relevant to the current role of all neighboring agents. For example, an agent performing a "breakout" role will automatically increase its attention weight to observations of nearby "supportive" agents through an attention mechanism. To leverage domain priors, this invention introduces type-pairing bias. This learnable parameter matrix pre-stores the coordination importance coefficients between different military branches, accelerating the establishment of communication links in the early stages of training. Simultaneously, to further suppress channel redundancy, a gating network... Based on the confidence generation coefficient of local observations When local observations are sufficient to support decision-making, gating will shut down external communication flows to prevent noise from interfering with the main branch strategy.

[0059] Specifically, construct a query-key-value model. (Intelligent agent) First, a linear projection matrix is ​​used. potential role vectors Mapped to query vector For each other intelligent agent j Encode its observations (Generated by the observation encoder) respectively through the projection matrix and Mapping to key Sum . , , The projection matrix is ​​shared by all agents.

[0060] Computational intelligent agents i For other intelligent agents j When assigning attention weights, this invention introduces a learnable type-pairing bias matrix. The matrix has dimensions K×K (K is the number of agent types), and the initial value of each element can be set based on domain prior knowledge (for example, in StarCraft, the Medivac's support for Marauders is highly important, so a larger initial bias value can be set). The attention aggregation formula is: , where d is the dimension of the key vector. Type pairing bias enables the attention mechanism to explicitly encode the importance of cooperation between different types of agents. For example, a fanatic performing an "assault" role will automatically increase the attention weight of observation information of nearby "medical boat" type agents.

[0061] To further suppress redundant information in the communication channel, this invention designs a role-aware gating network. The input of this gating network is the agent. i Local embedding (i.e., the output of the observation encoder) and latent role vectors The concatenation of these parameters is used to generate dynamic gating coefficients through a single-layer network with a sigmoid activation function. The gating coefficient, ranging from 0 to 1, reflects whether the agent currently needs external communication. For example, when local observation information is sufficient to support the current decision, the gating coefficient tends to 0, shutting down the communication flow to prevent noise interference.

[0062] Applying the gating coefficient to the aggregated message yields the final communication representation: This representation is then concatenated with the local observation code and input into the Action Value Assessment Network (Q-network) to calculate the individual action value.

[0063] In the intensive training phase, in order to accurately assess the contribution of each heterogeneous agent to the team's victory, this embodiment introduces the TEVE module. Figure 7 This is a diagram of the Type Enhanced Value Assessment and Multi-Headed Intrinsic Exploration (TEVE) module. Traditional QMIX hybrid networks often struggle to capture the decisive role of specific key units in the global Q-value when dealing with asymmetric, highly heterogeneous maps.

[0064] This invention addresses this problem through a type-enhanced gated hybrid mechanism. The weights of the hybrid network consist of basic components and type-incremental components. .

[0065] Global Gated Scalar g , It played a role in smoothing the transition: in the early stages of training, g When the model is relatively small, it prioritizes learning basic collaborative patterns; in the later stages of training, g As the system grows, it begins to utilize the type context of heterogeneous unit compositions to fine-tune the credit allocation logic, thereby significantly improving the generalizability of strategies for specific unit ratios.

[0066] Specifically, the basic hypernetwork is based on the global state. s (Including information such as the location, health, and type of all agents) Generate basic mixture parameters and To ensure monotonicity constraints (i.e., an increase in individual Q-values ​​does not lead to a decrease in global Q-values), the last layer of the base supernetwork uses an absolute value activation function or performs a non-negative mapping on the output.

[0067] Type-modulated supernetworks based on the type context of heterogeneous teams (For example, a vector representing the number or combination of various types of agents in the current team) Generate incremental parameters Δ W and Δ b This supernetwork is independent of the base supernetwork and learns how to adjust the credit allocation strategy based on the team composition. For example, when the team contains high-value protected units (such as Colossi), the type modulation supernetwork adds a positive bias to the hybrid weights associated with that unit, allowing the unit's contribution to be evaluated more accurately.

[0068] Introducing globally learnable gating scalars g (Initial values ​​are close to 0, gradually increasing with the number of training steps), the final hybrid network parameters are synthesized according to the following formula: .

[0069] In the early stages of training, g is relatively small, and the model mainly relies on basic co-patterns for learning, ensuring training stability. In the later stages of training, g increases, and the system begins to use type context to fine-tune credit allocation. This gating smoothing mechanism avoids noise interference introduced by type enhancement in the early training stages.

[0070] Furthermore, to address the issue that a single prediction model is insufficient to cover complex state spaces in heterogeneous scenarios, this embodiment employs a multi-head RND exploration mechanism.

[0071] ; By assigning independent prediction heads to each type of agent, the system can evaluate the novelty of different responsibility domains in parallel, providing dense reward spurs for rare cooperative states. This is particularly effective in adversarial maps like MMM2.

[0072] Specifically, to address the difficulty of exploration in sparse reward environments, this invention constructs a multi-head random network distillation module. Independent prediction heads are constructed for each of the K types of agents, and each prediction head contains a target network with fixed parameters. and a trainable prediction network For input state s Calculate the prediction error for each head. Since the target network parameters are fixed, if the state... s If it is a novel area (i.e., an area with less exploration), the prediction error will be larger.

[0073] Multi-head intrinsic reward Match agents according to their categories, perform normalization pruning (e.g., limit to the range [0, 5.0]), and then compare with external environmental rewards. Weighted fusion: ,in The intrinsic reward coefficient is set to 0.05 in this embodiment and linearly annealed to 0. This design provides differentiated exploration incentives for different types of agents. For example, the medical ship will receive an intrinsic reward for exploring new strategies to treat teammates, while the fanatic will receive an intrinsic reward for exploring offensive formations.

[0074] Jointly optimize the total loss function: ,in, L TD The standard time-series difference loss is calculated as the mean square error between the global Q-value and the target Q-value. L KL The KL divergence loss is used for the prior and posterior of the role. L RNDThis is a weighted sum of multi-head prediction errors. Through end-to-end joint optimization, the modules mutually reinforce each other: the role inference module provides accurate role vectors for communication and value decomposition; the communication module provides key information for value assessment; the value decomposition module provides accurate gradient signals for policy optimization; and the intrinsic exploration module helps the system escape the sparse reward trap.

[0075] Furthermore, the type pairing bias matrix The model is trained jointly with the entire system in an end-to-end manner, using Adam as the optimizer and a learning rate of 5×10⁻⁶. -4 The initialization can be set based on domain knowledge: for example, in the StarCraft MMM2 map, the initial bias value for pairing Marauders and Medivacs is set to 2.0, the initial bias value for pairing Marines and Medivacs is set to 1.5, and the initial bias value for within the same type is set to 0.5. During training, the bias matrix will be automatically adjusted according to the actual cooperation effect. It is worth noting that the trained bias matrix can be transferred to other multi-agent tasks with similar structures and used as initialization parameters for fine-tuning, accelerating the convergence of new tasks.

[0076] The proposed method employs a centralized training and distributed execution architecture. During the training phase, all modules (role inference VAE, attention communication network, hybrid network, and RND module) are collaboratively optimized on a server with sufficient computing resources. During the execution phase (e.g., deployment to drone formations or game AI), only a lightweight role inference module (retaining only the prior encoder and FiLM decoder) and a directional communication module (attention communication and gating network) need to be deployed, without running the posterior encoder, hybrid network, and RND module. This design enables the method to adapt to computationally limited embedded systems while maintaining low inference latency during the execution phase.

[0077] Further reference Figure 8 As an implementation of the methods shown in the above figures, this application provides an embodiment of a heterogeneous multi-agent cooperative system based on role-driven communication and heterogeneous perception enhancement. This system embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0078] Secondly, embodiments of the present invention also disclose a heterogeneous multi-agent collaborative system based on role-driven communication and heterogeneous perception enhancement, such as... Figure 8 As shown, it includes: a role inference module 81, a directional communication module 82, a value assessment module 83, and an inner exploration module 84.

[0079] In one specific embodiment, the role inference module 81 is used to infer continuous latent role variables from the local observations of each agent using a variational autoencoder architecture, and to perform feature linear modulation on the individual policy network based on the latent role variables to generate role-specific individual policies; the directional communication module 82 is used to use the latent role variables as query vectors, to aggregate teammate information by introducing a type-pairing bias attention mechanism, and to use role-aware gating to suppress redundancy in the aggregated messages to obtain a filtered communication representation.

[0080] The value assessment module 83 is used to generate basic mixing parameters and incremental parameters through the basic supernetwork and type modulation supernetwork respectively during the centralized training phase, and obtain the final mixed network parameters by using global gating scalar fusion to monotonically mix the values ​​of individual actions to achieve credit allocation; the intrinsic exploration module 84 is used to construct a multi-head random network distillation module, set independent prediction heads for different types of agents, generate intrinsic rewards and fuse them with extrinsic rewards, and jointly optimize the parameters of each module.

[0081] The functions and methods of the above modules correspond to each other, and will not be repeated here.

[0082] To verify the effectiveness of the invention, a comprehensive evaluation was conducted on the StarCraft II multi-agent cooperative benchmark platform SMAC.

[0083] The StarCraft II Collaborative Mapping (SMAC) environment is one of the most representative heterogeneous collaborative benchmarks in the field of multi-agent reinforcement learning. Experiments evaluate three types of maps with significant heterogeneous characteristics: basic heterogeneous collaborative maps. Figure 2 s3z (including Zealot and Stalker units), High-Value Unit Protected Area Figure 1 c3s5z (which includes three types of units: Colossus, Stalker, and Zealot) and the highly heterogeneous asymmetric map MMM2 (which includes three types of units: Marauder, Marine, and Medivac, with highly asymmetrical enemy and friendly configurations).

[0084] In this environment, the global state includes information such as unit coordinates, health, shield value, type ID, and skill cooldown; local observation only exposes local information within the field of vision to each agent; the action space is a discrete set of actions, and different types of agents have different sets of legal actions, such as... Figure 9 As shown.

[0085] The experiments involved maps of varying difficulty and heterogeneity. Table 1 details the general training parameters used in the experiments and the hyperparameter configurations of this invention: Table 1 Experimental Parameter Settings ; To quantify the convergence speed and final performance of the algorithm, the test win rate was used as the core metric in the experiment. This metric is the most intuitive indicator of the effectiveness of the algorithm's strategy. During training, training was paused at regular intervals, and 32 independent test rounds were run to calculate the percentage of times the agent defeated the opponent. All the result graphs in this invention use this metric to visually demonstrate the algorithm's dynamic learning ability and convergence speed throughout the entire interaction cycle.

[0086] .

[0087] Considering that multi-agent reinforcement learning may still experience minor oscillations due to environmental randomness at the end of training, this paper introduces the average test win rate in the quantitative comparison table to more objectively and rigorously evaluate and compare the absolute performance of different algorithms after convergence. This metric is calculated by averaging all test win rate data within the last 10% of steps of the training process, thereby effectively eliminating random errors in single evaluations and truly reflecting the collaborative level after the model has stabilized.

[0088] This invention addresses soft update coefficients in the MMM2 scenario. τ Communication gating initial bias and intrinsic reward coefficient η and type prior constraint coefficients λ KL Sensitivity analysis was performed, and the hyperparameter search results are shown in Table 2. The comparative evaluation is shown in Figures 10(a), 10(b), 10(c), and 10(d).

[0089] Table 2. Quantization results of key hyperparameters in the MMM2 scenario. ; Experiments show that when τ =0.005, bias=2.0 η =0.05、 λ KL When the value is 0, the model achieves the best balance between stability and final performance, with an average test win rate of 89.58%, a convergence step count of approximately 1.45M, and the smallest standard deviation.

[0090] Soft update coefficient τ Directly affects the tracking speed of the target network: τ When the target network is too small, updates are delayed, making it difficult to reflect environmental changes in a timely manner. τ Excessive bias amplifies noise, causing value estimation oscillations. The initial bias of the communication gating controls the message reception threshold of the agent in the early stages of training: too small a bias results in too many redundant messages entering the backbone network; too large a bias leads to premature suppression of effective cooperative messages. Intrinsic reward coefficient. η The strength of the impact of multi-head RND on the main task optimization is determined by: ηA value of 0.05 can effectively guide agents to discover high-value collaborative states; η If the coefficient is too large, the intrinsic reward noise will mask the extrinsic reward signal, leading to training oscillations and a decrease in win rate. Type prior coefficient λ KL Consistency constraints used to adjust for character distribution and type priors: In a highly dynamic heterogeneous adversarial environment like MMM2, excessively strong type constraints can limit the freedom of character evolution. λ KL The fact that 0 actually yielded the optimal result indicates that retaining sufficient flexibility in role representation is more conducive to the emergence of complex division of labor strategies.

[0091] To further verify the contributions of heterogeneous perception enhancement, role-driven communication, type-aware hybrid network, and intrinsic exploration mechanism modules to the overall performance of the RH-QMIX algorithm, ablation experiments were conducted on representative heterogeneous and asymmetric adversarial scenarios. The following four algorithm variants were constructed, and some key modules of the model were removed to evaluate their marginal effects.

[0092] (1) Remove intrinsic exploration (w / o RND): The multi-head RND module is removed, and training is performed using only the extrinsic rewards provided by the environment. This variant aims to verify the necessity of intrinsic incentive mechanisms in addressing the problem of sparse rewards and exploration difficulties in heterogeneous scenarios.

[0093] (2) Removal of the Communication Module (w / o Comm): The role-driven attention communication module and communication prediction auxiliary task are removed, and the agent makes decisions based solely on local observations. This variant aims to evaluate the role of explicit information interaction in compensating for the limitations of local perception and promoting cooperation.

[0094] (3) Remove Heterogeneity-Aware Enhancement (w / o Type): All agent type-related designs are removed, including type-role consistency constraints in HRIG, type incremental branches in hybrid networks, and type pairing biases in communication. This variant treats heterogeneous agents as homogeneous entities and aims to verify the contribution of explicit heterogeneity modeling to policy specialization and value assessment.

[0095] (4) Remove Role-Aware Mixer: Remove joint role context information from the input of the hybrid network, degenerating it into a structure that depends only on global state and party type. This variant aims to verify the impact of introducing dynamic role intent into the value decomposition process on improving credit allocation accuracy.

[0096] The ablation comparison curves are shown in Figures 11(a), 11(b), and 11(c), and the ablation results are shown in Table 3.

[0097] Table 3 Performance Quantification Results of Ablation Experimental Model ; After removing multi-head RNDs, the model mainly relies on external rewards for training, and the exploration speed slows down significantly under sparse reward conditions, especially in MMM2. After removing role-driven communication, agents can only make independent decisions based on local observations, and the collaborative efficiency drops significantly. After removing heterogeneous perception enhancement, the system degenerates into a generalized processing of different types of agents, and the accuracy of role expression and value assessment is affected. Type differences cannot be explicitly encoded, and role division tends to be blurred. After removing role-aware mixing, type context cannot be effectively entered into the mixing network, role context no longer participates in joint value construction, credit allocation error increases, and training standard deviation increases accordingly.

[0098] The ablation results show that the three modules HRIG, RDAC and TEVE correspond to "role formation, information filtering, value decomposition and exploration compensation" respectively. The three complement each other and are indispensable.

[0099] To comprehensively evaluate the performance of this invention in heterogeneous collaborative tasks, the following representative multi-agent reinforcement learning algorithms were also selected as baselines for comparison: (1) VDN: A basic value decomposition algorithm that assumes that the global Q value is the direct sum of all local Q values. It is suitable for simple collaborative scenarios and serves as the most basic performance reference.

[0100] (2) QMIX: The most mainstream value decomposition algorithm at present. By introducing a hybrid network and adding absolute value operation in weight generation, it realizes monotonicity constraint and can fit more complex joint value functions. It is the basic architecture of the RH-QMIX algorithm.

[0101] (3) ROMA: A representative algorithm for multi-agent role representation. This algorithm learns implicit role embeddings based on local observations, thereby adjusting individual strategies and promoting behavioral division of labor. Using ROMA as the core baseline, we can intuitively verify the significant advantages of the heterogeneous perception enhancement and explicit communication mechanism introduced in this study in breaking through the bottleneck of local observations and achieving global deep collaboration.

[0102] (4) Kaleidoscope: An adaptive partial parameter sharing method for heterogeneous multi-agent reinforcement learning. This method introduces learnable masks on the shared network parameters to dynamically control the degree of parameter sharing between different agents, thereby enhancing policy diversity while maintaining the sample efficiency brought by parameter sharing, enabling agents to learn differentiated policies, and showing good performance in various multi-agent tasks.

[0103] Comparison curves as follows Figure 12As shown in Table 4, the experimental results are as follows.

[0104] Table 4 Quantitative Results of Model Comparison Experiment ; Experimental results show that the present invention (RH-QMIX) performs particularly well in the highly heterogeneous and asymmetric MMM2 scenario, with an average test win rate of 89.58%, which is significantly better than QMIX's 79.24%, VDN's 86.04%, ROMA's 15.10%, and Kaleidoscope's 67.29%, improving by about 10.3 percentage points compared to the best baseline. At the same time, the standard deviation is only 0.0501, indicating that the present invention not only has stronger performance, but also more stable training.

[0105] In the 2s3z scenario, both role-driven communication and type-aware hybrid approaches can quickly form an effective division of labor. In the 1c3s5z scenario, the need to protect high-value units makes the advantages of role inference and gated communication more obvious. In the MMM2 scenario, due to the strong heterogeneity of enemy and friendly unit configurations and the more complex targets, this invention can still maintain a high win rate and a more stable training curve, indicating that role space and type context have significant modeling value for complex collaboration.

[0106] Among them, QMIX has good value decomposition capabilities, but lacks explicit role modeling and communication filtering mechanisms; VDN is limited by the linear summation assumption, and its value expression capabilities are even weaker; ROMA introduces the concept of roles, but lacks strong type prior support and communication filtering mechanisms; Kaleidoscope has good parameter sharing flexibility, but it is still insufficient to completely solve the communication redundancy and credit allocation problems in high-intensity heterogeneous adversarial scenarios.

[0107] Figure 13 The heatmap for attention in the MMM2 scene is shown, illustrating the distribution of attention weights at the type and individual levels.

[0108] The heatmap results show that the attention distribution among different types is uneven, indicating that the model has indeed learned to filter information based on type differences. Simultaneously, units of the same type are not completely isolated, but rather retain necessary local interactions, suggesting that the model has learned a sophisticated strategy of "when to communicate, how much to communicate, and to whom to communicate" rather than simple broadcasting. Observations of individual-level weights further indicate that a few key agents become information hubs at different times, reflecting that the model proactively selects communication partners with higher decision-making value based on the tactical situation. This behavior is closer to the information routing patterns in real-world team collaborations.

[0109] This non-uniform distribution characteristic proves that the RDAC module is not simply fully connected, but rather learns information routing patterns with tactical intent. Further clustering analysis of the latent role space reveals that agents at different stages form distinct role type clusters within the latent space. This directly demonstrates that the HRIG module successfully extracts continuous and dynamic behavioral roles from discrete unit types, providing solid theoretical and experimental support for solving heterogeneous multi-agent collaboration problems.

[0110] Based on the above experimental results, this invention effectively solves the problems of rigid division of labor, communication redundancy, and difficulty in credit allocation in heterogeneous multi-agent systems through a four-layer collaborative mechanism of "role inference - communication screening - type enhancement - intrinsic exploration". It can effectively improve the final win rate, convergence speed, and training stability in complex heterogeneous scenarios, and has obvious mechanism advantages in strongly heterogeneous asymmetric tasks. It is suitable for application in complex heterogeneous collaborative scenarios such as tactical simulation, cooperative search, heterogeneous formation control, and multi-role task allocation.

[0111] In practical engineering deployments, the method proposed in this invention can be adapted to various hardware platforms. For embedded systems with limited computing power, only the lightweight role inference and communication gating modules can be retained during the execution phase, while the heavier variational autoencoder training and multi-head RND modules can be placed on the offline training server.

[0112] The scope of protection of this invention should not be limited to a specific number of network layers, hidden dimensions, or a specific simulation map. Any technical solution that utilizes local observations to infer continuous latent variables, drives attention communication filtering, and introduces type context enhancement in value decomposition should fall within the substantive technical scope of this invention. Preferred combinations of parameters provided by this invention (such as...) τ =0.005, bias=2.0, η =0.05) has universal reference value in the field of complex heterogeneous collaboration.

[0113] The above embodiments are merely illustrative of the present invention and are not intended to limit the invention. Any changes or modifications to the above embodiments based on the technical essence of the present invention will fall within the scope of the claims of the present invention.

[0114] The following is for reference. Figure 14 It illustrates an electronic device suitable for implementing embodiments of the present invention (e.g., Figure 1 A schematic diagram of the structure of a computer device 1400 (shown as a server or terminal device). Figure 14 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0115] like Figure 14As shown, the computer device 1400 includes a central processing unit (CPU) 1401 and a graphics processing unit (GPU) 1402, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1403 or programs loaded from storage portion 1409 into random access memory (RAM) 1404. The RAM 1404 also stores various programs and data required for the operation of the device 1400. The CPU 1401, GPU 1402, ROM 1403, and RAM 1404 are interconnected via a bus 1405. An input / output (I / O) interface 1406 is also connected to the bus 1405.

[0116] The following components are connected to I / O interface 1406: an input section 1407 including a keyboard, mouse, etc.; an output section 1408 including a liquid crystal display (LCD), speakers, etc.; a storage section 1409 including a hard disk, etc.; and a communication section 1410 including a network interface card such as a LAN card, modem, etc. The communication section 1410 performs communication processing via a network such as the Internet. A drive 1411 may also be connected to I / O interface 1406 as needed. Removable media 1412, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1411 as needed so that computer programs read from it can be installed into storage section 1409 as needed.

[0117] In particular, according to embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1410, and / or installed from removable medium 1412. When the computer program is executed by central processing unit (CPU) 1401 and graphics processing unit (GPU) 1402, the functions defined in the methods of this invention are performed.

[0118] It should be noted that the computer-readable medium described in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with a system, apparatus, or device that executes instructions. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with a system, apparatus, or device that is executed by instructions. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0119] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using dedicated hardware-based devices that perform the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0121] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor.

[0122] In another aspect, the present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the following steps: inferring continuous latent role variables from the local observations of each agent using a variational autoencoder architecture, and performing feature linear modulation on the individual policy network according to the latent role variables to generate role-specific individual policies; using the latent role variables as query vectors, aggregating teammate information by introducing a type-pairing biased attention mechanism, and using role-aware gating to suppress redundancy in the aggregated messages to obtain a filtered communication representation; during the centralized training phase, generating basic mixing parameters and incremental parameters through a basic supernetwork and a type-modulation supernetwork respectively, and obtaining the final mixing network parameters through global gating scalar fusion, monotonically mixing the individual action values ​​to achieve credit allocation; constructing a multi-head random network distillation module, setting independent prediction heads for different types of agents, generating intrinsic rewards and fusing them with extrinsic rewards, and jointly optimizing the parameters of each module.

[0123] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A heterogeneous multi-agent collaboration method based on role-driven communication and heterogeneous perception enhancement, characterized in that, The method includes the following steps: S1. Using a variational autoencoder architecture, continuous latent role variables are inferred from the local observations of each agent, and the individual policy network is linearly modulated based on the latent role variables to generate role-specific individual policies. S2. Using the potential role variables as query vectors, teammate information is aggregated by introducing a type-pairing biased attention mechanism, and the aggregation message is redundancy suppressed by using role-aware gating to obtain the filtered communication representation. S3. During the intensive training phase, basic mixing parameters and incremental parameters are generated by the basic supernetwork and the type modulation supernetwork, respectively. The final mixing network parameters are obtained by global gating scalar fusion. The individual action values ​​are monotonically mixed to achieve credit allocation. S4. Construct a multi-head random network distillation module, set independent prediction heads for different types of agents, generate intrinsic rewards and fuse them with extrinsic rewards, and jointly optimize the parameters of each module.

2. The heterogeneous multi-agent cooperation method according to claim 1, characterized in that, Step S1 specifically includes: S11, the intelligent agent i Local observation at the current moment and type identifier Input the role prior encoder to generate the prior distribution. ; Represents intelligent agents Static type identifiers, such as military unit; Represents intelligent agents i At any moment t The dynamic potential role vector; S12, the intelligent agent i Local movements - observing historical trajectories Introducing a role posterior encoder to generate the posterior distribution during the training phase. ； S13. Using the KL divergence constraint, the prior distribution and posterior distribution are kept consistent. The calculation formula is as follows: And extract continuous latent role vectors through reparameterized sampling. ; Indicates the historical trajectory τ Mathematical expectation on Indicates the Kullback-Leibler divergence. Describe the prior distribution, This represents an approximate posterior distribution; S14, Transfer the potential role vector Input the character decoder to generate feature linear modulation parameters—scaling coefficients. Peaceful transfer Hidden features of the intermediate layer of the individual value network Perform modulation: This yields the intermediate features after character specialization.

3. The heterogeneous multi-agent cooperation method according to claim 1, characterized in that, Step S2 specifically includes: S21. Construct a role-driven query-key-value model with intelligent agents. Potential role vectors as query vector With other intelligent agents Observation coding As a key Sum ,in , , To share the projection matrix; S22. Introduce learnable type-pairing bias terms. A type-related correction term is added to the attention weight calculation to aggregate the communication messages. : ; Other intelligent agents j The transpose of the key vector This represents the hidden dimension of the key-value vector in the multi-head attention mechanism. S23, Regarding intelligent agents At any moment t Gated networks are based on local embedding and potential role vectors Generate dynamic gating coefficients : ,in σ It is the sigmoid function; S24. Apply the gating coefficient to the aggregated message to obtain the final communication representation. And input the action value assessment branch.

4. The heterogeneous multi-agent cooperation method according to claim 1, characterized in that, Step S3 specifically includes: S31, the basic hypernetwork is based on the global state s Generate basic mixing parameters and And monotonicity is guaranteed through nonnegative mapping; S32, The type modulation supernetwork generates incremental parameters Δ based on the type context of heterogeneous teams. W and Δ b ; S33. Introduce globally learnable gating scalars. g , The final hybrid network parameters are synthesized according to the following formula: 。 5. The heterogeneous multi-agent cooperation method according to claim 1, characterized in that, Step S4 specifically includes: S41. Construct the multi-head random network distillation module for... K Each intelligent agent is equipped with an independent prediction head to calculate the prediction error. : ; Indicates the first k The output feature vector of each prediction head Indicates the first k The output feature vector of each target head s Indicates the global environment state; S42. Integrate intrinsic rewards and extrinsic environmental rewards to adjust the total training reward. ; Indicates the intrinsic reward coefficient; This refers to external environmental rewards; Represents intrinsic reward; Clip(·) and Normalize(·) represent function operations that numerically truncate and normalize scalar data, respectively; S43, Joint optimization includes temporal difference loss. L TD KL divergence loss L KL and RND predicted loss L RND The total loss function is: 。 6. The heterogeneous multi-agent cooperation method according to claim 5, characterized in that, The RND prediction loss function is: ;in, Indicates the first in the current scene k The weighting of the number of intelligent agents. This indicates the stop gradient operator, used to ensure that the parameters of the target network remain fixed and are not updated during backpropagation.

7. The heterogeneous multi-agent cooperation method according to claim 4, characterized in that, The global gated scalar g The annealing strategy employs linear or exponential growth, making early training... g Approaching 0, the model prioritizes learning basic collaborative patterns in the later stages of training. g Approaching 1, the model uses type context to fine-tune credit allocation.

8. A heterogeneous multi-agent collaborative system based on role-driven communication and heterogeneous perception enhancement, characterized in that, include: The role inference module is used to infer continuous latent role variables from the local observations of each agent using a variational autoencoder architecture, and to perform feature linear modulation on the individual policy network based on the latent role variables to generate role-specific individual policies. The directional communication module is used to aggregate teammate information by introducing a type-pairing bias attention mechanism, using the potential role variables as query vectors, and to suppress redundancy in the aggregated messages by using role-aware gating to obtain a filtered communication representation. The value assessment module is used to generate basic mixing parameters and incremental parameters through the basic supernetwork and type modulation supernetwork respectively during the centralized training phase, and obtain the final mixing network parameters by global gating scalar fusion, so as to monotonically mix the individual action values ​​and realize credit allocation. The intrinsic exploration module is used to construct a multi-head random network distillation module, which sets independent prediction heads for different types of agents, generates intrinsic rewards and merges them with extrinsic rewards, and jointly optimizes the parameters of each module.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images