Credit distribution method, device and system in multi-agent cooperation

Abstract

The invention relates to a credit distribution method in multi-agent cooperation, and the method comprises the following steps: obtaining joint data of a multi-agent system, the joint data comprising observation information and execution actions of each agent; the joint data are input into a discriminator model, and the discriminator model carries out modeling on an interaction dependency relationship among multiple agents based on an interaction dependency relationship modeling module of an attention mechanism; generating an auxiliary reward signal corresponding to each agent through a discriminator model; generating a fused reward signal based on the auxiliary reward signal and a global reward signal from the environment; and training and updating the multi-agent strategy model by using the fused reward signal so as to optimize the cooperation strategy of the multi-agent system. By introducing a credit distribution mechanism based on a discriminator model, automatic decomposition and optimization generation of individual reward signals are realized, so that the learning efficiency and stability of a multi-agent system in a complex cooperative task are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-agent reinforcement learning, in particular to a credit assignment method, device and system in multi-agent cooperation. BACKGROUND

[0002] In the field of multi-agent reinforcement learning, especially in cooperative multi-agent tasks, credit assignment is a core challenge. Credit assignment refers to how to reasonably decompose and assign the global reward obtained by the team to each agent to accurately reflect the contribution of each agent to the success of the team.

[0003] The prior art usually adopts the following methods to solve the credit assignment problem: (1) Cooperative method based on value decomposition (VDN, QMIX, etc.). This method decomposes the joint value function into local values of each agent through structural constraints to achieve scalable training. However, this method relies on the additivity or monotonicity assumption and cannot express complex causal contribution relationships, so the reward assignment accuracy is limited in tasks where agent behavior is strongly coupled or dynamically dependent, and suboptimal strategies are prone to converge.

[0004] (2) Strategy gradient method based on centralized critic (such as COMA). This method trains through joint strategy advantage calculation and can theoretically provide differentiated reward signals to agents. However, the centralized critic parameters expand rapidly with the increase in the number of agents, and this method only outputs global strategy gradient feedback, making it difficult to directly explain the source of individual behavior contribution, and the gradient signal decays significantly in sparse or delayed reward environments, resulting in training instability and slow convergence.

[0005] (3) Method based on imitation learning or adversarial training (such as GAIL, MA-GAIL). This method compares the current policy trajectory with the expert policy trajectory (artificial demonstration of human demonstration trajectory or high-performance prior policy trajectory) through a discriminator to provide learning guidance. However, the traditional discriminator only outputs the overall distribution difference or global reward signal, and does not design a structured decomposition mechanism, so it cannot convert expert behavior guidance into local rewards that can be used for independent decision-making, which is not conducive to forming an interpretable credit assignment process.

[0006] In summary, due to the lack of a unified framework that can simultaneously support multi-agent interaction modeling, interpretable reward decomposition ability, scalable training mechanism, and adaptation to sparse reward environments, the existing methods have insufficient credit assignment accuracy, unstable learning, low convergence efficiency, and limited strategy performance.

[0007] Therefore, there is an urgent need for a new multi-agent collaboration training method with a learnable decomposable reward structure, attention-based interaction modeling capability, and support for different reward source expression forms to address the shortcomings of existing technologies.

[0008] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0009] The top technical problem to be solved by the present application is how to improve the overall learning efficiency, policy performance, and stability of a multi-agent reinforcement learning system in complex collaboration tasks. Specifically, it can be embodied as how to accurately and efficiently decompose and distribute the global reward signal to each agent in a multi-agent collaboration task to solve the problems of inaccurate credit distribution, slow convergence, and lack of modeling of interaction relationships between agents in existing technologies. To this end, the present application provides a credit distribution method, device, and system in multi-agent collaboration to solve the above technical problems.

[0010] The technical solution adopted by the present application to solve the above technical problems is as follows.

[0011] The first aspect of the present application provides a credit distribution method in multi-agent collaboration, comprising the following steps: S1, obtaining joint data of a multi-agent system, the joint data including observation information and executed actions of each agent; S2, inputting the joint data into a trained discriminator model, wherein the discriminator model uses an interaction dependency modeling module based on an attention mechanism to model the interaction dependency between multiple agents; S3, processing the joint data through the discriminator model to generate an auxiliary reward signal corresponding to each agent, wherein the auxiliary reward signal is used to decompose and distribute the global reward signal from the environment; S4, generating a fusion reward signal based on the auxiliary reward signal and the global reward signal from the environment; S5, training and updating the multi-agent policy model using the fusion reward signal to optimize the collaboration policy of the multi-agent system.

[0012] In some embodiments, the attention mechanism of step S2 is a multi-head attention mechanism, and the discriminator model is trained by: obtaining a training sample set, the training sample set including expert trajectory data generated by an expert policy or environment trajectory data labeled with a global reward signal feedback by the environment; and optimizing the parameters of the discriminator model to minimize the difference between the reward signal output by the discriminator model and the expert trajectory data or the global reward signal.

[0013] In some embodiments, step S4 specifically comprises generating the fusion reward signal according to the formula: , wherein, represents the fusion reward signal of the i-th agent at the time step t, represents the global reward signal at the time step t, represents the auxiliary reward signal of the i-th agent generated by the discriminator model, is a preset auxiliary reward weight coefficient.

[0014] In some embodiments, the process of the discriminator model processing the joint data in step S3 comprises the following steps: mapping the observation information and the action of each agent into a corresponding feature vector through an encoder network; inputting the feature vectors of all agents into a multi-head attention layer to calculate the attention weight between agents and output the new feature representation of each agent that is context-enhanced; inputting the new feature representation of each agent into a corresponding reward prediction head network to output the auxiliary reward signal of each agent.

[0015] The second aspect of the present application provides a credit allocation device in multi-agent cooperation, comprising: a data acquisition module configured to acquire joint data of a multi-agent system, the joint data comprising observation information and executed actions of each agent; a discriminator module comprising a neural network constructed based on an attention mechanism, configured to process the joint data to model the interaction dependency relationship between the plurality of agents and output an auxiliary reward signal corresponding to each agent; a reward fusion module configured to generate a fusion reward signal based on the auxiliary reward signal and a global reward signal from an environment; and a policy training module configured to train and update a multi-agent policy model using the fusion reward signal to optimize the cooperation strategy of the multi-agent system.

[0016] In some embodiments, the discriminator module further comprises: an encoder submodule configured to map the observation information and the action of each agent into a corresponding feature vector; a multi-head attention submodule configured to model the interaction dependency relationship of the feature vectors of all agents and output context-enhanced feature representations; and a reward prediction submodule configured to predict and output the auxiliary reward signal of each agent based on the context-enhanced feature representations.

[0017] The third aspect of the present application provides a multi-agent decision control system, comprising: the credit allocation device of claim 6 or 7; an environment interaction module configured to control the plurality of agents to perform actions in an environment and acquire a global reward signal; and a policy execution module configured to load the trained multi-agent policy model and generate a decision action for each agent according to the multi-agent policy model.

[0018] The fourth aspect of the present application provides an electronic device, comprising: one or more processors; a memory; one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and the one or more computer programs comprise instructions for executing the method of the first aspect of the present application.

[0019] The fifth aspect of the present application provides a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect of the present application.

[0020] The sixth aspect of the present application provides a multi-agent cooperation method based on credit distribution, comprising the following steps: The credit distribution method in the multi-agent cooperation of the first aspect of the present application is executed to generate a fusion reward signal for each agent; Based on the fusion reward signal, the decision strategy of each agent is distributedly trained or centrally trained by using a reinforcement learning algorithm to optimize the individual strategy; In the training process or after the training is completed, the updated individual strategy is used to control the corresponding agent to perform actions in the environment, thereby realizing multi-agent cooperation.

[0021] The present application has the following beneficial effects: The present application can explicitly model the interaction dependency relationship between agents by introducing a discriminator model based on an attention mechanism. The discriminator model can process joint data based on the interaction dependency relationship to generate refined auxiliary reward signals. This mechanism avoids the credit distribution bias caused by simple summation or linear decomposition in traditional methods, significantly improves the accuracy and rationality of reward distribution, and is especially suitable for scenarios where the contribution of agents is significantly different or there are complex dependency relationships, solving the problem of inaccurate credit distribution.

[0022] Further, the present application fuses the auxiliary reward signal generated by the discriminator model and the global reward signal of the environment by weighting, providing a more direct and timely learning signal for each agent. The fusion reward signal is based on the formula: Not only contains the overall performance information, but also incorporates the evaluation based on the contribution of the behavior of each agent, thereby effectively alleviating the learning difficulty caused by sparse rewards and delayed rewards, greatly accelerating the training process of the strategy model, enabling the multi-agent system to converge to a high-performance cooperation strategy more quickly, and improving the convergence speed.

[0023] Furthermore, the attention mechanism in this application is a multi-head attention mechanism, which automatically captures and quantifies the collaborative relationships between different agents. After the encoder maps the local observations and actions of each agent into feature vectors, the attention layer generates feature representations rich in contextual information by calculating queries. This enables the discriminator model to understand the dynamic dependencies between agents, and its output auxiliary reward signal has strong interpretability, reflecting the actual value of agent behavior in a specific context. This solves the problem of existing technologies lacking explicit and interpretable modeling of interactions between agents, and enhances the ability to model the interaction relationships between agents.

[0024] In summary, this application ensures the accuracy of credit allocation through a multi-head attention structure based on the discriminator model; it guarantees the consistency between the reward signal output by the discriminator model and the global objective through adversarial or regression training objectives; it combines accurate individual signals with global signals based on a reward fusion module, providing high-quality training signals for policy optimization; and iterative updates of the multi-agent policy model, in turn, provide new data for the discriminator model, forming an efficient collaborative optimization closed loop. It can be understood that this application, through the aforementioned technical features, constitutes an organically collaborative whole, enabling the system to not only perform excellently in the accuracy of credit allocation but also achieve significant overall improvements in convergence speed, stability, and final policy performance, realizing efficient, stable, and high-performance optimization of multi-agent collaborative policies.

[0025] Other beneficial effects of this application will be further described below. Attached Figure Description

[0026] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a diagram showing the overall system framework implementation corresponding to the method in this application; Figure 2 The method of this application is based on Figure 1 Flowchart of the iterative training method; Figure 3 The correlation analysis diagram shows the reward decomposition of the method in this application in nine GFootball multi-agent task environments with different complexities. Detailed Implementation

[0027] The following provides a detailed description of the embodiments of this application. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope and application of this application.

[0028] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0029] The embodiments of this application will be described in detail below with reference to the accompanying drawings, so that those skilled in the art can easily understand and implement this application. The following description is merely exemplary and is not intended to limit the scope of protection of this application.

[0030] The core of this application lies in providing a credit allocation method for multi-agent collaboration. Its basic principle is as follows: a discriminator model is designed, employing a multi-head attention module to interactively model the state-action features of each agent, thereby outputting a local reward estimate (per-agent reward) for each agent. This discriminator model can be trained through adversarial learning (distinguishing between policy trajectories and expert trajectories) or regression learning (making the sum of its outputs approximate the global environment reward). Finally, the auxiliary reward generated by the discriminator model is weighted and fused with the global environment reward to form a fused reward signal for policy optimization, thereby guiding the multi-agent policy to converge quickly and accurately towards efficient collaboration.

[0031] In some embodiments, the multi-agent cooperative method of this application is based on, for example, Figure 1 The system's overall framework is shown, and its setup includes the following steps: Provides a multi-agent policy module: each agent has a policy network (which can be a value-based or policy-based approach, such as a behavior network of QMIX / VDN or an actor-critic based actor).

[0032] Provide a discriminator module: a centralized neural network used to discriminate / estimate the joint trajectory or time step state-action of multiple agents and output auxiliary rewards for each agent.

[0033] Provides a coordinator (Trainer): responsible for data acquisition (environment interaction and playback buffer), discriminator model training, policy training, and hyperparameter scheduling.

[0034] Provides an optional expert sample library (Expert Buffer): for adversarial learning scenarios, it stores expert trajectories for the discriminator model to learn and align with the target.

[0035] In some embodiments, such as Figure 2 The diagram shows the operation flowchart of the method in this application based on the above system framework using an iterative training approach, which allows the policy and discriminator model to be updated alternately. The policy samples trajectories in the environment and stores them in the replay pool; the discriminator model receives the current policy trajectory and the target (expert or positive samples aggregated according to environment rewards) for training; the per-agent reward output by the discriminator model is used as an auxiliary term for policy optimization and is fused with the global environment reward according to preset weights into the policy loss, thereby completing the next round of policy update.

[0036] In some embodiments, the discriminator model design of this application includes the following steps: Set the input representation: The discriminator model at time step For all The inputs received by each agent include: 1. Local observations of each agent (For example, player's relative coordinates, speed, whether they have the ball, etc.); 2. Actions of each agent (Discrete or continuous); 3. Optional global information (e.g., global time step, score, etc.); 4. Optional historical context (short-term window or recurrent neural network (RNN) hidden state) (To handle delayed rewards)

[0037] 5. Input Encoding: For each agent, the input is first encoded using a shared encoder (Multilayer Perceptron (MLP) or a small convolutional / embedding layer). Observation-action pairs are mapped to vector representations .

[0038] Set up a multi-head attention interaction layer: Multi-head attention (similar to Transformer) is used to model the interactions of agent representations in order to capture the dependencies and contributions between agents: 1. The query, key, and value matrices are obtained by linear transformations: , , ,in .

[0039] 2. Adopt Each attention head has an output that is concatenated and passed through a linear layer to obtain the interaction representation. .

[0040] 3. Number of heads to focus on As a hyperparameter, the recommended value range is {1,2,4,8,16}. Experimental results show that 4-8 heads usually perform best.

[0041] 4. Residual connections and layer normalization (LayerNorm) can be added after the attention layer to improve training stability.

[0042] 5. This module outputs the context-enhanced representation for each agent. .

[0043] Configure the per-agent reward prediction header: based on The discriminator model outputs a scalar for each agent. (Auxiliary reward estimated by the discriminator model): , in, It can be a single or multiple MLP, and can include a normalization layer to control the scale.

[0044] Set the discriminator model loss function: The training objective of the discriminator model can be based on expert trajectories, employing adversarial imitation learning techniques. The optimization objective of the discriminator model is to distinguish between policy-generated trajectories and expert trajectories. For a single time step... Applying binary classification cross-entropy loss: in, Representing the state-action vectors of a joint or multi-agent system, the discriminator model can ultimately decompose and map them to the corresponding auxiliary rewards for each agent. Specifically, this is done by assigning the output layer vectors of the global discriminator model to agents.

[0045] The discriminator model output can be converted into a reward function for each agent. This serves as an intrinsic reward for the strategy.

[0046] In some embodiments, the policy update of this application includes the following steps: The overall loss of the strategy consists of the global reward from the environment and the auxiliary reward generated by the discriminator model for the agent. The modified reward function is: in: The policy objective of conventional models (such as maximizing expected return, PPO shearing loss, Q-learning TD error, etc.). To utilize the auxiliary objective output by the discriminator model, for example, by incorporating the per-agent auxiliary reward as an immediate reward into the advantage estimation: During algorithm training, this is used to modify the advantage in actor-critic: in, To assist in the weighting of rewards, the typical range is [0, 0.5], and based on experience, {0.01, 0.05, 0.1, 0.2} is often chosen.

[0047] The periodicity of policy updates can be controlled separately from the update frequency of the discriminator model (the number of update steps of the discriminator model is a key hyperparameter, with a recommended range of 10-40 times per policy update cycle, which can be optimized).

[0048] In some embodiments, the training process (pseudocode) of this application is as follows:

[0049] In some embodiments, the hyperparameters involved in this application are as follows:

[0050] In some embodiments, this application also provides the following implementation schemes: Implementation Plan 1: Best Implementation Plan (MACAD Algorithm) In this implementation plan, the experimental setup is as follows: This implementation plan was validated using the Google Research Football (GFootball) task set, and the specific scenarios included the following 9 sub-environments: Scene Name: 1.11_vs_11_easy (Simplified rules, easy difficulty) 2.11_vs_11_intermediate (Simplified rules, medium difficulty) 3.11_vs_11_hard (Simplified rules, high difficulty) 4.11_vs_11_stochastic_easy (random rules, easy difficulty) 5.11_vs_11_stochastic_intermediate (random rules, medium difficulty) 6.11_vs_11_stochastic_hard (random rules, high difficulty) 7. academy_3_vs_1_defense (3v1 defense scenario) 8. Academy 4 vs 4 Easy (4v4 Easy Difficulty) 9. academy_4_vs_4_hard (4v4 high difficulty) Number of agents: There are 11 agents (11v11) or 4 agents (4v4) in each scenario.

[0051] Neural network structure: Encoder: 2-layer MLP, 512 neurons per layer, with ReLU activation function.

[0052] Multi-head attention module: number of attention heads H=8, and the dimensions of the query, key, and value matrices are all 64.

[0053] Reward prediction head: 2-layer MLP, 256 neurons per layer, ReLU activation function, and linear activation for the output layer.

[0054] A multi-head attention discriminator model structure is adopted as the core module. The specific steps are as follows: 1. Input stage: The local observation vectors of each agent and their actions are combined to form input features, which are then mapped to a shared latent space by the encoder.

[0055] 2. Discriminator model structure: The discriminator model includes a multi-head self-attention module for calculating the correlation weights between different agents; the output layer uses Sigmoid activation to generate the individual reward prediction value for each agent.

[0056] 3. Optimization Goal: The global environment reward is used as the ground truth signal to minimize the difference between the discriminator model output and the true global reward; and the individual reward signal is fed back to the policy network through gradient backpropagation to achieve joint training.

[0057] 4. Training and Update Strategies: The discriminator model and the policy network are updated alternately, and the discriminator model is optimized every few training steps to ensure the stability and consistency of the reward signal.

[0058] 5. Effect Verification: Evaluations were conducted in nine sub-environments of the Google Research Football task set, and the results show that this implementation significantly outperforms existing baseline algorithms in terms of average win rate, convergence speed, and training stability. Specifically, as follows... Figure 3 As shown, the ability of the discriminator model to model collaborative behavior is evaluated by calculating the Pearson correlation coefficient, Spearman rank correlation coefficient, and mutual information between the individual rewards generated by the discriminator model and the global task objective.

[0059] Experimental results show that in simple collaborative scenarios (such as 3v1 with a goalkeeper and corner kick tactics), individual rewards and global rewards are positively correlated, with Spearman correlation coefficients exceeding 0.42 and mutual information values ​​ranging from 0.15 to 0.22 bits, indicating that the discriminator model can effectively identify the immediate contributions of each agent to the team's success.

[0060] In more complex coordination tasks (such as passing and shooting, and scoring through breakthroughs), the correlation is significantly improved, with Pearson correlation coefficients reaching 0.6–0.8 and Spearman correlation coefficients reaching 0.7–0.85. The mutual information reaches a maximum of 0.42 bits, indicating that the present invention can capture the temporal dependencies and cooperative patterns among multiple agents and achieve more semantically meaningful credit allocation.

[0061] Even in sparse or delayed reward scenarios (such as countering hard modes or fighting lazy opponents), although the linear correlation decreases, the mutual information remains positive (minimum 0.08 bits), indicating that the discriminator model can still extract nonlinear information that is valuable for individual learning and demonstrates good robustness.

[0062] In summary, the experimental results verify that the credit allocation mechanism provided by this invention can accurately and stably decompose the global reward into local signals reflecting individual contributions, especially in complex collaborative tasks, demonstrating stronger modeling capabilities and significantly improving the efficiency and stability of multi-agent policy learning.

[0063] Implementation Scheme 2: Modified Scheme A (Independent Optimized Version of Discriminator Model) In this scheme, the discriminator model and the policy network are optimized independently: the policy generates trajectory samples to maximize the expected reward through a reinforcement learning algorithm, while the discriminator model learns expert trajectory features and outputs the reward. Although this scheme weakens the discriminative power of the reward signal, it is equivalent to the main scheme in principle and still falls within the scope of protection of this invention.

[0064] Implementation Plan 3: Modified Plan B (Multi-task Migration Version) In this scheme, the discriminator model parameters are shared across multiple different tasks, and only the parameters of the attention module are adjusted to achieve task-adaptive transfer. This scheme maintains the consistency of reward learning and is applicable to cross-environment generalization tasks, representing a reasonable extension of this invention.

[0065] In some embodiments, this application also provides a credit allocation device in multi-agent cooperation, comprising: a data acquisition module for acquiring joint data of the multi-agent system, the joint data including observation information and actions performed by each agent; a discriminator model including a neural network constructed using a multi-head attention mechanism for processing the joint data to model the interaction dependencies between multiple agents and outputting an auxiliary reward signal corresponding to each agent; a reward fusion module for generating a fused reward signal based on the auxiliary reward signal and a global reward signal from the environment; and a policy training module for training and updating the multi-agent policy model using the fused reward signal to optimize the cooperative policy of the multi-agent system.

[0066] In some embodiments, the discriminator model further includes: an encoder submodule for mapping the observation information and actions of each agent to corresponding feature vectors; a multi-head attention submodule for modeling the interaction dependencies of the feature vectors of all agents and outputting context-enhanced feature representations; and a reward prediction submodule for predicting and outputting auxiliary reward signals for each agent based on the context-enhanced feature representations.

[0067] In some embodiments, this application also provides a multi-agent decision control system, including: the credit allocation device of this application; an environment interaction module for controlling multiple agents to perform actions in the environment and obtain global reward signals; and a policy execution module for loading a trained multi-agent policy model and generating decision actions for each agent according to the model.

[0068] In some embodiments, this application also provides an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and the one or more computer programs include instructions for performing the methods of this application.

[0069] In some embodiments, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of this application.

[0070] In summary, this invention introduces a discriminator-based credit assignment (MACAD) mechanism into the Multi-Agent Reinforcement Learning (MARL) framework, achieving automatic decomposition and optimized generation of individual reward signals. This significantly improves the learning efficiency and stability of multi-agent systems in complex collaborative tasks. Compared with existing technologies, this invention has the following technical effects and advantages: 1. A stable credit allocation process: The discriminator model network uses a multi-head attention module to perform interpretable decomposition of the global reward signal, modeling the contributions of different agents. This method avoids the high variance problem of traditional equal reward distribution or gradient-based backpropagation, making reward allocation more stable and consistent with task semantics. Experimental results show that in complex environments such as GFootball, MACAD exhibits a more stable credit allocation process compared to baseline algorithms (such as QMIX, VDN, and MA-GAIL), reducing the uncertainty of policy updates.

[0071] 2. Accelerated convergence and performance improvement: The individualized reward signal generated by the discriminator model can provide more direct optimization directions for each agent, significantly improving the training convergence speed. Experimental results show that this invention can achieve faster convergence speed and higher win rate than baseline algorithms (QMIX, VDN, MA-GAIL, etc.) in complex environments such as GFootball.

[0072] 3. Enhanced generalization and robustness: The multi-head attention structure enables the discriminator model to flexibly model inter-individual dependencies under various task and environment configurations, thereby improving the algorithm's generalization ability under unseen tasks and its robustness under noisy observations.

[0073] The multi-head attention structure enables the discriminator model to flexibly model inter-agent dependencies under various task and environmental configurations, improving the algorithm's generalization ability. Experimental results show that in complex coordination tasks (such as passing and shooting, and scoring through breakthroughs), the Pearson correlation coefficient reaches 0.6–0.8, indicating that this invention can capture temporal dependencies and collaborative patterns among multiple agents, achieving more semantically meaningful credit allocation. 4. Automated reward learning and interpretability enhancement: Generative discriminator models can automatically infer individual rewards without requiring manual reward function design. The key to this mechanism lies in the discriminator model's ability to model the behavior and contributions of different agents in collaborative tasks through a multi-head attention module, thereby generating personalized reward signals for each agent. The multi-head attention module quantifies the collaborative relationships between agents. For example, in a G-Football task, agent A's passing action plays a crucial role in agent B's score. The attention module reveals that agent B's attention weights are significantly concentrated on agent A's actions, indicating that agent A's behavior contributes more to agent B's success. Therefore, this invention innovatively improves the reward design and credit allocation mechanism in the traditional MARL training process, achieving more efficient, stable, and interpretable multi-agent collaborative learning.

[0074] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] The background section of this application may include background information about the problem or environment in which this application is made, and is not necessarily a description of prior art. Therefore, the content included in the background section does not constitute an admission of prior art by the applicant.

[0079] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of this application and should not be construed as limiting the specific implementation of this application to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of this application, and such substitutions or modifications should all be considered within the scope of protection of this application. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the described specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments and advantages of this application have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A credit allocation method in multi-agent cooperation, characterized in that, Includes the following steps: S1. Obtain joint data of the multi-agent system, the joint data including the observation information and actions performed by each agent; S2. Input the joint data into the trained discriminator model, wherein the discriminator model uses an attention-based interaction dependency modeling module to model the interaction dependencies between multiple agents. S3. The joint data is processed by the discriminator model to generate an auxiliary reward signal corresponding to each agent, wherein the auxiliary reward signal is used to decompose and distribute the global reward signal from the environment; S4. Generate a fused reward signal based on the auxiliary reward signal and the global reward signal from the environment; S5. The multi-agent policy model is trained and updated using the fused reward signal to optimize the cooperative strategy of the multi-agent system.

2. The method according to claim 1, characterized in that, The attention mechanism in step S2 is a multi-head attention mechanism. The discriminator model is trained as follows: a training sample set is obtained, which includes expert trajectory data generated by expert policies or environmental trajectory data labeled with global reward signals from environmental feedback; the parameters of the discriminator model are optimized with the goal of minimizing the difference between the reward signal output by the discriminator model and the expert trajectory data or the global reward signal.

3. The method according to claim 1, characterized in that, Step S4 specifically involves: following the formula: Generate a fusion reward signal, in which, This represents the fusion reward signal of the i-th agent at time step t. This represents the global reward signal at time step t. This represents the auxiliary reward signal generated by the discriminator model for the i-th agent. This is the preset auxiliary reward weighting coefficient.

4. The method according to claim 1, characterized in that, The discriminator model in step S3 processes the joint data as follows: mapping the observation information and actions of each agent to corresponding feature vectors through the encoder network; inputting the feature vectors of all agents to the multi-head attention layer, calculating the attention weights between agents, and outputting the new feature representation of each agent after context enhancement; The new feature representation of each agent is input into the corresponding reward prediction head network, and the auxiliary reward signal of each agent is output.

5. A credit allocation device for multi-agent cooperation, characterized in that, include: The data acquisition module is used to acquire joint data of the multi-agent system, which includes the observation information and actions performed by each agent. The discriminator module includes a neural network built on an attention mechanism, used to process the joint data to model the interaction dependencies between multiple agents and output an auxiliary reward signal corresponding to each agent. The reward fusion module is used to generate a fused reward signal based on the auxiliary reward signal and the global reward signal from the environment; The policy training module is used to train and update the multi-agent policy model using the fused reward signal to optimize the cooperative policy of the multi-agent system.

6. The credit allocation device according to claim 5, characterized in that, The discriminator module further includes: an encoder submodule, used to map the observation information and actions of each agent into corresponding feature vectors; a multi-head attention submodule, used to model the interaction dependencies of the feature vectors of all agents and output context-enhanced feature representations; and a reward prediction submodule, used to predict and output auxiliary reward signals for each agent based on the context-enhanced feature representations.

7. A multi-agent decision control system, characterized in that, include: The credit allocation device as described in claim 5 or 6; The environment interaction module is used to control multiple agents to perform actions in the environment and obtain global reward signals; The policy execution module is used to load the trained multi-agent policy model and generate decision actions for each agent based on the multi-agent policy model.

8. An electronic device, characterized in that, include: One or more processors; Memory; One or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, the one or more computer programs including instructions for performing the method as described in any one of claims 1-4.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method as described in any one of claims 1-4.

10. A multi-agent cooperative method based on credit allocation, characterized in that, Includes the following steps: The credit allocation method in multi-agent cooperation as described in any one of claims 1 to 4 is executed to generate a fusion reward signal for each agent; Based on the fused reward signal, a reinforcement learning algorithm is used to perform distributed or centralized training on the decision-making strategy of each agent in order to optimize its individual strategy. During or after the training process, the updated individual policies are used to control the corresponding agents to perform actions in the environment, thereby achieving multi-agent collaboration.

Images