Method and device for evaluating contribution degree of agent of multi-agent system, storage medium, electronic equipment and computer program product

Abstract

This application discloses a method, apparatus, storage medium, electronic device, and computer program product for evaluating the contribution of agents in a multi-agent system. The evaluation method includes: determining the graph neural network model corresponding to the multi-agent system based on received environmental state information, wherein the environmental state information includes at least agent state information, task requirements, and environmental information; training an action network and a graph neural network model based on the environmental state information; determining the action information of the nodes in the graph neural network model using the trained action network based on the environmental state information; and determining the contribution of the agents using the trained graph neural network model based on the environmental state information and the action information.

Description

Technical Field

[0001] This application relates to the field of multi-agent system technology, and more specifically, to a method, apparatus, storage medium, electronic device, and computer program product for evaluating the contribution of agents in a multi-agent system. Background Technology

[0002] Collaborative tasks in multi-agent systems typically involve complex interactions and dynamic environmental changes. For example, in rescue search scenarios, multiple agents need to quickly locate targets in unknown environments while avoiding interference with each other; in logistics sorting scenarios, agents need to dynamically adjust their division of labor based on real-time task requirements to improve overall efficiency.

[0003] Graph Neural Networks (GNNs) have demonstrated significant advantages in modeling complex relational networks. GNN models capture the interactions between agents through graph structures and aggregate local and global information using message passing mechanisms. For example, GNNs have proven effective in modeling dynamic topological relationships in social network analysis or traffic prediction.

[0004] However, the inventors of this application have discovered that the application of GNNs in multi-agent cooperative systems suffers from insufficient dynamic adjustment and a lack of effective contribution evaluation. For example, the graph structure of a multi-agent system needs to be dynamically adjusted according to task requirements or environmental changes, which places higher demands on the real-time performance of GNNs. Transforming the embedding vectors output by GNNs into interpretable "contribution" metrics and combining them with the task objective function is also a current research challenge.

[0005] The content of the background section is merely technology known to the public and does not necessarily represent existing technology in the field. Summary of the Invention

[0006] According to one aspect of this application, this application provides a method for evaluating the contribution of agents in a multi-agent system. The evaluation method includes: determining the graph neural network model corresponding to the multi-agent system based on received environmental state information, wherein the environmental state information includes at least agent state information, task requirements, and environmental information; training an action network and a graph neural network model based on the environmental state information; determining the action information of the nodes of the graph neural network model through the trained action network based on the environmental state information; and determining the contribution of the agents through the trained graph neural network model based on the environmental state information and the action information.

[0007] According to some embodiments of this application, determining the graph neural network model corresponding to the multi-agent cooperative system based on the received environmental state information includes: determining the node model based on the agent state information and task requirements; determining the edge structure based on the agent state information, task requirements, and environmental information; and determining the graph neural network model based on the node model and edge structure.

[0008] According to some embodiments of this application, training an action network and a graph neural network model based on environmental state information includes: performing at least one action network training step and at least one graph neural network model training step. The action network training step includes: determining training action information for the agent using the action network based on the environmental state information; determining the agent's value evaluation parameters using a value network based on the training action information, environmental state information, and a preset reward function; updating the environmental state information based on the value evaluation parameters; and training the action network based on the updated environmental state information. The graph neural network model training step includes: determining an experience trajectory based on the environmental state information, training action information, and updated environmental state information; determining global task performance based on the experience trajectory; and training the graph neural network model based on the global task performance.

[0009] According to some embodiments of this application, the preset reward function includes a preset cooperation function and a preset game competition function.

[0010] According to some embodiments of this application, determining the agent's contribution based on environmental state information and action information through a trained graph neural network model includes: mapping the environmental state information and action information to a high-dimensional embedding space through the first layer of the trained graph neural network to obtain the first feature vector of the node; determining the node's neighbor information based on the environmental state information; determining the node's updated feature vector based on the first feature vector and neighbor information through subsequent layers of the trained graph neural network; and determining the agent's contribution based on the updated feature vector through the linear layers of the trained graph neural network.

[0011] According to another aspect of this application, this application also provides an evaluation device for the contribution of agents in a multi-agent system. The evaluation device includes a processing module. The processing module determines the graph neural network model corresponding to the multi-agent system based on received environmental state information, wherein the environmental state information includes at least agent state information, task requirements, and environmental information; the processing module trains an action network and a graph neural network model based on the environmental state information; the processing module determines the action information of the nodes of the graph neural network model through the trained action network based on the environmental state information; and the processing module determines the contribution of the agents based on the environmental state information and the action information through the trained graph neural network model.

[0012] According to some embodiments of this application, the processing module determines the node model based on the agent's state information and task requirements; the processing module determines the edge structure based on the agent's state information, task requirements, and environmental information; and the processing module determines the graph neural network model based on the node model and edge structure.

[0013] According to another aspect of this application, this application also provides a non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is capable of implementing the evaluation method described above.

[0014] According to another aspect of this application, this application also provides an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the evaluation method described above.

[0015] According to another aspect of this application, this application also provides a computer program product, comprising: a computer program stored on a computer-readable storage medium; the computer program includes program instructions that, when executed by a computer, cause the computer to perform the evaluation method as described above. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating an evaluation method 1000 according to an embodiment of this application is shown;

[0018] Figure 2 A flowchart illustrating step S110 according to an embodiment of this application is shown;

[0019] Figure 3 A flowchart illustrating step S120 according to an embodiment of this application is shown;

[0020] Figure 4 A flowchart illustrating step S121 according to an embodiment of this application is shown;

[0021] Figure 5 A flowchart illustrating step S122 according to an embodiment of this application is shown;

[0022] Figure 6 A flowchart illustrating step S140 according to an embodiment of this application is shown;

[0023] Figure 7 A schematic diagram of the structure of an evaluation apparatus according to an embodiment of this application is shown;

[0024] Figure 8 A schematic diagram of an intelligent agent action network according to an embodiment of this application is shown.

[0025] Explanation of reference numerals in the attached figures:

[0026] Evaluation device 20; processing module 21. Detailed Implementation

[0027] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0028] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of these specific details, or other methods, components, materials, devices, etc. In these cases, well-known structures, methods, devices, implementations, materials, or operations will not be shown or described in detail.

[0029] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0030] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order.

[0031] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0032] The English terms used in this application, their full English names, and their corresponding Chinese definitions are as follows:

[0033] RL stands for Reinforcement Learning.

[0034] MARL stands for Multi-Agent Reinforcement Learning.

[0035] CTDE, Centralized Training with Decentralized Execution;

[0036] MADDPG, Multi-Agent Deep Deterministic Policy Gradient.

[0037] Collaborative tasks in multi-agent systems typically involve complex interactions and dynamic environmental changes. For example, in rescue search scenarios, multiple agents need to quickly locate targets in unknown environments while avoiding interference with each other; in logistics sorting scenarios, agents need to dynamically adjust their division of labor based on real-time task requirements to improve overall efficiency.

[0038] Traditional multi-agent cooperative methods suffer from several drawbacks: policy game theory and non-stationarity (e.g., policy game theory in multi-agent environments can lead to non-stationarity, where changes in other agents' policies can affect the current agent's decision-making process, resulting in unstable training); the contradiction between local observation and global cooperation (e.g., agents typically only acquire local observation information and struggle to fully perceive the global state, limiting their collaborative capabilities in complex tasks); and limitations in the fairness of contribution evaluation and reward allocation (e.g., in multi-agent systems, accurately evaluating each agent's contribution to the task and allocating rewards or resources accordingly is a long-standing challenge. Traditional methods often fail to reflect the actual role of agents, leading to efficiency losses).

[0039] To address the aforementioned issues, existing technologies have attempted to optimize them through the following methods: for example, adopting a centralized control and distributed execution approach, with some systems using a centralized controller to coordinate the behavior of intelligent agents. However, the inventors of this application have found that this method is less flexible in dynamic environments and is difficult to adapt to the collaborative needs of heterogeneous intelligent agents (with different capabilities or task types).

[0040] For example, by adopting an extension of reinforcement learning, multi-agent reinforcement learning alleviates the non-stationarity problem by introducing a centralized training and distributed execution (CTDE) framework. For instance, MADDPG (Multi-Agent Deep Deterministic Policy Gradient Algorithm) allows the Critic network to access global information to optimize the policy, but it still struggles to solve the problem of modeling complex dependencies among multiple agents.

[0041] For example, many systems rely on preset rules or static task allocation strategies when using static task assignment and fixed rules. However, the inventors of this application have discovered that this approach lacks adaptability to dynamic task changes. For instance, in a rescue scenario, if obstacles suddenly appear in the environment or task priorities are adjusted, the system may not be able to respond quickly.

[0042] For example, in terms of coarse-grained contribution assessment, existing technologies typically evaluate agent contributions based on simple task completion metrics (such as the number of tasks or time). However, the inventors of this application have found that this approach ignores the interaction relationships between agents and the dynamic changes in the overall system performance.

[0043] Graph Neural Networks (GNNs) have demonstrated significant advantages in modeling complex relational networks. GNN models capture the interactions between agents through graph structures and aggregate local and global information using message passing mechanisms. For example, GNNs have proven effective in modeling dynamic topological relationships in social network analysis or traffic prediction.

[0044] However, the inventors of this application have discovered that the application of GNNs in multi-agent cooperative systems suffers from insufficient dynamic adjustment and a lack of effective contribution evaluation. For example, the graph structure of a multi-agent system needs to be dynamically adjusted according to task requirements or environmental changes, which places higher demands on the real-time performance of GNNs. Transforming the embedding vectors output by GNNs into interpretable "contribution" metrics and combining them with the task objective function is also a current research challenge.

[0045] According to one aspect of this application, a method 1000 for evaluating the contribution of agents in a multi-agent system is provided. The evaluation method 1000 can be executed by a computer system, which, exemplarily, can be a host or server with data processing capabilities.

[0046] See Figure 1 Evaluation method 1000 can be applied for steps S110-S140.

[0047] In step S110, the computer system determines the graph neural network model corresponding to the multi-agent cooperative system based on the received environmental state information.

[0048] According to the example embodiment, a multi-agent system can be a system that can complete highly complex tasks in dynamic and uncertain environments through the collaborative work of multiple agents. The agents can be heterogeneous agents such as drones, unmanned vehicles, and handling robots, and the multi-agent system can be applied to complex scenarios such as rescue search and rescue, and logistics sorting.

[0049] Environmental state information can be the state of a multi-agent system and the information about its environment. Environmental state information includes at least agent state information, task requirements, and environmental information.

[0050] The state information of an agent can be information describing the state of all agents, such as speed, acceleration, and battery level.

[0051] Task requirements can be the state of the target task of a multi-agent system, such as task type, task priority, task deadline, task completion progress, and the resources required for the task (load capacity, power, sensor type), etc.

[0052] Environmental information can be information about the environment in which a multi-agent system is located. For example, environmental information may include the observation location, the location and size of obstacles, terrain features, communication conditions, and the location and motion state of dynamic objects.

[0053] Environmental state information can form a global state matrix. Each row of the global state matrix corresponds to a feature vector of an agent, such as its observation position, speed, battery level, and task information.

[0054] Multi-agent systems can be abstracted as graph structures. Agents and tasks can be considered as nodes in the graph, while the interactions between agents or between agents and tasks can be considered as edges.

[0055] Graph neural network models can be graph structures for multi-agent systems built based on environmental state information.

[0056] Computer systems can construct graph neural network models based on the nodes and edges of a graph structure.

[0057] In step S120, the computer system trains the action network and graph neural network model based on the environmental state information.

[0058] According to the example embodiment, the action network can be an Actor network. The computer system can be based on the MADDPG algorithm, with each agent configured with an Actor network and a Critic network. The computer system can train the action network using the MADDPG algorithm. The training phase employs a centralized training mode, allowing the Critic network to access global information, i.e., all environmental state information. The execution phase is decentralized, with each agent making autonomous decisions based solely on local observations and policy, determining its training action information. Local observations can be the agent's own environmental state information.

[0059] The Critic network takes the training action information and global state of all agents as input and outputs the value evaluation of the corresponding agent, thus solving the non-stationarity problem in multi-agent environments. Each Actor network adopts a policy gradient update mechanism, which optimizes the decision policy by maximizing the expected reward while ensuring continuous action output.

[0060] Computer systems can formalize multi-agent task collaboration problems as constrained Markov games. During the learning process, each agent interacts with other agents based on local observations, optimizing its strategy by maximizing its cumulative reward. In collaborative scenarios, agents may cooperate or compete; for example, when dividing tasks for a search, reasonable avoidance and cooperation are necessary. The centralized Critic structure of MADDPG can coordinate the strategies of each agent, promoting teamwork to achieve the task objective. The computer system can update environmental state information using the value assessment parameters output by the Critic network.

[0061] The computer system can determine an empirical trajectory based on environmental state information, training action information, and updated environmental state information; it can also determine global task performance based on this empirical trajectory. Furthermore, the computer system can train a graph neural network model based on the global task performance.

[0062] In step S130, the computer system determines the action information of the nodes of the graph neural network model based on the environmental state information and the trained action network.

[0063] According to the example embodiment, the action information of a node can be the action information of an agent. The computer system can determine the action information of each node (i.e., the agent) based on the environmental state information and through the trained Actor network.

[0064] In step S140, the computer system determines the contribution of the agent based on the environmental state information and action information, using the trained graph neural network model.

[0065] According to the example implementation, contribution can be a quantitative indicator of the positive impact of the agent on the completion of the target task.

[0066] For example, a computer system can use the first layer of a graph neural network to map environmental state information and action information into a high-dimensional embedding space to obtain the first feature vector of a node; the computer system can then determine the neighbor information of the node based on the environmental state information; the computer system can then use the subsequent layers of the trained graph neural network to determine the updated feature vector of the node based on the first feature vector and the neighbor information; and the computer system can then use the linear layers of the trained graph neural network to determine the contribution of the agent based on the updated feature vector.

[0067] Through the above embodiments, the technical solution of this application determines the graph neural network model corresponding to the multi-agent collaborative system using environmental state information. The technical solution of this application trains an action network and a graph neural network model using environmental state information. The technical solution of this application determines the action information of nodes in the graph neural network model using the trained action network based on environmental state information. The technical solution of this application determines the contribution of an agent using both environmental state information and action information, through the trained graph neural network model.

[0068] The evaluation method provided in this application organically combines MADDPG reinforcement learning with graph neural networks. MADDPG enables policy competition and coordination among agents, while centralized Criticism addresses the non-stationarity problem in multi-agent games. Simultaneously, graph neural networks dynamically evaluate the contribution of each agent at the cluster level, enhancing the system's perception of the overall structure. This application supports collaboration among heterogeneous agents and improves the adaptability of multi-agent systems to dynamic task changes.

[0069] Optionally, see Figure 2 Step S110 may include steps S111-S113.

[0070] In step S111, the computer system determines the node model based on the agent's state information and task requirements.

[0071] According to the example embodiment, the node model can be a collection of nodes in a graph structure. The computer system can use entities such as agents and tasks as nodes in the graph structure, and all nodes together form a node model. Each node includes node features, which can be represented as feature vectors based on agent state information and task requirements.

[0072] In step S112, the computer system determines the edge structure based on the agent's state information, task requirements, and environmental information.

[0073] According to the example embodiment, the edge structure can be a collection of edges from a graph structure. The computer system can use the interactions between agents or between agents and tasks as edges in the graph structure. All edges are collected into an edge structure. Each edge includes edge features, which can be represented as a feature vector based on agent state information, task requirements, and environmental information.

[0074] In step S113, the computer system determines the graph neural network model based on the node model and edge structure.

[0075] According to the example embodiment, the computer system can construct a graph neural network model of a multi-agent system based on the node model and edge structure.

[0076] For example, a graph neural network model can include a set of nodes V and a set of edges E.

[0077] The node set V represents all agents, and the edge set E represents the effective interaction relationships between agents, such as communication links, proximity, and task cooperation dependencies. The graph structure of the graph neural network model can be stored using an adjacency matrix or an edge list. The graph structure can be used as the input to the graph neural network model. The agent action space is represented using continuous vectors, and each agent has an independent set of action variables A, such as movement speed and direction, and pick-up / placement actions. i .

[0078] Optionally, see Figure 3 Step S120 may include steps S121 and S122.

[0079] Step S121 is the step in which the computer system executes the training action network. The computer system executes step S121 at least once.

[0080] See Figure 4 Step S121 may include steps S1211-S1214.

[0081] In step S1211, the computer system determines the training action information of the agent through the action network based on the environmental state information.

[0082] According to an example embodiment, training action information can be the execution action information of an agent during the training action network process.

[0083] The computer system can input environmental status information into the action network, and the action network can output training action information.

[0084] In step S1212, the computer system determines the value evaluation parameters of the agent through a value network based on training action information, environmental state information, and a preset reward function.

[0085] According to the example embodiment, the preset reward function can be a quantitative indicator of the agent's completed training actions. The value network can be a Critic network.

[0086] Optionally, the preset reward function may include a preset cooperation function and a preset game-theoretic competition function. The preset cooperation function can be a preset reward function when the target task is a cooperative task. The preset game-theoretic competition function can be a preset reward function when the target task is a game-theoretic competition task. The value evaluation parameter can be a parameter used to evaluate the value of the agent in completing the target task.

[0087] The computer system can update the state of the agent based on the received training action information and output an instant reward for each agent.

[0088] For example, the pre-defined cooperation function can include a globally corresponding joint pre-defined cooperation function and individual pre-defined cooperation functions for each agent. In multi-agent cooperation scenarios, multiple agents share a common goal, typically maximizing a global reward. The joint pre-defined cooperation function can be:

[0089]

[0090] in, Let i be a set of agents. The training action information for each agent i is a. i The environmental state information is s, and the total training action information is a = (a1, ..., a2) N R(s,a) is the global reward function, which is a single, uniform reward signal given by the environmental state-action sequence.

[0091] A joint pre-defined cooperation function can ensure that all agents receive the same reward, encouraging them to work together to complete the task.

[0092] The pre-defined cooperation function for each agent can be a weighted reward system for each agent. The pre-defined cooperation function for each agent can be:

[0093]

[0094] Individual pre-defined collaboration functions can be applied to collaborative tasks where individuals have different abilities or responsibilities.

[0095] For example, the preset game competition function can include a preset zero-sum game function and a preset partial competition function. In a game scenario, the goals of the agents are conflicting, and the reward function design should reflect zero-sum or partial zero-sum characteristics.

[0096] The default zero-sum game function can be:

[0097] r i (t)=-rj (t);

[0098] r i (t) is the zero-sum game function preset for agent i, r j (t) is the preset zero-sum game function of agent j. The preset zero-sum game function is mainly used in adversarial scenarios such as Go, competitive games, and capture and escape scenarios.

[0099] The pre-defined partial competition function can be:

[0100] r i (t)=R i (s t ,a t )-λ·R j (s t ,a t ),λ∈[0,1];

[0101] Among them, R i (s t ,a t R represents the self-reward of agent i; j (s t ,a t ) represents the gain of agent i's opponent (agent j); λ is the weight, which can control the degree of "profiting at the expense of others".

[0102] In λ=1 and R i =R j In the case of λ = 1, the pre-defined partial competition function degenerates into the pre-defined zero-sum game function; when λ = 1, agent i is a purely self-interested agent.

[0103] According to the example embodiment, the computer system can input training action information and environmental state information into the value network, and the value network outputs value evaluation parameters.

[0104] In step S1213, the computer system updates the environmental status information based on the value assessment parameters.

[0105] According to the example embodiment, after the agent performs training actions, it affects the environmental state information, causing the environmental state information to be updated. The computer system can update the environmental state information based on value assessment parameters.

[0106] In step S1214, the computer system trains the action network based on the updated environmental state information.

[0107] According to the example implementation, the Actor network can employ a policy gradient update mechanism to optimize the decision-making policy by maximizing the expected return while ensuring continuous action output.

[0108] The computer system can input the updated environmental state information into the action network to conduct the next round of training.

[0109] The computer system can also set up a replay buffer structure to store historical interaction records of multi-agent states, actions, rewards, and next states for offline training.

[0110] Step S122 is the step of the computer system executing the training graph neural network model.

[0111] See Figure 5 Step S122 may include steps S1221-S1223.

[0112] In step S1221, the computer system determines the experience trajectory based on the environmental state information, training action information, and updated environmental state information.

[0113] According to an example embodiment, an experience trajectory can be a sequence of states and actions of an agent. The experience trajectory may include environmental state information, training action information, and updated environmental state information. It may also include immediate rewards output based on the agent's training action information.

[0114] In step S1222, the computer system determines the global task performance based on the empirical trajectory.

[0115] According to the example embodiment, global task performance can be a parameter representing the overall contribution of the agent's experience trajectory to the target task.

[0116] Computer systems can use graph neural networks to output global task performance based on empirical trajectories.

[0117] In step S1223, the computer system trains a graph neural network model based on the global task performance.

[0118] According to the example embodiment, the computer system uses global task performance as a label to train a graph neural network model and adjust the model parameters so that the contribution values ​​of each agent output by the graph neural network model are more consistent with actual contributions. The computer system can also use contribution values ​​as auxiliary rewards or inputs to influence the Critic, prompting the policy to learn towards the global optimum.

[0119] Graph neural network model parameters can be trainable variables that are automatically learned and updated during training through the backpropagation algorithm. Graph neural network model parameters can include node feature updates, weight matrices, bias vectors, attention weights, gating parameters, residual mappings, and output layer parameters, etc.

[0120] For example, a computer system can determine the node feature updates for each layer of a graph neural network model using the following formula:

[0121]

[0122] in, Let L be the weight matrix of the l-th layer of the graph neural network model. Let d be the real number field and W be the number of d. (l) The dimension. b (l) Let σ be the bias vector, and σ(·) be the nonlinear activation function. This represents the set of adjacent nodes of node v. This is the node feature update for each layer of the graph neural network model, that is, the updated state of the current node i (i.e., the current agent i) at layer l+1. The hidden state after updating the current node i (i.e. the current agent i).

[0123] bias vector b (l) Bias terms can be introduced into each layer of a graph neural network model to enhance its expressive power. The computer system can determine the bias vector according to the following formula:

[0124]

[0125] In graph attention networks, the importance of neighboring nodes is modeled through the attention mechanism, and the computer system can determine the attention weights according to the following formula:

[0126]

[0127] Where a is a learnable attention vector; ||·|| denotes the feature concatenation operation. LeakyReLU is an activation function. W is a trainable parameter matrix; The hidden state of the current node i (i.e., the current agent i); This represents the hidden state of the neighboring nodes of the current node i (i.e., the current agent i).

[0128] By introducing a gating mechanism into a gated graph neural network model to control the flow of information, the computer system can determine the gating parameters according to the following formula:

[0129]

[0130] Where ⊙ represents the Hadamard product, z v To update the door, is The degree of retention; m v for Aggregated data; h v r represents the hidden state of the current node i (i.e., the current agent i); vThe reset gate represents the effect of the hidden state of the current node i (i.e., the current agent i) before the update on the hidden state after the update; h′ is a candidate hidden state for the current node i (i.e., the current agent i), which is a hidden state that incorporates the neighbor information of the current node i; v for h v The hidden state of the current node i (i.e., the current agent i) after its update. z W r W h with U z U r U h All of these are trainable parameter matrices.

[0131]

[0132] in, This represents a graph convolution or message passing function, specifically the residual mapping parameter Θ. (l) This is the set of parameters for this layer.

[0133] The output layer is typically a fully connected classification or regression layer. The computer system can determine the output layer formula based on the following formula:

[0134]

[0135] Among them, W out Let b be the weight matrix of the output layer. out Let L be the bias vector of the output layer, and L represent the last layer. These are the output layer parameters.

[0136] For example, see Figure 8 The agent action network can be the decision probability distribution of the agent. The agent action network integrates the graph neural network model and the Actor-Critic network structure.

[0137] The computer system can perform environmental initialization operations. It randomly deploys agents and target / task points based on the task scenario and assigns initial roles or capability parameters to each agent.

[0138] The computer system can receive environmental state data in real time. In each training round, multiple agents execute policies in the environment, and each agent generates training action information from local observations and information transmitted by the GNN (environmental state data). The multi-agent system receives the actions of all agents, updates the environmental state information, and outputs the immediate reward for each agent. The updated environmental state information and immediate rewards are stored in the replay buffer.

[0139] The computer system can sample a batch of training data from the replay buffer at fixed intervals to update the parameters of the Critic and Actor networks. For example, the system can update the Critic network parameters by iterating through the multi-agent joint temporal difference error. The system can also improve the policy performance of the Actor network using policy gradients. Updating the Critic and Actor network parameters can employ target networks and soft update mechanisms to ensure learning stability.

[0140] The computer system can train graph neural network models synchronously or alternately. It can utilize the experience trajectories collected in the current round to calculate global task performance. Using this global task performance as a label, the GNN is trained and its parameters adjusted to ensure that the output contributions of each agent more accurately reflect actual contributions. The computer system can also use these contributions as supplementary rewards or as input to influence the Critic, prompting the Actor network's policy to learn towards the global optimum.

[0141] The computer system can execute steps S121 and S122 multiple times until the graph neural network model converges (e.g., the loss function of the graph neural network model is below a threshold). The training process employs a centralized global Critic perspective and graph structure information fusion, enabling each agent to maintain decentralized execution capabilities while considering the overall interests, thus achieving effective collaboration in complex tasks.

[0142] After training, only the Actor policy and the graph neural network model are retained. During actual execution, each agent uses the trained Actor network to select action information based on the current environmental state information and the graph neural network model embeddings. The graph neural network model continuously updates the local contribution evaluation to guide task selection or fine-tuning of the cooperation strategy.

[0143] Through the above embodiments, the technical solution of this application can alleviate the high variance and instability problems of traditional RL in multi-agent environments by setting up an Actor network for each agent i, so that each agent will take into account the influence of the behavior of other agents when updating the policy.

[0144] The technical solution of this application utilizes the centralized Critic network of MADDPG, where each agent's policy considers global information during training, significantly improving collaborative efficiency. The graph neural network model enhances team perception, fusing inter-agent correlation information into decision-making, resulting in more accurate and consistent collaborative decisions. Experiments show that combining the information fusion mechanism of GNNs can significantly improve the multi-agent cooperation effect and convergence speed.

[0145] The technical solution of this application uses the task completion rate and resource utilization efficiency of the agent after completing the task as the global performance as the supervision target, and trains the graph neural network model by minimizing the difference between the contribution prediction and the actual performance.

[0146] Graph neural network models can be trained simultaneously with the main RL loop, enabling contribution evaluation to assist policy optimization. If necessary, graph neural networks can also incorporate attention mechanisms to dynamically adjust edge weights based on task urgency or agent scarcity, reflecting the relative importance of different agents in the current task.

[0147] Optionally, see Figure 6 Step S140 may include steps S141-S144.

[0148] In step S141, the computer system maps environmental state information and action information to a high-dimensional embedding space through the first layer of the trained graph neural network to obtain the first feature vector of the node.

[0149] According to the example embodiment, the first feature vector can be the feature vector of the node after mapping environmental state information and action information to a high-dimensional embedding space.

[0150] Graph neural network models can use multi-layer convolution or attention mechanisms for feature aggregation.

[0151] For example, a computer system can use the agent's current position, state features, and task information as input to the first layer of a graph neural network. The first layer of the graph neural network takes the local observations of each node as input. i The input features of the task state are mapped to a high-dimensional embedding space to obtain the first feature vector.

[0152] In step S142, the computer system determines the neighbor information of the node based on the environmental status information.

[0153] According to the example embodiment, neighbor information can be the status information of a node's neighboring nodes. For example, node information can include the status, location, task requirements, and environmental information of neighboring nodes.

[0154] In step S143, the computer system determines the aggregated feature vector of a node based on the first feature vector and neighbor information through the subsequent layers of the trained graph neural network.

[0155] According to the example implementation, the aggregated feature vector can be the node features aggregated by performing multi-layer convolution or attention mechanism on the nodes.

[0156] For example, subsequent layers of a graph neural network can continuously update nodes by aggregating neighbor information and the first feature vector.

[0157] In step S144, the computer system determines the agent's contribution based on the updated feature vector through the linear layer of the trained graph neural network.

[0158] According to an example embodiment, the computer system can convert aggregated feature vectors into contribution scores c through a linear layer. i .

[0159] Through the above embodiments, the technical solution of this application utilizes graph neural networks to model the cluster structure and agent contributions, achieving refined contribution evaluation. Compared with traditional simple accumulation or equal distribution of rewards, the technical solution of this application can allocate more fair and reasonable benefits based on the actual role of agents in a multi-agent system, avoiding efficiency losses caused by one-sided rewards. Accurate evaluation results help the system effectively incentivize and adjust different agents, further improving the overall performance of multi-agent systems.

[0160] According to another aspect of this application, this application also provides an evaluation device 20 for the agent contribution of a multi-agent system. The evaluation device 20 can perform the evaluation method 1000 described above. See also... Figure 7 The evaluation device 20 includes a processing module 21.

[0161] According to the example embodiment, the processing module 21 determines the graph neural network model corresponding to the multi-agent system based on the received environmental state information, wherein the environmental state information includes at least agent state information, task requirements, and environmental information.

[0162] The processing module 21 trains the action network and graph neural network model based on the environmental state information.

[0163] Based on the environmental state information, the processing module 21 determines the action information of the nodes in the graph neural network model through the trained action network.

[0164] The processing module 21 determines the contribution of the agent based on the environmental state information and action information, using the trained graph neural network model.

[0165] Environmental state information, graph neural network model, action network, action information, and contribution have been described in the above evaluation method 1000, and therefore will not be repeated here.

[0166] Through the above embodiments, the technical solution of this application determines the graph neural network model corresponding to the multi-agent collaborative system using environmental state information. The technical solution of this application trains an action network and a graph neural network model using environmental state information. The technical solution of this application determines the action information of nodes in the graph neural network model using the trained action network based on environmental state information. The technical solution of this application determines the contribution of an agent using both environmental state information and action information, through the trained graph neural network model.

[0167] This application organically combines MADDPG reinforcement learning with graph neural networks. MADDPG enables policy competition and coordination among agents, while centralized Criticism addresses the non-stationarity problem in multi-agent games. Simultaneously, graph neural networks evaluate the contribution of each agent at the cluster level, enhancing the system's perception of the overall structure. This application supports collaboration among heterogeneous agents and improves the adaptability of multi-agent systems to dynamic task changes.

[0168] Optionally, the processing module 21 determines the node model based on the agent's state information and task requirements.

[0169] The processing module 21 determines the edge structure based on the agent's state information, task requirements, and environmental information.

[0170] The processing module 21 determines the graph neural network model based on the node model and edge structure.

[0171] The node model and edge structure have been described in the evaluation method 1000 above, so they will not be repeated here.

[0172] Optionally, the processing module 21 executes the training action network step at least once.

[0173] Based on the environmental state information, the processing module 21 determines the training action information of the agent through the action network.

[0174] The processing module 21 determines the value evaluation parameters of the agent through a value network based on the training action information, environmental state information, and preset reward function.

[0175] The processing module 21 updates the environmental status information based on the value assessment parameters.

[0176] The processing module 21 trains the action network based on the updated environmental state information.

[0177] Processing module 21 executes the training graph neural network model step at least once.

[0178] The processing module 21 determines the experience trajectory based on the environmental state information, training action information, and updated environmental state information.

[0179] Processing module 21 determines the empirical trajectory based on the empirical trajectory.

[0180] Processing module 21 trains a graph neural network model based on the overall task performance.

[0181] Training action information, value networks, value assessment parameters, experience trajectories, and assessment methods described above (1000) will not be repeated here.

[0182] Optionally, the processing module 21 maps the environmental state information and action information to a high-dimensional embedding space through the first layer of the trained graph neural network to obtain the first feature vector of the node.

[0183] The processing module 21 determines the neighbor information of the node based on the environmental status information.

[0184] The processing module 21 determines the updated feature vector of a node based on the first feature vector and neighbor information through the subsequent layers of the trained graph neural network.

[0185] Processing module 21 determines the agent's contribution based on the updated feature vector through the linear layer of the trained graph neural network.

[0186] The first feature vector, neighbor information, and updated feature vector have been described in the evaluation method 1000 above, so they will not be repeated here.

[0187] According to another aspect of this application, this application also provides a non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is capable of implementing the evaluation method described above.

[0188] According to another aspect of this application, this application also provides an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the evaluation method described above.

[0189] According to another aspect of this application, this application also provides a computer program product, comprising: a computer program stored on a computer-readable storage medium; the computer program includes program instructions that, when executed by a computer, cause the computer to perform the evaluation method as described above.

[0190] Finally, it should be noted that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions of the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for evaluating the contribution of agents in a multi-agent system, characterized in that, The evaluation method includes: S110, Based on the received environmental state information, determine the graph neural network model corresponding to the multi-agent system, wherein the environmental state information includes at least agent state information, task requirements, and environmental information; S120, Train the action network and the graph neural network model based on the environmental state information; S130, Based on the environmental state information, determine the action information of the nodes of the graph neural network model through the trained action network; S140, Based on the environmental state information and the action information, the contribution of the agent is determined through the trained graph neural network model; S110, based on the received environmental state information, determines the graph neural network model corresponding to the multi-agent cooperative system, including: S111, determine the node model based on the agent's state information and the task requirements; S112, determine the edge structure based on the agent state information, the task requirements, and the environment information; S113, Determine the graph neural network model based on the node model and the edge structure; S120, based on the environmental state information, trains the action network and the graph neural network model, including: Execute S121 at least once to train the action network steps, including: S1211, Based on the environmental state information, determine the training action information of the agent through the action network; S1212, Based on the training action information, the environmental state information, and the preset reward function, the value evaluation parameters of the agent are determined through a value network; S1213, Update the environmental status information according to the value assessment parameters; S1214, Train the action network based on the updated environmental state information; The S122 step, training the graph neural network model, must be executed at least once, including: S1221, Determine the experience trajectory based on the environmental state information, the training action information, and the updated environmental state information; S1222, Determine the global task performance based on the aforementioned experience trajectory; S1223, Train the graph neural network model based on the global task performance; In step S140, based on the environmental state information and the action information, the contribution of the agent is determined using a trained graph neural network model, including: S141, through the first layer of the trained graph neural network, the environmental state information and the action information are mapped to a high-dimensional embedding space to obtain the first feature vector of the node; S142, Determine the neighbor information of the node based on the environmental state information; S143, through the subsequent layers of the trained graph neural network, the updated feature vector of the node is determined based on the first feature vector and the neighbor information; S144, through the linear layer of the trained graph neural network, the contribution of the agent is determined according to the updated feature vector; The intelligent agent can be a drone, an unmanned vehicle, or a transport robot. The agent's state information includes speed, acceleration, or battery level; The task requirements include task type, task priority, task deadline, task completion progress, and resources required for the task. The environmental information includes the observation location, the location and size of obstacles, terrain features, communication conditions, and the location and motion state of dynamic objects; The action network is an Actor network; The graph neural network model is a graph structure of the multi-agent system constructed based on the environmental state information; The action information of the node is the execution action information of the intelligent agent; The contribution is a quantitative indicator of the positive impact of the agent on the completion of the target task; The node model is a collection of all nodes, and the nodes are the agent and the task; The edge structure is a set of all edges, and the edges are between agents or between agents. Interaction relationships between tasks; The training action information refers to the action information executed by the agent during the training of the action network; The preset reward function is a quantitative indicator of the information on the agent's completed training actions. The value assessment parameters are parameters used to evaluate the value of an agent in completing a target task; The experience trajectory includes the environmental state information, the training action information, and the updated environmental state information; The global task performance is the overall contribution parameter of the agent's experience trajectory to the target task. The first feature vector is the feature vector of the node after mapping the environmental state information and the action information into a high-dimensional embedding space; The neighbor information includes the status, location, task requirements, and environmental information of nearby nodes; The updated feature vector is the node feature aggregated by performing multi-layer convolution or attention mechanism on the node.

2. The evaluation method according to claim 1, characterized in that, The preset reward function includes a preset cooperation function and a preset game competition function, wherein the preset cooperation function is a preset reward function when the target task is a collaborative task, and the preset game competition function is a preset reward function when the target task is a game competition task.

3. A device for evaluating the contribution of agents in a multi-agent system, characterized in that, The evaluation apparatus performs the evaluation method as described in any one of claims 1-2, and the evaluation apparatus includes: The processing module determines the graph neural network model corresponding to the multi-agent system based on the received environmental state information. The environmental state information includes at least agent state information, task requirements, and environmental information, including: The processing module determines the node model based on the agent's state information and the task requirements; The processing module determines the edge structure based on the agent's state information, the task requirements, and the environment information; The processing module determines the graph neural network model based on the node model and the edge structure; The processing module trains the action network and the graph neural network model based on the environmental state information; The processing module determines the action information of the nodes of the graph neural network model based on the environmental state information and through the trained action network. The processing module determines the agent's contribution based on the environmental state information and the action information, using a trained graph neural network model, including: The processing module maps the environmental state information and the action information into a high-dimensional embedding space through the first layer of the trained graph neural network to obtain the first feature vector of the node. The processing module determines the neighbor information of the node based on the environmental state information; The processing module determines the updated feature vector of the node based on the first feature vector and the neighbor information through the subsequent layers of the trained graph neural network. The processing module determines the contribution of the agent based on the updated feature vector through the linear layer of the trained graph neural network. The processing module executes the training action network step at least once, including: The processing module determines the training action information of the agent through the action network based on the environmental state information. The processing module determines the value evaluation parameters of the agent through a value network based on the training action information, the environmental state information, and the preset reward function. The processing module updates the environmental status information based on the value assessment parameters; The processing module trains the action network based on the updated environmental state information; The processing module executes the training graph neural network model step at least once, including: The processing module determines the experience trajectory based on the environmental state information, the training action information, and the updated environmental state information; The processing module determines the overall task performance based on the experience trajectory. The processing module trains the graph neural network model based on the global task performance.

4. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the evaluation method as described in any one of claims 1-2.

5. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the evaluation method as described in any one of claims 1-2.

6. A computer program product, characterized in that, The computer program includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the evaluation method as described in any one of claims 1-2.

Images