A large-scale multi-agent collaborative control method based on role allocation

Abstract

The present application relates to a large-scale multi-agent collaborative control method, apparatus, and computer equipment based on role allocation. The method comprises: based on each agent exploring in a specified environment, collecting at least one trajectory data of each agent interacting with the environment, and storing the trajectory data corresponding to each agent in an experience replay pool; executing steps S2-1 to S2-6 for each agent in the specified environment, obtaining the individual action value of each agent, generating a joint action value based on the individual action values of all agents using a QMIX network; and updating the individual action value of each agent using a Q-learning algorithm based on the joint action value. This method can improve the accuracy and stability of multi-agent collaborative control, and also provides an efficient and reliable solution for large-scale agent collaborative systems, which can significantly improve the overall performance of the system and its ability to adapt to complex tasks.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a large-scale multi-agent collaborative control method, apparatus, and computer equipment based on role allocation. Background Art

[0002] When dealing with complex tasks that require collaboration among multiple agents, multi-agent reinforcement learning typically requires agents to undertake different subtasks to achieve a common goal, where the agents share the same reward structure and a common policy exploration space. As the complexity of the joint task and policy space increases, the agents often struggle to find the optimal joint policy. Summary of the Invention

[0003] Based on this, it is necessary to provide a large-scale multi-agent collaborative control method, device and computer equipment based on role allocation that can overcome the challenges of independent learning strategies of intelligent agents, enable intelligent agents to focus on their own tasks, and achieve better division of labor and cooperation to complete joint tasks in response to the above technical problems.

[0004] In a first aspect, the present application provides a large-scale multi-agent collaborative control method based on role allocation, which is used to automatically allocate roles to agents, including:

[0005] Step S1: Based on the exploration of each agent in a specified environment, at least one trajectory data of each agent interacting with the environment is collected, and the trajectory data corresponding to each agent is stored in an experience replay pool;

[0006] Step S2: Execute steps S2-1 to S2-6 for each agent in the specified environment to obtain the individual action value of each agent. Use the QMIX network to generate the joint action value based on the individual action values of all agents, and then proceed to step S3.

[0007] Step S3: Update the individual action value of each agent through the Q-learning algorithm based on the joint action value.

[0008] In a second aspect, the present application also provides a large-scale multi-agent collaborative control device based on role allocation, comprising:

[0009] A data collection module is used to collect at least one trajectory data of each agent's interaction with the environment based on the agent's exploration in the specified environment, and store the trajectory data corresponding to each agent in the experience replay pool;

[0010] The joint action value generation module is used to execute steps S2-1 to S2-6 for each agent in the specified environment, obtain the individual action value of each agent, generate the joint action value based on the individual action values of all agents using the QMIX network, and then enter the individual action value update module;

[0011] The individual action value update module is used to update the individual action value of each agent through the Q-learning algorithm based on the joint action value.

[0012] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of a large-scale multi-agent collaborative control method based on role allocation.

[0013] The aforementioned large-scale multi-agent collaborative control method, apparatus, and computer device based on role assignment collect at least one trajectory data of each agent's interaction with the environment based on each agent's exploration within a specified environment. The corresponding trajectory data for each agent is stored in an experience replay pool. The individual action value of each agent in the specified environment is obtained. A QMIX network is used to generate a joint action value based on the individual action values of all agents. The joint action value is then updated using a Q-learning algorithm based on the joint action value. A MARL (Multi-Agent Reinforcement Learning) learning method based on neural networks and roles combines self-attention mechanisms, cross-attention mechanisms, and gated recurrent units. Agents are assigned different roles, encouraging them to focus on specific subtasks within the overall task. This approach aims to overcome the challenges of independent agent learning strategies by dynamically assigning roles to agents and limiting the action space for each role category. Furthermore, this application enables agents to focus on their own tasks, achieving better division of labor and collaboration to complete joint tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 1 is a flow chart of a large-scale multi-agent collaborative control method based on role allocation in one embodiment;

[0015] Figure 2 A topological architecture diagram of a large-scale multi-agent collaborative control method based on role allocation in one embodiment;

[0016] Figure 3 is a schematic structural diagram of an information sharing module in one embodiment;

[0017] Figure 4This is a diagram showing the effect of role allocation on the action space restriction of similar agents in a large-scale multi-agent cooperative control method based on role allocation during 100K training rounds in one embodiment;

[0018] Figure 5 This is a diagram showing the effect of role allocation on the action space restriction of similar agents in a large-scale multi-agent cooperative control method based on role allocation during 1 million training rounds in one embodiment;

[0019] Figure 6 This is a diagram showing the effect of role allocation on the action space restriction of similar agents in a large-scale multi-agent collaborative control method based on role allocation during 1.5 million training rounds in one embodiment;

[0020] Figure 7 A graph showing the win rate of a large-scale multi-agent cooperative control method based on role allocation in one embodiment, after training for 2 million rounds on the 2s3z map in the StarCraft SMAC experimental environment;

[0021] Figure 8 A graph showing the win rate of a large-scale multi-agent cooperative control method based on role allocation in one embodiment, after training for 2 million rounds on the 1c3s5z map in the StarCraft SMAC experimental environment;

[0022] Figure 9 A graph showing the win rate of a large-scale multi-agent cooperative control method based on role allocation in one embodiment, after training for 2 million rounds on the 8m_vs_9m map in the StarCraft SMAC experimental environment;

[0023] Figure 10 A schematic diagram of the results of a benchmark test of a large-scale multi-agent collaborative control method based on role allocation in a multi-agent particle environment in one embodiment. DETAILED DESCRIPTION

[0024] Multi-agent reinforcement learning, a cutting-edge research area in machine learning and artificial intelligence, has garnered widespread attention in recent years. A multi-agent system (MARL) is a complex system model in which multiple agents collaborate to complete a shared task and achieve a common goal. As an effective solution, cooperative multi-agent reinforcement learning (MARL) has been widely applied in complex decision-making scenarios, such as game-playing, multi-robot control, and intelligent systems, and has demonstrated strong potential. MARL not only improves the efficiency of inter-agent collaboration but also optimizes their task execution strategies. However, despite recent progress in multi-agent collaboration methods, the development of cooperative MARL still faces two key challenges that significantly limit its further application in practical scenarios. First, scalability: As the number of agents increases, the joint state-action space grows exponentially, leading to a sharp increase in computational complexity, making it difficult to meet the requirements of large-scale multi-agent systems. Second, partial observability: Because agents can only execute policies based on a partial observation history, decentralized policies must remain effective and reliable even when global information is unavailable. To address these two challenges, the "Centralized Training with Decentralized Execution" (CTDE) paradigm was proposed and became mainstream. It utilizes global information to optimize agent policy learning during the training phase, while allowing agents to independently make action decisions based on their local observations during the execution phase, thus addressing the scalability issue to a certain extent. However, centralized training still requires agents to learn policies in a joint state-action space, which often leads to agents tending to learn similar policies, thus affecting the effectiveness of multi-agent collaboration. Especially when there are a large number of agents, homogeneous policy learning can lead to inefficient collaboration and even cause the joint strategy to become suboptimal, making it difficult to meet the requirements of efficient collaboration for complex tasks. Therefore, how to achieve efficient heterogeneous collaboration in large-scale multi-agent systems remains one of the core challenges in current multi-agent reinforcement learning research, and innovative approaches are urgently needed to address it.

[0025] Currently, multi-agent reinforcement learning (RL) has attracted widespread attention and achieved significant progress within the "centralized training and decentralized execution" paradigm. CTDE addresses scalability issues by leveraging global information for training while relying solely on local observations of agents when executing decisions. Within this framework, a large number of multi-agent RL algorithms have been further developed. Researchers decompose the joint action value into a simple sum of the action values of individual agents, allowing each agent to independently execute action decisions while still learning a globally optimal policy during centralized training. However, simple linear decomposition cannot capture the complex nonlinear collaborative relationships between agents, and in some scenarios, this approach cannot guarantee the global optimal policy. Other studies have introduced a hybrid network with monotonicity constraints to nonlinearly combine the local action values of each agent to obtain a global joint action value. This nonlinear hybrid network design can handle more complex collaborative tasks beyond linear combination strategies and allows the method to be extended to more agent environments without being affected by the exponential growth of the joint action space. However, due to the monotonicity constraint, this method cannot express certain complex nonlinear policies. Furthermore, in high-dimensional environments, it requires a large number of samples and takes a long time to train. Some studies have also mapped the joint action-value function to a new, more easily factorizable joint action-value function, while maintaining the same optimal action. This approach ensures a more general joint action-value decomposition, thus covering a wider range of MARL tasks than the aforementioned methods. However, relying on a neural network to approximate the joint action-value function in this approach leads to inaccurate estimates, making policy learning very difficult and resulting in poor performance on actual tasks. Another study theoretically derived the general form of the joint action-value function and the action-value functions of individual agents using a Taylor expansion, and naturally used a multi-head attention mechanism to approximate the parameters in the formula to estimate the joint action value. This made the algorithm's individual decentralized strategy maximization solvable and strengthened the ability to decompose the joint action value. However, in practical applications, this approach restricts the parameters in the attention mechanism to non-negative values, and similarly encounters issues caused by monotonicity constraints, making it impossible to accurately express complex joint strategies. The researchers proposed a multi-agent action value learning method called Duplex Dueling, which uses the Duplex Dueling network structure to decompose the joint action value function. This Duplex Dueling structure transforms the IGM principle into an advantage function constraint that is easier to implement, thereby achieving efficient and robust action value function learning.

[0026] While the aforementioned work has achieved some success in decomposing individual agent action-value functions and optimizing individual agent policies, most approaches primarily focus on the decision-making capabilities of individual agents while neglecting inter-agent collaboration. Complex MARL tasks require multiple agents to undertake different subtasks to achieve a common goal. Therefore, collaboration is crucial for overall task success, a feature that remains insufficient in the aforementioned algorithms. Consequently, subsequent research has widely applied the concept of role-based multi-agent learning to the design of multi-agent systems to enhance inter-agent collaboration and reduce system design complexity. The core of this concept is to decompose the multi-agent collaborative task into a series of subtasks and assign different roles to the agents. These roles then learn policies within different constrained action spaces to solve the subtasks corresponding to their roles. By decomposing the agents' behaviors into role representations, researchers assign each agent a specific role, enabling it to select actions based on its role, and optimizing the role and policy selection during the learning process. This role-based approach effectively reduces the complexity of policy learning, particularly in complex multi-agent collaborative tasks. However, the proper definition and assignment of roles in this approach poses a significant challenge. Role design directly impacts the efficiency of agent collaboration. Furthermore, while dynamic role switching can improve flexibility, frequent role switching can lead to learning instability, necessitating a carefully designed optimization process for the role selection network. A role-based multi-agent reinforcement learning approach has been proposed. This approach uses a role selector to decompose the joint action space into a finite action space, corresponding to different roles. The agent's decision-making strategy is structured in two layers: first, a role is selected, and then a strategy is selected based on the role. This two-layer structure, characterized by role division and layering, addresses the issue of the excessively large joint action state space in multi-agent environments. However, this approach requires manual specification of the number of role categories, and these roles are predefined and fixed throughout the learning process, making it inflexible in adapting to diverse task scenarios. Furthermore, this approach utilizes action encoding with inherent planar clustering properties to assign agent roles. Consequently, its performance is sensitive to the parameters of the clustering algorithm employed, significantly impacting learning outcomes. Other studies have introduced a role allocator in the multi-agent reinforcement learning method, assigning agents to different roles, and using graph convolution modules combined with autoencoders to enhance the collaboration ability within the same role.

[0027] To address these challenges and current situations, this paper proposes a large-scale multi-agent collaborative control method based on neural networks and role assignment. By designing an information sharing module and an agent role assignment module, combined with self-attention mechanisms, cross-attention mechanisms, and gated recurrent units, this approach aims to achieve more efficient and reliable role assignment without human assistance, ensuring that agents collaborate within their respective stable roles and enabling efficient long-term strategy learning.

[0028] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0029] In an exemplary embodiment, Figure 1 As shown, a large-scale multi-agent collaborative control method based on role allocation is provided, which is used to automatically allocate roles to agents, including:

[0030] Step S1: Based on the exploration of each agent in a specified environment, at least one trajectory data of each agent interacting with the environment is collected, and the trajectory data corresponding to each agent is stored in an experience replay pool;

[0031] For example, training data collection; in this step, the agent is first allowed to freely explore the multi-agent environment, where the multi-agent environment is the designated environment; at the same time, records of the entire process of the agent's interaction with the environment are collected, specifically including the sequence of state, action, reward value and next state, and such trajectory data (Trajectory Data) is stored in the experience replay pool (Replay Buffer) of the agent learning until it is full.

[0032] Step S2: Execute steps S2-1 to S2-6 for each agent in the specified environment to obtain the individual action value of each agent. Use the QMIX network to generate the joint action value based on the individual action values of all agents, and then proceed to step S3.

[0033] Exemplarily, the joint action value of all agents is calculated, including repeating steps S2-1 to S2-6 for all agents to obtain the individual action value function of each agent, and using the QMIX network to process all individual action value functions to obtain the joint action value.

[0034] Step S3, updating the individual action value of each agent through the Q-learning algorithm based on the joint action value;

[0035] Exemplarily, the TD (Temporal Difference Error) error is calculated based on the joint action value and parameters are updated. This includes using the Q-learning algorithm, based on the obtained joint action value of all agents, using a bootstrap method to estimate the maximum estimate of the next state using the current action-value function, and using this estimate to update the value of the current action. The QMIX network is updated, and the gradients of each network parameter are calculated using the backpropagation algorithm. The parameters of each network are updated using gradient descent. Each network includes, but is not limited to, pre-trained role encoders containing MLP modules, LSTM modules, and Gaussian distribution modules, parameterized neural networks, and Q networks.

[0036] In an exemplary embodiment, steps S2-1 to S2-6 in step S2 include:

[0037] Step S2-1: extract the target trajectory data corresponding to the target agent from the experience replay pool and align the observation-action value of the target agent at the specified time step;

[0038] Specifically, align the observation-action values of multiple time steps above and below, including extracting a complete trajectory data from the experience replay pool and aligning the upper and lower Agents with time steps The observation-action value is used as the basis for subsequent role assignment of agents and strengthening the collaboration between agents; That is the target agent.

[0039] Step S2-2: Based on the aligned observation-action values of the target agent, a representation of the target agent's own reference state information is generated by combining the self-attention mechanism and the fully connected layer;

[0040] Specifically, generate an agent The coarse state information of the agent, including the aligned agent with multiple time steps The observation-action value of , combined with the self-attention mechanism and the fully connected layer, generates a representation of the intelligent agent The representation of the rough state information of the target agent, that is, the representation of the reference state information of the target agent.

[0041] Step S2-3: Using a cross-attention mechanism, based on the representation of the target agent's own reference state information and the observation-action values of non-target agents in the specified environment, generate personalized information about the target agent from non-target agents in the specified environment, and share the personalized information with the target agent.

[0042] Specifically, generate other agents to the agent The shared information of the agent after step S2-2 is processed using the cross attention mechanism Characterization and removal of Agents other than ) is processed to generate an intelligent agent For intelligent agents Personalized information is formulated and Share for the intelligent agent More reasonable and collaborative role allocation is provided to help; The representation of is the representation of the target agent’s own reference state information, except Agents other than ) refers to the non-target agents other than the target agent in the specified environment.

[0043] Step S2-4, based on a pre-trained role encoder comprising an MLP module, an LSTM module, and a Gaussian distribution module, generates a Gaussian distribution representing the role of the target agent according to the personalized information shared by the non-target agents received by the target agent and the observation-action value of the target agent;

[0044] Step S2-5, using a parameterized neural network to generate weight parameters of a target policy network for controlling the target agent based on a Gaussian distribution representing the role of the target agent;

[0045] In step S2-6, the individual action value of the target agent is output through the Q network based on the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent.

[0046] In an exemplary embodiment, the role encoder comprising an MLP module, an LSTM module, and a Gaussian distribution module pre-trained in step S2-4 includes:

[0047] An MLP module is constructed using the personalized information shared by the target agent and the target agent's own observation-action value aggregate vector as input to extract features and output the aggregate vector feature representation.

[0048] LSTM is used to build an LSTM module with aggregate vector feature representation as input to process time series and generate high-dimensional features of aggregate vectors;

[0049] A linear layer is used to construct a Gaussian distribution module that takes the high-dimensional features of the aggregated vector as input to generate a Gaussian distribution representing the role of the target intelligent agent;

[0050] The role encoder is composed of MLP module, LSTM module and Gaussian distribution module from input to output. The loss function is designed and used to train the role encoder.

[0051] Specifically, the character encoder generates a representative agent Gaussian distribution of roles, where the agent That is the target agent, agent That is, non-target agents other than the target agent in the specified environment, including:

[0052] Agent Get all agents in step S2-3 After sharing the personalized information, the shared personalized information is combined with its own local observation value-action decision through feature concatenation operation as an input vector to complete the aggregation process, and an aggregated vector is obtained, where the observation value-action decision is the observation-action value. A two-layer multi-layer perceptron (MLP) is used to perform preliminary feature extraction on the input vector (i.e., the aggregated vector), and the activation function uses the ReLU function, and the output is the feature representation.

[0053] Furthermore, in order to capture the dynamic information of the agent in multiple time steps, a single-layer long short-term memory network (LSTM) is used to perform time series processing on the feature representation to generate time-related high-dimensional features.

[0054] Furthermore, the high-dimensional features are mapped to the parameters of the character Gaussian distribution, including mean and variance, through a linear layer. The Gaussian distribution is expressed by the formula Indicates that represents the mean of the role encoding, Represents variance.

[0055] Furthermore, during the training process, the parameters of the above-mentioned MLP module, LSTM module and Gaussian distribution module are learned. In the MLP module, the weight matrix and bias parameters need to be learned to learn the high-level representation of the input features, that is, the weight matrix and bias parameters are learned and optimized; in the LSTM module, the weights and bias parameters of the input gate, forget gate, and output gate, as well as the update parameters of the hidden state need to be learned to model the time series features, that is, the weights and bias parameters of the input gate, forget gate, output gate, and the update parameters of the hidden state are learned and optimized; in the Gaussian distribution module, the mean value needs to be learned. and variance Parameters used to describe the distribution of the agent's role implicit characteristics, that is, the mean and variance The parameters of the model are learned and optimized through a trainable neural network. The model parameters are used as the role encoder of the agent, and the output represents the Gaussian distribution of the agent role, that is, the Gaussian distribution of the target agent role.

[0056] In an exemplary embodiment, designing a loss function includes:

[0057] Based on the reconstruction loss function, the KL divergence (Kullback-Leibler Divergence) regularization term is introduced to maximize the mutual information between the observation-action value of the target agent and the Gaussian distribution of the target agent role to obtain the loss function.

[0058] For example, the specific training process is mainly divided into the design of loss function and the process of gradient optimization. The loss function design includes reconstruction loss, regularization constraint and mutual information optimization. First, based on the reconstruction loss function, the error between the agent action value generated by the current policy network parameters and the ideal target action value is measured. Since the role encoding generated by the role encoder will be used to generate the parameters of the agent's policy network, minimizing the reconstruction error can optimize the quality of the role encoding and ensure that the representation generated by the role encoder can effectively guide the learning of the policy network; in addition, in order to avoid the distribution of roles being too concentrated, the KL divergence regularization term is introduced to make the generated role Gaussian distribution cover as large a representation space as possible; in the role encoding process, the mutual information between the observation-action value of the target agent and the Gaussian distribution of the target agent's role is maximized, and at the same time, the mutual information between the observation value of the agent and the role distribution is optimized to ensure the personalization and independence of the role.

[0059] In an exemplary embodiment, step S2-5 includes:

[0060] The Gaussian distribution of the target agent role is sampled to obtain the role representation of the target agent, and the role representation of the target agent is input into a parameterized neural network, where the parameterized neural network includes an MLP and various linear layers;

[0061] The role representation of the target agent is mapped through a parameterized neural network to generate the weight parameters of the target policy network used to control the target agent.

[0062] Specifically, the character decoder generates a representative agent The role strategy weight parameter of the agent The role strategy weight parameters are the weight parameters of the target strategy network used to control the target agent, including:

[0063] The Gaussian distribution of the target agent role obtained in step S2-4 is used as input. A parameterized neural network (called a hypernetwork) is used to generate role-specific policy parameters (weight parameters of the target policy network used to control the target agent). A hypernetwork is a neural network used to generate target network parameters. Its input is the role representation, and its output is the parameters of the individual agent policy network. Here, the target network is the policy network that controls the agent's action decisions. These weight parameters are fed into the parameterized hypernetwork to generate the corresponding role policy weight parameters, which influence the calculation of the value of the agent's individual actions.

[0064] In an exemplary embodiment, mapping the role representation of the target agent through a parameterized neural network to generate weight parameters of a target policy network for controlling the target agent includes:

[0065] The target agent's role representation is transformed through MLP to extract high-dimensional features, and the ReLU activation function is used to improve the nonlinear expression ability.

[0066] Each linear layer is used to output the weight parameters of each policy network corresponding to each linear layer based on the high-dimensional features of the role representation, where each policy network is used to control the corresponding intelligent agent, and the weight parameters of each policy network include the weight parameters of the target policy network used to control the target intelligent agent.

[0067] Specifically, for the parameterization process of the hypernetwork, the hypernetwork receives the role representation obtained by sampling from the role Gaussian distribution As input, the input role representation is first transformed through a set of multi-layer perceptrons (MLPs) to extract high-dimensional features, and the ReLU activation function is used to improve the nonlinear expression ability. Then the last layer of the hypernetwork is composed of multiple linear layers. The output of each linear layer corresponds to different parameters of the policy network, that is, the weight and bias of the policy network. The weights of these linear layers are optimized by the trainable parameters in the hypernetwork. In the specific process of the hypernetwork generating the role policy representation, the hypernetwork transforms the role representation into a Mapped to the weight parameters of the control strategy network , the weight parameters generated here are It will be directly used in the target policy network. Furthermore, the target policy network receives the local observation value of the target agent as input, and then combines it with the above-mentioned , output the value of the agent's individual action after role-based weight processing .

[0068] In an exemplary embodiment, step S2-6 includes:

[0069] The observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent are input into the Q network, which includes an input layer, a GRU module, a fully connected layer, and an output layer;

[0070] Receive the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent through the input layer;

[0071] The GRU module is used to extract time series information from the observation-action value of the target intelligent agent;

[0072] The observation-action value of the target agent processed by the GRU module is extracted through the fully connected layer, and the individual action value of the target agent is generated by combining the weight parameters of the target policy network used to control the target agent.

[0073] The output layer is used to output the individual action value of the target agent.

[0074] Specifically, the agent is calculated by combining the role strategy weight parameters The individual action value of the role strategy weight parameter is the weight parameter of the target strategy network used to control the target agent. The target agent includes:

[0075] The reinforcement learning Q network is used to calculate the action value of the agent. The Q network consists of an input layer, a GRU module, a fully connected layer, and an output layer. The input layer is the input of the agent. The local observations and action decisions (observation-action values), as well as the role strategy weight parameters generated by the super network in step S2-5; the GRU module is the core part of the Q network, which is used to process the local observations of the agent at multiple time steps and extract time series information; the fully connected layer further extracts features based on the GRU module and generates individual action values in combination with the role strategy weight parameters; finally, the output layer outputs the agent The individual action value of each action under the current observation. This design can capture the behavior pattern of the agent in a dynamic environment. The input is the agent The local observations and action decisions, combined with the agent obtained in step S2-5 The corresponding role strategy weight parameters are used as additional inputs to the neural network, and the output is the agent The value of individual actions.

[0076] In an exemplary embodiment, generating a joint action value based on the individual action values of all agents using the QMIX network in step S2 includes:

[0077] The QMIX network is used to process the individual action values of all intelligent agents, and the individual action values of all intelligent agents are mapped to the joint action value through nonlinear combination.

[0078] In a specific embodiment, StarCraft Multi-Agent Challenge (SMAC) is used as the experimental environment. The application first initializes the environment. In this example, each of our combat units is controlled by a separate reinforcement learning agent, and the enemy units are controlled by a computer with a difficulty level of 7. The agent can select actions to perform in a discrete action space, including moving in four basic directions (up, down, left, and right), selecting a local unit to attack, stopping, or taking no action. Each agent makes an action decision based on its local observation value at each time step. If there is a enemy units, then the action space of each agent consists of The objective is to control all friendly units to defeat enemy units within a set event limit and maximize the winning rate of each game. This is to freely explore in a multi-agent reinforcement learning environment.

[0079] In step S1, the application first controls the intelligent agent to randomly execute the action strategy, allowing our unit to freely explore the environment and accumulate game experience in the experience replay pool until it is full.

[0080] Step S2-1: Take a complete trajectory data from the experience replay pool , and extract the corresponding agent from it Trajectory data ,in, Indicates that at time step At this moment, the global observation information set of the entire multi-agent system; Representing an agent At time step The local observation value at time; Indicates that at time step At time t, the set of actions performed by all agents; Representing an agent At time step The action of the moment; Indicates that at time step At this moment, the reward set fed back to the entire system by the environment; Representing an agent At time step Always follow their actions and current status The local reward obtained. Then at the current time step Bottom alignment Agents with time steps The observation-action pair contains the vector representation of the context information as the subsequent agent The basis for assigning roles and executing action decisions; where the agent The observation-action pair is the observation-action value of the target agent.

[0081] Step S2-2, using the self-attention network to analyze the multi-time-step agent The observation-action pairs are processed and the output is the representation of the agent Brief information about the individual's status (That is, the representation of the target agent's own reference state information): ,in .in, is the key vector, through the weight matrix For input observations The feature representation obtained by linear transformation represents the characteristics of each input observation and is used to calculate the correlation with other observations. is the key vector The characteristic dimension of It is a scaling factor that prevents the dot product value of the key vector and the query vector from being too large, which causes the gradient of the softmax function to be too small, making the optimization process difficult; For the query vector, through the weight matrix For input observations The feature representation obtained by linear transformation is For use with key vector Perform a dot product to calculate the correlation score between the observation and other observations; The value vector is the observation value The feature representation generated by linear transformation contains the main information features of the input observation value and is used for subsequent weighted summation; Represents the time step The query vector of . The observation value is contained in the observation-action value.

[0082] Step S2-3, in each agent Agents other than the target agent (non-target agents in the specified environment, denoted as agents ) Received by the agent After briefly sharing information about the individual states of each agent The shared information and its own local observation representation are processed using a cross-attention network to generate each different agent Feedback to the agent Shared context information (i.e., personalized information) : ,in, The parameters and 、 and 、 and 、 and 、 and 、 and 、 and 、 and The concept corresponds to It is the representation of the target agent’s own reference state information.

[0083] Step S2-4, in all agents Complete the agent After the shared personalized information feedback, the agent Aggregate the obtained shared information representation (i.e., shared personalized information) with its own local observation value representation (contained in the observation-action value of the target agent): in, is an aggregate function, are trainable model parameters, For the obtained agent The local observations of all agents Feedback to the agent The intelligent agent after aggregating personalized favorable information (that is, shared personalized information) Information. The role encoder input is aggregated information , through Perform nonlinear mapping to generate intelligent agents The mean of the Gaussian distribution of and standard deviation , to describe the agent role characteristics and their uncertainty.

[0084] Among them, the role encoder is a multi-layer neural network, through the formula and To generate the desired agent The mean of the Gaussian distribution of and standard deviation .in To generate the mean neural network, To generate variance The neural network, by inputting Feature extraction is performed and obtained by non-linear mapping; and Representing a neural network and According to the output of the role encoder, the agent The role representation is represented by the following Gaussian distribution: in Represents a specific role representation sampled from the role distribution, which is used for subsequent strategy generation or task assignment. The training goal of the role encoder is to optimize the generation of role distribution so that it can effectively reflect the local characteristics and global collaboration requirements of the agent. The specific optimization goal is to minimize the loss function: in Represents the loss of the policy network generated based on the role distribution, that is, the temporal difference (TD) loss in policy optimization, which is used to ensure that the generated role representation has a guiding role in the agent's policy learning; represents the KL divergence between the role distribution and the standard normal distribution, where represents the identity matrix, represents the standard normal distribution, with KL divergence as the regularization term to encourage diversity in role distribution and prevent overfitting; is a scaling factor that controls the weight of the regularization term.

[0085] In step S2-5, the present application constructs a role decoder, which aims to generate role-related strategy weight parameters (i.e., the weight parameters of the target strategy network used to control the target agent) through the Gaussian distribution of the agent's role, which is used to control the action decision of the agent. The role decoder is a hyper-network for generating agent strategy parameters. It receives the specific role representation sampled from the role distribution. As input, it maps the generated policy network weight parameters related to the role .

[0086] Specifically, the role decoder first uses a multi-layer perceptron (MLP) to represent the role. Perform feature extraction and generate high-dimensional feature representation. The expression is: The activation function uses the ReLU function to enhance the nonlinear expression ability of the network. The output layer of the hypernetwork is further mapped to generate the weight parameters and bias parameters of the target policy network: in and are the weight matrix and bias vector of the policy network, respectively. The output layer consists of multiple linear layers, each corresponding to a different parameter part of the policy network. During the optimization of the role decoder, the role decoder and the policy network are trained using a joint optimization method. Optimization is performed by minimizing the following loss function: in The target action value is the ideal action value calculated based on the Bellman equation and used to guide the learning of the current Q network. The target action value mainly serves as a supervisory signal to minimize the error between the actual output action value and the target action value. is the KL divergence between the role distribution and the standard normal distribution, which is used to regularize the role Gaussian distribution.

[0087] Step S2-6, combining with intelligent agents Local status information , the role strategy weight parameter obtained in step S2-5 and aggregated information , through a gated recurrent unit (GRU) module, which additionally considers the high-dimensional feature representation obtained in steps S2-5 And output the agent action value: in Represents the set of all the parameters that need to be trained above is the observed value, It is a high-dimensional feature representation.

[0088] In step S2, the agent All agents repeat steps S2-1 to S2-6 and calculate The action value of each agent at the moment is processed by the Mixing Network of the QMIX algorithm to process the action value vector of all agents, which is expressed as: is the individual action value of each agent, where The global environment state information is an additional input, and the individual action value of each agent is mapped to the global action value, that is, the joint action value, through nonlinear combination.

[0089] In step S3, based on the obtained global action value, the temporal difference loss is calculated and the parameters of the QMIX network and other related networks are updated using the loss: in Represents the agent at time step Always follow their actions and the current state of the environment Partial rewards obtained, is the attenuation factor, which is a coefficient that controls the importance of future rewards in current decisions; For the future state of the environment, For the actions corresponding to the future moments, The parameters of the current policy network are periodically The parameters of the target policy network are copied from , and the copies are updated at regular time steps.

[0090] The above steps constitute the core technical solution of the present invention. Through the combination of self-attention mechanism, cross-attention mechanism, gated recurrent unit and reinforcement learning QMIX algorithm, the allocation and robust update of agent roles are realized, and efficient and stable agent strategy learning based on role division is completed in complex joint tasks.

[0091] This application proposes a large-scale multi-agent collaborative control technology based on role assignment. By introducing time-aware and information-sharing mechanisms, it significantly improves the collaborative efficiency of multi-agent systems in complex environments. First, this application innovatively uses the combined information of the agents' local observations and behaviors over multiple time steps as the basis for role assignment, rather than relying solely on current observation data. This time-aware role assignment approach enables agents to better capture and utilize historical information, thereby improving the stability and long-term effectiveness of the collaborative control process. Second, this application designs an information sharing module to further enhance the local information fusion and role assignment between agents. This module optimizes the collaborative process through a two-stage information transfer mechanism: in the first stage, the current agent generates brief contextual information and shares it with other agents. In the second stage, other agents provide customized feedback based on the received contextual information through an attention-based architecture, enabling each agent to obtain richer and more accurate environmental information. This information sharing mechanism effectively overcomes the limitations of local observation and enhances the global collaborative capabilities of the multi-agent system. Finally, by combining time-aware local observation-behavior information with shared context-aware information, this application establishes a more precise and efficient role assignment mechanism. Compared to traditional role assignment methods, this application can automatically implement role assignment without manual intervention, ensure that agents collaborate under stable roles, and achieve high efficiency in long-term strategy learning. In summary, the framework of this application not only improves the accuracy and stability of multi-agent collaborative control by optimizing role assignment and enhancing information sharing between agents, but also provides an efficient and reliable solution for large-scale agent collaboration systems, which can significantly improve the overall performance of the system and its ability to adapt to complex tasks.

[0092] In one embodiment, the topology diagram of the present application is provided as follows: Figure 2As shown in Figure 2, the main modules and their interaction processes in the training phase are described, with an emphasis on the role allocation mechanism of time perception and the role of the information sharing module. The agent freely explores in the multi-agent reinforcement learning environment, and the trajectory data obtained includes And store it in the experience replay pool. Sampling trajectory data, alignment Time step for the agent Observation-action value pairs Combining the self-attention mechanism with the fully connected layer to generate the representation of the target agent's own reference state information Processing self-representation using crisscross attention networks Observation-action value pairs with other agents , generates feedback from each other agent to the agent Shared context information (i.e., personalized information) ; The agent Local observation-action value pairs Feedback to the agent with all other agents Personalized and beneficial information Aggregate the shared personalized information, input the aggregated information into the role encoder to generate the role distribution, and input the role decoder to obtain the policy network weight parameters related to the role. , input GRU layer to get individual action value And additionally consider high-dimensional features becomes and Mixing Network after QMIX algorithm and combined with additional input global state information ,in and Are all weight matrices, get the joint action value A temporal difference loss is calculated and used to update the parameters of the QMIX network and other related networks. Action decisions are then fed back to the multi-agent reinforcement learning environment. In the implementation of this application, local information for each agent is first collected and generated through an information sharing module and aligned across multiple time steps. Specifically, in the first stage, the agent generates brief contextual information based on its current local observations and actions using an attention mechanism. This contextual information provides each agent with a rough description of the environment and behavior, which is then shared with other agents. In the second stage, after receiving this contextual information, other agents, using an attention-based architecture, combine their own observations and strategies to provide detailed feedback to the current agent. This feedback is weighted to ensure the pertinence and effectiveness of the information. The agents adjust their behavior based on the received contextual feedback, thereby completing the optimization process for role assignment. In this way, the framework of this application achieves more accurate and efficient role assignment based on local information and shared contextual information across multiple time steps. After these two optimization stages, the method of this application automatically completes role assignment between agents and ensures that the agents can collaborate and perform tasks under stable roles. Compared with the traditional collaborative method based on single-time-step local observations, this application adopts a time-aware dynamic role allocation mechanism, which enables the agents to better adapt to changing environments and task requirements, thereby significantly improving the collaborative stability and learning efficiency of multi-agent systems in long-term tasks.

[0093] The information sharing module structure of this application is as follows Figure 3 As shown in , it involves two stages of information sharing. In the first stage, the agent generates brief context information based on its local observation data. Specifically, each agent After collecting local environment information and self-behavior information for multiple time steps, these data are compressed into a concise context description. c i This contextual information c i It will then be shared with other agents j and k This process is achieved through information broadcasting, ensuring that all agents receive the same collaborative background information. In the second stage, after receiving the context information, other agents will process the context information and generate feedback through the attention mechanism based on their own local information and current behavior strategy. m ji and m kiAt this point, the agent uses its own attention network to selectively extract key information from the context based on its relevance and importance, and adjusts the current task accordingly. Through this feedback process, other agents provide appropriate suggestions and feedback based on the contextual information passed on, helping the target agent to better optimize its role allocation.

[0094] Figure 4 、 Figure 5 、 Figure 6 Describes the effect of the method of this application on limiting the action space of agents within the same role as the number of learning rounds of the agent increases, where Figure 4 The effect of this method on limiting the action space of agents within the same role when the number of training rounds is 100K. Figure 5 The effect of this application method on limiting the action space of agents within the same role when the number of training rounds is 1 million, Figure 6 The effectiveness of our method in constraining the action space of agents within the same role for 1.5 million training epochs. Our method was additionally tested on the StarCraft SMAC map 1c3s5z, which clearly demonstrates the heterogeneity of the agents. In this scenario, the agent must control one Colossi (three Colossi), three Stalkers (three Stalkers), and five Zealots (five Berserkers) to defeat all enemy units. In the early stages of training (100,000 training epochs) and mid-stage (500,000 to 1.5 million training epochs), all agents' action representations still explore the action space. Through detailed comparison, our method demonstrates its strong ability to constrain the exploration of the action space. In the early stages of training, when role assignments are just beginning to be learned, agents primarily learn their action strategies through random exploration. At this stage, all agents' action representations remain random and disordered. Later in training (2 million training epochs), after repeated learning, the agents were assigned different sub-roles, resulting in Colossi, Stalkers, and Zealots exhibiting significantly different motion tendencies and action patterns. Furthermore, coordination between units became closer, and the agents began to win battles, demonstrating the effectiveness of role-assignment learning for collaborative task allocation.

[0095] Figure 7 The winning rate curve of the proposed method after 2 million rounds of training on the 2s3z map in the StarCraft SMAC experimental environment is shown. Figure 8 The winning rate curve of the proposed method after 2 million rounds of training on the 1c3s5z map in the StarCraft SMAC experimental environment is shown. Figure 9The proposed method's win rate curves, obtained after 2 million rounds of training on the 8m_vs_9m map in the StarCraft SMAC experimental environment, are presented. The proposed method's performance on the three maps is compared with that of the traditional QMIX algorithm. The proposed method outperforms the QMIX algorithm on all three maps, achieving a 34.71% improvement in average win rate and a 6.34% reduction in variance. This dual improvement in win rate and variance demonstrates the proposed method's advantages in learning efficiency and stability. Specifically, the use of multi-time-step contextual information for role assignment reduces the frequent changes in agent roles and action tendencies, thereby facilitating more stable learning.

[0096] exist Figure 10 This paper demonstrates the performance of our approach in two benchmark tests: predator-prey and cooperative navigation in a multi-agent particle environment. Predator-prey is a cooperative multi-agent reinforcement learning task that simulates a test scenario in which agents act as predators and prey interact with each other. In this scenario, there are three predators and one prey. The predator must capture the prey to win, and the capture process depends on the agent's strategy. In the cooperative navigation scenario, the agents must cooperate to move on a plane and cover a predefined goal. The reward is calculated based on the distance between each agent and its assigned goal. Furthermore, in the cooperative navigation scenario, each agent occupies a region in two-dimensional space, and two agents are penalized if they "collide" (i.e., their activity areas overlap). Therefore, each agent must avoid collisions with other agents while moving towards their designated goal. Twenty independent experiments were conducted in each scenario, and the median performance results are presented. Compared with other algorithms, the method of the present application achieved the highest average reward in both predator-prey and cooperative navigation scenarios, and also showed significant advantages in learning efficiency, indicating the effectiveness of the invented method.

[0097] In summary, through a comprehensive comparison and analysis of the efficiency of agent collaboration in a variety of large-scale multi-agent collaborative control tasks, the framework proposed in this application performs significantly better than traditional methods and other reinforcement learning algorithms in multi-agent systems. This application framework combines the self-attention mechanism and the cross-attention mechanism, introduces a time-aware role allocation mechanism and an information sharing module, effectively reducing the inefficient exploration of agents in the action space, thereby improving collaboration efficiency and task completion. Especially in complex environments and long-term collaborative tasks, the application framework significantly improves the collaborative ability of agents and system stability through multi-time-step contextual information and feedback mechanisms. These improvements not only enhance the adaptability of multi-agent systems, but also make the collaboration of agents in dynamic environments more efficient. Therefore, the excellent performance of the application framework in a variety of complex scenarios verifies its feasibility and effectiveness as an efficient and stable multi-agent collaborative control method.

[0098] At least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages. These steps or stages may be executed at different times, or may be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0099] Based on the same inventive concept, the embodiment of the present application also provides a large-scale multi-agent collaborative control device based on role allocation for implementing the large-scale multi-agent collaborative control method based on role allocation involved above. The implementation solution provided by the device is similar to the implementation solution described in the above method, so the specific limitations in the embodiments of one or more large-scale multi-agent collaborative control devices based on role allocation provided below can be found in the above-mentioned limitations on the large-scale multi-agent collaborative control method based on role allocation, and will not be repeated here.

[0100] In an exemplary embodiment, a large-scale multi-agent collaborative control device based on role allocation is provided, comprising:

[0101] A data collection module is used to collect at least one trajectory data of each agent's interaction with the environment based on the agent's exploration in the specified environment, and store the trajectory data corresponding to each agent in the experience replay pool;

[0102] The joint action value generation module is used to execute steps S2-1 to S2-6 for each agent in the specified environment, obtain the individual action value of each agent, generate the joint action value based on the individual action values of all agents using the QMIX network, and then enter the individual action value update module;

[0103] The individual action value update module is used to update the individual action value of each agent through the Q-learning algorithm based on the joint action value.

[0104] Each module in the aforementioned large-scale multi-agent collaborative control device based on role allocation can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0105] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0106] Those skilled in the art will be able to make several modifications and improvements without departing from the concept of the present application, and these modifications and improvements are all within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be based on the appended claims.

Claims

1. A large-scale multi-agent collaborative control method based on role allocation, characterized in that: The method is used to automatically assign roles to intelligent agents, ensuring that the intelligent agents complete collaboration in their respective stable roles, and includes: Step S1: Based on the exploration of each agent in a specified environment, at least one trajectory data of each agent interacting with the environment is collected, and the trajectory data corresponding to each agent is stored in an experience replay pool; Step S2, executing steps S2-1 to S2-6 for each agent in the specified environment, obtaining the individual action value of each agent, generating a joint action value based on the individual action values of all agents using the QMIX network, and then proceeding to step S3; wherein generating the joint action value based on the individual action values of all agents using the QMIX network includes processing the individual action values of all agents using the QMIX network, and mapping the individual action values of all agents to obtain the joint action value through nonlinear combination; Among them, steps S2-1 to S2-6 in step S2 include: Step S2-1: extract the target trajectory data corresponding to the target agent from the experience replay pool and align the observation-action value of the target agent at the specified time step; Step S2-2: Based on the aligned observation-action values of the target agent, a representation of the target agent's own reference state information is generated by combining the self-attention mechanism and the fully connected layer; Step S2-3: Using a cross-attention mechanism, based on the representation of the target agent's own reference state information and the observation-action values of non-target agents in the specified environment, generate personalized information about the target agent from non-target agents in the specified environment, and share the personalized information with the target agent. Step S2-4, based on a pre-trained role encoder comprising an MLP module, an LSTM module, and a Gaussian distribution module, generates a Gaussian distribution representing the role of the target agent according to the personalized information shared by the non-target agents received by the target agent and the observation-action value of the target agent; The role encoder including the MLP module, the LSTM module and the Gaussian distribution module pre-trained in step S2-4 includes: An MLP module is constructed using the personalized information shared by the target agent and the target agent's own observation-action value aggregate vector as input to extract features and output the aggregate vector feature representation. LSTM is used to build an LSTM module with aggregate vector feature representation as input to process time series and generate high-dimensional features of aggregate vectors; A linear layer is used to construct a Gaussian distribution module that takes the high-dimensional features of the aggregated vector as input to generate a Gaussian distribution representing the role of the target intelligent agent; From input to output, the MLP module, LSTM module and Gaussian distribution module constitute the role encoder, and the loss function is designed and used to train the role encoder; Step S2-5, using a parameterized neural network to generate weight parameters of a target policy network for controlling the target agent based on a Gaussian distribution representing the role of the target agent; Step S2-6, outputting the individual action value of the target agent through the Q network based on the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent; Wherein, the step S2-6 includes: The observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent are input into the Q network, which includes an input layer, a GRU module, a fully connected layer, and an output layer; Receive the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent through the input layer; The GRU module is used to extract time series information from the observation-action value of the target intelligent agent; The observation-action value of the target agent processed by the GRU module is extracted through the fully connected layer, and the individual action value of the target agent is generated by combining the weight parameters of the target policy network used to control the target agent. Use the output layer to output the individual action value of the target intelligent agent; Step S3: Update the individual action value of each agent through the Q-learning algorithm based on the joint action value.

2. A large-scale multi-agent collaborative control method based on role allocation according to claim 1, characterized in that: The design loss function includes: On the basis of reconstructing the loss function, the KL divergence regularization term is introduced and the mutual information between the observation-action value of the target agent and the Gaussian distribution of the target agent role is maximized to obtain the loss function.

3. A large-scale multi-agent collaborative control method based on role allocation according to claim 2, characterized in that: The step S2-5 includes: The Gaussian distribution of the target agent role is sampled to obtain the role representation of the target agent, and the role representation of the target agent is input into a parameterized neural network, where the parameterized neural network includes an MLP and various linear layers; The role representation of the target agent is mapped through a parameterized neural network to generate the weight parameters of the target policy network used to control the target agent.

4. A large-scale multi-agent collaborative control method based on role allocation according to claim 3, characterized in that: The process of mapping the role representation of the target agent through a parameterized neural network to generate weight parameters of a target policy network for controlling the target agent includes: The target agent's role representation is transformed through MLP to extract high-dimensional features, and the ReLU activation function is used to improve the nonlinear expression ability. Each linear layer is used to output the weight parameters of each policy network corresponding to each linear layer based on the high-dimensional features of the role representation, where each policy network is used to control the corresponding intelligent agent, and the weight parameters of each policy network include the weight parameters of the target policy network used to control the target intelligent agent.

5. A large-scale multi-agent collaborative control device based on role allocation, characterized in that: The device comprises: A data collection module is used to collect at least one trajectory data of each agent's interaction with the environment based on the agent's exploration in the specified environment, and store the trajectory data corresponding to each agent in the experience replay pool; a joint action value generation module for executing steps S2-1 to S2-6 for each agent in a specified environment, obtaining the individual action value of each agent, generating a joint action value based on the individual action values of all agents using a QMIX network, and then entering an individual action value update module; wherein generating a joint action value based on the individual action values of all agents using a QMIX network includes processing the individual action values of all agents using a QMIX network, and mapping the individual action values of all agents to obtain a joint action value through nonlinear combination; Among them, steps S2-1 to S2-6 include: Step S2-1: extract the target trajectory data corresponding to the target agent from the experience replay pool and align the observation-action value of the target agent at the specified time step; Step S2-2: Based on the aligned observation-action values of the target agent, a representation of the target agent's own reference state information is generated by combining the self-attention mechanism and the fully connected layer; Step S2-3: Using a cross-attention mechanism, based on the representation of the target agent's own reference state information and the observation-action values of non-target agents in the specified environment, generate personalized information about the target agent from non-target agents in the specified environment, and share the personalized information with the target agent. Step S2-4, based on a pre-trained role encoder comprising an MLP module, an LSTM module, and a Gaussian distribution module, generates a Gaussian distribution representing the role of the target agent according to the personalized information shared by the non-target agents received by the target agent and the observation-action value of the target agent; The role encoder including the MLP module, LSTM module and Gaussian distribution module pre-trained in step S2-4 includes: An MLP module is constructed using the personalized information shared by the target agent and the target agent's own observation-action value aggregate vector as input to extract features and output the aggregate vector feature representation. LSTM is used to build an LSTM module with aggregate vector feature representation as input to process time series and generate high-dimensional features of aggregate vectors; A linear layer is used to construct a Gaussian distribution module that takes the high-dimensional features of the aggregated vector as input to generate a Gaussian distribution representing the role of the target intelligent agent; From input to output, the MLP module, LSTM module and Gaussian distribution module constitute the role encoder, and the loss function is designed and used to train the role encoder; Step S2-5, using a parameterized neural network to generate weight parameters of a target policy network for controlling the target agent based on a Gaussian distribution representing the role of the target agent; Step S2-6, outputting the individual action value of the target agent through the Q network based on the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent; Wherein, the step S2-6 includes: The observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent are input into the Q network, which includes an input layer, a GRU module, a fully connected layer, and an output layer; Receive the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent through the input layer; The GRU module is used to extract time series information from the observation-action value of the target intelligent agent; The observation-action value of the target agent processed by the GRU module is extracted through the fully connected layer, and the individual action value of the target agent is generated by combining the weight parameters of the target policy network used to control the target agent. Use the output layer to output the individual action value of the target intelligent agent; The individual action value update module is used to update the individual action value of each agent through the Q-learning algorithm based on the joint action value.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

Images