Large-scale multi-agent cooperative control method based on role allocation

Abstract

The invention relates to a large-scale multi-agent cooperative control method and device based on role allocation and computer equipment. The method comprises the following steps: exploring in a specified environment based on each agent, collecting at least one piece of track data of interaction between each agent and the environment, and storing the track data corresponding to each agent into an experience playback pool; the method comprises the following steps: respectively executing steps S2-1 to S2-6 for each intelligent agent in a specified environment to obtain an individual action value of each intelligent agent, and generating a joint action value by using a QMIX network based on the individual action values of all the intelligent agents; and on the basis of the joint action value, the individual action value of each agent is updated through a Q-learning algorithm. By adopting the method, the precision and the stability of multi-agent cooperative control can be improved, an efficient and reliable solution is provided for a large-scale agent cooperative system, and the overall performance and the capability of adapting to complex tasks of the system can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a large-scale multi-agent collaborative control method, device and computer equipment based on role allocation. Background Art

[0002] When dealing with complex tasks that require multiple agents to collaborate, multi-agent reinforcement learning usually requires agents to undertake different subtasks to achieve a common goal, where the agents share the same reward structure and a common strategy exploration space. As the complexity of the joint task and strategy space increases, it is often difficult for agents to find the optimal joint strategy. Summary of the invention

[0003] Based on this, it is necessary to provide a large-scale multi-agent collaborative control method, device and computer equipment based on role allocation, which can overcome the challenges of independent learning strategies of intelligent agents, enable intelligent agents to focus on their own tasks, and achieve better division of labor and cooperation to complete joint tasks.

[0004] In a first aspect, the present application provides a large-scale multi-agent collaborative control method based on role allocation, which is used to realize automatic role allocation for agents, including: Step S1, based on the exploration of each agent in a specified environment, collect at least one trajectory data of each agent interacting with the environment, and store the trajectory data corresponding to each agent into an experience replay pool; Step S2, executing steps S2-1 to S2-6 for each agent in the specified environment, obtaining the individual action value of each agent, generating the joint action value based on the individual action values ​​of all agents using the QMIX network, and then entering step S3; Step S3, based on the joint action value, the individual action value of each agent is updated through the Q-learning algorithm.

[0005] In a second aspect, the present application also provides a large-scale multi-agent collaborative control device based on role allocation, comprising: A data collection module is used to collect at least one trajectory data of each agent's interaction with the environment based on the exploration of each agent in the specified environment, and store the trajectory data corresponding to each agent in the experience replay pool; A joint action value generation module is used to execute steps S2-1 to S2-6 for each agent in the specified environment, obtain the individual action value of each agent, generate a joint action value based on the individual action values ​​of all agents using the QMIX network, and then enter the individual action value update module; The individual action value update module is used to update the individual action value of each agent through the Q-learning algorithm based on the joint action value.

[0006] In a third aspect, the present application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a large-scale multi-agent collaborative control method based on role allocation when executing the computer program.

[0007] The above-mentioned large-scale multi-agent collaborative control method, device and computer equipment based on role allocation, through exploration of each agent in a specified environment, collects at least one trajectory data of each agent interacting with the environment, and stores the trajectory data corresponding to each agent in the experience replay pool; for each agent in the specified environment, the individual action value of each agent is obtained, and the joint action value is generated using the QMIX network based on the individual action value of all agents, and then the individual action value of each agent is updated by the Q-learning algorithm based on the joint action value. The MARL (Multi-Agent Reinforcement Learning) learning method based on neural networks and roles combines self-attention mechanism, cross-attention mechanism and gated recurrent unit. Among them, agents are assigned different roles to encourage them to focus on specific subtasks in the overall task. It is intended to overcome the challenges of independent learning strategies of agents by dynamically assigning roles to agents and limiting the action space of each role category. In addition, the present application also enables agents to focus on their own tasks and achieve better division of labor and cooperation to complete joint tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A schematic flow chart of a large-scale multi-agent collaborative control method based on role allocation in one embodiment; Figure 2 A topological architecture diagram of a large-scale multi-agent collaborative control method based on role allocation in one embodiment; Figure 3 is a schematic diagram of the structure of an information sharing module in one embodiment; Figure 4 This is a diagram showing the effect of role allocation on the action space restriction of similar agents in a large-scale multi-agent cooperative control method based on role allocation when the number of training rounds is 100K in one embodiment; Figure 5 This is a diagram showing the effect of role allocation on the action space restriction of similar agents in a large-scale multi-agent cooperative control method based on role allocation when the number of training rounds is 1 million in one embodiment; Figure 6This is a diagram showing the effect of role allocation on the action space restriction of similar agents in a large-scale multi-agent cooperative control method based on role allocation when the number of training rounds is 1.5 million in one embodiment; Figure 7 A winning rate curve diagram of a large-scale multi-agent cooperative control method based on role allocation in one embodiment, which is trained for 2 million rounds on a 2s3z map in a StarCraft SMAC experimental environment; Figure 8 A winning rate curve diagram of a large-scale multi-agent cooperative control method based on role allocation in one embodiment after training 2 million rounds on the 1c3s5z map in the StarCraft SMAC experimental environment; Fig. 9 A winning rate curve diagram of a large-scale multi-agent cooperative control method based on role allocation in one embodiment, which is trained for 2 million rounds on the 8m_vs_9m map in the StarCraft SMAC experimental environment; Fig.10 A schematic diagram of the results of a benchmark test of a large-scale multi-agent collaborative control method based on role allocation in a multi-agent particle environment in one embodiment. DETAILED DESCRIPTION

[0009] As a cutting-edge research direction in the field of machine learning and artificial intelligence, multi-agent reinforcement learning has received widespread attention in recent years. Multi-agent systems are complex system models that achieve common goals by working together to complete shared tasks. As an effective solution, cooperative multi-agent reinforcement learning (Cooperative MARL) has been widely used in complex decision-making scenarios such as game confrontation, multi-robot control, and intelligent systems, and has shown great potential. Multi-agent reinforcement learning can not only improve the efficiency of collaboration between agents, but also optimize their task execution strategies. However, although recent research has made some progress in multi-agent collaboration methods, the development of cooperative multi-agent reinforcement learning still faces two key challenges, which greatly limits its further application in practical scenarios. First, the scalability problem: As the number of agents increases, the joint state-action space grows exponentially, resulting in a sharp increase in computational complexity, which is difficult to meet the needs of large-scale multi-agent systems; second, the partial observability problem: Since agents can only execute strategies based on local observation history, this requires decentralized strategies to ensure their effectiveness and reliability when global information cannot be obtained. In order to address these two challenges, the paradigm of "Centralized Training with Decentralized Execution" (CTDE) was proposed and became mainstream. It uses global information to optimize agent strategy learning in the training phase, and allows agents to independently make action decisions based on their local observations in the execution phase, thus solving the scalability problem to a certain extent. However, centralized training still requires agents to learn strategies in the joint state-action space, which usually causes agents to tend to learn similar strategies, which in turn affects the collaboration effect between multiple agents. Especially when there are a large number of agents, homogeneous strategy learning will cause low collaboration efficiency and even cause the joint strategy to fall into suboptimal, making it difficult to meet the needs of complex tasks for efficient collaboration. Therefore, how to achieve heterogeneous and efficient collaboration in large-scale multi-agent systems is still one of the core problems in current multi-agent reinforcement learning research, and it is urgently needed to be solved through innovative methods.

[0010] At present, multi-agent reinforcement learning has received widespread attention under the paradigm of "centralized training and decentralized execution" and has made significant progress. CTDE is committed to solving the scalability problem by using global information for training and relying only on local observations of agents when making decisions. Under this framework, a large number of multi-agent reinforcement learning algorithms have been further developed. By decomposing the joint action value into a simple sum of the action values ​​of individual agents, the researchers enable each agent to independently execute action decisions, while still being able to learn the global optimal strategy during the centralized training phase. However, the simple linear decomposition method cannot capture the nonlinear collaborative relationship between complex agents, and in some scenarios this method cannot guarantee the global optimal strategy. There are also studies that introduce a hybrid network and add monotonicity constraints to nonlinearly combine the local action values ​​of each agent to obtain the global joint action value. The design of the nonlinear hybrid network can handle more complex collaborative tasks rather than being limited to linear combination strategies, and enables the method to be extended to more agent environments without being affected by the exponential growth of the joint action space. However, due to the monotonicity constraint, this method cannot express some complex nonlinear strategies, and in high-dimensional environments, this method has a large demand for samples and requires a long training time. There are also studies that map the joint action value function to a new joint action value function that is easier to factorize and have the same optimal action. This method ensures a more general joint action value decomposition, so it can cover a wider range of MARL task categories than the above methods. However, relying on neural networks to approximate the joint action value function in this method will lead to inaccurate valuations, which makes policy learning very difficult and the actual performance on the task is not good. Other studies theoretically derive the general form of the joint action value function and the action value function of a single agent in the form of Taylor expansion, and naturally use the parameters in the multi-head attention mechanism approximation formula to estimate the joint action value, so that the maximization of the individual decentralized strategy of the algorithm is solvable, and the decomposition ability of the joint action value is strengthened. However, in the actual application of this method, the parameters in the attention mechanism are restricted to non-negative values, and problems caused by monotonicity constraints will also occur, making it impossible to accurately express complex joint strategies. The researchers proposed a multi-agent action value learning method called Duplex Dueling, which uses a Duplex Dueling network structure to decompose the joint action value function. This Duplex Dueling structure transforms the IGM principle into an advantage function constraint that is easier to implement, thereby achieving efficient and robust action value function learning.

[0011] Although the above work has achieved certain results in decomposing the action-value function of a single agent and optimizing the strategy of a single agent, most methods mainly focus on the decision-making ability of a single agent, while ignoring the collaboration between agents. In complex MARL tasks, multiple agents are required to undertake different subtasks to achieve common goals, so the ability to collaborate is crucial to the success of the overall task, which is still insufficient in the above algorithms. To this end, in subsequent research, the concept of role-based multi-agent learning has been widely used in the design of multi-agent systems to enhance the collaboration between agents and reduce the complexity of system design. The core of this concept is to decompose the multi-agent cooperation task into a series of subtasks, and assign different roles to the multi-agents. Different roles learn strategies in different restricted action spaces to solve different subtasks corresponding to their roles. By decomposing the behavior of the agent into the performance of the role, the researchers assign a specific role to each agent, make it choose actions according to the role, and optimize the choice of role and strategy during the learning process. Through this role-oriented approach, this method can effectively reduce the complexity of strategy learning, especially in complex multi-agent cooperation tasks. However, how to reasonably define and allocate roles in this method is an important issue. The design of roles directly affects the collaborative efficiency of agents. In addition, although dynamic role switching can improve flexibility, frequent role switching may lead to instability in the learning process, and the optimization process of the role selection network needs to be carefully designed. A role-based multi-agent reinforcement learning method has been proposed. A role selector is used to decompose the joint action space into a limited action space corresponding to different roles. When making decisions, the agent's strategy is divided into two layers. First, the role is selected, and then the strategy is selected based on the role. Through the two-layer structure of role division and layering, this method solves the problem of too large joint action state space in a multi-agent environment. However, this method requires the number of role categories to be manually specified, and these roles are predefined and fixed throughout the learning process, which is not flexible enough in adapting to different task scenarios. In addition, this method uses action encoding with intrinsic planar clustering characteristics to realize the allocation of agent roles. Therefore, the performance of the method is sensitive to the parameters of the clustering algorithm adopted, which has a great impact on the learning effect. Other studies have introduced a role allocator in the multi-agent reinforcement learning method, assigning agents to different roles, and using graph convolution modules combined with autoencoders to enhance the collaboration ability within the same role.

[0012] In view of the above problems and current situation, this paper proposes a large-scale multi-agent collaborative control method based on neural network and role allocation. By designing information sharing module and agent role allocation module, combined with self-attention mechanism, cross-attention mechanism and gated recurrent unit, it aims to achieve more effective and reliable role allocation without human assistance, ensuring that agents complete collaboration in their respective stable roles and conduct efficient long-term series strategy learning.

[0013] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0014] In an exemplary embodiment, Figure 1 As shown, a large-scale multi-agent collaborative control method based on role allocation is provided, which is used to realize automatic role allocation for agents, and includes: Step S1, based on the exploration of each agent in a specified environment, collect at least one trajectory data of each agent interacting with the environment, and store the trajectory data corresponding to each agent into an experience replay pool; Exemplarily, training data collection; in this step, first allow the agent to freely explore in the multi-agent environment, where the multi-agent environment is the specified environment; at the same time, collect records of the entire process of the agent's interaction with the environment, including the sequence of states, actions, reward values, and the next state, and store such trajectory data (Trajectory Data) in the experience replay pool (Replay Buffer) of the agent's learning until it is full.

[0015] Step S2, executing steps S2-1 to S2-6 for each agent in the specified environment, obtaining the individual action value of each agent, generating the joint action value based on the individual action values ​​of all agents using the QMIX network, and then entering step S3; Exemplarily, the joint action value of all agents is calculated, including repeating steps S2-1 to S2-6 for all agents to obtain the individual action value function of each agent, and using the QMIX network to process all individual action value functions to obtain the joint action value.

[0016] Step S3, updating the individual action value of each agent through the Q-learning algorithm based on the joint action value; Exemplarily, the TD (temporal difference error) error is calculated based on the joint action value and the parameters are updated, including the joint action value of all agents obtained, using the Q-learning algorithm, using the bootstrap method, that is, using the current action value function to estimate the maximum estimate of the next state, and using the estimated value to update the value of the current action, updating the QMIX network and calculating the gradient of each network parameter through the back propagation algorithm, and using the gradient descent to update the parameters of each network. Each network includes but is not limited to a pre-trained role encoder including an MLP module, an LSTM module and a Gaussian distribution module, a parameterized neural network, a Q network, etc.

[0017] In an exemplary embodiment, steps S2-1 to S2-6 in step S2 include: Step S2-1, extracting the target trajectory data corresponding to the target agent from the experience replay pool, and aligning the observation-action value of the target agent at the specified time step; Specifically, align the observation-action values ​​of multiple time steps above and below, including extracting a complete trajectory data from the experience replay pool and aligning the upper and lower Agents with time steps The observation-action value is used as the basis for subsequent role assignment of agents and strengthening the collaboration between agents. That is the target agent.

[0018] Step S2-2, based on the aligned observation-action value of the target agent, a representation of the target agent's own reference state information is generated by combining the self-attention mechanism and the fully connected layer; Specifically, generate an agent coarse state information of the agent, including the use of an aligned agent with multiple time-step scales The observation-action value of , combined with the self-attention mechanism and the fully connected layer, generates a representation of the intelligent agent The representation of the target agent’s own rough state information, that is, the representation of the target agent’s own reference state information.

[0019] Step S2-3, using a cross-attention mechanism, based on the representation of the target agent's own reference state information and the observation-action values ​​of the non-target agents other than the target agent in the specified environment, generates personalized information of the target agent from the non-target agents other than the target agent in the specified environment, and shares the personalized information with the target agent; Specifically, generate other agents to agents The shared information of the agent after step S2-2 is processed using the cross attention mechanism Characterization and removal Agents other than ) to process the observation-action value and generate an intelligent agent For intelligent agents Personalized information is formulated and sent to the agent Share for the intelligent agent The intelligent agent The representation of is the representation of the target agent’s own reference state information, except Agents other than ) refers to the non-target agents other than the target agent in the specified environment.

[0020] Step S2-4, based on a pre-trained role encoder including an MLP module, an LSTM module, and a Gaussian distribution module, a Gaussian distribution representing the role of the target agent is generated according to the personalized information shared by the non-target agents and the observation-action value of the target agent received by the target agent; Step S2-5, using a parameterized neural network to generate weight parameters of a target policy network for controlling the target agent according to a Gaussian distribution representing the role of the target agent; Step S2-6, through the Q network, based on the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent, output the individual action value of the target agent.

[0021] In an exemplary embodiment, the pre-trained role encoder including the MLP module, the LSTM module and the Gaussian distribution module in step S2-4 includes: An MLP module is constructed using MLP, which takes the personalized information shared by the target agent and the aggregate vector of the target agent's own observation-action value as input, and is used to extract features and output the aggregate vector feature representation; LSTM is used to construct an LSTM module with aggregate vector feature representation as input, which is used to process time series and generate aggregate vector high-dimensional features; A linear layer is used to construct a Gaussian distribution module with high-dimensional features of the aggregated vector as input to generate a Gaussian distribution representing the role of the target intelligent agent; The MLP module, LSTM module and Gaussian distribution module in the input-output direction constitute the role encoder. The loss function is designed and used to train the role encoder.

[0022] Specifically, the role encoder generates a representative agent The Gaussian distribution of roles, where the agent is the target agent, agent That is, non-target agents other than the target agent in the specified environment, including: Agent Get all agents in step S2-3 After sharing the personalized information, the shared personalized information is combined with its own local observation value-action decision through feature concatenation operation as an input vector to complete the aggregation processing and obtain an aggregated vector, where the observation value-action decision is the observation-action value, and a two-layer multi-layer perceptron (MLP) is used to perform preliminary feature extraction on the input vector (i.e., the aggregated vector), the activation function uses the ReLU function, and the output is the feature representation.

[0023] Furthermore, in order to capture the dynamic information of the agent in multiple time steps, a single-layer long short-term memory network (LSTM) is used to perform time series processing on the feature representation to generate time-related high-dimensional features.

[0024] Furthermore, the high-dimensional features are mapped to the parameters of the character Gaussian distribution, including mean and variance, through a linear layer. The Gaussian distribution is expressed by the formula Indicates that represents the mean of the role encoding, Represents variance.

[0025] Furthermore, the parameters of the above-mentioned MLP module, LSTM module and Gaussian distribution module are learned during the training process. In the MLP module, the weight matrix and bias parameters need to be learned to learn the high-level representation of the input features, that is, the weight matrix and bias parameters are learned and optimized; in the LSTM module, the weight and bias parameters of the input gate, forget gate, output gate, and the update parameters of the hidden state need to be learned to model the time series features, that is, the weight and bias parameters of the input gate, forget gate, output gate, and the update parameters of the hidden state are learned and optimized; in the Gaussian distribution module, the mean value needs to be learned. and variance , which is used to describe the distribution of the agent's role implicit characteristics, that is, the mean and variance The parameters of the model are learned and optimized through a trainable neural network, which serves as the role encoder of the agent and outputs the Gaussian distribution representing the role of the agent, that is, the Gaussian distribution representing the role of the target agent.

[0026] In an exemplary embodiment, designing a loss function includes: On the basis of reconstructing the loss function, the KL divergence (Kullback-Leibler Divergence) regularization term is introduced to maximize the mutual information between the observation-action value of the target agent and the Gaussian distribution of the target agent role to obtain the loss function.

[0027] Exemplarily, the specific training process is mainly divided into the design of loss function and the process of gradient optimization. The loss function design includes reconstruction loss, regularization constraint and mutual information optimization. First, based on the reconstruction loss function, the error between the agent action value generated by the current policy network parameters and the ideal target action value is measured. Since the role encoding generated by the role encoder will be used to generate the parameters of the agent's policy network, minimizing the reconstruction error can optimize the quality of the role encoding and ensure that the representation generated by the role encoder can effectively guide the learning of the policy network; in addition, in order to avoid the distribution of roles being too concentrated, the KL divergence regularization term is introduced to make the generated role Gaussian distribution cover as large a representation space as possible; in the role encoding process, the mutual information between the observation-action value of the target agent and the Gaussian distribution of the target agent role is maximized, and the mutual information between the observation value of the agent and the role distribution is optimized to ensure the personalization and independence of the role.

[0028] In an exemplary embodiment, step S2-5 includes: The Gaussian distribution of the target agent role is sampled to obtain the role representation of the target agent, and the role representation of the target agent is input into a parameterized neural network, wherein the parameterized neural network includes an MLP and various linear layers; The role representation of the target agent is mapped through a parameterized neural network to generate weight parameters of the target policy network used to control the target agent.

[0029] Specifically, the character decoder generates a representative agent The role strategy weight parameter of the agent The role strategy weight parameters are the weight parameters of the target strategy network used to control the target agent, including: The Gaussian distribution of the target agent role obtained from step S2-4 is used as input, and a parameterized neural network (called a hypernetwork) for generating role strategy representation is used to map and generate role-related strategy parameters (weight parameters of the target strategy network used to control the target agent). A hypernetwork is a neural network used to generate target network parameters. Its input is the role representation, and its output is the parameters of the agent individual strategy network. Here, the target network is the strategy network that controls the action decision of the agent. It is sent to the parameterized hypernetwork to generate the corresponding role strategy weight parameters, which will affect the calculation of the value of the agent's individual action.

[0030] In an exemplary embodiment, mapping the role representation of the target agent through a parameterized neural network to generate weight parameters of a target policy network for controlling the target agent includes: The target agent’s role representation is transformed through MLP to extract high-dimensional features of the target agent’s role representation, and the ReLU activation function is used to improve the nonlinear expression ability. Each linear layer is used to output the weight parameters of each policy network corresponding to each linear layer based on the high-dimensional features of the role representation, wherein each policy network is used to control the corresponding intelligent agent, and the weight parameters of each policy network include the weight parameters of the target policy network used to control the target intelligent agent.

[0031] Specifically, for the parameterization process of the hypernetwork, the hypernetwork receives the role representation obtained by sampling from the role Gaussian distribution As input, the input role representation is first transformed through a set of multi-layer perceptrons (MLPs) to extract high-dimensional features, and the ReLU activation function is used to improve the nonlinear expression ability. Then the last layer of the hypernetwork is composed of multiple linear layers. The output of each linear layer corresponds to different parameters of the policy network, that is, the weight and bias of the policy network. The weights of these linear layers are optimized through the trainable parameters in the hypernetwork. In the specific process of the hypernetwork generating role policy representation, the hypernetwork transforms the role representation Mapped to the weight parameters of the control strategy network , the weight parameters generated here The target policy network receives the local observations of the target agent as input, and then combines the local observations generated by the hypernetwork , output the value of the agent's individual action after role-based weight processing .

[0032] In an exemplary embodiment, step S2-6 includes: Input the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent into the Q network, which includes an input layer, a GRU module, a fully connected layer, and an output layer; Receive the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent through the input layer; The GRU module is used to extract time series information from the observation-action value of the target intelligent agent; The observation-action value of the target agent processed by the GRU module is feature extracted through the fully connected layer, and the individual action value of the target agent is generated by combining the weight parameters of the target policy network used to control the target agent; The output layer is used to output the individual action value of the target agent.

[0033] Specifically, the agent is calculated by combining the role strategy weight parameters The individual action value of the role strategy weight parameter is the weight parameter of the target strategy network used to control the target agent. That is the target agent, including: The reinforcement learning Q network is used to calculate the action value of the agent. The Q network consists of an input layer, a GRU module, a fully connected layer, and an output layer. The input layer is the input of the agent. The local observations and action decisions (observation-action values), as well as the role strategy weight parameters generated by the super network in step S2-5; the GRU module, as the core part of the Q network, is used to process the local observations of the agent at multiple time steps and extract time series information; the fully connected layer further extracts features based on the GRU module and generates individual action values ​​in combination with the role strategy weight parameters; finally, the output layer outputs the agent The individual action value of each action under the current observation. This design can capture the behavior pattern of the agent in a dynamic environment. The input is the agent The local observations and action decisions of the agent are combined with the agent obtained in step S2-5 The corresponding role strategy weight parameters are used as additional inputs to the neural network, and the output is the agent The value of individual actions.

[0034] In an exemplary embodiment, generating a joint action value using a QMIX network based on the individual action values ​​of all agents in step S2 includes: The QMIX network is used to process the individual action values ​​of all intelligent agents, and the individual action values ​​of all intelligent agents are mapped to obtain the joint action value through nonlinear combination.

[0035] In a specific embodiment, StarCraft Multi-Agent Challenge (SMAC) is used as the experimental environment. The application first initializes the environment. In this example, each combat unit of our side is controlled by a separate reinforcement learning agent, and the enemy unit is controlled by a computer with a difficulty level of 7. The agent can select actions to perform in a discrete action space, including moving in four basic directions (up, down, left, and right), selecting a local unit to attack, stopping, or taking no action. Each agent makes an action decision based on its local observations at each time step. If there is a enemy units, then the action space of each agent consists of The task goal is to control all our units to defeat the enemy units within the set event limit and maximize the winning rate of each game. That is, free exploration in the multi-agent reinforcement learning environment.

[0036] Step S1, this application first controls the intelligent agent to randomly execute the action strategy, so that our unit can freely explore the environment and accumulate game experience into the experience replay pool until it is full.

[0037] Step S2-1, take out a complete trajectory data from the experience replay pool , and extract the corresponding agent from it Trajectory data ,in, Indicates that at time step At this moment, the global observation information set of the entire multi-agent system; Representing an Agent At time step The local observation value at time; Indicates that at time step At time t, the set of actions performed by all agents; Representing an Agent At time step Momentary action; Indicates that at time step At this moment, the reward set fed back to the entire system by the environment; Representing an Agent At time step According to its action at all times and current status The local reward obtained. Then at the current time step Align bottom Agents with time steps The observation-action pair contains the vector representation of the context information as the subsequent agent The basis for assigning roles and executing action decisions; where the agent The observation-action pair is the observation-action value of the target agent.

[0038] Step S2-2, using the self-attention network to The observation-action pairs are processed and the output is the representation of the agent Brief information about the individual's status (That is, the representation of the target agent's own reference state information): ,in .in, is the key vector, through the weight matrix For input observations The feature representation obtained by linear transformation represents the characteristics of each input observation and is used to calculate the correlation with other observations. is the key vector The characteristic dimension of It is a scaling factor that prevents the dot product value of the key vector and the query vector from being too large, which causes the gradient of the softmax function to be too small, making the optimization process difficult; For the query vector, through the weight matrix For input observations The feature representation obtained by linear transformation is For use with key vector Perform a dot product to calculate the correlation score between this observation and other observations; The value vector is the observed value The feature representation generated by linear transformation contains the main information features of the input observations and is used for subsequent weighted summation; Represents the time step The query vector of . The observation value is contained in the observation-action value.

[0039] Step S2-3, in each except the intelligent agent Agents other than the target agent (non-target agents in the specified environment except the target agent, denoted as agents ) Received by the agent After briefly sharing information about the individual states of each agent The shared information and its own local observation representation are processed using a cross-attention network to generate each different agent Feedback to the Agent Shared context information (i.e. personalized information) : ,in, The parameters and , and , and , and , and , and , and , and The concept corresponds to It is the representation of the target agent’s own reference state information.

[0040] Step S2-4, in all agents Complete the agent After the shared personalized information feedback, the agent Aggregate the obtained shared information representation (i.e., shared personalized information) with its own local observation value (contained in the observation-action value of the target agent): in, is an aggregate function, are trainable model parameters, To obtain the agent The local observations of all agents Feedback to the Agent The intelligent agent after aggregating personalized favorable information (i.e., shared personalized information) Information. The input to the role encoder is aggregate information , through Perform nonlinear mapping to generate intelligent agents The mean of the Gaussian distribution of and standard deviation , to describe the agent role characteristics and their uncertainty.

[0041] Among them, the role encoder is a multi-layer neural network, through the formula and To generate the desired agent The mean of the Gaussian distribution of and standard deviation .in To generate the mean The neural network, To generate the variance The neural network, by inputting Perform feature extraction and obtain it through nonlinear mapping; and Representing a neural network and According to the output of the role encoder, the agent The role representation is represented by the following Gaussian distribution: in Represents a specific role representation sampled from the role distribution, which is used for subsequent strategy generation or task assignment. The training goal of the role encoder is to optimize the generation of role distribution so that it can effectively reflect the local characteristics and global collaboration requirements of the intelligent agent. The specific optimization goal is to minimize the loss function: in represents the loss of the policy network generated based on the role distribution, i.e., the temporal difference (TD) loss in policy optimization, which is used to ensure that the generated role representation has a guiding role in the policy learning of the agent; represents the KL divergence between the role distribution and the standard normal distribution, where represents the identity matrix, represents the standard normal distribution, with KL divergence as the regularization term, which is used to encourage diversity in role distribution and prevent overfitting; is a scaling factor that controls the weight of the regularization term.

[0042] Step S2-5, the present application constructs a role decoder, which aims to generate role-related strategy weight parameters (i.e., weight parameters of the target strategy network used to control the target agent) through the Gaussian distribution of the agent's role, which is used to control the action decision of the agent. The role decoder is a hyper-network for generating agent strategy parameters, which receives specific role representations sampled from the role distribution. As input, it maps the generated policy network weight parameters associated with the role .

[0043] Specifically, the role decoder first uses a multi-layer perceptron (MLP) to represent the role. Perform feature extraction and generate high-dimensional feature representation. The expression is: The activation function uses the ReLU function to enhance the nonlinear expression ability of the network. Furthermore, high-dimensional feature representation The output layer of the hypernetwork is further mapped to generate the weight parameters and bias parameters of the target policy network: in and are the weight matrix and bias vector of the policy network respectively. The output layer consists of multiple linear layers, each of which corresponds to a different parameter part of the policy network. In the optimization process of the role decoder, the training process of the role decoder and the policy network adopts a joint optimization method. The optimization is performed by minimizing the following loss function: in is the target action value, which is the ideal action value calculated based on the Bellman equation and is used to guide the learning of the current Q network. The main function of the target action value is to serve as a supervisory signal to minimize the error between the actual output action value and the target action value; is the KL divergence between the role distribution and the standard normal distribution, which is used to regularize the role Gaussian distribution.

[0044] Step S2-6, combining agents Local status information , the role strategy weight parameter obtained in step S2-5 Aggregate information , through a gated recurrent unit (GRU) module, which additionally considers the high-dimensional feature representation obtained in step S2-5 And output the agent action value: in Represents the set of all the parameters that need to be trained above is the observed value, It is a high-dimensional feature representation.

[0045] In step S2, the agent All agents repeat steps S2-1 to S2-6 and calculate The action value of each agent at the moment is processed by the Mixing Network of the QMIX algorithm to process the action value vector of all agents, which is expressed as: is the individual action value of each agent, where The global environment state information is an additional input. The individual action values ​​of each agent are mapped through nonlinear combination to obtain the global action value, that is, the joint action value.

[0046] Step S3, based on the obtained global action value, calculate the temporal difference loss, and use the loss to update the parameters of the QMIX network and other related networks: in Represents the agent at time step According to its action at all times and the current state of the environment Partial rewards received, is the decay factor, which is the coefficient that controls the importance of future rewards in current decisions; For the future state of the environment, For the actions corresponding to the future moments, To periodically update the parameters of the current policy network The parameters of the target policy network are copied from , and the copies are updated at regular time steps.

[0047] The above steps constitute the core technical solution of the present invention. Through the combination of self-attention mechanism, cross-attention mechanism, gated recurrent unit and reinforcement learning QMIX algorithm, the allocation and robust update of agent roles are realized, and efficient and stable agent strategy learning based on role division is completed in complex joint tasks.

[0048] This application proposes a large-scale multi-agent collaborative control technology based on role allocation. By introducing time perception and information sharing mechanisms, the collaborative efficiency of multi-agent systems in complex environments is significantly improved. First, this application innovatively uses the joint information of local observations and behaviors of agents at multiple time steps as the basis for role allocation, rather than relying solely on current observation data. This time-aware role allocation method enables agents to better capture and utilize historical information, thereby improving the stability and long-term effectiveness of the collaborative control process. Secondly, this application designs an information sharing module to further enhance the local information fusion and role allocation effects between agents. This module optimizes the collaboration process through a two-stage information transfer mechanism: in the first stage, the current agent generates brief context information and shares it with other agents; in the second stage, other agents provide customized feedback based on the received context information through an attention-based architecture, so that each agent can obtain richer and more accurate environmental information. This information sharing mechanism effectively compensates for the limitations of local observation and improves the global collaboration capability of multi-agent systems. Finally, the present invention also establishes a more accurate and efficient role allocation mechanism by combining time-aware local observation-behavior information and shared context-aware information. Compared with the traditional role allocation method, this application can automatically implement role allocation without human intervention, and can ensure that the agents collaborate under stable roles to achieve high efficiency in long-term strategy learning. In summary, the framework of this application not only improves the accuracy and stability of multi-agent collaborative control by optimizing role allocation and enhancing information sharing between agents, but also provides an efficient and reliable solution for large-scale agent collaboration systems, which can significantly improve the overall performance of the system and its ability to adapt to complex tasks.

[0049] In one embodiment, the topology diagram of the present application is provided as follows: Figure 2 As shown in Figure 1, the main modules and their interaction processes in the training phase are described, with an emphasis on the role allocation mechanism of time perception and the role of the information sharing module. The agent freely explores in the multi-agent reinforcement learning environment, and the trajectory data obtained includes And store it in the experience replay pool. Sample and align the trajectory data Time step for the agent The observation-action value pair Combining the self-attention mechanism with the fully connected layer to generate a representation of the target agent’s own reference state information Processing self-representation using criss-cross attention networks Observation-action value pairs with other agents , generating feedback from each other agent to the agent Shared context information (i.e. personalized information) ; The agent Local observation-action value pairs Feedback to the agent with all other agents Personalized and beneficial information The aggregated information is input into the role encoder to generate the role distribution, and then input into the role decoder to obtain the policy network weight parameters related to the role. , input GRU layer to get individual action value And additionally consider high-dimensional features becomes and Mixing Network after QMIX algorithm and combined with additional input global state information ,in and Are all weight matrices, get the joint action value The temporal difference loss is calculated, and the loss is used to update the parameters of the QMIX network and other related networks, and the action decision execution is fed back to the multi-agent reinforcement learning environment. In the implementation process of this application, the local information of each agent is first collected and generated through the information sharing module, and aligned at multiple time steps. Specifically, in the first stage, the agent generates short context information through the attention mechanism based on its current local observations and actions. This context information provides a rough description of the environment and behavior for each agent, and this information is then shared with other agents. In the second stage, after receiving the context information, other agents provide detailed feedback to the current agent through an architecture based on the attention mechanism, combined with their own observations and strategies. The feedback is weighted to ensure the pertinence and effectiveness of the information. The agent adjusts its behavior based on the received context feedback, thereby completing the optimization process of role allocation. In this way, the framework of this application can achieve more accurate and efficient role allocation based on local information and shared context information of multiple time steps. After the above two-stage optimization process, the method of this application can automatically complete the role allocation between agents and ensure that the agents can collaborate to perform tasks under stable roles. Compared with the traditional collaborative method based on single-time-step local observations, this application adopts a time-aware dynamic role allocation mechanism, which enables the agents to better adapt to changing environments and task requirements, thereby significantly improving the collaborative stability and learning efficiency of multi-agent systems in long-term tasks.

[0050] The information sharing module structure of this application is as follows Figure 3 As shown in Figure 2, it involves two stages of information sharing. In the first stage, the agent generates brief context information based on its local observation data. Specifically, each agent After collecting local environment information and self-behavior information for multiple time steps, the data is compressed into a concise context description. c i . This contextual information c i It will then be shared with other agents j and k This process is achieved through information broadcasting, ensuring that all agents receive the same collaborative background information. In the second stage, after other agents receive the context information, they will process the context information and generate feedback through the attention mechanism based on their local information and current behavior strategies. m ji and m ki At this point, the agent will use its own attention network to selectively extract key information from the context information based on its relevance and importance, and adjust the current task based on this information. Through this feedback process, other agents provide appropriate suggestions and feedback based on the transmitted context information, helping the target agent to better optimize the role allocation.

[0051] Figure 4 , Figure 5 , Figure 6 Describes the effect of limiting the action space of agents within the same role as the number of learning rounds of the agent increases. Figure 4 The effect of limiting the action space of agents in the same role by this application method when the number of training rounds is 100K. Figure 5 The effect of limiting the action space of agents in the same role by the method of this application when the number of training rounds is 1 million, Figure 6The effect of limiting the action space of agents within the same role by the method of the application when the number of training rounds is 1.5 million. The method of the application was additionally tested on the 1c3s5z map of StarCraft SMAC, which clearly shows the heterogeneity of the agents. In this scenario, the agent needs to control one Colossi (three colossi), three Stalkers (three hunters) and five Zealots (five berserkers) to defeat all enemy units. In the early stages of training (100K training rounds) and the mid-term training stage (500K to 1.5 million training rounds), the action representations of all agents are still exploring in the action space. Through detailed comparison, the application demonstrates a strong ability to limit the exploration of agents in the action space. In the early stages of training, when the learning of role assignments is just beginning, the agents mainly use random exploration to learn their action strategies. At this time, the action representations of all agents are still random and disordered. At the end of the training (2 million training rounds), after repeated learning, the agents were assigned different sub-roles, causing Colossi, Stalkers, and Zealots to show significantly different action tendencies and action patterns. In addition, the cooperation between units became closer, and the agents began to win battles, indicating that the role allocation learning method is effective in collaborative task allocation.

[0052] Figure 7 The winning rate curve of the proposed method after 2 million rounds of training on the 2s3z map in the StarCraft SMAC experimental environment is shown. Figure 8 The winning rate curve of the proposed method after 2 million rounds of training on the 1c3s5z map in the StarCraft SMAC experimental environment is shown. Fig. 9 The winning rate curve of the proposed method after 2 million rounds of training on the 8m_vs_9m map in the StarCraft SMAC experimental environment is shown. The performance of the proposed method on three maps is compared with that of the traditional QMIX algorithm. The proposed method performs better than the QMIX algorithm on all three maps, achieving a 34.71% increase in the average winning rate and a 6.34% reduction in the variance. The double increase in winning rate and variance reflects the advantages of the proposed method in learning efficiency and learning stability. Specifically, the use of multi-time-step context information for role assignment reduces the frequent changes in the roles and action tendencies of the agents, thus facilitating more stable learning.

[0053] exist Fig.10The performance of the method of the present application in two benchmark tests of predator-prey and cooperative navigation in a multi-agent particle environment is demonstrated in the paper. Predator-prey is a cooperative multi-agent reinforcement learning task that simulates a test scenario in which the agent acts as a predator and the prey interacts with the predator. In this scenario, three predators and one prey are set up. The predator needs to capture the prey to win, and the capture process depends on the strategy of the agent. In the cooperative navigation scenario, the agents need to cooperate to move on the plane and cover the predefined goals. The reward is calculated based on the distance between each agent and its assigned goal. In addition, in the cooperative navigation scenario, each agent occupies a certain area in the two-dimensional space, and if two agents "collide" (that is, their activity areas overlap), they will be punished. Therefore, each agent must avoid collisions with other agents while moving towards the designated goal. The method of the present application conducted 20 independent experiments in each of the two scenarios, and finally presented the median performance results. Compared with other algorithms, the method of the present application achieved the highest average reward in both predator-prey and cooperative navigation scenarios, and also showed significant advantages in learning efficiency, indicating the effectiveness of the invented method.

[0054] In summary, through a comprehensive comparison and analysis of the efficiency of agent collaboration in a variety of large-scale multi-agent collaborative control tasks, the framework proposed in this application performs significantly better than traditional methods and other reinforcement learning algorithms in multi-agent systems. This application framework combines the self-attention mechanism and the cross-attention mechanism, introduces a time-aware role allocation mechanism and an information sharing module, effectively reducing the inefficient exploration of agents in the action space, thereby improving collaboration efficiency and task completion. Especially in complex environments and long-term collaborative tasks, the application framework significantly improves the collaborative ability and system stability of agents through multi-time-step contextual information and feedback mechanisms. These improvements not only enhance the adaptability of multi-agent systems, but also make the collaboration of agents in dynamic environments more efficient. Therefore, the excellent performance of the application framework in a variety of complex scenarios verifies its feasibility and effectiveness as an efficient and stable multi-agent collaborative control method.

[0055] At least a portion of the steps in the flowcharts involved in the embodiments described above may include multiple steps or multiple stages. These steps or stages may be executed at different times, or may be executed in turn or alternation with other steps or at least a portion of the steps or stages in other steps.

[0056] Based on the same inventive concept, the embodiment of the present application also provides a large-scale multi-agent collaborative control device based on role allocation for implementing the large-scale multi-agent collaborative control method based on role allocation involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the large-scale multi-agent collaborative control device based on role allocation provided below can be referred to the limitations of the large-scale multi-agent collaborative control method based on role allocation above, and will not be repeated here.

[0057] In an exemplary embodiment, a large-scale multi-agent collaborative control device based on role allocation is provided, comprising: A data collection module is used to collect at least one trajectory data of each agent's interaction with the environment based on the exploration of each agent in the specified environment, and store the trajectory data corresponding to each agent in the experience replay pool; A joint action value generation module is used to execute steps S2-1 to S2-6 for each agent in the specified environment, obtain the individual action value of each agent, generate a joint action value based on the individual action values ​​of all agents using the QMIX network, and then enter the individual action value update module; The individual action value update module is used to update the individual action value of each agent through the Q-learning algorithm based on the joint action value.

[0058] Each module in the above-mentioned large-scale multi-agent cooperative control device based on role allocation can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0059] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0060] For those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these modifications and improvements are all within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A large-scale multi-agent collaborative control method based on role allocation, characterized in that: The method is used to realize automatic role allocation for an intelligent agent, and comprises: Step S1, based on the exploration of each agent in a specified environment, collect at least one trajectory data of each agent interacting with the environment, and store the trajectory data corresponding to each agent into an experience replay pool; Step S2, executing steps S2-1 to S2-6 for each agent in the specified environment, obtaining the individual action value of each agent, generating the joint action value based on the individual action values ​​of all agents using the QMIX network, and then entering step S3; Step S3, based on the joint action value, the individual action value of each agent is updated through the Q-learning algorithm.

2. A large-scale multi-agent collaborative control method based on role allocation according to claim 1, characterized in that: Steps S2-1 to S2-6 in step S2 include: Step S2-1, extracting the target trajectory data corresponding to the target agent from the experience replay pool, and aligning the observation-action value of the target agent at the specified time step; Step S2-2, based on the aligned observation-action value of the target agent, a representation of the target agent's own reference state information is generated by combining the self-attention mechanism and the fully connected layer; Step S2-3, using a cross-attention mechanism, based on the representation of the target agent's own reference state information and the observation-action values ​​of the non-target agents other than the target agent in the specified environment, generates personalized information of the target agent from the non-target agents other than the target agent in the specified environment, and shares the personalized information with the target agent; Step S2-4, based on a pre-trained role encoder including an MLP module, an LSTM module, and a Gaussian distribution module, a Gaussian distribution representing the role of the target agent is generated according to the personalized information shared by the non-target agents and the observation-action value of the target agent received by the target agent; Step S2-5, using a parameterized neural network to generate weight parameters of a target policy network for controlling the target agent according to a Gaussian distribution representing the role of the target agent; Step S2-6, through the Q network, based on the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent, output the individual action value of the target agent.

3. A large-scale multi-agent collaborative control method based on role allocation according to claim 2, characterized in that: The role encoder including the MLP module, the LSTM module and the Gaussian distribution module pre-trained in step S2-4 includes: An MLP module is constructed using MLP, which takes the personalized information shared by the target agent and the aggregate vector of the target agent's own observation-action value as input, and is used to extract features and output the aggregate vector feature representation; LSTM is used to construct an LSTM module with aggregate vector feature representation as input, which is used to process time series and generate aggregate vector high-dimensional features; A linear layer is used to construct a Gaussian distribution module with high-dimensional features of the aggregated vector as input to generate a Gaussian distribution representing the role of the target intelligent agent; The MLP module, LSTM module and Gaussian distribution module in the input-output direction constitute the role encoder. The loss function is designed and used to train the role encoder.

4. A large-scale multi-agent collaborative control method based on role allocation according to claim 3, characterized in that: The design loss function includes: On the basis of reconstructing the loss function, the KL divergence regularization term is introduced and the mutual information between the observation-action value of the target agent and the Gaussian distribution of the target agent role is maximized to obtain the loss function.

5. A large-scale multi-agent collaborative control method based on role allocation according to claim 4, characterized in that: The step S2-5 comprises: The Gaussian distribution of the target agent role is sampled to obtain the role representation of the target agent, and the role representation of the target agent is input into a parameterized neural network, wherein the parameterized neural network includes an MLP and various linear layers; The role representation of the target agent is mapped through a parameterized neural network to generate weight parameters of the target policy network used to control the target agent.

6. A large-scale multi-agent collaborative control method based on role allocation according to claim 5, characterized in that: The process of mapping the role representation of the target agent through a parameterized neural network to generate weight parameters of a target strategy network for controlling the target agent includes: The target agent’s role representation is transformed through MLP to extract high-dimensional features of the target agent’s role representation, and the ReLU activation function is used to improve the nonlinear expression ability. Each linear layer is used to output the weight parameters of each policy network corresponding to each linear layer based on the high-dimensional features of the role representation, wherein each policy network is used to control the corresponding intelligent agent, and the weight parameters of each policy network include the weight parameters of the target policy network used to control the target intelligent agent.

7. A large-scale multi-agent collaborative control method based on role allocation according to claim 6, characterized in that: The step S2-6 comprises: Input the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent into the Q network, which includes an input layer, a GRU module, a fully connected layer, and an output layer; Receive the observation-action value of the target agent and the weight parameters of the target policy network used to control the target agent through the input layer; The GRU module is used to extract time series information from the observation-action value of the target intelligent agent; The observation-action value of the target intelligent agent processed by the GRU module is feature extracted through the fully connected layer, and the individual action value of the target intelligent agent is generated by combining the weight parameters of the target policy network used to control the target intelligent agent; The output layer is used to output the individual action value of the target agent.

8. A large-scale multi-agent collaborative control method based on role allocation according to claim 7, characterized in that: In step S2, the joint action value is generated by using the QMIX network based on the individual action values ​​of all agents, including: The QMIX network is used to process the individual action values ​​of all intelligent agents, and the individual action values ​​of all intelligent agents are mapped to obtain the joint action value through nonlinear combination.

9. A large-scale multi-agent collaborative control device based on role allocation, characterized in that: The device comprises: A data collection module is used to collect at least one trajectory data of each agent's interaction with the environment based on the exploration of each agent in the specified environment, and store the trajectory data corresponding to each agent in the experience replay pool; A joint action value generation module is used to execute steps S2-1 to S2-6 for each agent in the specified environment, obtain the individual action value of each agent, generate a joint action value based on the individual action values ​​of all agents using the QMIX network, and then enter the individual action value update module; The individual action value update module is used to update the individual action value of each agent through the Q-learning algorithm based on the joint action value.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images