Multi-agent reinforcement learning method, electronic equipment and storage medium

Abstract

The invention provides a multi-agent reinforcement learning method, electronic equipment and a storage medium, and the method comprises the steps: an agent generates an action through an actor network by inputting the observation of the agent, and then inputs the action into a reviewer network of the agent to generate a value independent of the agent, and inputting the values generated by all the intelligent agents and the own characteristics of each intelligent agent into the attention hybrid network to generate the collective value of all the intelligent agents. Meanwhile, actions, observation and global states are input into a lesion random network distillation network to generate exploration contributions of each agent, and the exploration contributions are quantified by reward values. The generated reward value is input into another attention hybrid network, generating a collective target value, and each agent network is optimized by minimizing the collective value and the value of the collective target value. According to the method, the intelligent agent exploration contribution can be quantized, and the intelligent agent value contribution is not merely quantized.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning, and in particular to a multi-agent reinforcement learning method, electronic equipment and storage medium. Background Art

[0002] In recent years, multi-agent reinforcement learning (MARL) has attracted widespread attention and has been applied to real-world settings, showing great potential, such as fleet control, schedule optimization, resource management, and autonomous vehicles. Despite its unprecedented success in many fields and applications, many challenges still exist in the multi-agent reinforcement learning environment.

[0003] A major challenge in multi-agent reinforcement learning (MARL) is the coordination problem between agents (similar to machine players) due to the partially observable environment. From an individual perspective, each agent pursues its own optimal strategy when learning, rather than the optimal strategy of the entire team. However, in scenarios involving multiple agents, they need to effectively collaborate to achieve a common goal, rather than acting independently. This collaborative nature requires agents to coordinate and communicate with each other to take appropriate actions in complex environments. Sample inefficiency is another major challenge in MARL, resulting in millions of interactions required even for relatively simple problem settings, which is very detrimental to agent training. As the number of agents increases, the state-action space grows exponentially, exacerbating the sample inefficiency problem. Furthermore, each agent needs to learn in an uncertain environment, where multiple local optima may exist. Leveraging efficient exploration—that is, effectively exploring the unknown environment to gather the most information and experience that is most beneficial for policy learning—can alleviate sample inefficiency, discover potential optimal solutions, and avoid falling into suboptimal ones.

[0004] Credit-assigned value decomposition (CVA) has become a key approach for solving collaborative multi-agent tasks in recent years. It plays a crucial role in solving such tasks and is rooted in the centralized training and decentralized execution (CTDE) paradigm. In the CTDE paradigm, agents can share information and experience during training to better learn collaborative strategies. Specifically, during training, agents generate a value by inputting their own observations and actions. The values ​​generated by all agents are input into the hybrid network to generate a collective value for all agents, which is then used to optimize the agent network during training. During the execution phase, each agent can execute tasks in a decentralized and independent manner based on the learned strategy (agent network) to achieve the collaborative goal. Through the CVA method, agents can better understand the contributions of other agents to the task, enabling them to adjust their strategies to improve collaborative efficiency.

[0005] However, the single contribution generated by credit assignment is not sufficient to accurately characterize the role of the agent in learning, such as the exploration effect. Figure 1 Consider the scenario in Figure 1, where two agents are exploring four rooms. Initially, agent A explores room 2, and agent B explores room 3. This represents a new state for each agent and a new state from a global perspective. While exploration of this new state is encouraged, in the next step, agents A and B swap exploration paths, with agent A now exploring room 3 and agent B exploring room 2. From the perspective of each individual agent, they have indeed visited new states. However, from a global perspective, these states have already been explored. This exchange does not add any new meaning to the overall cooperative exploration. Agents should be encouraged to explore new states from a global perspective, such as agent A visiting room 1 and agent B visiting room 4. In this case, using a single contribution approach only considers the agent's contribution to the team reward, but cannot assess the agent's exploration contribution at the global level. Therefore, in current technical solutions, the value of agents and the collective value generated are generated through external stimuli (rewards from the environment) to optimize the agent network. Agent exploration, as a means of rapid algorithm convergence, cannot be quantified through environmental rewards. Summary of the Invention

[0006] In view of the above technical problems, the technical solution adopted by the present invention is:

[0007] According to a first aspect of the present invention, a multi-agent reinforcement learning method is provided, the method comprising the following steps:

[0008] S100, setting the training times counter C=1.

[0009] S200, if C≤C0, execute S300, otherwise, execute S900; C0 is a preset training times threshold.

[0010] S300, setting the data collection times counter N=1.

[0011] S400, if N≤N0, execute S500, otherwise, execute S800; N0 is the preset data collection times threshold.

[0012] S500, initialize the multi-agent environment and set the time step calculator p=1.

[0013] S600, if p≤p0, execute S700, otherwise, set N=N+1, execute S; p0 is the preset time step threshold.

[0014] S700, input the observation information currently observed by the intelligent agent into the corresponding actor network, obtain the corresponding action, and execute the action to obtain the corresponding external reward, life value and global state information, obtain the corresponding five-tuple information, and store it in the current sample training database as historical data; if the life values ​​of all intelligent agents are 0, set N=N+1 and execute S400, otherwise, set p=p+1 and execute S600; the five-tuple information includes observation information, action, external reward obtained by executing the action, life value and global state information.

[0015] S800, obtain k sample data groups from the current sample training database as the current training sample data, and use the current training sample data to train the current multi-agent reinforcement learning network model to obtain the corresponding current output result; if it is determined based on the current output result that the current multi-agent reinforcement learning network model meets the convergence condition, execute S900, otherwise, update the parameters of the current multi-agent reinforcement learning network model based on the current output result, and set C=C+1, and execute S200; the multi-agent reinforcement learning network includes an actor network, a critic network, a duel random network distillation network, a first attention hybrid network, a second attention hybrid network and a target value acquisition module; one sample data group corresponds to historical data obtained from a data collection.

[0016] S900, takes the current multi-agent reinforcement learning network model as the target multi-agent reinforcement learning network model.

[0017] Among them, the current multi-agent reinforcement learning network model is trained using the current training sample data to obtain the corresponding current output results, including:

[0018] S801: The observation information o corresponding to the sample data corresponding to the i-th agent in the current training sample data is i and action u i Input into the corresponding critic network to get the corresponding value Q i ; The value of i ranges from 1 to n, where n is the number of multi-agents.

[0019] S802: Input the values, agent feature information, and global state information corresponding to all agents into the first attention hybrid network to obtain the first collective value Q1 corresponding to all agents. tot , and the value, agent feature information and global state information corresponding to all sample data are input into the second attention hybrid network respectively, and the corresponding collective value Q2 of all agents is obtained tot, and input the observation information, actions and global state information corresponding to all agents into the duel random network distillation network to obtain the global exploration reward value r corresponding to all agents tot in .

[0020] S803, the total external reward corresponding to the current training sample data, Q2 tot and r tot in Input the target value acquisition module to obtain the collective target value Q corresponding to all agents Target tot , the output results include Q1 tot and Q Target tot .

[0021] According to a second aspect of the present invention, an electronic device is provided, comprising a processor and a memory; the processor is configured to execute the steps of the method according to the first aspect of the present invention by calling a program or instruction stored in the memory.

[0022] According to a second aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing a program or instructions for enabling a computer to execute the steps of the method according to the first aspect of the present invention.

[0023] The present invention has at least the following beneficial effects:

[0024] The multi-agent reinforcement learning method provided by an embodiment of the present invention uses an attention mechanism to model the local observations and global states of the agents to obtain hidden features that encapsulate the global dynamic relationship information of the agents. The observation representation containing the global dynamic relationship is merged with the global original state using a residual structure. Then, it is connected with each action and simultaneously input into a fixed network and a prediction network for prediction, including: encoding the input vector to better extract important features, using two feature extractors to extract global features and local features respectively, and using a decoder to integrate global and local features. Finally, the features generated by the prediction network and the fixed network are used to calculate the mean square error, thereby generating an intrinsic exploration reward for each agent, which can quantify the exploration contribution of the agent, rather than simply quantifying the value contribution of the agent.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 A schematic diagram of the exploration effect of existing multi-agents;

[0028] Figure 2 A flowchart of a multi-agent reinforcement learning method provided by an embodiment of the present invention;

[0029] Figures 3 to 7 Schematic diagram of experimental comparison. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0032] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of the steps can be performed in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. A process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0033] The embodiment of the present invention provides a multi-agent reinforcement learning method, such as Figure 1 As shown, the method may include the following steps:

[0034] S100, setting the training times counter C=1.

[0035] S200, if C≤C0, execute S300, otherwise, execute S900; C0 is a preset training times threshold.

[0036] In the embodiment of the present invention, C0 can be set based on actual needs and can be an empirical value, generally the number of times that can make the model converge.

[0037] S300, setting the data collection times counter N=1.

[0038] S400, if N≤N0, execute S500, otherwise, execute S800; N0 is the preset data collection times threshold.

[0039] In the embodiment of the present invention, N0 may be set based on actual needs and may be an empirical value. In an exemplary embodiment, N0 may be set to 20.

[0040] S500, initialize the multi-agent environment and set the time step calculator p=1.

[0041] In embodiments of the present invention, a multi-agent environment can be any scenario requiring collaborative exploration and decision-making, whether symmetrical or asymmetrical, skill-based or simply relying on cooperation. In one exemplary embodiment, it can be a gaming scenario, such as a multi-agent experimental environment for the game StarCraft. The multi-agents in a multi-agent environment are multiple agents in a collaborative relationship.

[0042] S600, if p≤p0, execute S700, otherwise, set N=N+1, execute S; p0 is the preset time step threshold.

[0043] In the embodiment of the present invention, p0 can be set based on actual needs and can be an empirical value. The number of time steps corresponding to different collection times can be the same or different. Under the same circumstances, in an exemplary embodiment, p0 can be set to 128.

[0044] S700, input the observation information currently observed by the agent into the corresponding actor network, obtain the corresponding action, and execute the action to obtain the corresponding external reward, life value and global state information, obtain the corresponding five-tuple information, and store it in the current sample training database as historical data; the five-tuple information includes observation information, action, external reward, life value and global state information obtained by executing the action; if the life values ​​of all agents are 0, it means that all agents can no longer perform the task, and this training ends, set N=N+1, and execute S400, otherwise, set p=p+1, and execute S600.

[0045] In the embodiment of the present invention, each agent corresponds to its own actor network. The structure of the actor network can be an existing network structure.

[0046] In this embodiment of the present invention, observation information refers to the environmental information that the current agent can observe within its field of view. Taking a game environment as an example, observation information may include the distance between the agent and other agents, the agent's x-coordinate relative to the origin on the map, the agent's y-coordinate relative to the origin on the map, the agent's health, the agent's shield, the agent's functional role type, the last action of allied agents, and the surrounding terrain features. Global state information refers to information corresponding to all agents in the current environment. For example, in a game environment, it may include the coordinates of all agents relative to the center of the map, the observation information of all agents, the agent's energy, the agent's cooldown time, and the last action of all agents.

[0047] Those skilled in the art know that the content defined by S300 to S700 is a data collection phase. In each data collection phase, the multi-agent environment will be initialized and the agent will be given an initial state so that it can interact with the environment.

[0048] S800, obtain k sample data groups from the current sample training database as the current training sample data, and use the current training sample data to train the current multi-agent reinforcement learning network model to obtain the corresponding current output result; if it is determined based on the current output result that the current multi-agent reinforcement learning network model meets the convergence condition, execute S900, otherwise, update the parameters of the current multi-agent reinforcement learning network model based on the current output result, and set C=C+1, and execute S200; the multi-agent reinforcement learning network includes an actor network, a critic network, a duel random network distillation network, a first attention hybrid network, a second attention hybrid network and a target value acquisition module; one sample data group corresponds to the historical data obtained from a data collection, that is, the data in a data collection stage.

[0049] In the embodiment of the present invention, k≥NO, and can be set based on actual needs. In an exemplary embodiment, k can be set to 32.

[0050] S900, takes the current multi-agent reinforcement learning network model as the target multi-agent reinforcement learning network model.

[0051] In an embodiment of the present invention, the current multi-agent reinforcement learning network model is trained using the current training sample data to obtain the corresponding current output result, which may specifically include:

[0052] S801: The observation information o corresponding to the sample data corresponding to the i-th agent in the current training sample data is i and action u i Input into the corresponding critic network to get the corresponding value Q i; The value of i ranges from 1 to n, where n is the number of multi-agents.

[0053] It is known to those skilled in the art that i Includes the observation information of the i-th agent at all time steps corresponding to the current training sample data, u i For o i The actions corresponding to all corresponding time steps.

[0054] S802: Input the values, agent feature information, and global state information corresponding to all agents into the first attention hybrid network to obtain the first collective value Q1 corresponding to all agents. tot , and the value, agent feature information and global state information corresponding to all sample data are input into the second attention hybrid network respectively, and the corresponding collective value Q2 of all agents is obtained tot , and input the observation information, actions and global state information corresponding to all agents into the duel random network distillation network to obtain the global exploration reward value r corresponding to all agents tot in .

[0055] S803, the total external reward corresponding to the current training sample data, Q2 tot and r tot in Input the target value acquisition module to obtain the collective target value Q corresponding to all agents Target tot ;

[0056] In the embodiment of the present invention, the output result includes Q1 tot and Q Target tot .

[0057] In the embodiment of the present invention, Q Target tot =r ex +βr tot in +γQ2 tot , β and γ are both hyperparameters, r ex is the total extrinsic reward corresponding to the current training sample data.

[0058] In the embodiment of the present invention, if Q2 tot and Q Target tot The absolute value of the difference between the two values ​​is less than the set threshold and remains stable during several consecutive training sessions. The set threshold can be a number infinitely close to 0.

[0059] In the embodiment of the present invention, based on Q2 totand Q Target tot , update the parameters of the current network model using gradient descent, any Q2-based tot and Q Target tot , and the updating method of updating the parameters of the current network model using the gradient descent method all fall within the protection scope of the present invention.

[0060] Furthermore, in an embodiment of the present invention, the duel random network distillation network may include an attention module, a first duel exploration reward calculation module, and a second duel exploration reward calculation module. The attention module is used to model the local observation information and global state of the agent to obtain hidden features that encapsulate the agent's global dynamic relationship information and establish the relationship between the agent and the global environment. The first duel exploration reward calculation module and the second duel exploration reward calculation module are used to evaluate the agent's exploration effect, so that the agent achieves a balance between the local and global perspectives, thereby obtaining the intrinsic exploration reward generated by each agent.

[0061] In the embodiment of the present invention, r tot in It can be obtained based on the hidden features obtained by the attention module and the intrinsic exploration reward obtained by the duel exploration reward calculation module. Specifically, it can be obtained through the following steps:

[0062] S10, o i 、u i and S t Input into the attention module to obtain the corresponding attention representation feature AT i ;S t is the global state information corresponding to the current environment corresponding to the current sample training data; is the attention coefficient obtained by the u-th attention head of the attention module, where u ranges from 1 to h, and h is the number of attention heads of the attention module. Wk(o i ) means to change o i Convert to key vector, Wq(ES t ) means to convert S t The embedding vector ES i Converted to query vector, Wv(o i ) means to change o i Converted to a value vector, · represents dot product; Softmax() represents normalization operation; × represents cross product.

[0063] It is known to those skilled in the art that S t It may include the global state information of the agent corresponding to all time steps corresponding to the current sample training data.

[0064] In the embodiment of the present invention, the embedding vector can be obtained through one or two layers of embedding transformation, and the specific transformation method may belong to the existing technology. In the embodiment of the present invention, the observation information uses original data as input, which can better preserve the original information.

[0065] It is known to those skilled in the art that any method of converting a feature vector into a key vector, a query vector, or a value vector falls within the scope of protection of the present invention.

[0066] In a multi-agent environment, the state space is huge. If there is no sufficient representation to eliminate a large amount of irrelevant information, even similar states may show huge metric differences, resulting in inaccurate evaluation of the agent's intrinsic reward. i ) operation converts the original observation value into individual value and performs dimensionality reduction, which makes it easier for the subsequent network to extract important features.

[0067] S11, AT i and S t Perform residual splicing to obtain the corresponding splicing feature vector PF i .

[0068] S12, PF i and u i Input into the first duel exploration reward calculation module and the second duel exploration reward calculation module respectively to obtain the corresponding first exploration reward feature vector DF1 i and the second exploration reward feature vector DF2 i .

[0069] In an embodiment of the present invention, the first duel exploration reward calculation module and the second duel exploration reward calculation module have the same structure but different network parameters, and both include an encoder, a global feature extractor, a local feature extractor and a decoder.

[0070] Among them, the parameters of the first duel exploration reward calculation module are updated during each training, and the parameters of the second duel exploration reward calculation module are fixed.

[0071] Among them, DF1 i DF2 is the feature vector obtained by the decoder of the first duel exploration reward calculation module. i Feature vector obtained by the decoder of the second duel exploration reward calculation module.

[0072] In an embodiment of the present invention, F i E is the feature vector obtained by the encoder of the first duel exploration reward calculation module, σ1 G (F iE ) means F i E Input the global feature vector obtained from the global feature extractor of the first duel exploration reward calculation module, is the first local eigenvector after correction, The dth feature in is σ1 L (F i E ), the dth feature in , the value of d ranges from 1 to D, D is σ1 L (F i E ), σ1 L (F i E ) means F i E Input the local feature vector obtained by the local feature extractor of the first duel exploration reward calculation module.

[0073] F i E The feature vector obtained by the encoder of the second duel exploration reward calculation module, σ2 G (F i E ) means F i E Input the global feature vector obtained from the global feature extractor of the second duel exploration reward calculation module, is the modified second local eigenvector, The dth feature in is σ2 L (F i E ), the dth feature in the equation, d ranges from 2 to D, and D is σ2 L (F i E ), σ2 L (F i E ) means F i E The local feature vector obtained by the local feature extractor of the second duel exploration reward calculation module is input.

[0074] S13, based on DF1 i and DF2 i Get the intrinsic reward corresponding to the i-th agent f1 ig DF1 i The g-th feature in f2ig DF2 i The gth feature in , the value of g ranges from 1 to X, and X is the number of features in the exploration reward feature, that is, the dimension of the exploration reward feature.

[0075] In an embodiment of the present invention, the intrinsic reward of the intelligent agent is merged into the corresponding dimension of the feature vector. During the merging process, the average value of all dimensions is subtracted from each dimension of the local feature, which can reduce the estimation variance.

[0076] In the embodiment of the present invention, by the minimum r i in To optimize the parameters of the duel exploration reward calculation module. As the network encounters more common states, r i in will be reduced, providing a small exploration bonus for spaces that do not need to be explored.

[0077] S14, based on and r i in Get

[0078] In this embodiment of the present invention, the attention weights obtained by the attention module are used as weights for calculating the global exploration reward. This prevents loss of agent contribution information and allows for adaptation to dynamic relationship settings. Thus, the agent's exploration contribution is evaluated and integrated into the global exploration reward for policy optimization. In this embodiment of the present invention, by quantifying the agent's exploration contribution and awarding it an exploration reward, exploration is promoted, thereby improving sample efficiency and accelerating convergence.

[0079] Furthermore, in an embodiment of the present invention, the first attention hybrid network and the second attention hybrid network have the same structure but different network parameters, and both include an attention module and a weight calculation module.

[0080] The parameters of the first attention hybrid network are updated during each training, and the parameters of the second attention hybrid network are updated in each set period by copying the parameters of the first attention hybrid network.

[0081] In multi-agent reinforcement learning, simply evaluating exploration contributions is insufficient. In keeping with the original goal of reinforcement learning, it is also important to evaluate the global goal-oriented contribution. This paper uses a value decomposition approach to achieve this, but in a different way than existing approaches. While simply combining the monotonically nonlinear values ​​of each agent into a matrix value using a hybrid network has achieved significant results, it is still insufficient to comprehensively evaluate the goal-oriented contributions of the agents. As mentioned earlier, the relationships between agents are dynamic, and the importance of different agents may vary at different time steps. To account for this, this paper reintroduces the attention mechanism, dynamically establishing connections between individual and global features and applying this to adjust the values ​​of the agents. This paper adjusts the values ​​output by each critic by incorporating global information to better facilitate the hybrid network's calculation of the collective value. The input value is converted into an attention-based value through the attention mechanism, which is then fed into the hybrid network to generate Qtot.

[0082] Q1 tot Obtained through the following steps:

[0083] S20, Q i 、AC i and S t Input into the attention module of the first attention hybrid network to obtain the corresponding first attention-based value Q1 a i , and S i Input into the weight calculation module of the first attention hybrid network to obtain the corresponding first multiplication weight matrix W1 1i , the first addition weight matrix b1 1i , the second multiplication weight matrix W1 2i and the second summing weight matrix b1 2i ;AC i is the agent feature corresponding to the i-th agent in the current sample training data, S t It is the global state information corresponding to the current environment corresponding to the current sample training data.

[0084] In the embodiment of the present invention, the weight calculation module may be a weight calculation module in a network used in an existing credit allocation value decomposition method, and the specific method for obtaining the weight may be the existing technology.

[0085] In an embodiment of the present invention, is the attention coefficient obtained by the u1th attention head of the attention module of the first attention hybrid network, where u1 ranges from 1 to h1, and h1 is the number of attention heads of the attention module of the first attention hybrid network. Wk1(ACi ) indicates that the attention module of the first attention hybrid network is used to convert AC i Convert to key vector, Wq1(ES t ) indicates that the attention module of the first attention hybrid network is used to t The embedding vector ES t Converted to a query vector, · represents dot product; Softmax() represents normalization operation; × represents cross product.

[0086] S21, based on Q1 a i 、W1 1i 、b1 1i 、W1 2i and b1 2i Get Q1 tot =((Q1 a i W1 1i )+b1 1i )·W1 2i +b1 2i .

[0087] Q2 tot Obtained through the following steps:

[0088] S30, Q i 、AC i and S i Input into the attention module of the second attention hybrid network to obtain the corresponding second attention-based value Q2 a i , and S i Input into the weight calculation module of the second attention hybrid network to obtain the corresponding first multiplication weight matrix W2 1i , the first addition weight matrix b2 1i , the second multiplication weight matrix W2 2i and the second summing weight matrix b2 2i .

[0089] in, is the attention coefficient obtained by the u2th attention head of the attention module of the second attention hybrid network. The value of u2 ranges from 1 to h2. h2 is the number of attention heads of the attention module of the second attention hybrid network. Wk2(AC i ) indicates that the attention module of the second intention-force hybrid network is used to i Convert to key vector, Wq2(ES t ) indicates that the attention module of the second intention-force hybrid network is used to t The embedding vector ESt Convert to a query vector.

[0090] S31, based on Q2 a i 、W2 1i 、b2 1i 、W2 2i and b2 2i Get Q2 tot =((Q2 a i W2 1i )+b2 1i )·W2 2i +b2 2i .

[0091] In an embodiment of the present invention, by adopting the attention mechanism to model the dynamic relationship between the intelligent agent and the global environment and applying it to adjust the value, the global relationship information can be merged before entering the hybrid network, which is conducive to more comprehensive calculations for goal-oriented exploration and enhances the effectiveness of the hybrid network.

[0092] In the present embodiment, SMAC is used as an experimental scenario (as shown in Table 1 below) to illustrate the convergence effect of the method provided by the present invention. SMAC is based on the popular real-time strategy (RTS) game "StarCraft II." Unlike the full "StarCraft II," SMAC focuses on the micro-strategy of each agent, specifically controlling each agent to defeat the enemy. Each unit is controlled by a single agent, and its decision-making is entirely based on local observations within a limited perspective centered on that unit. In different scenarios, the agents need to coordinate various skills to achieve victory.

[0093] Table 1:

[0094]

[0095] Through multiple experiments, we compared the existing DIFFER (Decomposing Individual Reward for Fair Experience Replay: a method for decomposing individual rewards for fair experience replay), FACMAC (Factored Multi-Agent Centralised Policy Gradients: a method for decomposing multi-agent centralized policy gradients), LIIR (Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning: a method for learning individual intrinsic rewards in multi-agent reinforcement learning), MASIA (Multi-Agent communication via Self-supervised Information Aggregation: a method for multi-agent communication via self-supervised information aggregation), Qatten (a practical multi-head attention based Q-value mixing network: a practical multi-head attention based Q-value mixing network), QMIX (Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning: a method for decomposing monotonic value functions for deep multi-agent reinforcement learning), and Qplex (DUPLEX DUELING MULTI-AGENT Compared with seven other methods such as Q-LEARNING (Dual Duel Multi-Agent Q Learning Method), the method provided by the embodiment of the present invention can achieve fast convergence under different experimental environments, such as Figures 3 to 7 As shown, Figure 3 This is the experimental effect diagram under the MMM2 scene. Figure 4 This is the experimental effect diagram under the 3s_vs_3z scene. Figure 5 This is the effect diagram of the 2c_vs_64zg scene. Figure 6 This is the experimental effect diagram under the MMM scenario. Figure 7The figure shows the effect of the 8m scenario, in which the method provided by the embodiment of the present invention is represented as DGEC in the figure. MMM2 is considered to be a very difficult asymmetric scenario, that is, our intelligent agent (ally) units are overwhelmed by enemy units in terms of numbers, and concentrated firepower strategies and extensive policy coordination are required to enable the allied units to defeat the numerically superior enemy units. Through experiments, compared with simple and difficult scenarios, the time for all methods to converge to the optimal solution is delayed. However, the method provided by the embodiment of the present invention still shows excellent performance. Affected by collaborative exploration and decision-making, the algorithm quickly reaches a suboptimal state in the early stage of training and then tends to converge stably. In contrast, other methods begin to converge in the later stage of training. Specifically, the method provided by the embodiment of the present invention achieves a 60% win rate as early as the time step of 8e5 steps, while the win rate of other methods before 1e6 steps is almost below 40%, or even 0%.

[0096] Experiments show that the method provided by the present invention is faster than existing techniques in solving SMAC scenarios and converges to the ideal decision-making strategy more quickly during training. Furthermore, the method provided by the present invention can be applied to a variety of scenarios requiring collaborative exploration and decision-making, whether symmetric or asymmetric, skill-based or simply relying on simple cooperation. It can effectively and robustly perform tasks even in complex environments.

[0097] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.

[0098] An embodiment of the present invention further provides a non-transitory computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.

[0099] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved. This is not limited herein.

[0100] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A multi-agent reinforcement learning method, characterized in that: The method comprises the following steps: S100, setting training times counter C=1; S200, if C≤C0, execute S300, otherwise, execute S900; C0 is a preset training times threshold; S300, setting the data collection times counter N=1; S400, if N≤N0, execute S500, otherwise, execute S800; N0 is the preset data collection times threshold; S500, initialize the multi-agent environment and set the time step calculator p=1; S600, if p≤p0, execute S700, otherwise, set N=N+1 and execute S; p0 is the preset time step threshold; S700, inputting the observation information currently observed by the agent into the corresponding actor network, obtaining the corresponding action, and executing the action to obtain the corresponding external reward, life value and global state information, obtaining the corresponding five-tuple information, and storing it in the current sample training database as historical data; if the life values ​​of all agents are 0, set N=N+1 and execute S400, otherwise, set p=p+1 and execute S600; the five-tuple information includes observation information, action, external reward obtained by executing the action, life value and global state information; S800, obtaining k sample data groups from the current sample training database as current training sample data, and using the current training sample data to train the current multi-agent reinforcement learning network model to obtain the corresponding current output result; If it is determined based on the current output result that the current multi-agent reinforcement learning network model meets the convergence condition, execute S900; otherwise, update the parameters of the current multi-agent reinforcement learning network model based on the current output result, set C=C+1, and execute S200; the multi-agent reinforcement learning network includes an actor network, a critic network, a duel random network distillation network, a first attention hybrid network, a second attention hybrid network and a target value acquisition module; a sample data group corresponds to historical data obtained from a data collection; S900, taking the current multi-agent reinforcement learning network model as the target multi-agent reinforcement learning network model; Among them, the current multi-agent reinforcement learning network model is trained using the current training sample data to obtain the corresponding current output results, including: S801: The observation information o corresponding to the sample data corresponding to the i-th agent in the current training sample data is i and action u i Input into the corresponding critic network to get the corresponding value Q i ; The value of i ranges from 1 to n, where n is the number of multi-agents; S802: Input the values, agent feature information and global state information corresponding to all agents into the first attention hybrid network respectively, and obtain the first collective value Q1 corresponding to all agents. tot , and the values ​​corresponding to all sample data, agent feature information, and global state information are input into the second attention hybrid network to obtain the corresponding collective value Q2 of all agents tot , and input the observation information, actions and global state information corresponding to all agents into the duel random network distillation network to obtain the global exploration reward value r corresponding to all agents tot in ; S803, the total external reward corresponding to the current training sample data, Q2 tot and r tot in Input the target value acquisition module to obtain the collective target value Q corresponding to all agents Target tot ; The output results include Q1 tot and Q Target tot .

2. The multi-agent reinforcement learning method according to claim 1, characterized in that: The duel random network distillation network includes an attention module, a first duel exploration reward calculation module, and a second duel exploration reward calculation module; r tot in Obtained through the following steps: S10, o i 、u i and S t Input into the attention module to obtain the corresponding attention representation feature AT i ; S t is the global state information corresponding to the current environment corresponding to the current sample training data; is the attention coefficient obtained by the u-th attention head of the attention module, where u ranges from 1 to h, and h is the number of attention heads of the attention module. Wk(o i ) means to convert o i Convert to key vector, Wq(ES t ) means to convert S t The embedding vector ES i Converted to query vector, Wv(o i ) means to convert o i Convert to a value vector, · represents dot product; Softmax() represents normalization operation; × represents cross product; S11, AT i and S t Perform residual splicing to obtain the corresponding splicing feature vector PF i ; S12, PF i and u i Input into the first duel exploration reward calculation module and the second duel exploration reward calculation module respectively to obtain the corresponding first exploration reward feature vector DF1 i and the second exploration reward feature vector DF2 i ; S13, based on DF1 i and DF2 i Get the intrinsic reward corresponding to the i-th agent f1 ig For DF1 i The g-th feature in f2 ig For DF2 i The gth feature in , where g ranges from 1 to X, and X is the number of features in the exploration reward feature; S14, based on and r i in Get 3. The method according to claim 2, characterized in that The first duel exploration reward calculation module and the second duel exploration reward calculation module have the same structure but different network parameters, and both include an encoder, a global feature extractor, a local feature extractor, and a decoder; Among them, DF1 i DF2 is the feature vector obtained by the decoder of the first duel exploration reward calculation module i The feature vector obtained by the decoder of the second duel exploration reward calculation module; F i E is the feature vector obtained by the encoder of the first duel exploration reward calculation module, σ1 G (F i E ) indicates that F i E Input the global feature vector obtained from the global feature extractor of the first duel exploration reward calculation module, is the first local eigenvector after correction, The dth feature in is σ1 L (F i E ), the value of d ranges from 1 to D, and D is σ1 L (F i E ), σ1 L (F i E ) indicates that F i E Input the local feature vector obtained in the local feature extractor of the first duel exploration reward calculation module; F i E is the feature vector obtained by the encoder of the second duel exploration reward calculation module, σ2 G (F i E ) indicates that F i E Input the global feature vector obtained from the global feature extractor of the second duel exploration reward calculation module, is the modified second local eigenvector, The dth feature in is σ2 L (F i E ), the value of d ranges from 2 to D, and D is σ2 L (F i E ), σ2 L (F i E ) indicates that F i E The local feature vector obtained by the local feature extractor of the second duel exploration reward calculation module is input.

4. The method according to claim 1, characterized in that: The first attention hybrid network and the second attention hybrid network have the same structure but different network parameters, and both include an attention module and a weight calculation module; Q1 tot Obtained through the following steps: Q i , AC i and S t Input into the attention module of the first attention hybrid network to obtain the corresponding first attention-based value Q1 a i , and S i Input into the weight calculation module of the first attention hybrid network to obtain the corresponding first multiplication weight matrix W1 1i , the first addition weight matrix b1 1i , the second multiplication weight matrix W1 2i and the second addition weight matrix b1 2i ;AC i is the agent feature corresponding to the i-th agent in the current sample training data, S t The global state information corresponding to the current environment corresponding to the current sample training data; Based on Q1 a i 、W1 1i 、b1 1i 、W1 2i and b1 2i Get Q1 tot =((Q1 a i W1 1i )+b1 1i )·W1 2i +b1 2i ; Q2 tot Obtained through the following steps: Q i , AC i and S i Input into the attention module of the second attention hybrid network to obtain the corresponding second attention-based value Q2 a i , and S i Input into the weight calculation module of the second attention hybrid network to obtain the corresponding first multiplication weight matrix W2 1i , the first addition weight matrix b2 1i , the second multiplication weight matrix W2 2i and the second addition weight matrix b2 2i ; Based on Q2 a i 、W2 1i 、b2 1i 、W2 2i and b2 2i Get Q2 tot =((Q2 a i ·W2 1i )+b2 1i )·W2 2i +b2 2i .

5. The method according to claim 4, characterized in that Q Target tot =r ex +βr tot in +γQ2 tot , β and γ are both hyperparameters, r ex is the total extrinsic reward corresponding to the current training sample data.

6. The method according to claim 4, characterized in that is the attention coefficient obtained by the u1th attention head of the attention module of the first attention hybrid network, where u1 ranges from 1 to h1, and h1 is the number of attention heads of the attention module of the first attention hybrid network. Wk1(AC i ) indicates that the attention module of the first attention hybrid network is used to convert AC i Convert to key vector, Wq1(ES t ) indicates that the attention module of the first attention hybrid network is used to t The embedding vector ES t Converted to query vector, · represents dot product; Softmax() represents normalization operation; × represents cross product; is the attention coefficient obtained by the u2th attention head of the attention module of the second attention hybrid network. The value of u2 ranges from 1 to h2. h2 is the number of attention heads of the attention module of the second attention hybrid network. Wk2(AC i ) indicates that the attention module of the second intention-force hybrid network is used to convert AC i Convert to key vector, Wq2(ES t ) indicates that the attention module of the second intentional hybrid network is used to t The embedding vector ES t Convert to a query vector.

7. An electronic device, characterized in that: including a processor and a memory; The processor is used to execute the steps of the method according to any one of claims 1 to 6 by calling the program or instruction stored in the memory.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store a program or instruction, and the program or instruction enables a computer to execute the steps of the method according to any one of claims 1 to 6.

Images