Multi-agent decision-making method based on combination of MAPPO and hierarchical attention mechanism

Abstract

The invention provides a multi-agent decision-making method based on the combination of MAPPO and a hierarchical attention mechanism, and the method comprises the steps: inputting the observation state information of a current moment into a strategy network, which introduces a local attention mechanism, of each agent, and outputting a first action; second actions of other intelligent agents are combined to be input into the value network with the global attention mechanism introduced, and combined actions of the intelligent agents are output; the local attention mechanism of the strategy network can make the intelligent agents focus on key local information around the intelligent agents, the global attention mechanism of the value network can collectively collect the macroscopic situation of the whole battlefield, and the task value of each intelligent agent can be reasonably evaluated. When the situation of a battlefield is changed, dynamic adjustment can be rapidly made by means of the attention mechanism of the double view angles, and the MAPPO algorithm has the strengthening effect on agent cooperation, cooperates with the attention mechanism, dynamically weights according to task requirements and real-time situations, makes more optimized decisions and improves the success rate of combat.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-agent decision-making, and specifically relates to a multi-agent decision-making method based on the combination of MAPPO and hierarchical attention mechanism. Background Art

[0002] As an important means of simulating war, wargaming is of crucial significance in the fields of military research, strategic planning, etc. With the development of artificial intelligence technology, the application of multi-agent reinforcement learning in wargaming has gradually attracted attention. In reinforcement learning, the Markov decision process is a core concept. The agent selects an action based on the current state, the environment transfers to a new state according to this action, and gives the agent a feedback, that is, a reward. The agent optimizes its policy through learning to maximize the cumulative reward.

[0003] However, there are many problems in the existing technology when facing the complex environment of wargaming. Currently, some research focuses on the individual decision-making optimization of agents. By improving the reinforcement learning algorithm, the agent can make more reasonable action choices in a specific environment. There is a lack of an effective cooperation mechanism among multi-agents. When performing tasks such as offense, defense, and capturing key points, each agent often only considers its own interests or local information, and fails to make decisions from the overall team goal, which easily leads to repeated actions or action conflicts, reducing the efficiency of task execution. The agent's perception and utilization of environmental information are not sufficient. In a partially observable wargaming environment, the agent may make decisions only relying on local observation information, ignoring the importance of the global situation, and thus making sub-optimal or even wrong decisions. For example, when judging the overall strategic intention of the enemy, relying only on the action information of local enemy agents may lead to misjudgment. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a multi-agent decision-making method based on the combination of MAPPO and hierarchical attention mechanism. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0005] A multi-agent decision-making method based on the combination of MAPPO and hierarchical attention mechanism includes:

[0006] Each agent perceives the surrounding environment at the current moment to obtain observation state information; the observation state information includes the state information of itself and other agents and terrain information;

[0007] Each agent inputs the observed state information at the current moment into its own policy network with a local attention mechanism, so that the policy network outputs a first action; inputs the first action and the second actions of other agents into the value network with a global attention mechanism, and calculates the reward generated if the agent executes the first action; selects the first action with the maximum reward as the joint action of the agent.

[0008] Each agent executes its corresponding joint action.

[0009] Beneficial effects:

[0010] In the present invention, the agents in the wargame fully integrate the global situation and local observation information with the help of the attention mechanism, and optimize the decision-making by dynamically weighting according to the tasks and the situation, thereby improving the success rate of combat. In previous multi-agent wargames, it was difficult for traditional methods to accurately grasp the local situations of each agent and the overall battlefield situation for effective task allocation. In the present invention, the local attention mechanism of the actor network enables the agent to focus on the key local information around itself, such as the specific actions of nearby enemy agents, the terrain characteristics in a small area, etc., and thus execute the corresponding task actions more accurately based on this information. At the same time, the global attention of the critic network can overview the macroscopic situation of the entire battlefield, such as the overall strength comparison between the enemy and us, the strategic layout, etc. Based on this, the task value of each agent can be reasonably evaluated to achieve a more scientific task allocation. When the battlefield situation changes, such as enemy reinforcements or our strategic adjustments, with the help of this dual-perspective attention mechanism, dynamic adjustments can be made quickly. Moreover, the reinforcement effect of the MAPPO algorithm itself on agent cooperation, in coordination with the attention mechanism, dynamically weights according to the task requirements and the real-time situation to make more optimized decisions and improve the success rate of combat.

[0011] The following will further elaborate on the present invention in conjunction with the drawings and embodiments. Description of the Drawings

[0012] Figure 1 is a schematic flowchart of a multi-agent decision-making method based on the combination of MAPPO and hierarchical attention mechanism provided by the present invention;

[0013] Figure 2 is a schematic diagram of multi-agent decision-making provided by the present invention;

[0014] Figure 3 is a schematic diagram of the policy network provided by the present invention;

[0015] Figure 4 is a schematic diagram of the value network provided by the present invention;

[0016] Figure 5It is a schematic diagram of the program operation of the multi-agent decision-making method based on the combination of MAPPO and hierarchical attention mechanism provided by the present invention. Specific Embodiments

[0017] The present invention will be further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0018] As Figure 1 shown, the present invention provides a multi-agent decision-making method based on the combination of MAPPO and hierarchical attention mechanism, including:

[0019] S100. Each agent perceives the surrounding environment at the current moment to obtain observation state information; the observation state information includes the state information of itself and other agents and terrain information.

[0020] This step clarifies the number and type of agents included in the multi-agent system and the wargaming environment in which they are located. Determine the state space in the environment, covering the global state, the overall force distribution of both sides in the entire wargaming, the occupation situation of important strategic points and other macroscopic information, as well as the local observation state of the agent, the position of enemy agents within a certain range around the agent itself, and the characteristics of the local terrain and other microscopic information. At the same time, define the action space, that is, various actions that the agent can take, such as moving, attacking, etc.

[0021] In the wargaming scenario, the agent can obtain rich state information, which is organized in vector form and input into the neural network. The attention mechanism screens and weights the information according to rules to highlight key content. A reasonable data transmission and sharing strategy is designed among agents to ensure timely information interaction. Experience data is effectively collected and stored in the replay buffer for calculation and model update. By continuously interacting with the environment and updating data, the agent can learn the optimal strategy to adapt to environmental changes.

[0022] S200. Each agent inputs the observation state information at the current moment into its own policy network introducing the local attention mechanism, so that the policy network outputs the first action; inputs the first action and the second actions of other agents into the value network introducing the global attention mechanism, and calculates the reward generated if the agent executes the first action; selects the first action with the maximum reward as the joint action of the agent.

[0023] The present invention designs a reward function for evaluating the behavior and decisions of an agent. The reward function comprehensively considers multiple factors, including obtaining positive rewards for completing task objectives, corresponding reward values for causing damage to the enemy, its own survival, and reward values obtained for complying with rules, etc. To guide the agent to learn better strategies, the reward function sets different weights according to the importance and difficulty of the task. The reward weight for occupying key capture points is relatively high, while the reward weight for ordinary actions is relatively low.

[0024] Combined with Figure 2 and Figure 3 , the present invention integrates the attention mechanism module into the policy network in the network architecture of MAPPO, making the output of the attention mechanism a part of the input of the policy network to help the policy network better select actions. The basic network architecture MLPBase is selected according to the dimension of the observation space, and an external attention layer is inserted after the second layer of the multi-layer perceptron, and the output of the external attention mechanism is used as the input of the next layer. When generating the action probability distribution, the attention mechanism highlights the important information related to the current state, enabling the policy network to more accurately estimate the value of each action. After passing through the network processing flow, the logarithmic probability of the given action and the corresponding action distribution entropy are output. The update of the policy network is based on whether to use maximum gradient norm clipping, obtaining the gradient norm and performing the optimizer update operation.

[0025] Among them, the policy network includes an input layer, a local attention mechanism module, an MLP layer, and an output layer; the local attention mechanism module is used to perform a linear transformation on the input observed state information to obtain a query vector Q 1 , a key vector K 1 , and a value vector V 1 , that is, Q 1 = W Q x, K 1 = W K x, V 1 = W V x, where W Q , W K , W V are learnable parameter matrices. Using the query vector Q 1 , the key vector K 1 to calculate the attention weights, expressed as Using the attention value weights to perform a weighted sum on the value vector V 1 to obtain the attention degree y of each agent 1 = A 1 V 1 ; passing the y 1 through the MLP layer and the output layer to obtain the probability distribution of each action, and selecting the one with the maximum probability as the first action output. In the formula, d k1denotes the dimension of the key vector, and the softmax function is used to normalize the attention weights so that their sum is 1. The input layer of the policy network is used to input the observation states of itself and other agents at the current moment. The MLP layer selects actions in the experience replay buffer according to the y, and then obtains the probability distribution of each action through the output layer.

[0026] The present invention can also use a location-based attention mechanism to replace the hierarchical attention mechanism introduced in the policy network. The former allocates attention weights according to the positional relationship of agents in the environment and is applicable to scenarios where location information is crucial for task execution.

[0027] Reference Figure 4 , the present invention also introduces a global attention mechanism, that is, the multi-head attention mechanism module focuses on features from multiple perspectives in parallel, automatically learning the importance weights of features related to different agents, and more effectively fusing this information for value function estimation. The self-attention mechanism will calculate the similarity between each agent and the other 3 agents, thereby dynamically allocating attention weights. The observation vector of each agent is linearly transformed into Q, K, and V vectors, and linear transformation is performed to calculate the attention weights to obtain the final output vector. By analyzing the global state and local observation state, the multi-head attention mechanism evaluates the importance of the current state, thereby more accurately predicting future rewards. It judges the possible benefits brought by the actions of agents in different states.

[0028] The value network includes an input layer, a central network, and an output layer connected in sequence. The central network includes a backbone network, a global attention mechanism module, and an MLP layer connected in sequence; the backbone network is used to extract features from the input first action and all second actions, and input the extracted features into the global attention mechanism module; the global attention mechanism module adopts a multi-head attention mechanism and is used to perform a linear transformation on the first action and all second actions to obtain a query vector Q 2 , key vector K 2 and value vector V 2 ; using the query vector Q 2 , key vector K 2 calculate the attention weights, denoted as where W i Q , W i K , W i V is the weight matrix of the query, key, and value of the i-th attention head; using the attention value weights to perform a weighted sum on the value vector V 2 to obtain the attention degree y of each agent 2 = A 2 V 2 ; taking the y2 The reward distribution of each first action is obtained through the MLP layer and the output layer; the first action with the largest reward is selected as the joint action.

[0029] The present invention designs a unique hierarchical attention mechanism, which performs coarse-grained and fine-grained attention allocation on information by constructing different levels, effectively focuses on key information in a multi-agent system, avoids agents being interfered by a large amount of irrelevant information, improves the accuracy and efficiency of decision-making, and at the same time pays attention to information interaction between agents, helps agents better understand the intentions and actions of teammates, and realizes efficient cooperation.

[0030] S300, each agent executes its corresponding joint action.

[0031] Each agent obtains the corresponding state when executing its corresponding joint action, and stores the joint action, the corresponding state, and the reward in the experience replay buffer.

[0032] Multiple agents interact with the environment according to the policy generated by the current policy network, and collect experience data during the interaction process, including information such as state, action, reward, and next state. In a wargame, agents perform actions such as moving and attacking on the map, obtain rewards according to the environmental feedback, and record the corresponding state information. Agents select actions in a certain state, the environment gives rewards according to the actions of the agents, and transfers to the next state, and this information will be stored in the experience replay buffer.

[0033] In a wargame scenario, agents can obtain rich state information, and the information is organized in vector form and input into the neural network. The attention mechanism filters and weights the information according to rules to highlight key content. A reasonable data transmission and sharing strategy is designed between agents to ensure timely information interaction. Experience data is effectively collected and stored in the replay buffer for calculation and model update. By continuously interacting with the environment and updating data, agents can learn the optimal strategy to adapt to environmental changes.

[0034] The present invention can also simplify the state representation by extracting key features, only retaining the information directly related to the task to reduce the state space dimension, or adopting state aggregation based on region division, dividing the map into multiple regions and aggregating and representing the information within the regions, which are respectively applicable to the requirements of different wargame scenarios.

[0035] Each agent calculates the advantage function using the experience data in the experience replay buffer, and the advantage function is used to measure the advantage of an agent taking a certain action relative to the average action; the advantage function adopts generalized advantage estimation, which is expressed as:

[0036]

[0037] Among them, γ is the discount factor and λ is the GAE parameter. V w (s t ) represents the estimated value function of the current state.

[0038] In a specific embodiment of the present invention, each agent updates the policy network and the value network using the data in the experience replay buffer after a period of time; among them, the policy network is updated every first number of iterations, the value network is updated every second number of iterations, and the first number of iterations is different from the second number of iterations.

[0039] The policy network updates its parameters by calculating the policy gradient and using an optimization algorithm; the calculation formula of the policy gradient is where π θ (a t |s t ) is the probability distribution of the action generated by the policy network at the current moment; a t is the action at the current moment, and s t is the state at the current moment.

[0040] Randomly sample a small batch of data from the experience replay buffer and use this data to update the policy network and the value network. The update objective of the policy network is to maximize the cumulative reward, and the parameters are updated by calculating the policy gradient and using an optimization algorithm (such as the Adam optimizer). During the update process, in order to prevent the policy update from being too large and causing performance degradation, the proximal policy optimization (PPO) method can be adopted to stabilize the training process by restricting the difference between the new policy and the old policy, that is, both the policy network and the value network use the objective function of the proximal policy optimization method to limit the update process. The objective function of the proximal policy optimization method is expressed as:

[0041]

[0042] where ε is the truncation parameter, represents the probability of executing the action a t under the old policy in the state s t .

[0043] The present invention can also use the A2C algorithm or the TRPO algorithm to replace the MAPPO algorithm. The A2C algorithm has a lower computational complexity and is suitable for situations where the computing resources are limited, the real-time requirement is high, and the requirement for the final performance is not particularly stringent. The TRPO algorithm has stronger convergence guarantees and is suitable for situations where the requirements for model stability and convergence are high and the computing resources permit.

[0044] The value network of the present invention updates parameters by minimizing the mean squared error loss function, and the loss function wherein is the target value function, which is estimated by the Monte Carlo method or the temporal difference method; V(s t , θ) is the state value estimated by the value network.

[0045] The present invention can also adopt a value function approximation based on a decision tree or a value function approximation based on a kernel method to replace the neural network approximation value function, which has advantages in dealing with discrete and structured state spaces.

[0046] In terms of improving the environmental dynamics modeling in environmental modeling and state representation of the present invention, in different embodiments, an implicit dynamic model learning can be adopted to allow the reinforcement learning algorithm to automatically learn the environmental dynamics, or an environmental dynamic prediction module can be introduced to predict future environmental changes, and the two are respectively applicable to environmental situations with different levels of complexity and modeling difficulties.

[0047] Since the current technical environment provides the required technical foundation for implementing the present invention, high-performance computing devices GPU and the deep learning framework PyTorch provide sufficient computing power and convenient development tools for building and training a multi-agent model combining MAPPO and the attention mechanism, and the present invention is easy to implement technically. The MAPPO algorithm has been deeply studied in the field of multi-agent reinforcement learning, and its principle and implementation method have been widely verified; the successful application of the attention mechanism in other fields proves its effectiveness and feasibility in actual operation. These technical conditions enable the effective combination of the two in theory, and by reasonably designing the network architecture and adjusting parameters, a multi-agent system suitable for the wargame scenario can be constructed.

[0048] In the wargame scenario, the agent can obtain rich state information, and the information is organized in vector form and input into the neural network. The attention mechanism filters and weights the information according to rules to highlight key content. A reasonable data transmission and sharing strategy is designed among agents to ensure timely information interaction. Experience data is effectively collected and stored in the replay buffer for calculation and model update. By continuously interacting with the environment and updating data, the agent can learn the optimal strategy to adapt to environmental changes.

[0049] The experiments of the present invention were carried out on a machine running Ubuntu 20.04, with configurations including Python3.8, PyTorch 2.4.1, and CUDA 11.8. The machine is equipped with an Intel(R) Core(TM) i7-14650HX CPU@2.20GHz processor and an NVIDIA GeForce RTX 4070 Laptop GPU.

[0050] ReferenceFigure 5 As shown, after the program starts running, the present invention first parses the command line parameters, determines the device, sets the random seed, and creates the running target. Create a parallel training and evaluation environment based on the configuration. Select the runner according to the policy sharing, and then the training cycle begins. During the training process, data is first collected, the advantage value is calculated and standardized, the PPO update cycle is used, the policy information and loss are calculated, the policy network is updated, and then the value loss is calculated to update the value network. Determine whether the training is finished. If it is finished, close the environment, export the indicators and close the log resources. If the training is not finished, return to data collection.

[0051] The present invention deeply combines the MAPPO algorithm with the attention mechanism to synergistically improve the strategy generation and value assessment process, provide targeted information input for the policy network to generate more appropriate action strategies, enable the value network to more accurately assess the state value, and use the collaboration of the two to effectively handle the complex relationships between multiple agents, helping agents make wise decisions in scenarios such as cooperation, competition, and resource sharing.

[0052] The present invention improves the ability of intelligent agents to adapt to environmental changes in real time by combining an innovative attention mechanism with the MAPPO algorithm, enabling them to quickly perceive and respond to environmental dynamics such as enemy intelligent agent deployment and terrain changes in wargames, adjust decision-making strategies accordingly, and enhance cross-scenario versatility. The attention mechanism automatically adjusts the information processing method according to the characteristics of different scenarios, combined with the performance stability of the MAPPO algorithm, effectively solving the problem of task allocation in various wargame scenarios.

[0053] It is worth noting that the terms "first" and "second" in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0054] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components or steps.

[0055] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. It cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A multi-agent decision-making method based on MAPPO combined with a hierarchical attention mechanism, characterized in that: The multi-agent decision-making method comprises: Each intelligent agent senses the surrounding environment at the current moment and obtains observation state information; the observation state information includes state information of itself and other intelligent agents and terrain information; Each agent inputs the current observation state information into its own strategy network that introduces the local attention mechanism, so that the strategy network outputs the first action; the first action and the second action of other agents are input into the value network that introduces the global attention mechanism, and the reward generated by the agent if the agent executes the first action is calculated; the first action with the largest reward is selected as the joint action of the agent; Each agent performs its corresponding joint action.

2. The multi-agent decision-making method based on combining MAPPO with a hierarchical attention mechanism according to claim 1, characterized in that: The strategy network includes an input layer, a local attention mechanism module, an MLP layer and an output layer; the local attention mechanism module is used to perform a linear transformation on the input observation state information to obtain a query vector Q1, a key vector K1 and a value vector V1; the query vector Q1 and the key vector K1 are used to calculate the attention weight, which is expressed as The attention value weight is used to perform weighted summation on the value vector V1 to obtain the attention degree y1=A1V1 of each agent; the y1 is passed through the MLP layer and the output layer to obtain the probability distribution of each action, and the one with the largest probability is selected as the first action output, where d k1 represents the dimension of the key vector, and the softmax function is used to normalize the attention weights so that their sum is 1.

3. The multi-agent decision-making method based on combining MAPPO with a hierarchical attention mechanism according to claim 1, characterized in that: The input layer of the policy network is used to input the observed states of itself and other agents at the current moment. The MLP layer selects actions in the experience replay buffer according to y, and then obtains the probability distribution of each action through the output layer.

4. The multi-agent decision-making method based on combining MAPPO with a hierarchical attention mechanism according to claim 1, characterized in that: The value network includes an input layer, a central network and an output layer connected in sequence, and the central network includes a backbone network, a global attention mechanism module and an MLP layer connected in sequence; the backbone network is used to extract features of the input first action and all second actions, and input the extracted features into the global attention mechanism module; the global attention mechanism module adopts a multi-head attention mechanism to perform linear transformation on the first action and all second actions to obtain a query vector Q2, a key vector K2 and a value vector V2; the attention weight is calculated using the query vector Q2 and the key vector K2, which is expressed as Where W i Q , W i K , W i V is the weight matrix of the query, key and value of the i-th attention head; the value vector V2 is weightedly summed using the attention value weight to obtain the attention degree y2=A2V2 of each agent; the y2 is passed through the MLP layer and the output layer to obtain the reward distribution of each first action; the first action with the largest reward is selected as the joint action.

5. The multi-agent decision-making method based on combining MAPPO with a hierarchical attention mechanism according to claim 1, characterized in that: Each agent obtains a corresponding state when executing its corresponding joint action, and stores the joint action, the corresponding state and the reward in the experience replay buffer.

6. The multi-agent decision-making method based on combining MAPPO with a hierarchical attention mechanism according to claim 5, characterized in that: Each agent uses the experience data in the experience replay buffer to calculate the advantage function, which is used to measure the advantage of the agent taking a certain action relative to the average action; the advantage function uses a generalized advantage estimate, expressed as: Among them, γ is the discount factor, λ is the GAE parameter, Represents the estimated value of the value function for the current state.

7. The multi-agent decision-making method based on combining MAPPO with a hierarchical attention mechanism according to claim 6, characterized in that: Each agent updates the policy network and the value network using data in the experience replay buffer after a period of time; wherein the policy network is updated at intervals of a first number of iterations, and the value network is updated at intervals of a second number of iterations, and the first number of iterations is different from the second number of iterations.

8. The multi-agent decision-making method based on combining MAPPO with a hierarchical attention mechanism according to claim 7, characterized in that: The policy network calculates the policy gradient and uses the optimization algorithm to update the parameters; the calculation formula of the policy gradient is: Among them, π θ (a t |s t ) is the probability distribution of the action generated by the policy network at the current moment; a t is the action at the current moment, s t It is the state at the current moment.

9. The multi-agent decision-making method based on combining MAPPO with a hierarchical attention mechanism according to claim 8, characterized in that: The value network updates its parameters by minimizing the mean square error loss function. in, is the target value function, estimated by Monte Carlo method or temporal difference method; V(s t ,θ) is the state value estimated by the value network.

10. The multi-agent decision-making method based on combining MAPPO with a hierarchical attention mechanism according to claim 9, characterized in that: The policy network and the value network both use the objective function of the proximal policy optimization method to limit the update process. The objective function of the proximal policy optimization method is expressed as: in, ε is the cutoff parameter, Indicates the state s under the old strategy t Next, perform action a t probability.

Images