Multi-agent implicit collaborative decision-making method based on dark tactical guidance

Abstract

The invention discloses a multi-agent implicit collaborative decision-making method based on dark tactical guidance, which is used for solving the problems of unreliable communication and even complete lack of collaboration in a multi-agent system. A'dark tactical 'concept is introduced to guide behaviors of the intelligent agent. The method comprises three core components: a tactical distribution network, an intelligent agent strategy network and a group of dark tactics, and the three components are synchronously optimized in a training stage; during execution, each agent selects a dark tactics which is most adaptive to the current overall and individual states through the tactics distribution network according to information of other agents which are observed by the agent and can be acquired by the agent; the selected dark tactics are then entered into the policy network along with individual observations of the agents to generate action decisions. According to the method, cooperation and competition among the intelligent agents can be considered, the benefit of the overall strategy of the intelligent agent cluster is maximized, and the cooperation efficiency of the multiple intelligent agents in an uncertain environment is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-agent reinforcement learning, and in particular relates to a multi-agent implicit collaborative decision-making method based on dark tactic guidance. Background Art

[0002] Multi-Agent Reinforcement Learning (MARL) is an extension of reinforcement learning (RL) that focuses on scenarios where multiple agents learn and make decisions in a shared environment. Multi-Agent Reinforcement Learning is a cutting-edge technology that has been widely used in many fields in recent years, such as self-driving cars, drone formation flying, robot collaboration, multi-player games, and other technical fields. However, although existing MARL methods have made significant progress in theoretical research and simulation experiments, they still face many challenges in practical applications, especially in environments with unstable communications and high uncertainty. These difficulties limit the performance of MARL in more complex and dynamic scenarios.

[0003] Traditional multi-agent reinforcement learning methods mostly rely on the centralized training and decentralized execution (CTDE) framework. In the training phase, the information of all agents is used for centralized learning; but in the execution phase, each agent can only make independent decisions based on its own local observations. This problem of local observation is particularly prominent in complex and dynamic environments. Agents often make suboptimal decisions because they cannot obtain global information.

[0004] To address this limitation, communication mechanisms have been introduced in recent years, allowing agents to make up for the lack of observation by sharing information during the execution phase. Such methods usually rely on attention mechanisms or graph neural networks to integrate information from teammates, and most of them are based on the premise that the communication environment is ideal, continuous, and delay-free. However, in reality, problems such as communication interruptions, delays, or noise often occur, resulting in unstable information transmission, which in turn affects the decision-making of agents. Therefore, these methods have significant limitations in practical applications. In addition, the instability of communication causes the agent to receive inconsistent information from teammates at different time steps, disrupting the learning process of the policy network, which further leads to unstable decisions. Studies have shown that in an environment with uncertain communication, the performance of agents may be significantly reduced.

[0005] like Figure 1As shown in the figure, preliminary experimental results further demonstrate the impact of this communication instability on existing methods. Based on the QMIX framework, the researchers extended a module that enables the agent to simultaneously perceive its own information and messages from teammates. The system performed best when communication was completely unimpeded (QMIX_100%), verifying that sharing teammate information can improve decision quality. However, the performance of QMIX_40% (representing moderate communication uncertainty) was even worse than the baseline without communication (QMIX_0%). The researchers believe that the reason lies in the high sensitivity of the policy network to input instability. The set of teammate information received at different time steps may fluctuate greatly. For example, information from teammates 1, 2, and 3 is received at t=1, and only information from teammates 2 and 4 is received at t=2. This fluctuation disrupts the learning process and leads to suboptimal decisions. Therefore, in uncertain collaborative MARL environments, more robust methods need to be developed to cope with the challenges brought by communication uncertainty and input fluctuations. Summary of the invention

[0006] To solve the above problems, the purpose of the present invention is to provide a multi-agent implicit collaborative decision-making method based on dark tactics guidance, which uses a dark tactics learning (Learning Latent Tactics, LLT) framework based on deep learning to deal with multi-agent implicit collaborative tasks under uncertainty. The dark tactics learning framework is a multi-stage negotiation framework based on tactic generation and action decision-making. By introducing dark tactics, it helps the agent to more effectively deal with the problem of missing observation information of teammates in the decision-making process, thereby improving the robustness of the policy network and the stability of decision-making; the dark tactics learning framework is divided into a negotiation stage and an action stage, aiming to improve the cooperation efficiency of the multi-agent system in an uncertain environment.

[0007] In order to achieve the above-mentioned invention object, the present invention adopts the following technical scheme:

[0008] A multi-agent implicit collaborative decision-making method based on dark tactics guidance includes the following steps:

[0009] S1. The partially observable Markov decision process POMDP is used as a model. Each agent obtains its own local observation information and aggregates the observation information from other agents through the attention mechanism to generate a comprehensive representation that reflects the key features of the overall environment and the status of teammates. This comprehensive representation is input into the tactic allocation network to generate the Q value of the dark tactic. In the training phase, the agent selects the dark tactic through the ∈-greedy strategy. In the testing phase, the agent always selects the dark tactic with the highest Q value.

[0010] S2, each agent combines its own observation information and the selected dark tactics in step S1 to generate the Q value of the action through the agent strategy network; in the training phase, the agent selects executable actions through the ∈-greedy strategy; in the testing phase, the agent always selects the executable action with the highest Q value;

[0011] S3. The dark tactical learning framework jointly optimizes the parameters of the tactical allocation network in step S1 and the agent strategy network in step S2 through a two-layer hybrid network, thereby optimizing the tactical Q value in step S1 and the action Q value in step S2; the first layer of the hybrid network is used to optimize the global tactical Q value, and the second layer of the hybrid network is used to optimize the global action Q value; the entire dark tactical learning framework performs efficient training and optimization of the model by minimizing the joint loss function.

[0012] Furthermore, in the above step S1, the model of the partially observable Markov decision process POMDP is defined by the following tuple:

[0013] G=<S,A,P,R,Z,m,γ>

[0014] Among them, S is the state space, which represents the set of all possible states; A is the action space, which represents the set of all possible actions that the agent can take in each state; P is the state transition function, which represents the probability of state change after executing an action; R is the reward function, which is the immediate reward for taking an action in a given state; Z is the observation space, which is the set of observations that the agent can receive; n is the observation probability, which represents the probability of obtaining a certain observation after performing a specific action in a specific state; γ is the discount factor, which represents the discount on future rewards and is used to balance the importance of current rewards and future rewards;

[0015] Suppose there is a set X = {1, 2, ..., k} of k dark tactics, each tactic corresponds to a learnable vector in m-dimensional space At each time step t, each agent selects a tactic x from the tactic set i ∈X, used to guide one’s own action selection a i ∈A; the formed joint tactics x joint ∈X n and joint action a joint ∈A n According to the state transfer function P(s'|s,a joint ) leads to a new global state s' and obtains a shared reward r = R(s,a) joint ).

[0016] Furthermore, the above step S1 includes the following sub-steps:

[0017] S1.1. Obtaining individual observation information

[0018] At each time step t, agent i obtains its local observation information z i , and can also receive some observation information z from other agents -i ; Combine the current observation of the agent and the observation of other agents, generate the Q value of the dark tactic through the tactic allocation network, and select the dark tactic according to the Q value; expressed as:

[0019]

[0020] Among them, x i is the dark tactic vector of agent i; z i is the observation information of agent i itself, z -i is the observation information from other agents; j is the observation information of agent j itself, z -j is the observation information from other agents; for (i,j)∈{1,…,n}, there exists |z -i |≠|z -j |;

[0021] S1.2. Aggregate information using attention mechanism

[0022] For agent i, its own observation information z i As the query vector z i W Q , and the observation information z from teammate j j ∈z -i , respectively as key vector z j W K Sum value vector z j W V ;

[0023] The attention score is calculated by the dot product of the query vector and the key vector, and obtained after scaling and Softmax operation, expressed as:

[0024]

[0025] in, is a learnable weight matrix, the aggregate information vector c of agent i i It is composed of its own information and attention-weighted teammates’ information.

[0026]

[0027] in, is the observation information aggregated by agent i;

[0028] S1.3. Generate Q value of dark tactics

[0029] (1) At each time step t, the gated recurrent unit GRU network generates the hidden state of the next time step through the following update formula:

[0030] h′ i =GRU(MLP(c i ),h i )

[0031] in, is the current GRU hidden state; is the hidden state at the next time step;

[0032] (2) After obtaining the updated hidden state h′ i After that, the Q value of the dark strategy is Q x Calculated using the following formula:

[0033] Q x (z i ,z -i ,.)=h′ i W x

[0034] in, is a learnable parameter, k represents the number of tactics, and the hidden state h′ i Mapped to the tactical Q-value space, each element in this space represents the Q-value of the agent's corresponding tactic;

[0035] S1.4. Choose dark tactics

[0036] During the training phase, the tactic with the highest current Q value is selected with probability 1-∈, and other tactics are randomly selected with probability ∈. The formula is as follows:

[0037]

[0038] During the testing phase, the agent always chooses the tactic with the highest Q value.

[0039] Furthermore, the above step S2 includes the following sub-steps:

[0040] S2.1. Generating Q-values ​​of actions based on dark tactics

[0041] Combined with the current observation and dark tactics of the agent, the Q value of the executable action is generated through the agent strategy network, and the executable action is selected according to the Q value; it is expressed as:

[0042]

[0043] Among them, a i is the executable action of agent i; z i is the observation information of agent i itself, x iis the dark tactic vector of agent i; the operation is:

[0044] S2.11. The observation information z of the intelligent agent i With dark tactics vector x i To splice,

[0045] [z i ;x i ]

[0046] Among them, the splicing operation [z i ;x i ] is to combine the observation information with the dark tactics to form a new input vector;

[0047] S2.12. The concatenated input vector is passed into a multi-layer perceptron (MLP) network, through which high-level feature representations are extracted:

[0048] MLP([z i ;x i ])

[0049] S2.13, the information extracted in step S2.12 is passed to the gated recurrent single GRU network, which updates the hidden state based on the current and historical information Updated hidden state Obtained by the following formula:

[0050]

[0051] in, represents the hidden state of the previous time step; is the updated hidden state at the current time step;

[0052] S2.14. Based on hidden state Generate the Q value of each action that the agent can currently perform through linear transformation:

[0053]

[0054] The linear layer transforms the hidden state Mapped to the action Q-value space, each element in the space represents the Q-value of the corresponding action of the agent; each Q-value corresponds to the evaluation of the corresponding action taken by the agent in the current state, indicating the expected return of the action;

[0055] S2.2. Select executable actions

[0056] During the training phase, the agent randomly selects an action with probability ∈ and selects the action with the highest Q value with probability 1-∈, as follows:

[0057]

[0058] Among them, A represents the set of all possible actions; Q a (z i ,a|x i ) represents the Q value of action a;

[0059] During the testing phase, the agent chooses the action with the highest Q value.

[0060] Furthermore, the above step S3 includes the following sub-steps:

[0061] S3.1. Global Tactical Q-Value Optimization

[0062] The tactical Q-value of each agent i The first layer of hybrid network is aggregated to generate the global tactical Q value Among them, s represents the global state, x joint ={x i ,x -i} represents the joint strategy of all agents; the global strategy Q value Calculated by the following formula:

[0063]

[0064] Among them, f θ′ It is the first hybrid network trained to integrate the individual tactical Q-values;

[0065] During the training process, the global tactical Q value is optimized through the temporal difference loss function, so that each agent can make better tactical decisions in the next time step; the loss function The definition is as follows:

[0066]

[0067] Where r is the reward at the current time step; γ is the discount factor; s′ and x joint ′ is the state and joint strategy of the next time step; is the global tactical Q-value of the next time step estimated by the target network;

[0068] By minimizing the loss function Optimize the dark tactical Q-values ​​generated by the tactical allocation network;

[0069] S3.2. Global Action Q-Value Optimization

[0070] The Q value of each action of agent i Summarize through the second layer of hybrid network to generate the global action Q value Among them, a joint ={a i ,a-i} represents the joint action of all agents at the current time step, and the action is selected based on their respective dark tactics x; the global action Q value The calculation formula is as follows:

[0071]

[0072] Among them, f φ′ is a second trainable hybrid network that computes the global Q-value of the joint action;

[0073] Optimization is based on TD loss function:

[0074]

[0075] in, is the global action Q value of the next time step estimated by the target network;

[0076] By minimizing the loss function Optimize the joint action Q-value generated by the agent policy network;

[0077] S3.3. Total loss function

[0078] The entire training process uses a comprehensive total loss function Used to optimize tactical Q-value and action Q-value simultaneously; this loss function converts tactical Q-value loss and action Q-value loss Perform weighted summation, the formula is as follows:

[0079]

[0080] Among them, Θ = {θ, θ′, φ, φ′} is the set of all parameters to be optimized;

[0081] By minimizing the total loss Parameters of the tactical allocation network and the agent policy network are optimized simultaneously.

[0082] The above-mentioned tactical allocation network and agent strategy network are both neural networks composed of multi-layer perceptron MLP and gated recurrent unit GRU. The parameters to be trained and the functions to be completed are different.

[0083] Due to the adoption of the above-mentioned technical solution, the present invention has the following advantages:

[0084] The present invention is based on a multi-agent implicit collaborative decision-making method guided by dark tactics. It constructs an agent implicit collaborative training framework by optimizing the policy network and the tactical decision network. During the execution phase of the framework, the agent will make tactical choices based on its own observation information and the observation information of its teammates, and perform specific actions according to the policy network and the selected tactics. Even in uncertain collaborative scenarios, that is, when the observation information of some or all teammates is temporarily missing, it can still work normally, and is suitable for open collaborative environments in reality.

[0085] The present invention is based on a multi-agent implicit collaborative decision-making method guided by dark tactics. Each agent uses the attention mechanism to dynamically aggregate observation information from other agents. The agent can perform weighted processing on the received information and focus on key data related to the current task, thereby improving the effectiveness and robustness of the decision. The agent can achieve flexible aggregation between different observation data through the attention mechanism, thereby enhancing the decision-making ability of the system under uncertain conditions. In the training stage, the agent selects dark tactics and executable actions through the θ-greedy strategy to ensure a balance between exploration and utilization. The dark tactics learning framework can take into account the collaboration and competition between agents and maximize the benefits of the overall strategy of the agent cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is a graph showing the experimental results of QMIX in the prior art in StarCraft II;

[0087] Figure 2 It is a flow chart of the multi-agent implicit collaborative decision-making method based on dark tactics guidance of the present invention;

[0088] Figure 3 It is a structural diagram of the dark tactics learning framework in the multi-agent implicit collaborative decision-making method based on dark tactics guidance of the present invention;

[0089] Figure 4 It is a visualization diagram of tactical allocation in one iteration of SMAC scenario 6h_vs_8z in one embodiment of the present invention. DETAILED DESCRIPTION

[0090] The technical solution of the present invention is further described in detail below through the drawings and embodiments.

[0091] like Figure 2As shown, the present invention uses the partially observable Markov decision process POMDP as a model; each agent obtains its own local observation information, and aggregates the observation information from other agents through the attention mechanism to generate a comprehensive representation to reflect the key features of the overall environment and the status of teammates. This comprehensive representation is input into the tactic allocation network to select the dark tactic vector. In the training phase, the agent selects dark tactics and executable actions through the ∈-greedy strategy to ensure a balance between exploration and utilization;

[0092] The model of the above partially observable Markov decision process POMDP is defined by the following tuple:

[0093] G=<S,A,P,R,Z,n,γ>

[0094] Among them, S is the state space, which represents the set of all possible states; A is the action space, which represents the set of all possible actions that the agent can take in each state; P is the state transition function, which represents the probability of state change after executing an action; R is the reward function, which is the immediate reward for taking an action in a given state; Z is the observation space, which is the set of observations that the agent can receive; n is the observation probability, which represents the probability of obtaining a certain observation after performing a specific action in a specific state; γ is the discount factor, which represents the discount on future rewards and is used to balance the importance of current rewards and future rewards;

[0095] Suppose there is a set X = {1, 2, ..., k} of k dark tactics, each tactic corresponds to a learnable vector in m-dimensional space At each time step t, each agent selects a tactic x from the tactic set i ∈X, used to guide one’s own action selection a i ∈A; the formed joint tactics x joint ∈X n and joint action a joint ∈A n According to the state transfer function P(s'|s,a joint ) leads to a new global state s' and obtains a shared reward r = R(s,α joint ). The purpose of this invention is to train an agent to achieve an optimal joint action representation by dark tactics to maximize the expected global reward.

[0096] like Figures 2 to 4 As shown, a multi-agent implicit collaborative decision-making method based on dark tactics guidance includes the following steps:

[0097] S1, Negotiation Phase

[0098] S1.1. Obtaining individual observation information

[0099] At each time step t, agent i obtains its local observation information z i , and can also receive some observation information z from other agents -i ; Combine the current observation of the agent and the observation of other agents, generate the Q value of the dark tactic through the tactic allocation network, and select the dark tactic according to the Q value; the observation information includes the current state of the agent, environmental feedback, and key data related to the task; expressed as:

[0100]

[0101] Among them, x i is the dark tactic vector of agent i; z i is the observation information of agent i itself, z -i is the observation information from other agents; j is the observation information of agent j itself, z -j is the observation information from other agents; for (i,j)∈{1,…,n}, there exists |z -i |≠|z -j |, indicating that the amount and content of information received by each agent from other agents are not completely consistent, thus exacerbating the information asymmetry of the system; for example, in a certain period of time, agent i receives the observation information of all other agents, while agent j only receives the observation information of some agents due to the unstable communication environment, which will lead to the problem of inconsistent input length. In order to manage this variable length input, the attention mechanism is used, so that the agent can achieve flexible aggregation between different observation data, enhancing the decision-making ability of the system under uncertain conditions;

[0102] S1.2. Aggregate information using attention mechanism

[0103] Since the amount of teammate observation information received by agent i may be inconsistent with that of agent j, a scaled dot-product attention mechanism is introduced to dynamically aggregate this information; the attention mechanism can not only handle variable-length inputs, but also automatically assign appropriate weights to different information and automatically extract the key data that is most important for the current decision;

[0104] For agent i, its own observation information z i As the query vector z i W Q , and the observation information z from teammate j j ∈z -i , respectively as key vector z j W K Sum value vector z j W V;

[0105] The attention score is calculated by the dot product of the query vector and the key vector, and obtained after scaling and Softmax operation, expressed as:

[0106]

[0107] in, is a learnable weight matrix, the aggregate information vector c of agent i i It is composed of its own information and attention-weighted teammates’ information.

[0108]

[0109] in, is the observation information aggregated by agent i; the attention mechanism ensures that even if the teammate information is missing, the agent can still make decisions based on its own observations. In addition, the learnable weight matrix W Q ,W K ,W V Enable the system to adaptively optimize during training and continuously improve the effect of multi-agent collaboration;

[0110] S1.3. Generate Q value of dark tactics

[0111] The aggregate information c in step S1.2 i Input into the tactical allocation network to generate the agent's dark tactic vector x i ; It can not only capture the characteristics of the current state of the agent, but also maintain the temporal correlation between time steps through the gated recurrent unit GRU, enhancing the continuity and robustness of tactical decision-making; the operation is:

[0112] (1) At each time step t, the gated recurrent unit GRU network generates the hidden state of the next time step through the following update formula:

[0113] h′ i =GRU(MLP(c i ),h i )

[0114] in, is the current GRU hidden state; is the hidden state at the next time step;

[0115] (2) After obtaining the updated hidden state h′ i After that, the Q value of the dark strategy is Q x Calculated using the following formula:

[0116] Q x (z i ,z -i,.) = h′ i W x

[0117] in, is a learnable parameter, k represents the number of tactics, and the hidden state h′ i Mapped to the tactical Q-value space, each element in this space represents the Q-value of the agent's corresponding tactic;

[0118] S1.4. Choose dark tactics

[0119] In the process of tactic selection, the ∈-greedy strategy is used to ensure the balance between exploration and utilization; in the training phase, the tactic with the highest current Q value is selected with probability 1-∈, and other tactics are randomly selected with probability ∈ to ensure sufficient exploration; the formula is as follows:

[0120]

[0121] During the testing phase, the agent always chooses the tactic with the highest Q value to ensure the optimality of the decision;

[0122] S2. Action Phase

[0123] S2.1. Generating Q-values ​​of actions based on dark tactics

[0124] In the action phase, each agent’s action decision depends not only on its own observation information z i , which is also affected by the dark tactic vector x generated in the negotiation phase i The core goal of the action phase is that each agent combines its own observation information and selected dark tactics, generates the Q value of the executable action through the agent strategy network, and selects the executable action according to the Q value; it is expressed as:

[0125]

[0126] Among them, a i is the executable action of agent i; z i is the observation information of agent i itself, x i is the dark tactic vector of agent i; the operation is:

[0127] S2.11. The observation information z of the intelligent agent i With dark tactics vector x i To splice,

[0128] [z i ;x i ]

[0129] Among them, the splicing operation [z i ;x i] is to combine the observation information with the dark tactics to form a new input vector;

[0130] S2.12. The concatenated input vector is passed into a multi-layer perceptron (MLP) network, through which high-level feature representations are extracted:

[0131] MLP([z i ;x i ])

[0132] S2.13, the information extracted in step S2.12 is passed to the gated recurrent single GRU network, which updates the hidden state based on the current and historical information Updated hidden state Obtained by the following formula:

[0133]

[0134] in, represents the hidden state of the previous time step; is the updated hidden state at the current time step;

[0135] S2.14. Based on hidden state Generate the Q value of each action that the agent can currently perform through linear transformation:

[0136]

[0137] The linear layer transforms the hidden state Mapped to the action Q-value space, each element in the space represents the Q-value of the corresponding action of the agent; each Q-value corresponds to the evaluation of the corresponding action taken by the agent in the current state, indicating the expected return of the action;

[0138] S2.2. Select executable actions

[0139] After obtaining the Q value corresponding to each action, the agent uses the ∈-greedy strategy to select executable actions based on these values. Specifically, during the training phase, the agent randomly selects an action with probability ∈ to ensure that different strategy spaces are explored to avoid falling into local optimal solutions; the action with the highest Q value is selected with probability 1-∈, as shown in the following formula:

[0140]

[0141] Among them, A represents the set of all possible actions; Q a (Z i ,a|x i ) represents the Q value of action a;

[0142] During the testing phase, the agent selects the action with the highest Q value to ensure the optimality and stability of the strategy.

[0143] In this way, the action phase combines dark tactics and observation information to generate high-quality action decisions, ensuring that the agent can demonstrate robust decision-making capabilities in complex task environments;

[0144] S3, training and optimization

[0145] The dark tactic learning framework is designed with two layers of QMIX-style hybrid networks, which are used to optimize the joint strategy of tactical selection and action decision-making respectively; the two layers of hybrid networks are used to optimize the Q value of different stages respectively: the first layer of hybrid network is used to optimize the global tactical Q value in the negotiation stage, and the second layer of hybrid network optimizes the global action Q value in the action stage;

[0146] S3.1. Global Tactical Q-Value Optimization

[0147] In the negotiation phase, the tactical Q value of each agent i is The first layer of hybrid network is aggregated to generate the global tactical Q value Among them, s represents the global state, x joint ={x i ,x -i} represents the joint strategy of all agents; the global strategy Q value Calculated by the following formula:

[0148]

[0149] Among them, f θ′ It is the first trainable hybrid network for integrating individual tactical Q-values;

[0150] During the training process, the global tactical Q value is optimized through the temporal difference (TD) loss function, so that each agent can make better tactical decisions in the next time step; the loss function is defined as follows:

[0151]

[0152] Where r is the reward at the current time step; γ is the discount factor; s′ and x joint ′ is the state and joint strategy of the next time step; is the global tactical Q-value of the next time step estimated by the target network;

[0153] By minimizing the loss function Optimize the dark tactic Q value generated by the tactic allocation network in the negotiation phase, thereby improving the effectiveness of each agent's tactic selection;

[0154] S3.2. Global Action Q-Value Optimization

[0155] In the action phase, the Q value of each agent i’s action Summarize through the second layer of hybrid network to generate the global action Q value where a joint ={a i ,a -i} represents the joint action of all agents at the current time step, and the action is selected based on their respective dark tactics x; the global action Q value The calculation formula is as follows:

[0156]

[0157] Among them, f φ′ is a second trainable hybrid network that computes the global Q-value of the joint action;

[0158] Similar to the optimization of tactical Q value, the optimization of action phase is also based on TD loss function:

[0159]

[0160] in, is the global action Q value of the next time step estimated by the target network;

[0161] By minimizing the loss function , optimize the joint action Q value generated by the agent strategy network in the action phase to ensure that the agent can take the best action based on tactical selection;

[0162] like Figure 3 As shown in the figure, in one round in the 6h_vs_8z scenario, different agents perform different actions according to the selected tactics, optimizing the behavior of the entire team. For example, at t=2, the agents disperse according to the learned tactics to maximize the attack output against the enemy; while at t=12, some agents with low health (such as agents 4, 5, and 6) choose the second tactic to retreat to avoid excessive damage, while other agents (such as agents 1, 2, and 3) choose the third tactic to go to the front line and bear the damage. This dynamic optimization of the action Q value allows each agent to flexibly adjust its behavior based on the environment, and ultimately maximizes the damage output and ensures its own survival at t=21;

[0163] S3.4. Total loss function

[0164] The entire training process uses a comprehensive total loss function It is used to simultaneously optimize the tactical Q value in the negotiation phase and the action Q value in the action phase; this loss function converts the tactical Q value loss and action Q-value loss Perform weighted summation, the formula is as follows:

[0165]

[0166] Among them, Θ = {θ, θ′, φ, φ′} is the set of all parameters to be optimized;

[0167] By minimizing the total loss It can simultaneously optimize the parameters of the tactical allocation network in the negotiation phase and the agent strategy network in the action phase, so that the agent can make more reasonable tactical and action decisions under the joint strategy.

[0168] In summary, the present invention is based on the multi-agent implicit collaborative decision-making method guided by dark tactics. It optimizes the global tactical Q value and the global action Q value respectively through a two-layer hybrid network. The dark tactic learning framework is continuously adjusted and improved on the basis of the joint strategy, thereby improving the overall collaboration and decision-making efficiency of the intelligent agents.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-agent implicit collaborative decision-making method based on dark tactics guidance, characterized by: It includes the following steps: S1. The partially observable Markov decision process POMDP is used as a model. Each agent obtains its own local observation information and aggregates the observation information from other agents through the attention mechanism to generate a comprehensive representation that reflects the key features of the overall environment and the status of teammates. This comprehensive representation is input into the tactic allocation network to generate the Q value of the dark tactic. In the training phase, the agent selects the dark tactic through the ∈-greedy strategy. In the testing phase, the agent always selects the dark tactic with the highest Q value. S2, each agent combines its own observation information and the selected dark tactics in step S1 to generate the Q value of the action through the agent strategy network; in the training phase, the agent selects executable actions through the ∈-greedy strategy; in the testing phase, the agent always selects the executable action with the highest Q value; S3. The dark tactical learning framework jointly optimizes the parameters of the tactical allocation network in step S1 and the agent strategy network in step S2 through a two-layer hybrid network, thereby optimizing the tactical Q value in step S1 and the action Q value in step S2; the first layer of the hybrid network is used to optimize the global tactical Q value, and the second layer of the hybrid network is used to optimize the global action Q value; the entire dark tactical learning framework performs efficient training and optimization of the model by minimizing the joint loss function.

2. The multi-agent implicit collaborative decision-making method based on dark tactic guidance according to claim 1 is characterized by: step In S1, the model of the partially observable Markov decision process POMDP is defined by the following tuple: G=<S,A,P,R,Z,n,γ> Among them, S is the state space, which represents the set of all possible states; A is the action space, which represents the set of all possible actions that the agent can take in each state; R is the state transition function, which represents the probability of state change after executing an action; P is the reward function, which is the immediate reward for taking an action in a given state; Z is the observation space, which is the set of observations that the agent can receive; n is the observation probability, which represents the probability of obtaining a certain observation after performing a specific action in a specific state; γ is the discount factor, which represents the discount on future rewards and is used to balance the importance of current rewards and future rewards; Suppose there is a set X = {1, 2, ..., k} of k dark tactics, each tactic corresponds to a learnable vector in m-dimensional space At each time step t, each agent selects a tactic x from the tactic set i ∈X, used to guide one’s own action selection a i ∈A; the formed joint tactics x joint ∈X n and joint action a joint ∈A n According to the state transfer function P(s′|s, a joint ) leads to a new global state s′ and obtains a shared reward r = R(s, a joint ).

3. The multi-agent implicit collaborative decision-making method based on dark tactic guidance according to claim 1 is characterized by: The step S1 comprises the following sub-steps: S1.

1. Obtaining individual observation information At each time step t, agent i obtains its local observation information z i , and can also receive some observation information z from other agents -i ; Combine the current observation of the agent and the observation of other agents, generate the Q value of the dark tactic through the tactic allocation network, and select the dark tactic according to the Q value; expressed as: Among them, x i is the dark tactic vector of agent i; z i is the observation information of agent i itself, z -i is the observation information from other agents; j is the observation information of agent j itself, z -j is the observation information from other agents; for (i, j)∈{1,...,n}, there exists |z -i |≠|z -j |; S1.

2. Aggregate information using attention mechanism For agent i, its own observation information z i As the query vector z i W Q , and the observation information z from teammate j j ∈z -i , respectively as key vector z j W K Sum value vector z j W V ; The attention score is calculated by the dot product of the query vector and the key vector, and obtained after scaling and Softmax operation, expressed as: Among them, W Q , W K , is a learnable weight matrix, the aggregate information vector c of agent i i It is composed of its own information and attention-weighted teammates’ information. in, is the observation information aggregated by agent i; S1.

3. Generate Q value of dark tactics (1) At each time step t, the gated recurrent unit GRU network generates the hidden state of the next time step through the following update formula: h′ i =GRU(MLP(c i ),h i ) in, is the current GRU hidden state; is the hidden state at the next time step; (2) After obtaining the updated hidden state h′ i After that, the Q value of the dark strategy is Q x Calculated using the following formula: Q x (z i ,z -i ,.)=h′ i W x in, is a learnable parameter, k represents the number of tactics, and the hidden state h′ i Mapped to the tactical Q-value space, each element in this space represents the Q-value of the agent's corresponding tactic; S1.

4. Choose dark tactics In the training phase, the tactic with the highest current Q value is selected with probability 1=∈, and other tactics are randomly selected with probability ∈. The formula is as follows: During the testing phase, the agent always chooses the tactic with the highest Q value.

4. The multi-agent implicit collaborative decision-making method based on dark tactic guidance according to claim 1 is characterized by: The step S2 comprises the following sub-steps: S2.

1. Generating Q-values ​​of actions based on dark tactics Combined with the current observation and dark tactics of the agent, the Q value of the executable action is generated through the agent strategy network, and the executable action is selected according to the Q value; it is expressed as: Among them, a i is the executable action of agent i; z i is the observation information of agent i itself, x i is the dark tactic vector of agent i; the operation is: S2.

11. The observation information z of the intelligent agent i With dark tactics vector x i To splice, [z i ;x i ] Among them, the splicing operation [z i ;x i ] is to combine the observation information with the dark tactics to form a new input vector; S2.

12. The concatenated input vector is passed into a multi-layer perceptron (MLP) network, through which a high-level feature representation is extracted: MLP([from i ;x i ]) S2.13, the information extracted in step S2.12 is passed to the gated recurrent single GRU network, which updates the hidden state based on the current and historical information Updated hidden state Obtained by the following formula: in, represents the hidden state of the previous time step; is the updated hidden state at the current time step; S2.

14. Based on hidden state Generate the Q value of each action that the agent can currently perform through linear transformation: The linear layer transforms the hidden state Mapped to the action Q-value space, each element in the space represents the Q-value of the corresponding action of the agent; each Q-value corresponds to the evaluation of the corresponding action taken by the agent in the current state, indicating the expected return of the action; S2.

2. Select executable actions During the training phase, the agent randomly selects an action with probability ∈ and selects the action with the highest Q value with probability 1-∈, as follows: Among them, A represents the set of all possible actions; Q a (z i ,a|x i ) represents the Q value of action a; During the testing phase, the agent chooses the action with the highest Q value.

5. The multi-agent implicit collaborative decision-making method based on dark tactic guidance according to claim 1 is characterized by: The step S3 comprises the following sub-steps: S3.

1. Global Tactical Q-Value Optimization The tactical Q-value of each agent i The global tactical Q value is generated through the first layer of hybrid network aggregation Among them, s represents the global state, x joint ={x j ,x -i } represents the joint strategy of all agents; the global strategy Q value Calculated by the following formula: Among them, f θ′ It is the first trainable hybrid network for integrating individual tactical Q-values; During the training process, the global tactical Q value is optimized through the temporal difference loss function, so that each agent can make better tactical decisions in the next time step; the loss function The definition is as follows: Where r is the reward at the current time step; γ is the discount factor; s′ and x joint ′ is the state and joint strategy of the next time step; is the global tactical Q-value of the next time step estimated by the target network; By minimizing the loss function Optimize the dark tactical Q-values ​​generated by the tactical allocation network; S3.

2. Global Action Q Value Optimization The Q value of each action of agent i Summarize through the second layer of hybrid network to generate the global action Q value Among them, a joint ={a i , a -i } represents the joint action of all agents at the current time step, and the action is selected based on their respective dark tactics x; the global action Q value The calculation formula is as follows: Among them, f φ′ is a second trainable hybrid network that computes the global Q-value of the joint action; Optimization is based on TD loss function: in, is the global action Q value of the next time step estimated by the target network; By minimizing the loss function Optimize the joint action Q-value generated by the agent policy network; S3.

3. Total loss function The entire training process uses a comprehensive total loss function Used to optimize tactical Q-value and action Q-value simultaneously; this loss function converts tactical Q-value loss and action Q-value loss Perform weighted summation, the formula is as follows: Wherein, Θ = {θ, θ′, φ, φ′} is the set of all parameters to be optimized; By minimizing the total loss Parameters of the tactical allocation network and the agent policy network are optimized simultaneously.

Images