Multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics

Abstract

The present invention proposes a multi-agent reinforcement learning method based on the dynamic generation of group confrontation tactics. By introducing a dynamic generation method when solving group confrontation using the multi-agent reinforcement learning method, it can better understand the scene, overcome the technical challenge of understanding complex scene tasks, and fully understand the scene tasks through tactics; by introducing a position heat map, it effectively summarizes and records the action trajectory of the agent in group confrontation, and forms a probability distribution form based on the association with success. By using a self-imitation learning mechanism based on the position heat map, the agent can tend to fully explore the environment in the early stage of the task with macro actions, and concentrate the macro actions on the key positions explored and continuously optimize them in the middle and late stages of the task.

Description

Technical Field

[0001] The present invention belongs to the field of deep reinforcement learning technology, and specifically relates to a multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics. Background Art

[0002] The problem of group confrontation has important research significance in real life. For example, in unmanned equipment (drones and unmanned ships) systems, multiple unmanned equipment form a multi-agent system. The system needs to rely on certain algorithms to complete the exploration of the environment, and ultimately be able to navigate and act in the environment and complete confrontation tasks with possible opponents.

[0003] For multi-agent systems composed of unmanned vehicles, there are two main approaches: knowledge-based (i.e., fixed-rule approaches) and data-driven approaches. The former relies on human experience and requires the use of classical control algorithms to design the multi-agent system's actions. This requires the support of expert knowledge and lacks the ability for dynamic optimization. The latter, data-driven approaches, primarily utilize deep reinforcement learning algorithms to control the multi-agent system. However, existing multi-agent reinforcement learning methods rely on allocating credit among multiple agents based on a single reward signal. This means they focus on micro-level decisions about specific actions within the multi-agent system, lacking the macro-level consideration of the entire combat process, as human experts do. All decisions are data-driven, lacking long-term tactical guidance at the macro level. Therefore, they are not well suited for complex group combat situations.

[0004] Most current multi-agent deep reinforcement learning methods fail to consider the tactical nature of group confrontations, focusing solely on decision-making and optimization of specific actions. Consequently, these methods fail to fully understand the key aspects of the task, systematically and hierarchically explore the environment, and ultimately guide the agents in completing cooperative confrontation tasks. While a few methods utilize communication or hierarchical modeling information to assist multi-agent decision-making, the duration of these high-level information relies on predetermined human experience and must be consistent and unchangeable across agents. Consequently, they lack the flexibility to adapt to diverse environments.

[0005] In view of the current multi-agent scenario of group confrontation, the main technical challenges faced by existing technologies are as follows:

[0006] Technical Challenge 1: Complex scenario tasks are difficult to understand. Existing methods make decisions from the perspective of specific actions of multiple agents, and therefore cannot fully understand the scenario tasks. This is especially true in difficult adversarial scenarios and tasks with sparse rewards. The reward function is sparse, and if the agent cannot understand the task from a macro perspective, it will be limited to the local optimal solution.

[0007] Technical Challenge 2: Insufficient utilization of spatiotemporal information. Spatiotemporal information in group confrontation scenarios is key to understanding and completing tasks. However, how to summarize more information based on trajectory experience in existing exploration rounds and use it to guide the intelligent agent to make further decisions in group confrontation is a technical challenge. Existing methods cannot obtain more information from spatiotemporal information, nor can they use spatiotemporal information to make better action decisions. Therefore, it can be said that spatiotemporal information is not fully utilized.

[0008] Technical Challenge 3: Human prior knowledge is highly dependent on this approach. Using macroscopic information to help agents decide specific actions is a better way to complete tasks. However, existing research either requires manual definition of the specific content of macroscopic information or relies on empirically determined usage timeframes. This necessitates the introduction of human prior knowledge to assist multi-agent group confrontations, making it inapplicable to diverse scenarios, inherently limited, and highly dependent on human involvement.

[0009] Based on the above-mentioned technical problems existing in the prior art, the present invention provides a multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics. Summary of the Invention

[0010] The present invention proposes a multi-agent reinforcement learning method based on the dynamic generation of group confrontation tactics.

[0011] The present invention adopts the following technical solutions:

[0012] The present invention provides a multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics, comprising:

[0013] Step 1: Establish an upper-layer macro-action controller and a lower-layer micro-operation controller, as well as an upper-layer macro-action optimizer and a lower-layer micro-operation optimizer, and initialize the network parameters respectively;

[0014] Step 2: The agent uses the upper-level macro-action controller to generate the target position and macro-action duration;

[0015] Step 3, n->n+1, where n represents the nth agent. Repeat step 2 for different agents until all agents use the upper-level macro-action controller to generate the target position and macro-action duration, that is, n=N, where N is the total number of multi-agent systems.

[0016] Step 4: The agents in the multi-agent system use the underlying micro-operation controller to generate specific actions;

[0017] Step 5, n->n+1, where n represents the nth agent. Repeat step 4 for each agent until all agents use the underlying micro-manipulation controller to generate specific actions, that is, n=N, where N is the number of agents.

[0018] Step 6: The agent performs specific actions in the environment;

[0019] Step 7: The agent repeatedly executes steps 2 to 6 until t = T, the end of the i-th round, where T is the maximum number of time steps in a round and t is the time step of the interaction between the agent and the environment;

[0020] Step 8: The agent introduces the location heat map and updates it;

[0021] Step 9: The agent introduces a loss function of the self-imitation learning mechanism based on the heat map;

[0022] Step 10: The agent optimizes the upper-level macro-action controller;

[0023] Step 11, n->n+1, where n represents the nth agent. Different agents execute steps 9 to 10 separately until all agents optimize the upper-level macro action controller, that is, n=N, where N is the number of agents.

[0024] Step 12: The agent introduces auxiliary rewards containing spatiotemporal information;

[0025] Step 13: The agent optimizes the lower-level micro-operation controller;

[0026] Step 14: Repeat steps 2 to 13 until the upper macro-action controller and the lower micro-operation controller of the agent no longer change or the maximum number of rounds is reached;

[0027] Step 15: The intelligent agent uses the optimized upper-level macro-action controller to generate the target position and macro-action duration, and guides the lower-level micro-operation controller to generate specific actions.

[0028] Furthermore, step 2 includes:

[0029] Step 21: The agent obtains local observations from the environment

[0030] Step 22: The agent observes the local and previous upper-level macro actions Use neural network abstract representations;

[0031] Step 23: The agent generates implicit information using a recurrent neural network

[0032] Step 24: The agent uses the neural network to pass implicit information Generate upper-level macro actions Include duration and target location

[0033] Step 25: The agent selects the upper macro action. When the duration of the upper macro action currently being taken is When it is greater than 1, the previous upper-level macro action is still used at time t, and the duration of the upper-level macro action is reduced by one; when the duration of the upper-level macro action currently being taken is less than 1, the upper-level macro action newly generated by the upper-level macro action controller is used at time t; as shown in the following formula (1):

[0034]

[0035] Furthermore, step 4 includes:

[0036] Step 41: The agent obtains local observations from the environment

[0037] Step 42: The agent obtains the upper macro action from the upper macro action controller

[0038] Step 43: The agent observes the local Upper-level macro actions And previous specific actions Use neural network abstract representations;

[0039] Step 44: The agent generates implicit information using a recurrent neural network

[0040] Step 45: The agent uses the neural network to pass implicit information Generate specific actions

[0041] Furthermore, step 6 includes:

[0042] Step 61: The agents integrate their independent specific actions and combine them into a joint action for all agents.

[0043] Step 62: The agent inputs actions into the environment and collects information from the environment.

[0044] Step 63: The agent obtains state information from the environment

[0045] Step 64: The agent stores the output of the upper-level macro-action and lower-level micro-operation controller as well as the environmental information into the experience pool.

[0046] Furthermore, step 8 includes:

[0047] Step 81: The agent determines whether it has successfully completed the confrontation task in this round based on the final information provided by the environment.

[0048] Step 82: The agent maps the position at time t in round i to a number in the position heat map space Right now:

[0049] Step 83: The agent updates the location heat map based on whether it is successful.

[0050] Step 84, t->t+1, the agent repeatedly executes steps 82 to 83 until t=T, and completes the update using all the data of the i-th round;

[0051] Step 85, n->n+1, different agents execute steps 82 to 84 respectively until all agents use the round information to update the position heat map, that is, n=N, where N is the number of agents;

[0052] Step 86: The heat map updates the position index g by increasing by 1, and steps 82 to 85 are repeated until g=G, where G is the size of the heat map space dimension, completing the update of all position representations of the position heat map.

[0053] Furthermore, step 9 includes:

[0054] Step 91: The agent calculates the probability distribution of the target position g in the position heat map, as shown in the following formula (2):

[0055]

[0056] Step 92: The agent calculates the cross entropy loss function between the value distribution of the upper-level macro action controller and the target position distribution in the position heat map, as shown in the following formula (3):

[0057]

[0058] Furthermore, step 10 includes:

[0059] Step 101: The agent collects data from the experience pool: Among them, r is the reward and s represents the state;

[0060] Step 102: The agent sets the interval between the start time and the end time to The upper macro action is calculated as the new reward, as shown in the following formula (4):

[0061]

[0062] Where γ is the discount coefficient, represents the time from time step t to time step historical trajectory;

[0063] Step 103: The agent uses the upper-level macro-action optimizer to combine the value outputs of the N agent upper-level macro-action controllers into an overall upper-level macro-action value.

[0064] Step 104: The agent calculates the TD loss function for agent n based on the data in the experience pool according to the following formula (5):

[0065]

[0066] Among them, Θ - represents the target network parameters of the upper-level macro action controller;

[0067] Step 105: The agent uses the heat map loss function and TD loss function Combine them to get the upper macro action controller optimization loss function for agent n As shown in formula (6):

[0068]

[0069] Among them, λ is a hyperparameter;

[0070] Step 106: The agent uses the loss function and the gradient descent method to optimize the upper-level macro action controller.

[0071] Furthermore, step 13 includes:

[0072] Step 131: The agent collects data from the experience pool. Among them, B represents the dimension size of a batch of data (size);

[0073] Step 132: The agent uses the lower-level micro-operation optimizer to combine the value outputs of the N lower-level micro-operation controllers of the agent into an overall action value;

[0074] Step 133: The agent calculates the overall TD loss function based on the data in the experience pool and the auxiliary rewards obtained in step 12 according to the following formula (7):

[0075]

[0076] Among them, β is a hyperparameter, θ - It is the target network of the lower-level micro-operation controller;

[0077] Step 134, b=b+1, the agent repeats steps 131 to 133 for a batch of data from the experience pool until b=B, where B is the size of the batch data sampled from the experience pool;

[0078] Step 135: The agent sums the loss functions obtained in step 134 to obtain the loss function of the lower-level micro-operation controller.

[0079] Furthermore, in step 81, if the adversarial environment has a winning signal, obtaining the winning signal is considered a success; if the environment does not have a winning signal, but only rewards with different values, obtaining the maximum reward value that can be obtained within a time step is considered a success.

[0080] Furthermore, in step 83, if successful, the agent determines the position at time t in round i. Is it equal to the number g of the heat map update? If so, the heat map C i (g) The value increases by 1. If it is different, no update is performed; if it is unsuccessful, no update operation is performed.

[0081] Compared with the prior art, the present invention has the following advantages:

[0082] 1. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics described in the present invention introduces a dynamic generation method when using the multi-agent reinforcement learning method to solve group confrontation. When exploring and utilizing in the scene, it not only focuses on how to perform specific confrontation actions at each time step, but also adds dynamic generation of macro actions such as when and where to perform from a macro perspective. Compared with previous methods, it can better understand the scene, overcome the technical challenges of difficult understanding of complex scene tasks, and fully understand the scene tasks through tactics, especially in difficult confrontation scenes and tasks with sparse rewards, thereby obtaining the global optimal solution.

[0083] 2. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics described in the present invention effectively summarizes and records the action trajectory of the agent in group confrontation by introducing a position heat map, and forms a probability distribution form based on the association with success. By using a self-imitation learning mechanism based on a position heat map, the multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics described in the present invention can make the agent tend to fully explore the environment in the early stage of the task, and concentrate the macro-action on the key positions explored and continuously optimize them in the middle and late stages of the task. The upper-level macro-action is dynamically adjusted according to spatial information and contains spatiotemporal information, which is used to guide specific actions to better complete cooperation in cluster confrontation tasks. Compared with existing methods, the multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics described in the present invention takes into account spatiotemporal information, overcomes the technical challenge of insufficient utilization of spatiotemporal information, and enhances sample utilization and exploration efficiency.

[0084] 3. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics described in the present invention constructs an asynchronous dynamic update scheme based on duration. The agent generates position and duration through a controller instead of relying on human experience to give it in advance, thus overcoming the technical challenge of strong dependence on human prior knowledge; the technical solution of the multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics described in the present invention is closer to the confrontation scenarios in real life, and the agent's tactical strategy is more flexible. Combining the above two points, the multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics described in the present invention can better adapt to different scenarios and does not require too much human experience to help when changing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 This is a schematic diagram of a multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics according to an embodiment of the present invention;

[0086] Figure 2 A schematic diagram comparing the impact of multi-agent reinforcement learning based on dynamic generation of group confrontation tactics on solving specific tasks GRF-keeper according to an embodiment of the present invention with existing methods;

[0087] Figure 3 A schematic diagram comparing the impact of multi-agent reinforcement learning based on dynamic generation of group confrontation tactics in solving a specific GRF-hard task with existing methods according to an embodiment of the present invention;

[0088] Figure 4 A schematic diagram showing the impact of multi-agent reinforcement learning based on dynamic generation of group confrontation tactics on solving the specific task SMAC-6h_vs_8z in an embodiment of the present invention compared with existing methods;

[0089] Figure 5 A schematic diagram comparing the impact of multi-agent reinforcement learning based on dynamic generation of group confrontation tactics on solving the specific task SMAC-2c_vs_64zg according to an embodiment of the present invention with existing methods;

[0090] Figure 6 Schematic diagram comparing the impact of multi-agent reinforcement learning based on dynamic generation of group confrontation tactics on solving specific tasks MMM2 as described in an embodiment of the present invention with existing methods. DETAILED DESCRIPTION

[0091] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other unless there is a conflict.

[0092] Example

[0093] The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics includes:

[0094] Step 1: Establish an upper-layer macro-action controller and a lower-layer micro-operation controller, as well as an upper-layer macro-action optimizer and a lower-layer micro-operation optimizer, and initialize the network parameters respectively;

[0095] Step 2: The agent uses the upper-level macro-action controller to generate the target position and macro-action duration;

[0096] Specifically, step 2 includes:

[0097] Step 21: The agent obtains local observations from the environment

[0098] Step 22: The agent observes the local and previous upper-level macro actions Use neural network abstract representations;

[0099] Step 23: The agent generates implicit information using a recurrent neural network

[0100] Step 24: The agent uses the neural network to pass implicit information Generate upper-level macro actions Include duration and target location

[0101] Step 25: The agent selects the upper macro action. If the duration of the upper macro action currently being taken is If it is greater than 1, the previous upper-level macro action will still be used at time t, and the duration of the upper-level macro action will be reduced by one; if the duration of the upper-level macro action currently being taken is less than 1, the upper-level macro action newly generated by the upper-level macro action controller will be used at time t; that is:

[0102]

[0103] Step 3, n->n+1, where n represents the nth agent. Repeat step 2 for different agents until all agents use the upper-level macro-action controller to generate the target position and macro-action duration, that is, n=N, where N is the total number of multi-agent systems.

[0104] Step 4: The agent units in the multi-agent system use the underlying micro-operation controller to generate specific actions;

[0105] Specifically, step 4 includes:

[0106] Step 41: The agent obtains local observations from the environment

[0107] Step 42: The agent obtains the upper macro action from the upper macro action controller

[0108] Step 43: The agent observes the local Upper-level macro actions And previous specific actions Use neural network abstract representations;

[0109] Step 44: The agent generates implicit information using a recurrent neural network

[0110] Step 45: The agent uses the neural network to pass implicit information Generate specific actions

[0111] Step 5, n->n+1, where n represents the nth agent. Different agents execute step 4 separately until all agents use the lower-level micro-operation controller to generate specific actions, that is, n=N, where N is the number of agents;

[0112] Step 6: The agent performs specific actions in the environment;

[0113] Specifically, step 6 includes:

[0114] Step 61: The agents integrate their independent specific actions and combine them into a joint action for all agents.

[0115] Step 62: The agent inputs actions into the environment and collects information from the environment.

[0116] Step 63: The agent obtains state information from the environment

[0117] Step 64: The agent stores the output of the upper-level macro-action and lower-level micro-operation controllers and the environment information into the experience pool;

[0118] Step 7: The agent repeatedly executes steps 2 to 6 until t = T, the end of the i-th round, where T is the maximum number of time steps in a round and t is the time step of the interaction between the agent and the environment;

[0119] Step 8: The agent introduces the location heat map and updates it;

[0120] Specifically, step 8 includes:

[0121] Step 81: The agent determines whether it has successfully completed the confrontation task in this round based on the final information provided by the environment. If the confrontation environment has a winning signal, obtaining the winning signal is considered a success. If the environment does not have a winning signal, but only rewards with different values, obtaining the maximum reward value that can be obtained within a time step is considered a success.

[0122] Step 82: The agent maps the position at time t in round i to a number in the position heat map space Right now:

[0123] Step 83: The agent updates the position heat map based on whether it succeeds. If successful, the agent determines the position at time t in round i. Is it equal to the number g of the heat map update? If so, the heat map C i (g) The value increases by 1. If it is different, no update is performed. If it is unsuccessful, no update operation is performed.

[0124] Step 84, t->t+1, the agent repeatedly executes steps 82 to 83 until t=T, and completes the update using all the data of the i-th round;

[0125] Step 85, n->n+1, different agents execute steps 82 to 84 respectively until all agents use the round information to update the position heat map, that is, n=N, where N is the number of agents;

[0126] Step 86: The heat map update position index g is increased by 1, and steps 82 to 85 are repeated until g=G, where G is the size of the heat map space dimension, completing the update of all position representations of the position heat map;

[0127] Step 9: The agent introduces a loss function of the self-imitation learning mechanism based on the heat map;

[0128] Specifically, step 9 includes:

[0129] Step 91: The agent calculates the probability distribution of the target position g in the position heat map:

[0130]

[0131] Step 92: The agent calculates the cross entropy loss function between the value distribution of the upper-level macro action controller and the target position distribution in the position heat map:

[0132]

[0133] Step 10: The agent optimizes the upper-level macro-action controller;

[0134] Specifically, step 10 includes:

[0135] Step 101: The agent collects data from the experience pool: Where r is the reward, s represents the state, which is given by the environment here;

[0136] Step 102: The agent sets the interval between the start time and the end time to The upper macro action is calculated as the new reward:

[0137]

[0138] Where γ is the discount factor, represents the time from time step t to time step historical trajectory;

[0139] Step 103: The agent uses the upper-level macro-action optimizer to combine the value outputs of the N agent upper-level macro-action controllers into an overall upper-level macro-action value.

[0140] Step 104: The agent calculates the TD loss function for agent n based on the data in the experience pool:

[0141]

[0142] Among them, Θ - represents the target network parameters of the upper-level macro action controller;

[0143] Step 105: The agent uses the heat map loss function and TD loss function Combine them to get the upper macro action controller optimization loss function for agent n

[0144]

[0145] Among them, λ is a hyperparameter;

[0146] Step 106: The agent uses the loss function to optimize the upper-level macro action controller using the gradient descent method.

[0147] Step 11, n->n+1, where n represents the nth agent. Different agents execute steps 9 to 10 separately until all agents optimize the upper-level macro action controller, that is, n=N, where N is the number of agents.

[0148] Step 12: The agent introduces auxiliary rewards containing spatiotemporal information;

[0149] Step 13: The agent optimizes the lower-level micro-operation controller;

[0150] Specifically, step 13 includes:

[0151] Step 131: The agent collects data from the experience pool. Among them, B represents the dimension size of a batch of data (size);

[0152] Step 132: The agent uses the lower-level micro-operation optimizer to combine the value outputs of the N lower-level micro-operation controllers of the agent into an overall action value;

[0153] Step 133: The agent calculates the overall TD loss function based on the data in the experience pool and the auxiliary rewards obtained in step 12:

[0154]

[0155] Among them, β is a hyperparameter, θ - It is the target network of the lower-level micro-operation controller;

[0156] Step 134, b=b+1, the agent repeats steps 131 to 133 for a batch of data from the experience pool until b=B, where B is the size of the batch data sampled from the experience pool;

[0157] Step 135: The agent sums the loss functions obtained in step 134 to obtain the loss function of the lower-level micro-operation controller;

[0158] Step 14: Repeat steps 2 to 13 until the upper macro-action controller and the lower micro-operation controller of the agent no longer change or the maximum number of rounds is reached;

[0159] Step 15: The intelligent agent uses the optimized upper-level macro-action controller to generate the target position and macro-action duration, and guides the lower-level micro-operation controller to generate specific actions, thus realizing the multi-agent reinforcement learning method in group confrontation.

[0160] like Figure 1 Said method mainly includes four parts, namely, an upper-layer macro-action controller, a lower-layer micro-operation controller, an upper-layer macro-action optimizer, and a lower-layer micro-operation optimizer;

[0161] Among them, the upper-layer macro-action controller and the lower-layer micro-operation controller are used in the training and execution stages, and the upper-layer macro-action optimizer and the lower-layer micro-operation optimizer are used in the training stage;

[0162] Specifically, the upper macro action controller uses a three-layer neural network, which is a linear network MLP, a recurrent memory unit GRU and a parallel two-layer linear network MLP. The upper macro action controller is based on the local observation of the intelligent agent. Generate upper-level macro actions Contains two items: duration and target location The action controller uses a three-layer neural network, which is a linear network MLP, a recurrent memory unit GRU and a linear network MLP. The role of the action controller is to and the upper macro action generated by the upper macro action controller Generate specific underlying micro-operations The upper macro action optimizer and the lower micro operation optimizer use the same two-layer monotonically increasing network structure. For example, the first layer contains two linear networks W1 and B1. The two linear networks are combined using a two-layer linear network MLP with an activation function ReLU added in the middle. The two linear networks are based on the global state s t The parameters w1 and b1 for linear combination are obtained, the absolute value of the parameter w1 is taken, and then the values ​​of multiple agents are monotonically combined using the parameters w1 and b1. After passing through the ReLU activation function, it passes through the second layer network again, which also contains two layers of linear network MLP and adds the activation function ReLU in the middle. Finally, the reward value after the combination of multiple agents is obtained, which is used to optimize the upper-level macro action controller and the lower-level micro operation controller of the agent.

[0163] In the specific implementation, there are two possible application scenarios, such as in the process of robot football confrontation, or the group confrontation scenario of unmanned equipment. However, it is very difficult to design real robots and form a multi-agent system in the real world, which requires large equipment support, and its confrontation process has certain risks. Therefore, in this embodiment, a simulation platform from the mainstream academia is used to simulate the application of unmanned unit battles. The specific simulation environments are: Google Football Simulation Environment (GRF) and StarCraft Multi-Agent Challenge (SMAC). Specifically in the Google Football Simulation Environment, two scenarios are used, namely 3v1 and hard. Specifically in the StarCraft Multi-Agent Challenge, three scenarios are used, namely 6h_vs_8z, 2c_vs_64zg and MMM2. In the simulation scenarios simulating the real-world group confrontation process, the total task duration T is set to 200. If the total duration is exceeded, the current round is directly terminated and a new round is restarted. At the same time, the current round task is considered a failure.

[0164] In order to evaluate the performance of the method of the present invention, in the specific embodiment of the present invention, the following will be combined with Figure 2-6 The test results of the method described in this invention are evaluated in detail and compared with the following ten benchmark algorithms:

[0165] VDN: This method is a classic multi-agent deep reinforcement learning method that uses a directly summed linear network to make decisions and optimize the specific actions of the agents. It lacks a tactical understanding of the environment and the dynamic application of tactics to guide specific actions to better complete group confrontations.

[0166] QMIX: This method is a classic multi-agent deep reinforcement learning method. It uses a monotonically increasing two-layer linear network to make decisions and optimize the specific actions of the agents. It lacks a tactical understanding of the environment and the dynamic use of tactics to guide specific actions to better complete group confrontations.

[0167] QTRAN: This method is a classic multi-agent deep reinforcement learning method. It proposes a decomposition relationship between overall optimality and local optimality at the level of specific actions, and theoretically seeks the decomposition form of the optimal action. However, it lacks a tactical understanding of the environment and the dynamic application of tactics to guide specific actions to better complete group confrontations. Therefore, it is not easy to converge to specific optimal actions during training.

[0168] OW-QMIX: This method is a classic multi-agent deep reinforcement learning method. It proposes to use an optimistic weighted scheme to relax the original monotonicity assumption to improve the decomposition and fitting of the overall optimal action at the specific action level. It lacks a tactical understanding of the environment and the dynamic application of tactics to guide specific actions to better complete group confrontations. It has certain effects in simple environments such as grid worlds.

[0169] CW-QMIX: This method is a classic multi-agent deep reinforcement learning method. It proposes to use a center-weighted scheme to relax the original monotonicity assumption to improve the decomposition and fitting form of the overall optimal action at the specific action level. It lacks a tactical understanding of the environment and the dynamic application of tactics to guide specific actions to better complete group confrontations. It requires more environmental exploration and has a low sample utilization rate. Therefore, it is generally ineffective in complex group confrontations.

[0170] QPLEX: This method is a cutting-edge multi-agent deep reinforcement learning method. It proposes the use of a complex competitive network to derive the overall optimal action and the individual optimal action in a sufficient and necessary form at the level of specific agent actions. However, it lacks a tactical understanding of the environment and the dynamic application of tactics to guide specific actions to better complete group confrontations. It requires more environmental exploration and has a low sample utilization rate. Therefore, it is generally not effective in complex group confrontations.

[0171] FACMAC: This method is a cutting-edge multi-agent deep reinforcement learning method based on the policy gradient algorithm. It generalizes the specific action scenarios of the agents to the continuous action space, but lacks a tactical understanding of the environment and the dynamic application of tactics to guide specific actions and better achieve group confrontation.

[0172] MAIC: This method introduces a communication mechanism between agents. Agents learn models to predict the actions of their teammates and, based on this information, pass it on to their teammates, thereby optimizing their specific action choices. This method attempts to enhance the agent's decision-making through communication, but the layering is not clear, and the spatiotemporal information in the communication is not clear.

[0173] RODE: This method is a hierarchical reinforcement learning approach that uses roles to partition the agent's action space. The agent then selects roles and actions sequentially. While this method utilizes a hierarchical approach to enhance the agent's decision-making regarding specific actions, its upper-level information, namely the roles, is fixed at the start of training and lacks dynamic adjustment based on spatiotemporal information. Furthermore, the number of time steps for role selection in this method is a manually designed constant that remains constant for all agents, making it poorly adaptable to diverse environments.

[0174] MASER: This method uses sub-goals generated from experience pool data to formulate auxiliary rewards for optimizing specific actions. However, the sub-goals in this method are defined as the value at a specific moment in a round. The definition and optimization of sub-goals are relatively abstract and lack the use of spatiotemporal information.

[0175] like Figure 2-6 As shown in the figure, the multi-agent reinforcement learning method (HiMacMic) based on dynamic generation of group confrontation tactics of the present invention achieved the highest test results in different scenarios in a simulation environment simulating the real world. In the experiment, the parameters of the benchmark method in different test environments were adjusted to fully demonstrate the better performance of the benchmark algorithm and conduct a full comparison. Figure 4 Taking the most difficult scenario of the StarCraft multi-agent challenge as an example, the method described in this embodiment, by introducing dynamically generated tactics, nearly doubled the probability of winning compared to the best result (OW-QMIX) of an algorithm that only focuses on the specific actions of the agents. Compared to the best result (QPLEX) of an algorithm that only focuses on the specific actions of the agents, the method of the present invention has a 20% improvement in winning rate. This demonstrates that the method of the present invention can overcome the technical challenges of difficult understanding of complex scenario tasks, and by introducing dynamically generated macro-actions to fully understand the scenario tasks, especially in difficult confrontation scenarios and tasks with sparse rewards, a better group confrontation strategy can be obtained.

[0176] according to Figure 2-6 As shown in the figure, compared with other methods of improving the specific actions of intelligent agents, MAIC, RODE and MASER, MAIC is Figure 4 and 5 The effect is not good, MASER is Figure 3 and 6While it can complete the task to a certain extent in the context of the task, it performs poorly in other scenarios. Unlike other benchmark methods, this method achieves good test results in multiple benchmark scenarios. This demonstrates that this method introduces position heatmaps and auxiliary rewards containing spatiotemporal information to guide the optimization process of tactics and specific actions. This fully utilizes spatiotemporal information, addresses the technical challenge of insufficient spatiotemporal information utilization in group confrontation scenarios, obtains more information from spatiotemporal information, and uses spatiotemporal information to make better action decisions, ultimately achieving a better group confrontation strategy. At the same time, the dynamic generation of upper-level macro-actions makes tactics flexible and adaptable, overcoming the technical challenge of relying heavily on human prior knowledge. Based on spatiotemporal information and task characteristics, different tactics are dynamically and asynchronously generated to adapt to different scenarios and task requirements, ultimately achieving a better group confrontation strategy.

[0177] The present invention is not limited to the above-described embodiments. The above-described embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims.

Claims

1. A multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics, characterized by: include: Step 1: Establish an upper-layer macro-action controller and a lower-layer micro-operation controller, as well as an upper-layer macro-action optimizer and a lower-layer micro-operation optimizer, and initialize the network parameters respectively; Step 2: The agent uses the upper-level macro-action controller to generate the target position and macro-action duration: Step 3, n->n+1, where n represents the nth agent. Repeat step 2 for different agents until all agents use the upper-level macro-action controller to generate the target position and macro-action duration, that is, n=N, where N is the number of agents. Step 4: The agents in the multi-agent system use the underlying micro-operation controller to generate specific actions: Step 5, n->n+1, where n represents the nth agent. Different agents execute step 4 separately until all agents use the lower-level micro-operation controller to generate specific actions, that is, n=N, where N is the number of agents; Step 6: The agent performs specific actions in the environment: Step 7: The agent repeatedly executes steps 2 to 6 until t = T, the end of the i-th round, where T is the maximum number of time steps in a round and t is the time step of the interaction between the agent and the environment; Step 8: The agent introduces the location heat map and updates it; Step 9: The agent introduces the loss function of the self-imitation learning mechanism based on the heat map: The loss function is the cross entropy loss function, as shown in the following formula (3): Among them, p i (g) is the probability distribution of the target position g, G is the spatial dimension of the heat map, and Q is the value of the upper-level macro action controller; Step 10: The agent optimizes the upper-level macro action controller: The agent uses the upper-level macro-action optimizer to combine the value outputs of N agent upper-level macro-action controllers into an overall upper-level macro-action value; The TD loss function is calculated according to the following formula (5): Among them, Θ -,n represents the target network parameters of the upper-level macro action controller, For the reward, Q Ma Output value for the upper macro action optimizer, s represents the state, the symbol subscript represents the time, t represents the time t, is the upper-level macro action generated by the upper-level macro action controller, and γ is the discount coefficient; The agent uses the cross entropy loss function and TD loss function Combine them to get the upper macro action controller optimization loss function for agent n As shown in formula (6): Among them, λ is a hyperparameter; Step 11, n->n+1, where n represents the nth agent. Different agents execute steps 9 to 10 separately until all agents optimize the upper-level macro action controller, that is, n=N, where N is the number of agents. Step 12: The agent introduces auxiliary rewards containing spatiotemporal information; Step 13: The agent optimizes the lower-level micro-operation controller: The agent uses the lower-level micro-operation optimizer to combine the value outputs of N lower-level micro-operation controllers of the agent into an overall action value; The overall TD loss function is calculated according to the following formula (7): Among them, β is a hyperparameter, θ - is the target network parameter of the lower-level micro-operation controller, For the next moment joint micro-operation of the selected agent, For auxiliary rewards, Q Mi Output value to the underlying micro-operation optimizer; Step 14: Repeat steps 2 to 13 until the upper macro-action controller and the lower micro-operation controller of the agent no longer change or the maximum number of rounds is reached; Step 15: The intelligent agent uses the optimized upper-level macro-action controller to generate the target position and macro-action duration, and guides the lower-level micro-operation controller to generate specific actions.

2. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics according to claim 1 is characterized in that: Step 2 includes: Step 21: The agent obtains local observations from the environment Step 22: The agent observes the local and previous upper-level macro actions Use neural network abstract representations; Step 23: The agent generates implicit information using a recurrent neural network Step 24: The agent uses the neural network to pass implicit information Generate upper-level macro actions Include duration and target location Step 25: The agent selects the upper macro action. When the duration of the upper macro action currently being taken is When it is greater than 1, the previous upper-level macro action is still used at time t, and the duration of the upper-level macro action is reduced by one; when the duration of the upper-level macro action currently being taken is less than 1, the upper-level macro action newly generated by the upper-level macro action controller is used at time t; as shown in the following formula (1):

3. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics according to claim 1 is characterized in that: Step 4 includes: Step 41: The agent obtains local observations from the environment Step 42: The agent obtains the upper macro action from the upper macro action controller Step 43: The agent observes the local Upper-level macro actions And previous specific actions Use neural network abstract representations; Step 44: The agent generates implicit information using a recurrent neural network Step 45: The agent uses the neural network to pass implicit information Generate specific actions 4. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics according to claim 1 is characterized in that Step 6 includes: Step 61: The agents integrate the individual specific actions into a joint action of all agents. Step 62: The agent inputs actions into the environment and collects information from the environment. Step 63: The agent obtains state information s from the environment t ,award Step 64: The agent stores the output of the upper-level macro-action and lower-level micro-operation controller as well as the environmental information into the experience pool.

5. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics according to claim 1 is characterized in that: Step 8 includes: Step 81: The agent determines whether it has successfully completed the confrontation task in this round based on the final information provided by the environment. Step 82: The agent maps the position at time t in round i to a number in the position heat map space Right now: Step 83: The agent updates the location heat map based on whether it is successful. Step 84, t->t+1, the agent repeatedly executes steps 82 to 83 until t=T, completing the update using all the data from the i-th round; Step 85, n->n+1, different agents execute steps 82 to 84 respectively until all agents use the round information to update the position heat map, that is, n=N, where N is the number of agents; Step 86: The heat map updates the position index g by increasing by 1, and steps 82 to 85 are repeated until g=G, where G is the size of the heat map space dimension, completing the update of all position representations of the position heat map.

6. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics according to claim 1 is characterized in that Step 9 includes: Step 91: The agent calculates the probability distribution of the target position g in the position heat map, as shown in the following formula (2): Among them, G is the size of the heat map space dimension; Step 92: The cross entropy loss function is as follows (3): Among them, p i (g) is the probability distribution of the target position g, G is the spatial dimension of the heat map, and Q is the value of the upper-level macro action controller.

7. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics according to claim 2 is characterized in that: Step 10 includes: Step 101: The agent collects data from the experience pool: Where r is the reward and s represents the state; Step 102: The agent calculates the new reward based on the data in the experience pool, as shown in the following formula (4): Where γ is the discount factor, represents the time from time step t to time step historical trajectory; Step 103: The agent uses the upper-level macro-action optimizer to combine the value outputs of the N agent upper-level macro-action controllers into an overall upper-level macro-action value. Step 104: Calculate the TD loss function according to the following formula (5): Among them, Θ -,n represents the target network parameters of the upper-level macro action controller, For rewards; Step 105: The agent uses the cross entropy loss function and TD loss function Combine them to get the upper macro action controller optimization loss function for agent n As shown in formula (6): Among them, λ is a hyperparameter; Step 106: The agent uses the loss function and the gradient descent method to optimize the upper-level macro action controller.

8. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics according to claim 1 is characterized in that: Step 13 includes: Step 131: The agent collects data from the experience pool. Among them, B represents the size of a batch of data, r is the reward, and s represents the state; Step 132: The agent uses the lower-level micro-operation optimizer to combine the value outputs of the N lower-level micro-operation controllers of the agent into an overall action value; Step 133: Calculate the overall TD loss function according to the following formula (7): Among them, β is a hyperparameter, θ - is the target network parameter of the lower-level micro-operation controller; Step 134, b=b+1, the agent repeats steps 131 to 133 for a batch of data from the experience pool until b=B; Step 135: The agent sums the loss functions obtained in step 134 to obtain the loss function of the lower-level micro-operation controller.

9. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics according to claim 5, characterized in that: In step 81, if the adversarial environment has a winning signal, obtaining the winning signal is considered a success; if the environment does not have a winning signal, but only rewards with different values, obtaining the maximum reward value that can be obtained within a time step is considered a success.

10. The multi-agent reinforcement learning method based on dynamic generation of group confrontation tactics according to claim 5, characterized in that: In step 83, if successful, the agent determines its position at time t in round i. Is it equal to the number g of the heat map update? If so, the heat map C i (g) The value increases by 1. If it is different, no update is performed; if it is unsuccessful, no update operation is performed.

Images