Deep reinforcement learning cluster confrontation method and system based on attention mechanism

Abstract

The present invention provides a deep reinforcement learning cluster confrontation method and system based on an attention mechanism, including the training of a deep reinforcement learning model with an embedded attention module in a small-scale confrontation scenario, as well as group division and determination of group members in a large-scale confrontation scenario, and confrontation guidance based on the deep reinforcement learning model. In this solution, based on the attention mechanism, it is possible to configure the individual weights of agents of different camps and types and the weights of their state information, thereby improving the decision-making efficiency and accuracy of agents in a highly dynamic environment. In a large-scale scenario, group division and determination of members within the group can ensure that each agent can effectively execute tasks according to the current situation. By integrating the attention mechanism with deep reinforcement learning technology, the high-efficiency cooperative combat ability of agents in a dynamic and complex environment can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and more particularly, to a deep reinforcement learning cluster confrontation method and system based on an attention mechanism. Background Art

[0002] With the development of artificial intelligence technology, multi-agent systems (MAS) are increasingly widely used in complex environments. Especially in fields such as military simulations, network security, and robot competitions, multi-agent systems need to process a large amount of dynamic information and perform efficient cooperative operations and task assignments. Traditional multi-agent systems often rely on manually designed rules and strategies and are difficult to adapt to highly dynamic and uncertain environments.

[0003] In recent years, deep reinforcement learning (DRL) has been widely used in multi-agent systems due to its self-learning ability and sequential decision-making ability. However, standard DRL algorithms face many challenges when dealing with large-scale multi-agent systems, mainly including the exponential growth of the state space and the coordination problem between agents. Especially in highly dynamic complex confrontation environments, multi-agents of different camps and types interact with the environment, and these agents have different input dimensions and physical meanings, further increasing the processing difficulty of traditional DRL algorithms. To solve these problems, researchers have proposed various improvement methods, and among them, introducing the attention mechanism has become one of the effective means to improve the adaptability and decision-making quality of the model.

[0004] The attention mechanism can quickly identify and respond to key events in complex environments by dynamically adjusting the attention degree of agents to environmental information, reducing the interference of invalid information. This mechanism can effectively process the dynamic inputs of different agents and improve the adaptability and decision-making quality of the model. However, the attention mechanism has a high demand for the number of parameters and is not suitable for directly applying to the training of ultra-large-scale agents. Therefore, how to flexibly design the network structure according to the characteristics of the attention mechanism and make it adapt to the confrontation scenarios of large-scale agents has become an important technical problem. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a deep reinforcement learning cluster confrontation method and system based on an attention mechanism to achieve the efficient cooperative combat ability of agents in dynamic complex environments.

[0006] In a first aspect, the present invention provides a deep reinforcement learning cluster confrontation method based on an attention mechanism, the method comprising:

[0007] Construct a cluster confrontation model, where the cluster confrontation model includes an agent cluster and an adversarial simulation scenario. The agent cluster includes multiple agents of different camps and different types. The adversarial simulation scenario includes an adversarial simulation environment and protection points;

[0008] Select some agents and protection points in the cluster confrontation model to train a deep reinforcement learning model embedded with an attention module;

[0009] Based on all the agents in the agent cluster, conduct group division, and for each agent in each group of all groups, select agents of the same camp and agents of different camps to which the agent belongs, and determine the protection points within each group;

[0010] Input the agents and protection points within all groups into the trained deep reinforcement learning model to output the action information of each agent;

[0011] Use the action information to guide each agent to conduct simulation confrontation in the adversarial simulation environment.

[0012] In an alternative implementation, the step of selecting some agents and protection points in the cluster confrontation model to train a deep reinforcement learning model embedded with an attention module includes:

[0013] For some agents in the cluster confrontation model, construct an embedded representation of the agent based on the Embed function, and input the embedded representation into the attention module of the deep reinforcement learning model;

[0014] Based on the attention module, output the embedded representations of other agents corresponding to each agent, and splice and process the embedded representation of the agent and the embedded representations of other agents in a fully connected layer to obtain decision-making information;

[0015] Train the deep reinforcement learning model based on the loss function constructed for each agent and the decision-making information until the training stops when a preset stop condition is met.

[0016] In an alternative implementation, the step of constructing the embedded representation of the agent based on the Embed function includes:

[0017] Use the fully connected layer in the deep reinforcement learning model to process the observation information and action information of the agent respectively;

[0018] Splice the processing results of the observation information and action information with the attention output results of the agents of the same camp, agents of different camps to which the agent belongs, and the protection points;

[0019] Process the splicing result using a fully connected layer to obtain the embedded representation of the agent.

[0020] In an alternative embodiment, the step of outputting the embedded representations of other agents corresponding to each agent based on the attention module includes:

[0021] For each agent, obtain the embedded representations of each other agent corresponding to the agent based on the attention module;

[0022] Obtain the attention weights of each other agent based on a scoring function;

[0023] Accumulate the embedded representations of all other agents based on the attention weights of each other agent to obtain the embedded representation of other agents corresponding to the agent.

[0024] In an alternative embodiment, the step of obtaining the attention weights of each other agent based on a scoring function includes:

[0025] Obtain the degree of attention of the agent to each other agent based on a scoring function;

[0026] For each other agent, obtain the attention weight of the other agent according to the degree of attention of the agent to the other agent and the cumulative value of the degrees of attention of the agent to all other agents.

[0027] In an alternative embodiment, the agent includes a red - side agent and a blue - side agent belonging to different camps;

[0028] The step of performing group division based on all agents in the agent cluster includes:

[0029] Obtain the number of red - side agents and blue - side agents in the selected agent cluster;

[0030] Based on the set agent ratio pattern within a group and the numbers of red - side agents and blue - side agents, determine the number of groups and the number of subgroups within each group.

[0031] In an alternative embodiment, the step of determining the number of groups and the number of subgroups within each group based on the set agent ratio pattern within a group and the numbers of red - side agents and blue - side agents includes:

[0032] Determine the number of groups based on the set agent ratio pattern within a group;

[0033] Minimize the constructed objective function based on the number of groups, the number of the red agents, and the number of the blue agents to determine the number of subgroups within each group.

[0034] In an alternative embodiment, the step of selecting agents of the same camp and agents of different camps for each of the agents within each subgroup of all groups and determining protection points within each of the subgroups includes:

[0035] Select red agents of the same camp according to the position information of each of the red agents and the group division result to form multiple subgroups;

[0036] Determine the blue agents to be added to each of the subgroups based on the distances between each of the blue agents and the average positions of the red agents within each of the subgroups;

[0037] For each of the subgroups, determine the protection points to be added to each of the subgroups based on the distances between the blue agents within the subgroup and each of the protection points.

[0038] In an alternative embodiment, the action information includes acceleration information;

[0039] The step of guiding each of the agents to perform simulation confrontation within the confrontation simulation environment with the action information includes:

[0040] At each time step of each simulation confrontation, guide each of the agents to move within the confrontation simulation environment with the acceleration information obtained at the time step;

[0041] Statistically analyze the confrontation results of the agents of different camps in multiple simulation confrontations and analyze the success rate and failure rate of the agents of different camps.

[0042] In a second aspect, the present invention provides a deep reinforcement learning cluster confrontation system based on an attention mechanism, the system includes:

[0043] A construction module, configured to construct a cluster confrontation model, the cluster confrontation model includes an agent cluster and a confrontation simulation scenario, the agent cluster includes multiple agents of different camps and different types, and the confrontation simulation scenario includes a confrontation simulation environment and protection points;

[0044] A training module, configured to select some of the agents and protection points in the cluster confrontation model to train a deep reinforcement learning model embedded with an attention module;

[0045] A division module, configured to perform group division based on all the agents in the agent cluster, select agents of the same camp and agents of different camps for each of the agents within each subgroup of all groups, and determine the protection points within each of the subgroups;

[0046] A processing module, configured to input agents and protection points within all groups into a trained deep reinforcement learning model, and output action information of each of the agents;

[0047] A simulation confrontation module, configured to guide each of the agents to perform simulation confrontation within the confrontation simulation environment according to the action information. Description of the Drawings

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments of the present invention. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as a limitation of the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0049] Figure 1 It is a flowchart of a deep reinforcement learning cluster confrontation method based on an attention mechanism provided by an embodiment of the present invention;

[0050] Figure 2 It is a schematic design diagram of Actor-Critic in an attention module provided by an embodiment of the present invention;

[0051] Figure 3 It is one of the schematic diagrams of offense and defense confrontation games on a small scale provided by an embodiment of the present invention;

[0052] Figure 4 It is the second of the schematic diagrams of offense and defense confrontation games on a small scale provided by an embodiment of the present invention;

[0053] Figure 5 It is a schematic diagram of group division provided by an embodiment of the present invention;

[0054] Figure 6 It is a schematic diagram of a large-scale offense and defense confrontation game provided by an embodiment of the present invention;

[0055] Figure 7 It is a functional module block diagram of a deep reinforcement learning cluster confrontation system based on an attention mechanism provided by an embodiment of the present invention;

[0056] Figure 8 It is a structural block diagram of an electronic device provided by an embodiment of the present invention. Detailed Embodiments

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. The components of the embodiments of the present invention usually described and illustrated in the drawings here can be arranged and designed in various different configurations.

[0058] Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0059] It should be noted that like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0060] It should be noted that in this document, the terms "include", "comprise", or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the presence of additional identical elements in the process, method, article, or device including the element.

[0061] The following will describe in detail some embodiments of the present invention with reference to the accompanying drawings. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0062] Please refer to Figure 1 , which is a flowchart of the deep reinforcement learning cluster confrontation method based on the attention mechanism provided by the embodiment of the present invention. The deep reinforcement learning cluster confrontation method based on the attention mechanism can be executed by a deep reinforcement learning cluster confrontation system based on the attention mechanism. The deep reinforcement learning cluster confrontation system based on the attention mechanism can be implemented by software and / or hardware and can be configured in an electronic device, and the electronic device can be a computer device. The detailed steps of the deep reinforcement learning cluster confrontation method based on the attention mechanism are introduced as follows.

[0063] S11. Construct a cluster confrontation model, where the cluster confrontation model includes an agent cluster and an adversarial simulation scenario. The agent cluster includes multiple agents of different camps and different types, and the adversarial simulation scenario includes an adversarial simulation environment and a protection point.

[0064] S12. Select some agents and protection points in the cluster confrontation model to train the deep reinforcement learning model embedded with an attention module.

[0065] S13. Group all the agents in the agent cluster, and within each group of all groups, select the agents belonging to the same camp and the agents belonging to different camps for each agent, and determine the protection points within each group.

[0066] S14. Input the agents and protection points within all groups into the trained deep reinforcement learning model to output the action information of each agent.

[0067] S15. Use the action information to guide each agent to conduct simulation confrontation in the confrontation simulation environment.

[0068] In this embodiment, the agent cluster includes agents of different camps, such as attacker agents and defender agents. The types of agents include reconnaissance type, strike type, interference type, interception type, etc. Each type of agent undertakes different functions and tasks.

[0069] The confrontation simulation scenario includes a simulated environment in a defined area. One or more protection points are included in this area. These attackers and defenders conduct complex attack and game tasks in this area. The task objectives include but are not limited to protecting key target protection points in the area, preventing agents of the other camp from entering a specific area, or destroying the protection points of the other camp, etc.

[0070] The agents in this embodiment can be robot agents, drone agents, network node agents, etc. This solution can be applied to scenarios such as military simulation, network security simulation, robot competition simulation, economic game simulation, etc. For example, in the network security scenario, the confrontation between hacker attacks and defenses can be simulated. In the robot competition, the coordinated operation of multiple robots can be simulated. In the economic game, the competition strategies in the market can be simulated. In different application scenarios, the types and task objectives of the agents can be customized based on actual needs.

[0071] In this embodiment, the numbers of attackers, defenders, and protection points can also be set based on the confrontation simulation requirements. In the setting of the cluster confrontation scenario, considering that agents may die when performing strike functions, the number of red agents (defender agents) should be greater than that of blue agents (attacker agents) initially. However, if during the confrontation process, due to tactics such as deception and interference by the blue side, the number of red agents is less than that of blue agents at a certain moment, unless there is a time limit, the red agents will be regarded as failed, and at this time, it can be considered that the blue agents have won.

[0072] In this embodiment, an attention module is designed. The attention module includes an Actor network and a Critic network. The attention module can process the input information of agents of different camps and different types to improve the adaptability and decision-making quality of the model.

[0073] The attention module is embedded in the constructed deep reinforcement learning model, which is a multi-agent deep deterministic policy gradient (MADDPG) algorithm model. In this embodiment, variants of other deep reinforcement learning models, such as PPO, TRPO, etc., can also be used, or combined with other advanced attention mechanisms. In addition, the learning process can be accelerated and the initial performance of the agent can be improved by introducing external knowledge or pre-trained models.

[0074] In this embodiment, a small-scale agent and protection points are selected from the cluster confrontation model to train the deep reinforcement learning model. Then, in a large-scale scenario, the agents are grouped and teammates (agents of the same camp), opponents (agents of different camps), and protection points are determined to form multiple groups. The trained deep reinforcement learning model is used to guide the confrontation simulation of the large-scale agent cluster divided into multiple groups. The guidance information includes the action information of the agents, mainly including acceleration information. Thus, each agent can perform the confrontation simulation under the guidance of the acceleration information output by the deep reinforcement learning model.

[0075] The trained deep reinforcement learning model can improve the confrontation level of agents on both sides of different camps and avoid the situation where one side of the agents is too simple and the winning rate of the other side is too high. Therefore, this method is applicable to both attackers and defenders and is not limited to the application restrictions of a specific confrontation scenario.

[0076] In this embodiment, the attention module includes an Actor network and a Critic network. The two can share some network parameters, and their training processes are similar. When training the deep reinforcement learning model with the attention module embedded based on small-scale agents, it can be achieved in the following way:

[0077] For some agents in the cluster confrontation model, an embedded representation of the agent is constructed based on the Embed function, and the embedded representation is input into the attention module of the deep reinforcement learning model; based on the attention module, the embedded representations of other agents corresponding to each agent are output, and the embedded representation of the agent is concatenated and processed with the embedded representations of other agents in the fully connected layer to obtain decision-making information; the deep reinforcement learning model is trained based on the loss function constructed for each agent and the decision-making information until the preset stop condition is met and the training stops.

[0078] In this embodiment, the design of the Critic network includes two blocks, namely the overall design of the Critic network and the specific design of the Embed function.

[0079] The overall design of the Critic network can be expressed as:

[0080]

[0081] Among them, and are fully connected networks; is the embedded representation of each agent (constructed by the Embed function), and its input is its own observation (local state information of the surrounding area detected by sensors based on the configuration) and the action concatenated vector; is the output of the attention module, which is an embedded representation of all other agents except itself.

[0082] As Figure 2 shown, where the left side represents the overall design of the Critic network and the right side is the specific design of the Embed function. Embedding this architecture into MADDPG for training can train an adversarial strategy in a small-scale scenario.

[0083] Combined with the above formula, it can be seen that the embedded representation of the agent is constructed based on the Embed function , and the embedded representation is processed by a fully connected layer (FC) . After processing, it is concatenated with the embedded representations of other agents and processed together by a fully connected layer to obtain decision-making information. Based on the decision-making information and the constructed loss function, the deep reinforcement learning model is trained. When the training reaches convergence or the number of training iterations meets the requirements, etc., the training stops.

[0084] Among them, different types of agents can have different reward functions and loss functions to reflect their specific task objectives.

[0085] In this embodiment, when constructing the embedded representation of the agent based on the Embed function, it can be achieved through the following methods:

[0086] Use the fully connected layers in the deep reinforcement learning model to process the observation information and action information of the agent respectively; concatenate the processing results of the observation information and action information with the attention output results of the agents in the same camp, the agents in different camps, and the protection points to which the agent belongs; use the fully connected layer to process the concatenated results to obtain the embedded representation of the agent.

[0087] In this embodiment, the specific design of the Embed function is as follows:

[0088]

[0089] Among them, 、 、 、 and are fully connected networks, The network of this type is the first-layer embedding for the input. They are all attention mechanism networks under different types or camps, and their definitions are similar to the overall design of the above-mentioned Critic network.

[0090] That is, according to the above formula, it can be known that based on the Embed function, the observation information and action information of the agent are first processed through the fully connected layer: 、 。 respectively represent the attention output results of agents in the same camp as the agent, the attention output results of agents in different camps, and the attention output results of the protection points.

[0091] The processing results of the agent's observation information and action information are concatenated with the above three types of attention output results , and then the fully connected layer is used to process the concatenated result, and finally the embedding representation of the agent is obtained.

[0092] In the above formula, is the output of the attention module, which is an embedding representation of all other agents except itself. In the above step of outputting the embedding representations of other agents corresponding to each agent based on the attention module, it can be achieved in the following way:

[0093] For each agent, obtain the embedding representations of the other agents corresponding to the agent based on the attention module; obtain the attention weights of the other agents based on the scoring function; accumulate the embedding representations of all other agents based on the attention weights of the other agents to obtain the embedding representation of the other agents corresponding to the agent.

[0094] In this embodiment, given , the embedding representation of other agents can be expressed as:

[0095]

[0096] Among them, is the attention weight, which can be determined based on the scoring function. Specifically, it can be determined in the following way:

[0097] The attention degree of the agent to each other agent is obtained based on a scoring function; for each other agent, according to the attention degree of the agent to the other agent and the cumulative value of the attention degrees of the agent to all other agents, the attention weight of the other agent is obtained.

[0098] Specifically, the calculation formula of the attention weight is as follows:

[0099]

[0100] is a scoring function for measuring the attention degree of the agent to the agent which can be comprehensively calculated based on factors such as the distance, speed, and direction between agents. Optionally, a normalized inner product is used as the scoring function. Optionally, the self-attention mechanism can be used to set learnable parameters to participate in training.

[0101] The overall idea of the design of the Critic network is that it can separately evaluate the state value of each agent, and then comprehensively consider to obtain the final value evaluation. This enables the Critic to flexibly process the inputs of agents of different camps and different types. If the agent is not within the observation range of the Critic network, then it should be 0, so as not to affect the evaluation of the Critic network.

[0102] Among them, the agent embedding networks of the same camp and type can share parameters, and the attention modules between different agents can share parameters.

[0103] The design of the Actor network is similar to the design of the Embed function. Only the dimension of the last layer of the Embed function needs to be modified to be used as the action output of the Actor network, and the network parameters of the remaining levels can be shared with the Embed function in the Critic network, which will not be elaborated in this embodiment.

[0104] In this embodiment, the inputs of the Actor network and the Critic network not only include agents of different camps and different types, but also include target protection points with non-decision-making functions. This enables the method to adapt to the cluster confrontation scenario of asymmetric targets and meet the input requirements of agents of different camps and different types.

[0105] The attention mechanism can dynamically adjust the degree of attention of each agent to the state information of other agents. By adaptively changing the weights, it enhances the sensitivity of the model to key information, improving the decision-making efficiency and accuracy. Especially in high-dynamic and uncertain environments, the attention mechanism can quickly identify and respond to key events, reduce the interference of invalid information, and improve the response speed and decision-making quality of the system.

[0106] Key information includes but is not limited to the flight action information of the agent, the value evaluation results, as well as the survival status of the agent and whether it exceeds the observation range. This information is crucial for the agent's decision-making. The attention mechanism can dynamically identify and process this key information. By increasing the weights of this information, the model can adapt to information inputs of different lengths, improving the decision-making accuracy. Especially in the case of a large number of agents and a rapidly changing environment, it effectively prevents misjudgments and decision-making errors from occurring and avoids a sharp decline in decision-making ability.

[0107] Specifically, in the Actor network, the attention mechanism is used to weight the state information of different agents in order to more effectively learn individual behavior strategies, enabling each agent to make optimal decisions based on the current environment and the states of other agents.

[0108] In the Critic network, the attention mechanism assigns weights to different agents to more accurately evaluate the effects of multi-agent joint actions, thereby optimizing the overall strategy and improving the task completion rate and system performance.

[0109] Finally, the attention module is embedded in the deep reinforcement learning model, and the training of the deep reinforcement learning model is carried out while the attention module is being trained.

[0110] During the training process of the deep reinforcement learning model, the Experience Replay and Target Network Update strategies are adopted to stabilize the learning process. The observations, actions, etc. of the agent at each time step are stored in the experience replay pool and trained through random sampling to reduce the correlation between samples. Every certain number of time steps or training rounds, the Soft Update strategy is used to update the parameters of the target network to stabilize the training process. The specific formula is:

[0111]

[0112] where, represents the parameters of the current network, represents the parameters of the target network, is a small positive number, usually taking a value of 0.01.

[0113] In this embodiment, during the process of training the deep reinforcement learning model, the confrontation effects of agents in different camps under the guidance of the trained deep reinforcement learning model can also be observed. When the confrontation effects of the two gradually tend to be stable, that is, when the training reaches convergence, the training can be completed.

[0114] Among them, Figure 3 The figure shows the confrontation effects of the red and blue agents in a 1:1 ratio under the guidance of the trained deep reinforcement learning model in a small-scale scenario. In this small scale, after training with the deep reinforcement learning model, the red and blue agents are guided to confront. During the task process, if the red agent fails to strike and is killed, the task will not stop, but will continue until the blue agent successfully strikes the target point.

[0115] In this task, the step size of each round is 100, and it converges after training 8000 rounds. It can be seen that the converged agents have a significant level of intelligent gaming. For the red agent, it will continuously predict the blue agent's attack route according to the situation of the blue agent, and always be on guard and intercept near the line connecting the blue agent and the protection point. For the blue agent, it will continuously change its own attack route according to the position of the red agent, giving full play to the first-mover advantage of the attacking side as the active side, and moving at high speed to prevent the red agent from changing the route in time. In Figure 3 the example, the blue agent wins by a narrow margin.

[0116] Figure 4 The figure shows the confrontation effects of the red and blue agents in a 2:1 ratio. In this task, the step size of each round is 100, and it converges after training 15000 rounds. It can be seen that the converged agents have a significant level of intelligent gaming. For the red agent, it will continuously predict the blue agent's attack route according to the situation of the blue agent, and when the action range of the blue agent is confirmed, it will perform double-insurance interception one after the other. For the blue agent, it will continuously change its own attack route according to the position of the red agent, and will reduce its speed when about to be intercepted to flexibly respond to unexpected situations. In Figure 4 the example, the blue agent is successfully intercepted.

[0117] After the above-mentioned agents in the small-scale complete the training of deep reinforcement learning, the trained deep reinforcement learning model can be applied to large-scale confrontation scenarios. First, it is necessary to divide all the agents in the agent cluster in the large-scale scenario into groups. Specifically, the group division can be achieved through the following methods:

[0118] Obtain the number of red agents and blue agents in the selected agent cluster; determine the number of groups and the number of subgroups within each group based on the set agent ratio mode within the group and the number of red agents and blue agents.

[0119] In this embodiment, the agent ratio mode within a group is characterized by the number of red agents: the number of blue agents: the number of protection points, and specifically may include, for example, 1:1:1, 2:1:1, 3:1:1, etc.

[0120] Specifically, the step of determining the number of groups and the number of subgroups within each group based on the set agent ratio mode within a group and the numbers of red agents and blue agents can be achieved in the following manner:

[0121] Determine the number of groups based on the set agent ratio mode within a group; based on the number of groups, the numbers of red agents and blue agents, minimize the constructed objective function to determine the number of subgroups within each group.

[0122] In this embodiment, one ratio mode may correspond to one group. For example, if the ratio modes include 1:1:1 and 2:1:1, it indicates that the number of groups is 2.

[0123] Combined with Figure 5 as shown, in task grouping, assume there are red agents, blue agents, and protection points, divided into a total of groups. Each group contains several red agents (red), 1 blue agent (blue), and 1 target protection point (target). Under the condition of ensuring that all red agents have opponents and the total number of selected opponents is greater than the actual number of opponents, make the number of small-scale groups as large as possible.

[0124] Assume the number of the group is , then solve the following objective function:

[0125]

[0126]

[0127] This optimization problem ensures that all red agents have corresponding blue opponents and forms small-scale combat units as much as possible to achieve the balance between resources and the winning rate.

[0128] Optionally, the minimized objective function can also be replaced according to requirements. Here, the requirements are that for the red side, a small-scale team can achieve a relative balance between the winning rate and resources; for the blue side, only considering the most threatening red agents can achieve a relative balance between the winning rate and attention. The purpose of both sides is to conduct combat on as small a scale as possible.

[0129] When , it is obtained that ; in this case, n being 2 indicates that there are two mixing ratio modes of 1:1:1 and 2:1:1. The number of groups in the first mixing ratio mode is 20, and the number of groups in the second mixing ratio mode is 10.

[0130] When , it is obtained that ; in this case, n being 2 indicates that there are two mixing ratio modes of 1:1:1 and 2:1:1. The number of groups in the first mixing ratio mode is 3, and the number of groups in the second mixing ratio mode is 12.

[0131] When , it is obtained that ; in this case, n being 3 indicates that there are three mixing ratio modes of 1:1:1, 2:1:1, and 3:1:1. The number of groups in the first mixing ratio mode is 26, the number of groups in the second mixing ratio mode is 16, and the number of groups in the third mixing ratio mode is 0.

[0132] When , it is obtained that ; in this case, n being 3 indicates that there are three mixing ratio modes of 1:1:1, 2:1:1, and 3:1:1. The number of groups in the first mixing ratio mode is 0, the number of groups in the second mixing ratio mode is 2, and the number of groups in the third mixing ratio mode is 4.

[0133] It can be seen that the task assignment will arrange small-scale combat situations as much as possible. And when , the intelligence level and computing resources of the agents trained can achieve a better balance, which can support the second-level decision-making goal after task assignment.

[0134] After completing the group division and determining the number of groups within each group, it is necessary to determine the specific members within each group. In this embodiment, within each group of all groups, agents belonging to the same camp and agents belonging to different camps are selected for each agent, and the protection points within each group are determined. Specifically, it can be achieved through the following methods:

[0135] According to the position information of each red agent and selecting red agents belonging to the same camp according to the group division results, multiple groups are formed; based on the distances between each blue agent and the average positions of the red agents within each group, the blue agents to be added to each group are determined; for each group, based on the distances between the blue agents within the group and each protection point, the protection points to be added to each group are determined.

[0136] In this embodiment, the confrontation units on both sides operate independently to ensure that both the red and blue sides can complete the confrontation tasks independently. For the red side, it is ensured that all blue agents have red agents selected as opponents. If multiple red agents select the same opponent, the cooperation resources are organized according to the existing training model to carry out coordinated strikes on the blue agents. For the blue side, it is not necessary to consider all red agents. Only a few of the most threatening red agents need to be selected as the interception targets so as to bypass them and attack the target point. During the confrontation process, the composition of the confrontation units can be adjusted in real time according to the scene changes, and the task allocation of the large-scale cluster confrontation scene is realized by solving the small integer programming problem.

[0137] In the opponent selection, the red and blue agents can independently select opponents (agents from different camps). The principle for the red agents to select opponents is to try their best to select opponents they are good at dealing with and guard the nearest protection point. The principle for the blue agents to select opponents is to select several opponents that pose the greatest threat to them and select the protection point (the target point for the blue side) according to the difficulty of the attack.

[0138] If starting from the perspectives of the red agents and the blue agents respectively, the ways for the red agents to select opponents include: first, select teammates according to the initial positions of all red agents to form several groups, and the total number of groups is denoted as ; then, select blue opponents according to the distance between the blue agents and the groups; finally, select the protection point according to the distance between the blue agents and the protection point.

[0139] For the red agents , the set of its teammates is denoted as , then:

[0140]

[0141] Among them, is the number of teammates (agents of the same camp) required in the group where is located; represents the distance between ; represents the set of indices of the remaining red agents in the current step. When calculating for the first time, is the set of all red agent indices. Iterate according to this formula until is .

[0142] If satisfies the following formula, then is the blue agent opponent selected by :

[0143]

[0144] Among them, is the imaginary agent corresponding to the average position of the group where is and the average value of the distances from all agents in the group where represents the index set of the blue agents.

[0145] In when it is determined that if satisfies:

[0146]

[0147] then is the selected opponent of the blue agent. Among them, represents the index set of the protection points.

[0148] The way for the blue agent to select an opponent includes: first, select several red agents with the greatest threat as opponents according to the distance; then select protection points according to the distances between the blue agent and several opponents.

[0149] For the blue agent , its opponent set is denoted as , then:

[0150]

[0151] Among them, is the number of opponents.

[0152] If satisfies the following formula, then is the selected offensive protection point:

[0153]

[0154] Among them, is the imaginary agent corresponding to the average position of the group where is a constant selected according to the environment.

[0155] In this embodiment, it is assumed that only two groups are considered, that is, the ratio of red agents to blue agents is 1:1 and 2:1. That is to say, the blue agent can select at most two red agents as its decision range. Therefore, in this stage, it is necessary to select its own teammates, opponents, and protection points as the decision range to call the appropriate deep reinforcement learning algorithm for decision-making.

[0156] First, the teammate selection iteration process of the red agents is carried out in sequence according to the order of the agents and the number of red agents in each group. For the red agent in the group of red 2 and blue 1 , its teammate is denoted as , then:

[0157]

[0158] is equivalent to:

[0159]

[0160] And the red agent in the group of red 1 and blue 1 has no teammate, so the iteration ends.

[0161] Next, the red agent selects the decision range. Denote the selected opponent as , then satisfies:

[0162]

[0163] Next, when is determined, if satisfies:

[0164]

[0165] then is the selected blue agent opponent. Thus, a group of agents in the red side is all determined.

[0166] Finally, it is the blue agent's turn to select the decision range. Since the blue agent does not know the grouping information of the red agent, two red agents are uniformly selected as the decision range. For the blue agent , its opponent set is denoted as , then:

[0167]

[0168] Next, the protection point is selected according to the threat level of the red agent at this time. Here should not be too large relative to the size of the venue. Optionally, in the venue, meets the requirements of task allocation. Therefore, if satisfies:

[0169]

[0170] then is the selected offensive protection point.

[0171] After the group division and the determination of the members within each group are completed based on the above method, the trained deep reinforcement learning model can be called to guide each member within each group to perform adversarial simulation. The information including the current observation information and action information of each agent is input into the deep reinforcement learning model. After being processed by the deep reinforcement learning model, the action information for the next moment can be output to guide the agent to conduct confrontation. The action information mainly includes acceleration information. When guiding the simulation confrontation based on the deep reinforcement learning model, it can be achieved through the following methods:

[0172] In each time step of each simulation confrontation, the acceleration information obtained at the time step is used to guide each agent to move within the adversarial simulation environment; the confrontation results of the agents in different camps in multiple simulation confrontations are statistically analyzed, and the success rate and failure rate of the agents in different camps are analyzed.

[0173] Figure 6 The large-scale cluster confrontation effect under this solution is shown, and exemplarily, . According to the deep reinforcement learning and task assignment algorithm, the system can achieve intelligent decision-making at the second level, autonomously select opponents and conduct games. In this example, the blue agents penetrated through layers of blockades and successfully attacked the protection point, so the blue side finally won. However, the red side also achieved the interception of most blue agents and showed excellent performance. Under this scenario setting, the final winning rate of the red side stabilized at about 66%.

[0174] In summary, the solution of the present invention provides a deep reinforcement learning cluster confrontation method and system based on the attention mechanism, including the design of a reinforcement learning algorithm based on the attention mechanism in the case of small-scale combat, and the proposal of a method based on task assignment in the case of large-scale combat.

[0175] The purpose of this method is to achieve the efficient cooperative combat ability of agents in a dynamic and complex environment by integrating the attention mechanism and deep reinforcement learning technology. Specifically, this solution first designs an Actor-Critic network architecture integrating the attention mechanism for the small-scale combat scenario, which can allocate corresponding attention modules according to the characteristics of agents in different camps and types, thereby improving the decision-making efficiency and accuracy of agents in a highly dynamic environment. On this basis, the MADDPG algorithm is used to train the agents so that they can show high-level game strategies in confrontation scenarios such as one-on-one or two-on-two.

[0176] Furthermore, to meet the combat requirements of larger scales, this solution proposes a large-scale confrontation strategy based on task allocation. This strategy uses linear programming methods to solve the task grouping problem in large-scale scenarios, ensuring that each agent can effectively execute tasks according to the current battlefield situation. During the task allocation process, not only the distance between agents is considered, but also factors such as threat levels are comprehensively considered to achieve more reasonable and efficient resource allocation. In addition, through the dynamic cooperation between agents, the system can support decision-making responses at the second level, thus maintaining a high degree of flexibility and adaptability in complex battlefield environments.

[0177] Based on the same inventive concept, please refer to Figure 7 , the embodiment of the present invention also provides a schematic diagram of the functional modules of a deep reinforcement learning cluster confrontation system based on an attention mechanism. This embodiment can divide the functional modules of the deep reinforcement learning cluster confrontation system based on an attention mechanism according to the above method embodiment. For example, corresponding functional modules can be divided for each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0178] For example, in the case of dividing each functional module corresponding to each function, Figure 7 the deep reinforcement learning cluster confrontation system based on an attention mechanism shown is only a schematic diagram of a device. The deep reinforcement learning cluster confrontation system based on an attention mechanism may include a construction module, a training module, a division module, a processing module, and a simulation confrontation module. The functions of each functional module of the deep reinforcement learning cluster confrontation system based on an attention mechanism will be elaborated in detail below.

[0179] The construction module is used to construct a cluster confrontation model. The cluster confrontation model includes an agent cluster and an adversarial simulation scenario. The agent cluster includes multiple agents of different camps and different types. The adversarial simulation scenario includes an adversarial simulation environment and protection points.

[0180] The training module is used to select some agents and protection points in the cluster confrontation model to train a deep reinforcement learning model embedded with an attention module.

[0181] The division module is used to group all the agents in the agent cluster and select agents of the same camp and agents of different camps for each agent within each group of all groups, and determine the protection points within each group.

[0182] A processing module, configured to input the agents and protection points within all groups into the trained deep reinforcement learning model, and output the action information of each said agent;

[0183] A simulation confrontation module, configured to guide each said agent to perform simulation confrontation within the confrontation simulation environment according to the action information.

[0184] It can be understood that the above-mentioned construction module, training module, partitioning module, processing module and simulation confrontation module can be used to execute the above S11 to S15. The detailed implementation manners of the construction module, training module, partitioning module, processing module and simulation confrontation module can refer to the content related to the above S11 to S15.

[0185] The deep reinforcement learning cluster confrontation system based on the attention mechanism provided in this embodiment has the same, similar or corresponding technical features as the deep reinforcement learning cluster confrontation method based on the attention mechanism in the above embodiment, and has the same technical effects. For the relevant content of this system, reference can be made to the description related to the above method, and this embodiment will not be elaborated here.

[0186] Please refer to Figure 8 , which is a structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device may be a computer device or the like. The electronic device includes a memory, a processor and a communication module. Each element of the memory, the processor and the communication module is directly or indirectly electrically connected to each other to realize data transmission or interaction. For example, these elements may be electrically connected to each other through one or more communication buses or signal lines.

[0187] Among them, the memory is used to store computer programs or data. The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), etc.

[0188] The processor is configured to read / write the data or program stored in the memory, and execute the deep reinforcement learning cluster confrontation method based on the attention mechanism provided in any embodiment of the present invention.

[0189] The communication module is configured to establish a communication connection between the electronic device and other communication terminals through a network, and is configured to receive and send data through the network.

[0190] It should be understood that Figure 8 the structure shown is only a schematic diagram of the structure of the electronic device, and the electronic device may further include more or fewer components than those shown Figure 8 in it, or have a configuration different from that shown Figure 8 in it.

[0191] Furthermore, an embodiment of the present invention further provides a computer-readable storage medium storing machine-executable instructions, and when the machine-executable instructions are executed, the depth reinforcement learning cluster confrontation method based on the attention mechanism provided in the above embodiment is implemented.

[0192] Specifically, the computer-readable storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the computer-readable storage medium runs, it can execute the above-mentioned depth reinforcement learning cluster confrontation method based on the attention mechanism. Regarding the process involved when the machine-executable instructions in the computer-readable storage medium run, reference can be made to the relevant descriptions in the above method embodiments, and details are not described here again.

[0193] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0194] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any reference signs in the claims should not be regarded as limiting the claims involved.

Claims

1. A deep reinforcement learning cluster confrontation method based on an attention mechanism, characterized in that To achieve the efficient cooperative combat ability of agents in a dynamic and complex environment, the method includes: Construct a cluster confrontation model, which includes an agent cluster and a confrontation simulation scenario. The agent cluster includes multiple agents of different camps and different types. The confrontation simulation scenario includes a confrontation simulation environment and protection points. The agents are robot agents, UAV agents or network node agents; Select some agents and protection points in the cluster confrontation model to train a deep reinforcement learning model embedded with an attention module; Based on all the agents in the agent cluster, group them, and select agents of the same camp and agents of different camps for each agent within each group of all groups, and determine the protection points within each group; Input the agents and protection points within all groups into the trained deep reinforcement learning model to output the action information of each agent. Among them, input the current observation information and action information of each agent into the deep reinforcement learning model to output the action information of the next moment. The action information includes acceleration information. The attention module in the deep reinforcement learning model dynamically adjusts the degree of attention of each agent to the state information of other agents; Use the action information to guide each agent to perform simulation confrontation in the confrontation simulation environment. Among them, in each time step of each simulation confrontation, use the acceleration information obtained at the time step to guide each agent to move in the confrontation simulation environment, count the confrontation results of agents of different camps in multiple simulation confrontations, and analyze the success rate and failure rate of agents of different camps; The step of selecting some agents and protection points in the cluster confrontation model to train a deep reinforcement learning model embedded with an attention module includes: For some agents in the cluster confrontation model, construct the embedding representation of the agent based on the Embed function, and input the embedding representation into the attention module of the deep reinforcement learning model; Based on the attention module, output the embedding representations of other agents corresponding to each agent, and splice and process the embedding representation of the agent and the embedding representations of other agents in the fully connected layer to obtain decision-making information; Train the deep reinforcement learning model based on the loss function constructed for each agent and the decision-making information until the preset stop condition is met and then stop training.

2. The method for cluster confrontation of deep reinforcement learning based on the attention mechanism according to claim 1, wherein The step of constructing the embedding representation of the agent based on the Embed function includes: Use the fully connected layer in the deep reinforcement learning model to process the observation information and action information of the agent respectively; Splice the processing results of the observation information and action information with the attention output results of the agents of the same camp, agents of different camps and the protection points of the agent; Use the fully connected layer to process the splicing result to obtain the embedding representation of the agent.

3. The method for cluster confrontation of deep reinforcement learning based on attention mechanism according to claim 1, characterized in that, The step of outputting the embedding representations of other agents corresponding to each agent based on the attention module includes: For each of the agents, obtain the embedding representations of the other agents corresponding to the agent based on the attention module; Obtain the attention weights of the other agents based on a scoring function; Accumulate the embedding representations of all other agents based on the attention weights of the other agents to obtain the embedding representation of the other agents corresponding to the agent.

4. The method for cluster confrontation of deep reinforcement learning based on attention mechanism according to claim 3, characterized in that The step of obtaining the attention weights of the other agents based on a scoring function includes: Obtain the degree of attention of the agent to each of the other agents based on a scoring function; For each of the other agents, obtain the attention weight of the other agent according to the degree of attention of the agent to the other agent and the cumulative value of the degrees of attention of the agent to all other agents.

5. The method for cluster confrontation of deep reinforcement learning based on attention mechanism according to claim 1, characterized in that, The agents include red agents and blue agents belonging to different camps; The step of grouping all the agents in the agent cluster includes: Obtain the numbers of red agents and blue agents in the selected agent cluster; Determine the number of groups and the number of subgroups in each group based on the set agent ratio pattern within the group and the numbers of red agents and blue agents.

6. The method for cluster confrontation of deep reinforcement learning based on attention mechanism according to claim 5, characterized in that The step of determining the number of groups and the number of subgroups in each group based on the set agent ratio pattern within the group and the numbers of red agents and blue agents includes: Determine the number of groups based on the set agent ratio pattern within the group; Minimize the constructed objective function based on the number of groups, the numbers of red agents and blue agents to determine the number of subgroups in each group.

7. The method for cluster confrontation based on attention mechanism and deep reinforcement learning according to claim 5, characterized in that The step of selecting agents belonging to the same camp and agents belonging to different camps for each agent in each subgroup of all groups and determining the protection points in each subgroup includes: Select red agents belonging to the same camp according to the position information of each red agent and the grouping result to form multiple subgroups; Determine the blue agents to be added to each subgroup based on the distances between each blue agent and the average positions of the red agents in each subgroup; For each subgroup, determine the protection points to be added to each subgroup based on the distances between the blue agents in the subgroup and each protection point.

8. A deep reinforcement learning cluster confrontation system based on an attention mechanism, characterized in that A system for implementing the attention mechanism-based deep reinforcement learning cluster confrontation method according to any one of claims 1-7, the system includes: A construction module for constructing a cluster confrontation model, the cluster confrontation model including an agent cluster and an adversarial simulation scenario, the agent cluster including multiple agents of different camps and different types, and the adversarial simulation scenario including an adversarial simulation environment and protection points; A training module for selecting some agents and protection points in the cluster confrontation model to train a deep reinforcement learning model embedded with an attention module; A division module, configured to perform group division based on all agents in the agent cluster, select agents belonging to the same camp and agents belonging to different camps for each agent within each subgroup of all groups, and determine protection points within each subgroup; A processing module, configured to input the agents and protection points within all subgroups into a trained deep reinforcement learning model, and output action information of each agent; A simulation confrontation module, configured to guide each agent to perform simulation confrontation within the confrontation simulation environment based on the action information.

Images