Deep reinforcement learning method and system for swarm intelligence capture the flag game

Abstract

The application discloses a kind of deep reinforcement learning method and system for group wisdom flag capture game, solve the problem of group wisdom path planning and flag capture under the condition of external competition, internal cooperation of imperfect information.Specifically, first, the picture features are extracted based on the split channels of convolutional neural network;Then, based on the graph attention network, the competitive relationship between agents under the condition of imperfect information is quantitatively determined according to the observed information and the received teammate observation information, so that the agent fully considers the state of other agents when making decisions;Finally, according to the attention value of different agents on the field, the multi-agent double duel deep Q network is assisted to realize the path planning and autonomous navigation of multi-agent in a two-dimensional maze environment, optimize the group wisdom flag capture strategy, to realize fast and accurate two-to-two flag capture.

Description

TECHNICAL FIELD

[0001] The present application relates to a crowd wisdom capture-the-flag game strategy end-to-end solution method and system, in particular to multi-agent deep reinforcement learning path planning and graph attention network quantization game competition relationship. BACKGROUND

[0002] Game is the strategy optimization process of participants in cooperative or competitive behavior. When game theory is introduced into multi-agent system, with the widespread use of basic perception units such as sensor modules and intelligent wearable devices, through Internet collaboration, efficient data collection and task collaborative planning in multi-agent system are realized, making it possible to complete the crowd wisdom attack and defense task in complex battlefield environment. Under this background, by constructing and solving the game model, the success rate of intelligent agent accurate attack and defense in high confrontation and high dynamic battlefield environment is improved.

[0003] The attack and defense game problem can be modeled as a capture-the-flag game model, in which the attacker achieves the capture of the defender's target, and the defender tries to destroy the attacker to prevent the attack. When the game is low-dimensional, existing literature (for example, Huang H, Ding J, Zhang W, et al. Automation-assisted capture-the-flag: A differential game approach. IEEE Trans Control Syst Technol, 2014, 23: 1014-1028) often tries to build a differential game model to solve the optimal strategy of 1 vs. 1 capture-the-flag game in an analytical form. However, when the capture-the-flag game is extended to the case of multi-player game in complex battlefield environment, the optimal strategy of the intelligent agent often depends on the action selection of other intelligent agents, causing problems such as explosion of partial differential equation dimension, which makes it impossible to analytically solve the optimal strategy of crowd wisdom capture-the-flag game.

[0004] Convolutional neural networks can extract features of input images, but from the perspective of analyzing the state of each step of the intelligent agent, it is impossible to quantitatively determine whether the decision of the intelligent agent at the next moment is biased towards cooperation or competition, thereby affecting the formulation of the final multi-agent capture-the-flag game strategy.

[0005] Deep reinforcement learning algorithms can solve the path planning problem in the space with a large number of obstacles under the input of battlefield environment images, but for multi-agent systems, especially multi-agent systems with competition and cooperation, different cooperative agents and competitive agents have different reward functions, so it is impossible to use the same deep reinforcement learning network to search for strategies. SUMMARY

[0006] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0007] The purpose of the present application is to solve the above problems, provide a deep reinforcement learning method and system for group intelligence capture the flag game, solve the problem of overcoming imperfect observation information in strong confrontation and high dynamic battlefield environment, and realize group intelligence efficient attack and defense capture the flag.

[0008] The technical scheme of the present application is: the present application discloses a deep reinforcement learning method for group intelligence capture the flag game, the method comprises:

[0009] Step 1: Abstract the complex battlefield environment into a two-dimensional maze simulation environment including multiple intelligent agents;

[0010] Step 2: Extract visual features of the input image based on the convolutional neural network channel by channel;

[0011] Step 3: Use the reinforcement learning double duel deep Q network to initialize the corresponding network for the intelligent agent of the subunit respectively;

[0012] Step 4: According to the task goal of group intelligence capture the flag, construct the reward function, and set the target to be reached according to the capture the flag rule abstracted from the battlefield environment;

[0013] Step 5: Add a graph attention network to the double duel deep Q network structure;

[0014] Step 6: Set the maximum iteration round;

[0015] Step 7: According to the observation information of the intelligent agent at this moment and the communication information with the teammates, calculate the attention value between each intelligent agent at each step, and the intelligent agent selects the attack / defense action according to the attention value;

[0016] Step 8: According to the selected reward function, execute the action, and obtain the experience replay four-tuple;

[0017] Step 9: Obtain the new environment partial observation, update the experience replay four-tuple, and calculate the double duel deep Q network target value;

[0018] Step 10: Repeat steps 7-9, update the double duel deep Q network, until the iteration number reaches the maximum iteration number, realize the group intelligence capture the flag game in the maze environment.

[0019] According to an embodiment of the deep reinforcement learning method for the crowd wisdom capture the flag game, step 1 further comprises:

[0020] define N agents, including a set of local observations a set of actions and a set of states S and state transition function for each agent i, the local observation obtained by the basic perception unit

[0021] According to an embodiment of the deep reinforcement learning method for the crowd wisdom capture the flag game, step 2 further comprises:

[0022] Add a residual network module after the convolutional neural network, and connect the features of the convolutional neural network and the residual network module connected in front and back together, for reducing overfitting.

[0023] According to an embodiment of the deep reinforcement learning method for the crowd wisdom capture the flag game, step 4 further comprises:

[0024] The goal to be achieved is that the agent of our side captures the flag of the enemy agent under the condition that the agent of our side is not killed by the enemy agent.

[0025] The application also discloses a deep reinforcement learning system for a crowd wisdom capture the flag game, the system comprising:

[0026] The simulation module is configured to abstract the complex battlefield environment into a two-dimensional maze simulation environment including a plurality of agents;

[0027] The feature extraction module is configured to extract visual features of the input image based on a convolutional neural network;

[0028] The initialization module is configured to initialize the corresponding network of the agent of the subunit using a reinforcement learning double duel deep Q network;

[0029] The reward function construction module is configured to construct a reward function according to the crowd wisdom capture the flag task goal, and set the goal to be achieved according to the capture the flag rule abstracted from the battlefield environment;

[0030] The graph attention network module is configured to add a graph attention network in the double duel deep Q network structure;

[0031] The maximum iteration round setting module is configured to set the maximum iteration round;

[0032] The attention value calculation module is configured to calculate the attention value between each agent at each step according to the observation information of the agent at the moment and the communication information with the teammates, and the agent selects the attack / defense action according to the attention value;

[0033] The experience replay quadruple obtaining module is configured to obtain an experience replay quadruple by performing an action according to the selected reward function;

[0034] The network target value calculating module is configured to obtain a new environment partial observation, update the experience replay quadruple, and calculate a double duel deep Q network target value;

[0035] The attention value calculating module, the experience replay quadruple obtaining module and the network target value calculating module are repeatedly executed to update the double duel deep Q network until the number of iterations reaches the maximum number of iterations, so that the crowd intelligence flag capture game in the maze environment is realized.

[0036] According to an embodiment of the deep reinforcement learning system for the crowd intelligence flag capture game, the simulation module is further configured to:

[0037] define N agents, which include a set of local observations a set of actions and a set of states S and a state transition function for each agent i, obtain the local observation through a basic perception unit

[0038] According to an embodiment of the deep reinforcement learning system for the crowd intelligence flag capture game, the feature extraction module is further configured to:

[0039] A residual network module is added after the convolutional neural network, and features of the convolutional neural network and the residual network module connected in front and back are connected together, so as to reduce overfitting.

[0040] According to an embodiment of the deep reinforcement learning system for the crowd intelligence flag capture game, the reward function construction module is further configured to:

[0041] The target to be achieved is that the agent of our side captures the flag of the enemy agent under the condition that the agent of our side is not killed by the enemy agent.

[0042] The present application has the following beneficial effects compared with the prior art: first, the double duel deep Q network algorithm is used to train multiple agents in the present application, which solves the problem of network deviation in path planning in a complex maze environment, ensures that the agent can obtain significant rewards whether or not the state is transferred, and thus realizes more accurate state estimation and improves the training efficiency. Second, the graph attention network is used to focus on the most relevant agent in the flag capture game process in the present application, which quantifies the mutual relationship between agents in real time and assists the double duel deep Q network algorithm in decision-making, efficiently optimizing the flag capture strategy of multiple agents. Third, the path planning and autonomous navigation strategy of multiple agents is trained end-to-end in the present application in view of external competition and internal cooperation, realizing the flag capture game of multiple agents under imperfect information in a two-dimensional maze environment. Attached Figure Description

[0043] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0044] Figure 1 This is a schematic diagram of a capture-the-flag maze environment for the intelligent agents.

[0045] Figure 2 This is the overall network framework involved in an embodiment of the deep reinforcement learning method for capture-the-flag game of the present invention.

[0046] Figure 3 A graph showing the change in attention value during the capture-the-flag game.

[0047] Figure 4 A schematic diagram of an embodiment of the deep reinforcement learning system for capture-the-flag game of the present invention is shown. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0049] like Figure 1 As shown, the capture-the-flag game framework in this embodiment suffers from imperfect observation information for the agents. The perfect information an agent can obtain at each step is its own position relative to the flag; imperfect information includes non-friendly information outside the agent's field of vision on the maze map. Taking agent 1 of team A as an example, its observation range is limited, such as... Figure 1 The area within the center circle is shown. When the enemy flag is within the observation range, the agent explores with that flag as its target; when the enemy flag is not within the observation range, the agent explores randomly. After successfully capturing a flag, if the entire observation range consists of enemy territory, the agent explores randomly until it reaches the boundary of its own territory; if there is friendly territory within the observation range, its target is the set of points on the boundary line of its own territory within the observation range, such as... Figure 1 As shown in the shaded area, the agent continuously optimizes its target while evading enemy pursuit, selecting a target point that is both safe to reach and relatively close, in order to prompt the agent to quickly return with the flag and score.

[0050] like Figure 2 As shown in the figure, this embodiment provides a deep reinforcement learning method for capture-the-flag game with collective intelligence, and its specific steps are as follows.

[0051] Step 1: Abstract the complex battlefield environment into a two-dimensional maze simulation environment, define N agents, which contain a set of local observations A set of actions And a set of states S and state transition functions For each agent i, the local observation obtained by the basic perception unit

[0052] In step 1, the maze map is center-symmetric, and A and B teams are stationed in the left and right half regions respectively. It is assumed that the A and B team agents are stationed (initial starting position) at the lower left and upper right corners of the map respectively. The agents of the same team can keep communication through the Internet, and the observable range of the agents of different teams is within the constraint of the maze environment, that is, the range with the agent as the center and the radius of 5 Euclidean distance. The action set refers to the combination of 5*5=25 actions of the action set of all agents of the same team. Taking the A team as an example, the action set of agent 1 {up1, down1, left1, right1, stop1} and the action set of agent 2 {up2, down2, left2, right2, stop2} are combined. The goal is to capture the flag in the enemy area. Specifically, when the agent is located in the home map, the goal of this stage is to protect the home flag while killing the invading enemy agents (the two agents are located at the same coordinate position, which means killing. When the agent is killed, it is revived at the fixed station); when the agent is located in the enemy map, the goal of this stage is to capture the enemy flag while avoiding the killing of the enemy agent. The moving speed of the agent is the same, and the action is selected from the action set at each step, and the game ends when the agent moves to 1200 steps. Under the constraint of the two-dimensional maze map environment, after the agent captures the flag, it is considered to be successful only when it successfully returns to the home territory (scores one point), and the team with the highest final score wins. If the final score is zero, the game ends in a draw.

[0053] Step 2: Feature extraction step, extract the visual features of the input image. It is assumed that the panorama of the maze map is FOV, and since the agent can only observe the environmental information within a certain distance of the field of view, the observation information of the agent is divided into 3 channels, representing the maze map, the agent and the flag respectively.

[0054] Among them, a residual network module is added after the convolutional neural network, which connects the features of the convolutional neural network and the residual network module connected before and after together, which is used to reduce overfitting and improve performance. Specifically, a 3-stack ResNet-101 feature extractor is implemented, the stride of the first convolutional block in the convolutional layer conv5 stage is modified from 2 to 1 to improve the resolution of the feature map, and finally the image feature map of each step is obtained.

[0055] Step 3: Initialize the respective networks for the agents of the two teams using the reinforcement learning double dueling deep Q network, such as initializing two networks for the agents of the two teams respectively. Follow the policy Select an action from the set of agent actions with the action probability and, through interaction with the maze environment, transition to the next state and obtain the reward to determine the goodness of the flag capture strategy. The initial state is determined by the initial state distribution ρ: S → [0, 1]. Each agent i tries to maximize the cumulative discounted reward where T is the expected time range, γ ∈ [0, 1] is the discount parameter.

[0056] where the reinforcement learning double dueling deep Q network is trained from scratch using Nvidia Quadro RTX 8000 GPU to convergence, the weights of the evaluation network are initialized as θ, θ', the memory D is set to store experience replay, the maximum size of the replay buffer is N r , the training batch size is N b′ , and the target network replacement frequency is N rf ; the target distance threshold d arrive , the collision distance threshold d collision , and the number of rounds M are set.

[0057] The duel deep Q network estimates the state value and the advantage of each action by setting the advantage function as the difference between the advantage function of a single action and the average advantage function of all actions, and then calculates the final Q value:

[0058]

[0059] where θ is the weight, α, β are the parameters of the main and auxiliary deep Q networks in the double deep Q network, s t , a t represent the state and action of the agent at time t respectively.

[0060] Step 4: Construct the reward function according to the goal of the crowd wisdom flag capture task. According to the flag capture rules abstracted from the battlefield environment, the following goals need to be achieved: the friendly agent captures the enemy flag without being killed by the enemy agent.

[0061] where the specific reward function design formula is as follows:

[0062] If the attack action is encouraged, the reward r offensive contains the reward for the agent being killed by the enemy agent: r1 = -30; the reward for the agent capturing the enemy flag: r2 = 300; the boundary reward obtained by the agent after capturing the flag: When the flag is captured, the agent gets a distance reward:

[0063] If the defensive action is encouraged, the reward r defensive The reward for the agent killing an enemy agent: r5=30; the reward for the flag being captured: r6=-300; the boundary reward obtained by the agent after the flag is captured: The distance reward obtained by the agent after the flag is captured:

[0064] In addition, the general exploration map reward r else The reward for the agent obtaining an enemy guide: r9=10, the reward for the agent touching the maze obstacle: r 10 =-500, and the reward for the agent moving each step: r 11 =-5.

[0065] where d enemy represents the distance between the agent and the enemy agent; d boundary represents the distance between the agent and the A and B team map boundaries.

[0066] Step 5: Add a graph attention network to the double duel deep Q network structure.

[0067] The graph attention network can process the local observation values of all agents, thereby quantitatively determining the competition relationship between the agents, which helps to pay attention to the enemy agents that constitute a threat in time, thereby avoiding being killed. The specific calculation process is as follows:

[0068] First, a shared parameterized matrix W∈R F is used, where F represents the feature dimension of the feature vector set, the feature vector of the target agent i is h i , and the feature vectors of the neighborhood agents {j∈N i} are h j . The process of calculating the attention weight between agents in the maze map is as follows:

[0069]

[0070] where e ij is the contribution degree of the agent o j in the maze map to the agent o i , is an attention coefficient calculation network. Then, the softmax function is used to standardize all neighborhood agents of each agent, thereby obtaining the standardized attention coefficient a ij , so that the attention coefficients between different agents are easy to compare, and the calculation process is as follows:

[0071]

[0072] where N i represents all the neighboring agents of the target agent i (including i itself). The a ij corresponding linear combination of the agent features as the final output of each agent, i.e.

[0073]

[0074] where h′ i contains the feature information of all other agents in the maze map, so the output of the abstract vector of different agents represents different, regardless of whether the agent class is the same, so it can be used to predict the position of different agents.

[0075] Step 6: Set the maximum iteration round.

[0076] where the maximum iteration round is 1200.

[0077] Step 7: According to the observation information of the agent at this moment and the communication information with the teammates, the attention value between each agent at each step is calculated. According to the attention value, the agent selects the action a t .

[0078] where the calculation process of the attention value between each agent at each step is as follows:

[0079] The graph attention layer trains a parameterized matrix to perform feature transformation on the agent vector, and obtains the shared parameterized matrix W ∈ R F and the attention coefficient calculation network Let the observation vector of agent i to the surrounding environment at time t be After extracting the features, pass them to the graph attention network to obtain the normalized attention coefficient a ij of agent j to agent i, thereby enhancing the features extracted from the local observation of the agent. In addition, using a multi-layer perception, the dimension of the features extracted in the dimension reduction convolutional neural network is reduced, which effectively reduces the feature dimension that can be shared by the graph attention network. Specifically, at each step, the distance d t of the agent to the target at time t, part of the observation o t of the environment, and the attention weight a ij of all agents are obtained, then according to a ij , the next step action bias attack / defense is selected. The agent selects the bias action a t with a probability of ε, otherwise, and update d t with the new depth information.

[0080] Step 8: Perform the action according to the selected reward function, and obtain the experience replay quadruple.

[0081] wherein, under the condition of d t ≤ d arrive and d t > d collision , if the agent is in an offensive state, the reward is: r(o t , a t ) = c(d t-1 -d t )+r offensive +r9+r 11 ; if the agent is in an offensive state, the reward at this time is: r(o t , a t ) = c(d t-1 -d t )+r defensive +r9+r 11 . Under the condition of d t > d arrive and d t > d collision , the reward is: r(o t , a t ) = r9+r 11 . Under other conditions, the reward is: r(o t , a t ) = r 10 +r 11 .

[0082] After that, the agent sets the memory D to store the experience replay, obtains the experience replay quadruple, and stores the local buffer B t and the experience pool M t of n time experience replay quadruples (o t , a t+1 , r u , o u ). After the agent obtains the reward, the experience replay quadruple (o t , a t , r t , o t+1 ) is stored in the buffer B i in chronological order.

[0083] Step 9: Obtain a new environment partial observation o t+1 , update the experience replay quadruple, and calculate the double duel deep Q network target value.

[0084] wherein, the process of the agent updating the experience replay quadruple and calculating the double duel deep Q network target value is as follows:

[0085] First, the agent obtains a new environment partial observation o t+1 , the replay buffer maximum size is set to N r , the training batch size is N b′ , and the target network replacement frequency is N rf . The transition group (o t , a t , r t , o t+1 ) is stored in the memory D, when |D|≥N r , the old tuple is replaced. Then, a batch of N b′ transition groups (o k , a k , r k , o k+1 ) is randomly selected in the memory D. On this basis, the double duel deep Q network target value y k is calculated using the time difference, a batch of N b′ transition groups (o k , a k , r k , o k+1 ) is randomly selected in the memory D,

[0086] For each transition group:

[0087]

[0088] y k =r k +γQ(o k+1 , a max (o k+1 |θ)|θ′)

[0089] wherein γ is a discount factor, taking a value of 0.9, and y k is updated by gradient descent with respect to (y k -Q(o, a|θ)) 2 loss.

[0090] Step 10: Steps 7-9 are repeatedly performed to update the double duel deep Q network until the number of iterations reaches the maximum number of iterations.

[0091] After step 10 is completed, the crowd intelligence flag capture game in the maze environment can be implemented, and the agent takes an action of biasing cooperation / competition according to the local observation range and the exploration of the maze map at each step, and finally quickly and accurately captures the flag while avoiding being attacked by the enemy.

[0092] For example Figure 3As shown, the embodiment shows the change of attention value in the process of one flag capture game. Taking agent 1 of team A as an example, since the standard attention value is obtained by solving the graph attention network, the attention value of the multi-agent satisfies a 12 +a 13 +a 14 =1 at each step. When the flag capture starts, the agents of the two teams start from the base. Due to the limitation of the observation range of the agent, enemy agents appear in the observation range of agent 1 at about 150 steps. Therefore, before the encounter, a 12 =1, a 13 =a 14 =0. For the agents 1 and 2 of the same team, when a 12 =1, agent 1 only cooperates with agent 2 to plan the path; when a 12 ≠1, agent 1 not only cooperates with agent 2 to plan the path, but also competes with the agents of the other team to avoid being killed. Similarly, for the agents 1 and 3 of the other team, when a 13 =1, agent 1 only competes with agent 3; when a 13 ≠1, agent 1 not only competes with agent 3, but also cooperates with the agents of the same team. Therefore, the graph attention network can not only quantify the situation of cooperation and competition of the agents in the maze map under imperfect information, but also assist the agents in optimizing the flag capture game strategy, thereby improving the success rate of flag capture.

[0093] Figure 4 The principle of one embodiment of the deep reinforcement learning system for crowd wisdom flag capture game of the application is shown. Please refer to Figure 4 , the system of the embodiment includes a simulation module, a feature extraction module, an initialization module, a reward function construction module, a graph attention network module, a maximum iteration round setting module, an attention value calculation module, an experience replay four-tuple acquisition module, and a network target value calculation module.

[0094] The simulation module is configured to abstract the complex battlefield environment into a two-dimensional maze simulation environment including a plurality of agents. The simulation module is further configured to define N agents, including a set of local observations a set of actions and a set of states S and a state transition function For each agent i, the local observation

[0095] The specific implementation content of the simulation module is the same as step 1 of the above method embodiment, which will not be repeated here.

[0096] The feature extraction module is configured to extract visual features of the input image based on a convolutional neural network.

[0097] The specific implementation of the feature extraction module is the same as step 2 of the method embodiment described above, and will not be repeated here.

[0098] The initialization module is configured to initialize the corresponding network for the agent of the team using the reinforcement learning double duel deep Q network.

[0099] The specific implementation of the initialization module is the same as step 3 of the method embodiment described above, and will not be repeated here.

[0100] The reward function construction module is configured to construct a reward function according to the goal of the crowd wisdom flag capture task, and set the target to be achieved according to the flag capture rules abstracted from the battlefield environment. The reward function construction module is further configured to: the target to be achieved is that the agent of the own side captures the flag of the enemy side without being killed by the agent of the enemy side.

[0101] The specific implementation of the reward function construction module is the same as step 4 of the method embodiment described above, and will not be repeated here.

[0102] The graph attention network module is configured to add a graph attention network in the double duel deep Q network structure. The specific implementation of the graph attention network module is the same as step 5 of the method embodiment described above, and will not be repeated here.

[0103] The maximum iteration round setting module is configured to set the maximum iteration round. The specific implementation of the maximum iteration round setting module is the same as step 6 of the method embodiment described above, and will not be repeated here.

[0104] The attention value calculation module is configured to calculate the attention value between each agent at each step according to the observation information of the agent at that moment and the communication information with the teammates, and the agent selects the attack / defense action according to the attention value.

[0105] The specific implementation of the attention value calculation module is the same as step 7 of the method embodiment described above, and will not be repeated here.

[0106] The experience replay four-tuple acquisition module is configured to acquire the experience replay four-tuple according to the selected reward function and the action.

[0107] The specific implementation of the attention value calculation module is the same as step 8 of the method embodiment described above, and will not be repeated here.

[0108] The network target value calculation module is configured to obtain a new environment part observation, update the experience replay four-tuple, and calculate a double duel deep Q network target value.

[0109] The specific implementation content of the network target value calculation module is the same as step 9 of the method embodiment described above, and will not be described here again.

[0110] The attention value calculation module, the experience replay four-tuple acquisition module, and the network target value calculation module are repeatedly executed to update the double duel deep Q network until the number of iterations reaches the maximum number of iterations, and the crowd intelligence capture-the-flag game in the maze environment is realized. The agent takes the action of biasing cooperation / competition according to the local observation range combined with the exploration of the maze map at each step, and finally quickly and accurately captures the flag while avoiding being attacked by the enemy.

[0111] Although the above methods are illustrated and described as a series of actions for the sake of simplicity of explanation, it should be understood and appreciated that the methods are not limited by the order of the actions, because according to one or more embodiments, some actions can occur in different orders and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but can be appreciated by those skilled in the art.

[0112] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality, without limitation. Whether such functionality is implemented in hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0113] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these designed to perform the functions described herein. The general-purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0114] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0115] In one or more exemplary embodiments, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0116] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A deep reinforcement learning method for group-based capture-the-flag games, characterized in that, The methods include: Step 1: Abstract the complex battlefield environment into a two-dimensional maze simulation environment that includes multiple intelligent agents; Step 2: Extract visual features of the input image by channel based on a convolutional neural network; Step 3: Use a reinforcement learning dual-duel deep Q network to initialize the corresponding networks for the agents in each team. This includes following the policy to obtain the action probability of selecting actions from the agent's action set, and transitioning to the next state and obtaining rewards through interaction with the maze environment to judge the quality of the capture-the-flag strategy. The initial state is determined by the initial state distribution. Step 4: Construct a reward function based on the objectives of the capture-the-flag mission, and set the objectives to be achieved based on the capture-the-flag rules abstracted from the battlefield environment; Step 5: Add a graph attention network to the dual-duel deep Q-network structure; Step 6: Set the maximum number of iterations; Step 7: Based on the agent's observation information and communication information with teammates at the current moment, calculate the attention value between agents at each step, and the agent selects an offensive / defensive action based on the attention value; Step 8: Execute actions according to the selected reward function, obtain experience replay quadruplets, where the agent sets memory to store experience replays, obtains experience replay quadruplets, and the experience replay quadruplets store n time points; Step 9: Acquire new environmental observations, update the experience replay quadruple, and calculate the dual duel depth Q-network objective value; Step 10: Repeat steps 7-9 to update the dual-battle deep Q network until the maximum number of iterations is reached, thus realizing the capture-the-flag game in a maze environment.

2. The deep reinforcement learning method for capture-the-flag game according to claim 1, characterized in that, Step 1 further includes: Define N agents, each containing a set of local observations. A set of actions and a set of states S and state transition functions For each agent i, the local observations obtained through the basic sensing unit 3. The deep reinforcement learning method for capture-the-flag game according to claim 1, characterized in that, Step 2 further includes: Adding a residual network module after the convolutional neural network connects the features of the convolutional neural network and the residual network module, thereby reducing overfitting.

4. The deep reinforcement learning method for capture-the-flag game according to claim 1, characterized in that, Step 4 further includes: The objective is for our agent to capture the enemy's flag without being killed by the enemy agent.

5. A deep reinforcement learning system for group capture-the-flag games, characterized in that the system... include: The simulation module is configured to abstract a complex battlefield environment into a two-dimensional maze simulation environment that includes multiple intelligent agents. The feature extraction module is configured to extract visual features of the input image by channel based on a convolutional neural network; The initialization module is configured to use a reinforcement learning dual-duel deep Q-network to initialize the corresponding networks for the agents in each team. This includes following the policy to obtain the action probability of selecting an action from the agent's action set, and interacting with the maze environment to transition to the next state and obtain a reward to judge the quality of the capture-the-flag strategy. The initial state is determined by the initial state distribution. The reward function construction module is configured to construct a reward function based on the objective of the capture-the-flag mission, and set the target to be achieved based on the capture-the-flag rules abstracted from the battlefield environment. The graph attention network module is configured to incorporate a graph attention network into a dual-duel depth Q-network structure; The maximum iteration rounds setting module is configured to set the maximum iteration rounds. The attention value calculation module is configured to calculate the attention value between agents at each step based on the agent's observation information and communication information with teammates at the current moment. The agent selects an offensive / defensive action based on the attention value. The experience replay quadruple acquisition module is configured to execute actions based on the selected reward function to acquire experience replay quadruples. The agent sets up memory to store experience replays and acquires experience replay quadruples, which store n time points. The network objective value calculation module is configured to acquire new environmental partial observations, update the experience replay quadruple, and calculate the dual duel depth Q network objective value. The attention value calculation module, experience replay quadruple acquisition module, and network target value calculation module are repeatedly executed to update the dual duel deep Q network until the maximum number of iterations is reached, thus realizing a capture-the-flag game in a maze environment.

6. The deep reinforcement learning system for capture-the-flag game according to claim 5, characterized in that, The simulation module is further configured as follows: Define N agents, each containing a set of local observations. A set of actions and a set of states S and state transition functions For each agent i, the local observations obtained through the basic sensing unit 7. The deep reinforcement learning system for capture-the-flag game according to claim 5, characterized in that, The feature extraction module is further configured as follows: Adding a residual network module after the convolutional neural network connects the features of the convolutional neural network and the residual network module, thereby reducing overfitting.

8. The deep reinforcement learning system for capture-the-flag game according to claim 5, characterized in that, The reward function constructor module is further configured as follows: The objective is for our agent to capture the enemy's flag without being killed by the enemy agent.

Images