A method for acquiring cooperative strategies in war game simulation based on deep reinforcement learning model
By building a multi-agent system in a war chess deduction environment, using QMIX algorithm combined with sub-target reward shaping, the reward sparseness problem in multi-agent reinforcement learning is solved, and efficient learning and strategy acquisition of the agent in a war chess environment is achieved.
Patent Information
- Application Number
- CN202210270021.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-18
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-03-18
AI Technical Summary
In the war game deduction environment, it is difficult for multiple agents to guide agents to learn better learning strategies, mainly due to the problem of sparse rewards, which leads to difficulty in training and slow convergence.
Using a method based on deep reinforcement learning model, a multi-agent system environment is built, and a decision model is generated in combination with preset tuples. Sub-objectives are designed for reward shaping, and a multi-agent reinforcement learning architecture with distributed decision-making and centralized training is combined with Hierarchical reward shaping framework, generate QMIX algorithms to solve the policy model and obtain the collaboration strategy.
Effectively guide multiple agents to learn better collaboration strategies in a war chess environment, improve learning efficiency and strategy quality, and overcome training difficulties caused by reward sparseness.
Smart Images

Figure CN114722701B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of war game simulation, and in particular to a method for acquiring war game simulation collaboration strategies based on a deep reinforcement learning model. Background Art
[0002] The problem of sparse rewards also exists in the research of Multi-Agent Reinforcement Learning (MARL). Wargames are a game tool that simulates real battlefields and combat processes. Compared with traditional reinforcement learning environments, wargame environments have the characteristics of incomplete information, uncertainty, and simultaneous decision-making by multiple operators. It is necessary to consider the coordination and game between operators. These characteristics increase the difficulty of reinforcement learning research on the one hand, and provide a research platform for the further development of reinforcement learning on the other.
[0003] Researchers at home and abroad have been conducting research on artificial intelligence in wargame environments for a long time. The victory of the chess agent AlphaGo over the world champion of Go, Lee Sedol, demonstrated the advantages of reinforcement learning technology in the field of intelligent decision-making. Compared with the chess environment, the incomplete observation of wargame simulation makes its research more challenging. At present, there are not many applications of multi-agent reinforcement learning on wargame platforms. MARL requires the agent to interact with the environment and obtain a large amount of data for training the agent. The action space and state space increase exponentially with the increase in the number of operators, which will lead to the occurrence of sparse reward problems, further hindering the training and convergence of reinforcement learning.
[0004] Therefore, how to guide multiple agents to learn better learning strategies in a war game environment has become a technical problem that needs to be solved urgently.
[0005] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention
[0006] The main purpose of the present invention is to provide a method for acquiring collaborative strategies in wargame simulations based on a deep reinforcement learning model, aiming to solve the technical problem that the existing technology cannot guide multiple agents to learn better learning strategies in a wargame environment.
[0007] To achieve the above object, the present invention provides a method for acquiring a war game cooperation strategy based on a deep reinforcement learning model, the method comprising:
[0008] Build a multi-agent system environment on the war game platform;
[0009] Modeling is performed according to the multi-agent system environment in combination with preset tuples to generate a decision model;
[0010] Design sub-goals based on expert prior knowledge, and use the sub-goals for reward shaping;
[0011] Select a multi-agent reinforcement learning architecture with distributed decision-making and centralized training combined with a hierarchical reward shaping framework to generate a QMIX algorithm based on the reward shaping;
[0012] The decision model is solved by the QMIX algorithm to obtain the war game collaboration strategy.
[0013] Optionally, the step of generating a decision model by modeling the multi-agent system environment in combination with a preset tuple includes:
[0014] The decision model of the multi-agent system environment is modeled using the improved tuple G = <S, U, P, r, Z, O, n, γ>, where s∈S describes the observation of the red team on the global situation of the training environment. At each time step, each agent n has a choice of actions in the optional action set U n Select an action from the list, and the actions selected by all agents form a joint action u∈U≡U n , the joint action drives the state transition in the environment, the state transition is defined as follows P(s'|s,u):S×U×S→[0,1], all agents share the same reward function r(s,u):S×U→γ and γ∈[0,1).
[0015] Optionally, the step of designing sub-goals according to expert prior knowledge and using the sub-goals for reward shaping includes:
[0016] Determine the sub-goal sequence based on expert prior knowledge;
[0017] Determine an abstract state learned by reinforcement learning and construct a Markov decision model on the abstract state;
[0018] Building a neural network to solve the Markov decision model;
[0019] The neural network is trained, and after the neural network training converges, the obtained V(Z) value is used as the potential energy of the state to perform reward shaping.
[0020] Optionally, the step of building a neural network to solve the Markov decision model includes:
[0021] In the neural network, the input space dimension is four-dimensional. When the first sub-goal is reached, the first-dimensional input is 1, otherwise it is 0. The same is true for the second, third, and fourth dimensions. After a hidden layer with a dimension of 10, a normal distribution with a mean of 0 and a standard deviation of 0.01 is selected for parameter initialization, and the abstract state function value is output to solve the Markov decision model.
[0022] Optionally, the step of selecting a distributed decision-making, centralized training multi-agent reinforcement learning architecture combined with a hierarchical reward shaping framework to generate a QMIX algorithm based on the reward shaping comprises:
[0023] A multi-agent reinforcement learning architecture with distributed decision-making and centralized training is selected, and the QMIX algorithm is built in combination with a hierarchical reward shaping framework. The QMIX algorithm includes a group of DRQNs and a hybrid network. Each DRQN network in the group corresponds to a distributed execution strategy. The hybrid network weightedly combines a group of Qa output by a group of DRQN networks in a complex nonlinear pattern to output Qtot.
[0024] Optionally, after the step of selecting a distributed decision-making, centralized training multi-agent reinforcement learning architecture combined with a hierarchical reward shaping framework to generate a QMIX algorithm based on the reward shaping, the step further includes:
[0025] The input information is passed through a layer of MLP network, and a normal distribution with a mean of 0 and a standard deviation of 1 is selected for parameter initialization, and a set of feature vectors is output;
[0026] Using a rectified linear unit to perform nonlinear processing on the feature vector;
[0027] The processed feature vector and the hidden layer information stored in the previous step are input into the RNN layer. According to the requirements of time series training, the feature vector of the current time step and the hidden layer information of the previous step are fused in the RNN layer to generate a new feature vector and new hidden layer information.
[0028] The output of the RNN layer is passed to the last MLP layer to obtain the Q value;
[0029] Adopt the epislon-greedy method. When the randomly generated probability is less than the preset probability, use the argmax function to select the action with the largest state-action value and execute it.
[0030] When the generated probability is less than the set probability, the action is randomly selected with equal probability and the action obtained at the current step is Deposit it in the replay pool and withdraw it when the network is updated.
[0031] Optionally, before the step of solving the decision model by using the QMIX algorithm to obtain the war game cooperation strategy, the step further includes:
[0032] Get the Q value list of M agents in the preset address;
[0033] Get the global state St and pass it through a separate linear layer for feature processing;
[0034] After another layer of absolute value activation function, the features are processed nonlinearly to generate the weights of the hybrid network;
[0035] The output is four vectors <W1, W2, bias1, bias2>, which are used to perform a weighted combination of the Q-value lists of the x1th agent to the xmth agent;
[0036] According to the Q value list as the input of the hybrid network, after calculation, the output Q tot (τ,a) value.
[0037] The present invention builds a multi-agent system environment on a war game platform; models the multi-agent system environment in combination with preset tuples to generate a decision model; designs sub-goals based on expert prior knowledge, and uses the sub-goals for reward shaping; selects a multi-agent reinforcement learning architecture with distributed decision-making and centralized training in combination with a hierarchical reward shaping framework to generate a QMIX algorithm based on the reward shaping; solves the decision model through the QMIX algorithm to obtain a war game simulation collaboration strategy, thereby guiding the multi-agents to learn a better learning strategy in the war game environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A flowchart of a first embodiment of a method for acquiring a war game collaboration strategy based on a deep reinforcement learning model according to the present invention;
[0039] Figure 2 This is a state space diagram of the first embodiment of the method for acquiring a war game cooperation strategy based on a deep reinforcement learning model of the present invention;
[0040] Figure 3 A battlefield situation diagram of the first embodiment of the method for acquiring a war game cooperation strategy based on a deep reinforcement learning model of the present invention;
[0041] Figure 4 This is a schematic diagram of the aircraft action space of the first embodiment of the method for acquiring a war game cooperation strategy based on a deep reinforcement learning model of the present invention;
[0042] Figure 5 This is a network architecture diagram of the first embodiment of the method for acquiring collaborative strategies for wargame simulations based on a deep reinforcement learning model of the present invention.
[0043] Figure 6 A hybrid network diagram of the first embodiment of the method for acquiring a war game collaboration strategy based on a deep reinforcement learning model of the present invention
[0044] Figure 7 This is a neural network update process diagram of the first embodiment of the method for acquiring war game cooperation strategy based on the deep reinforcement learning model of the present invention. DETAILED DESCRIPTION
[0045] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0046] The embodiment of the present invention provides a method for acquiring a war game cooperation strategy based on a deep reinforcement learning model. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for acquiring a war game collaboration strategy based on a deep reinforcement learning model of the present invention.
[0047] In this embodiment, the method for acquiring a war game collaboration strategy based on a deep reinforcement learning model includes the following steps:
[0048] Step S10: Building a multi-agent system environment on the war game platform.
[0049] It should be noted that the experimental environment used in this embodiment is the war game simulation platform of the National War Game Simulation Competition. As a real-time strategy war game simulation platform, the National War Game Simulation Competition platform has the following characteristics:
[0050] 1. The platform covers joint operations in all domains of land, sea, air, space and electric power, and displays them in a unified visual field, which is easily accessible through the global situation interface.
[0051] 2. Rich combat units. The ground facilities include: command posts, radars, airports, armored vehicles, ground missiles, anti-aircraft guns, etc.; the surface facilities include: destroyers, aircraft carriers, minesweepers and other surface ships, submarines, etc.; the air facilities include: fighters, bombers, early warning aircraft and other aircraft; in addition, it also includes satellites and other weapon platforms that may be involved in modern warfare, as well as merchant ships, fishing boats, civilian aircraft and other civilian platforms.
[0052] 3. Complete equipment data. Each combat unit in the platform has the model parameters of the sensors it carries, and also includes different types of mounts, including missiles, jammers, etc.
[0053] 4. Diverse missions. The combat units in the platform support the execution of missions such as interception, escort, patrol, and strike, and can also achieve more advanced missions through formations or mission combinations.
[0054] Based on the above characteristics, the platform can simulate military problems in the real world more realistically with the help of computer simulation technology, and represent the dynamic game process under asymmetric and incomplete information. Research on this platform is of great significance to the research on reinforcement learning of high-dimensional complex environments and multi-agent interactions. This embodiment builds a multi-agent training environment based on the ocean air combat assumption of the National War Game Simulation Competition Platform. The assumed battlefield environment is located over the ocean, which is a rectangular combat area from 151.5 degrees to 161.0 degrees east longitude and 23.5 degrees to 28.0 degrees north latitude, including combat sea areas and combat airspaces, with a north-south height of about 500km and an east-west length of about 1000km. This embodiment uses the MARL algorithm to control the combat units of the Red Party. Through interactive learning with the environment, it is hoped that a sea-air coordinated combat strategy can be learned. The experimental situation is described as follows: the Red Party's aircraft carrier is located in the area of 25.52 degrees north latitude and 153.25 degrees east longitude, and the Blue Party's aircraft carrier is located in the area of 26.32 degrees north latitude and 158.65 degrees east longitude. According to the satellite reconnaissance of the Red and Blue sides, the Red side's Arleigh Burke-class guided missile destroyer is roughly located in the sea area of 25.7 degrees north latitude and 155.3 degrees east longitude. The Blue side's Arleigh Burke-class guided missile destroyer is roughly located in the sea area of 25.8 degrees north latitude and 157.5 degrees east longitude. The Blue side operator uses the traditional planning method to make control decisions. The Blue side commander issues offensive-oriented combat orders in a task-oriented command manner, and first clarifies the combat time and combat tasks to subordinates. For the four Super Hornet fighters, they have good air combat capabilities. First, they are organized into two double-aircraft formations and move towards the target airspace. Their task is to determine the specific location of the enemy destroyer. Their second task is to find the main force of the enemy fighter and fight; the two Lightning fighters have the ability to attack surface ships. After the Hornet fighter finds the specific location of the enemy ship, it sets out to attack the enemy ship and try to avoid engaging with the main force of the enemy air force. Compared with fighters, because the destroyer has a slower maneuvering speed, it patrols on the sea surface and sets a patrol area depending on our combat situation. The Red Army's aircraft operators are controlled using the MARL algorithm, and through interactive learning with the environment, they hope to learn a sea-air coordinated combat strategy. The destroyer uses a rule-based algorithm to control the regional patrol algorithm.
[0055] In the specific implementation, the observation distance of a single aircraft in this embodiment is limited, so there is a local observation state space The centralized training center can obtain intelligence information from each aircraft and integrate it into the global observation state space St. First, the local observation value of each aircraft is introduced The detection range of the x1th aircraft that needs to make a decision is a set of fixed values, including air detection range and sea detection range. At a certain moment, there are N enemy aircraft and M ships in its field of view. The situation data observed by the x1th aircraft is processed, and the information of aircraft, ships and missiles of both sides is spliced as local observation information and input into the decision network of the aircraft, such as Figure 2 As shown. The input St of the hybrid network, the global state St at time T consists of the information of all our aircraft, the information of all enemy aircraft, the information of all our ships, the information of all enemy ships, and the information of all missiles, as shown Figure 3 The aircraft's action space and speed have four gears, as shown in the following table:
[0056]
[0057] Aircraft speed gear table
[0058] At the same time, the aircraft can choose a heading angle of 360 degrees. In order to reduce the decision space, this embodiment discretizes the heading angle and divides it into six directions, each with an interval of sixty degrees. The aircraft's maneuvering action is determined by the gear position and the aircraft's heading angle, and there are a total of 4*6=24 actions. The aircraft's action space also includes strike actions. The enemy has a total of six fighters and a destroyer, so a single-step strike action is seven actions. The action space of a single aircraft has 31 actions. This embodiment assumes that it takes a certain amount of time for the aircraft to execute a single-step action, so only one action can be executed at each decision time step. Because the selection actions contained in the action space of a single aircraft will be different at different times, in actual experiments, the aircraft must first determine the optional actions. When the action output by the neural network is in the list of optional actions, the corresponding action is executed. Figure 4 shown.
[0059] Step S20: Modeling is performed according to the multi-agent system environment in combination with preset tuples to generate a decision model.
[0060] Furthermore, the step of generating a decision model by modeling the multi-agent system environment in combination with a preset tuple includes: using an improved tuple G = <S, U, P, r, Z, O, n, γ> to model the decision model of the multi-agent system environment, where s∈S describes the observation of the red side on the global situation of the training environment, and at each time step, each agent n has a decision choice in the optional action set U n Select an action from the list, and the actions selected by all agents form a joint action u∈U≡U n, the joint action drives the state transition in the environment, the state transition is defined as follows P(s'|s,u):S×U×S→[0,1], all agents share the same reward function r(s,u):S×U→γ and γ∈[0,1).
[0061] Step S30: Design sub-goals based on expert prior knowledge, and use the sub-goals for reward shaping.
[0062] In the specific implementation, this embodiment first introduces the original reward function. The score of the two aircraft is 139 points for destroying an enemy aircraft, and 139 points for losing an aircraft. The score of destroying an enemy destroyer is 1843 points, and the score of losing a destroyer is 1843 points. The score is determined by the assumption itself. At the end of the game, the winner is determined by the scores of both sides. In war game simulation, due to the high dimension of the combat space, many choices of action space, and the enemy units often bypass our units and directly attack destroyers, our aircraft often face the situation of making many decisions but not getting any rewards, and finally just spinning in place. In order to solve the problem of sparse rewards, this paper adopts the reward shaping method of sub-goals generated by expert knowledge. In this embodiment, in the sea-air coordinated combat mission, the sub-goals and related details of the reward shaping reinforcement learning algorithm are designed according to the expert prior knowledge. In the sea-air coordinated combat mission, according to the commander's prior knowledge, there are some sub-tasks from which sub-goals can be extracted. This study starts from the overall combat situation and considers the tactical global sub-goals. At the same time, there are some expert experiences related to the action command and control of a single agent on the battlefield, which can also be used in the design of sub-goals. The sub-goals of the action control of the single agent itself are not considered in this study, and they need to be studied in the future. The red plane departs from the airport, and its main combat goal is to destroy the enemy destroyer and defend our destroyer. The approximate longitude and latitude coordinates of the enemy destroyer have been obtained by satellite reconnaissance and shared with all our aircraft. The primary task after takeoff is to search and detect enemy aircraft, so as to provide guidance for the formulation of the next step of tactics. In this embodiment, the reconnaissance and discovery of enemy aircraft is first set as the sub-goal Sg1. Before and after achieving this sub-goal, the combat mission changes greatly, from the reconnaissance-based task to the attack on enemy targets. After discovering the enemy target, our agent needs to analyze and judge according to the current position of the air combat operator, the number, type and longitude and latitude position coordinates of the enemy's detected operators, and judge whether to conduct air combat or naval combat first under the current situation by interacting with the environment. In order to guide the agent to learn this combat process, this embodiment sets the enemy aircraft entering the range of our anti-aircraft missile firepower as Sg2, and sets the enemy ship entering the range of our anti-sea strike missile as Sg3. When our situation is unfavorable, our aircraft should enter the defensive position, so the sub-goal of reaching the defensive position is set to Sg4. When reaching different sub-goals, the agent has different potential energy functions, and it is hoped that the agent will be guided to transfer to a state with high potential energy through learning. Examples of sub-goals are shown in the following table.
[0063]
[0064] Example of sub-goals set based on expert knowledge
[0065] By analogy, in the sea-air coordinated combat mission, according to the commander's prior knowledge, multiple combat intentions at different stages will be formulated in advance, and different combat intentions will also be converted to a certain extent according to different situations. Before and after the execution of the combat intention to achieve the combat goal, the battlefield situation will change greatly. This embodiment can set the combat intention as a sub-goal of reinforcement learning according to the commander's prior knowledge. After the sub-goal is set, for different sub-goals, this embodiment will set the judgment condition of whether the sub-goal is achieved. For sub-goal Sg1, this embodiment analyzes whether the enemy fighter is found from the intelligence at each decision step. For Sg2, after the enemy fighter is found, the distance between our fighter and the enemy fighter is calculated in real time. When the distance is less than the given threshold, it is confirmed that the sub-goal has been achieved. The setting method of sub-goal judgment Sg3 is the same. The algorithm pre-sets the defense position around our destroyer. Our fighter can intercept the enemy fighter together with the destroyer at this position. When our fighter arrives at the position around the destroyer, it is confirmed that the sub-goal Sg4 has been reached.
[0066] Furthermore, the step of designing sub-goals based on expert prior knowledge and using the sub-goals for reward shaping includes: determining a sub-goal sequence based on expert prior knowledge; determining an abstract state learned by reinforcement learning, and constructing a Markov decision model on the abstract state; building a neural network to solve the Markov decision model; training the neural network, and using the obtained V(Z) value as the potential energy of the state after the neural network training converges to perform reward shaping.
[0067] It should be noted that after determining the sub-goal sequence {Sg1, Sg2, Sg3, Sg4}, the abstract state learned by reinforcement learning is also determined. Different from the model introduced in the principle, the size of the abstract state space here is 24 = 16, that is, whether to reach Sg1 (2 states) * whether to reach Sg2 (2 states) * whether to reach Sg3 (2 states) * whether to reach Sg4 (2 states), the abstract state space is expressed as {Z0, Z1, ..., Z 16}. A Markov decision model is constructed on the abstract state, and a neural network is built to solve the model. The input space dimension is four-dimensional. When the first sub-goal is reached, the first-dimensional input is 1, otherwise it is 0. The same is true for the second, third, and fourth dimensions. After a hidden layer with a dimension of 10, a normal distribution with a mean of 0 and a standard deviation of 0.01 is selected for parameter initialization. Finally, the output layer outputs the abstract state function value. The neural network is updated in real time. After the neural network training converges, the obtained V(Z) value is used as the potential energy of the state to shape the reward. The value function is related to the strategy. In the training of the abstract model, the strategy we adopt is obtained from the specific function training. Then, as the specific neural network training proceeds, the strategy learned by the lower-level specific reinforcement learning gradually improves, and the value function learned by the abstract model is closer to the true value of the state. This makes the reward function set according to the value function guide the specific reinforcement learning algorithm closer to the final goal.
[0068] Furthermore, the step of building a neural network to solve the Markov decision model includes:
[0069] In the neural network, the input space dimension is four-dimensional. When the first sub-goal is reached, the first-dimensional input is 1, otherwise it is 0. The same is true for the second, third, and fourth dimensions. After a hidden layer with a dimension of 10, a normal distribution with a mean of 0 and a standard deviation of 0.01 is selected for parameter initialization. Finally, the output layer outputs the abstract state function value to solve the Markov decision model.
[0070] Step S40: Select a multi-agent reinforcement learning architecture with distributed decision-making and centralized training and combine it with a hierarchical reward shaping framework to generate a QMIX algorithm based on the reward shaping.
[0071] It should be noted that in MAS, due to the existence of some constraints, a single agent is often difficult to observe global information, especially in real-time strategy game environments such as war games with fog of war. Sometimes there is communication between agents to share information, but in practical applications, communication will also have many limitations. For the above reasons, for each agent, it is often necessary to obtain local observations based on the specific environment in which it is located, and establish corresponding discrete strategies based on local observations. When the number of agents increases, the joint action decision space composed of the discretized strategy is too large, which will cause the traditional single-agent algorithm to fail. In response to the above problems, this embodiment proposes a learning paradigm of centralized training and distributed execution. Centralized training requires the agent to obtain a Qtot based on the global state and joint action. When the number of operators is too large, Qtot is difficult to learn directly. Even if it can be learned, there is no direct way to convert it into each operator based on a separate observation. To solve this problem, Tabish Rashid et al. proposed the QMIX algorithm. The algorithm consists of a group of DRQN (Deep Recurrent Q-Network) and a mixing network. Each DRQN network in the group corresponds to a distributed execution strategy. The mixing network combines a group of Qa output by a group of DRQN networks in a complex nonlinear mode, thereby outputting Qtot. Therefore, the QMIX algorithm can represent the centralized action value function in a factorized representation method. This representation method can have good scalability according to the number of agents, and allows decentralized strategies to easily obtain results in linear time through a single argmax operation.
[0072] In order to ensure consistency, the algorithm needs to ensure that the global optimum is composed of the local optimum of all operators. The algorithm constrains the parameters of the hybrid network to be positive numbers so that the above requirements are met:
[0073]
[0074] Next, we will introduce the specific structure of the neural network in the QMIX algorithm. At time T, for each agent a, there is a DRQN network that outputs its own value function, taking the current local observation value as input at each time step, such as Figure 5 shown.
[0075] The input information is passed through a layer of MLP network. The MLP layer first processes the situation information. For the fully connected layer, a normal distribution with a mean of 0 and a standard deviation of 1 is selected for parameter initialization, and a set of feature vectors is output. Next, the rectified linear unit (RELU) is used to perform nonlinear processing on the feature vector. Then, the network inputs the processed feature vector and the hidden layer information stored in the previous step into the RNN layer. According to the requirements of time series training, the feature vector of the current time step of the agent and the hidden layer information of the previous step are fused in the RNN layer to generate a new feature vector and new hidden layer information. Finally, the output of the RNN layer is passed to the last layer of MLP to obtain the Q value. Here, the Q value is an N-dimensional vector that represents the state-action value function value of the agent a selecting H different actions. After obtaining the Q value vector of agent a, the algorithm adopts the epislon-greedy method. When the probability of random generation is less than the set probability, the argmax function is used to select the action with the largest state-action value and execute it. When the probability of generation is less than the set probability, the action is randomly selected with equal probability. Then the current step is obtained. Store it in the playback pool and take it out when the network is updated. Next, let's introduce the composition of the hybrid network. Assuming that we have a total of M agents, according to this method, we can obtain the Q value list of our x1th agent to the xmth agent. This Q value list is used as the input of the hybrid network. After the network calculation, a Q value list is output. tot (τ,a) values. The weights of the hybrid network are generated by independent super-networks. Each super-network takes the global state St as input and generates a layer of parameters of the hybrid network. Figure 6 The hybrid network and its super network are explained. The global state St first passes through a separate linear layer for feature processing, and then passes through a layer of absolute value activation function to perform nonlinear processing on the features, and then generates the weights of the hybrid network. The linear formation and absolute value activation function are collectively referred to as the super network. In order to ensure that the weights of the hybrid network are non-negative, the absolute value activation function is an indispensable super network component. The output of the super network is four vectors <W1, W2, bias1, bias2>. These four vectors are used to perform weighted combination of the Q value list of our x1th agent to the xmth agent. In order to meet the weighting requirements, these four vectors are reshaped into four matrices of a certain size. Next, the update process of the neural network is introduced, such as Figure 7 First, The list is input into the hybrid network to generate Qtot, using the target network and the stored next frame observation action reward information at the same time, according to the formula:
[0076] y tot = r + γmax u′ Q tot (τ′,u′,s′;θ - )
[0077] Calculate the target network's estimate of the current state-action pair and calculate y tot and Q tot This difference is defined as the loss function and back-propagated in the hybrid network and DRQN network, where θ - is the parameter of the target network. The loss function is defined as follows, where b is the number of state transition quadruplets (s, a, s', r) sampled from the memory pool.
[0078]
[0079] Furthermore, the step of selecting a multi-agent reinforcement learning architecture with distributed decision-making and centralized training combined with a hierarchical reward shaping framework to generate a QMIX algorithm based on the reward shaping includes: selecting a multi-agent reinforcement learning architecture with distributed decision-making and centralized training, and building a QMIX algorithm in combination with a hierarchical reward shaping framework, wherein the QMIX algorithm includes a group of DRQNs and a hybrid network, wherein each DRQN network in the group corresponds to a distributed execution strategy, and the hybrid network weights a group of Qa output by a group of DRQN networks in a complex nonlinear mode, thereby outputting Q tot .
[0080] Furthermore, after the step of selecting a multi-agent reinforcement learning architecture with distributed decision-making and centralized training combined with a hierarchical reward shaping framework to generate a QMIX algorithm based on the reward shaping, it also includes: passing the input information through a layer of MLP network, selecting a normal distribution with a mean of 0 and a standard deviation of 1 for parameter initialization, and outputting a set of feature vectors; using a rectified linear unit to perform nonlinear processing on the feature vector; inputting the processed feature vector and the hidden layer information stored in the previous step into the RNN layer, and according to the requirements of time series training, fusing the feature vector of the current time step and the hidden layer information of the previous step in the RNN layer to generate a new feature vector and new hidden layer information; passing the output of the RNN layer into the last layer of MLP to obtain the Q value; adopting the epislon-greedy method, when the randomly generated probability is less than the preset probability, using the argmax function, selecting the action with the largest state action value and executing it; when the generated probability is less than the set probability, randomly extracting actions with equal probability and executing the action obtained at the current step Deposit it in the replay pool and withdraw it when the network is updated.
[0081] Step S50: Solve the decision model using the QMIX algorithm to obtain a war game collaboration strategy.
[0082] Furthermore, before the step of solving the decision model by the QMIX algorithm to obtain the war game cooperation strategy, it also includes: obtaining the Q value list of M agents in the preset address; obtaining the global state St and performing feature processing through a separate linear layer; and then performing nonlinear processing on the features through a layer of absolute value activation function to generate the weight of the hybrid network; the output result is four vectors <W1, W2, bias1, bias2>, which are used to perform weighted combination of the Q value list of the x1th agent to the xmth agent; according to the Q value list as the input of the hybrid network, Q is output after calculation. tot (τ,a) value.
[0083] This embodiment builds a multi-agent system environment on a war game platform; models the multi-agent system environment in combination with preset tuples to generate a decision model; designs sub-goals based on expert prior knowledge, and uses the sub-goals for reward shaping; selects a multi-agent reinforcement learning architecture with distributed decision-making and centralized training in combination with a hierarchical reward shaping framework to generate a QMIX algorithm based on the reward shaping; solves the decision model through the QMIX algorithm to obtain a war game simulation collaboration strategy, thereby guiding multiple agents to learn better learning strategies in a war game environment.
[0084] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0085] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0087] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for acquiring war game cooperation strategy based on deep reinforcement learning model, characterized in that: The method comprises: Build a multi-agent system environment on the wargame platform; Modeling is performed according to the multi-agent system environment in combination with preset tuples to generate a decision model; Design sub-goals based on expert prior knowledge, and use the sub-goals for reward shaping; Select a multi-agent reinforcement learning architecture with distributed decision-making and centralized training combined with a hierarchical reward shaping framework to generate a QMIX algorithm based on the reward shaping; The decision model is solved by the QMIX algorithm to obtain the war game cooperation strategy; the step of modeling according to the multi-agent system environment combined with the preset tuple to generate the decision model includes: The decision model of the multi-agent system environment is modeled using the improved tuple G = <S, U, P, r, Z, O, n, γ>, where s∈S describes the observation of the red team on the global situation of the training environment. At each time step, each agent n has a choice of actions in the optional action set U n Select an action from the list, and the actions selected by all agents form a joint action u∈U≡U n , the joint action drives the state transition in the environment, the state transition is defined as follows P(s'|s,u):S×U×S→[0,1], all agents share the same reward function r(s,u):S×U→γ and γ∈[0,1); The step of designing sub-goals based on expert prior knowledge and using the sub-goals for reward shaping includes: Determine the sub-goal sequence based on expert prior knowledge; Determine an abstract state learned by reinforcement learning and construct a Markov decision model on the abstract state; Building a neural network to solve the Markov decision model; Training the neural network, and after the neural network training converges, using the obtained V(Z) value as the potential energy of the state to perform reward shaping; The step of building a neural network to solve the Markov decision model includes: In the neural network, the input space dimension is four-dimensional. When the first sub-goal is reached, the first-dimensional input is 1, otherwise it is 0. The same is true for the second, third, and fourth dimensions. After a hidden layer with a dimension of 10, a normal distribution with a mean of 0 and a standard deviation of 0.01 is selected for parameter initialization, and the abstract state function value is output to solve the Markov decision model.
2. The method according to claim 1, characterized in that The step of selecting a multi-agent reinforcement learning architecture with distributed decision-making and centralized training combined with a hierarchical reward shaping framework to generate a QMIX algorithm based on the reward shaping comprises: A multi-agent reinforcement learning architecture with distributed decision-making and centralized training is selected, and the QMIX algorithm is built in combination with a hierarchical reward shaping framework. The QMIX algorithm includes a group of DRQNs and a hybrid network. Each DRQN network in the group corresponds to a distributed execution strategy. The hybrid network weightedly combines a group of Qa output by a group of DRQN networks in a complex nonlinear pattern to output Qtot.
3. The method according to claim 2, characterized in that After the step of selecting a distributed decision-making, centralized training multi-agent reinforcement learning architecture combined with a hierarchical reward shaping framework to generate a QMIX algorithm based on the reward shaping, the step further includes: The input information is passed through a layer of MLP network, and a normal distribution with a mean of 0 and a standard deviation of 1 is selected for parameter initialization, and a set of feature vectors is output; Using a rectified linear unit to perform nonlinear processing on the feature vector; The processed feature vector and the hidden layer information stored in the previous step are input into the RNN layer. According to the requirements of time series training, the feature vector of the current time step and the hidden layer information of the previous step are fused in the RNN layer to generate a new feature vector and new hidden layer information. The output of the RNN layer is passed to the last MLP layer to obtain the Q value; Adopt the epislon-greedy method. When the randomly generated probability is less than the preset probability, use the argmax function to select the action with the largest state-action value and execute it. When the generated probability is less than the set probability, the action is randomly selected with equal probability and the action obtained at the current step is Deposit it in the replay pool and withdraw it when the network is updated.
4. The method according to any one of claims 1 to 3, characterized in that Before the step of solving the decision model by the QMIX algorithm to obtain the war game cooperation strategy, the method further includes: Get the Q value list of M agents in the preset address; Get the global state St and pass it through a separate linear layer for feature processing; After another layer of absolute value activation function, the features are processed nonlinearly to generate the weights of the hybrid network; The output is four vectors <W1, W2, bias1, bias2>, which are used to perform a weighted combination of the Q-value lists of the x1th agent to the xmth agent; According to the Q value list as the input of the hybrid network, after calculation, the output Q tot (τ,a) value.
