Intelligent war game deduction method based on reinforcement learning

Abstract

The invention discloses an intelligent war game deduction method based on reinforcement learning, and the method comprises the steps: constructing a dynamic battlefield model: constructing an adjustable battlefield simulation platform, and defining the landform, resources and army distribution elements in a battlefield; a reinforcement learning strategy is generated and optimized, an intelligent strategy generation algorithm based on deep reinforcement learning is designed, and an efficient combat strategy is generated in multiple rounds of training by constructing a state space and an action space and combining a situation reward function; a double-agent chess playing training mechanism is introduced, black parties and white parties are modeled into reinforcement learning agents, red parties and blue parties are modeled into reinforcement learning agents, and a real battlefield game is simulated through multiple rounds of alternate training; result visualization deduction: a dynamic decision visualization function is provided, and battlefield situation, troop dynamics and a strategy execution process can be displayed in real time; according to the method, the problems of rule solidification, limited strategy generation capability, insufficient antagonism and the like in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to intelligent decision-making technology, and in particular to an intelligent war game deduction method based on reinforcement learning. Background Art

[0002] Wargaming is a key tool for simulating military operations and strategic decision-making, widely used in military training, operational planning, and decision support in complex scenarios. By simulating multiple factors, such as battlefield dynamics, troop deployment, and combat processes, wargaming can effectively evaluate the pros and cons of different tactical options, providing important insights for military command.

[0003] Current wargaming techniques primarily fall into the following categories: rule-based methods, strategy optimization methods based on game theory, and intelligent wargaming methods based on artificial intelligence. Rule-based methods use expert experience to predefine a fixed set of rules and simulate troop movements. While their advantage lies in their simplicity, they struggle to cope with complex and dynamic battlefield environments. Game-theory-based methods attempt to optimize the strategies of both sides through mathematical models. While their theoretical nature is strong, their applicability to actual battlefields is limited. Artificial intelligence-based wargaming methods, on the other hand, focus on leveraging machine learning techniques to generate strategies. In particular, the application of reinforcement learning algorithms to adaptive decision-making has become a growing research hotspot. For example, deep reinforcement learning (DRL) algorithms enable intelligent agents to start from scratch and acquire optimal strategies through multiple rounds of interactive learning. This technology not only overcomes the limitations of traditional methods but can also effectively cope with changing environments and uncertain adversary behavior.

[0004] However, introducing reinforcement learning into wargaming systems still faces multiple challenges: first, how to efficiently model complex battlefield environments so that they can accurately reflect the terrain, resource distribution, and troop dynamics of a real battlefield; second, how to design a reinforcement learning framework that can generate diverse tactical strategies and adapt them to the dynamically changing enemy and friendly situation; and third, how to use adversarial training to enable the intelligent agents of the red and blue sides to simulate real dynamic games, thereby improving the authenticity of the simulation and its decision-making reference value. Summary of the Invention

[0005] The purpose of the invention is to provide an intelligent war game simulation method based on reinforcement learning, in order to solve the technical problems existing in the existing technology.

[0006] The technical solution is an intelligent war game simulation method based on reinforcement learning, which includes the following steps:

[0007] Step S1: Build a dynamic battlefield model: Build an adjustable battlefield simulation platform, define the terrain, resources, and troop distribution elements in the battlefield, support dynamic switching and rule adjustment of multiple scenarios, and provide continuous state space input for the reinforcement learning agent;

[0008] Step S2: Generate and optimize reinforcement learning strategies: Design an intelligent strategy generation algorithm based on deep reinforcement learning. By constructing a state space and an action space and combining it with a situation reward function, an efficient combat strategy is generated over multiple rounds of training. The intelligent strategy generation algorithm based on deep reinforcement learning uses an improved strategy optimization algorithm to improve the stability and efficiency of strategy generation.

[0009] Step S3: Develop a red-blue confrontation mechanism: Introduce a dual-agent game training mechanism, modeling the red and blue agents as reinforcement learning agents, and simulate real battlefield games through multiple rounds of alternating training. During the training process, the red and blue agents optimize their respective strategies through dynamic games to simulate the confrontation and complexity of real wars.

[0010] According to a further improvement of the present invention, the construction of a dynamic battlefield model represents the battlefield environment through a grid map, and models the terrain, resources, unit status and rules in the battlefield in a digital manner, specifically including:

[0011] Step S11: Construct the battlefield map into a two-dimensional grid space, where each grid cell represents a location on the battlefield. The attributes of the grid include terrain type, accessibility, and resource distribution, and are defined as:

[0012] G={g i,j |1≤i≤N,1≤j≤M}; where N and M are the number of rows and columns of the map respectively. The state of each grid can be further represented by the vector g i,j =[T,P,R], where T is the terrain type, P is the accessibility attribute, and R is the resource attribute;

[0013] Step S12: Model the status of the troop units on the map with dynamic attributes, including two types: friendly units and enemy units. The attributes of each unit include position p = (x, y), health value H, firepower range F, movement speed V and resource status R. u , the initial state matrix S records the basic information of all units and is dynamically updated during the deduction process. The unit state is expressed as:

[0014] S unit ={(p,H,F,V,R u )}; The unit's firing range F is defined as the Euclidean distance: d(u,v) = (x u -x v ) 2+(y u -y v ) 2 , only enemy units that satisfy d(u,v)≤F will be attacked;

[0015] Step S13: Define battlefield rules to regulate the actions of the troops and the conditions for winning or losing in the deduction. The basic rules include: the movement rule is that the unit can move in any direction at a movement speed V each round, but cannot enter an impassable grid; the attack rule is that the unit can only attack enemy targets within its firepower range, and the damage value is determined by the unit's attack power and the enemy's defense power. The damage calculation formula is:

[0016] D=AD f , where A is the attack power of the attacking unit, D f is the defense of the defending unit, and D is the final damage value;

[0017] Victory conditions: The deduced victory conditions include one of the following two: all enemy units are defeated; the team successfully occupies the preset target area.

[0018] According to a further improvement of the present invention, the generation of a reinforcement learning strategy and optimization of the reinforcement learning model optimizes the strategy execution effect of the intelligent agent in a dynamic battlefield environment by constructing a state space, an action space, and a reward function, combined with a deep learning algorithm, specifically including:

[0019] Step S21, setting state space S: The state space S is the basis for the agent to perceive the battlefield situation, including battlefield map information, attributes of its own units and enemy units, specifically expressed as:

[0020] S=[G,S units ]; where G is the battlefield grid map, which contains terrain, resource distribution and obstacle information; S units It is a collection of attributes of all units, including position p, fire range F, health H and resources R. The state space vector S is dynamically updated to reflect battlefield changes in real time.

[0021] Step S22, set the action space A: The action space A defines all possible actions of the agent, including: moving to the target location, attacking a specific unit, defending to improve survivability, and resupplying to restore health or ammunition. Each action behavior is represented by a vector ai, and its selection is based on the value output by the reinforcement learning policy network. The action space is represented as:

[0022] A={a i ∣a i ∈valid move of piece i}; where each action a i Corresponds to an action behavior;

[0023] Step S23, design reward function R: The reward function R is used to evaluate the effectiveness of the agent's behavior and guide the optimization of the strategy. Its design includes the following key indicators: the damage caused to the enemy D damage 、Own losses D loss and completion of mission objectives C objective , the specific formula is:

[0024] R=α1·D damage -α2·D loss +α3·C objective ; Among them, α1, α2, and α3 are the weight parameters of the reward function, which are used to balance the importance of different goals;

[0025] Step S24: Generate reinforcement learning strategy: Generate strategy through reinforcement learning model, use deep Q learning algorithm to optimize agent strategy, and fit state-action value function Q(s,a) through neural network. The Q value update rule is:

[0026] Q(s,a)=Q(s,a)+η·[r+γ·maxQ(s ′ ,a ′ )-Q(s,a)]; where η is the learning rate, r is the immediate reward, γ is the discount factor used to balance short-term and long-term benefits, s ′ is the next state, a ′ For the next action;

[0027] Step S25: During the training process, the agent stores each interaction data into the experience pool, randomly samples the data in batches for network update, reduces sample correlation, improves training efficiency, and uses the mean square error (MSE) to optimize Q-value prediction. The specific definition is as follows:

[0028] Where N is the number of batch training samples and γ is the discount factor used to weigh current rewards and future rewards.

[0029] According to a further improvement of the present invention, the formulation of the red-blue confrontation mechanism includes:

[0030] Step S31, modeling the red and blue agents: The red and blue agents are each used as a reinforcement learning model. Each agent is trained and competes in the same battlefield environment. Each agent includes a state space S and an action space A.

[0031] The state space includes battlefield map G, troop attributes S blue 、S red The battlefield map G is a grid-like two-dimensional representation. Troop attributes include position p, fire range F, health H, and resources R. The red and blue agents share the same state representation:

[0032] S=[G,S blue ,S red ];

[0033] The action space defines the operations that the agent can perform, including movement, attack, defense, and supply. Each action is represented by a vector a t Indicates that a t =[move,attack target], where move is the direction command and attack target is the specified enemy unit;

[0034] Step S32: using dual agents to conduct adversarial training and optimize the mechanism, specifically including the following steps:

[0035] Step S321, initializing the battlefield environment: At the beginning of each round of training, the battlefield map and the status of both sides' troops are initialized to ensure that both sides start the confrontation under fair conditions;

[0036] Step S322, Red and Blue agents make decisions alternately: The Red and Blue agents perform actions in turn according to the round-robin system. Each round of deduction is divided into three steps: Perceive the current state s t ; Select action a based on the current policy network t =argmaxQ(s t ,a); perform action a t , update battlefield status s t+1 ;

[0037] Step S323, update the reward function: assign rewards to both agents based on the action results. The reward function R is designed to guide the agents to optimize their strategies. The reward function formula is:

[0038] R=α1·D damage -α2·D loss +α3·C objective Among them, D damage is the damage caused by the agent to the enemy, D loss For your own losses, C objective is the completion of the task goal, α1, α2, and α3 are the weight parameters of the reward function adjusted according to training requirements;

[0039] Step S33, reinforcement learning model optimization: The reinforcement learning algorithm is used to optimize the strategies of the red and blue agents. The optimization strategies of both agents are based on the deep Q learning algorithm, and their state value function Q(s,a) is updated by the following formula:

[0040] Q(s,a)=Q(s,a)+η·[r+γ·maxQ(s ′ ,a ′)-Q(s,a)]; where η is the learning rate, r is the immediate reward, γ is the discount factor used to balance short-term and long-term benefits, s ′ is the next state, a ′ For the next action;

[0041] Use the experience replay mechanism to store training data, batch sample N data for back propagation to optimize network parameters, and the loss function is:

[0042]

[0043] Step S34, adversarial training convergence: setting the adversarial balance mechanism and convergence conditions to ensure the balance of the red and blue strategies and the stability of the training, including:

[0044] Counterbalance mechanism: When the unilateral winning rate of the red or blue team is too high, the winning team's strategy disturbance is increased or the reward weight of the weaker team is adjusted to balance the strategies of both sides;

[0045] Convergence conditions: Adversarial training stops when one of the following conditions is met: the winning rate of both sides is close to equilibrium, that is, the winning rate is 50±5%; the update amplitude of the strategy is less than the set threshold, that is, the Q value change rate is less than 0.01.

[0046] According to a further improvement of the present invention, the visual deduction results specifically include:

[0047] Dynamic battlefield display: Based on the grid battlefield map, the battlefield status is updated in real time, including unit positions, resource distribution and action status; the battlefield map is presented in the form of a two-dimensional grid, and each grid unit is g i,j Indicates the terrain attributes and status of the current location. Friendly and enemy units are distinguished by different colors and symbols, and their status is marked on the corresponding units in the form of numerical values or icons. The firepower range is represented by a circular area centered on the unit. The area within the range will be dynamically highlighted to show the firepower coverage. The firepower intensity is displayed through a heat map. The firepower coverage value is calculated as follows:

[0048] Among them, F i is the firepower value of unit i, d i is the distance from the current grid (x, y) to unit i;

[0049] Dynamic display of action paths: Records the movement and attack trajectory of each unit during the simulation. The unit's movement path is marked with arrows, and color coding indicates the change of unit status. Attack actions are connected by lines to represent the relationship between the target and the attacker. Attack damage values are displayed at the target position. The damage value calculation formula is:

[0050] D=AD f; Among them, A is the attack power of the attacking unit, D f The defense of the defending unit;

[0051] Key indicators and data statistics display: Loss statistics display friendly and enemy unit loss trends in the form of line graphs or bar graphs, including health reduction and number of units lost; battlefield control is displayed through a dynamic change chart of area occupation rate, and the dynamic changes in friendly control of key areas are described in the form of area charts or percentages; resource consumption is displayed through time series graphs to help users analyze whether resource allocation in the game is reasonable;

[0052] Decision report presentation: The report content includes strategy effect analysis, unit efficiency statistics and improvement suggestions: Strategy effect analysis quantifies the advantages and disadvantages of one's own strategy; unit efficiency statistics lists efficient and inefficient units; improvement suggestions combine deduction data to propose optimization solutions.

[0053] Beneficial effects: Based on traditional war game simulation, the present invention constructs a flexible and adjustable battlefield simulation environment that supports a variety of battlefield scenarios and rules; through the reinforcement learning algorithm, the intelligent agent can generate dynamic and adaptive strategies; through the dual-agent confrontation mechanism, the red and blue sides are continuously optimized in the alternating game, which improves the confrontation and authenticity of the simulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is an overall flow chart of the solution of the present invention.

[0055] Figure 2 It is a structural diagram of the deep Q network of the present invention.

[0056] Figure 3 This is a flow chart of the red-blue confrontation mechanism setting of the present invention. DETAILED DESCRIPTION

[0057] like Figure 1 The figure shows the overall process of the solution of the present invention. The present invention provides an intelligent war game simulation method based on reinforcement learning. Through the design of dynamic battlefield modeling, reinforcement learning strategy generation and optimization, red-blue confrontation mechanism and decision visualization module, a complete intelligent simulation system is formed, which can effectively solve the problems of rule rigidity, insufficient strategy generation capability and weak confrontation in the existing technology.

[0058] The specific implementation follows these steps: First, a flexible and adjustable battlefield simulation environment is established, defining the state space, action space, and battlefield rules. Then, reinforcement learning algorithms are used to optimize troop action strategies. Next, a red-blue confrontation mechanism is used for strategy training, making the simulation process simulate a real battlefield game. Finally, a visualization system is used to display the simulation results and generated strategies to help understand and improve the simulation process. The following describes each step and implementation process in detail:

[0059] Step 1: Dynamic battlefield modeling

[0060] Dynamic battlefield modeling is the first step of this invention. It is responsible for building a flexible and adjustable battlefield environment to provide a realistic and dynamic training scenario for reinforcement learning strategy generation and optimization. This module represents the battlefield environment through a grid map and models the terrain, resources, unit status and rules in a digital way.

[0061] First, the battlefield map is constructed as a two-dimensional grid space, where each grid cell represents a location on the battlefield. The attributes of the grid include terrain type, accessibility (0 means impassable, 1 means passable), and resource distribution, which are defined as:

[0062] G={g i,j |1≤i≤N,1≤j≤M};

[0063] Where N and M are the number of rows and columns of the map respectively; the state of each grid can be further represented by the vector g i,j =[T,P,R], where T is the terrain type, P is the accessibility attribute, and R is the resource attribute.

[0064] Secondly, the status of the troop units is modeled on the map with dynamic attributes, including two types of units: friendly units and enemy units. The attributes of each unit include position p = (x, y), health value H, firepower range F, movement speed V and resource status R u The initial state matrix S records the basic information of all units and is dynamically updated during the deduction process. The unit state is expressed as:

[0065] S unit ={(p,H,F,V,R u )};

[0066] The unit's firing range F is defined as Euclidean distance:

[0067]

[0068] Only enemy units that satisfy d(u,v)≤F will be attacked.

[0069] Third, define battlefield rules to regulate troop actions and the conditions for victory or defeat in the game. Basic rules include:

[0070] Movement rules: Each round, the unit can move in any direction at a speed of V, but cannot enter an inaccessible grid. Attack rules: The unit can only attack enemy targets within its firepower range. The damage value is determined by the unit's attack power and the enemy's defense power. The damage calculation formula is: D = AD f; Among them, A is the attack power of the attacking unit, D f is the defense of the defending unit, and D is the final damage value (if D<0, it is 0);

[0071] Victory conditions: The deduced victory conditions include one of the following two: all enemy units are defeated; the team successfully occupies the preset target area.

[0072] Step 2: Reinforcement Learning Strategy Generation and Optimization

[0073] This step aims to train a single agent through reinforcement learning algorithms to generate efficient combat strategies. The reinforcement learning model optimizes the agent's strategy execution in a dynamic battlefield environment by constructing a state space, action space, and reward function, combined with a deep learning algorithm.

[0074] First, define the state space S, action space A, and reward function R:

[0075] The state space S is the basis for the agent to perceive the battlefield situation. It contains battlefield map information, the attributes of its own units, and the attributes of the enemy units. It is specifically expressed as:

[0076] S=[G,S units ];

[0077] Among them, G is the battlefield grid map, which contains terrain, resource distribution and obstacle information; S units It is a collection of attributes of all units, including position p, fire range F, health H, and resources R. The state space vector S is dynamically updated to reflect battlefield changes in real time;

[0078] The action space S defines all possible behaviors of the agent, including: moving to the target location; attacking a specific unit; defending to improve survivability; resupplying to restore health or ammunition. Each action is represented by a vector a t indicates that its selection is based on the value output by the reinforcement learning policy network;

[0079] The reward function R is used to evaluate the effectiveness of the agent's behavior and guide the optimization of the strategy; its design includes the following key indicators: the damage caused to the enemy D damage 、Own losses D loss and completion of mission objectives C objective , the specific formula is:

[0080] R=α1·D damage -α2·D loss +α3·C objective ;

[0081] Among them, α1, α2, and α3 are the weight parameters of the reward function, which are used to balance the importance of different objectives.

[0082] Then, the strategy is generated by the reinforcement learning model, and the Deep Q-Network (DQN) algorithm is used to optimize the agent strategy. The state-action value function Q(s,a) is fitted through the neural network. The network structure is a three-layer fully connected network, such as Figure 2 Specifically, it includes:

[0083] The input layer receives the current state vector s t ;

[0084] The hidden layer consists of 128 neurons, and the activation function is ReLU;

[0085] The output layer corresponds to the Q value of each action in the action space;

[0086] Among them, the update rule of Q value is:

[0087] Q(s,a)=Q(s,a)+η·[r+γ·maxQ(s ′ ,a ′ )-Q(s,a)];

[0088] Among them, η is the learning rate, r is the immediate reward, γ is the discount factor used to balance short-term and long-term benefits, and s ′ is the next state, a ′ For the next action.

[0089] During training, the agent stores each interaction data (state, action, reward, next state) in the experience pool, and randomly samples the data in batches for network updates to reduce sample correlation and improve training efficiency. The mean square error (MSE) is used to optimize Q-value prediction, which is specifically defined as follows:

[0090]

[0091] Where N is the number of batch training samples and γ is the discount factor used to weigh current rewards and future rewards.

[0092] Step 3: Red vs. Blue Mechanism Setup

[0093] A dynamic game system of two reinforcement learning agents, the red team (enemy) and the blue team (self), is constructed to simulate the confrontation process in the real battlefield, thereby optimizing the robustness and intelligence of the strategy. Through multiple rounds of confrontation training, the mechanism continuously improves the adaptability of the strategies of both sides, making the deduction results closer to the actual battlefield situation. The red-blue confrontation mechanism includes agent modeling, confrontation training process, reward update mechanism and strategy convergence judgment. The entire confrontation mechanism setting process is as follows: Figure 3 As shown, specifically:

[0094] (1) Modeling of Red and Blue Agents: The Red and Blue agents are each used as a reinforcement learning model. Each agent is trained and competes in the same battlefield environment. Each agent consists of a state space S and an action space A.

[0095] The state space includes the battlefield map G and the unit attributes S blue 、S red The battlefield map G is a grid-like two-dimensional representation. Troop attributes include position p, fire range F, health H, and resources R. The red and blue agents share the same state representation: S = [G, S blue ,S red ];

[0096] The action space defines the operations that the agent can perform, including movement, attack, defense, and supply; each action is represented by a vector a t Indicates, for example, a t =[move, attack target], where move is the direction instruction and attack target is the specified enemy unit.

[0097] (2) Adversarial training process: The adversarial training process adopts a dual-agent parallel optimization mechanism, which specifically includes the following steps:

[0098] Initialize the battlefield environment: At the beginning of each round of training, initialize the battlefield map and the status of both sides' troops to ensure that both sides start the confrontation under fair conditions;

[0099] The red and blue agents make decisions alternately: The red and blue agents perform actions in turn according to the turn-based system. Each round of deduction is divided into three steps: Perceive the current state s t ; Select action a based on the current policy network t =argmaxQ(s t ,a); perform action a t , update battlefield status s t+1 ;

[0100] Update the reward function: assign rewards to both agents based on the results of their actions. The reward function R is designed to guide the agents to optimize their strategies. The reward function formula is:

[0101] R=α1·D damage -α2·D loss +α3·C objective ;

[0102] Among them, D damage is the damage caused by the agent to the enemy, D loss For your own losses, C objective is the completion of the task goal, and α1, α2, and α3 are the weight parameters of the reward function, which are adjusted according to training requirements.

[0103] (3) Reinforcement learning model optimization: The strategies of the red and blue agents are optimized by the reinforcement learning algorithm. The core of the training lies in calculating the Q-value update and adjusting the policy network parameters. Specifically, the strategies of both agents are based on the deep Q-learning (DQN) algorithm, and the state value function Q(s,a) is updated by the following formula:

[0104] Q(s,a)=Q(s,a)+η·[r+γ·maxQ(s ′ ,a ′ )-Q(s,a)];

[0105] Among them, η is the learning rate, r is the immediate reward, γ is the discount factor used to balance short-term and long-term benefits, and s ′ is the next state, a ′ For the next action;

[0106] Use the experience replay mechanism to store training data, batch sample N data for back propagation to optimize network parameters, and the loss function is:

[0107]

[0108] (4) Convergence of adversarial training: In order to ensure the balance of the red and blue strategies and the stability of training, the adversarial balance mechanism and convergence conditions are set:

[0109] Counterbalance mechanism: When the unilateral winning rate of the red or blue team is too high, the winning team's strategy disturbance is increased (such as increasing the exploration probability) or the reward weight of the weaker team is adjusted to balance the strategies of both sides.

[0110] Convergence conditions: Adversarial training stops when one of the following conditions is met: the winning rate of both sides is close to equilibrium (50±5%); the update amplitude of the strategy is less than the set threshold (the Q value change rate is less than 0.01).

[0111] Step 4: Visualization of deduction results

[0112] First, the dynamic display of the simulation results is based on a grid battlefield map, which updates the battlefield status in real time, including unit positions, resource distribution, and action status. The battlefield map is presented in the form of a two-dimensional grid, and each grid unit g i,j Indicates the terrain attributes and status of the current location; friendly units and enemy units are displayed separately with different colors and symbols, and their status (such as health points and resource amounts) are marked on the corresponding units in the form of values or icons; the firepower range is represented by a circular area centered on the unit, and the area within the range will be dynamically highlighted to show the firepower coverage. The firepower intensity is displayed through a heat map. The firepower coverage value is calculated as follows:

[0113]

[0114] Among them, Fi is the firepower value of unit i, d i The distance from the current grid (x, y) to unit i.

[0115] Secondly, the dynamic visualization of the action path records the movement and attack trajectory of each unit during the simulation. The unit's movement path is marked with arrows, and color coding indicates the change of unit status, such as green for healthy status and red for damaged or near-destruction status. Attack actions are connected by lines to represent the relationship between the target and the attacker, and the attack damage value is displayed at the target location. The damage value calculation formula is:

[0116] D=AD f ;

[0117] Among them, A is the attack power of the attacking unit, D f The defense power of the defending unit.

[0118] In addition, the system generates key indicators and data statistics for the simulation through a real-time data monitoring module; loss statistics display the unit loss trends of both the friendly and enemy sides in the form of line graphs or bar graphs, including the reduction in health points, the number of unit losses, etc.; battlefield control is displayed through a dynamic change graph of the area occupation rate, using area graphs or percentages to describe the dynamic changes in the friendly side's control over key areas; resource consumption is displayed through a time series graph to help users analyze whether resource allocation in the simulation is reasonable.

[0119] Finally, after the simulation is complete, the system automatically generates a decision report. This report includes a strategy effectiveness analysis, unit efficiency statistics, and improvement suggestions. The strategy effectiveness analysis quantifies the strengths and weaknesses of one's own strategy; the unit efficiency statistics list high- and low-efficiency units; and the improvement suggestions combine simulation data to propose optimization solutions.

[0120] Testing has verified that this invention can generate adaptive strategies in a variety of complex battlefield scenarios. Compared with traditional rule-based deduction methods, it significantly improves deduction efficiency, strategy robustness, and adversarial resistance. The red-blue adversarial training mechanism makes strategies more diverse and intelligent, while dynamic battlefield modeling further enhances the system's versatility and adaptability. This invention provides a new solution for intelligent wargaming, with important military applications and broad prospects for promotion.

[0121] The preferred embodiments of the present invention are described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the scope of protection of the present invention.

Claims

1. An intelligent war game deduction method based on reinforcement learning, characterized in that: The following steps are involved: Step S1: Build a dynamic battlefield model: Build an adjustable battlefield simulation platform, define the terrain, resources, and troop distribution elements in the battlefield, support dynamic switching and rule adjustment of multiple scenarios, and provide continuous state space input for the reinforcement learning agent; Step S2: Generate and optimize reinforcement learning strategies: Design an intelligent strategy generation algorithm based on deep reinforcement learning. By constructing a state space and an action space and combining it with a situation reward function, an efficient combat strategy is generated over multiple rounds of training. The intelligent strategy generation algorithm based on deep reinforcement learning uses an improved strategy optimization algorithm to improve the stability and efficiency of strategy generation. Step S3: Develop a red-blue confrontation mechanism: Introduce a dual-agent game training mechanism, model the red and blue sides as reinforcement learning agents, and simulate real battlefield games through multiple rounds of alternating training; During training, the Red and Blue agents optimize their strategies through dynamic game play to simulate the confrontation and complexity of real warfare. Step S4, visualize the simulation results: Provide dynamic decision visualization function, which can display the battlefield situation, troop dynamics and strategy execution process in real time; at the same time, generate a simulation report and present the victory and defeat analysis, strategy advantages and disadvantages and improvement suggestions, providing reference for military training and decision support.

2. The intelligent war game deduction method based on reinforcement learning according to claim 1, characterized in that: The construction of the dynamic battlefield model uses a grid map to represent the battlefield environment, and models the terrain, resources, unit status and rules in the battlefield in a digital manner, specifically including: Step S11: Construct the battlefield map into a two-dimensional grid space, where each grid cell represents a location on the battlefield. The attributes of the grid include terrain type, accessibility, and resource distribution, and are defined as: G={g i,j |1≤i≤N,1≤j≤M}; where N and M are the number of rows and columns of the map respectively. The state of each grid can be further represented by the vector g i,j =[T,P,R], where T is the terrain type, P is the accessibility attribute, and R is the resource attribute; Step S12: Model the status of the troop units on the map with dynamic attributes, including two types: friendly units and enemy units. The attributes of each unit include position p = (x, y), health value H, firepower range F, movement speed V and resource status R. u , the initial state matrix S records the basic information of all units and is dynamically updated during the deduction process. The unit state is expressed as: S unit ={(p,H,F,V,R u )}; The unit's firing range F is defined as Euclidean distance: Only enemy units that satisfy d(u,v)≤F will be attacked; Step S13: Define battlefield rules to regulate the actions of the troops and the conditions for winning or losing in the deduction. The basic rules include: the movement rule is that the unit can move in any direction at a movement speed V each round, but cannot enter an impassable grid; the attack rule is that the unit can only attack enemy targets within its firepower range, and the damage value is determined by the unit's attack power and the enemy's defense power. The damage calculation formula is: D=AD f , where A is the attack power of the attacking unit, D f is the defense of the defending unit, and D is the final damage value; Victory conditions: The deduced victory conditions include one of the following two: all enemy units are defeated; the team successfully occupies the preset target area.

3. The intelligent war game deduction method based on reinforcement learning according to claim 1, characterized in that: The generation of reinforcement learning strategies and optimization of reinforcement learning models optimizes the strategy execution effect of intelligent agents in dynamic battlefield environments by constructing state space, action space and reward function, combined with deep learning algorithms, specifically including: Step S21, setting state space S: The state space S is the basis for the agent to perceive the battlefield situation, including battlefield map information, attributes of its own units and enemy units, specifically expressed as: S=[G,S units ]; where G is the battlefield grid map, which contains terrain, resource distribution and obstacle information; S units It is a collection of attributes of all units, including position p, fire range F, health H and resources R. The state space vector S is dynamically updated to reflect battlefield changes in real time. Step S22, set action space A: Action space A defines all possible actions of the agent, including: moving to the target location, attacking a specific unit, defending to improve survivability, and resupplying to restore health or ammunition. Each action behavior is represented by vector a. i Indicates that its selection is based on the value output by the reinforcement learning policy network, and the action space is expressed as: A={a i ∣a i ∈valid move of piece i}; where each action a i Corresponds to an action behavior; Step S23, design reward function R: The reward function R is used to evaluate the effectiveness of the agent's behavior and guide the optimization of the strategy. Its design includes the following key indicators: the damage caused to the enemy D damage 、Own losses D loss and completion of mission objectives C objective , the specific formula is: R=α1·D damage -α2·D loss +α3·C objective ; Among them, α1, α2, and α3 are the weight parameters of the reward function, which are used to balance the importance of different goals; Step S24: Generate reinforcement learning strategy: Generate strategy through reinforcement learning model, use deep Q learning algorithm to optimize agent strategy, and fit state-action value function Q(s,a) through neural network. The Q value update rule is: Q(s,a)=Q(s,a)+η·[r+γ·maxQ(s′,a′)-Q(s,a)]; where η is the learning rate, r is the immediate reward, γ is the discount factor used to balance short-term and long-term benefits, s′ is the next state, and a′ is the next action; Step S25: During the training process, the agent stores each interaction data into the experience pool, randomly samples the data in batches for network update, reduces sample correlation, improves training efficiency, and uses the mean square error (MSE) to optimize Q-value prediction. The specific definition is as follows: Where N is the number of batch training samples and γ is the discount factor used to weigh current rewards and future rewards.

4. The intelligent war game deduction method based on reinforcement learning according to claim 1, characterized in that: The formulation of the red-blue confrontation mechanism includes: Step S31, modeling the red and blue agents: The red and blue agents are each used as a reinforcement learning model. Each agent is trained and competes in the same battlefield environment. Each agent includes a state space S and an action space A. The state space includes battlefield map G, troop attributes S blue 、S red The battlefield map G is a grid-like two-dimensional representation. Troop attributes include position p, fire range F, health H, and resources R. The red and blue agents share the same state representation: S=[G,S blue ,S red ]; The action space defines the operations that the agent can perform, including movement, attack, defense, and supply. Each action is represented by a vector a t Indicates that a t =[move,attack target], where move is the direction command and attack target is the specified enemy unit; Step S32, adversarial training process: using dual agents to conduct adversarial training and perform mechanism optimization, specifically including the following steps: Step S321, initializing the battlefield environment: At the beginning of each round of training, the battlefield map and the status of both sides' troops are initialized to ensure that both sides start the confrontation under fair conditions; Step S322, Red and Blue agents make decisions alternately: The Red and Blue agents perform actions in turn according to the round-robin system. Each round of deduction is divided into three steps: Perceive the current state s t ; Select action a based on the current policy network t =argmaxQ(s t ,a); perform action a t , update battlefield status s t+1 ; Step S323, update the reward function: assign rewards to both agents based on the action results. The reward function R is designed to guide the agents to optimize their strategies. The reward function formula is: R=α1·D damage -α2·D loss +α3·C objective Among them, D damage is the damage caused by the agent to the enemy, D loss For your own losses, C objective is the completion of the task goal, and α1, α2, and α3 are the weight parameters of the reward function, which are adjusted according to training requirements.

5. The intelligent war game deduction method based on reinforcement learning according to claim 4 is characterized in that: The formulation of the red-blue confrontation mechanism also includes: Step S33, reinforcement learning model optimization: The reinforcement learning algorithm is used to optimize the strategies of the red and blue agents. The optimization strategies of both agents are based on the deep Q-learning algorithm, and their state value functions Q(s,a) are updated by the following formula: Q(s,a)=Q(s,a)+η·[r+γ·maxQ(s′,a′)-Q(s,a)]; where η is the learning rate, r is the immediate reward, γ is the discount factor used to balance short-term and long-term benefits, s′ is the next state, and a′ is the next action; Use the experience replay mechanism to store training data, batch sample N data for back propagation to optimize network parameters, and the loss function is: Step S34, adversarial training convergence: setting the adversarial balance mechanism and convergence conditions to ensure the balance of the red and blue strategies and the stability of the training, including: Counterbalance mechanism: When the unilateral winning rate of the red or blue team is too high, the winning team's strategy disturbance is increased or the reward weight of the weaker team is adjusted to balance the strategies of both sides; Convergence conditions: Adversarial training stops when one of the following conditions is met: the winning rate of both sides is close to equilibrium, that is, the winning rate is 50±5%; the update amplitude of the strategy is less than the set threshold, that is, the Q value change rate is less than 0.

01.

6. The intelligent war game deduction method based on reinforcement learning according to claim 1, characterized in that: The visualization deduction results specifically include: Dynamic battlefield display: Based on the grid battlefield map, the battlefield status is updated in real time, including unit positions, resource distribution and action status; the battlefield map is presented in the form of a two-dimensional grid, and each grid unit is g i,j Indicates the terrain attributes and status of the current location. Friendly and enemy units are distinguished by different colors and symbols, and their status is marked on the corresponding units in the form of numerical values or icons. The firepower range is represented by a circular area centered on the unit. The area within the range will be dynamically highlighted to show the firepower coverage. The firepower intensity is displayed through a heat map. The firepower coverage value is calculated as follows: Among them, F i is the firepower value of unit i, d i is the distance from the current grid (x, y) to unit i; Dynamic display of action paths: Records the movement and attack trajectory of each unit during the simulation. The unit's movement path is marked with arrows, and color coding indicates the change of unit status. Attack actions are connected by lines to represent the relationship between the target and the attacker. Attack damage values are displayed at the target position. The damage value calculation formula is: D=AD f ; Among them, A is the attack power of the attacking unit, D f The defense of the defending unit.

7. The intelligent war game deduction method based on reinforcement learning according to claim 6, characterized in that: The visualization deduction results also include: Key indicators and data statistics display: Loss statistics display friendly and enemy unit loss trends in the form of line graphs or bar graphs, including health reduction and number of units lost; battlefield control is displayed through a dynamic change chart of area occupation rate, and the dynamic changes in friendly control of key areas are described in the form of area charts or percentages; resource consumption is displayed through time series graphs to help users analyze whether resource allocation in the game is reasonable; Decision report presentation: The report content includes strategy effect analysis, unit efficiency statistics and improvement suggestions: Strategy effect analysis quantifies the advantages and disadvantages of one's own strategy; unit efficiency statistics lists efficient and inefficient units; improvement suggestions combine deduction data to propose optimization solutions.

Images