A reinforcement learning combat decision explanation method for a war game and related products

Abstract

This application provides a reinforcement learning-based method and related products for interpreting wargaming operational decisions. The method first visualizes the raw wargaming situation data to generate visualized situation information. Then, it performs structured feature analysis on the visualized situation information to extract situation feature vectors containing multiple operational elements. Further, it utilizes a pre-constructed DQN-LSTM hybrid neural network, combined with a preset set of tactical actions and the situation feature vectors, to evaluate the benefits of tactical actions, generating a visualized decision-making chain while outputting the target tactical actions. Finally, by executing the target tactical actions and acquiring environmental state change data, it performs reward decomposition on the changed data to obtain the reward decomposition result for the target tactical action. This quantitatively evaluates the decision-making effect from multiple operational evaluation dimensions, effectively improving the interpretability of operational decisions.

Description

Technical Field

[0001] This application relates to the field of wargaming technology, and in particular to a reinforcement learning-based method for interpreting wargaming operational decisions and related products. Background Technology

[0002] Currently, while reinforcement learning has achieved a high win rate in wargaming, its deep neural network models remain a black box throughout the entire process from situational input to tactical output. Commanders cannot ascertain the specific basis for the agent's decisions, leading to insufficient trust in the decision support system. Existing general interpretability methods are mostly geared towards general tasks such as image classification, with outputs consisting only of heatmaps or single feature contribution values. They cannot be mapped to military semantic information such as troop strength comparisons or terrain advantages, and are insufficient to reflect the temporal reasoning process of sequential decision-making. Furthermore, existing reward functions provide feedback with a single scalar value, making it impossible for commanders to distinguish whether the reward comes from specific factors such as eliminating the enemy, target advancement, or friendly losses. The value assessment of actions is also implicit within the model. Therefore, the current application of reinforcement learning in wargaming suffers from poor interpretability of operational decisions in wargaming scenarios. Summary of the Invention

[0003] In view of the above problems, in order to improve the interpretability of wargaming-based operational decisions, this application provides a reinforcement learning-based operational decision interpretation method and related products for wargaming.

[0004] The embodiments of this application disclose the following technical solutions: In a first aspect, embodiments of this application provide a reinforcement learning-based method for interpreting combat decisions in wargaming simulations, the method comprising: Visualized situation information is obtained by visualizing the original situation data in the wargaming simulation environment; the wargaming simulation environment includes multiple types of wargaming combat units. Based on the visualized situation information, battlefield situation characteristics are analyzed to obtain a situation feature vector; Based on a pre-built DQN-LSTM hybrid neural network, the benefits of tactical actions are evaluated according to a preset set of tactical actions and the situation feature vector, and the target tactical actions and the visualized decision-making chain for the target tactical actions are determined. The target tactical actions control each of the wargaming combat units to execute tactical actions, and determine the environmental state change data of the wargaming simulation environment corresponding to each wargaming combat unit after executing the tactical actions; The reward decomposition is performed based on the environmental state change data to obtain the reward decomposition result for the target tactical action; the reward decomposition result is used to characterize the evaluation result of the target tactical action under different combat evaluation dimensions.

[0005] In one possible implementation, the preset tactical action set includes multiple candidate tactical actions; The pre-built DQN-LSTM hybrid neural network evaluates the benefits of tactical actions based on a preset set of tactical actions and the situation feature vector, determines the target tactical actions, and establishes a visualized decision-making chain for the target tactical actions, including: Based on the DQN-LSTM hybrid neural network, the estimated action benefit value corresponding to each candidate tactical action is calculated according to the situation feature vector. Based on the flexible maximum value strategy function, the action selection probability is transformed according to the estimated action benefit value corresponding to each candidate tactical action to obtain a candidate action probability set; the candidate action probability set includes the action selection probability corresponding to each candidate tactical action individually. The candidate tactical action with the highest action selection probability in the candidate action probability set is determined as the target tactical action; The visual decision link is obtained by semantically transforming the situation features corresponding to the target tactical actions within the situation feature vector.

[0006] In one possible implementation, after calculating the estimated action gain value corresponding to each candidate tactical action based on the situation feature vector, the method further includes: Based on the action advantage function, the action advantage degree of each candidate tactical action is calculated according to the estimated action benefit value of each candidate tactical action; and, Based on the state value function, the state baseline value for all the candidate tactical actions is calculated according to the estimated action benefit value of each candidate tactical action. Based on the action advantage degree of each candidate tactical action, the estimated action benefit value, and the state baseline value for all candidate tactical actions, action value display information is generated for each candidate tactical action.

[0007] In one possible implementation, the target tactical action controls each of the wargaming combat units to execute tactical actions, and determines the environmental state change data corresponding to the wargaming simulation environment after each of the wargaming combat units executes the tactical actions, including: Based on the target tactical actions, tactical planning is performed to generate tactical combat instructions for each of the wargame combat units; Send the corresponding tactical combat instructions to each of the wargaming combat units, and obtain the execution results of the combat instructions from each of the wargaming combat units; Environmental state analysis is performed based on the execution results of combat instructions fed back by each wargaming combat unit to obtain environmental state change data.

[0008] In one possible implementation, the environmental state change data includes: enemy casualties, friendly casualties, mission progress changes, terrain utilization indicators, and action execution time. The step of decomposing the reward based on the preset reward coefficient corresponding to the target tactical action and the environmental state change data to obtain the reward decomposition result for the target tactical action includes: Based on the type of tactical action corresponding to the target tactical action, the enemy's battle loss reward coefficient, mission objective reward coefficient, friendly loss penalty coefficient, and terrain reward coefficient corresponding to the type of tactical action are determined from a preset reward coefficient mapping table. The following calculations are performed: Calculate the enemy elimination reward value based on the enemy casualty count and the enemy casualty elimination reward coefficient; calculate the mission objective reward value based on the mission progress change and the mission objective reward coefficient; calculate the friendly casualty penalty value based on the friendly casualty count and the friendly casualty penalty coefficient; calculate the execution time penalty value based on the fixed time penalty constant and the action execution time; and calculate the terrain utilization reward value based on the terrain utilization identifier and the terrain reward coefficient. The reward decomposition result is determined by the reward value for eliminating the enemy, the reward value for the mission objective, the penalty value for friendly losses, the penalty value for execution time, and the reward value for terrain utilization.

[0009] In one possible implementation, after obtaining the reward decomposition result for the target tactical action, the method further includes: Based on the reward analysis results and the target tactical actions, a state transition experience is constructed; The state transition experience is stored in the experience replay buffer, and a preset batch of data is sampled from the experience replay buffer; Based on the preset batch data, the temporal difference error is calculated according to the temporal difference learning method, and the network parameters of the DQN-LSTM hybrid neural network are updated according to the temporal difference error to obtain the updated network parameters.

[0010] In one possible implementation, the situation feature vector includes: force situation features, terrain environment features, combat effectiveness features, and tactical situation features.

[0011] In one possible implementation, the method is applied to an agent, and after obtaining the reward decomposition result for the target tactical action, the method further includes: The exploration rate of the agent is exponentially decayed based on the ε-greedy strategy control exploration algorithm.

[0012] Secondly, embodiments of this application provide a reinforcement learning-based combat decision interpretation system for wargaming simulations, the system comprising: The first visualization module is used to perform visualization processing on the original situation data in the wargaming simulation environment to obtain visualized situation information; the wargaming simulation environment includes multiple types of wargaming combat units; The feature analysis module is used to perform battlefield situation feature analysis based on the visualized situation information to obtain a situation feature vector; The second visualization module is used to evaluate the benefits of tactical actions based on a pre-built DQN-LSTM hybrid neural network, according to a preset set of tactical actions and the situation feature vector, to determine the target tactical actions and the visualized decision-making chain for the target tactical actions. The action execution module is used to control each of the wargaming combat units to execute tactical actions through the target tactical actions, and to determine the environmental state change data of the wargaming simulation environment corresponding to each wargaming combat unit after executing the tactical actions. The reward decomposition module is used to decompose rewards based on the environmental state change data to obtain reward decomposition results for the target tactical actions; the reward decomposition results are used to characterize the evaluation results of the target tactical actions under different combat evaluation dimensions.

[0013] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any possible wargaming method for reinforcement learning-based operational decision interpretation in the first aspect.

[0014] Compared to existing technologies, this application offers the following advantages: This application provides a reinforcement learning-based operational decision interpretation method and related products for wargaming simulations. First, the original wargaming situation data is visualized to generate visualized situation information. Then, structured feature analysis is performed on the visualized situation information to extract situation feature vectors containing multiple operational elements. Further, a pre-constructed DQN-LSTM hybrid neural network is used, combined with a preset set of tactical actions and situation feature vectors, to evaluate the benefits of tactical actions. Simultaneously, a visualized decision-making chain is generated while outputting the target tactical actions, thus fully demonstrating the reasoning process from feature analysis to action value evaluation, improving the interpretability of single tactical decisions. Finally, by executing the target tactical actions and acquiring environmental state change data, reward decomposition is performed on the change data to obtain the reward decomposition result for the target tactical action, quantifying the decision-making effect from multiple operational evaluation dimensions. These steps are sequentially linked, forming a complete interpretation mechanism from situational awareness, feature analysis, decision reasoning to effect evaluation, effectively improving the interpretability of operational decisions. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a reinforcement learning-based combat decision interpretation method for wargaming provided in this application embodiment; Figure 2 A flowchart illustrating a target tactical maneuver and its visualization decision-making link determination method provided in an embodiment of this application; Figure 3 A schematic diagram of a visual decision-making chain provided in an embodiment of this application; Figure 4 A flowchart illustrating a method for determining environmental state change data provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a reinforcement learning combat decision interpretation system for wargaming provided in an embodiment of this application. Detailed Implementation

[0017] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0018] As described earlier, while reinforcement learning has achieved a high win rate in wargaming, its deep neural network models remain a black box throughout the entire process from situational input to tactical output. Commanders cannot ascertain the specific basis for the agent's decisions, leading to insufficient trust in the decision support system. Existing general interpretability methods are mostly geared towards general tasks such as image classification, with outputs consisting only of heatmaps or single feature contribution values. They cannot be mapped to military semantic information such as troop strength comparisons or terrain advantages, and are insufficient to reflect the temporal reasoning process of sequential decision-making. Furthermore, existing reward functions provide feedback with a single scalar value, making it impossible for commanders to distinguish whether the reward comes from specific factors such as eliminating the enemy, target advancement, or friendly losses. The value assessment of actions is also implicit within the model. Therefore, the current application of reinforcement learning in wargaming suffers from poor interpretability of operational decisions in wargaming.

[0019] Based on this, this application provides a reinforcement learning-based operational decision interpretation method and related products for wargaming simulations. In this method, the original situational data from the wargaming simulation is first visualized to generate visualized situational information. Then, structured feature analysis is performed on the visualized situational information to extract situational feature vectors containing multiple operational elements. Further, a pre-constructed DQN-LSTM hybrid neural network is used, combined with a preset set of tactical actions and the situational feature vectors, to evaluate the benefits of tactical actions. A visualized decision-making chain is generated simultaneously with the output of the target tactical actions, thus fully demonstrating the reasoning process from feature analysis to action value evaluation, improving the interpretability of single tactical decisions. Finally, by executing the target tactical actions and acquiring environmental state change data, reward decomposition is performed on the change data to obtain the reward decomposition result for the target tactical action, quantifying the decision-making effect from multiple operational evaluation dimensions. These steps are sequentially linked, forming a complete interpretation mechanism from situational awareness, feature analysis, decision reasoning to effect evaluation, effectively improving the interpretability of operational decisions.

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0021] It should be noted in advance that the reinforcement learning combat decision interpretation method for wargaming provided in this application embodiment can be applied to intelligent agents that provide combat decision services, or to non-intelligent agent scenarios such as command decision deployment platforms. This embodiment does not limit the specific execution object of this method.

[0022] See Figure 1 The figure is a flowchart illustrating a reinforcement learning-based combat decision interpretation method for wargaming provided in an embodiment of this application, specifically including the following steps: S101: Visualize the original situation data in the wargaming simulation environment to obtain visualized situation information; the wargaming simulation environment includes multiple types of wargaming combat units.

[0023] In a wargaming simulation environment, it is necessary to acquire raw situational data in real time at the current simulation moment. This data includes unstructured or semi-structured data such as battlefield terrain grids, the positions and types of combat units on both sides (enemy and friendly), and the status information of each unit. To improve the efficiency of commanders' perception of the battlefield situation, the system visualizes the above raw data. First, the battlefield terrain is discretized into a two-dimensional grid of M rows and N columns. Each grid cell is labeled with six terrain attributes: plains, mountains, forests, water areas, urban areas, and roads, and distinguished by different backgrounds or colors. Second, each combat unit on both sides (such as mechanized infantry companies, tank companies, self-propelled artillery companies, reconnaissance platoons, etc.) is marked with specific icons at the corresponding grid positions. Red units are deployed in the area below the grid, and blue units are deployed in the area above the grid, with the middle area serving as a combat buffer zone. The system also automatically calculates the troop strength comparison based on the number of units on both sides and statistically analyzes quantitative indicators such as the number of threat areas, the coverage of the red side's visible range, and the proportion of the red side's controlled area. At the same time, it determines the comprehensive situational assessment level based on the red-blue troop strength ratio. All the aforementioned graphical elements, quantitative statistical results, and level determination information are integrated into unified visual situation information and simultaneously presented on the display terminal of the command personnel.

[0024] For a specific example of visualized situational information, taking a typical simulation scenario as an example, the battlefield terrain is represented by a 10x12 two-dimensional grid, totaling 120 grid cells. The area below the grid (rows 6 to 9) contains Red Force combat units: a mechanized infantry company at grid (8,3), a tank company at (8,5), a self-propelled artillery company at (9,2), a reconnaissance platoon at (7,8), an army aviation unit at (9,10), and a support unit at (9,6), totaling 6 Red Force units. The area above the grid (rows 0 to 3) contains Blue Force units: one Blue Force unit each at (0,4) and (0,7) in row 0, one Blue Force unit each at (1,2), (1,5), and (1,9) in row 1, and one Blue Force unit each at (2,6) and (2,8) in row 2, totaling 7 Blue Force units. The central area (rows 4 and 5) is a combat buffer zone, currently undefended. Regarding terrain attributes, columns 0-3 on the left are mountains (dark brown), columns 4-8 in the middle are plains (light green), and columns 9-11 on the right are woodlands (dark green). A river runs from row 3 across columns 4-6 (marked in blue). The left panel of the visualization interface displays the terrain and unit deployments in a grid format, with red units represented by green icons and blue units by red icons. The statistics panel on the right side of the grid displays in real-time: 6 red units, 7 blue units, a force ratio of 0.86, 15 threat areas, 42% red visibility coverage, 35% red controlled area, and an overall situation assessment level automatically determined as disadvantageous based on the force ratio (blue force greater than red force but not exceeding 1.5 times).

[0025] S102: Based on the visualized situation information, perform battlefield situation feature analysis to obtain a situation feature vector.

[0026] After obtaining the visualized situational information, the information is structured and quantitatively calculated to extract situational features that characterize the current key state of the battlefield. Specifically, the basic data, such as battlefield terrain grid data, the location and type statistics of red and blue force combat units, the state parameters of each unit, and the distribution of control areas, are first extracted from the visualized situational information. Based on this, feature values ​​are calculated according to four categories: force situation, terrain environment, combat effectiveness, and tactical situation. The situational feature vector includes force situation features, terrain environment features, combat effectiveness features, and tactical situation features. Among them, force situation features include the ratio of the number of red and blue force combat units, the degree of advantage of armored units, the degree of concentration of firepower in a unit area, and the comprehensive evaluation value of the overall mobility of the troops. The calculation of these features requires traversing the type and location information of all combat units. The terrain environment features are calculated based on the terrain attribute labels of each grid unit and the current position of the red force, calculating the degree of terrain advantage for combat operations, the terrain obstruction range of enemy observation and firepower, the terrain accessibility of the forward direction, and the control ratio of key high points on the battlefield.

[0027] Combat effectiveness characteristics reflect the Red Force's logistical and status support capabilities, including the proportion of remaining ammunition to the authorized strength, the proportion of remaining vehicle fuel to full capacity, the combat readiness rate of participating personnel, and the smoothness of the command and communication system. Tactical situation characteristics focus on battlefield layout and risk control, including the degree of protection of flanks against enemy threats, the completeness of the defensive position depth configuration, the availability of reserve forces, and the feasibility of rearward transfer routes. All 16 characteristics are normalized during calculation, mapping the raw data to the range of 0 to 1 to ensure comparability of characteristics with different dimensions. Each characteristic is expanded into a 3-dimensional sub-vector through a three-layer fully connected network, and the sub-vectors of the 16 characteristics are finally concatenated to form a 48-dimensional real-valued feature vector. For interpretability, the names, values, and categories of the 16 characteristics are displayed synchronously in a grouped list, with color coding indicating the quality level of the feature values ​​(green for good, blue for average, and orange for poor), enabling commanders to clearly understand the specific situational factors and their current status that are of concern in decision-making.

[0028] S103: Based on a pre-built DQN-LSTM hybrid neural network, the tactical action benefit is evaluated according to the preset tactical action set and the situation feature vector, and the target tactical action and the visualized decision-making link for the target tactical action are determined.

[0029] In this embodiment, the preset set of tactical actions includes multiple candidate tactical actions. For example, these candidate tactical actions include: frontal assault, flanking maneuver, fire suppression; defensive fortification; tactical withdrawal; reconnaissance and infiltration; artillery barrage; and coordinated attack. These eight actions constitute the selectable action space for each decision cycle. Each action has a clear tactical semantics, which facilitates subsequent value assessment and decision interpretation.

[0030] After obtaining the situation feature vector, a pre-constructed DQN-LSTM hybrid neural network is used as input. This vector, combined with multiple candidate tactical actions from a pre-defined set of tactical actions, is used to evaluate the benefit of each candidate action in the current wargaming simulation environment. This hybrid neural network integrates a Deep Q-Network (DQN) and a Long Short-Term Memory (LSTM) network. The LSTM layer captures the temporal dependencies of historical decision cycles, enabling the model to adjust its evaluation based on the dynamic evolution of the battlefield situation. The DQN part outputs the estimated benefit value (denoted as the Q-value) for each candidate tactical action through fully connected layers. Based on these Q-values, the final target tactical action to be executed in the simulation environment is selected from the multiple candidate actions.

[0031] Next, with reference to the accompanying drawings of a specific process embodiment, the process of determining the target tactical actions and the corresponding visualized decision-making link in step S103 will be described.

[0032] See Figure 2 The figure is a flowchart illustrating a target tactical maneuver and its visualization decision-making link determination method provided in an embodiment of this application, specifically including the following steps: S1031: Based on the DQN-LSTM hybrid neural network, calculate the estimated action benefit value corresponding to each candidate tactical action according to the situation feature vector.

[0033] First, based on the pre-constructed DQN-LSTM hybrid neural network, the estimated action reward value, i.e., the Q-value, for each candidate tactical action in the preset tactical action set is calculated according to the situation feature vector generated in step S102. For example, the preset tactical action set includes eight candidate actions with clear military semantics, including frontal assault, flanking maneuver, fire suppression, defensive fortification, tactical withdrawal, reconnaissance and infiltration, artillery barrage, and coordinated attack. The DQN-LSTM network captures the temporal dependencies of historical decision cycles through its LSTM layers, enabling the evaluation of the current Q-value to reference the dynamic evolution of the battlefield situation, rather than viewing the current state in isolation. Subsequently, the fully connected layers of the network output the Q-value corresponding to each action, which quantitatively reflects the expected cumulative reward that the agent can obtain after performing a certain tactical action under the current battlefield situation.

[0034] In this embodiment, the design of the DQN-LSTM hybrid neural network fully considers the temporal dependence of wargame decision-making. The Long Short-Term Memory (LSTM) layer is responsible for sequentially modeling the situational feature vectors across multiple decision cycles to capture the dynamic evolution of the battlefield situation. For example, trends such as the gradual contraction of enemy forces, continuous depletion of friendly ammunition, and increasing flank threats over time. The hidden state output by the LSTM layer contains historical context information for the current moment and is then fed into the fully connected layer of the Deep Q-Network (DQN). After a nonlinear transformation, the fully connected layer outputs an 8-dimensional real-valued vector. Each component of this vector corresponds to the estimated reward value of a preset tactical action (frontal assault, flanking maneuver, fire suppression, defensive fortification, tactical retreat, reconnaissance and infiltration, artillery barrage, coordinated attack). The mathematical meaning of Q(s,a) is: the expected cumulative discounted reward that can be obtained by continuing to act according to the optimal strategy after performing action a under the current situational feature vector s. A higher value indicates that the model favors the long-term combat effect of the action in the current battlefield environment. This calculation process allows the value of actions, which was originally implicit within deep neural networks, to be explicitly quantified, providing a directly comparable numerical basis for subsequent strategy selection and value comparison.

[0035] S1032: Based on the flexible maximum value strategy function, the action selection probability is transformed according to the estimated action benefit value corresponding to each candidate tactical action to obtain a candidate action probability set; the candidate action probability set includes the action selection probability corresponding to each candidate tactical action individually.

[0036] Furthermore, in determining the target tactical action, instead of directly selecting the action with the highest Q value, a flexible maximum value policy function is introduced. This function converts the estimated action benefits of each candidate tactical action calculated above into corresponding action selection probabilities, thus forming a candidate action probability set. Through this conversion, Q values, which originally had large absolute differences, are mapped to a probability distribution with a sum of 1. The role of this probability distribution is twofold: firstly, it preserves the relative merits of Q values, with actions having higher Q values ​​having a greater probability of selection; secondly, by adjusting the temperature parameter in the flexible maximum value policy function, the steepness of the probability distribution can be controlled, thereby balancing exploration and exploitation in decision-making. In this way, the numerical judgments within the model can be transformed into probabilistic decision tendencies, providing commanders with intuitive quantitative indicators to understand the degree of hesitation or certainty of the agent among different options.

[0037] S1033: The candidate tactical action with the highest action selection probability in the candidate action probability set is determined as the target tactical action.

[0038] Subsequently, the candidate tactical action with the highest probability from the candidate action probability set is selected as the target tactical action to be executed in the current decision-making cycle. This step clarifies the final decision output, namely, the optimal tactical action selected after comprehensively considering the current situation characteristics and the long-term benefits of all candidate tactical actions. This target tactical action will serve as the basis for generating specific combat instructions and driving the actions of wargaming combat units.

[0039] S1034: Based on the target tactical actions, perform semantic transformation on the situation features corresponding to the situation feature vector to obtain the visualized decision link.

[0040] Finally, in step S1034, the process of generating a visualized decision-making link begins. During this process, based on the target's tactical maneuver, the most relevant situational feature values ​​are located and extracted from the situational feature vector obtained in step S102, and then semantic transformation is performed. Specifically, different tactical maneuvers are sensitive to different situational features. For example, if the selected target's tactical maneuver is a flanking maneuver, features such as troop strength comparison, maneuver index, and flank safety in the situational feature vector are automatically associated, and specific normalized values ​​of these features are extracted (e.g., maneuver index 0.68, flank safety 0.72). Subsequently, these quantified values ​​are combined with information such as the reasons for the action selection (e.g., a weak point on the right flank of the blue force's defense), troop deployment suggestions (e.g., suggesting the deployment of a tank company for encirclement), and decision confidence to automatically generate a natural language reasoning text that conforms to military thinking habits. This text is the core content of the visualized decision-making link. This fundamentally solves the black-box problem of opaque decision-making processes in traditional deep reinforcement learning models, thereby improving the interpretability of operational decisions.

[0041] For an example of visualizing the decision-making process, please refer to [link / reference]. Figure 3 The diagram below illustrates a visualized decision-making chain. The chain clearly presents the complete reasoning process from Q-value evaluation to strategy selection in a side-by-side panel format. The left-hand Q-value evaluation panel lists eight tactical actions and their corresponding Q-values ​​and advantage function values, intuitively displaying the expected returns of each action in the current situation. The right-hand strategy selection panel converts Q-values ​​into action selection probabilities based on a flexible maximum strategy, displaying candidate actions from highest to lowest probability. Each action is accompanied by auxiliary explanatory indicators such as expected return, risk assessment, and confidence level. Below the panel, the attention weight distribution displays the normalized weights of eight feature dimensions—troop strength, firepower, maneuverability, terrain, ammunition, communication, flanks, and depth—in a bar chart format. The bottom decision reasoning chain automatically converts the above quantitative information into natural language text. This chain fully connects the logical chain of why a particular tactical action was chosen as the target action, allowing commanders to intuitively trace the agent's decision-making basis.

[0042] The above is an introduction to the target tactical actions and visualization decision-making link determination process in step S103. Next, we will continue to combine... Figure 1 The following describes the subsequent steps S104 and S105 in the method of this embodiment.

[0043] S104: Control each of the wargaming combat units to execute tactical actions through the target tactical actions, and determine the environmental state change data corresponding to the wargaming simulation environment after each of the wargaming combat units executes the tactical actions.

[0044] In step S104, the target tactical action determined in step S103 is used as a high-level decision-making instruction. Through the instruction parsing and distribution mechanism built into the wargaming simulation environment, coordinated control is exercised over various wargaming combat units on the battlefield. Since the target tactical action itself is an abstract concept with military semantics, it cannot be directly executed by a single combat unit. Therefore, it is first necessary to decompose the abstract tactical action into a series of executable instructions for specific combat units based on the current battlefield situation, the current status of each combat unit, and the standard operating procedures in the tactical doctrine library. For example, if the target action is a flanking maneuver, the main force unit performing the maneuver (such as a tank company and mechanized infantry company) will be identified, a maneuver path will be planned for it, and a maneuver instruction will be issued; simultaneously, units requiring fire support (such as self-propelled artillery companies) will be identified, and a suppressive fire instruction will be issued; units requiring frontal containment will be identified, and a feint attack or defensive instruction will be issued. These instructions are sent to the corresponding combat unit model in the form of structured data packets through the simulation environment's communication interface, driving each unit to perform maneuvering, engagement, reconnaissance, and support actions synchronously or asynchronously in the virtual battlefield. The entire control process follows the time sequence and spatial relationship of combat coordination, ensuring that the actions of each unit form an organic coordination at the tactical level, thereby realistically simulating the command and control process in actual combat.

[0045] Correspondingly, after each wargaming combat unit completes its tactical actions, the wargaming simulation engine calculates the changes in the entire battlefield environment based on underlying simulation rules such as the engagement model, damage model, and maneuver model, and uses this to determine the environmental state change data. The environmental state change data reflects the quantitative differences in the battlefield situation from before to after execution, specifically including: enemy casualties, friendly casualties, changes in mission progress, terrain utilization indicators, and action execution time. Environmental state change data is the basis for understanding the consequences of actions and making subsequent decision adjustments. By comparing the environmental state before and after execution, the actual combat benefits brought by the target's tactical actions can be quantified, such as how many enemy units were eliminated, how much friendly casualties were incurred, how much the mission progressed, and whether terrain advantages were fully utilized. Next, the process of determining the environmental state change data in step S104 will be described in conjunction with the accompanying drawings of a specific process embodiment.

[0046] See Figure 4 The figure is a flowchart illustrating a method for determining environmental state change data according to an embodiment of this application, specifically including the following steps: S1041: Perform tactical planning based on the target tactical actions, and generate tactical combat instructions for each of the multiple wargaming combat units.

[0047] After determining the target tactical maneuver, the tactical planning phase begins. Since the target tactical maneuver is an abstract concept with high-level military semantics, it cannot be directly executed by a single combat unit. Therefore, based on the current battlefield situation, the type and status of each combat unit, and the standard operating procedures in the pre-set tactical doctrine library, the abstract maneuver needs to be decomposed into a series of tactical operational instructions oriented towards specific combat units. Taking a flanking maneuver as an example, this process requires identifying the main force units (such as tank companies and mechanized infantry companies) performing the flanking maneuver, planning specific maneuver routes for them, and generating instructions to maneuver to the target coordinates. Simultaneously, units requiring fire support (such as self-propelled artillery companies) need to be identified, and instructions are issued to suppress the enemy's frontal defenses to divert their attention. Through this planning process, an abstract tactical maneuver is transformed into a set of specific instructions that can be executed in parallel or sequentially, enabling various combat units to coordinate and execute the target tactical maneuver.

[0048] S1042: Send the corresponding tactical combat instructions to each of the wargaming combat units, and obtain the execution results of the combat instructions from each of the wargaming combat units.

[0049] After generating tactical combat instructions, the instructions generated in the previous step are sent to the corresponding combat unit models asynchronously or synchronously through the instruction distribution interface built into the wargaming simulation environment. Each combat unit has an independent decision-making and behavioral model in the simulation environment, capable of receiving instructions, parsing instruction parameters, and executing corresponding actions based on its own state and battlefield environment rules (such as the impact of terrain on maneuver speed and the probability of fire strikes). During execution, each combat unit will produce execution result feedback in real time or in stages. This feedback includes: whether the instruction was successfully executed, the actual start and end time of execution, unexpected events encountered during execution, changes in its own state, and, if the instruction involves an attack, a report on the enemy damage caused. The execution results of all combat units are collected to form the original execution result dataset.

[0050] S1043: Based on the execution results of the combat instructions fed back by each wargaming combat unit, perform environmental state analysis to obtain the environmental state change data.

[0051] After obtaining the command execution results of all relevant combat units, the environmental state analysis phase begins. The core task of this phase is to aggregate the dispersed execution results and compare them with the pre-execution battlefield baseline state, thereby calculating the quantitative changes in the entire wargaming simulation environment before and after the execution of the target tactical action. Specifically, key indicators are first extracted from the execution results: statistics on the losses of all Red Force combat units, the losses of all Blue Force combat units, changes in the positions of each unit, the total consumption of ammunition and fuel, changes in mission objective progress, and the simulation time consumed in executing this tactical action. Subsequently, these indicators are compared item by item with the situation snapshot before the action, generating structured environmental state change data. In one possible implementation, auxiliary information can also be recorded, such as whether favorable terrain was utilized, or whether specific tactical events were triggered. This environmental state change data directly reflects the actual combat effects of the target tactical action selected by the agent in the real simulation environment.

[0052] The above is an introduction to the method for determining environmental state change data in step S104. Next, we will continue to combine... Figure 1 The following steps S105 will be described in detail.

[0053] S105: Perform reward decomposition based on the environmental state change data to obtain the reward decomposition result for the target tactical action; the reward decomposition result is used to characterize the evaluation result of the target tactical action under different combat evaluation dimensions.

[0054] In step S105, the system uses the environmental state change data obtained in step S104 as input to perform a reward decomposition operation, thereby obtaining the reward decomposition result for the target tactical action. Reward decomposition refers to breaking down the single scalar total reward value in traditional reinforcement learning into several sub-reward components with clear military meanings according to the multidimensional semantics of combat evaluation. In wargaming scenarios, the effects of a single tactical action are often complex; it may simultaneously eliminate enemy units, advance the mission objective, but also cause friendly losses, consume time, and utilize or not utilize terrain advantages. Traditional single reward values ​​cannot accurately represent the multiple impacts of the target tactical action on the battlefield after its execution. Commanders can only see an abstract number, but have no way of knowing how much of it comes from eliminating the enemy, how much from mission advancement, and how much is offset by friendly losses. The reward decomposition mechanism in this embodiment is designed to solve this information black box problem. Based on various objective indicators in the environmental state change data, including enemy casualties, friendly casualties, mission progress changes, action execution time, and terrain utilization indicators, multiple sub-reward components are generated according to preset contribution calculation methods for different combat evaluation dimensions. Each sub-reward component corresponds to an independent evaluation dimension, such as contribution to enemy elimination, mission progress contribution, friendly survival cost, time efficiency cost, and terrain utilization gain. These sub-reward components can be positive or negative, and their algebraic sum is the total reward value in the traditional sense. Through this decomposition, the comprehensive effect of a decision is broken down into multiple independently examineable semantic units, providing a dimensional quantitative basis for subsequent interpretable presentation.

[0055] Specifically, the reward decomposition process in step S105 is implemented through the following three steps: Step 1: Based on the tactical action type corresponding to the target tactical action, determine the enemy casualty reward coefficient, mission objective reward coefficient, friendly casualty penalty coefficient, and terrain reward coefficient corresponding to the tactical action type from the preset reward coefficient mapping table.

[0056] Different tactical maneuvers differ fundamentally in their operational objectives, risk preferences, and expected effects; therefore, a uniform reward coefficient cannot be used to evaluate all maneuvers. For example, flanking maneuvers emphasize mobility and surprise, and their success often depends on the utilization of terrain and flank protection, thus requiring a higher terrain reward coefficient. Conversely, frontal assaults rely more on troop superiority and firepower density, with a higher expected direct lethality to the enemy, thus requiring a higher enemy casualty reward coefficient. To this end, this embodiment pre-constructs a reward coefficient mapping table. This table uses tactical maneuver type as an index and assigns four types of differentiated coefficients to each maneuver: enemy casualty reward coefficient, mission objective reward coefficient, and terrain reward coefficient. During reward decomposition, the type identifier of the target tactical maneuver determined in step S103 (such as flanking maneuver or coordinated attack) is first obtained. Then, using this identifier as a key, the four types of coefficient values ​​corresponding to the tactical maneuver type are quickly retrieved from the pre-defined reward coefficient mapping table. Taking flanking maneuvers as an example, the mapping configuration might be: enemy casualty reward coefficient 0.10, mission objective reward coefficient 0.006, friendly casualty penalty coefficient 0.12, and terrain reward coefficient 0.03. This differentiated configuration allows the reward decomposition results to accurately reflect the intrinsic value of different tactical actions, thereby improving the rationality and interpretability of the assessment.

[0057] Step 2: Calculate the enemy elimination reward value based on the enemy casualty count and the enemy casualty elimination reward coefficient; calculate the mission objective reward value based on the mission progress change and the mission objective reward coefficient; calculate the friendly casualty penalty value based on the friendly casualty count and the friendly casualty penalty coefficient; calculate the execution time penalty value based on the fixed time penalty constant and the action execution time; and calculate the terrain utilization reward value based on the terrain utilization identifier and the terrain reward coefficient.

[0058] After completing the coefficient retrieval, the five sub-reward components are calculated using the specific indicators in the environmental state change data obtained in step S104. The specific calculation methods are as follows: First, the enemy elimination reward value = enemy casualties × enemy casualty reward coefficient. The enemy casualties are derived from the number of blue team combat units reduced in the environmental state change data. Multiplying this by the reward coefficient corresponding to the action yields a positive contribution value for eliminating the enemy. Second, the mission objective reward value = mission progress change × mission objective reward coefficient. The mission progress change reflects the increase or decrease in the percentage of mission completion after executing the tactical action. Multiplying this by the coefficient yields a positive benefit in mission advancement. Third, the friendly loss penalty value = -(friendly casualties × friendly loss penalty coefficient). The friendly casualties are the number of red team combat units lost. Since loss is a negative cost, the calculation result is negative, forming a penalty item. Fourth, the execution time penalty value = -(fixed time penalty constant × action execution time). The fixed-time penalty constant is a preset positive number (e.g., 0.01). The action execution time is measured in simulation steps or seconds. The product of the two is negative, used to incentivize the agent to complete tactical actions within a reasonable time and avoid unnecessary delays. Fifth, the terrain utilization reward value = terrain utilization flag × terrain reward coefficient. The terrain utilization flag is a Boolean value or a 0 / 1 variable. It is set to 1 when the combat unit actually enters favorable terrain or utilizes terrain concealment during the action, and 0 otherwise. Multiplying it by the terrain reward coefficient yields a positive terrain utilization gain. These five sub-reward values ​​characterize the comprehensive effect of the tactical action from five dimensions: firepower strike, mission advancement, survivability protection, time efficiency, and terrain utilization. Each sub-reward value has a clear physical meaning.

[0059] Step 3: Determine the reward decomposition result as the reward value for eliminating the enemy, the reward value for the mission objective, the penalty value for friendly losses, the penalty value for execution time, and the reward value for terrain utilization.

[0060] Finally, the reward values ​​calculated in step two—enemy elimination reward, mission objective reward, friendly loss penalty, execution time penalty, and terrain utilization reward—are collectively determined as the reward decomposition result for the tactical action targeting that objective. This result exists in the form of structured data, typically containing a total reward field and five independent sub-reward fields. In the visualization interface for commanders, the reward decomposition result is usually presented in the form of a grouped list or dashboard. This result allows commanders to clearly identify that the decision performed well in terms of strike effect (eliminating 2 enemy units), but also incurred the loss of 1 friendly unit, while terrain utilization brought some benefits. Through reward decomposition, the multi-dimensional evaluation information originally hidden behind a single scalar reward is explicitly presented, allowing commanders to accurately understand the details of the gains and losses of each decision step without reverse inference.

[0061] In one possible implementation, after obtaining the reward decomposition result for the target tactical action, the reward decomposition result can also be used as the basis for updating the network parameters of the DQN-LSTM hybrid neural network. This is specifically achieved through the following three steps: Step 1: Based on the reward analysis results and the target tactical actions, construct state transition experience.

[0062] After completing a full decision-making cycle, the following information is accumulated: the situational feature vector before executing the target tactical action, the index number of the selected target tactical action, the reward decomposition result, and the situational feature vector at the next moment obtained by updating the environmental state change data after executing the action. This information is combined into a structured four-tuple, called a state transition experience. It is important to note that the reward value used in this step is the total reward re-aggregated after reward decomposition. Because the reward decomposition process has semantically quantified the combat effects across multiple dimensions such as enemy elimination, mission progress, friendly losses, time consumption, and terrain utilization, the aggregated total reward has higher interpretability and rationality, and can more accurately reflect the true quality of the agent's decision-making. The process of constructing state transition experiences essentially encapsulates the input, output, and feedback of a tactical decision into a sample that can be directly consumed by the training algorithm, providing a data foundation for the experience replay mechanism.

[0063] Step 2: Store the state transition experience in the experience replay buffer, and sample a preset batch of data from the experience replay buffer.

[0064] A first-in, first-out (FIFO) experience replay buffer with a fixed preset capacity is maintained. Each completed state transition experience is pushed to the end of the buffer; when the buffer is full, the oldest sample is automatically discarded to ensure that the buffer always stores decision-making experience from the most recent period. This experience replay mechanism aims to break the temporal correlation of data between consecutive decision-making cycles. In wargame simulations, there is a high temporal dependency between states in adjacent decision-making cycles (e.g., the situational feature vectors of cycle t and cycle t+1 often do not change much). If these experiences are used directly in chronological order for learning, it will lead to large variance in neural network parameter updates, and may even cause the system to get stuck in local optima. By randomly sampling from the buffer, experiences from different periods and scenarios can be mixed, ensuring that the training samples satisfy the assumption of independent and identically distributed (IOD), thereby improving training stability and convergence speed. Specifically, at each model update time, a sample set of a preset batch size is randomly drawn from the experience replay buffer, for example, a batch size of 64. This batch of data contains multiple state transition experiences, each from different decision-making cycles or even different inference rounds, exhibiting diverse situational characteristics and action choices, and can provide balanced learning signals for network updates.

[0065] Step 3: Based on the preset batch data, calculate the temporal difference error using the temporal difference learning method, and update the network parameters of the DQN-LSTM hybrid neural network according to the temporal difference error to obtain the updated network parameters.

[0066] In step three, based on batch data randomly sampled from the experience replay buffer, the parameters of the DQN-LSTM hybrid neural network are updated using a temporal difference learning method. Specifically, for each state transition experience in the batch, two networks are used simultaneously for computation: one is an online network currently being trained, and the other is a target network with the same structure as the online network but whose parameter updates are more lagging. The online network is responsible for outputting the estimated reward value for performing the selected tactical action in the current state based on the experience. The target network is responsible for calculating the maximum estimated reward value among all possible tactical actions in the next state based on the experience. Subsequently, the system adds the actual immediate reward obtained after performing the action to the maximum estimated reward value of the next state after scaling by a discount factor to obtain a corrected target reward value. This target reward value is considered as a more accurate value estimate that the current decision should achieve.

[0067] The learning error is defined as the difference between the estimated reward value output by the online network and the target reward value. If the estimated reward value is lower than the target reward value, it means the online network underestimated the value of the action and needs to be adjusted to increase it; conversely, if the estimated reward value is higher than the target reward value, it means it overestimated it and needs to be appropriately reduced. The system uses a backpropagation algorithm to propagate this error signal layer by layer to each connection weight in the network, and uses an optimizer to fine-tune the weights with a preset learning rate, enabling the online network to output a more accurate action value assessment when facing similar situations in the future. After several such batch updates, the parameters of the online network are periodically copied to the target network to maintain the relative stability of the target network and avoid drastic oscillations in the learning process. Through this mechanism, the actual feedback of each decision, including the total reward aggregated from multiple dimensions such as eliminating the enemy, advancing the mission, suffering losses, consuming time, and utilizing terrain, is transformed into a learning signal that drives the network evolution. This allows the agent to continuously improve its ability to judge the value of tactical actions through continuous interaction and gradually converge to a better decision strategy. The updated network parameters will be directly used for Q-value evaluation in the next decision-making cycle, forming a complete closed-loop learning process.

[0068] Regarding step S1031 of the aforementioned embodiment, after obtaining the estimated action benefit value of each candidate tactical action, action value display information for each candidate tactical action can be calculated based on a pre-set action advantage function and state value function to present the value of each candidate tactical action to the commander. This is specifically achieved through the following three steps: Step 1: Based on the action advantage function, calculate the action advantage degree of each candidate tactical action according to the estimated action benefit value of each candidate tactical action; The action advantage function is defined as the difference between the action value function and the state value function. Its physical meaning is the expected cumulative reward that can be obtained by performing a specific action a given current situation s. To calculate the action advantage, the baseline state value V(s) of the current state is first obtained (the specific calculation of this value will be explained in detail in step two). After obtaining V(s), for each candidate tactical action a, the system subtracts the baseline state value V(s) from its estimated action reward value Q(s,a) to obtain the action advantage A(s,a). The action advantage can be positive or negative: a positive value indicates that the expected reward of the action is higher than the average level of the current state, and the larger the value, the more significant the advantage; a negative value indicates that the expected reward of the action is lower than the average level, and performing the action may lead to results inferior to conventional strategies. For example, if the baseline value V(s) of the current situation is 0.500, and the Q value of "frontal assault" is 0.687, then its action advantage is +0.187, meaning that this action has a relative advantage of 37.4% higher than the average level. Conversely, the Q value of "tactical retreat" is 0.253, so its action advantage is -0.247, indicating that this action is significantly inferior to the average level. By calculating the action advantage, the originally abstract Q value is transformed into a relative advantage index with clear comparative significance, enabling commanders to intuitively identify which tactical actions have excess benefits in the current situation and which actions should be avoided as much as possible.

[0069] Step 2: Based on the state value function, calculate the state baseline value for all the candidate tactical actions according to the estimated action benefit value of each candidate tactical action.

[0070] Step two is executed simultaneously with step one. The state value function V(s) is used to characterize the baseline level of the expected cumulative reward that the agent can obtain by choosing an action according to the current strategy under the current situation s. In the technical solution of this invention, the state value function V(s) is calculated based on the estimated action reward values ​​of all candidate tactical actions. The specific calculation method is as follows: take the maximum value among all candidate tactical actions' Q values ​​as an approximate estimate of the state baseline value. This method of taking the maximum value reflects the theoretically best expected reward that can be achieved under the current situation. Taking eight tactical actions as an example, if their Q values ​​are: frontal assault 0.687, flanking maneuver 0.667, fire suppression 0.577, defensive fortification 0.454, tactical retreat 0.253, reconnaissance and infiltration 0.535, artillery barrage 0.577, and coordinated attack 0.667, then the state baseline value V(s) = 0.687. This value conveys key information to the commander: under the current battlefield situation, even if the agent takes the optimal action, the upper limit of its expected cumulative reward is approximately 0.687. The baseline value of the situation provides a necessary reference point for calculating the advantage of action in step one, and also enables commanders to assess the overall value level of the current situation. If V(s) is high, it indicates that the current situation is favorable to the commander and there is a large decision-making space; if V(s) is low or even negative, it indicates that the situation is unfavorable and the expected benefits of any action are limited.

[0071] Step 3: Generate action value display information for each candidate tactical action based on the action advantage degree, the estimated action benefit value, and the state baseline value for all candidate tactical actions.

[0072] After completing the calculations in the two steps above, the estimated action benefit value, action advantage degree, and global state baseline value of each candidate tactical action are structurally integrated to generate action value display information for commanders. This display information is usually presented in the form of a table or matrix in the visualization interface. Specifically, each row corresponds to a tactical action and contains three columns of core values: the first column displays Q(s,a), presented as a signed floating-point number, the value of which reflects the expected absolute benefit; the second column displays A(s,a), also presented as a signed floating-point number, and can be color-coded (green for positive values, red for negative values) to enhance the distinction between positive and negative values; the third column can be supplemented with a bar chart or column chart of Q values ​​to achieve a visual comparison of the values. At the same time, the current value of the state baseline value V(s) is displayed separately at the bottom or side of the table as a comparison benchmark for all actions. In addition, the system can automatically highlight the action with the highest advantage degree (i.e., the best action) according to the ranking of action advantage degree, and use different background colors to distinguish the action areas with positive and negative advantage degrees. In this way, by displaying multi-dimensional action value information, commanders can not only see the absolute expected benefits of each action, but also clearly understand the gap between each action and the optimal action, as well as the degree of superiority or inferiority of each action relative to the average level. This allows for a deeper understanding of the decision-making logic of the agent and effectively improves the interpretability of combat decisions.

[0073] On the other hand, in one possible implementation, when the method provided in the embodiments of this application is applied to an agent, the exploration rate of the agent can also be exponentially decayed based on the ε-greedy strategy control exploration algorithm.

[0074] Specifically, in the training process of reinforcement learning, the agent needs to strike a balance between exploration and exploitation. This embodiment employs an ε-greedy strategy to control this balance, with the core mechanism as follows: In each decision cycle, the agent randomly selects an action from a preset set of tactical actions with a certain exploration probability, i.e., performs an exploration behavior; while with the remaining probability, it selects the target tactical action with the highest current value assessment, i.e., performs an exploitation behavior. The value of the exploration probability directly determines the agent's enthusiasm for exploring new tactics. A higher exploration probability encourages the agent to try different tactical actions widely, helping to avoid getting trapped in local optima; while a lower exploration probability makes the agent tend to repeat verified effective tactics, improving the stability of decision-making.

[0075] To fully explore the tactical space in the early stages of training and gradually converge to a high-quality strategy in the later stages, this invention dynamically adjusts the exploration probability using an exponential decay method. Specifically, after each decision cycle, the exploration probability undergoes a multiplicative decay according to a preset decay coefficient, causing the exploration probability to gradually decrease at an exponential rate. This process begins with an initially set exploration probability, gradually decreasing as training rounds progress until it reaches a preset minimum threshold, after which it stops decaying. This exponential decay method achieves a smooth and gradual transition between exploration and utilization. The higher exploration probability in the early stages of training allows the agent to widely engage with diverse tactical actions such as frontal assaults, flanking maneuvers, and reconnaissance infiltration, accumulating rich experience samples. As training deepens, the exploration probability decreases exponentially, and the agent gradually focuses on validated high-yield tactical actions, thereby ensuring the stability and combat effectiveness of the final strategy. Through this mechanism, the agent can achieve adaptive learning in a dynamic wargaming environment, avoiding the possibility of missing better tactics due to premature utilization and preventing the instability of decision quality caused by over-exploration, thus improving the reliability of decision-making.

[0076] This application provides a reinforcement learning-based method for interpreting operational decisions in wargaming. First, the original situational data from the wargaming exercise is visualized to generate visualized situational information. Then, structured feature analysis is performed on the visualized situational information to extract situational feature vectors containing multiple operational elements. Further, a pre-constructed DQN-LSTM hybrid neural network is used, combined with a preset set of tactical actions and the situational feature vectors, to evaluate the benefits of tactical actions. A visualized decision-making chain is generated simultaneously with the output of the target tactical actions, thus fully demonstrating the reasoning process from feature analysis to action value evaluation, improving the interpretability of single tactical decisions. Finally, by executing the target tactical actions and acquiring environmental state change data, reward decomposition is performed on the changed data to obtain the reward decomposition result for the target tactical action, quantifying the decision-making effect from multiple operational evaluation dimensions. These steps are sequentially linked, forming a complete interpretation mechanism from situational awareness, feature analysis, decision reasoning to effect evaluation, effectively improving the interpretability of operational decisions.

[0077] The following describes a reinforcement learning-based war game decision interpretation system provided by an embodiment of this application. The reinforcement learning-based war game decision interpretation system described below can be referred to in correspondence with the reinforcement learning-based war game decision interpretation method described above.

[0078] See Figure 5 The figure is a schematic diagram of the structure of a reinforcement learning combat decision-making system for wargaming provided in an embodiment of this application, specifically including the following modules: The first visualization module 100 is used to perform visualization processing on the original situation data in the war game simulation environment to obtain visualized situation information; the war game simulation environment includes multiple types of war game combat units. The feature analysis module 200 is used to perform battlefield situation feature analysis based on the visualized situation information to obtain a situation feature vector; The second visualization module 300 is used to evaluate the benefits of tactical actions based on a pre-built DQN-LSTM hybrid neural network, according to a preset set of tactical actions and the situation feature vector, to determine the target tactical actions and the visualized decision-making link for the target tactical actions. The action execution module 400 is used to control each of the war game combat units to execute tactical actions through the target tactical actions, and to determine the environmental state change data of the war game simulation environment corresponding to each of the war game combat units after executing the tactical actions. The reward decomposition module 500 is used to decompose the rewards based on the environmental state change data to obtain the reward decomposition results for the target tactical actions; the reward decomposition results are used to characterize the evaluation results of the target tactical actions under different combat evaluation dimensions.

[0079] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a computer-readable storage medium storing computer instructions for causing the computer to execute the reinforcement learning combat decision interpretation method for wargaming as described in any of the above embodiments.

[0080] The computer-readable media in this application embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0081] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the reinforcement learning combat decision interpretation method of wargaming as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0082] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the system, method, and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The system, method, electronic device, and medium embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment solution according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0083] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for explaining combat decision of reinforcement learning in war game, characterized in that, The method includes: Visualized situation information is obtained by visualizing the original situation data in the wargaming simulation environment; the wargaming simulation environment includes multiple types of wargaming combat units. Based on the visualized situational information, battlefield situational characteristics are analyzed to obtain a situational feature vector. Based on a pre-built DQN-LSTM hybrid neural network, the benefits of tactical actions are evaluated according to a preset set of tactical actions and the situation feature vector, and the target tactical actions and the visualized decision-making chain for the target tactical actions are determined. The target tactical actions control each of the wargaming combat units to execute tactical actions, and determine the environmental state change data of the wargaming simulation environment corresponding to each wargaming combat unit after executing the tactical actions; The reward decomposition is performed based on the environmental state change data to obtain the reward decomposition result for the target tactical action; the reward decomposition result is used to characterize the evaluation result of the target tactical action under different combat evaluation dimensions.

2. The method of claim 1, wherein, The preset set of tactical actions includes multiple candidate tactical actions; The pre-built DQN-LSTM hybrid neural network evaluates the benefits of tactical actions based on a preset set of tactical actions and the situation feature vector, determines the target tactical actions, and establishes a visualized decision-making chain for the target tactical actions, including: Based on the DQN-LSTM hybrid neural network, the estimated action benefit value corresponding to each candidate tactical action is calculated according to the situation feature vector. Based on the flexible maximum value strategy function, the action selection probability is transformed according to the estimated action benefit value corresponding to each candidate tactical action to obtain a candidate action probability set; the candidate action probability set includes the action selection probability corresponding to each candidate tactical action individually. The candidate tactical action with the highest action selection probability in the candidate action probability set is determined as the target tactical action; The visual decision link is obtained by semantically transforming the situation features corresponding to the target tactical actions within the situation feature vector.

3. The method of claim 2, wherein, After calculating the estimated action benefit value corresponding to each candidate tactical action based on the situation feature vector, the method further includes: Based on the action advantage function, the action advantage degree of each candidate tactical action is calculated according to the estimated action benefit value of each candidate tactical action; and, Based on the state value function, the state baseline value for all the candidate tactical actions is calculated according to the estimated action benefit value of each candidate tactical action. Based on the action advantage degree of each candidate tactical action, the estimated action benefit value, and the state baseline value for all candidate tactical actions, action value display information is generated for each candidate tactical action.

4. The method of claim 1, wherein, By controlling the target tactical actions to execute tactical actions of each wargaming combat unit, the environmental state change data corresponding to the wargaming simulation environment after each wargaming combat unit executes the tactical actions are determined, including: Based on the target tactical actions, tactical planning is performed to generate tactical combat instructions for each of the wargame combat units; Send the corresponding tactical combat instructions to each of the wargaming combat units, and obtain the execution results of the combat instructions from each of the wargaming combat units; Environmental state analysis is performed based on the execution results of combat instructions fed back by each wargaming combat unit to obtain environmental state change data.

5. The method of claim 1, wherein, The environmental status change data includes: enemy casualties, friendly casualties, mission progress changes, terrain utilization indicators, and action execution time. The step of decomposing the reward based on the preset reward coefficient corresponding to the target tactical action and the environmental state change data to obtain the reward decomposition result for the target tactical action includes: Based on the type of tactical action corresponding to the target tactical action, the enemy's battle loss reward coefficient, mission objective reward coefficient, friendly loss penalty coefficient, and terrain reward coefficient corresponding to the type of tactical action are determined from a preset reward coefficient mapping table. The following calculations are performed: Calculate the enemy elimination reward value based on the enemy casualty count and the enemy casualty elimination reward coefficient; calculate the mission objective reward value based on the mission progress change and the mission objective reward coefficient; calculate the friendly casualty penalty value based on the friendly casualty count and the friendly casualty penalty coefficient; calculate the execution time penalty value based on the fixed time penalty constant and the action execution time; and calculate the terrain utilization reward value based on the terrain utilization identifier and the terrain reward coefficient. The reward decomposition result is determined by the reward value for eliminating the enemy, the reward value for the mission objective, the penalty value for friendly losses, the penalty value for execution time, and the reward value for terrain utilization.

6. The method according to claim 1, characterized in that, After obtaining the reward decomposition result for the target tactical action, the method further includes: Based on the reward analysis results and the target tactical actions, a state transition experience is constructed; The state transition experience is stored in the experience replay buffer, and a preset batch of data is sampled from the experience replay buffer; Based on the preset batch data, the temporal difference error is calculated according to the temporal difference learning method, and the network parameters of the DQN-LSTM hybrid neural network are updated according to the temporal difference error to obtain the updated network parameters.

7. The method according to claim 1, characterized in that, The situational feature vector includes: troop situational features, terrain and environment features, combat effectiveness features, and tactical situational features.

8. The method according to claim 1, characterized in that, The method is applied to an intelligent agent, and after obtaining the reward decomposition result for the target tactical action, the method further includes: The exploration rate of the agent is exponentially decayed based on the ε-greedy strategy control exploration algorithm.

9. A reinforcement learning-based operational decision interpretation system for wargaming, characterized in that, The system includes: The first visualization module is used to perform visualization processing on the original situation data in the wargaming simulation environment to obtain visualized situation information; the wargaming simulation environment includes multiple types of wargaming combat units; The feature analysis module is used to perform battlefield situation feature analysis based on the visualized situation information to obtain a situation feature vector; The second visualization module is used to evaluate the benefits of tactical actions based on a pre-built DQN-LSTM hybrid neural network, according to a preset set of tactical actions and the situation feature vector, to determine the target tactical actions and the visualized decision-making chain for the target tactical actions. The action execution module is used to control each of the wargaming combat units to execute tactical actions through the target tactical actions, and to determine the environmental state change data of the wargaming simulation environment corresponding to each wargaming combat unit after executing the tactical actions. The reward decomposition module is used to decompose rewards based on the environmental state change data to obtain reward decomposition results for the target tactical actions; the reward decomposition results are used to characterize the evaluation results of the target tactical actions under different combat evaluation dimensions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the reinforcement learning-based wargaming decision interpretation method according to any one of claims 1-8.

Images