A Wargaming Agent-Assisted Decision-Making Method Based on Situational Awareness Interaction
The situational awareness-based interactive wargaming agent-assisted decision-making method solves the problem of high-dimensional decision-making in wargaming simulations, improves decision-making efficiency and accuracy, and supports high-quality decision-making by agents in complex battlefield environments.
Patent Information
- Application Number
- CN202411012209.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-07-26
AI Technical Summary
In wargaming simulations and adversarial exercises, the complex and ever-changing battlefield situation and the large number of combat entities result in a high-dimensional state-action space for command and decision-making processes. Combat missions are difficult to translate into reasonable reward functions, and timely and accurate feedback cannot be provided. This increases the difficulty of training strategy models and makes it difficult to make high-quality decisions.
By using a wargaming-based intelligent agent-assisted decision-making method based on situational awareness interaction, battlefield situation data is acquired, key battlefield situation parameters are filtered, a decision-making action module is constructed, the probability of decision-making actions is output, key battlefield data is statistically analyzed, weighted integration is performed, the optimal decision-making action is generated, and feedback and filtering are performed in conjunction with the upper-level intelligent agent.
It improves the efficiency and accuracy of agent decision-making, provides system administrators with a basis for deduction and review, ensures that decisions conform to real battlefield scenarios, and supports the correction of agent training parameters.
Smart Images

Figure CN118966356B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the research field of key technologies for wargaming simulation systems, specifically involving a wargaming agent-assisted decision-making method based on situational awareness interaction. Background Technology
[0002] With the ever-accelerating pace of modern warfare, future joint operations are becoming increasingly complex and diverse. Intelligent combined arms battalion formations are crucial for gaining battlefield initiative, achieving speed over slowness, rapidly and flexibly focusing forces, and accurately and efficiently striking enemy targets, playing a vital role in future joint operations.
[0003] However, in wargaming simulations and adversarial exercises, the complex and ever-changing battlefield situation and numerous combat entities lead to a high-dimensional state-action space in the command and decision-making process. Specifically, combat objectives are difficult to directly translate into reasonable reward functions, failing to provide immediate and accurate feedback. This results in problems such as sparse, delayed, and inaccurate feedback in combat actions. Furthermore, the highly dynamic battlefield environment and continuously changing game tasks increase the difficulty of training strategy models, making it a significant challenge to make high-quality decisions under different game tasks.
[0004] Multi-agent technology refers to a technique that utilizes the interaction and collaboration among multiple intelligent agents to accomplish tasks. An intelligent agent is a machine or software entity with autonomous decision-making capabilities. Combining multiple intelligent agents forms an intelligent system that can collaborate and communicate like humans. By integrating multi-agent technology with combined arms battalion formations, each branch of the armed forces can be viewed as an intelligent agent. They can autonomously reason and plan to select appropriate strategies, simplifying the command and decision-making process and thus better completing tasks.
[0005] This invention proposes a multi-agent-assisted decision-making method, combined with existing wargaming systems, aiming to provide decision support for agents through intelligent algorithms and data analysis. This method can interact with real-time battlefield situation data to assist agents in decision analysis, scenario assessment, and tactical planning, thereby improving the efficiency and accuracy of agent decision-making. Summary of the Invention
[0006] The purpose of this invention is to provide a multi-agent auxiliary decision-making method based on wargaming, which provides necessary auxiliary information support for agents when making decisions, enabling them to make decisions based entirely on current battlefield situation data that conform to real battlefield scenarios. It also provides system administrators with a basis for post-war game analysis, facilitating timely correction of agent training parameters and ensuring the most realistic wargaming simulation results.
[0007] The technical solution adopted in this invention is a wargaming agent-assisted decision-making method based on situational awareness interaction, which is implemented according to the following steps:
[0008] Step 1: Acquire situational awareness data of the battlefield environment in which the wargaming agent is located;
[0009] Step 2: Use the acquired situational awareness data as input to the action decision module of the wargaming agent. Analyze the impact of military factors on military operations and select the main battlefield situational data parameters that affect the agent's decision-making actions in the current specific scenario.
[0010] Step 3: Construct a decision-making action module. The main battlefield situation data parameters obtained from Step 2 are used as inputs to the decision-making action module, and the probability of all possible decision actions is output.
[0011] Step 4: Collect key battlefield data for the current combat scenario. Based on the mission requirements of the current round, collect key battlefield data for each wargaming agent in the scenario, such as survival index, deterrence index, artillery suppression index, combat effectiveness, etc., to provide comprehensive information support.
[0012] Step 5: Integrate the action probabilities output in Step 3 with the key battlefield data collected in Step 4, and classify them according to the agent ID.
[0013] Step 6: Input the weighted integrated data from Step 5 as decision support information into the upper-level agent for evaluation and selection of decision actions, provide feedback to the lower-level agent, and generate the optimal decision action.
[0014] The invention is further characterized in that,
[0015] Understanding and perceiving the battlefield situation mainly includes environmental data such as combat maps and simulation data;
[0016] Environmental data such as combat maps follow specific examples of military rules, such as structured representations of terrain.
[0017] The simulation data mainly includes constraint rule data, self-game data, and replay data, which provide a learning and training foundation for the wargaming agent.
[0018] The agent will perceive the current battlefield environment state based on the state space. Its data comes from a wargaming system and is acquired through interaction with the system. In a particular combat simulation, the agent will perceive the following at the start of the round:
[0019] (1) Battlefield map information: hexagonal grid terrain information, grid interior features information, grid inter-grid features information, elevation information, etc.;
[0020] (2) Wargame information: Wargame category information, whether the wargame is dead, the wargame's position on the hexagonal grid, morale and fatigue value, formation and orientation, number of ammunition carried, etc.
[0021] (3) Operational scenario rule information: weapon system rule information, reconnaissance system rule information, mobility system rule information, etc.;
[0022] In step 1, the situational awareness data of the battlefield environment in which the wargame agent is located is obtained by simulating the environment around the agent based on the geographical information of the hexagonal chessboard and collecting battlefield situational information, including enemy situation, friendly situation and various battlefield environment data.
[0023] Furthermore, the specific implementation steps of step 1 are as follows:
[0024] Step 1.1: Based on the characteristics of the battlefield situation data in wargaming simulation and the functional requirements of the wargaming agent, collect a large amount of data that meets the distribution characteristics of the battlefield environment on the basis of small sample simulation data;
[0025] The data that meets the battlefield environment distribution characteristics under the wargaming platform mainly includes: hexagonal grid location information, grid-to-grid feature information, elevation information, wargaming type information, and ammunition carrying information, etc., which are battlefield situation environment data.
[0026] Step 1.2: Classify the battlefield situation and environment data collected in Step 1.1 through data mining, and integrate the classification results by combining the relationships between battlefield situation elements.
[0027] After collecting battlefield situational data, the purpose of data mining is to perform classification and clustering algorithms on the raw data, and combine it with the battlefield situational element map to obtain a discontinuous set of data elements. Then, by combining the relationships between battlefield situational elements, the data element set is further integrated and processed to obtain coherent situational information, thereby achieving accurate battlefield perception.
[0028] Furthermore, step 1.2 is specifically implemented according to the following steps:
[0029] Step 1.2.1: Construct a military application model library for the logical layer and a data mining algorithm library for the algorithm layer;
[0030] Step 1.2.2: Through the combined application of the logic layer and the algorithm layer, the battlefield situation environment feature data is processed by data mining to form various categories of battlefield situation element data;
[0031] Step 1.2.3: Integrate the data mining results by combining the relationships between battlefield situational elements to form coherent situational awareness data and complete accurate battlefield awareness.
[0032] In step 2, the situational awareness data obtained in step 1 is used as the input to the action decision module of the wargaming agent. Based on the influence of military factors on military operations, the main battlefield situational data parameters that affect the agent's decision-making actions in the current specific scenario are analyzed and filtered.
[0033] Wargaming systems improve the accuracy of battlefield situation data by simulating real combat environments, but the relationships between data require commanders to analyze, filter, and accurately assess the battlefield situation using military operational theory. These relationships are generated through the correlation between data.
[0034] Furthermore, the specific implementation steps of step 2 are as follows:
[0035] Step 2.1: Using the battlefield situational awareness data from Step 1 as input, perform preliminary screening, cleaning, and other data preprocessing operations.
[0036] Step 2.2: Construct a labeled dataset based on a series of formatted samples with positive and negative annotations obtained from actual adversarial scenarios and custom rules;
[0037] Step 2.3: Based on the tag dataset constructed in Step 2.2, filter the basic situational characteristic data at the combat location;
[0038] Step 2.4: Encode the basic situation features from Step 2.3 and perform diverse feature fusion and filtering. Associate them from multiple dimensions to form a structured situation feature dataset, which is then stored as the main battlefield situation data parameter.
[0039] In step 3, the main battlefield situation data parameters obtained from step 2 are used as input to the decision-making action module, and the probability of all possible decision actions of the wargaming agent is output.
[0040] Furthermore, the specific implementation steps of step 3 are as follows:
[0041] Step 3.1: Construct the action space;
[0042] The action space includes upper-level macro-level tasks based on military prior knowledge and lower-level agent behaviors based on deep reinforcement learning. Both upper and lower-level actions are discrete in form.
[0043] Furthermore, step 3.1 is specifically implemented according to the following steps:
[0044] Step 3.1.1: Construct the action space based on the hexagonal grid target;
[0045] In such tasks, the multi-agent squad first selects a subset of hexagonal grids from a defined set of available hexagonal grid targets, and then performs tasks related to those hexagonal grids. These tasks typically involve maneuvering to the corresponding hexagonal grid and then performing discrete operations such as stopping, maneuvering upwards, maneuvering downwards, seizing control, or concealing oneself.
[0046] Step 3.1.2: Construct the action space based on the wargame objective.
[0047] In this type of mission, the multi-agent squad first selects a subset from all surviving enemy wargame targets, and then performs tasks related to that subset of wargames. These tasks typically involve moving to a square at a suitable distance from the piece and then performing discrete operations such as stopping, firing, or concealing. The purpose of stopping or concealing is similar to surveillance.
[0048] Step 3.2: Using the main battlefield situation data from Step 2 as input, and combining it with the action space, the lower-level policy network generates a set of action probabilities.
[0049] The set of action probabilities can be obtained through the policy function. The lower-level agent generates a policy function π based on the observed state s, which is used as input to obtain the probability of taking action 'a' in the current state. The policy function π is a probability density function that describes the likelihood of taking action 'a' in the current state s.
[0050] According to the definition of the policy function, this function can serve as the basis for the command and decision-making output of the lower-level intelligent agent's wargame in the current battlefield state.
[0051] Furthermore, step 3.2 is specifically implemented according to the following steps:
[0052] Step 3.2.1: Use the policy network π(a|s;θ) to approximate the policy function π, and give the executable command decision a under the current battlefield state s;
[0053] Step 3.2.2: Update the network parameters θ using the gradient ascent method in the policy gradient algorithm;
[0054] Step 3.2.3: The battlefield situation perceived by the lower-level agents based on the situation matrix and the prediction results of the target behavior are used as inputs to generate a set of decision action probabilities for each agent through the policy network.
[0055] Step 3.3: Construct the action-value function Q π , used to improve the policy network π(a|s;θ).
[0056] The Action-Value Function is defined as follows:
[0057]
[0058] The above equation describes the action a that an agent chooses when it observes the environmental state s at time t. t Whether the outcome is wise depends on the discounted return U. t Regarding (s) t ,a t ) expectations.
[0059] According to the discount return U t Due to the randomness of the action, the agent cannot immediately observe the value of the action at time t. Therefore, expectation is introduced to estimate the reward of the action value at time t. π Only with s t ,a t related.
[0060] Furthermore, step 3.3 is implemented in the following specific steps:
[0061] Step 3.3.1: Construct a value network q(s,a;w) to approximate the action-value function Q. π The quality of executable command decision 'a' is evaluated, and this evaluation space is used to improve the strategy network.
[0062] Step 3.3.2: Update the network parameters w using gradient descent in the temporal difference (TD) algorithm.
[0063] In step 4, it is necessary to collect key battlefield data for each wargaming agent based on the combat mission requirements of the current round, such as survival index, deterrence index, artillery suppression index, and combat effectiveness, to provide comprehensive information support for agent decision-making.
[0064] Furthermore, the specific implementation steps of step 4 are as follows:
[0065] Step 4.1: Construct a battlefield element entity classification model;
[0066] The battlefield situation element entity classification model combines battlefield situation elements in a supervised manner to help train the entity classification model. After the model is trained, it will extract and classify features from battlefield map information and wargaming information, and output a set of key battlefield entity elements in the combat scenario.
[0067] Furthermore, step 4.1 is implemented in the following specific steps:
[0068] Step 4.1.1: Construct the CNN model architecture;
[0069] The CNN model consists of two convolutional layers, two pooling layers, one fully connected layer, and one softmax regression layer. It uses the Adam algorithm, which is based on a first-order gradient and a stochastic objective function, to optimize the entire neural network.
[0070] Step 4.1.2: Weight initialization;
[0071] To create this model, a large number of weights and biases need to be created. The weights in this model require a small amount of data noise during initialization to break symmetry and avoid zero gradients, thereby improving the accuracy of battlefield element entity classification. The model uses the ReLU activation function and initializes the biases with a small positive number to avoid the problem of neuron node outputs always being zero. The ReLU activation function is defined as:
[0072] f(x) = max(0,x)
[0073] Where x is the input neuron.
[0074] Step 4.1.3: Convolution and pooling.
[0075] The task of the convolutional layer is to select features from different angles of the battlefield element feature data from the previous layer. After obtaining the convolutional features, they are divided into non-overlapping regions, and then the maximum feature of these regions is used to obtain the pooled convolutional features. The first layer consists of a convolution followed by a max-pooling operation. Each 5×5 convolutional kernel yields 32 features, and each output channel has a corresponding bias. Similarly, in the second layer, each 5×5 convolutional kernel yields 64 features.
[0076] Step 4.2: Construct a fully connected layer;
[0077] A fully connected layer with 1024 neurons is constructed to process the entire wargame battlefield environment data. Simultaneously, the vector output from the pooling layer is transformed into a one-dimensional vector, multiplied by the weight matrix, a bias is added, and the ReLU activation function is applied.
[0078] Step 4.3: Construct the output layer;
[0079] To reduce overfitting in battlefield element data classification, Dropout is added before the output layer, which can effectively improve the generalization ability of large neural networks. Finally, a Softmax layer is added. In Softmax regression, it is assumed that the class label y can take k distinct values. For a given test input x, the hypothesis function p(y=j|x:w) is the probability that x belongs to class j:
[0080]
[0081] Where w represents the parameters of the model.
[0082] Step 4.4: Extract and classify features from battlefield map information and wargame information;
[0083] Furthermore, the specific implementation steps of step 4.4 are as follows:
[0084] Step 4.4.1: Input the battlefield map information and wargame information into the model in matrix form. The convolutional layer is used for feature learning of battlefield elements.
[0085] Step 4.4.2: Input the battlefield element features output in Step 4.4.1 into the fully connected layer for classification to obtain key battlefield element entity data.
[0086] Step 4.5: Compile the battlefield element feature classification results output in Step 4.4 into key battlefield data for the current combat scenario.
[0087] In step 5, the action probabilities output in step 3 and the key battlefield data collected in step 4 need to be weighted and integrated. The processed data is classified according to the agent number and serves as auxiliary information for action decision-making, influencing the agent's final decision action.
[0088] Furthermore, the specific implementation steps of step 5 are as follows:
[0089] Step 5.1: Determine the weighting details;
[0090] By using methods such as military expert knowledge base assessment and wargaming historical data analysis, the relative importance of the output action probability in step 3 to key battlefield data (such as survival index, deterrence index, artillery suppression index, combat effectiveness, etc.) in the current scenario is determined and presented in the form of weight ratios.
[0091] Step 5.2: Verify and adjust the weight allocation;
[0092] The allocation of battlefield key data weights in step 5.1 is verified and fine-tuned to ensure that the weight ratio of each battlefield key data meets the needs of the current scenario and can effectively support the action decision-making process under different circumstances.
[0093] Step 5.3: Perform a weighted integration of the action probabilities output in Step 3 and the key battlefield data collected in Step 4;
[0094] Each key battlefield data point is multiplied by its corresponding weight ratio, and all processed key battlefield data points are summed to obtain a comprehensive score for the probability of the action, ensuring that the contribution of different factors to the final decision action is comprehensively considered.
[0095] Step 5.4: Based on the actual combat situation in the current scenario, quantify and rank the comprehensive scores of the probability of each action after the integration of the key battlefield data weights in Step 5.3;
[0096] Step 5.5: Classify the output results of step 5.4 according to the agent number.
[0097] By identifying the unique ID field of each agent, the probabilities of all actions and their comprehensive scores quantified in step 5.4 are classified into the corresponding agent attribute fields, providing personalized decision support for each agent.
[0098] In step 6, the output of step 5 is used as decision-making auxiliary information and input into the upper-level agent for action selection and evaluation. The optimal action probability sequence that conforms to the combat mission in the current scenario is output, and the feedback is given to the lower-level agent to output the optimal decision action to be executed.
[0099] Furthermore, the specific implementation steps of step 6 are as follows:
[0100] Step 6.1: Construct a high-level knowledge reasoning intelligent agent model;
[0101] Knowledge-based reasoning agents are intelligent agent models that form domain knowledge by manually extracting unstructured information such as combat rules, regulations, and the experience of combat commanders, and then use this knowledge to make reasoning decisions.
[0102] Knowledge-based reasoning agent models require no training and can solve the cold start problem for agents; they can reflect strategic-level decisions that conform to military common sense, and their high-level intelligence level depends on the completeness of military domain knowledge.
[0103] Furthermore, step 6.1 is implemented in the following specific steps:
[0104] Step 6.1.1: Establish a military prior knowledge base;
[0105] Based on unstructured information such as military operational rules and the experience of operational commanders, data information in the field of wargaming is acquired, and a military prior knowledge base is established and presented in the form of a knowledge graph.
[0106] Step 6.1.2: Build the inference engine and output the overall task of the lower-level intelligent agent.
[0107] Based on the military prior knowledge base established in step 6.1.1, the set of combat tasks in the current combat scenario is classified to determine whether the overall task of the current lower-level intelligent agent is an offensive, retreating, or defensive combat task.
[0108] Step 6.2: Input auxiliary information for action evaluation and filtering;
[0109] The action probabilities and corresponding comprehensive scores of each agent in step 5 are input into the upper-level knowledge reasoning agent as decision-aid information. Based on the overall task of the current agent in step 6.1.2, the action is evaluated and filtered, and the set of decision actions with higher comprehensive scores that meet the overall task is selected as the optimal decision action sequence.
[0110] Step 6.3: Output the optimal decision action sequence for feedback;
[0111] Based on the optimal decision action sequence obtained in step 6.2, the action probabilities in the attribute fields of the lower-level data learning driven agent are adjusted: the action probabilities outside the optimal decision action sequence are modified to 0, and evenly distributed to the action probabilities corresponding to the optimal decision action sequence, ensuring that the sum of the action probabilities of the lower-level agent is 1.
[0112] Step 6.4: Input the optimal decision action probability sequence from Step 6.3 into the decision model of the intelligent war game simulation system, and the policy network in the decision model will output the optimal action decision of the current agent. Attached Figure Description
[0113] Figure 1 This is the flowchart upon which the present invention provides a wargaming agent-assisted decision-making method based on situational awareness interaction is based;
[0114] Figure 2 This is a model diagram of the battlefield situation awareness data mining process in a wargame intelligent agent-assisted decision-making method based on situation awareness interaction according to the present invention.
[0115] Figure 3 This is a diagram of the overall framework of a knowledge- and data-based wargaming agent in a situation-aware interactive wargaming agent-assisted decision-making method of the present invention.
[0116] Figure 4 This is a schematic diagram of the overall structure upon which the situation-aware interactive wargame agent-assisted decision-making method of the present invention is based.
[0117] Figure 5 This is a schematic diagram illustrating an application scenario of a wargaming-based intelligent agent-assisted decision-making method based on situational awareness interaction. Detailed Implementation
[0118] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0119] Reference Figure 1 The present invention provides a method for assisting decision-making in wargaming-based intelligent agents based on situational awareness interaction. The specific steps are as follows:
[0120] Step S1. Obtain situational awareness data of the battlefield environment in which the wargame agent is located;
[0121] Based on the geographic information of the hexagonal chessboard, the system simulates the environment around the intelligent agent and collects and acquires battlefield situation information, including enemy situation, friendly situation, and various battlefield environment data.
[0122] Step S1 specifically includes:
[0123] S1.1) Based on the characteristics of battlefield situation data in wargaming simulations and the functional requirements of wargaming intelligent agents, a large amount of data that meets the distribution characteristics of the battlefield environment is collected on the basis of small sample simulation data;
[0124] For example, battlefield situational data such as hexagonal grid location information, grid-to-grid feature information, elevation information, wargame category information, and ammunition carrying information.
[0125] S1.2) Use data mining to classify the battlefield situation and environment data collected in step S1.1, and integrate the classification results by combining the relationships between battlefield situation elements.
[0126] After collecting battlefield situational data, the data is combined with the battlefield situational element map to obtain a set of discontinuous data elements. By combining the relationships between the battlefield situational elements, the data element set is further integrated and processed to obtain more coherent situational information, thereby achieving accurate battlefield perception.
[0127] S1.2 specifically includes:
[0128] S1.2.1) Construct a military application model library for the logic layer and a data mining algorithm library for the algorithm layer;
[0129] S1.2.2) Through the combined application of the logic layer and the algorithm layer, battlefield situation environment characteristic data are processed by data mining to form various categories of battlefield situation element data;
[0130] S1.2.3) Integrate the data mining results by combining the relationships between battlefield situational elements to form coherent situational awareness data and achieve accurate battlefield awareness.
[0131] It should be noted that, in order to better demonstrate the feasibility of the present invention, the number of intelligent agents is set to 3 in this embodiment, and the action space is set as follows: maneuver upward, maneuver to the upper left, maneuver to the upper right, direct aim, indirect aim, and concealment.
[0132] In this embodiment, the battlefield situation information to be collected specifically includes:
[0133] Map information: area ID (area_evn_id), number of rows (hex_rows), number of columns (hex_columns), terrain (hexgon_terrain), elevation (hexgon_elevation), etc.
[0134] Piece Information (Enemy / Friendly Situation): Piece ID (chessman_obj_id), Piece Faction (chessman_force_id), Group (chessman_spawn_id), Current Unit Quantity (contain_numb), Direct-fire Ammo Quantity (anti_ammo_num), Indirect-fire Ammo Quantity (ammo_num), Pass-through Hexagonal Grid Index Information (pass_index), Non-Pass-through Hexagonal Grid Index Information (unpass_index), etc.
[0135] Step S2. Use the acquired situational awareness data as input to the action decision module of the wargaming agent. Analyze the impact of military factors on military operations and select the main battlefield situational data parameters that affect the agent's decision-making actions in the current specific scenario.
[0136] Step S2 specifically includes:
[0137] S2.1) Using the battlefield situational awareness data from step S1 as the input, perform preliminary screening, cleaning, and other data preprocessing operations.
[0138] By preprocessing the data, attribute components with low relevance to decision-making are removed from the battlefield situation awareness data. In this embodiment, weather and intelligence factors are removed.
[0139] S2.2) Construct a labeled dataset based on a series of formatted samples with positive and negative annotations obtained from actual adversarial and custom rules;
[0140] S2.3) Based on the label dataset constructed in step S2.2, filter the basic situational characteristic data at the combat location;
[0141] S2.4) The basic situation features in step S2.3 are encoded and various features are fused and filtered. They are correlated from multiple dimensions to form a structured situation feature dataset, which is then stored as the main battlefield situation data parameters.
[0142] For the wargaming system's operational environment in this embodiment, the main battlefield situational characteristics data can be considered as the global system state division s = σ. E ,σ1,σ2,..,σ N ,σ -1 ,…,σ -N Where σ E This refers to all information about the area outside of the pieces on the board, also known as map information; for each piece i∈I∪I on both sides - i, σ iThis represents all the information contained within the piece itself, also known as piece information.
[0143] The map information includes the attribute information of all the grids on the map and their topological relationships. The piece information includes the piece's original dynamic attributes and intrinsic attributes. Due to the importance of its position, the piece's positional information is further expanded by combining the grid it is located on with other information.
[0144] In this embodiment, the basic situational features (area ID (area_evn_id), terrain (hexgon_terrain), elevation (hexgon_elevation), piece ID (chessman_obj_id), piece's faction (chessman_force_id), direct-fire ammunition quantity (anti_ammo_num), indirect-fire ammunition quantity (ammo_num), pass-through hexagonal grid index information (pass_index), etc.) contained in the map information and piece information are encoded and fused to form a structured situational feature dataset, which is saved in JSON data format, as shown below.
[0145]
[0146] Step S3. Construct a decision-making action module, using the main battlefield situation data parameters obtained from the filtering and processing in step S2 as input to the decision-making action module, and outputting the probabilities of all possible decision actions;
[0147] Step S3 specifically includes:
[0148] S3.1) Constructing the action space;
[0149] An action space is constructed, including upper-level macro-level tasks based on military prior knowledge and lower-level agent behaviors based on deep reinforcement learning. Both upper and lower-level actions are discrete in form.
[0150] S3.1 specifically includes:
[0151] S3.1.1) - Constructing the action space based on hexagonal grid targets;
[0152] In such tasks, the multi-agent squad first selects a subset of hexagonal grids from a defined set of available hexagonal grid targets, and then performs tasks related to those hexagonal grids. These tasks typically involve maneuvering to the corresponding hexagonal grid and then performing discrete operations such as stopping, maneuvering upwards, maneuvering downwards, seizing control, or concealing oneself.
[0153] S3.1.2)_Constructing the action space based on wargame objectives.
[0154] In this type of task, the multi-agent squad first selects a subset from all surviving enemy wargame targets, and then performs tasks related to that subset of wargames. These tasks typically involve moving to a square at a suitable distance from the piece and then performing discrete operations such as stopping, firing, or concealing.
[0155] For the wargaming system platform in this embodiment, at each step, for agent ∈ I, its action is within the optional task set G. i Select a task g i ∈G i The task is generally represented as g. <c g ,x g >, which includes the specific type number c of the task g and the target location x involved in the task g ; Optional task set G = {g (1) ,g (1) ,...,g (K) This includes K operational missions permitted by environmental rules, the specific details of which depend on observation information, such as... Figure 4 As shown.
[0156] In this embodiment, the action space is constructed as follows: [Move upwards (mobi_up), move leftwards (mobi_left_up), move rightwards (mobi_right_up), direct aim (dir_aim), indirect aim (anti_dir_aim), conceal (conceal)]
[0157] S3.2) Taking the main battlefield situation data from step S2 as input, and combining it with the action space, the lower-level policy network generates a set of action probabilities.
[0158] The set of action probabilities can be obtained through the policy function. The lower-level agent generates a probability density function π, which uses the observed state s as input to the policy function π to obtain the probability of taking action 'a' in the current state. π describes the likelihood of taking action 'a' in the current state s.
[0159] According to the definition of a policy function, this function can serve as the basis for the decision-making output of the agent's command actions in the current battlefield state.
[0160] S3.2 specifically includes:
[0161] S3.2.1) Use the policy network π(a|s;θ) to approximate the policy function π, and give the executable command decision a under the current battlefield state s;
[0162] S3.2.2) Update network parameters θ using the gradient ascent method in the policy gradient algorithm;
[0163] S3.2.3) The lower-level agents take the battlefield situation perceived by the situation matrix and the prediction results of the target behavior as input, and generate the decision action probability set of each agent through the policy network.
[0164] In this embodiment, the battlefield state s is input, and the policy network outputs the probability set of decision actions for all current wargaming agents. The decision actions in this embodiment include: moving up (mobi_up), moving left and up (mobi_left_up), moving right and up (mobi_right_up), direct aiming (dir_aim), indirect aiming (anti_dir_aim), and concealment (conceal), as shown in the table below.
[0165] Probability set table of agent decision actions
[0166]
[0167] S3.3)_Construction Action-Value Function Q π , used to improve the policy network π(a|s;θ).
[0168] The Action-Value Function is defined as follows:
[0169]
[0170] This describes how an agent, at time t, observes the environmental state s and chooses action a. t Whether the outcome is wise depends on the discounted return U. t Regarding (s) t ,a t ) expectations.
[0171] According to the discount return U t Due to the randomness of the action, the agent cannot immediately observe the value of the action at time t. Therefore, expectation is introduced to estimate the reward of the action value at time t. π Only with s t ,a t related.
[0172] S3.3 is as follows:
[0173] S3.3.1) Construct a value network q(s,a;w) to approximate the action-value function Q. π The quality of executable command decision 'a' is evaluated, and this evaluation space is used to improve the strategy network.
[0174] S3.3.2) Update network parameters w using gradient descent in the temporal difference (TD) algorithm.
[0175] Specifically, in this embodiment, the agent's action decision is based on autonomous decision-making through deep reinforcement learning. At each step, for agent ∈ I, the neural network outputs a set of possible action probabilities d. i d i This indicates the probability of all executable actions of the current agent. And according to the evaluation space rules... Get an actual action a i .
[0176] For an agent ∈ I, the evaluation space rule The significance lies in the fact that when the agent obtains the probability set d of possible actions... i Through the task g selected by the upper layer i Constrain and correct d i The probability of a possible action is obtained as d' i ,Right now Finally, select the action with the highest probability, a. i And execute, that is
[0177] Step S4. Collect key battlefield data for the current combat scenario. Based on the mission requirements of the current round, collect key battlefield data for each wargaming agent in the scenario, such as survival index, deterrence index, artillery suppression index, combat effectiveness, etc., to provide comprehensive information support.
[0178] Step S4 specifically includes:
[0179] S4.1) Construct a battlefield element entity classification model;
[0180] S4.1 specifically includes:
[0181] S4.1.1) Constructing the CNN model architecture;
[0182] The CNN model consists of two convolutional layers, two pooling layers, one fully connected layer, and one softmax regression layer. It uses the Adam algorithm, which is based on a first-order gradient and a stochastic objective function, to optimize the entire neural network.
[0183] S4.1.2) Weight initialization;
[0184] To create this model, a large number of weights and biases need to be created. The weights in this model require a small amount of data noise during initialization to break symmetry and avoid zero gradients, thereby improving the accuracy of battlefield element entity classification. The model uses the ReLU activation function and initializes the biases with a small positive number to avoid the problem of neuron node outputs always being zero. The ReLU activation function is defined as:
[0185] f(x) = max(0,x)
[0186] Where x is the input neuron.
[0187] S4.1.3) Convolution and Pooling.
[0188] The task of the convolutional layer is to select features from different angles of the battlefield element feature data from the previous layer. After obtaining the convolutional features, they are divided into non-overlapping regions, and then the maximum feature of these regions is used to obtain the pooled convolutional features. The first layer consists of a convolution followed by a max-pooling operation. Each 5×5 convolutional kernel yields 32 features, and each output channel has a corresponding bias. Similarly, in the second layer, each 5×5 convolutional kernel yields 64 features.
[0189] S4.2) Construct a fully connected layer;
[0190] A fully connected layer with 1024 neurons is constructed to process the entire wargame battlefield environment data. Simultaneously, the vector output from the pooling layer is transformed into a one-dimensional vector, multiplied by the weight matrix, a bias is added, and the ReLU activation function is applied.
[0191] S4.3) Construct the output layer;
[0192] To reduce overfitting in battlefield element data classification, Dropout is added before the output layer, which can effectively improve the generalization ability of large neural networks. Finally, a Softmax layer is added. In Softmax regression, it is assumed that the class label y can take k distinct values. For a given test input x, the hypothesis function p(y=j|x:w) is the probability that x belongs to class j:
[0193]
[0194] Where w represents the parameters of the model.
[0195] S4.4) - Extract and classify features from battlefield map information and wargame information;
[0196] Furthermore, the specific implementation steps of step S4.4 are as follows:
[0197] S4.4.1) The battlefield map information and wargame information are input into the model in matrix form, and the convolutional layer is used for feature learning of battlefield elements;
[0198] S4.4.2) Input the battlefield element features output in step S4.4.1 into the fully connected layer for classification to obtain key battlefield element entity data.
[0199] S4.5) The battlefield element feature classification results output in step S4.4 are statistically analyzed into key battlefield data in the current combat scenario.
[0200] The key battlefield data collected in this embodiment for the current combat scenario are as follows:
[0201] Survival index, deterrence index, artillery suppression index, and combat effectiveness.
[0202] Step S5. The action probabilities output in step S3 are weighted and integrated with the key battlefield data statistically collected in step S4, and then classified according to the agent number.
[0203] Step S5 specifically includes:
[0204] S5.1) Determine the weighting details;
[0205] By using methods such as military expert knowledge base assessment and wargaming historical data analysis, the relative importance of the action probability output in step S3 to key battlefield data (such as survival index, deterrence index, artillery suppression index, combat effectiveness, etc.) in the current scenario is determined and presented in the form of weighted ratios.
[0206] S5.2) Verify and adjust weight allocation;
[0207] The allocation of battlefield key data weights in step S5.1 is verified and fine-tuned to ensure that the weight ratio of each battlefield key data meets the needs of the current scenario and can effectively support the action decision-making process under different circumstances.
[0208] S5.3) Perform a weighted integration of the action probabilities output in step S3 and the key battlefield data statistically collected in step S4;
[0209] Each key battlefield data point is multiplied by its corresponding weight ratio, and all processed key battlefield data points are summed to obtain a comprehensive score for the probability of the action, ensuring that the contribution of different factors to the final decision action is comprehensively considered.
[0210] S5.4) Based on the actual combat situation in the current scenario, the probability of each action after the integration of the key battlefield data weights in step S5.3 is quantitatively ranked according to its comprehensive score;
[0211] S5.5) Classify the output results of step S5.4 according to the agent number.
[0212] By identifying the unique ID field of each agent, the probabilities of all actions and their comprehensive scores quantified in step S5.4 are classified into the corresponding agent attribute fields, ensuring that personalized decision support is provided for each agent.
[0213] In this embodiment, the action probability set and key battlefield data statistically collected in step S3 are integrated, classified according to the agent number, and the results are stored in the form of a complete JSON dataset, as shown below.
[0214]
[0215] Step S6. Input the integrated data from step S5 as decision support information into the upper-level agent for evaluation and selection of decision actions, and then provide feedback to the lower-level agent to assist in generating the optimal decision action.
[0216] Step S6 specifically includes:
[0217] S6.1) Constructing a high-level knowledge-reasoning intelligent agent model;
[0218] Knowledge-based reasoning agents are intelligent agent models that form domain knowledge by manually extracting unstructured information such as combat rules, regulations, and the experience of combat commanders, and then use this knowledge to make reasoning decisions.
[0219] S6.1 specifically includes:
[0220] S6.1.1) Establish a military prior knowledge base;
[0221] Based on unstructured information such as military operational rules and the experience of operational commanders, data information in the field of wargaming is acquired, and a military prior knowledge base is established and presented in the form of a knowledge graph.
[0222] S6.1.2) _Build the inference engine and output the overall task of the lower-level intelligent agent.
[0223] Based on the military prior knowledge base established in step S6.1.1, the set of combat tasks in the current combat scenario is classified to determine whether the overall task of the current lower-level intelligent agent is an offensive, retreating, or defensive combat task.
[0224] S6.2) Input auxiliary information for action evaluation and filtering;
[0225] The action probabilities and corresponding comprehensive scores of each agent in step S5 are input into the upper-level knowledge reasoning agent as decision-aid information. Based on the overall task of the current agent in step S6.1.2, the actions are evaluated and filtered, and the decision actions with higher comprehensive scores that meet the overall task are selected as the optimal decision action sequence.
[0226] S6.3) Output the optimal decision action sequence for feedback;
[0227] Based on the optimal decision action sequence obtained in step S6.2, the action probabilities in the attribute fields of the lower-level data learning driven agent are adjusted: the action probabilities outside the optimal decision action sequence are modified to 0, and evenly distributed to the action probabilities corresponding to the optimal decision action sequence, ensuring that the sum of the action probabilities of the lower-level agent is 1.
[0228] In this embodiment, the agent generates the optimal decision action sequence according to the above-described auxiliary decision-making steps as follows:
[0229]
[0230] S6.4) Input the optimal decision action probability sequence from step S6.3 into the decision model of the intelligent war game simulation system, and the policy network in the decision model outputs the optimal action decision of the current agent.
[0231] It should be noted that, as used herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0232] The sequence numbers of the embodiments of the present invention described above are merely for convenience and do not represent the superiority or inferiority of the embodiments.
[0233] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0234] The above are merely embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A wargaming agent-assisted decision-making method based on situational awareness interaction, comprising the following steps: S1. Acquire situational awareness data of the battlefield environment in which the wargame agent is located. The situational awareness data includes hexagonal grid position information, grid-to-grid feature information, elevation information, wargame category information, and ammunition carrying information. S2. The acquired situational awareness data is used as the input to the action decision module of the wargaming agent. The impact of military factors on military operations is analyzed, and the main battlefield situational data parameters that affect the agent's decision-making actions in the current specific scenario are selected. S3. Construct a decision-making action module, using the main battlefield situation data parameters obtained from step S2 as input to the decision-making action module, and outputting the probability of decision-making actions. Step S3 specifically includes: S3.1) Constructing the action space: Constructing the action space includes upper-level macro-level tasks based on military prior knowledge and lower-level agent behaviors based on deep reinforcement learning. Both upper and lower level actions are in discrete form. S3.2) Generate a set of possible action probabilities based on the lower-level policy network; S3.3) Constructing the action-value function Used to improve policy networks ; Specifically, step S3.1 includes: S3.1.1) Constructing an action space based on a hexagonal grid target; In such tasks, the multi-agent squad first selects a subset of hexagonal grids from a defined set of available hexagonal grid targets, and then performs tasks related to the hexagonal grids; such tasks refer to maneuvering to the corresponding hexagonal grid and then performing discrete operations, including: stopping, maneuvering upwards, maneuvering downwards, seizing control, or concealing. S3.1.2) Construct the action space based on the wargame objective; In this type of task, the multi-agent squad first selects a subset from the set of all surviving enemy wargame targets, and then performs tasks related to the wargames of that subset; the tasks related to the wargames of that subset refer to moving to a grid at a suitable distance from the wargame and then performing discrete operations. Specifically, step S3.2 includes: S3.2.1) Using a policy network Approximation policy function Provide the current battlefield status. Executable command decisions ; S3.2.2) Update network parameters using the gradient ascent method in the policy gradient algorithm. ; S3.2.3) The lower-level agents take the battlefield situation perceived by the situation matrix and the prediction results of the target behavior as input, and generate the decision action probability set of each agent through the policy network; Step S3.3) specifically includes: S3.3.1) Building a value network Used for approximating action-value functions Assess the feasibility of command decisions The degree of good or bad is used as an evaluation space to improve the policy network; S3.3.2) Update network parameters using gradient descent in the time difference algorithm. ; S4. Collect key battlefield data for the current combat scenario. Based on the mission requirements of the current round, collect key battlefield data for each wargaming agent in the current scenario to provide comprehensive information support. The key battlefield data includes survival index, deterrence index, artillery suppression index, and combat effectiveness. S5. The action probabilities output in step S3 are weighted and integrated with the key battlefield data collected in step S4, and then classified according to the agent ID; specifically including: S5.1) Determine the weighting details; By using the methods of military expert knowledge base evaluation and wargaming historical data analysis, the relative importance of the action probability output in step S3 to the key battlefield data in the current scenario is determined and presented in the form of weight ratio. The key battlefield data includes survival index, deterrence index, artillery suppression index, and combat effectiveness. S5.2) Verify and adjust weight allocation; The allocation of battlefield key data weights in step S5.1) is verified and fine-tuned to ensure that the weight ratio of each battlefield key data meets the needs of the current scenario and can effectively support the action decision-making process under different circumstances. S5.3) Perform a weighted integration of the action probabilities output in step S3 and the key battlefield data statistically collected in step S4; Each key battlefield data point is multiplied by its corresponding weight ratio, and all processed key battlefield data points are summed to obtain a comprehensive score for the probability of the action, ensuring that the contribution of different factors to the final decision action is comprehensively considered. S5.4) Based on the actual combat situation in the current scenario, the probability of each action after the integration of the key battlefield data weights in step S5.3 is quantitatively ranked according to its comprehensive score; S5.5) Classify the output results of step S5.4 according to the agent number; By identifying the unique ID field of each agent, the probabilities of all actions and their comprehensive scores that were quantified and arranged in step S5.4) are classified into the corresponding agent attribute fields to ensure that personalized decision support is provided for each agent. S6. Input the weighted integrated data from step S5 as decision support information into the upper-level agent to evaluate and filter decision actions, provide feedback to the lower-level agent, and generate the optimal decision action. Step S6 specifically includes: S6.1) Constructing a high-level knowledge-reasoning intelligent agent model; The knowledge-reasoning intelligent agent is an intelligent agent model that forms domain knowledge by manually extracting unstructured information such as combat rules, regulations, and the experience of combat commanders, and makes reasoning decisions based on this knowledge. Specifically, step S6.1 includes: S6.1.1) Establish a military prior knowledge base; Based on unstructured information from military operational rules and the experience of operational commanders, data information in the field of wargaming is acquired, a military prior knowledge base is established, and presented in the form of a knowledge graph. S6.1.2) Build the inference engine and output the overall task of the lower-level intelligent agent; Based on the military prior knowledge base established in step S6.1.1), the set of combat tasks in the current combat scenario is classified to obtain the overall task of the current lower-level intelligent agent as an offensive, retreating, or defensive combat task. S6.2) Input auxiliary information for action evaluation and filtering; The action probabilities and corresponding comprehensive scores of each agent in step S5 are used as decision-making auxiliary information and input into the upper-level knowledge reasoning agent. Based on the overall task of the current agent in step S6.1.2), the action is evaluated and filtered, and the decision action with a higher comprehensive score that meets the overall task is selected as the optimal decision action sequence. S6.3) Output the optimal decision action sequence for feedback; Based on the optimal decision action sequence obtained in step S6.2), the action probabilities in the attribute fields of the lower-level data learning driven agent are adjusted: the action probabilities outside the optimal decision action sequence are modified to 0 and evenly distributed to the action probabilities corresponding to the optimal decision action sequence, ensuring that the sum of the action probabilities of the lower-level agent is 1. S6.4) Input the optimal decision action probability sequence from step S6.3) into the decision model of the intelligent war game simulation system, and the policy network in the decision model outputs the optimal action decision of the current agent.
2. The wargaming agent-assisted decision-making method based on situational awareness interaction according to claim 1, characterized in that, Step S1 specifically includes: S1.1) Data Acquisition: Based on the characteristics of battlefield situation data in wargaming simulations and the functional requirements of wargaming intelligent agents, a large amount of data that meets the distribution characteristics of the battlefield environment is generated on the basis of small sample simulation data. S1.2) Data Mining: After collecting battlefield situational data, the battlefield situational element map is combined to obtain a set of discontinuous data elements. By combining the relationships between battlefield situational elements, more coherent situational information is obtained, thus achieving accurate battlefield perception.
3. The wargaming agent-assisted decision-making method based on situational awareness interaction according to claim 2, characterized in that, Step S1.2) data mining specifically includes: S1.2.1) Construct a military application model library for the logic layer and a data mining algorithm library for the algorithm layer; S1.2.2) Through the combined application of the logic layer and the algorithm layer, the battlefield situation environment feature data is processed by data mining to form various types of battlefield situation element data, which serve as situational awareness data of the battlefield environment in which the wargame agent is located.
4. The wargaming agent-assisted decision-making method based on situational awareness interaction according to claim 3, characterized in that, Step S2 specifically includes: S2.1) Data preprocessing: Using raw battlefield situational awareness data as input, perform preliminary screening and cleaning operations; S2.2) Label Dataset Construction: The label dataset is constructed from a series of formatted samples with positive and negative labels obtained based on actual adversarial and custom rules; S2.3) Construction of situation feature dataset: Based on the labeled dataset, calculate and store basic situation feature data at the combat position, encode the basic situation features and perform diversified feature fusion and filtering, and associate them from multiple dimensions to form a structured situation feature dataset.
5. The wargaming agent-assisted decision-making method based on situational awareness interaction according to claim 4, characterized in that, Step S4 specifically includes: S4.1) Construct a battlefield element entity classification model; S4.2) Construct a fully connected layer: S4.3) Construct the output layer: S4.4) Perform feature extraction and classification on battlefield map information and wargame information.
Citation Information
Patent Citations
Knowledge-driven war game deduction intelligent decision-making method
CN113435598A
Wargame deduction cooperation strategy obtaining method based on deep reinforcement learning model
CN114722701A
