Wargame agent auxiliary decision-making method based on situation awareness interaction

Through the situation-aware interaction, the problem of sparse feedback and complexity of decision-making in wargame deduction is solved, efficient and accurate decision support and systematic training and correction are achieved, and the authenticity and accuracy of war situations are improved.

CN118966356AActive Publication Date: 2024-11-15XIAN UNIV OF TECH
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202411012209.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-15
Estimated Expiration
2044-07-26

AI Technical Summary

Technical Problem

In wargame deduction, the complex and changeable battlefield situation and high dynamic environment lead to the problem of sparse feedback, delay and inaccurate feedback in the command and decision-making process, which increases the difficulty of training the strategy model and makes it difficult to make high-quality decisions.

Method used

Using a war chess agent-assisted decision-making method based on situational awareness interaction, we generate the best by obtaining battlefield situation data, analyzing military factors, building decision-making action modules, counting key battlefield data, and integrating weights, and using upper-level agents to make decision-making judgments and screening. Decision-making actions.

Benefits of technology

It improves the efficiency and accuracy of the decision-making of the agent, provides decision-making support that conforms to the real battlefield situation, and provides system managers with a basis for deduction and review, ensures the correction of training parameters, and achieves the deduction effect that is appropriate for the war situation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118966356A_ABST
    Figure CN118966356A_ABST
Patent Text Reader

Abstract

The invention discloses a war game agent auxiliary decision-making method based on situation awareness interaction. The method specifically comprises the steps of obtaining situation awareness data of a battlefield environment where a war game agent is located; screening the situation awareness data according to a military factor theory; constructing a decision action module, inputting battlefield situation data, and outputting a decision action probability; counting battlefield key data of the current battlefield scene; carrying out weight integration on the decision action probability and the statistical battlefield key data, and carrying out classification according to the number of the agent; inputting the weight integrated data as decision-making auxiliary information into an upper-layer agent to judge and screen decision-making actions, and feeding back a lower-layer agent to generate an optimal decision-making action; through the above steps, decision support can be provided for the war game intelligent agent in combination with an intelligent war game deduction system, and the decision efficiency and accuracy of the intelligent agent are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the research field of key technologies of war game simulation systems, and specifically relates to a war game intelligent agent auxiliary decision-making method based on situation awareness interaction. Background Art

[0002] As the pace of modern warfare continues to accelerate, the future joint combat situation will become more complex and diverse. Intelligent synthetic battalion formations will help to seize the initiative on the battlefield, achieve the goal of defeating the slow with the fast, quickly and flexibly focus forces, and accurately and efficiently strike enemy targets, playing a very important role in future joint operations.

[0003] However, in the war game simulation and confrontation process, the battlefield situation is complex and changeable, and there are many combat entities, resulting in the command decision-making process facing a high-dimensional state-action space. Specifically, it is difficult to directly convert the combat mission objectives into a reasonable reward function, and it is impossible to provide immediate and accurate feedback, resulting in combat behavior facing problems such as sparse feedback, delays, and inaccuracy. In addition, the highly dynamic battlefield environment and the continuously changing game tasks increase the difficulty of training the strategy model, making it a major challenge to make high-quality decisions under different game tasks.

[0004] Multi-agent technology refers to a technical means of completing tasks by using the interaction and collaboration between multiple agents. An agent is a machine or software entity with autonomous decision-making capabilities. By combining multiple agents, an intelligent system can be formed that can collaborate and communicate like humans. By combining multi-agents with synthetic battalion formations, each branch of the military can be regarded as an agent that can select appropriate strategies through autonomous reasoning and planning, simplifying the process of command decision-making, and thus better completing tasks.

[0005] The invention proposes a multi-agent-based decision-making assistance method, combined with the existing war game simulation system, to provide decision support for the agent through intelligent algorithms and data analysis. This method can assist the agent in decision analysis, scenario assessment and tactical planning by interacting with real-time battlefield situation data, thereby improving the efficiency and accuracy of the agent's decision-making. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-agent decision-making assistance method based on war game simulation, which provides necessary auxiliary information support for the agent when making decisions, so that it can make decision actions that conform to the real battlefield scenario based entirely on the current battlefield situation data. At the same time, it also provides a basis for system managers to conduct simulation review, so as to facilitate timely correction of agent training parameters and ensure the most appropriate simulation effect for the real war scenario.

[0007] The technical solution adopted by the present invention is a method for assisting decision-making of a war game agent based on situation awareness interaction, which is specifically implemented according to the following steps:

[0008] Step 1: Obtain situational awareness data of the battlefield environment where the war game agent is located;

[0009] Step 2: Use the acquired situational awareness data as the input of the action decision module of the war game agent, analyze the impact of military factors on military operations, and screen out the main battlefield situation data parameters that affect the agent's decision-making actions in the current specific scenario;

[0010] Step 3: construct a decision action module, use the main battlefield situation data parameters obtained by screening and processing in step 2 as input to the decision action module, and output all possible decision action probabilities;

[0011] Step 4: Count the key battlefield data of the current combat scenario. According to the task requirements of the current round, count the key battlefield data of each war game agent in the scenario, such as survival index, deterrence index, artillery suppression index, combat effectiveness and other data, to provide comprehensive information support;

[0012] Step 5: weight the action probability output in step 3 and the key battlefield data counted in step 4, and classify them according to the agent number;

[0013] Step 6: Input the weighted integrated data in step 5 as decision-making auxiliary information to the upper-level intelligent agent to judge and screen the decision-making actions, provide feedback to the lower-level intelligent agent, and generate the optimal decision-making action.

[0014] The present invention is also characterized in that:

[0015] The understanding and perception of battlefield situation mainly includes environmental data such as combat maps and deduction data;

[0016] Environmental data such as combat maps are specific examples of military rules, such as structured representation data of terrain;

[0017] The deduction data mainly includes constraint rule data, self-game data, replay data, etc., which provide a learning and training basis for wargame agents.

[0018] The agent will perceive the current battlefield environment state based on the state space. Its data comes from the war game simulation system and is obtained through interaction with the war game simulation system. In a certain battle simulation, the agent will perceive the following at the beginning of the round:

[0019] (1) Battlefield map information: hexagonal grid terrain information, grid feature information, grid feature information, elevation information, etc.

[0020] (2) Game information: game category information, whether the game is dead, the location of the game in the hexagonal grid, morale and fatigue values, formation and direction, number of ammunition carried, etc.

[0021] (3) Combat scenario rule information: weapon system rule information, reconnaissance system rule information, mobility system rule information, etc.;

[0022] In step 1, the situation awareness data of the battlefield environment in which the war game agent is located is obtained by simulating the environment around the agent based on the geographic information of the hexagonal chessboard, and collecting battlefield situation information including enemy situation, our situation and various battlefield environment data.

[0023] Furthermore, the specific implementation steps of step 1 are as follows:

[0024] Step 1.1: Based on the characteristics of battlefield situation data in wargame simulation and the functional requirements of wargame agents, a large amount of data that meets the distribution characteristics of the battlefield environment is collected on the basis of small sample simulation data;

[0025] The data that meets the distribution characteristics of the battlefield environment on the war game simulation platform mainly include: hexagonal grid position information, grid feature information, elevation information, war game category information, ammunition carried information and other battlefield situation environment data.

[0026] Step 1.2: Classify the battlefield situation environment data collected in step 1.1 through data mining, and integrate the classification results based on the relationship between battlefield situation elements.

[0027] After collecting battlefield situation and environment data, the purpose of data mining is to classify and cluster the original data and obtain a set of incoherent data elements by combining the battlefield situation element map. Then, the data element set is further integrated and processed based on the relationship between battlefield situation elements to obtain coherent situation information, thereby completing accurate battlefield perception.

[0028] Furthermore, the step 1.2 is specifically implemented according to the following steps:

[0029] Step 1.2.1: Build a military application model library at the logic layer and a data mining algorithm library at the algorithm layer;

[0030] Step 1.2.2, after the combined application of the logic layer and the algorithm layer, the battlefield situation environment feature data is processed through data mining to form multiple categories of battlefield situation element data;

[0031] Step 1.2.3: Integrate the data mining results based on the relationship between battlefield situation elements to form coherent situation awareness data and complete accurate battlefield perception.

[0032] In step 2, the situation awareness data obtained in step 1 is used as the input of the action decision module of the war game agent. According to the impact of military factors on military operations, the main battlefield situation data parameters that affect the agent's decision-making actions in the current specific scenario are analyzed and screened.

[0033] The battlefield situation data of the war game simulation system improves the accuracy of the data on the basis of simulating the real combat environment, but the relationship between the data requires combat commanders to use military combat theoretical knowledge to analyze, screen and accurately assess the battlefield situation through the correlation between the data.

[0034] Furthermore, the specific implementation steps of step 2 are as follows:

[0035] Step 2.1: Take the battlefield situation awareness data in step 1 as the object input and perform preliminary screening, cleaning and other data preprocessing operations;

[0036] Step 2.2: Construct a labeled dataset based on a series of formatted samples with positive and negative annotations obtained based on actual adversarial and custom rules;

[0037] Step 2.3, based on the label data set constructed in step 2.2, filter the basic situation feature data at the combat position;

[0038] Step 2.4: Data encode the basic situation features in step 2.3 and perform diversified feature fusion screening, associate them from multiple dimensions to form a structured situation feature data set and store it as the main battlefield situation data parameters.

[0039] In step 3, the main battlefield situation data parameters obtained by screening and processing in step 2 are used as the input of the decision-making action module, and the probabilities of all possible decision actions of the war game agent are output.

[0040] Furthermore, the specific implementation steps of step 3 are as follows:

[0041] Step 3.1, construct the action space;

[0042] The action space includes upper-level macro tasks based on military prior knowledge and lower-level intelligent agent behaviors based on deep reinforcement learning. Both upper and lower-level actions are in discrete form.

[0043] Furthermore, the step 3.1 is specifically implemented according to the following steps:

[0044] Step 3.1.1: Construct action space based on the task of hexagonal grid target;

[0045] In this type of task, the multi-agent team first selects some hexagonal grids from the set of available hexagonal grid targets, and then performs tasks related to the hexagonal grids. Such tasks usually refer to maneuvering to the corresponding hexagonal grid and then performing some discrete operations, such as: stopping, maneuvering upward, maneuvering downward, seizing control or hiding.

[0046] Step 3.1.2: Construct the action space based on the tasks of the war game objectives.

[0047] In this type of mission, the multi-agent team first selects a subset from the set of all surviving enemy game targets and then performs a task related to that subset of game. This task usually involves moving to a grid at a suitable distance from the chess piece and then performing some discrete actions, such as: stop, shoot, or hide. The purpose of the stop or hide mission is similar to that of stalking.

[0048] Step 3.2: Taking the main battlefield situation data in step 2 as input and combining it with the action space, the lower-level strategy network generates a set of feasible action probabilities;

[0049] The set of possible action probabilities can be obtained through the policy function The lower-level agent uses the observed state s as the input of the policy function π to obtain the probability of taking action a in the current state. The policy function π is a probability density function that describes the possibility of taking action a in the current state s.

[0050] According to the definition of the strategy function, this function can be used as the basis for the lower-level intelligent war game to make command decisions under the current battlefield status.

[0051] Furthermore, the step 3.2 is specifically implemented according to the following steps:

[0052] Step 3.2.1, use the policy network π(a|s; θ) to approximate the policy function π, and give the executable command decision a under the current battlefield state s;

[0053] Step 3.2.2, update the network parameter θ by gradient ascent in the policy gradient algorithm;

[0054] Step 3.2.3: The battlefield situation perceived by the lower-level agent based on the situation matrix and the predicted results of the target behavior are used as input, and the decision action probability set of each agent is generated through the strategy network.

[0055] Step 3.3: Construct the action-value function Q π , used to improve the policy network π(a|s;θ).

[0056] The action-value function is defined as:

[0057]

[0058] The above formula describes the agent's choice of action a when it observes the environment state s at time t. t The result is wise, which is the discounted return U t About t ,a t ) expectations.

[0059] Based on the discount return U t The agent cannot observe the randomness of the action immediately at time t, so the expectation is introduced to estimate the benefit of the action value. Finally, the action-value function Q at time t is π Only with s t ,a t related.

[0060] Furthermore, step 3.3 is specifically implemented according to the following steps:

[0061] Step 3.3.1: Construct a value network q(s,a;w) to approximate the action-value function Q π , judge the quality of executable command decision a, and use it as an evaluation space to improve the strategy network;

[0062] Step 3.3.2: Update the network parameter w by gradient descent in the temporal difference (TD) algorithm.

[0063] In step 4, it is necessary to collect statistics on the key battlefield data of each wargame agent in the scene according to the combat mission requirements of the current round, such as survival index, deterrence index, artillery suppression index, combat effectiveness and other data, to provide comprehensive information support for the agent's decision-making.

[0064] Furthermore, the specific implementation steps of step 4 are as follows:

[0065] Step 4.1, construct a battlefield element entity classification model;

[0066] The battlefield situation element entity classification model combines battlefield situation elements in a supervised manner to help train the entity classification model. After the model training is completed, it will extract and classify features of battlefield map information and war game information to output a set of key battlefield entity elements in the combat scenario.

[0067] Furthermore, step 4.1 is specifically implemented according to the following steps:

[0068] Step 4.1.1, build the CNN model architecture;

[0069] The CNN model consists of two convolutional layers, two pooling layers, a fully connected layer and a Softmax regression layer. The Adam first-order gradient-based stochastic objective function algorithm is used to optimize the entire neural network.

[0070] Step 4.1.2, weight initialization;

[0071] In order to create this model, a large number of weights and bias terms need to be created. When initializing the weights in this model, a small amount of data noise needs to be added to break the symmetry and avoid zero gradients to improve the accuracy of battlefield element entity classification. The model uses the ReLU activation function and initializes the bias term with a small positive number to avoid the problem of the output of the neuron node being always 0. The ReLU activation function is defined as:

[0072] f(x)=max(0,x)

[0073] Among them, x is the input neuron.

[0074] Step 4.1.3, convolution and pooling.

[0075] The task of the convolution layer is to select features from different angles of the feature data of battlefield elements in the previous layer. After obtaining the convolution features, the convolution features are divided into non-overlapping areas, and then the maximum features of these areas are used to obtain the pooled convolution features. The first layer is completed by a convolution followed by a Maxpooling. The convolution obtains 32 features in each 5×5 convolution kernel, and there is a corresponding offset for each output channel. Similarly, in the second layer, each 5×5 convolution kernel obtains 64 features.

[0076] Step 4.2: Build a fully connected layer;

[0077] A fully connected layer with 1024 neurons is constructed to process the entire war game battlefield environment data. At the same time, the vector output by the pooling layer is converted into a one-dimensional vector, multiplied by the weight matrix, added with the bias, and the RELU activation function is used.

[0078] Step 4.3: Construct the output layer.

[0079] In order to reduce overfitting of battlefield element data classification, adding Dropout before the output layer can effectively improve the generalization ability of large neural networks. Finally, add the Softmax layer. In Softmax regression, it is assumed that the class label y can take k different values. For a given test input x, assume that the function p(y=j|x:w) is the probability value of x belonging to category j:

[0080]

[0081] Among them, w is the parameter of the model.

[0082] Step 4.4, extracting and classifying features of battlefield map information and war game information;

[0083] Furthermore, the specific implementation steps of step 4.4 are as follows:

[0084] Step 4.4.1: Input battlefield map information and wargame information into the model in matrix form, and the convolutional layer is used for feature learning of battlefield elements;

[0085] Step 4.4.2: Input the battlefield element features output in step 4.4.1 into the fully connected layer for classification to obtain the key battlefield element entity data.

[0086] Step 4.5: Count the battlefield element feature classification results output from step 4.4 as key battlefield data in the current combat scenario.

[0087] In step 5, the action probability output from step 3 and the key battlefield data counted in step 4 need to be weighted and integrated. The processed data are classified according to the agent number and serve as auxiliary information for action decision-making to influence the final decision action of the agent.

[0088] Furthermore, the specific implementation steps of step 5 are as follows:

[0089] Step 5.1, determine the weight details;

[0090] Through methods such as military expert knowledge base evaluation and war game historical data analysis, the relative importance of the action probability output in step 3 to key battlefield data (such as survival index, deterrence index, artillery suppression index, combat effectiveness, etc.) in the current scenario is determined and presented in the form of a weighted ratio.

[0091] Step 5.2: Verify and adjust weight distribution;

[0092] Verify and fine-tune the distribution of battlefield key data weights in step 5.1 to ensure that the weight ratio of each battlefield key data meets the needs of the current scenario and can effectively support the action decision-making process in different situations.

[0093] Step 5.3: weighted integration of the action probability output in step 3 and the key battlefield data counted in step 4;

[0094] Each battlefield key data is multiplied by its corresponding weight ratio, and all processed battlefield key data are added up as the comprehensive score of the action probability, ensuring that the contribution of different factors to the final decision-making action is comprehensively considered.

[0095] Step 5.4: According to the actual combat situation in the current scenario, the probability of each action after the integration of the battlefield key data weights in step 5.3 is quantitatively arranged according to its comprehensive score;

[0096] Step 5.5: Classify the output results of step 5.4 according to the agent number.

[0097] By identifying the unique number field for each agent, all action probabilities and their comprehensive scores quantified and arranged in step 5.4 are classified into the corresponding agent attribute fields to provide personalized decision support for each agent.

[0098] In step 6, the output result of step 5 is input into the upper-level intelligent agent as decision-making auxiliary information for action screening and evaluation, and the optimal action probability sequence that meets the combat mission in the current scenario is output. Feedback is given to the lower-level intelligent agent and the optimal decision action is output.

[0099] Furthermore, the specific implementation steps of step 6 are as follows:

[0100] Step 6.1: Construct an upper-level knowledge reasoning agent model;

[0101] The knowledge-reasoning intelligent agent is an intelligent agent model that forms domain knowledge by manually extracting unstructured information such as combat rules, regulations, and combat commanders' experience, and makes reasoning decisions based on this.

[0102] The knowledge-reasoning intelligent agent model does not require training and can solve the cold start problem of the intelligent agent. It can reflect strategic-level decisions that are consistent with military common sense, and its high-level intelligence level depends on the degree of perfection of military knowledge.

[0103] Furthermore, step 6.1 is specifically implemented according to the following steps:

[0104] Step 6.1.1, establish a military prior knowledge base;

[0105] Based on unstructured information such as military combat rules and combat commanders' experience, data information in the field of war game simulation is obtained, and a military prior knowledge base is established and presented in the form of a knowledge graph.

[0106] Step 6.1.2: Build an inference engine to output the overall task of the lower-level agent.

[0107] According to the military prior knowledge base established in step 6.1.1, the combat task set in the current combat scenario is classified to obtain the overall task of the current lower-level intelligent agent, such as attack, retreat, or defense.

[0108] Step 6.2, input auxiliary information for action evaluation and screening;

[0109] The action probability and corresponding comprehensive score of each agent in step 5 are input into the upper-level knowledge reasoning agent as decision-making auxiliary information, and the actions are judged and screened according to the overall task of the current agent in step 6.1.2, and the decision action set with a higher comprehensive score that meets the overall task is selected as the optimal decision action sequence.

[0110] Step 6.3: Output the optimal decision action sequence for feedback;

[0111] According to the optimal decision action sequence screened in step 6.2, the action probabilities in the attribute fields of the lower-level data learning-driven intelligent agent are adjusted: the action probabilities outside the optimal decision action sequence are modified to 0, and evenly supplemented to the action probabilities corresponding to the optimal decision action sequence, ensuring that the sum of the action probabilities of the lower-level intelligent agent is 1.

[0112] Step 6.4: Input the optimal decision action probability sequence in step 6.3 into the decision model of the intelligent war game simulation system, and the policy network in the decision model outputs the optimal action decision of the current intelligent agent. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 It is a flow chart based on a situation awareness interaction-based war game agent decision-making assistance method of the present invention;

[0114] Figure 2 It is a model diagram of battlefield situation awareness data mining process in a war game intelligent agent auxiliary decision-making method based on situation awareness interaction of the present invention;

[0115] Figure 3 It is an overall framework diagram of a war game intelligent agent based on knowledge and data in a war game intelligent agent auxiliary decision-making method based on situation awareness interaction of the present invention;

[0116] Figure 4 It is a schematic diagram of the overall structure of a war game agent auxiliary decision-making method based on situation awareness interaction of the present invention;

[0117] Figure 5 This is a schematic diagram of the application scenario of a war game agent-assisted decision-making method based on situational awareness interaction; DETAILED DESCRIPTION

[0118] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0119] Reference Figure 1 The present invention provides a method for assisting decision-making of a war game agent based on situation awareness interaction. The specific steps are as follows:

[0120] Step S1. Obtain situational awareness data of the battlefield environment in which the war game agent is located;

[0121] The surrounding environment of the intelligent agent is simulated according to the hexagonal chessboard geographic information, and battlefield situation information including enemy situation, our situation and various battlefield environment data are collected and obtained.

[0122] Step S1 specifically includes:

[0123] S1.1)_According to the characteristics of battlefield situation data in wargame simulation and the functional requirements of wargame agents, a large amount of data that meets the distribution characteristics of the battlefield environment is collected based on small sample simulation data;

[0124] For example, battlefield situation and environmental data such as hexagonal grid position information, grid feature information, elevation information, game category information, and ammunition carried.

[0125] S1.2)_Use data mining to classify the battlefield situation environment data collected in step S1.1, and integrate the classification results based on the relationship between battlefield situation elements.

[0126] After collecting battlefield situation and environmental data, we combine it with the battlefield situation element map to obtain an incoherent set of data elements. Combined with the relationship between battlefield situation elements, we further integrate and process the data element set to obtain more coherent situation information, thereby achieving accurate battlefield perception.

[0127] S1.2 specifically includes:

[0128] S1.2.1)_Build a military application model library at the logic layer and a data mining algorithm library at the algorithm layer;

[0129] S1.2.2)_After the combined application of the logic layer and the algorithm layer, the battlefield situation environment feature data is processed through data mining to form multiple categories of battlefield situation element data;

[0130] S1.2.3)_Integrate the data mining results based on the relationships between battlefield situation elements to form coherent situation awareness data and achieve accurate battlefield perception.

[0131] It should be noted that, in order to better demonstrate the feasibility of the present invention, the number of agents is set to 3 in this embodiment, and the action space is set to: maneuver upward, maneuver to the upper left, maneuver to the upper right, direct aim, indirect aim, and concealment.

[0132] The battlefield situation information collected in this embodiment specifically includes:

[0133] Map information: area number (area_evn_id), number of area rows (hex_rows), number of area columns (hex_columns), terrain (hexgon_terrain), elevation (hexgon_elevation), etc.

[0134] Chess piece information (enemy situation / our situation): chess piece number (chessman_obj_id), chess piece's faction (chessman_force_id), group (chessman_spawn_id), current number of units (contain_numb), direct aiming ammunition (anti_ammo_num), indirect aiming ammunition (ammo_num), line of sight hexagonal grid index information (pass_index), unpassable line of sight hexagonal grid index information (unpass_index), etc.

[0135] Step S2. The acquired situational awareness data is used as the input of the action decision module of the war game agent, and the impact of military factors on military operations is analyzed to screen out the main battlefield situation data parameters that affect the agent's decision-making actions in the current specific scenario;

[0136] Step S2 specifically includes:

[0137] S2.1)_Use the battlefield situation awareness data in step S1 as the object input, and perform preliminary screening, cleaning and other data preprocessing operations;

[0138] Through data preprocessing, attribute components with low relevance to decision-making in the battlefield situation awareness data are eliminated. In this embodiment, weather and intelligence factors are eliminated.

[0139] S2.2)_Build a labeled dataset based on a series of formatted samples with positive and negative annotations obtained based on actual adversarial and custom rules;

[0140] S2.3) Based on the label data set constructed in step S2.2, the basic situation feature data at the combat position is screened;

[0141] S2.4)_Data encoding and diversified feature fusion screening are performed on the basic situation features in step S2.3, and association is performed from multiple dimensions to form a structured situation feature data set and store it as the main battlefield situation data parameters.

[0142] For the war game simulation system combat environment in this embodiment, the main battlefield situation characteristic data can be considered as the division of the global system state s = σ E ,σ1,σ2,..,σ N ,σ -1 ,…,σ -N where σ E Refers to all the information on the field other than the chess pieces of both sides, also known as map information; for each chess piece i∈I∪I - i,σ iRepresents all the information contained in the chess piece itself, also known as chess piece information.

[0143] For map information, it includes the attribute information of all grids on the map and the topological relationship between them. For chess piece information, it includes the original dynamic attributes and intrinsic attribute information of the chess piece. Due to the importance of the position, the position information is separately expanded by combining a series of information such as the grid where the chess piece is located.

[0144] In this embodiment, the basic situation features (area number (area_evn_id), terrain (hexgon_terrain), elevation (hexgon_elevation), chess piece number (chessman_obj_id), chess piece camp (chessman_force_id), direct aiming ammunition (anti_ammo_num), indirect aiming ammunition (ammo_num), line of sight hexagonal grid index information (pass_index), etc.) contained in the map information and chess piece information are encoded and feature fused to form a structured situation feature data set saved in JSON data format, as shown below.

[0145]

[0146] Step S3. construct a decision action module, use the main battlefield situation data parameters obtained by the screening process in step S2 as the input of the decision action module, and output all possible decision action probabilities;

[0147] Step S3 specifically includes:

[0148] S3.1)_Build action space;

[0149] Construct an action space, including upper-level macro tasks based on military prior knowledge and lower-level intelligent agent behaviors based on deep reinforcement learning. Both upper and lower-level actions are in discrete form.

[0150] S3.1 specifically includes:

[0151] S3.1.1)_Construct action space based on the task of hexagonal grid target;

[0152] In this type of task, the multi-agent team first selects some hexagonal grids from the set of available hexagonal grid targets, and then performs tasks related to the hexagonal grids. Such tasks usually refer to maneuvering to the corresponding hexagonal grid and then performing some discrete operations, such as: stopping, maneuvering upward, maneuvering downward, seizing control or hiding.

[0153] S3.1.2)_Construct the action space based on the tasks of the war game objectives.

[0154] In this type of task, the multi-agent team first selects a subset from the set of all surviving enemy game objects and then performs a task related to that subset of game objects. Such tasks usually involve moving to a grid at a suitable distance from the chess piece and then performing some discrete actions, such as: stop, shoot, or hide.

[0155] For the war game simulation system platform in this embodiment, at each step, for the agent ∈I, its action is in the optional task set G i Select a task from i ∈G i Among them, the task is generally represented by g <c g ,x g >, which includes the specific type number of the task c g And the target position x involved in the task g ; Optional task set G = {g (1) ,g (1) ,...,g (K)} includes K combat tasks allowed by environmental rules, the specific content of which depends on observation information, such as Figure 4 shown.

[0156] In this embodiment, the action space is constructed: [mobi_up, mobi_left_up, mobi_right_up, dir_aim, anti_dir_aim, conceal]

[0157] S3.2)_ Taking the main battlefield situation data in step S2 as input, combined with the action space, the lower-level strategy network generates a set of feasible action probabilities;

[0158] The set of possible action probabilities can be obtained through the policy function The lower-level agent uses the observed state s as the input of the policy function π to obtain the probability of taking action a in the current state, which is a probability density function. It describes the possibility of taking action a in the current state s.

[0159] According to the definition of the strategy function, this function can be used as the basis for the intelligent agent to make command action decisions under the current battlefield status.

[0160] S3.2 specifically includes:

[0161] S3.2.1)_Use the policy network π(a|s;θ) to approximate the policy function π, giving the executable command decision a under the current battlefield state s;

[0162] S3.2.2)_ Update the network parameters θ by gradient ascent in the policy gradient algorithm;

[0163] S3.2.3)_The battlefield situation perceived by the lower-level intelligent agent based on the situation matrix and the predicted results of the target behavior are taken as input, and the decision action probability set of each intelligent agent is generated through the strategy network.

[0164] In this embodiment, the battlefield state s is input, and the strategy network outputs the decision action probability set of all current wargame agents. The decision actions in this embodiment include: maneuvering upward (mobi_up), maneuvering to the upper left (mobi_left_up), maneuvering to the upper right (mobi_right_up), direct aiming (dir_aim), indirect aiming (anti_dir_aim), and concealment (conceal), as shown in the following table.

[0165] Agent decision action probability set table

[0166]

[0167] S3.3)_Build action-value function Q π , used to improve the policy network π(a|s;θ).

[0168] The action-value function is defined as:

[0169]

[0170] Describes the agent's choice of action a when it observes the environment state s at time t t The result is wise, which is the discounted return U t About t ,a t ) expectations.

[0171] Based on the discount return U t The agent cannot observe the randomness of the action immediately at time t, so the expectation is introduced to estimate the benefit of the action value. Finally, the action-value function Q at time t is π Only with s t ,a t related.

[0172] S3.3 is as follows:

[0173] S3.3.1)_Build a value network q(s,a;w) to approximate the action-value function Q π , judge the quality of executable command decision a, and use it as an evaluation space to improve the strategy network;

[0174] S3.3.2)_Update the network parameters w by gradient descent in the temporal difference (TD) algorithm.

[0175] In particular, in this embodiment, the action decision of the agent is determined autonomously based on deep reinforcement learning. At each step, for the agent ∈I, the set of possible action probabilities d output by the neural network is i , d i It shows the probability of all executable actions of the current agent. And according to the evaluation space rules Get an action that is actually executed i .

[0176] For an agent ∈I, the evaluation space rules The significance of is that when the agent obtains the feasible action probability set d i , through the task g selected by the upper layer i Constrain and modify d i The probability of the action is d' i ,Right now Finally, we select action a with high probability. i And execute, that is

[0177] Step S4. Count the key battlefield data of the current combat scenario. According to the task requirements of the current round, count the key battlefield data of each war game agent in the scenario, such as survival index, deterrence index, artillery suppression index, combat effectiveness and other data, to provide comprehensive information support;

[0178] Step S4 specifically includes:

[0179] S4.1)_Build a battlefield element entity classification model;

[0180] S4.1 specifically includes:

[0181] S4.1.1)_Build CNN model architecture;

[0182] The CNN model consists of two convolutional layers, two pooling layers, a fully connected layer and a Softmax regression layer. The Adam first-order gradient-based stochastic objective function algorithm is used to optimize the entire neural network.

[0183] S4.1.2)_weight initialization;

[0184] In order to create this model, a large number of weights and bias terms need to be created. When initializing the weights in this model, a small amount of data noise needs to be added to break the symmetry and avoid zero gradients to improve the accuracy of battlefield element entity classification. The model uses the ReLU activation function and initializes the bias term with a small positive number to avoid the problem of the output of the neuron node being always 0. The ReLU activation function is defined as:

[0185] f(x)=max(0,x)

[0186] Among them, x is the input neuron.

[0187] S4.1.3)_Convolution and pooling.

[0188] The task of the convolution layer is to select features from different angles of the feature data of battlefield elements in the previous layer. After obtaining the convolution features, the convolution features are divided into non-overlapping areas, and then the maximum features of these areas are used to obtain the pooled convolution features. The first layer is completed by a convolution followed by a Maxpooling. The convolution obtains 32 features in each 5×5 convolution kernel, and there is a corresponding offset for each output channel. Similarly, in the second layer, each 5×5 convolution kernel obtains 64 features.

[0189] S4.2)_Build a fully connected layer;

[0190] A fully connected layer with 1024 neurons is constructed to process the entire war game battlefield environment data. At the same time, the vector output by the pooling layer is converted into a one-dimensional vector, multiplied by the weight matrix, added with the bias, and the RELU activation function is used.

[0191] S4.3)_Build the output layer;

[0192] In order to reduce overfitting of battlefield element data classification, adding Dropout before the output layer can effectively improve the generalization ability of large neural networks. Finally, add the Softmax layer. In Softmax regression, it is assumed that the class label y can take k different values. For a given test input x, assume that the function p(y=j|x:w) is the probability value of x belonging to category j:

[0193]

[0194] Among them, w is the parameter of the model.

[0195] S4.4)_Extract and classify the battlefield map information and war game information;

[0196] Furthermore, the specific implementation steps of step S4.4 are as follows:

[0197] S4.4.1)_The battlefield map information and war game information are input into the model in matrix form, and the convolution layer is used for feature learning of battlefield elements;

[0198] S4.4.2)_Input the battlefield element features output in step S4.4.1 into the fully connected layer for classification to obtain the key battlefield element entity data.

[0199] S4.5)_The battlefield element feature classification results output by step S4.4 are counted as key battlefield data in the current combat scenario.

[0200] In this embodiment, the key battlefield data counted for the current combat scenario are:

[0201] Survival index (survival_index), deterrence index (deterrence_index), artillery suppression index (artillery_suppression_index), combat effectiveness (combat_effectiveness).

[0202] Step S5. Weighting and integrating the action probability outputted from step S3 and the key battlefield data counted from step S4, and classifying them according to the agent number;

[0203] Step S5 specifically includes:

[0204] S5.1) Determine the weighting details;

[0205] Through methods such as military expert knowledge base evaluation and war game historical data analysis, the relative importance of the action probability output in step S3 to key battlefield data (such as survival index, deterrence index, artillery suppression index, combat effectiveness, etc.) in the current scenario is determined and presented in the form of a weighted ratio.

[0206] S5.2)_Verify and adjust weight distribution;

[0207] Verify and fine-tune the allocation of battlefield key data weights in step S5.1 to ensure that the weight ratio of each battlefield key data meets the needs of the current scenario and can effectively support the action decision-making process in different situations.

[0208] S5.3)_Weighted integration of the action probability output in step S3 and the key battlefield data counted in step S4;

[0209] Each battlefield key data is multiplied by its corresponding weight ratio, and all processed battlefield key data are added up as the comprehensive score of the action probability, ensuring that the contribution of different factors to the final decision-making action is comprehensively considered.

[0210] S5.4)_Based on the actual combat situation in the current scenario, the probability of each action after the battlefield key data weight integration in step S5.3 is quantitatively arranged according to its comprehensive score;

[0211] S5.5)_Classify the output results of step S5.4 according to the agent number.

[0212] By identifying the unique number field of each agent, all action probabilities and their comprehensive scores quantified and arranged in step S5.4 are classified into the corresponding agent attribute fields, ensuring personalized decision support for each agent.

[0213] In this embodiment, the action probability set and battlefield key data counted in step S3 are integrated and classified according to the agent number, and the result is stored in the form of an overall JSON data set, as shown below.

[0214]

[0215] Step S6. Input the integrated data in step S5 as decision-making auxiliary information to the upper-level intelligent agent for judging and screening the decision-making actions, and then provide feedback to the lower-level intelligent agent to assist in generating the optimal decision-making action.

[0216] Step S6 specifically includes:

[0217] S6.1)_Build an upper-level knowledge reasoning agent model;

[0218] The knowledge-reasoning intelligent agent is an intelligent agent model that forms domain knowledge by manually extracting unstructured information such as combat rules, regulations, and combat commanders' experience, and makes reasoning decisions based on this.

[0219] S6.1 specifically includes:

[0220] S6.1.1)_Build a military prior knowledge base;

[0221] Based on unstructured information such as military combat rules and combat commanders' experience, data information in the field of war game simulation is obtained, and a military prior knowledge base is established and presented in the form of a knowledge graph.

[0222] S6.1.2)_Build an inference engine to output the overall task of the lower-level agent.

[0223] According to the military prior knowledge base established in step S6.1.1, the combat task set in the current combat scenario is classified to obtain the overall task of the current lower-level intelligent body, which is an attack, retreat, or defense combat task.

[0224] S6.2)_ Input auxiliary information for action evaluation and screening;

[0225] The action probability and corresponding comprehensive score of each agent in step S5 are input into the upper-level knowledge reasoning agent as decision-making auxiliary information, and the actions are evaluated and screened according to the overall task of the current agent in step S6.1.2, and the decision actions with higher comprehensive scores that meet the overall task are selected as the optimal decision action sequence.

[0226] S6.3)_Output the optimal decision action sequence for feedback;

[0227] According to the optimal decision action sequence screened in step S6.2, the action probabilities in the attribute fields of the lower-level data learning-driven intelligent agent are adjusted: the action probabilities outside the optimal decision action sequence are modified to 0, and evenly added to the action probabilities corresponding to the optimal decision action sequence to ensure that the sum of the action probabilities of the lower-level intelligent agent is 1.

[0228] In this embodiment, the agent generates the optimal decision action sequence according to the above auxiliary decision steps as follows:

[0229]

[0230] S6.4)_The optimal decision action probability sequence in step S6.3 is input into the decision model of the intelligent war game simulation system, and the policy network in the decision model outputs the optimal action decision of the current intelligent agent.

[0231] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0232] The serial numbers of the embodiments of the present invention described above are only for convenience of description and do not represent the advantages or disadvantages of the embodiments.

[0233] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0234] The above are only embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A war game agent assisted decision-making method based on situation awareness interaction is implemented in the following steps: S1. Obtain situational awareness data of the battlefield environment in which the war game agent is located; S2. Use the acquired situational awareness data as the input of the action decision module of the war game agent, analyze the impact of military factors on military operations, and screen out the main battlefield situation data parameters that affect the agent's decision-making actions in the current specific scenario; S3, constructing a decision action module, taking the main battlefield situation data parameters obtained by screening and processing in step S2 as input of the decision action module, and outputting all possible decision action probabilities; S4. Count the key battlefield data of the current combat scenario. According to the task requirements of the current round, count the key battlefield data of each wargame agent in the scenario, such as survival index, deterrence index, artillery suppression index, combat effectiveness and other data, to provide comprehensive information support; S5, weighting and integrating the action probability outputted in step S3 and the key battlefield data counted in step S4, and classifying them according to the agent number; S6. Input the weighted integrated data in step S5 as decision-making auxiliary information to the upper-level intelligent agent to judge and screen the decision-making actions, provide feedback to the lower-level intelligent agent, and generate the optimal decision-making action.

2. The method for assisting decision-making by a wargame agent based on situational awareness interaction according to claim 1, characterized in that: The step S1 specifically includes: S1.1)_Data collection: According to the characteristics of battlefield situation data in wargame simulation and the functional requirements of wargame agents, a large amount of data that meets the distribution characteristics of battlefield environment is generated based on small sample simulation data; S1.2)_Data mining: After collecting battlefield situation and environmental data, the battlefield situation element map is combined to obtain an incoherent set of data elements; combined with the relationship between battlefield situation elements, more coherent situation information is obtained, thereby obtaining accurate battlefield perception.

3. The method for assisting decision-making by a wargame agent based on situational awareness interaction according to claim 2 is characterized in that: The step S1.2 specifically includes: S1.2.1) Build a military application model library at the logic layer and a data mining algorithm library at the algorithm layer. S1.2.2) After the combined application of the logic layer and the algorithm layer, the battlefield situation environment feature data is processed through data mining to form multiple categories of battlefield situation element data, which serve as the situation awareness data of the battlefield environment in which the war game intelligent agent is located.

4. The method for assisting decision-making by a wargame agent based on situational awareness interaction according to claim 2, characterized in that: The step S2 specifically includes: S2.1) Data preprocessing: Use the original battlefield situation awareness data as input and perform preliminary screening and cleaning operations. S2.2) Label dataset construction: A labeled dataset is constructed from a series of formatted samples with positive and negative annotations obtained based on actual adversarial and custom rules. S2.3) Construction of situation feature data set: Based on the label data set, the basic situation feature data at the combat position is calculated and stored, the basic situation features are encoded and diversified feature fusion screening is performed on the basic situation features, and association is performed from multiple dimensions to form a structured situation feature data set.

5. The method for assisting decision-making by a wargame agent based on situational awareness interaction according to claim 4 is characterized in that: The step S3 specifically includes: S3.1) Constructing the action space: Construct an action space, including upper-level macro tasks based on military prior knowledge and lower-level intelligent agent behaviors based on deep reinforcement learning. Both upper and lower-level actions are in discrete form. S3.2) Generate a set of feasible action probabilities based on the underlying policy network. S3.3) Constructing the action-value function Q π , used to improve the policy network π(a|s;θ).

6. The method for assisting decision-making by a war game agent based on situation awareness interaction according to claim 5 is characterized in that: The step S3.1 specifically includes: S3.1.1) Constructing action space based on the task of hexagonal grid target; In this type of task, the multi-agent team first selects some hexagonal grids from the set of available hexagonal grid targets, and then performs tasks related to the hexagonal grids. Such tasks usually refer to maneuvering to the corresponding hexagonal grid and then performing some discrete operations, such as: stopping, maneuvering upward, maneuvering downward, seizing control or hiding. S3.1.2) Constructing action space based on the tasks of the war game objectives; In this type of task, the multi-agent team first selects a subset from the set of all surviving enemy game objects and then performs a task related to the subset game. This task usually means moving to a grid at a suitable distance from the chess piece and then performing some discrete actions.

7. The method for assisting decision-making by a wargame agent based on situational awareness interaction according to claim 6, characterized in that: The step S3.2 specifically includes: S3.2.1) Use the policy network π(a|s; θ) to approximate the policy function π and give the executable command decision a under the current battlefield state s. S3.2.2) Update the network parameters θ using the gradient ascent method in the policy gradient algorithm. S3.2.3) The battlefield situation perceived by the lower-level intelligent agent based on the situation matrix and the predicted results of the target behavior are used as input, and the decision action probability set of each intelligent agent is generated through the strategy network.

8. The method for assisting decision-making by a war game agent based on situational awareness interaction according to claim 7 is characterized in that: The step S3.3 specifically includes: S3.3.1) Construct a value network q(s,a;w) to approximate the action-value function Q π , judge the quality of executable command decision a, and use it as an evaluation space to improve the strategy network. S3.3.2) Update the network parameters w by gradient descent in the temporal difference (TD) algorithm.

9. The method for assisting decision-making by a war game agent based on situation awareness interaction according to claim 7, characterized in that: The step S4 specifically includes: S4.1) constructing a battlefield element entity classification model; S4.2) Build a fully connected layer: S4.3) Construct the output layer: S4.4) Extract and classify features of battlefield map information and war game information.

Citation Information

Cited By

  • Intelligent agent situation information enhancement method and system based on knowledge reasoning

    CN119416815A

  • An Agent Situation Information Enhancement Method and System Based on Knowledge Reasoning

    CN119416815B

  • Parent-child interaction decision information generation method and device, equipment and storage medium

    CN120525064A

  • Wargame deduction strategy determination method and related product

    CN121009998A

  • A method for determining a wargaming strategy and related products

    CN121009998B