An effectiveness evaluation method based on modeling and analysis of massive adversarial simulation data

Abstract

The present invention proposes an effectiveness evaluation method based on the modeling and analysis of massive confrontation simulation deduction data. The method adopts orthogonal experimental design means, and determines the experimental factors and level elements through methods such as data dimensioning, constructs a decision-making evaluation scenario that meets the "five-more" requirements of "high information incompleteness", "multiple force deployment strategies for both sides", "multiple combat command and control strategies", "multiple battlefield environment elements", and "multiple enemy and friendly weapon equipment performance elements". An indicator weight matrix and an evaluation objective function are established. Through a multi-layer intelligent perception machine and membership optimization, the correlation between indicator parameters and decision effects is determined, and an optimized distribution result of the indicator weight is generated. The method adopts big data and knowledge fusion methods to comprehensively evaluate the massive game deduction results, improve the objectivity and accuracy of equipment effectiveness evaluation, and simultaneously construct a mapping relationship between interval parameters related to the "five-more" and corresponding indicators to form a matrix evaluation result query table, thereby improving the comprehensiveness of equipment effectiveness evaluation.

Description

Technical Field

[0001] The present invention relates to an effectiveness evaluation method based on modeling and analysis of massive confrontation simulation deduction data, belonging to the field of aerospace equipment effectiveness evaluation. Background Art

[0002] Equipment model capability assessments primarily focus on traditional aspects such as system contribution, system maturity, and system satisfaction. For system contribution assessments, experimental assessment methods, simulations, technology assessment methods based on scientific advisory panels, and quantitative technology assessments are primarily used. For technology system maturity assessments, methods based on technology readiness (TRL), integration readiness level (IRL), and system readiness level (SRL) are primarily used. System satisfaction assessments primarily assess whether the equipment system meets various capability and equipment requirements, as well as the degree of satisfaction with these requirements. A comprehensive assessment and analysis method combining qualitative and quantitative methods is primarily employed.

[0003] Traditional equipment model evaluation methods only use a small number of fixed scenario evaluations, mainly under the premise of fixed confrontation scenarios, fixed confrontation environments, transparent situation of both sides, determined equipment capability boundaries, and specified red-blue game confrontation strategies. In general, equipment effectiveness evaluation is carried out under the conditions of a single environment, single location deployment and quantity configuration, single equipment capability, single information completeness, and single strategy. However, the highly dynamic, uncertain, and complex scenarios in red-blue confrontations bring huge challenges to traditional equipment capability evaluation methods for static and deterministic scenarios. The main challenges are: incomplete factor information in red-blue confrontation scenarios, and it is difficult to fully model and analyze key factors, and traditional modeling analysis and evaluation methods are difficult to adapt; the environment in red-blue confrontation scenarios changes dynamically, and static analysis methods are difficult to apply; the strategies of both sides are uncertain, and traditional fixed-process simulation and deduction evaluation methods cannot cover strategy diversity; changes in equipment capabilities and different cognitions result in great differences in the results of different evaluation methods, and the evaluation results are not convincing. Summary of the Invention

[0004] The technical problem solved by the present invention is: to overcome the shortcomings of the existing technology and provide a comprehensive effectiveness evaluation method based on multiple scenarios. This method relies on a game deduction system and, in multiple scenarios such as incomplete blue party information and uncertain deployment strategies, uses a matrix evaluation method based on massive deduction data to solve the problems of traditional equipment model effectiveness evaluation being insufficiently comprehensive, accurate, and objective, providing simulation modeling designers and military users with a friendlier and more effective evaluation method.

[0005] The technical solution of the present invention is: a performance evaluation method based on modeling and analysis of massive adversarial simulation data, the steps of which are as follows:

[0006] 1) Establish an evaluation index system for equipment capabilities and design an evaluation test process;

[0007] 2) Design decision-making and evaluation scenarios for the "five more" needs through orthogonal experiments, and edit scenario files using a drag-and-drop scenario editing tool;

[0008] 3) Based on scenario assumptions, configure corresponding equipment models and decision-making agents, and generate massive amounts of simulation data through the simulation platform;

[0009] 4) Based on data statistical tools and the generated deduction data, a mapping relationship between the interval parameters related to the "five mores" and the corresponding indicators is constructed to form a matrix evaluation result query table.

[0010] In step 1), when constructing the evaluation index system, the target decomposition method is used based on the principles of purpose, integrity, and measurability. The specific steps are as follows:

[0011] 11) Determine the comprehensive effectiveness indicators that affect equipment capabilities based on the equipment's mission and tasks;

[0012] 12) Analyze the main factors affecting each comprehensive performance indicator and determine the individual performance indicators;

[0013] 13) Decompose each individual performance indicator until the lowest performance parameter of the system, i.e. the basic indicator.

[0014] The capability assessment test process described in step 1) is designed using the orthogonal experimental design method. The entire capability assessment test process includes six steps: parameter preprocessing, important factor screening, initial experimental design, sequential experimental design, prediction model construction and prediction error analysis, and decision maker sensitive factor analysis.

[0015] In step 2), the decision-making evaluation scenarios of the "five more" requirements are five decision-making evaluation uncertainty factor scenarios: uncertainty of incomplete information, uncertainty of the deployment strategy of the enemy and our forces, uncertainty of the combat command and control strategy, uncertainty of the environmental factors designed in the deduction scenario, and uncertainty of the enemy and our forces.

[0016] In step 2), the draggable scene editing tool refers to a method of quickly constructing scene scenarios by dragging and dropping, including the number of equipment, equipment positions, equipment parameters, map size, scene constraints, winning and losing conditions, and character design.

[0017] In step 3), the simulation deduction platform includes a user configuration module, a simulation engine module, an intelligent drive module, an interactive display module and a data management module.

[0018] Parameter splitting, parameter merging, and indicative function representation methods were used for parameter preprocessing; the uniform design method under the deviation criterion was used for important factor screening, initial experimental design, and sequential experimental design; nonparametric estimation methods and polynomial regression models were used for predictive model construction; cross-validation methods and mean square error models were used for predictive model construction and prediction error analysis; and the Bonferroni simultaneous test method was used for decision maker sensitive factor analysis.

[0019] The uncertainty of information incompleteness, the uncertainty of the force deployment strategy of both sides, the uncertainty of combat command and control strategy, the uncertainty of the environmental factors designed in the simulation scenario, and the uncertainty of the performance factors of the equipment of both sides are specifically as follows:

[0020] a) Information incompleteness refers to the presence of the fog of war during the game, where the opponent's information is partially detectable. The focus is on the proportion of invisible defense unit elements of the opponent. By setting a ratio and randomly changing the number of invisible elements in different scenarios within a certain constraint, the degree of information incompleteness can be adjusted.

[0021] b) Deployment strategy uncertainty refers to the uncertainty in the number and location of the opponent's defense units, making it difficult for us to know the exact deployment strategy of the opponent. Based on certain knowledge constraints, the uncertainty of the deployment strategy can be adjusted by randomly changing the number and location of the opponent's defense units.

[0022] c) Uncertainty in the charge strategy refers to uncertainty in the opponent’s defensive strategy, defensive formation, and defensive needs;

[0023] d) Environmental uncertainty refers to the complexity of the adversary's interference environment and the natural environment. Scenario design must consider the impact of different interference environments on detection and command equipment, and construct a countermeasure simulation environment that interferes with friendly detection. Furthermore, the impact of the Earth model on the simulation dynamics model must also be considered.

[0024] e) Uncertainty in equipment performance factors means that the opponent's equipment exhibits different performance in different scenarios, and there are many factors that affect equipment performance.

[0025] The data management module in the simulation and deduction platform is capable of recording and replaying massive game deduction data, supporting data collection during the deduction process and storage management of scenes and equipment libraries, and providing data selection, playback, and pause functions.

[0026] The simulation engine module in the simulation deduction platform adopts a multi-mode, variable-step simulation confrontation advancement mechanism to support instant acceleration and deceleration during the confrontation game between the two sides.

[0027] The beneficial effects of the present invention compared with the prior art are:

[0028] The present invention proposes an effectiveness evaluation method based on the modeling and analysis of massive confrontation simulation and deduction data. In view of the problem that the traditional equipment model evaluation method only adopts a small number of fixed scenarios for evaluation, resulting in non-objective, inaccurate and incomplete results, the orthogonal experimental design method is adopted to determine the experimental factors and level elements, and construct a decision-making evaluation scenario that meets the "five more" requirements of "more information incompleteness", "more force deployment strategies for both sides", "more combat command and control strategies", "more battlefield environment elements" and "more enemy and friendly weapon equipment performance elements". Relying on the game deduction platform, a matrix evaluation method for massive data is developed to form an evaluation result query table, which realizes the rapid, effective, objective and comprehensive evaluation of the equipment model, provides a comprehensive set of effectiveness analysis and evaluation means for equipment model effectiveness evaluation, promotes the deep integration of simulation deduction and simulation effectiveness evaluation, and facilitates users to conduct a comprehensive evaluation of simulation effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flowchart for equipment capability deduction and assessment;

[0030] Figure 2 To deduce the data storage flow chart;

[0031] Figure 3 This is a diagram of the equipment capability deduction and evaluation platform;

[0032] Figure 4 This is a flowchart for the deduction and evaluation of the deduction platform. DETAILED DESCRIPTION

[0033] The present invention provides an effectiveness evaluation method based on the modeling and analysis of massive adversarial simulation and deduction data, forming a process for comprehensive equipment effectiveness evaluation. Through orthogonal experimental design of "five more" scenarios, massive deduction data is generated based on the simulation and deduction platform, and a mapping table of interval parameters and corresponding indicators is constructed to achieve comprehensive evaluation of equipment effectiveness.

[0034] In terms of specific implementation methods, an effectiveness evaluation method based on massive confrontation simulation data modeling and analysis is divided into four parts: test design verification, capability evaluation method, combat scenario design, and core support platform. Figure 1 shown.

[0035] This method takes the game simulation platform as the core. Through the rapid integration of functional-level equipment and the accelerated simulation engine, it edits and designs diverse game confrontation scenarios, covers the randomness and uncertainty of equipment performance and confrontation process, constructs a reasonable equipment capability evaluation index system, designs evaluation test processes, generates a large amount of data through rapid and massive game simulation, conducts review data mining and statistical analysis, and realizes the effectiveness evaluation of equipment.

[0036] Specifically, the steps of the present invention are as follows:

[0037] (1) Experimental design phase. This phase focuses on evaluating the effectiveness of equipment in confrontation scenarios. Equipment effectiveness index systems are usually affected by parameters such as the blue team’s deployment location, blue team strategy, and information completeness. Therefore, it is necessary to use massive game simulations with multiple scenarios, multiple strategies, and multiple information to obtain the values ​​of various indicators under different scenarios, strategies, and information completeness conditions.

[0038] Based on the set typical scenarios and the knowledge-driven Blue Army, the scenario parameters are set and a massive number of simulations are carried out using the game simulation platform. 1,000 to 10,000 simulations are conducted under each scenario. In each simulation, various indicators of equipment effectiveness and other data are recorded, and the process data of the simulation is stored.

[0039] The evaluation and deduction test factors, levels and experimental design are shown in the following table:

[0040] Table 1 Experimental factors and their levels

[0041] Serial number factor Level 1 Level 2 Level 3 1. Information completeness 50％ 75％ 100％ 2. Blue Strategy 1 2 3 3. ...... ...... ...... ......

[0042] Table 2 Experimental scenario design

[0043]

[0044]

[0045] (2) Capability assessment stage. The highly dynamic, uncertain, and complex scenarios in the red-blue confrontation pose huge challenges to the traditional equipment capability assessment methods for static and deterministic scenarios. The main challenges are: incomplete information on elements in the red-blue confrontation scenario, making it difficult to fully model and analyze key elements, and traditional modeling analysis and assessment methods are difficult to adapt; the dynamic environment in the red-blue confrontation scenario is large, and static analysis methods are difficult to apply; the strategies of both sides are uncertain, and the traditional fixed-process simulation and deduction assessment method cannot cover the diversity of strategies; the changes in equipment capabilities and different cognitions result in great differences in the results of different assessment methods, and the assessment results are not convincing.

[0046] In response to the above challenges and problems, during the capability assessment phase, a game simulation architecture for equipment capabilities is designed to form a simulation system with massive game simulation capabilities. Based on the constructed equipment capability assessment indicator system, a rapid game simulation method is adopted to cover the diversity of incomplete, uncertain, and highly dynamic confrontation scenarios through massive simulations, forming a comprehensive and objective assessment capability. Based on intelligent game simulations at the million-level or above, it can cover major scenario assumptions, combat elements, and dynamic changes in blue force strategies. The test coverage of capability verification tests significantly exceeds that of traditional assessment methods.

[0047] (2.1) Construction of indicator system

[0048] Consider the ultimate purpose of the assessment, clarify the requirements for capability assessment, and, based on the principles of purposefulness, integrity, and measurability, establish a mapping relationship between equipment capabilities and the various factors that influence their performance. Utilize the target decomposition method to construct an evaluation indicator system. Fully considering the causal and subordinate relationships between indicators, decompose them step by step until the decomposed indicators meet measurable requirements, thereby forming a top-down indicator system hierarchy. Drawing on this approach, the steps for constructing an indicator system are as follows: Step 1: Based on the equipment's mission and tasks, determine the comprehensive effectiveness indicators that influence equipment capabilities. Step 2: Analyze the main factors influencing each comprehensive effectiveness indicator and determine individual effectiveness indicators. Step 3: Decompose each individual effectiveness indicator until the lowest-level performance parameters of the system, namely the foundational indicators.

[0049] (2.2) Massive game deduction

[0050] For the red-blue confrontation scenario, a game deduction architecture based on the "evolutionary game theory" is formed to form a "game evolution environment" simulation system with the ability to deduce games of massive types of equipment, to meet the deduction analysis needs and equipment performance requirements of the deduction results analysis. This technology mainly considers the requirements of scenario design and adopts simulation engine technology with adjustable granularity and equipment model architecture technology with flexibly defined styles to achieve massive game deduction.

[0051] (2.3) Data review and analysis

[0052] In view of the massive amount of game simulations conducted on the simulation platform and the accompanying generation of a large amount of game data, only on the basis of effective recording and storage of human-human, human-machine, and machine-machine confrontation data can the feature analysis and efficient review of the data be carried out. In the simulation simulation platform, the recording and playback of simulation data are generally realized through the data storage and reading module, which supports data collection during the simulation process and storage management of scenes and equipment libraries, and provides functions such as data selection, playback, and pause. The process of realizing the storage / playback function of simulation data is as follows: ① Classify the simulation data and determine the storage format of the process data; ② Receive the process data of the battle or simulation and record it in the memory, and write it to the specified file; ③ Read the simulation process data from the specified file and send the data to the interface. The data storage process of the simulation platform is as follows: Figure 2 shown.

[0053] In order to facilitate data management, a data storage management class attribute table is established, as shown in the following table.

[0054] Table 3 Data management attribute table

[0055]

[0056]

[0057] (3) Confrontation scenario design stage. Traditional equipment effectiveness evaluation is mainly conducted under the premise of a fixed confrontation scenario, a fixed confrontation environment, a transparent situation between the two sides, a determined equipment capability boundary, and a specified red-blue game confrontation strategy. In general, equipment effectiveness evaluation is conducted under the conditions of a single environment, a single position deployment and quantity configuration, a single equipment capability, a single information completeness, and a single strategy. However, the current confrontation scenario is faced with "many" information incompleteness, "many" force deployment strategies on both sides, "many" command and control strategies, "many" environmental factors, and "many" equipment performance factors on both sides. To achieve a comprehensive evaluation of equipment effectiveness, it is first necessary to analyze the "five-many" characteristics and construct a confrontation scenario with the "five-many" characteristics through a random deviation method based on knowledge constraints, which will serve as the basis for equipment capability evaluation methods and experimental verification.

[0058] (3.1) Incomplete information

[0059] Incomplete information refers to the fog of war that exists during offensive and defensive confrontations, meaning that the enemy's information is partially detectable. When designing incomplete information for confrontation scenarios, the primary consideration is the proportion of defense unit elements that are invisible to the opponent. By setting this proportion and randomly varying the number of invisible elements in different scenarios within certain constraints, the degree of incomplete information can be adjusted.

[0060] (3.2) Deployment strategy uncertainty

[0061] Deployment strategy uncertainty refers to the uncertainty in the number and location of the opponent's defense units, making it difficult for us to know the exact deployment strategy of the opponent. Based on certain knowledge constraints, the deployment strategy uncertainty can be adjusted by randomly changing the number and location of the opponent's defense units.

[0062] (3.3) Strategy uncertainty

[0063] Uncertainty in the charging strategy refers to uncertainty in the opponent’s defensive strategy, uncertainty in the defensive formation, and uncertainty in the main needs of the defense.

[0064] (3.4) Complex environmental factors

[0065] The complexity of the adversarial environment primarily refers to the complexity of the adversary's interference environment and the natural environment. Scenario design considers the impact of different interference environments on detection and command equipment, creating a simulated adversarial environment designed to disrupt friendly detection. Furthermore, the impact of the Earth model on the simulation dynamics model must be considered.

[0066] (3.5) Equipment performance factors are numerous

[0067] The multiple performance factors of the opponent's equipment mean that the opponent's equipment exhibits different performance in different scenarios, and there are many factors that affect the performance of the equipment.

[0068] (4) Core support platform.

[0069] To meet the needs of equipment assessment and incorporate "five-more" combat scenarios, we research simulation advancement mechanisms and application environment technologies suitable for multi-agent joint confrontations. We employ visual editing tools and randomized configuration to manage models of both sides in the confrontation. We build a scalable, flexible, and fast human-computer interaction platform, enabling capabilities such as confrontation process advancement, confrontation progress management, model management, event and message management, data recording and acquisition, and a human-computer interaction interface. Based on an asymmetric self-playing architecture, we design a value-based model for assessing the power of both sides, a task / behavior decision-making model for incomplete information, and the use of massive deduction and self-play methods. Through data / scenario-driven deduction and evaluation, we provide a platform tool for effectively and accurately assessing the system capabilities and effectiveness of equipment.

[0070] The equipment capability simulation and evaluation platform has the ability to simulate multiple deployments, multiple information, multiple strategies, multiple capabilities, and multiple environments. That is, the platform supports massive game simulations in multiple scenarios, massive game simulations with adjustable information incompleteness, multiple red AI decision-making algorithms and intelligent blue forces, and parameter intervals for blue equipment models. It also supports environmental diversity settings such as electromagnetic and different lighting. Through massive game simulations, the platform covers the uncertainty of the situation, the uncertainty of equipment performance, the uncertainty of strategies, the diversity of the environment, and the diversity of information to improve the accuracy and confidence of performance evaluation. The platform consists of a user configuration module, a simulation engine module, an intelligent drive module, an interactive display module, and a data management module. Figure 3 shown.

[0071] The equipment capability deduction and evaluation process is generally divided into three stages: initial preparation stage, simulation deduction stage and simulation analysis stage. The process is as follows: Figure 4 As shown in the figure, during the initial preparation phase, users configure scenario scenarios and their respective equipment models through configuration files (including scenario maps, combat environments, force distribution, capabilities, etc.). This configuration information is then transmitted to the simulation via data communication. During the simulation phase, the simulation begins based on the pre-set scenario information. During the simulation, situational information is displayed in real time and transmitted to the AI. The AI ​​then makes decisions based on the current situational information and controls the equipment models. During the simulation analysis phase, the AI's decision-making effectiveness is evaluated by analyzing the simulation data.

[0072] This project innovatively proposes an effectiveness assessment method based on the modeling and analysis of massive adversarial simulation data. This method addresses the objectivity, incompleteness, and inaccuracy inherent in traditional equipment capability assessment methods targeting static, deterministic scenarios, as they are subject to the highly dynamic, uncertain, and complex scenarios encountered in red-blue confrontations. The "five-plus" scenarios designed using orthogonal methods provide a more comprehensive and focused coverage of the key elements required for equipment assessment. Based on the principles of purposefulness, integrity, and measurability, the project employs a target decomposition method to construct an assessment indicator system. Relying on an equipment capability simulation and assessment platform, the project generates massive simulation data, constructs a mapping relationship between interval parameters and assessment indicator results, and forms a matrix-like mapping table, enhancing the comprehensiveness of model assessments.

[0073] The contents not described in detail in the specification of the present invention belong to the common knowledge of professionals in this field.

Claims

1. A performance evaluation method based on massive adversarial simulation data modeling and analysis, characterized by Here are the steps: 1) Establish an evaluation index system for equipment capabilities and design an evaluation test process; 2) Design decision-making and evaluation scenarios for the "Five More" needs through orthogonal experiments, and create scenario files using a drag-and-drop scenario editing tool; 3) Based on scenario assumptions, configure corresponding equipment models and decision-making agents, and generate massive amounts of simulation data through the simulation platform; 4) Based on statistical tools and the generated deduction data, a mapping relationship between interval parameters related to the "Five Mores" and corresponding indicators is constructed to form a matrix-style evaluation result query table; In step 1), when constructing the evaluation index system, the target decomposition method is used based on the principles of purpose, integrity, and measurability. The specific steps are as follows: 11) Determine the comprehensive effectiveness indicators that affect equipment capabilities based on the equipment's mission and tasks; 12) Analyze the main factors affecting each comprehensive performance indicator and determine the individual performance indicators; 13) Decompose each individual performance indicator until the lowest performance parameter of the system, i.e., the basic indicator; The capability evaluation test process in step 1) is designed using the orthogonal experimental design method. The entire capability evaluation test process includes six steps: parameter preprocessing, important factor screening, initial experimental design, sequential experimental design, prediction model construction and prediction error analysis, and decision maker sensitive factor analysis; In step 2), the decision-making evaluation scenarios for the "five more" requirements are five decision-making evaluation uncertainty factor scenarios: uncertainty in incomplete information, uncertainty in the deployment strategies of both enemy and friendly forces, uncertainty in combat command and control strategies, uncertainty in environmental factors designed in the simulation scenario, and uncertainty in the forces of both enemy and friendly forces. In step 2), the draggable scenario editing tool refers to a method for quickly constructing scenario scenarios by dragging and dropping, including the number of equipment, equipment positions, equipment parameters, map size, scenario constraints, win and loss conditions, and character design; In step 3), the simulation deduction platform includes a user configuration module, a simulation engine module, an intelligent drive module, an interactive display module and a data management module; Parameter preprocessing uses parameter splitting, parameter merging, and characteristic function representation methods; The uniform design method under the deviation criterion was used for important factor screening, initial experimental design, and sequential experimental design; nonparametric estimation methods and polynomial regression models were used for predictive model construction; cross-validation methods and mean square error models were used for predictive model construction and prediction error analysis; and the Bonferroni simultaneous test method was used for decision maker sensitive factor analysis. The uncertainty of information incompleteness, the uncertainty of the force deployment strategy of both sides, the uncertainty of combat command and control strategy, the uncertainty of the environmental factors designed in the simulation scenario, and the uncertainty of the performance factors of the equipment of both sides are specifically as follows: a) Information incompleteness refers to the presence of the fog of war during the game, where the opponent's information is partially detectable. The focus is on the proportion of invisible defense unit elements of the opponent. By setting a ratio and randomly changing the number of invisible elements in different scenarios within a certain constraint, the degree of information incompleteness can be adjusted. b) Deployment strategy uncertainty refers to the uncertainty in the number and location of the opponent's defense units, making it difficult for us to know the exact deployment strategy of the opponent. Based on certain knowledge constraints, the uncertainty of the deployment strategy can be adjusted by randomly changing the number and location of the opponent's defense units. c) Uncertainty in the charge strategy refers to uncertainty in the opponent’s defensive strategy, defensive formation, and defensive needs; d) Environmental uncertainty refers to the complexity of the adversary's interference environment and the natural environment. Scenario design must consider the impact of different interference environments on detection and command equipment, and construct a countermeasure simulation environment that interferes with friendly detection. Furthermore, the impact of the Earth model on the simulation dynamics model must also be considered. e) Uncertainty in equipment performance factors refers to the fact that the opponent's equipment exhibits different performance in different scenarios, and there are many factors that affect equipment performance; The data management module in the simulation platform is capable of recording and replaying massive amounts of game simulation data, supporting data collection during the simulation process and storage management of scenarios and equipment libraries, and providing data selection, playback, and pause functions. The simulation engine module in the simulation deduction platform adopts a multi-mode, variable-step simulation confrontation advancement mechanism to support instant acceleration and deceleration during the confrontation game between the two sides.

Images