Intelligent decision-making test method and test system based on reinforcement learning

Abstract

The invention provides an intelligent decision test method and test system based on reinforcement learning, and the method comprises the steps: defining scene test case parameters, defining difficulty levels of different scenes, and forming a scene parameter space; sampling from the scene parameter space to obtain a group of scene parameters, and converting the scene parameters obtained by current sampling into a scenario through a reinforcement learning model; the intelligent decision-making model is used for carrying out loop simulation test according to the loaded scenario, and the intelligent decision-making level of the intelligent decision-making model is evaluated; and calculating the confidence coefficient of the reinforcement learning model according to the scene parameters obtained by sampling and an intelligent decision level evaluation result, judging whether the confidence coefficient reaches a preset requirement, if not, repeating sampling and simulation testing, and if so, outputting an evaluation conclusion for the intelligent decision model. By applying the technical scheme of the invention, the technical problem of lack of an intelligent decision model decision level evaluation method in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent battlefield testing, and in particular to an intelligent decision-making testing method and a testing system based on reinforcement learning. Background Art

[0002] With the rapid development of global military technology, especially the deep integration of military intelligence technology and advanced information network technology, traditional combat modes are undergoing unprecedented changes. The traditional combat mode of large-scale concentrated forces and firepower coverage can no longer meet the urgent needs of modern warfare for rapid response, precision strikes, and efficient coordination. In this context, the miniaturization, decentralization, unmanned and intelligentization of weapon platforms have become the mainstream trend of military development in various countries.

[0003] At present, intelligent and autonomous systems, as the core components of emerging combat forces, are gradually penetrating into all aspects of war, such as reconnaissance, surveillance, attack, and assessment. These systems can use advanced sensor technology, artificial intelligence algorithms, big data analysis and other means to achieve comprehensive perception and accurate analysis of the battlefield environment, and then make autonomous decisions and respond flexibly, greatly enhancing the flexibility and adaptability of operations.

[0004] However, the tight coupling and fixed formation mode between combat units in the traditional combat style can no longer adapt to the highly decentralized and flexible combat requirements. Therefore, the emergence of new combat styles has become inevitable, such as manned / unmanned collaborative combat, unmanned cluster autonomous combat, etc. They use a large number of decentralized, heterogeneous, and fast and flexible combat units to dynamically build combat networks according to battlefield situations and command requirements, forming a distributed killing network with diverse, adaptive, highly flexible, and highly resistant characteristics, and achieving all-round and multi-level strikes on the enemy. However, there is currently no method to test and evaluate the decision-making level of intelligent decision-making models in this new combat style. Summary of the invention

[0005] The present invention provides an intelligent decision-making testing method and testing system based on reinforcement learning, which can solve the technical problem of the lack of decision-making level evaluation method of intelligent decision-making models in the prior art.

[0006] According to one aspect of the present invention, there is provided an intelligent decision-making test method based on reinforcement learning, the method comprising:

[0007] S1, define scenario test case parameters, and define the difficulty levels of different scenarios based on the scenario test case parameters to form a scenario parameter space. The scenario test case parameters include the number of red teams, the density of red team deployment, the balance of red team unit categories, the number of blue teams, the concentration of blue team attacks, the pre-war loss ratio of the red team, and the criticality of the pre-war loss units of the red team;

[0008] S2, build a reinforcement learning model based on the PPO algorithm and initialize it;

[0009] S3, sampling a set of scene parameters from the scene parameter space, and transforming the currently sampled scene parameters into a scenario through the reinforcement learning model;

[0010] S4, loading the assumption obtained in S3 in the simulation, using the intelligent decision-making model to perform a cyclic simulation test according to the loaded assumption, performing basic evaluation calculations on the decision-making performance of the intelligent decision-making model under the current assumption according to static indicators, dynamic indicators and adaptability indicators, and evaluating the intelligent decision-making level of the intelligent decision-making model according to the basic evaluation calculation results;

[0011] S5, calculating the confidence of the reinforcement learning model according to the scene parameters sampled in S3 and the intelligent decision-making level evaluation result obtained in S4, and judging whether the confidence meets the preset requirements, if not, go to S6, if yes, go to S7;

[0012] S6, repeat S3 to S5 until the confidence reaches the preset requirement;

[0013] S7, outputs the evaluation conclusion of the intelligent decision-making model.

[0014] Furthermore, static indicators include:

[0015] Q R％ =|V R | / |V|,

[0016]

[0017] Q var[θ] =-∑ k ∑ i p i,k[θ] ×log(p i,k[θ] ),

[0018]

[0019] Q F％ =|V F | / |V|,

[0020]

[0021] In the above formula, Q R％ Represents the detection scale, V R represents the set of detection nodes, V represents all nodes, Indicates the detection range, R i represents the i-th detection node, x i ,y i 、ri represents the abscissa, ordinate and radius of the node, Q var[θ] Indicates the degree of detection diversity, p i,k[θ] represents the probability distribution of the k-th dimension capability attribute in space θ, Q R∩％ Indicates detection coverage, Area D represents the defense area, ρ c Indicates the communication scale, N c represents the total number of communication nodes, N represents the total number of nodes, Q comm％ represents the communication coverage, V F represents the firepower node set, Q C,C represents the communication network cost, cost(V i ,V j ) represents the communication cost between any two nodes, Indicates the communication balance, V C represents the set of accused nodes, Q F％ Indicates the scale of firepower, Indicates the fire coverage range. Indicates firepower coverage, F i represents the i-th firepower node, Indicates the diversity of the firepower network, f t represents the critical threshold, <k>, <k 2 >represent the first-order moment and second-order moment of the network degree distribution respectively.

[0022] Furthermore, dynamic indicators include:

[0023]

[0024]

[0025] In the above formula, Q p (π) represents the interception success rate, m represents the number of test rounds, E represents the target set, E′ represents the intercepted target set, Q η (π) represents the interception effectiveness, N i Indicates the target quantity consumed, Q prior represents the interception priority efficiency, r(e i ) represents the interception target e i Priority scoring, l(e i |τ j ) represents the chain τ j For e i The interception distance, l(e i ) means e i The standard quantity, Q M(p(E),∈) represents the task matching degree, Q blc Indicates resource balance, r(R i ) represents the consumption ratio of each detection node, r(F i ) represents the consumption ratio of each firepower node, represents the communication efficiency, τ(e i ) represents the target e i The decision chain of G C Represents the communication network graph, d(G C ) represents G C The diameter, Q info represents the closure of the information chain, λ1 and λ2 represent weighting parameters, and b(e i |τ(e i )) represents the information chain τ(e i ) bandwidth requirement, d(e i |τ(e i )) represents the information chain τ(e i ) jump number, Q R％ Denotes the detection duty cycle, d ％ (e i |T(V,G)) represents the target e i The detectable path duty cycle, represents the detection communication cost, b C (e i |T(V,G)) represents the target e i The communication cost, D I Indicates the interception depth of architecture deployment, T M represents the target track, represents the i-th firepower killing zone, Q RD represents the target detection resource allocation capability, R D represents the detection resource set, E(t i ,r j ) represents the target t i Detected resources j Efficiency of identification and tracking, Q RF represents the target firepower resource allocation capability, R F represents the firepower resource set, E(t i ,f j ) represents the target t i Firepower resources j Interception efficiency, Q RG represents the target-guided resource allocation capability, R G represents the set of guiding resources, E(t i ,g j ) represents the target t i Guided resources j The efficiency of precise interception.

[0026] Further, the adaptability indicators include:

[0027] Q loss =1-P loss ,

[0028]

[0029] Rob rand =Q(T(V, G)) / Q(T(V′ rand , G′ rand )),

[0030] Rob key =Q(T(V, G)) / Q(T(V′ key , G′ key )),

[0031] In the above formula, Q loss Indicates the damage resistance rate, P loss represents the damage ratio, Q δ represents the self-healing capability, G′ represents the architecture diagram after damage, and G″ represents the new architecture diagram obtained after the interaction topology T(V′,G′) after damage is adjusted according to the preset conditions. represents the firepower network elasticity, N represents the number of sampling times, and p i represents the i-th sampling point, Indicates firepower node F i Coverage, It indicates the detection of network resilience. Represents the detection node R i Coverage, represents the elasticity of the communication network, C i represents the ith charge node, C j represents the jth charge node, Q(·) represents the performance function vector, T(V, G) is the original system topology, Rob rand Indicates the robustness to random attacks, Rob key represents the robustness against targeted attacks, T(V′ rand , G′ rand ) represents the topological structure of the system after random attack, T(V′ key , G′ key ) represents the system topology after being attacked in a targeted manner.

[0032] Furthermore, the level of intelligent decision-making includes six dimensions: structural attributes, communication attributes, response attributes, strike attributes, robust attributes and resilience attributes.

[0033] Furthermore, evaluating the intelligent decision-making level of the intelligent decision-making model according to the basic evaluation calculation results includes: evaluating the structural attributes of the intelligent decision-making model according to the detection range, detection scale, detection coverage, communication range, communication scale, firepower scale and firepower coverage range.

[0034] Furthermore, evaluating the intelligent decision-making level of the intelligent decision-making model based on the basic evaluation calculation results also includes: evaluating the communication attributes of the intelligent decision-making model based on communication range, communication scale, communication network capacity, communication balance, communication network elasticity and self-healing capability.

[0035] Furthermore, evaluating the intelligent decision-making level of the intelligent decision-making model based on the basic evaluation calculation results also includes: evaluating the response attributes of the intelligent decision-making model based on detection coverage, firepower coverage, task matching, communication efficiency, information chain closure, detection communication cost, detection duty cycle and detection communication cost.

[0036] Furthermore, evaluating the intelligent decision-making level of the intelligent decision-making model based on the basic evaluation calculation results also includes: evaluating the strike attributes of the intelligent decision-making model based on the detection range, firepower scale, firepower coverage range, firepower network diversity, firepower coverage rate, task matching degree, interception success rate, interception effectiveness, interception priority effectiveness, information chain closure, architecture deployment interception depth, target detection resource allocation capability, target firepower resource allocation capability and target guidance resource allocation capability.

[0037] Furthermore, evaluating the intelligent decision-making level of the intelligent decision-making model based on the basic evaluation calculation results also includes: evaluating the robust properties of the intelligent decision-making model based on the degree of detection diversity, firepower network diversity, resource balance, information chain closure, anti-destruction rate, target detection resource allocation capability, target firepower resource allocation capability, target guidance resource allocation capability, random attack robustness, targeted attack robustness and critical threshold.

[0038] Furthermore, evaluating the intelligent decision-making level of the intelligent decision-making model based on the basic evaluation calculation results also includes: evaluating the resilience properties of the intelligent decision-making model based on critical threshold, task matching, anti-destruction rate, target detection resource allocation capability, target firepower resource allocation capability, target guidance resource allocation capability, firepower network resilience, detection network resilience, network resilience, random attack robustness and targeted attack robustness.

[0039] Furthermore, the intelligent decision-making level of the intelligent decision-making model is evaluated according to the basic evaluation calculation results through the following formula:

[0040]

[0041] In the above formula, W k is the weight vector of the kth dimension, is the indicator bias vector used to calculate the k-th dimension evaluation result, ∑ k is the scale change vector used to calculate the k-th dimension evaluation result, and∑ k Obtained by fitting historical data.

[0042] According to another aspect of the present invention, there is also provided an intelligent decision-making test system based on reinforcement learning, the system comprising a scenario parameter space construction module, a scenario generation module, an optimization judgment module and a simulation test module;

[0043] The scenario parameter space construction module is used to define scenario test case parameters, and define the difficulty levels of different scenarios based on the scenario test case parameters to form a scenario parameter space. The scenario test case parameters include the number of red teams, the density of red team deployment, the balance of red team unit categories, the number of blue teams, the concentration of blue team attacks, the pre-war loss ratio of the red team, and the criticality of the pre-war loss units of the red team;

[0044] The scenario generation module is used to construct and initialize a reinforcement learning model based on the PPO algorithm; a set of scenario parameters are sampled from the scenario parameter space, and the currently sampled scenario parameters are converted into a scenario through the reinforcement learning model;

[0045] The simulation test module is used to load the assumptions generated by the assumption generation module in the simulation, and use the intelligent decision-making model to perform cyclic simulation tests according to the loaded assumptions, and perform basic evaluation calculations on the decision-making performance of the intelligent decision-making model under the current assumptions according to static indicators, dynamic indicators and adaptability indicators, and evaluate the intelligent decision-making level of the intelligent decision-making model according to the basic evaluation calculation results;

[0046] The optimization judgment module is used to calculate the confidence of the reinforcement learning model based on the scene parameters obtained by sampling and the intelligent decision-making level evaluation results, and to determine whether the confidence meets the preset requirements. If not, the assumption generation module is controlled to perform the next round of sampling and generate new assumptions, and the simulation test module is controlled to perform cyclic simulation tests based on the new assumptions. If yes, the evaluation conclusion for the intelligent decision-making model is output.

[0047] The technical solution of the present invention is applied to provide an intelligent decision-making test method and test system based on reinforcement learning. The method is oriented to the needs of new intelligent combat systems and copes with increasingly flexible and complex combat systems. The complex combat confrontation scenes are abstracted into parameterizable "scenarios", and various defense scenarios from simple to complex are described in a standardized language. The scenario parameterization is supported, so that complex defense scenarios can be accurately and consistently described. Reinforcement learning is applied to the field of intelligent decision-making testing, especially for intelligent battlefield environments. The scenario generation problem is defined as a Markov decision process (MDP) using a reinforcement learning algorithm, and the policy network parameters are adjusted through a policy gradient optimization algorithm (PPO) to achieve efficient and comprehensive Scenario generation supports coverage testing of all functions of the decision-making system and accelerated and intelligent testing of key scenarios. By introducing reinforcement learning, the test process can intelligently adapt to different scenarios and conditions. Through continuous trial and error and learning, the test strategy is optimized, which greatly improves the confidence of the test. Through a multi-dimensional evaluation system, the adaptability and robustness of the system can be comprehensively evaluated, which helps the system maintain stable performance in a complex and changeable battlefield environment, improve survivability and combat effectiveness, and achieve comprehensive evaluation and efficient testing of decision-making capabilities. It provides a scientific basis for the research and development and upgrading of military equipment, and can provide strong support for the realization of new combat styles. It is of great significance to promote the advancement of military science and technology and improve combat effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A flow chart of an intelligent decision-making testing method based on reinforcement learning provided according to a specific embodiment of the present invention is shown. DETAILED DESCRIPTION

[0050] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0052] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0053] like Figure 1 As shown, according to a specific embodiment of the present invention, a smart decision-making test method based on reinforcement learning is provided, the method comprising:

[0054] S1, define scenario test case parameters, and define the difficulty levels of different scenarios based on the scenario test case parameters to form a scenario parameter space. The scenario test case parameters include the number of red teams, the density of red team deployment, the balance of red team unit categories, the number of blue teams, the concentration of blue team attacks, the pre-war loss ratio of the red team, and the criticality of the pre-war loss units of the red team;

[0055] S2, build a reinforcement learning model based on the PPO algorithm and initialize it;

[0056] S3, sampling a set of scene parameters from the scene parameter space, and transforming the currently sampled scene parameters into a scenario through the reinforcement learning model;

[0057] S4, loading the assumption obtained in S3 in the simulation, using the intelligent decision-making model to perform a cyclic simulation test according to the loaded assumption, performing basic evaluation calculations on the decision-making performance of the intelligent decision-making model under the current assumption according to static indicators, dynamic indicators and adaptability indicators, and evaluating the intelligent decision-making level of the intelligent decision-making model according to the basic evaluation calculation results;

[0058] S5, calculating the confidence of the reinforcement learning model according to the scene parameters sampled in S3 and the intelligent decision-making level evaluation result obtained in S4, and judging whether the confidence meets the preset requirements, if not, go to S6, if yes, go to S7;

[0059] S6, repeat S3 to S5 until the confidence reaches the preset requirement;

[0060] S7, outputs the evaluation conclusion of the intelligent decision-making model.

[0061] By applying this configuration method, a reinforcement learning-based intelligent decision-making test method is provided. This method is oriented to the needs of new intelligent combat systems and copes with increasingly flexible and complex combat systems. The complex combat confrontation scenes are abstracted into parameterizable "scenarios", and various defense scenarios from simple to complex are described in a standardized language. The scenario parameterization is supported, so that complex defense scenarios can be described accurately and consistently. Reinforcement learning is applied to the field of intelligent decision-making testing, especially for intelligent battlefield environments. The scenario generation problem is defined as a Markov decision process (MDP) using a reinforcement learning algorithm, and the policy network parameters are adjusted through a policy gradient optimization algorithm (PPO) to achieve efficient and comprehensive scenario generation. , supports coverage testing of various functions of the decision-making system and acceleration and intelligent testing of key scenarios. By introducing reinforcement learning, the test process can intelligently adapt to different scenarios and conditions. Through continuous trial and error and learning, the test strategy is optimized, which greatly improves the confidence of the test; through a multi-dimensional evaluation system, it can comprehensively evaluate the adaptability and robustness of the system, which helps the system to maintain stable performance in a complex and changeable battlefield environment, improve survivability and combat effectiveness, and achieve comprehensive evaluation and efficient testing of decision-making capabilities; it provides a scientific basis for the research and development and upgrading of military equipment, and can provide strong support for the realization of new combat styles. It is of great significance to promote the advancement of military science and technology and improve combat effectiveness. Compared with the prior art, the technical solution of the present invention can solve the technical problem of the lack of decision-making level evaluation method of intelligent decision-making model in the prior art.

[0062] In actual applications, the scenario test case parameters are defined as follows:

[0063] red_richness: the number of red players;

[0064] red_density_level: the density of red team deployment;

[0065] red_balance: red side unit type balance;

[0066] blue_richness: the number of blue players;

[0067] blue_concentration: the concentration of blue attack;

[0068] kill_intensity: Red side's loss ratio before the battle;

[0069] kill_vitality: The criticality of the Red side's pre-battle loss units.

[0070] In addition, common parameters are defined as follows:

[0071] DELTA_NUMBER: The variable range of each parameter, which refers to the floating range around the default value, the default value is 20;

[0072] Red team node position: coordinates of each red team node, in polar coordinates (r, theta) or rectangular coordinates (x, y);

[0073] Blue node position: coordinates of each node, in polar coordinates (r, theta) or rectangular coordinates (x, y);

[0074] Combat range: The more concentrated the blue side's targets are, the greater the difficulty; the more dispersed the targets are, the easier it is.

[0075] The corresponding parameter space can be constructed through the above parameters and their value ranges.

[0076] Furthermore, for the difficulty level of the defined scenario, according to the complexity factor, we believe that the difficulty is positively correlated with the size of the strategy space. That is, the larger the strategy space, the larger the search space, the more difficult the process of finding the optimal solution, and therefore the greater the difficulty of the scenario. Due to the one-to-one correspondence between the strategy space and the parameter space, the above positive correlation still holds true between the parameter space and the difficulty. Specifically, since there is an exponential relationship between the size of the parameter space and the variable range of the DELTA_NUMBER parameter defined in the scenario sample, therefore, in an embodiment of the present invention, a part of the difficulty index is defined as the weighted variable range of the DELTA_NUMBER parameter.

[0077] In addition, the difficulty index is also positively correlated with the combat range, that is, the larger the combat range, the greater the difficulty. In other words, the more concentrated the blue team's targets are, the greater the difficulty; the more dispersed the targets are, the smaller the difficulty is. Therefore, in another embodiment of the present invention, we sample the average distance between the blue team and the cluster center as an indicator to describe the degree of dispersion. The more concentrated, the shorter the average distance and the greater the difficulty; vice versa. Considering that the blue team's coordinates are expressed in rectangular coordinate form, that is, (x, y), we use the blue team's node coordinate center As cluster centers:

[0078]

[0079] Then the average distance can be defined as:

[0080]

[0081] Therefore, the second part of the difficulty index can be defined as a weighted form of the average distance.

[0082] In addition, in an embodiment of the present invention, the PPO algorithm is used as the optimizer of the reinforcement learning model, that is, the policy gradient optimization algorithm (PPO) is used to adjust the policy network parameters, and the loss function of the policy gradient algorithm is as follows:

[0083]

[0084] Policy gradient is an on-policy algorithm, which requires that the sampling strategy and the update strategy are the same strategy. Therefore, it will take a lot of time to sample data. After the agent interacts with the environment, it will update the parameters. These data can only be used to update the parameters once. Next, it is necessary to collect data again before updating the parameters again. In order to reduce the number of sampling times and improve the efficiency of sample utilization, PPO uses the importance sampling theorem to improve the policy gradient algorithm and turn it into an off-policy capability. The importance sampling theorem does not require the random variable to satisfy the distribution of p(x) when calculating the expectation of a distribution (denoted as p(x)). Even if the random variable is assumed to obey other distributions (denoted as q(x)), the expectation of the random variable under the p(x) distribution can still be calculated. The specific mathematical expression is as follows:

[0085]

[0086] By substituting the importance sampling formula into the policy gradient loss function, and then using a series of mathematical approximations and derivation techniques, we can obtain the loss function of PPO:

[0087]

[0088] Among them, the A function represents the advantage function, which is usually defined by subtracting the state value from the state-action value, and the state value is estimated more closely to be unbiased and with small variance through the generalized advantage estimation (GAE). The PPO function achieves a small update when the strategy is updated through a simple clip upper and lower limit operation, thereby ensuring that the update operation is performed in the trust domain, stabilizing the training process, and achieving robust training.

[0089] The following specifically introduces the test case generation algorithm based on reinforcement learning proposed in the present invention. In this algorithm, the test case generation problem is constructed as a Markov decision process (MDP), where the state, action, and reward are defined as follows:

[0090] 1) Status: includes the tested scenario parameters and the decision-making behavior of the defense system intelligent decision-making algorithm in the scenario. Scenario parameters include deployment entity parameters (such as entity category, quantity, location) and defense entity parameters (such as entity category, quantity, location). Decision-making behavior includes architecture deployment decision-making behavior and firepower allocation decision-making behavior.

[0091] 2) Action: Generated test scenario parameters. Based on the state input, the scenario generator generates deployment entity parameters and defense entity parameters in the test scenario.

[0092] 3) Rewards: Evaluation indicators of intelligent decision-making algorithms in scene testing, including comprehensive indicators jointly constructed by scene static indicators, scene dynamic indicators, scene adaptation indicators, etc.

[0093] In this MDP problem, the scenario generator can be regarded as an intelligent agent that generates demand-oriented test scenarios, including comprehensive test scenarios and challenging test scenarios, by observing the current state.

[0094] By constructing the test scenario generation process as an MDP, different reinforcement learning algorithms can be used to achieve controllable test scenario generation. The generated scenarios will cover all scenarios under the preset parameter space, including general scenarios and challenging scenarios, thereby enabling comprehensive and targeted testing of intelligent decision-making algorithms.

[0095] The core of intelligent testing is how to find key scenarios that decision-making algorithms are difficult to deal with more quickly in order to discover strategy defects. Based on constructing the scenario generation process as MDP, this study proposes a scenario generation method based on reinforcement learning to accelerate the test and evaluation process of intelligent decision-making algorithms.

[0096] The test scenario generation method based on reinforcement learning aims to generate critical test scenarios that may cause strategy failure. It takes into account that most of the generated test scenarios are non-critical, that is, the decision algorithm can better cope with most normalized scenarios and some abnormal scenarios, and these scenario states will not provide information for building critical scenarios. Therefore, the proposed reinforcement learning method focuses on how to eliminate non-critical state data to accelerate the critical scenario discovery process.

[0097] When using the reinforcement learning model to generate scenarios, first, the scenario assumption requirement description is constructed as a configuration file to describe the test scenario parameters and test case space. Then, based on the constructed demand-oriented test case space, the automatic sampling generation of test samples for the scenario assumption is realized based on the reinforcement learning algorithm. Finally, in the test case generation process, the difficulty of the assumption is graded based on the constructed evaluation index system and the complexity of the scenario. In other words, it mainly includes two processes: first, the performance indicators of the scenario test are used to identify critical states or scenarios. For example, the failure of the decision algorithm to achieve effective defense in the architecture deployment test is used as a performance indicator. The key state and scenario parameters that cause the phenomenon are traced back to the source, and the evaluation of the state is constructed through reward plasticity. Then, the Markov process of the scenario generation and evolution process is edited and pruned, non-critical data is discarded, and the remaining data is used for reinforcement learning to train the policy gradient, so as to accelerate the process of discovering key scenarios.

[0098] In order to evaluate the credibility of the generated results by the scene generation algorithm, a confidence index is defined in the embodiment of the present invention, and its design idea is to evaluate the confidence of the reward value of the current result in the historical reward distribution. The specific calculation method is as follows: record the historical reward value, and calculate the expectation μ and standard deviation σ of the random variable. Assuming that the reward value obeys the Gaussian distribution N(μ, σ), then for the current newly sampled reward value r, its confidence interval can be expressed as (-|r-μ|, |r-μ|), then the confidence of the scene sampling can be defined as the confidence of the confidence interval in the Gaussian distribution N(μ, σ), that is, under this Gaussian distribution, what is the probability of falling in this interval. Let the number of sampling times be n, and the size of the sampled space be C c , the total action space size is C, then the scene sampling confidence c is calculated as follows:

[0099]

[0100] In the early stage of reinforcement learning, the action space is randomly sampled for extensive exploration to ensure comprehensive sample coverage. Subsequently, the algorithm focuses on areas with high variance of the evaluation index and increases the sampling density of these areas and their neighborhoods, aiming to reduce the uncertainty of the evaluation and improve the confidence level of the area. Since high-variance areas mean that the uncertainty of the evaluation results is large, reinforcement learning guides sampling to these areas through the designed reward mechanism, prompting new samples to concentrate more on these key areas, thereby accelerating convergence and enhancing the reliability of the evaluation results. Finally, the entire scene sampling space is covered through exploration, and the number of samples in the neighborhood with large evaluation variance is increased through exploitation (sampling along the direction of reward function improvement), thereby accelerating sampling efficiency and improving the confidence of the evaluation results.

[0101] Furthermore, the present invention adopts a multi-dimensional analysis method based on the statistics of scenario indicator test results, and evaluates and analyzes the network from different angles and levels to obtain a more comprehensive and in-depth understanding. In the embodiment of the present invention, the decision-making performance of the intelligent decision-making model under the current assumption is evaluated and calculated based on static indicators, dynamic indicators and adaptability indicators. Specifically, static indicators evaluate the theoretical overall performance of the SSL composed of our combat system in the absence of interaction; dynamic indicators evaluate the overall performance of our combat system in one type of scenario under the condition of red and blue interaction; adaptability indicators evaluate the overall generalization performance of our combat system in different types of scenarios under the condition of red and blue interaction. There are 33 indicators for evaluating intelligent decision-making capabilities.

[0102] 1. Static indicators include:

[0103] (1) Detection scale

[0104] Input: Interaction topology T(V, G), where V represents all nodes and G represents all edges of the topology. T(V, G) has the same meaning below. The set of probe nodes is V. R .

[0105] The detection scale of the interaction topology T(V, G) is calculated by the following formula:

[0106] Q R％ =|V R | / |V|.

[0107] (2) Detection range

[0108] Input: interaction topology T(V, G), detection node set V R .

[0109] Each detection node R i Contains three main attributes: position coordinates and radius: R i = <x i ,y i , r i >,x i Represents the horizontal axis, y i Represents the vertical axis, r i Represents the radius.

[0110] Detection range Calculated by the following formula:

[0111] Represents the union of the detection ranges of all detection nodes.

[0112] (3) Detection of diversity

[0113] Input: real-time optimization index interaction topology T(V, G), where the real-time optimization index detection unit belongs to K r detection unit types, each unit type includes N r detection units, the capability of each unit type can be expressed as k capability attributes, namely

[0114] The distribution probability of each unit type category i is:

[0115]

[0116] The diversity of unit types can be calculated by the following formula:

[0117]

[0118] Discretize the unit type capability space Θ into several intervals, and regard each interval as having the same value, then we can get the unit type K r The probability distribution of the ability attributes in is P i[θ] = <p i，1 , p i，2 , ..., p i，k >, then the diversity of unit type capabilities is Q var[θ] Calculated by the following formula:

[0119] Q var[θ] =-∑ k ∑ i p i,k[θ] ×log(p i,k[θ] ),

[0120] In the above formula, p i,k[θ] Represents the probability distribution of the k-th dimension ability attribute in space θ.

[0121] (4) Detection coverage

[0122] Input: interaction topology T(V, G), defense area Area D , the detection node set is V R .

[0123] Each detection node contains three main attributes: position coordinates and radius: i = <x i ,y i , r i >, x i Represents the horizontal axis, y i Represents the vertical axis, r i Represents the radius.

[0124] Calculate the detection range based on the interaction topology T(V, G):

[0125]

[0126] Then the detection coverage rate Q R∩％ Calculated by the following formula:

[0127]

[0128] (5) Communication scale

[0129] The number of communication nodes is Nc, and the communication capacity of the communication node is set to R C , can be the same as R C (Military wireless communication equipment can transmit about tens of kilometers, and the specific data needs to refer to the specific equipment parameters).

[0130] Input: total number of communication nodes Nc, total number of nodes N.

[0131] Communication scale ρ c Calculated by the following formula:

[0132]

[0133] (6) Communication coverage

[0134] Input: interaction topology T(V, G), detection node set V R , firepower node set V F .

[0135] First, calculate the degree matrix of the interactive topology graph Then the communication coverage rate Q comm％ Calculated by the following formula:

[0136]

[0137] (7) Communication network costs

[0138] Input: interaction topology T(V, G), charge node set V C .

[0139] Take the subgraph G formed by the communication subnetwork of the interactive topology graph G C , then the communication cost cost(V i ,V j ) can be calculated by the following formula:

[0140] Represents any two points i and j in the subgraph G C The shortest distance on

[0141] Then the communication cost Q of the instant optimization index system is C,C It can be calculated by the following formula:

[0142]

[0143] (8) Communication balance

[0144] Input: interaction topology T(V, G), charge node set V C .

[0145] First, calculate the degree matrix of the interactive topology graph Communication balance Calculated by the following formula:

[0146]

[0147] (9) Firepower scale

[0148] Input: interaction topology T(V, G), the firepower node set is V F .

[0149] The firepower scale Q of the interaction topology T(V, G) F％ It can be calculated by the following formula:

[0150] Q F％ =|V F | / |V|.

[0151] (10) Fire coverage

[0152] Input: interaction topology T(V, G), the firepower node set is V F .

[0153] Each firepower node contains three main attributes: position coordinates and radius: i = <x i ,y i , r i >, x i Represents the horizontal axis, y i Represents the vertical axis, r i Represents radius. Fire coverage area Calculated by the following formula:

[0154]

[0155] (11) Firepower Network Diversity

[0156] Input: real-time optimization index interaction topology T(V, G), where the real-time optimization index firepower unit belongs to K f detection unit types, each unit type includes N f detection units, the capability of each unit type can be expressed as k capability attributes, namely

[0157] The distribution probability of each unit type category i is:

[0158]

[0159] The diversity of unit types can be calculated by the following formula:

[0160]

[0161] Discretize the unit type capability space Θ into several intervals, and regard each interval as having the same value, then we can get the unit type K f The probability distribution of the ability attributes in is P i[θ] = <p i，1 , p i，2 , ..., p i，f >, then the diversity of unit type capabilities It can be calculated by the following formula:

[0162]

[0163] (12) Firepower coverage

[0164] Input: system architecture T(V, G), defense area Area D , the firepower node set is V R Each firepower node contains three main attributes: location coordinates and radius: F i = <x i ,y i , r i >,

[0165] Calculation method: Calculate the detection range based on the system architecture T(V, G):

[0166]

[0167] The detection coverage technology is as follows:

[0168]

[0169] (13) Critical Threshold

[0170] Input: interaction topology T(V, G).

[0171] Calculation formula:

[0172]

[0173] In the above formula, f t represents the critical threshold, <k>and<k^2> are the first and second moments of the network degree distribution.

[0174] (II) Dynamic indicators include:

[0175] (14) Interception success rate

[0176] Input: interaction topology T(V, G), target set E, fire decision strategy π.

[0177] Carry out m rounds of tests. In each test, the interaction topology T(V, G) and the target situation E editing scenario are sent to the simulation system for simulation. The set of intercepted targets is E′. Then the interception success rate of a single test is calculated as

[0178] p(π)=|E′| / |E|,

[0179] Then the average interception success rate of m rounds of trials is Q p (π) is calculated as follows:

[0180]

[0181] (15) Interception effectiveness

[0182] Input: interaction topology T(V, G), target situation E, fire decision strategy π.

[0183] Carry out m rounds of experiments. In each experiment, the interactive topology T(V, G) and the target situation E editing scenario are sent to the simulation system for simulation. The number of targets consumed is N. i , i = 1, 2, ..., m pieces, then the single test consumption ratio is calculated as:

[0184] η(π)=N i / |E|,

[0185] The average interception efficiency of m rounds of trials is Q η (π) is calculated as follows:

[0186]

[0187] (16) Interception priority efficiency

[0188] Input: interaction topology T(V, G), target situation E, firepower decision strategy π, target priority score r(e i ).

[0189] First, the kill net is generated by combining the interaction topology T(V, G), the target situation E, and the firepower decision strategy. The kill net contains i Several chains T(e i )={τ j (e i )}, chain τ j For e i The interception distance is denoted as l(e i |τ j ). For each interception target e i , find the firepower node F that can be intercepted as far as possible j , and F j For i The maximum interception distance is e i The standard quantity is denoted as l(e i ). Then the interception priority efficiency Q prior Calculated as follows:

[0190]

[0191] (17) Task Matching

[0192] Input: interaction topology T(V, G), target situation distribution E i ~p(E), chain number threshold∈.

[0193] Sampling the target situation m times, we get {E i }. For each target case E i Constructing a Kill Net:

[0194]

[0195] For the target case E i Each target in j , whether the task can be matched is calculated by the following formula:

[0196]

[0197] For E i The task matching degree is calculated as:

[0198]

[0199] Then the interaction topology T(V, G) faces the target situation distribution E i The task matching degree Q of ~p(E) M(p(E),∈) Calculated as:

[0200]

[0201] (18) Resource balance

[0202] Input: interaction topology T(V, G), where V = (C, R, F) includes accusation C, detection node R, fire node F, interception strategy π, and target situation E.

[0203] The interactive topology T(V, G) constructs a kill net according to the target situation E, and applies the interception strategy to form an interception net, and then forms the target allocation. Calculate each detection node R i With firepower node F i The consumption ratio r(R i ), r(F i ): The variance of the consumption ratio of each node on the interactive topology is the resource balance Q blc :

[0204]

[0205] (19) Communication efficiency

[0206] Input: interaction topology T(V, G), target situation E, fire decision strategy π.

[0207] First, the kill net is generated by combining the interaction topology T(V, G), the target situation E, and the firepower decision strategy. The kill net contains i Several chains of τ(e i )=R i →{C j }→F k , where the communication link {C j } is a subset of chains, whose length is |τ(e i )-2. The graph architecture diagram of the interaction topology T(V, G) is G, and the communication self-network diagram is G c , Figure G c The diameter of the c ).

[0208] Therefore, the communication efficiency It can be calculated by the following formula:

[0209]

[0210] (20) Information chain closure

[0211] Input: interaction topology T(V, G), target situation E

[0212] First calculate the chain:

[0213] τ(e i )=R i →{C j }F k ,

[0214] The required bandwidth is calculated as follows:

[0215]

[0216] The communication delay is |τ(e i )|,

[0217] When the information chain is closed, the communication bandwidth consumption is minimal and the communication distance and number of communication jumps are minimal, so the information chain closure Q info Calculated by the following formula:

[0218]

[0219] In the above formula, λ1 and λ2 represent weighting parameters, b(e i |τ(e i )) represents the information chain τ(e i ) bandwidth requirement, d(e i |τ(e i )) represents the information chain τ(e i ) of the jumps.

[0220] (21) Detection duty cycle

[0221] Input: interaction topology T(V, G), target situation E

[0222] First calculate the kill net:

[0223] T(e i )={τ(e i )|T(V,G)},

[0224] Each chain is:

[0225] τ(e i )=R i →{C j }→F k ,

[0226] Then the detection duty cycle is τ for all chains i For i The union of the detection ranges of i The flight path of a single detection unit is P. i The detectable path is The calculation is as follows:

[0227]

[0228] The average detection duty cycle is calculated as follows:

[0229]

[0230] Among them, d ％ (e i |T(V,G)) represents the target e i The detectable path duty cycle.

[0231] (22) Detection communication cost

[0232] Input: interaction topology T(V, G), target situation E,

[0233] First calculate the chain:

[0234] τ(e i )=R i →{C j }→F k ,

[0235] The required bandwidth is calculated as follows:

[0236]

[0237] Detection communication cost C (e i |T(V,G)) can be calculated as follows:

[0238]

[0239] Therefore, the average detection communication cost is obtained by the following formula:

[0240]

[0241] (23) Architecture deployment interception depth

[0242] Input: Target track T M , represents the target’s trajectory. The target’s trajectory can be represented as a series of space-time coordinate points, T M ={<t1,x1,y1> ,<t2,x2,y2> ,...}; Fire kill zone F A , indicating the geographical location and range of firepower deployment. The firepower killing area can be represented as a series of circular areas, F A ={<X a1 ,y a1 , r a1 >,<x a2 ,y a2 , r a2 >, ...}.

[0243] Interception Depth D I The target track T can be calculated M Fire kill zone F A The specific calculation formula is:

[0244]

[0245] in, represents the projection of the overlapping part of the target track and the i-th fire kill zone on the horizontal plane, and Length(·) represents the length of the calculated projection.

[0246] (24) Target detection resource allocation capability

[0247] Input: target set E, detection resource set V F .

[0248] The target detection resource allocation capability is evaluated by calculating the efficiency of the detection resources allocated to each target. The specific calculation formula is:

[0249]

[0250] Among them, E(t i ,r j ) represents the target t i Detected resources j Identification and tracking efficiency.

[0251] (25) Target firepower resource allocation capability

[0252] Input: target set E, firepower resource set V F .

[0253] The target firepower resource allocation capability is evaluated by calculating the efficiency of the firepower resources allocated to each target. The specific calculation formula is:

[0254]

[0255] Among them, η F (F i , e i ) represents the target e i Firepower Resource F i The efficiency of interception.

[0256] (26) Ability to guide resource allocation

[0257] Input: target set E, communication resource set V C .

[0258] The target guidance resource allocation capability is evaluated by calculating the efficiency of the guidance resources allocated to each target. The specific calculation formula is:

[0259]

[0260] Among them, τ represents a line for the target e i The decision chain, |τ| represents the number of jumps, η τ (τ,e i ) represents the effect of τ on the target e i guiding ability.

[0261] (III) Adaptability indicators include:

[0262] (27) Destruction rate

[0263] Input: system topology T(V, G), target situation E, performance loss threshold p ％ , attack strategy π.

[0264] First, calculate the task matching degree Q of the system topology architecture T(V, G) M (T(V, G), E). Calculate the connectivity of each node in the system topology diagram G and sort them from large to small. Damage ratio P loss Starting from 0 to 1, according to the attack strategy π, remove the first P loss The nodes are obtained from V', forming the damaged architecture diagram G'. Calculate the task matching degree Q under the new interaction topology T(V', G') M (T(V′, G′), E). Plot the task matching degree versus damage ratio until:

[0265]

[0266] The damage ratio at this time can be used to calculate the anti-destruction rate:

[0267] Q loss =1-P loss .

[0268] (28) Self-healing ability

[0269] Input: interaction topology T(V, G), target situation E, loss ratio δ, function ratio σ, damage strategy π δ .

[0270] First, calculate the task matching degree M(T(V, G), E) of the interaction topology T(V, G) under the target situation E. Make the interaction topology follow the damage strategy π δ Then, we can get the damaged node V′, which satisfies:

[0271] |V′|≤(1-δ)|V|,

[0272] The topology diagram at this time is recorded as G′.

[0273] Adjust the interaction topology T′(V′, G′) to obtain a new interaction topology G″, which is required to satisfy:

[0274] M(T″ (v′，G″) ,E)>σ·M(T(V,G),E),

[0275] Then the minimum adjustment is G″-G′, and the self-healing ability of the interactive topology can be obtained:

[0276]

[0277] (29) Firepower Network Resilience

[0278] Input: interaction topology T(V,G), defense area Area D , the firepower node set is V F Each firepower node contains three main attributes: position coordinates and radius: i = <x i ,y i , r i >.

[0279] The Monte Carlo method is used to calculate the repeated coverage rate of the firepower network in the defense area. D Randomize N sampling points p i ∈Area D .right <p1,...,p N >Traverse, any p i Coverage in the interactive topology T(V, G):

[0280]

[0281] The firepower network elasticity can be calculated by the following formula:

[0282]

[0283] in, Indicates firepower node F i coverage.

[0284] (30) Detecting network resilience

[0285] Input: interaction topology T(V, G), defense area Area D , the detection node set is V R Each detection node contains three main attributes: position coordinates and radius: R i = <x i ,y i , r i >.

[0286] The Monte Carlo method is used to calculate the repeated coverage of the detection network in the defense area. D Randomize N sampling points p i ∈Area D ,right <p1,...,p N >Traverse, any p i Coverage in the interactive topology T(V, G):

[0287]

[0288] The detection network elasticity can be calculated by the following formula:

[0289]

[0290] (31) Communications network resilience

[0291] Input: accusation node set V C Each control node contains three main attributes: position coordinates and radius: C i = <x i ,y i , r i >.

[0292] For each charging node C i Traversing, the number of nodes covered by other accused nodes is:

[0293]

[0294] The communication node elasticity can be calculated as follows:

[0295]

[0296] (32) Random Attack Robustness

[0297] Input: interaction topology T(V, G);

[0298] The random attack robustness of the interaction topology T(V, G) can be calculated as follows:

[0299] Rob rand =Q(T(V, G)) / Q(T(V′ rand , G′ rand )),

[0300] Q(·) represents the performance function vector, T(V, G) is the original system topology, T(V′ rand , G′ rand ) represents the system topology after being subjected to random attacks.

[0301] The value of random attack robustness is usually between 0 and 1, and the closer it is to 1, the more robust the network is. If the performance of the network decreases slightly after randomly deleting nodes, then the network has high random attack robustness.

[0302] (33) Robustness against targeted attacks

[0303] Input: interaction topology T(V, G);

[0304] The targeted attack robustness can be calculated as follows:

[0305] Rob key =Q(T(V, G)) / Q(T(V′ key , G′ key )),

[0306] Where Q(·) represents the performance function vector, T(V, G) is the original system topology, and T(V′ key , G′ key ) represents the system topology after being subjected to random attacks.

[0307] The value of targeted attack robustness is usually between 0 and 1, and the closer it is to 1, the more robust the network is. If the degradation of network performance after targeted node deletion is small, then the network has high targeted attack robustness.

[0308] Furthermore, in the embodiment of the present invention, the intelligent decision-making level includes six dimensions: structural attributes, communication attributes, response attributes, attack attributes, robust attributes, and elastic attributes. By analyzing the indicators of these six main dimensions, a quantitative dimension is given to each key performance indicator of the network, and a six-dimensional evaluation model of the intelligent decision-making system is obtained.

[0309] Specifically, in an embodiment of the present invention, evaluating the intelligent decision-making level of the intelligent decision-making model according to the basic evaluation calculation results includes: evaluating the structural attributes of the intelligent decision-making model according to the detection range, detection scale, detection coverage, communication range, communication scale, firepower scale and firepower coverage range; evaluating the communication attributes of the intelligent decision-making model according to the communication range, communication scale, communication network capability, communication balance, communication network elasticity and self-healing capability; evaluating the response attributes of the intelligent decision-making model according to the detection coverage, firepower coverage, task matching, communication efficiency, information chain closure, detection communication cost, detection duty cycle and detection communication cost; evaluating the response attributes of the intelligent decision-making model according to the detection range, firepower scale, firepower coverage range, firepower network diversity, firepower coverage, task matching, interception success rate, interception effectiveness, interception priority efficiency The strike attributes of the intelligent decision-making model are evaluated according to the degree of detection diversity, firepower network diversity, resource balance, information chain closure, anti-destruction rate, target detection resource allocation capability, target firepower resource allocation capability, target guidance resource allocation capability, random attack robustness, targeted attack robustness and critical threshold; the resilience attributes of the intelligent decision-making model are evaluated according to the critical threshold, task matching, anti-destruction rate, target detection resource allocation capability, target firepower resource allocation capability, target guidance resource allocation capability, firepower network elasticity, detection network elasticity, network elasticity, random attack robustness and targeted attack robustness.

[0310] Based on the above embodiment, in the embodiment of the present invention, the attributes of each dimension are calculated by various indicators belonging to the dimension. First, each indicator is independently normalized. The bias and scale of the normalized processing are obtained through historical data. Then, the weighted sum of the normalized indicators is calculated to obtain the ability evaluation result under the dimension. Assume that the indicator system vector obtained by the experiment is represented by V, then the evaluation result s of the kth dimension in the six dimensions k It is calculated as follows:

[0311]

[0312] In the above formula, W k is the weight vector of the kth dimension, is the indicator bias vector used to calculate the k-th dimension evaluation result, Σ k is the scale change vector used to calculate the k-th dimension evaluation result, and∑ k Obtained by fitting historical data.

[0313] Based on the results of multi-dimensional evaluation, the system's intelligent decision-making capabilities can be analyzed, and scenarios can be generated based on the evaluation results to enhance the system's intelligent decision-making capabilities.

[0314] According to another aspect of the present invention, there is also provided an intelligent decision-making test system based on reinforcement learning, the system comprising a scenario parameter space construction module, a scenario generation module, an optimization judgment module and a simulation test module;

[0315] The scenario parameter space construction module is used to define scenario test case parameters, and define the difficulty levels of different scenarios based on the scenario test case parameters to form a scenario parameter space. The scenario test case parameters include the number of red teams, the density of red team deployment, the balance of red team unit categories, the number of blue teams, the concentration of blue team attacks, the pre-war loss ratio of the red team, and the criticality of the pre-war loss units of the red team;

[0316] The scenario generation module is used to construct and initialize a reinforcement learning model based on the PPO algorithm; a set of scenario parameters are sampled from the scenario parameter space, and the currently sampled scenario parameters are converted into a scenario through the reinforcement learning model;

[0317] The simulation test module is used to load the assumptions generated by the assumption generation module in the simulation, and use the intelligent decision-making model to perform cyclic simulation tests according to the loaded assumptions, and perform basic evaluation calculations on the decision-making performance of the intelligent decision-making model under the current assumptions according to static indicators, dynamic indicators and adaptability indicators, and evaluate the intelligent decision-making level of the intelligent decision-making model according to the basic evaluation calculation results;

[0318] The optimization judgment module is used to calculate the confidence of the reinforcement learning model based on the scene parameters obtained by sampling and the intelligent decision-making level evaluation results, and to determine whether the confidence meets the preset requirements. If not, the assumption generation module is controlled to perform the next round of sampling and generate new assumptions, and the simulation test module is controlled to perform cyclic simulation tests based on the new assumptions. If yes, the evaluation conclusion for the intelligent decision-making model is output.

[0319] In summary, the present invention provides an intelligent decision-making test method and test system based on reinforcement learning. The method is oriented to the needs of new intelligent combat systems and copes with increasingly flexible and complex combat systems. The complex combat confrontation scenes are abstracted into parameterizable "scenarios", and various defense scenarios from simple to complex are described by standardized language. The scenario parameterization is supported, so that complex defense scenarios can be accurately and consistently described. Reinforcement learning is applied to the field of intelligent decision-making testing, especially for intelligent battlefield environments. The scenario generation problem is defined as a Markov decision process (MDP) using a reinforcement learning algorithm, and the policy network parameters are adjusted by a policy gradient optimization algorithm (PPO) to achieve efficient and comprehensive scenario Generate, support coverage testing of various functions of the decision-making system and acceleration and intelligent testing of key scenarios. By introducing reinforcement learning, the test process can intelligently adapt to different scenarios and conditions. Through continuous trial and error and learning, the test strategy is optimized, which greatly improves the confidence of the test; through a multi-dimensional evaluation system, it can comprehensively evaluate the adaptability and robustness of the system, which helps the system to maintain stable performance in a complex and changeable battlefield environment, improve survivability and combat effectiveness, and achieve comprehensive evaluation and efficient testing of decision-making capabilities; it provides a scientific basis for the research and development and upgrading of military equipment, and can provide strong support for the realization of new combat styles, which is of great significance for promoting the advancement of military science and technology and improving combat effectiveness. Compared with the prior art, the technical solution of the present invention can solve the technical problem of the lack of decision-making level evaluation method of intelligent decision-making model in the prior art.

[0320] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used herein to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figures. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figures. For example, if the device in the drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" may include both "above" and "below". The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used herein are interpreted accordingly.

[0321] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0322] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / k> < / k>

Claims

1. An intelligent decision-making test method based on reinforcement learning, characterized in that: The method comprises: S1, defining scenario test case parameters, and defining the difficulty levels of different scenarios based on the scenario test case parameters to form a scenario parameter space, wherein the scenario test case parameters include the number of red teams, the density of red team deployment, the balance of red team unit categories, the number of blue teams, the concentration of blue team attacks, the pre-war loss ratio of the red team, and the criticality of the pre-war loss units of the red team; S2, build a reinforcement learning model based on the PPO algorithm and initialize it; S3, sampling a set of scene parameters from the scene parameter space, and converting the currently sampled scene parameters into a scenario through the reinforcement learning model; S4, loading the assumption obtained in S3 in the simulation, using the intelligent decision-making model to perform a cyclic simulation test according to the loaded assumption, performing basic evaluation calculations on the decision-making performance of the intelligent decision-making model under the current assumption according to static indicators, dynamic indicators and adaptability indicators, and evaluating the intelligent decision-making level of the intelligent decision-making model according to the basic evaluation calculation results; S5, calculating the confidence of the reinforcement learning model according to the scene parameters sampled in S3 and the intelligent decision-making level evaluation result obtained in S4, and judging whether the confidence meets the preset requirements, if not, go to S6, if yes, go to S7; S6, repeating S3 to S5 until the confidence reaches the preset requirement; S7, output the evaluation conclusion of the intelligent decision-making model.

2. The method according to claim 1, characterized in that The static indicators include: Q R％ =|V R | / |V|, Q var[θ] =-Σ k Σ i p i,k[θ] ×log(p i,k[θ] ), Q F％ =|V F | / |V|, In the above formula, Q R％ Represents the detection scale, V R represents the set of detection nodes, V represents all nodes, Indicates the detection range, R i represents the i-th detection node, x i ,y i 、r i represents the abscissa, ordinate and radius of the node, Q var[θ] Indicates the degree of detection diversity, p i,k[θ] represents the probability distribution of the k-th dimension capability attribute in space θ, Q R∩％ Indicates detection coverage, Area D represents the defense area, ρ c Indicates the communication scale, N c represents the total number of communication nodes, N represents the total number of nodes, Q comm％ represents the communication coverage, V F represents the firepower node set, Q C,C represents the communication network cost, cost(V i ,V j ) represents the communication cost between any two nodes, Indicates the communication balance, V C represents the set of accused nodes, Q F％ Indicates the scale of firepower, Indicates the fire coverage range. Indicates firepower coverage, F i represents the i-th firepower node, Indicates the diversity of the firepower network, f t represents the critical threshold, <k>, <k 2 >represent the first-order moment and second-order moment of the network degree distribution respectively.< / k> 3. The method according to claim 2, characterized in that The dynamic indicators include: Q p (π) represents the interception success rate, m represents the number of test rounds, E represents the target set, E′ represents the intercepted target set, Q η (π) represents the interception effectiveness, N i Indicates the target quantity consumed, Q prior represents the interception priority efficiency, r(e i ) represents the interception target e i Priority scoring, l(e i |τ j ) represents the chain τ j For e i The interception distance, l(e i ) means e i The standard quantity, Q M(p(E),∈) represents the task matching degree, Q blc Indicates resource balance, r(R i ) represents the consumption ratio of each detection node, r(F i ) represents the consumption ratio of each firepower node, represents the communication efficiency, τ(e i ) represents the target e i The decision chain of G C Represents the communication network graph, d(G C ) represents G C The diameter, Q info represents the closure of the information chain, λ1 and λ2 represent weighting parameters, and b(e i |τ(e i )) represents the information chain τ(e i ) bandwidth requirement, d(e i τ(e i )) represents the information chain τ(e i ) jump number, Q R％ Denotes the detection duty cycle, d ％ (e i T(V,G)) represents the target e i The detectable path duty cycle, represents the detection communication cost, b C (e i |T(V,G)) represents the target e i The communication cost, D I Indicates the interception depth of architecture deployment, T M represents the target track, represents the i-th firepower killing zone, Q RD represents the target detection resource allocation capability, R D represents the detection resource set, E(t i ,r j ) represents the target t i Detected resources j Efficiency of identification and tracking, Q RF represents the target firepower resource allocation capability, R F represents the firepower resource set, E(t i ,f j ) represents the target t i Firepower resources j Interception efficiency, Q RG represents the target-guided resource allocation capability, R G represents the set of guiding resources, E(t i ,g j ) represents the target t i Guided resources j The efficiency of precise interception.

4. The method according to claim 3, characterized in that The adaptability indicators include: Q loss =1-P loss , Rob rand =Q(T(V,G)) / Q(T(V′ rand ,G′ rand )), Rob key =Q(T(V,G)) / Q(T(V′ key ,G′ key )), In the above formula, Q loss Indicates the damage resistance rate, P loss represents the damage ratio, Q δ represents the self-healing capability, G′ represents the architecture diagram after damage, and G″ represents the new architecture diagram obtained after the interaction topology T(V′,G′) after damage is adjusted according to the preset conditions. represents the firepower network elasticity, n represents the number of sampling times, and p i represents the i-th sampling point, Indicates firepower node F i Coverage, It indicates the detection of network resilience. Represents the detection node R i Coverage, represents the elasticity of the communication network, C i represents the ith charge node, C j represents the jth charge node, Q(·) represents the performance function vector, Rob rand Indicates the robustness to random attacks, Rob key represents the robustness of targeted attacks, T(V, G) is the original system topology, T(V′ rand , G′ rand ) represents the topological structure of the system after random attack, T(V′ key , G′ key ) represents the system topology after being attacked in a targeted manner.

5. The method according to claim 4, characterized in that The intelligent decision-making level includes six dimensions: structural attributes, communication attributes, response attributes, strike attributes, robust attributes and elastic attributes.

6. The method according to claim 5, characterized in that Evaluating the intelligent decision-making level of the intelligent decision-making model according to the basic evaluation calculation results includes: evaluating the structural attributes of the intelligent decision-making model according to the detection range, detection scale, detection coverage, communication range, communication scale, firepower scale and firepower coverage range.

7. The method according to claim 6, characterized in that Evaluating the intelligent decision-making level of the intelligent decision-making model based on the basic evaluation calculation results also includes: evaluating the communication attributes of the intelligent decision-making model based on communication range, communication scale, communication network capacity, communication balance, communication network elasticity and self-healing capability.

8. The method according to claim 7, characterized in that Evaluating the intelligent decision-making level of the intelligent decision-making model based on the basic evaluation calculation results also includes: evaluating the response attributes of the intelligent decision-making model based on detection coverage, firepower coverage, task matching, communication efficiency, information chain closure, detection communication cost, detection duty cycle and detection communication cost.

9. The method according to claim 8, characterized in that Evaluating the intelligent decision-making level of the intelligent decision-making model based on the basic evaluation calculation results also includes: evaluating the strike attributes of the intelligent decision-making model based on detection range, firepower scale, firepower coverage range, firepower network diversity, firepower coverage rate, task matching degree, interception success rate, interception effectiveness, interception priority effectiveness, information chain closure, architecture deployment interception depth, target detection resource allocation capability, target firepower resource allocation capability and target guidance resource allocation capability.

10. The method according to claim 9, characterized in that Evaluating the intelligent decision-making level of the intelligent decision-making model based on the basic evaluation calculation results also includes: evaluating the robust properties of the intelligent decision-making model based on the degree of detection diversity, firepower network diversity, resource balance, information chain closure, anti-destruction rate, target detection resource allocation capability, target firepower resource allocation capability, target guidance resource allocation capability, random attack robustness, targeted attack robustness and critical threshold.

11. The method according to claim 10, characterized in that Evaluating the intelligent decision-making level of the intelligent decision-making model based on the basic evaluation calculation results also includes: evaluating the elastic properties of the intelligent decision-making model based on critical threshold, task matching, anti-destruction rate, target detection resource allocation capability, target firepower resource allocation capability, target guidance resource allocation capability, firepower network elasticity, detection network elasticity, network elasticity, random attack robustness and targeted attack robustness.

12. The method according to any one of claims 6 to 11, characterized in that The intelligent decision-making level of the intelligent decision-making model is evaluated according to the basic evaluation calculation results using the following formula: In the above formula, W k is the weight vector of the kth dimension, is the indicator bias vector used to calculate the k-th dimension evaluation result, ∑ k is the scale change vector used to calculate the k-th dimension evaluation result, and∑ k Obtained by fitting historical data.

13. An intelligent decision-making test system based on reinforcement learning, characterized in that: The system includes a scenario parameter space construction module, a scenario generation module, an optimization judgment module and a simulation test module; The scenario parameter space construction module is used to define scenario test case parameters, and define the difficulty levels of different scenarios based on the scenario test case parameters to form a scenario parameter space, wherein the scenario test case parameters include the number of red teams, the density of red team deployment, the balance of red team unit categories, the number of blue teams, the concentration of blue team attacks, the pre-war loss ratio of the red team, and the criticality of the pre-war loss units of the red team; The scenario generation module is used to construct a reinforcement learning model based on the PPO algorithm and initialize it; sample a set of scene parameters from the scene parameter space, and convert the currently sampled scene parameters into a scenario through the reinforcement learning model; The simulation test module is used to load the assumptions generated by the assumption generation module in the simulation, use the intelligent decision model to perform a cyclic simulation test according to the loaded assumptions, perform basic evaluation calculations on the decision performance of the intelligent decision model under the current assumptions according to static indicators, dynamic indicators and adaptability indicators, and evaluate the intelligent decision level of the intelligent decision model according to the basic evaluation calculation results; The optimization judgment module is used to calculate the confidence of the reinforcement learning model based on the scene parameters obtained by sampling and the intelligent decision-making level evaluation results obtained, and judge whether the confidence meets the preset requirements. If not, the assumption generation module is controlled to perform the next round of sampling and generate new assumptions, and the simulation test module is controlled to perform cyclic simulation tests based on the new assumptions. If yes, the evaluation conclusion of the intelligent decision-making model is output.

Images