An equipment combat system intelligent level evaluation method

Abstract

The present application relates to a kind of equipment combat system intelligent level evaluation method, belong to combat system capability evaluation technical field, solve the rough evaluation method in prior art, inaccurate, high uncertainty, the problems such as complex evaluation process and poor interpretability.The evaluation method of the present application is from the intelligent degree of the model used in the introduced intelligent technology itself as the starting point, the bottom layer index of the minimum decomposable layer is decomposed to the minimum decomposable layer of equipment combat system layer by layer, the measurement factors of bottom layer index are analyzed from multiple aspects, the evaluation result is accurate, and have good interpretability.The evaluation model used in the evaluation bottom layer index value of the present application, step is simple, simple to realize, easy to integrate into simulation system or evaluation software, have good computability and scalability.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of combat system capability evaluation, and particularly relates to an equipment combat system intelligent level evaluation method. BACKGROUND

[0002] Modern war has the characteristics of high informatization, high coupling of combat elements, and rapid change of battlefield environment situation, which makes the deduction and decision-making of combat more and more complex. With the increase of the number of equipment, the uncertainty and complexity of deduction and decision-making also increase exponentially, and it is urgent to improve the intelligent level of equipment combat system through intelligent technology to adapt to the high-speed, complex and variable battlefield environment.

[0003] The intelligent level of equipment combat system will be one of the key factors to determine the result of future war. However, for a long time, how to measure the intelligent level of equipment combat system is still in the stage of fuzzy description and qualitative analysis. In recent years, some methods for quantitatively evaluating the intelligent level of system have been proposed. Some of the methods construct a structured index system, give the bottom index value, and use a comprehensive analysis method to obtain the evaluation results at different levels, but the index system is relatively rough, the measurement factors of the bottom index value are single, and the influence of the introduced intelligent technology on the intelligent level of the equipment combat system is not considered, so that the evaluation results are one-sided. Some methods introduce a machine learning model as an evaluation model, determine the bottom index value and the parameters of the model based on the index system and expert experience and experimental data, and finally infer the evaluation results at different levels, but the evaluation process is complex and the model has poor interpretability, which has great uncertainty risk in engineering application. SUMMARY

[0004] In view of the above problems, the application provides an equipment combat system intelligent level evaluation method, which solves the problems of rough evaluation method, inaccurate and uncertain evaluation results, complex evaluation process and poor interpretability in the prior art.

[0005] The application provides an equipment combat system intelligent level evaluation method, which specifically comprises the following steps:

[0006] Step 1, the intelligent levels of a plurality of intelligent models used by the equipment combat system are respectively decomposed layer by layer to obtain corresponding intermediate indexes and bottom indexes;

[0007] Step 2, for all the bottom indexes, the measurement factors of the corresponding bottom indexes are analyzed from the autonomy, interpretability, evolvability of each intelligent model used by the bottom index and the characteristics of the intelligent model, and the timeliness, completeness, accuracy and stability of the task involved by the bottom index; and an evaluation model of the level values of all the measurement factors is constructed;

[0008] Step 3, obtaining the bottom-level index value of each bottom-level index based on the level value of the metric factor in step 2 by using the evaluation model;

[0009] Step 4, obtaining the index value of each layer index in the equipment combat system by using a comprehensive evaluation method based on all the bottom-level index values; and obtaining the intelligent evaluation value of the equipment combat system from the index value of each layer index.

[0010] Optionally, when the corresponding intermediate index and bottom-level index are obtained by decomposition in step 1, the next-level index is obtained by decomposition from the corresponding previous-level index, and the bottom-level index is an index that cannot be decomposed.

[0011] Optionally, the specific steps of step 3 are as follows:

[0012] Step 31, obtaining input data of all corresponding metric factor level values required for calculating the bottom-level index value of each bottom-level index;

[0013] Step 32, obtaining the corresponding metric factor level value according to the evaluation model of each level value in step 2;

[0014] Step 33, mapping all the metric factor level values obtained in step 32 to a preset range to obtain metric factor mapping values by using a suitable normalization method;

[0015] Step 34, synthesizing all the metric factor mapping values required for calculating the bottom-level index value of each bottom-level index to obtain the bottom-level index value of each bottom-level index.

[0016] Compared with the prior art, the present application has at least the following beneficial effects:

[0017] (1) The evaluation method of the present application has low complexity, simple and clear evaluation process, and is easy to be applied in engineering.

[0018] (2) The evaluation method of the present application analyzes and constructs the intelligent level index system from the multiple perspectives of detection, control, resistance, attack and protection, covers all aspects and the whole process of the equipment combat system, is closely integrated with the specific combat task, and has good generalization.

[0019] (3) The evaluation method of the present application analyzes the metric factors of the bottom-level index from the intelligent degree of the model used by the introduced intelligent technology itself as a starting point, and has accurate evaluation results and good interpretability.

[0020] (4) The evaluation method of the present application uses the evaluation model for evaluating the bottom-level index value, which has simple steps, simple implementation, is easy to be integrated into a simulation system or evaluation software, and has good computability and expandability. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application.

[0022] Figure 1 An evaluation method for the application;

[0023] Figure 2 An index system grading index diagram of the equipment combat system intelligent level for the evaluation method of the application;

[0024] Figure 3 A decomposition diagram of the measurement factors of the underlying index for the evaluation method of the application;

[0025] Figure 4 A construction diagram of the intelligent level of early warning detection of each index for the evaluation method of the application;

[0026] Figure 5 A construction diagram of the intelligent level of command and control of each index for the evaluation method of the application;

[0027] Figure 6 A construction diagram of the intelligent level of striking of each index for the evaluation method of the application;

[0028] Figure 7 A construction diagram of the intelligent level of protection of each index for the evaluation method of the application;

[0029] Figure 8 A construction diagram of the intelligent level of support of each index for the evaluation method of the application. DETAILED DESCRIPTION

[0030] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. In addition, the present application can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0031] One specific embodiment of the present application, such as Figures 1-8 , discloses an equipment combat system intelligent level evaluation method, specifically comprising the following steps:

[0032] Step 1, the intelligent level of a plurality of intelligent models used by the equipment combat system is respectively decomposed layer by layer to obtain corresponding intermediate indexes and underlying indexes;

[0033] Among them, the next level index is obtained by decomposing the corresponding upper level index, and the underlying index is an index that cannot be decomposed.

[0034] Further, the equipment combat system is decomposed layer by layer according to the multi-perspective of early warning detection, command control, attack, protection and support, and the complex combat task is decomposed into a plurality of basic task links with clear target and clear process, until the corresponding bottom index is obtained.

[0035] Step 2, for all bottom indexes, analyzing the measurement factors of the corresponding bottom index from the autonomy, explainability, evolvability and intelligent model characteristics of each intelligent model used by the bottom index, and the timeliness, completeness, accuracy and stability of the task involved by the bottom index; constructing an evaluation model of the level value of all measurement factors;

[0036] It can be understood that autonomy is a combination of elements such as the number of environment types to which the intelligent model adapts, the range of environment to which the intelligent model adapts, the probability of normal work, and the frequency of human-machine interaction, and the autonomy reflects the ability of the intelligent model to adapt to the environment and autonomously run. Among them, the number of environment types to which the intelligent model actually works is the number of environment types to which the intelligent model actually works, the range of environment to which the intelligent model normally works in the actual working environment is the ratio of the range of environment to which the intelligent model normally works in the actual working environment to the total range of environment, the probability of normal work is the ratio of the time length of normal work of the intelligent model in the actual working environment to the total time length of work, and the frequency of human-machine interaction is the ratio of the time length of work with human participation of the intelligent model in the actual working environment to the total time length of work.

[0037] The evaluation model of the level value of the measurement factor is respectively:

[0038] The expression of the autonomy level value A of the intelligent model is:

[0039]

[0040] Wherein, N i is the number of environment types to which the intelligent model actually works in the i-th test, i = 1, 2, …, m, and m is the total number of tests; is the range of environment normally worked in the j-th environment of the i-th test, j = 1, 2, …, N i , is the total range of environment worked in the j-th environment of the i-th test; is the time length of normal work in the j-th environment of the i-th test, is the time length of work with human participation in the j-th environment of the i-th test, is the total time length of work in the j-th environment of the i-th test. The autonomy score A has a value range of [0, 1], and the larger the value is, the better.

[0041] The explainability level value I of the intelligent model is:

[0042] The explainability level value I corresponds to 1, 0.5 and 0 respectively according to the level value of the white-box, gray-box or black-box model of the intelligent model; the explainability level value I describes the degree to which the working mode of the intelligent model can be understood by humans, and the larger the value is, the better.

[0043] The evolvability level value E of the intelligent model:

[0044] The evolvability level value E corresponds to 1, 0.5 and 0 respectively according to the level value of the intelligent model having, partially having or not having online evolution using new data; the evolvability level value E describes the ability of the intelligent model to learn new things, and the larger the value is, the better.

[0045] The comprehensive level value of the intelligent model characteristics:

[0046] As shown in Figure 2 , the type of the intelligent model is determined; the comprehensive level value of the intelligent model is obtained according to the type of the intelligent model.

[0047] The type of the intelligent model includes a cognitive-based intelligent model, a search-based intelligent model, an optimization-based intelligent model and a learning-based intelligent model.

[0048] It can be understood that the measurement factors of various types of intelligent models and the evaluation model of the comprehensive level value have great differences, and specifically:

[0049] (1) The evaluation model of the comprehensive level value of the intelligent model characteristics of the cognitive-based intelligent model is:

[0050] The cognitive-based intelligent model includes rule sets, behavior trees, finite state machines and case base reasoning, etc., and is used for decision-making in the combat process. The measurement factors of the cognitive-based intelligent model include the total number of elements, the effective proportion and the utilization rate.

[0051] The total number of elements is:

[0052] The element is a unified description of concepts such as atomic rules of rule sets, behavior nodes of behavior trees, states of finite state machines and cases of case base reasoning. The total number of elements is the total number of elements contained in the cognitive-based intelligent model, which describes the capability limit of the cognitive-based intelligent model. The total number of elements N e The value range is [1, +∞), and the larger the value is, the better.

[0053] The effective proportion level value is:

[0054] The effective proportion is the proportion of the number of elements in the union of active elements to the total number of elements, and the active element refers to an element that is successfully matched or used in a single test. The effective proportion level value η v describes the theoretical effectiveness of the cognitive-based intelligent model, and the expression of the effective proportion level value η

[0055]

[0056] wherein, is the set of effective elements in the i-th test. The effective proportion level value η v The value range is [0, 1], and the larger the value is, the better.

[0057] The utilization level value is:

[0058] The utilization is the average of the proportion of the total number of effective elements to the total number of elements in multiple tests, which describes the actual effectiveness of the cognitive-based intelligent model, and the utilization level value is η u The value range is [0, 1], and the larger the value is, the better, and the expression is:

[0059]

[0060] It can be understood that the explainability level value of the cognitive-based intelligent model is:

[0061] The cognitive-based intelligent model works similarly to human cognition and has good explainability, so the explainability level value of the cognitive-based intelligent model I = 1.

[0062] It can be understood that the evolvability level value of the cognitive-based intelligent model is:

[0063] The cognitive-based intelligent model can correct and save the solution of new problems with the aid of human experience or other technologies, and has certain learning ability, so the evolvability level value of the cognitive-based intelligent model E = 0.5.

[0064] (2) The evaluation model of the comprehensive level value of the intelligent model characteristics of the search-based intelligent model is:

[0065] The search-based intelligent model includes graph search-based methods and sampling-based search methods, the graph search-based methods include dijkstra model and A* model, etc., the sampling-based search methods include rapid random expansion tree RRT model and derivative models of rapid random expansion tree RRT model; the graph search-based methods and the sampling-based search methods obtain results according to given parameters. The measurement factors of the search-based method include completeness level value, optimality level value, coverage level value, solution path length ratio level value and search times.

[0066] The completeness level value is:

[0067] An intelligent model is complete if it can always find the solution when the solution exists and can find that there is no solution when there is no solution in a limited time. An intelligent model is probabilistically complete if the probability of finding the solution converges to 1 as the time tends to infinity when the solution exists. The completeness level C is 1, 0.5 or 0, corresponding to complete, probabilistically complete or non-complete, respectively, and characterizes the ability of an intelligent model to find the solution. s The value range is [0, 1], and the larger the value is, the better.

[0068] Generally, the graph search-based method is complete, corresponding to the completeness level C s = 1; the sampling-based method is probabilistically complete, corresponding to the completeness level C s = 0.5.

[0069] The optimality level is:

[0070] An intelligent model is optimal if the solution obtained by the intelligent model is optimal in the evaluation index (e.g., the length of the solution path). An intelligent model is asymptotically optimal if the solution obtained after a finite number of iterations is a suboptimal solution that is closer to the optimal solution and is closer to the optimal solution after each iteration. The optimality level O is 1, 0.5 or 0, corresponding to optimal, asymptotically optimal or non-optimal, respectively, and characterizes the efficiency of an intelligent model in finding the optimal solution. s The value range is [0, 1], and the larger the value is, the better.

[0071] Generally, the graph search-based method is optimal, corresponding to the optimality level O s = 1; the sampling-based method, the rapidly expanding tree RRT and RRT-connect, is non-optimal, corresponding to the optimality level O s = 0, while the RRT* is asymptotically optimal, corresponding to the optimality level O s = 0.5.

[0072] The coverage level is:

[0073] Coverage uses the ratio of the search region range of an intelligent model to the feasible region range to measure the ability of an intelligent model to search globally. The coverage level η c The value range is [0, 1], and the larger the value is, the better, and the expression is:

[0074]

[0075] wherein, is the search region range in the i-th trial, The feasible region is the range of the i-th trial. The search region range and the feasible region range include the number of discrete nodes, the area of ​​the two-dimensional region, and the volume of the three-dimensional region, etc.

[0076] The ratio of the diameter length to the horizontal value is:

[0077] The solution path length ratio is the ratio of the lengths of the solutions obtained by the intelligent model to the reference optimal solution, characterizing the intelligent model's ability to find the optimal solution. The solution path length ratio level value η p The value range is [1, +∞), and the smaller the value, the better. The expression is:

[0078]

[0079] in, Let be the length of the solution obtained by the intelligent model in the i-th trial. The length of the reference optimal solution in the i-th trial. The meanings of the length of the solution obtained by the intelligent model and the length of the reference optimal solution include discrete node depth, Euclidean distance between two-dimensional and three-dimensional regions, etc.

[0080] Search count:

[0081] The number of searches is the average of the total number of searches performed by the intelligent model across multiple trials, characterizing the efficiency of the intelligent model in solving the problem. The number of searches N s The value range is [0, +∞), and the smaller the value, the better. The expression is:

[0082]

[0083] in, Let be the number of times the intelligent model is searched in the i-th trial.

[0084] Understandably, the interpretability level of a search-based intelligent model is:

[0085] The working method of search-based intelligent models is easy to understand and has good interpretability. Therefore, the interpretability level value of search-based intelligent models is I = 1.

[0086] Understandably, the evolvability level of a search-based intelligent model is:

[0087] Search-based intelligent models typically cannot learn to evolve online, therefore E=0.

[0088] (3) The evaluation model for the comprehensive level value of the intelligent model characteristics based on the optimized intelligent model is as follows:

[0089] The optimization-based intelligent model includes an evolutionary model and a swarm intelligence model, and the evolutionary model includes a single-objective evolutionary model and a multi-objective evolutionary model. Such an intelligent model usually does not generate a parameterized model, but runs a model process according to given parameters to obtain a result, and is commonly used for iterative optimization of a task allocation scheme. The metric factors of the optimization-based intelligent model include completeness, optimality, and coverage, and the definitions of the three factors are consistent with those in the search-based method. In addition, the swarm intelligence model and the single-objective evolutionary model further include factors such as optimization accuracy and optimization stability, and the multi-objective evolutionary model further includes factors such as inverse generational distance (IGD) and hyper-volume index (HV).

[0090] The completeness level value is:

[0091] The optimization-based intelligent model is usually complete, and therefore C s = 1.

[0092] The optimality level value is:

[0093] The optimization-based intelligent model is usually non-optimal, and therefore O s = 0.

[0094] The coverage level value is:

[0095] Coverage uses the ratio of the search region range of the intelligent model to the feasible region range to measure the ability of the intelligent model to search the global. The coverage level value η c ranges from 0 to 1, and the greater the value, the better. The expression is:

[0096]

[0097] wherein, is the search region range in the i-th test, is the feasible region range in the i-th test, and the search region range and the feasible region range include discrete node numbers, areas of two-dimensional regions, and volumes of three-dimensional regions.

[0098] It can be understood that the explainability level value of the optimization-based intelligent model is:

[0099] The optimization-based intelligent model has good explainability in terms of working mode, and therefore the explainability level value I of the optimization-based intelligent model is 1.

[0100] It can be understood that the evolvability level value of the optimization-based intelligent model is:

[0101] The optimization-based intelligent model usually cannot evolve online, and therefore E = 0.

[0102] Further, for swarm intelligence model and single-objective evolutionary model, the optimization accuracy and optimization stability are also included:

[0103] The optimization accuracy level value is:

[0104] The optimization accuracy is measured by the absolute error between the solution obtained by the intelligent model and the actual optimal solution, which describes the ability of the intelligent model to solve the optimal solution. The optimization accuracy level value E a The value range is [0, +∞), and the smaller the value is, the better it is, and the expression is:

[0105]

[0106] Wherein, is the solution obtained by the intelligent model in the ith test, is the actual optimal solution in the ith test.

[0107] The optimization stability level value is:

[0108] The optimization stability is measured by the standard deviation of the optimization accuracy, which describes the robustness of the intelligent model to solve the optimal solution. The optimization stability level E s The value range is [0, +∞), and the smaller the value is, the better it is, and the expression is:

[0109]

[0110] Further, for multi-objective evolutionary model:

[0111] The inverse generation distance IGD level value is:

[0112] The inverse generation distance IGD is the average distance from each reference point on the Pareto approximate frontier to the nearest solution in the solution set, which describes the convergence and diversity of the solution set. The inverse generation distance IGD level value takes the value range [0, +∞), and the smaller the value is, the better it is, and the expression is:

[0113]

[0114]

[0115] Wherein, P i is the preset Pareto approximate frontier in the ith test, is the solution set obtained by the intelligent model in the ith test, and F(*) is the objective function; is the distance from the r1th reference point on the preset Pareto approximate frontier in the ith test to the nearest solution in the solution set; is the r1th reference point on the preset Pareto approximate frontier in the ith test.

[0116] The hyper-volume indicator HV level value is:

[0117] The hyper-volume indicator HV is the space size of the union of the non-dominated solutions in the solution set and the hypercube formed by the reference points as the diagonal, which characterizes the convergence and diversity of the solution set. The hyper-volume indicator HV level value ranges from [0, +∞), and the larger the value is, the better it is. The expression is:

[0118]

[0119]

[0120] wherein n is the dimension of the objective function, is the solution set of the i-th test is the r2-th dominated solution of ref is the reference point related to the nadir point, is the k-th dimension of the reference point, H(*) is the unit step function, is the hypercube formed by and z ref as the diagonal, F k (*) is the k-th dimension of the objective function.

[0121] (4) The evaluation model of the comprehensive level value of the intelligent model characteristics of the learning-based intelligent model is:

[0122] The learning-based intelligent model includes a machine learning intelligent model, preferably a deep learning intelligent model. Such an intelligent model usually generates a parameterized intelligent model, which is used according to the intelligent model process when facing new data to obtain results, and is commonly used for classification, regression and other tasks. The learning-based intelligent model includes a supervised learning intelligent model, an unsupervised learning intelligent model and a reinforcement learning intelligent model. The supervised learning intelligent model includes intelligent models for classification, regression, object detection, single target tracking and multi-target tracking. The unsupervised learning intelligent model includes intelligent models for clustering and dimensionality reduction. The reinforcement learning intelligent model includes intelligent models for action value and policy gradient. The measurement factors of these intelligent models include dataset size, interpretability and evolvability. In addition, models for different task types have their own measurement factors.

[0123] The dataset size level value is:

[0124] The dataset size is the number of data samples used to train the intelligent model, which characterizes the boundary of the generalization performance of the intelligent model. The dataset size level value is N d , and the larger the value is, the better it is.

[0125] It can be understood that the interpretability level value of the learning-based intelligent model is:

[0126] The learning-based intelligent model includes a neural network model, a decision tree model, a support vector machine model, and a Bayesian network model, etc. The decision tree model, the support vector machine model, and the Bayesian network model have good interpretability, so I = 1; the neural network model has poor interpretability, so I = 0.

[0127] It can be understood that the evolvability level value of the optimization-based intelligent model is:

[0128] The learning-based intelligent model can generally be combined with online learning to obtain the ability of online learning evolution, so E = 1.

[0129] The learning-based intelligent model for different task types respectively includes its respective measurement factors:

[0130] a. For the intelligent model for classification of the supervised learning intelligent model:

[0131] The F1-score level value is:

[0132] The F1-score is the F1 score of the intelligent model for classification, which describes the accuracy of the intelligent model classification. The F1 level value is in the range of [0, 1], and the larger the value is, the better it is, and the expression is:

[0133]

[0134] Wherein, TP i is the number of true cases in the i-th test, FP i is the number of false positives in the i-th test, and FN i is the number of false negatives in the i-th test.

[0135] The AUC area level value is:

[0136] The AUC area is the area under the ROC curve of the intelligent model for classification, which describes the accuracy of the intelligent model classification. The AUC area is in the range of [0, 1], and the larger the value is, the better it is, and the expression is:

[0137]

[0138] Wherein, AUC i is the AUC score in the i-th test.

[0139] b. For the intelligent model for regression of the supervised learning intelligent model:

[0140] The R-squared fitting degree level value is:

[0141] The R-squared goodness-of-fit is measured using the sum of squared errors between predicted and true labels and the sample variance, and characterizes the goodness of fit of the regression intelligent model to the data regularity. The R-squared goodness-of-fit level value ranges from (-∞, 1], and the larger the value is, the better, and the expression is:

[0142]

[0143]

[0144] wherein n i is the number of data samples in the ith experiment, is the true label of the j1th sample in the ith experiment, is the predicted label of the j1th sample in the ith experiment.

[0145] The error MAPE level value is:

[0146] The error MAPE is measured using the difference between the sample true label and the predicted label and the true label, and characterizes the relative error of the regression intelligent model. The MAPE value ranges from [0, +∞), and the smaller the value is, the better, and the expression is:

[0147]

[0148] c. For the intelligent model of the supervised learning intelligent model for target detection:

[0149] The IoU accuracy level value is:

[0150] The IoU accuracy is the intersection over union of the detection box and the true box, and characterizes the accuracy of the intelligent model detection. The IoU accuracy level value ranges from [0, 1], and the larger the value is, the better, and the expression is:

[0151]

[0152] wherein is the total number of detection boxes in the ith experiment, is the total number of true boxes in the ith experiment, is the area of the detection box matched with the j2th true box in the ith experiment, and is 0 if there is no match, is the area of the j2th true box in the ith experiment.

[0153] The mAP accuracy level value is:

[0154] The mAP accuracy is the accuracy of the classification of the detection intelligent model. The mAP accuracy level value ranges from [0, 1], and the larger the value is, the better, and the expression is:

[0155]

[0156] wherein mAP i is the mAP score in the ith experiment.

[0157] d. For the intelligent model for target tracking of the supervised learning intelligent model:

[0158] The AOR intersection-over-union value level is:

[0159] The AOR intersection-over-union is the intersection-over-union of the predicted box and the real box, which characterizes the accuracy of the tracking model. The AOR intersection-over-union value level takes a value in the range [0, 1], and the larger the value is, the better it is, and the expression is:

[0160]

[0161] wherein, is the total number of frames in the ith experiment, is the area of the ith experiment that the predicted box matched with the j2th real box intersects with, is the area of the j2th real box in the ith experiment.

[0162] The AUC1 area level value is:

[0163] The AUC1 area is the area under the success rate curve of the tracking model, and the success rate curve is a curve with the intersection-over-union threshold as the abscissa and the success rate (the proportion of predicted boxes greater than the intersection-over-union threshold) as the ordinate, which characterizes the accuracy of the tracking intelligent model. The AUC1 area level value takes a value in the range [0, 1], and the larger the value is, the better it is, and the expression is:

[0164]

[0165] wherein AUC1 i is the AUC1 score in the ith experiment.

[0166] e. For the intelligent model for clustering of the unsupervised learning intelligent model:

[0167] The purity level value is:

[0168] The purity is the ratio of the number of correct samples to the total number of samples, which characterizes the accuracy of the intelligent model clustering. The purity P u level value takes a value in the range [0, 1], and the larger the value is, the better it is, and the expression is:

[0169]

[0170] wherein n i1 is the number of clusters in the ith1 experiment, is the set of samples of the jth3 cluster in the clustering result of the ith experiment, is the set of real samples of the kth1 cluster in the ith experiment.

[0171] The horizontal value of the silhouette coefficient is:

[0172] The silhouette coefficient uses the intra-cluster dissimilarity and the inter-cluster dissimilarity of the clustering result to measure the rationality of the clustering, and the horizontal value of the silhouette coefficient S c The value range is [-1, 1], and the larger the value is, the better it is, and the expression is:

[0173]

[0174]

[0175]

[0176] wherein, is the tth1 sample of the jth3 cluster in the clustering result of the ith experiment; is the sth sample of the set of samples of the jth3 cluster in the clustering result of the ith experiment; is the set of samples of the kth2 cluster in the clustering result of the ith experiment; is the sth sample of the set of samples of the kth3 cluster in the clustering result of the ith experiment.

[0177] f. For the dimension reduction intelligent model of the unsupervised learning intelligent model:

[0178] The horizontal value of the remaining dimension ratio is:

[0179] The remaining dimension ratio is the ratio of the dimension of the data after dimension reduction to the dimension of the original data, which describes the size of the dimension reduction. The horizontal value of the remaining dimension ratio η d The value range is (0, 1], and the smaller the value is, the better it is, and the expression is:

[0180]

[0181] wherein, is the dimension of the data after dimension reduction in the ith experiment, is the dimension of the original data in the ith experiment.

[0182] The horizontal value of the cumulative variance contribution rate is:

[0183] The cumulative variance contribution rate is the mean of the ratio of the variance of the data after dimension reduction to the variance of the original data, which describes the size of the information retention. The horizontal value of the cumulative variance contribution rate η s The value range is (0, 1], and the larger the value is, the better it is, and the expression is:

[0184]

[0185] wherein, is the variance of the jth 4-dimensional data in the ith trial.

[0186] g. For the reinforcement learning intelligent model:

[0187] The training step number level value is:

[0188] The training step number is the total training step number of the reinforcement learning model, which characterizes the limit of the generalization performance of the model. The training step number level value N t The value range is [0, +∞), and the larger the value is, the better.

[0189] Single session interaction frequency:

[0190] The single session interaction frequency is the average of the interaction frequency of the reinforcement learning model in multiple trials, which characterizes the ability of the intelligent model to make optimal decisions. The single session interaction frequency N a The value range is [1, +∞), and the larger the value is, the better, if the task requires “continuous performance”, and the smaller the value is, the better, if the task requires “ending as soon as possible”, the expression is:

[0191]

[0192] wherein, is the interaction frequency in the ith trial.

[0193] Single session reward level value:

[0194] The single session reward level value is the average of the discounted reward obtained by the reinforcement learning model in multiple trials, which characterizes the ability of the model to make optimal decisions. The single session reward level value R e The value range is (-∞, +∞), and the larger the value is, the better, the expression is:

[0195]

[0196] wherein, γ is the reward discount factor of the reinforcement learning model, γ t2 is the t2 power of γ, r t2 is the immediate reward value of the t2 step.

[0197] Step 3, obtaining the bottom-level index value of each bottom-level index based on the calculation method of the level value of the measurement factor in step 2;

[0198] Step 31, obtaining the input data of all corresponding measurement factor level values required for calculating the bottom-level index value of each bottom-level index;

[0199] Step 32, obtaining the corresponding measurement factor level value according to the calculation method of each measurement factor in step 2;

[0200] Step 33, mapping all the metric factor level values obtained in step 32 to a preset range (such as 0 to 1) to obtain metric factor mapping values using a suitable normalization method (such as a linear mapping quantization method);

[0201] Step 34, linearly weighting and synthesizing all the metric factor mapping values required for calculating the bottom layer index value of each bottom layer index to obtain the bottom layer index value of each bottom layer index.

[0202] Step 4, based on all the bottom layer index values, calculating the index value of each level in the equipment combat system by a comprehensive evaluation method such as analytic hierarchy process, fuzzy evaluation method or power index method; obtaining the intelligent evaluation value of the equipment combat system from the index value of each level.

[0203] In order to illustrate the effectiveness of the method of the present application, the above technical solutions of the present application are described in detail below through a specific embodiment, and the specific implementation steps are as follows:

[0204] Step 1, as shown in the figure, the intelligent level of the equipment combat system is the overall evaluation target, and the intelligent level of the equipment combat system is decomposed into multiple levels of indexes; each upper level index is decomposed into a corresponding lower level index until the lower level index cannot be decomposed, to obtain the bottom layer index.

[0205] Among them, the intelligent level of the equipment combat system is the first node, and the early warning reconnaissance intelligent level, the command control intelligent level, the protection intelligent level, the strike intelligent level and the support intelligent level are introduced as the first level indexes;

[0206] The early warning reconnaissance intelligent level refers to the intelligent level of using various detectors on land, sea, air and sky to detect outer space targets, atmospheric targets, water surface and underwater targets and land targets, and to uniformly plan and distribute information so that each branch can timely share and obtain complete and accurate information;

[0207] The command control intelligent level refers to the intelligent level of relying on an integrated command information system, the command center plans and coordinates the combat actions of each branch of equipment in the combat preparation and combat process according to the combat purpose;

[0208] The protection intelligent level refers to the intelligent level of effectively preventing the detection of the equipment combat system, avoiding damage when detected, and actively intercepting enemy attacks;

[0209] The strike intelligent level refers to the intelligent level of the equipment combat system under the decision command of the command center, intercepting, destroying or paralyzing enemy command posts, aircraft and / or missile buildings or fire equipment to make them lose combat capability;

[0210] The intelligence level of protection refers to the intelligence level of actions implemented by command agencies at all levels to ensure the safety of our side and the smooth completion of tasks.

[0211] Further, taking the first-level index as the second root node, the second-level index of the first-level index is introduced.

[0212] The second-level index of the intelligence level of early warning reconnaissance includes the intelligence level of information acquisition, the intelligence level of information processing, and the intelligence level of information transmission.

[0213] The second-level index of the intelligence level of command and control includes the intelligence level of state and posture acquisition, the intelligence level of intelligence analysis, the intelligence level of command decision, and the intelligence level of action implementation.

[0214] The second-level index of the intelligence level of protection includes the intelligence level of position protection and the intelligence level of equipment protection.

[0215] The second-level index of the intelligence level of attack includes the intelligence level of navigation and guidance and the intelligence level of hit and damage.

[0216] The second-level index of the intelligence level of protection includes the intelligence level of combat support, the intelligence level of logistics support, and the intelligence level of equipment support.

[0217] Further, taking the second-level index as the third root node, the third-level index (bottom-level index) of the second-level index is introduced.

[0218] Specifically, the intelligence level of information acquisition in the second-level index includes the intelligence level of detection and discovery, the intelligence level of target identification, the intelligence level of target positioning, and the intelligence level of target tracking.

[0219] The intelligence level of information processing in the second-level index includes the intelligence level of multi-source information fusion, the intelligence level of trajectory prediction, the intelligence level of threat assessment, and the intelligence level of damage assessment.

[0220] The intelligence level of information transmission in the second-level index includes the intelligence level of communication routing and the intelligence level of anti-interference.

[0221] The bottom-level index of other second-level indexes is described in Figures 4-8 .

[0222] Step 2: Taking the bottom-level index of detection and discovery intelligence level as an example, the detection and discovery process usually uses a classification algorithm to determine whether the echo signal contains a target. Here, it is assumed that a classification algorithm based on a deep neural network is used, and all the measurement factors of this bottom-level index are obtained.

[0223] Step 21: Since a classification algorithm is used, the measurement factors related to the characteristics of the intelligent model are: data set size, F1-score, and AUC, and the autonomy can be calculated according to the aforementioned expression.

[0224] Step 22: Since the classification algorithm is based on a deep neural network, its interpretability I = 0, evolvability E = 1.

[0225] Step 3, take the bottom index detection found intelligent level and its measurement factor F1-score as an example to build a basic evaluation model. According to the idea of the example, the basic evaluation model of other bottom indexes can be obtained.

[0226] Step 31: Take the detection found intelligent level as an example;

[0227] Step 32: Take F1-score as an example;

[0228] Step 33: Calculate the data required for the bottom index value, such as

[0229] Table 1 shows;

[0230] Table 1 Basic evaluation model input data of detection found intelligent level

[0231]

[0232] Step 34: Take the average of the F1-score of the classification network in all sensors, and the calculation formula is as follows

[0233]

[0234] Where F1 k4 is the F1-score of the k4th sensor, and the calculation method is as shown above.

[0235] Assume that the data in a certain test is:

[0236] N = 2, m = 3,

[0237]

[0238] Then the calculated F1-score is F1 = 0.9430;

[0239] Step 35: When F1-score is x0 = 0.5, the performance of the classification network is equivalent to that of a random classifier, so it can be considered as very poor, mapped to the minimum value 0, when F1-score reaches x1 = 1.0, it is considered to be good enough, mapped to the maximum value 1, the mapping formula is:

[0240]

[0241] Where x0 is …, x1 is …;

[0242] The normalized value obtained by mapping is

[0243] Step 36: assuming that all the metric factors are calculated, jump to step 307;

[0244] Step 37: assuming that the normalized values of all the metric factors are obtained, according to the characteristics of the detection discovery process, different proportion factors are given to the metric factors, and the linear summation model is used for synthesis, the normalized values and the proportion factors of the metric factors are shown in Table 2, and the calculated underlying index value is 0.7490;

[0245] Table 2: Normalized values and proportion factors of the metric factors of the intelligent level of detection discovery

[0246] metric factor dataset size F1-score AUC autonomy explainability evolvability normalized value 0.7611 0.8860 0.7324 0.8652 0 1 scaling factor 0.2 0.2 0.25 0.1 0.1 0.15

[0247] Step 38: based on the underlying index value, the index values of each level in the equipment combat system are calculated by using a comprehensive evaluation method such as the analytic hierarchy process, fuzzy evaluation method or power index method, and the intelligent evaluation value of the equipment combat system is obtained from the index values of each level.

[0248] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. An equipment combat system intelligent level evaluation method, characterized in that, Specifically comprising the following steps: Step 1, respectively decompose the intelligent level of a plurality of intelligent models used by the combat system to obtain corresponding intermediate indicators and bottom indicators; Wherein, when the corresponding intermediate indicators and bottom indicators are obtained, the next level indicators are obtained by decomposing the corresponding upper level indicators, and the bottom indicators are indicators that cannot be further decomposed; Step 2, for all bottom indicators, analyze the measurement factors of the corresponding bottom indicators from the autonomy, explainability, evolvability and intelligent model characteristics of each intelligent model used by the bottom indicators, and the timeliness, completeness, accuracy and stability of the tasks involved in the bottom indicators; construct an evaluation model of the level value of all measurement factors; Wherein, the expression of the autonomy level value A of the intelligent model is: wherein N i is the number of environment categories in which the intelligent model actually works in the i-th test, i = 1, 2, …, m, and m is the total number of tests; is the size of the environment range in which the intelligent model works normally in the j-th environment in the i-th test, j = 1, 2, …, N i , is the total size of the environment range in which the intelligent model works in the j-th environment in the i-th test; is the length of time in which the intelligent model works normally in the j-th environment in the i-th test, is the length of time in which the intelligent model works with human participation in the j-th environment in the i-th test, is the total length of time in which the intelligent model works in the j-th environment in the i-th test; Step 3, obtain the bottom indicator value of each bottom indicator based on the evaluation model of the level value of the measurement factors in step 2; Step 4, based on all bottom indicator values, obtain the indicator value of each layer indicator in the combat system by a comprehensive evaluation method; obtain the intelligent evaluation value of the combat system from the indicator value of each layer indicator.

2. The method of claim 1, wherein, The specific steps of step 3 are as follows: Step 31, obtain the input data of all corresponding measurement factor level values required to calculate the bottom indicator value of each bottom indicator; Step 32, obtain the corresponding measurement factor level value according to the evaluation model of each level value in step 2; Step 33, use a suitable normalization method to map all measurement factor level values obtained in step 32 to a preset range to obtain measurement factor mapping values; Step 34, synthesize all measurement factor mapping values required to calculate the bottom indicator value of each bottom indicator to obtain the bottom indicator value of each bottom indicator.

Images