Intelligent system intelligence level evaluation method based on multi-capability index fusion

Abstract

The invention provides an intelligent level evaluation method for an intelligent system based on multi-ability index fusion, and an index system for evaluating the intelligent level of the intelligent system is constructed by means of an OODA behavior model (including four links of perception, analysis and judgment, decision making and action) based on the existing unmanned intelligent system research. According to the invention, comprehensive technical design and implementation are carried out for intelligent grade evaluation of an intelligent system, and a unified intelligent grade evaluation framework of the intelligent system is constructed through four aspects of construction of an evaluation index system, measurement of bottom layer indexes, index weight distribution, design of a grade evaluation method and display of an evaluation result. And comprehensive and efficient support is provided for intelligent evaluation of an intelligent system.

Description

Technical Field

[0001] The present invention relates to the field of intelligent system intelligence level evaluation, and specifically to an intelligent system intelligence level evaluation method based on the fusion of multiple capability indicators. Background Art

[0002] At present, the development of intelligent systems has gone through decades of exploration. From the initial mechanization and informatization to today's intelligence, technological progress has brought about a qualitative leap. With the application of technologies such as deep learning, neural networks, and big data analysis, intelligent systems no longer rely solely on pre-programmed rules, but can conduct autonomous learning, prediction, and decision-making in a data-driven way. Scientifically evaluating the intelligence level of intelligent systems is not only to judge their technological advancement, but also to ensure that the system can effectively cope with complex environments and improve efficiency in practical applications. In this field, China has proposed a number of intelligent level evaluation methods, such as the level method, the dual coordinate axis method, and the spider web model, and has formed an evaluation framework with autonomy, security, and synergy as the core. In addition, domestic scholars have further evaluated the availability, credibility, and effectiveness of intelligent systems through a variety of methods (such as ADC model, hierarchical analysis method, gray system theory, etc.). Foreign research on intelligent system evaluation is at the forefront. For example, the United States has proposed an evaluation framework based on multi-dimensional indicators such as perception, cognitive decision-making, and autonomous capabilities, and attaches importance to the response and adaptability of the system in complex environments. Foreign countries also emphasize the evaluation of the adversarial and robustness of intelligent systems. In addition, Europe, Israel and other countries are promoting the internationalization of intelligent assessment standards through cross-border projects.

[0003] As the level of intelligence increases, the system is no longer a simple tool, but an "intelligent agent" with a certain degree of autonomy and adaptability. However, the traditional evaluation method of the intelligence level of intelligent systems mostly relies on functional testing and performance, while ignoring the characteristics of intelligent systems in terms of perception, cognition, decision-making and action capabilities. In addition, there is a lack of unified standards and systems for how these systems perform in actual applications and how to quantify their intelligence levels.

[0004] At the same time, although domestic evaluation methods have achieved good results in the construction of indicator systems, if they are directly used in the evaluation of the intelligent capabilities of intelligent systems, they are still not enough to be fully summarized. In addition, domestic research focuses more on building test sites and simulated test environments, focusing on the research and development of system platforms. The research on the testing and evaluation technology of the system is still insufficient, and a complete research system and method have not been formed. In addition, the current testing and evaluation system still has problems of lack of systematicity and scientificity. Therefore, it is necessary to propose a scientific, systematic, and quantifiable intelligent system intelligence level evaluation method. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes an intelligent system intelligence level evaluation method based on the fusion of multiple capability indicators. This patent will carry out comprehensive technical design and implementation for the intelligent level evaluation of intelligent systems. Through the construction of evaluation index system, measurement of underlying indicators, allocation of indicator weights, design of level evaluation method and display of evaluation results, a unified intelligent system intelligence level evaluation framework is constructed to provide comprehensive and efficient support for the intelligent evaluation of intelligent systems.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for evaluating the intelligence level of an intelligent system based on the fusion of multiple capability indicators includes the following steps:

[0008] Step 1: Construction of indicator system based on OODA theory;

[0009] Step 2: indicator weight allocation;

[0010] Step three: Intelligent level evaluation based on multi-indicator fusion.

[0011] As a further improvement of the present invention, the step 1 is to construct an indicator system based on the OODA theory, which is specifically as follows:

[0012] 1.OODA modeling theory;

[0013] OODA consists of four parts: perception, analysis and judgment, decision-making and action, among which:

[0014] (1) Perception: extracting data features;

[0015] (2) Cognition: cognition of data characteristics;

[0016] (3) Decision-making: making decisions in different environments;

[0017] (4) Action: Acting in a realistic and complex environment;

[0018] Decompose the capabilities of intelligent systems into four links: intelligent perception, intelligent cognition, intelligent decision-making and intelligent action;

[0019] 2. Construction of indicator evaluation system

[0020] Based on the four indicator system principles of core focus, taking into account characteristics, comprehensiveness and scalability, each capability layer is subdivided from four dimensions: autonomy, collaboration, learning and security. At the same time, the technical layer is divided according to relevant documents, and then the indicator system for evaluating the intelligent capabilities of intelligent systems is obtained. The technical layer is divided into autonomy, collaboration, learning and security. Each layer has corresponding underlying indicators to reflect the intelligent capabilities of the intelligent system.

[0021] As a further improvement of the present invention, the perception technology capability layer is specifically as follows:

[0022] (1) Perception capability technology layer;

[0023] By decomposing the intelligence dimensions of the intelligent system, the perception capability is divided into autonomy, collaboration, learning and safety. The technical layer corresponds to four underlying indicators: feature vector clustering, feature vector similarity, training convergence speed and adversarial training performance improvement.

[0024] 1) Autonomy of perception:

[0025] Meaning: Whether it is possible to perceive data features autonomously;

[0026] Measurement: Clustering evaluation of feature vectors. During the training process of the intelligent system, the intelligent system extracts the features of the training data to form feature vectors. By clustering the feature vectors and evaluating the clustering effect, the quality of the feature vectors perceived by the intelligent system can be evaluated.

[0027] Use the dimensionality reduction algorithm to reduce the dimension of the feature vector, then use the clustering algorithm to cluster, and finally use the clustering evaluation algorithm to evaluate the clustering effect. The dimensionality reduction algorithm used is TSNE, the clustering algorithm used is KMeans algorithm, and the clustering effect evaluation method used is silhouette coefficient;

[0028] Calculation formula:

[0029]

[0030] Where a(i) represents the average distance from sample i to other samples in its cluster, i.e., intra-cluster dissimilarity, b(i) represents the average distance from sample i to the nearest cluster, i.e., inter-cluster dissimilarity, and N represents the total number of training data samples. m1 is what this metric seeks;

[0031] Reasonable value range: the value range of the silhouette coefficient. Silhouette is a clustering algorithm effect evaluation indicator used to evaluate the compactness and separation of clustering results. Its value range is [-1, 1]. The closer the index is to 1, the better the perception ability autonomy.

[0032] 2) Synergy of perception capabilities;

[0033] Meaning: Whether it can collaborate with other intelligent systems to perceive data features;

[0034] Measurement: Similarity of feature vectors. The similarity between the feature vectors extracted from the system under test and the baseline system is used to characterize the degree of coordination between the system under test and the baseline system. The higher the similarity, the better the coordination between the systems under test. The cosine similarity value is used to calculate the similarity between feature vectors.

[0035] Calculation formula:

[0036]

[0037] Where F1 and F2 represent the feature vector perceived by the model to be tested and the feature vector perceived by the baseline model, respectively, i represents the i-th training data, N represents the number of samples in the training set, and m2 is what the metric requires;

[0038] Reasonable value range: the value range of cosine similarity. The value range of cosine similarity is [-1,1]. The closer the value of cosine similarity is to 1, the more consistent the directions of the two vectors are, and the more similar they are. Therefore, the closer this indicator is to 1, the better the synergy of perception ability is.

[0039] 3) Learnability of Perception Ability:

[0040] Meaning: Whether the perceived data features can be learned quickly;

[0041] Metrics: Training convergence speed. Intelligent systems usually improve their ability to perceive data features by learning from input data. Therefore, the convergence speed of the intelligent system during training can characterize the learning of the perception ability. The convergence speed of the loss value during training is used to measure the learning of the intelligent system's perception ability.

[0042] Calculation formula:

[0043]

[0044] Where L(t) represents the loss function value, T represents the total number of iterations, and t represents the current iteration step. m3 is what the metric is looking for;

[0045] Reasonable value range: the value range of the convergence speed of the loss value. The value range is (-∞,1], and the closer the index is to 1, the better the learning ability of perception is;

[0046] 4) Security of perception ability:

[0047] Meaning: Whether the characteristics of the data can be sensed safely;

[0048] Measurement: Adversarial training performance improvement. Use adversarial samples to retrain the intelligent system. Statistical analysis of the performance improvement of the intelligent system on adversarial samples before and after training to reflect the security of the system's perception capabilities.

[0049] Calculation formula:

[0050] m4=ACC adv -ACC origin

[0051] ACC adv ACC represents the accuracy of the model on adversarial samples after adversarial training. origin It represents the accuracy of the original model in adversarial samples, and m4 is what this metric requires;

[0052] Reasonable value range: the value range is [0,ACC adv ], the closer this indicator is to ACC adv This indicates that the better the perceived safety is;

[0053] (2) Cognitive capability technology layer;

[0054] 1) Autonomy of cognitive abilities:

[0055] Meaning: Whether the perceived data features can be recognized autonomously;

[0056] Metrics: Feature contribution, which is the degree of influence of the features of the data itself on the prediction results when the intelligent system predicts the data. The greater the performance difference, the greater the contribution of the features of the data itself to the prediction results of the intelligent system, which means that the intelligent system is too dependent on the appearance of the data and its cognitive ability is weaker.

[0057] Calculation formula:

[0058] m5=ACC origin -ACC masked

[0059] ACC origin Indicates the accuracy of the original model in the test set, ACC masked It represents the accuracy of the original model on the test set with mask added, and m5 is what this metric seeks;

[0060] Reasonable value range: the value range is [0,ACC origin ] The closer this indicator is to ACC origin This indicates that the greater the autonomy of cognitive ability, the better;

[0061] 2) Synergy of cognitive abilities:

[0062] Meaning: Can the system's ability to recognize data features be improved with the assistance of other intelligent systems?

[0063] Metrics: Performance improvement with the assistance of other models;

[0064] Calculation formula:

[0065] m6=ACC Ch atGLM -ACC origin

[0066] ACC Ch atGLM It indicates the accuracy of the model on the enhanced data after the data is enhanced by ChaGLM, AC Corigin It represents the accuracy of the model on the original data, and m6 is what this metric seeks;

[0067] Reasonable value range: the value range is [0,ACC Ch atGLM ] The closer this indicator is to ACC Ch atGLM This indicates that the better the synergy of cognitive abilities;

[0068] 3) Learnability of cognitive abilities:

[0069] Meaning: Whether the cognitive ability of the system can be steadily improved during the model training process;

[0070] Measurement: Smoothness of loss decline. The intelligent system improves its ability to recognize data features by learning from input data. The smoothness of the loss curve during training is used to measure the learning ability of the intelligent system's cognitive ability. That is, the faster the smoothness, the stronger the learning ability of the cognitive ability.

[0071] Calculation formula:

[0072]

[0073] where L(t) is the change in loss function over time or training steps, is the first-order derivative of the loss function, Var represents the variance, and m7 is what the metric seeks;

[0074] Reasonable value range: The value range is [0, +∞]. The closer the index is to 0, the better the cognitive ability and learning ability.

[0075] 4) Cognitive safety:

[0076] Meaning: Whether the characteristics of the data can be recognized safely;

[0077] Measurement: Adversarial attack tolerance. Use the performance improvement of adversarial training to characterize the security of cognitive ability. Set a threshold for the success rate of attacks on intelligent systems. Adjust the perturbation value in the process of generating adversarial samples by adversarial attacks until the success rate of adversarial attacks exceeds or reaches the threshold. The perturbation value is the measurement indicator of adversarial attack tolerance. The greater the adversarial attack tolerance, the higher the security of the system's cognitive ability.

[0078] Calculation formula:

[0079]

[0080] Where x is the original input, δ is the perturbation, f is the output function of the model, and m8 is the metric required;

[0081] Reasonable value range: the value range of δ. The range of δ is set to [0.1, 0.9]. The closer this index is to 0.9, the better the cognitive ability security is;

[0082] (3) Decision-making capability technical layer;

[0083] 1) Autonomy of decision-making ability:

[0084] Meaning: Whether the perceived data features can be recognized autonomously;

[0085] Measurement: Decision accuracy. The autonomy of decision-making ability is measured by the accuracy of the statistical intelligent system on the test set. That is, the higher the accuracy of the decision, the stronger the autonomy of the decision-making ability;

[0086] Calculation formula:

[0087]

[0088] Where N is the total number of test set samples, y i is the true label of the i-th sample, is the predicted label of the ith sample, It means when y i equal 1 when , otherwise 0. m9 is the metric required;

[0089] Reasonable value range: The value range is [0,1]. The closer the index is to 1, the better the decision-making autonomy is.

[0090] 2) Synergy of decision-making ability:

[0091] Meaning: Can it make decisions in collaboration with other intelligent systems in a decision-making environment?

[0092] Measurement: Similarity of feature vectors in the decision environment. The similarity between the feature vectors extracted from the test system and the baseline system in the test set is used to characterize the degree of coordination between the test system and the baseline system, and the cosine similarity value is used to calculate the similarity between feature vectors.

[0093] Calculation formula:

[0094]

[0095] Where F1 and F2 represent the feature vector perceived by the model to be tested and the feature vector perceived by the baseline model, respectively, i represents the i-th test data, and N represents the number of samples in the test set.10 That is what the measure seeks;

[0096] Reasonable value range: the value range of cosine similarity. Therefore, the closer this index is to 1, the better the decision-making ability coordination is.

[0097] 3) Learnability of decision-making ability:

[0098] Meaning: Whether the decision-making ability of the system can be quickly improved during the model training process;

[0099] Metrics: Accuracy convergence speed in training environment;

[0100] Calculation formula:

[0101]

[0102] Where A(t) represents the accuracy function value, T represents the total number of iterations, t represents the current iteration step, and m 11 That is what the measure seeks;

[0103] Reasonable value range: The value range is (-∞,1]. The closer the index is to 1, the better the decision-making ability is.

[0104] 4) Security of decision-making ability:

[0105] Meaning: Can decisions be made safely on input data?

[0106] Metric: Accuracy at a specific confidence level. The higher the accuracy at a specific confidence level, the higher the decision safety of the intelligent system.

[0107] Calculation formula:

[0108] wh en confidence=th resh old

[0109] Where N is the total number of test set samples, y i is the true label of the i-th sample, is the predicted label of the ith sample, It means when y i equal 1 when the confidence level is 0, otherwise, th resh old is the confidence threshold to be specified, which is used to characterize the accuracy under a specific confidence level. 12 That is what the measure seeks;

[0110] Reasonable value range: The value range is [0,1]. The closer the index is to 1, the better the decision-making ability security is;

[0111] (4) Action capability technology layer;

[0112] 1) Autonomy of action:

[0113] Meaning: Whether the person can autonomously complete actions in a complex real-life action environment;

[0114] Calculation formula:

[0115]

[0116] Where N is the total number of test set samples with added noise, y i is the true label of the i-th sample, is the predicted label of the ith sample, It means when y i equal 1 when it is, otherwise 0, in order to measure the accuracy of the model to be tested on the noise data set, m 13 That is what the measure seeks;

[0117] Reasonable value range: The value range is [0,1]. The closer the index is to 1, the better the autonomy of action ability.

[0118] 2) Synergy of action planning capabilities:

[0119] Meaning: Can it collaborate with other intelligent systems to make predictions in action?

[0120] Measurement: Jaccard correlation coefficient of the decision of the system under test and the baseline system. The intelligent system will predict the input data and generate prediction labels. The correlation coefficient between the prediction labels generated by the system under test and the baseline system is used to represent it. The Jaccard correlation coefficient between the prediction labels is used to measure it. The larger the Jaccard value, the stronger the coordination of action capabilities.

[0121] Calculation formula:

[0122]

[0123] Among them, M o ∩M i Indicates that the model to be tested and the baseline model predict the same label for the sample, M o ∪M i Indicates that both models have at least one predicted sample, m 14 That is what the measure seeks;

[0124] Reasonable value range: the value range of the Jaccard correlation coefficient. The value range of the Jaccard correlation coefficient is [0,1]. The closer the index is to 1, the better the coordination of action capabilities.

[0125] 3) Learning of action ability:

[0126] Meaning: Can the system's action error be quickly reduced during the model training process using a real and complex training set?

[0127] Measurement: The convergence speed of training loss in a noisy environment. The intelligent system is trained on a noisy training set. The convergence speed of the training loss curve during the training process is used to measure the learning ability of the action ability. That is, the faster the convergence speed, the stronger the learning ability of the action ability.

[0128] Calculation formula:

[0129]

[0130] Where L(t) represents the loss function value of the model to be tested under the noise data set, T represents the total number of iterations, t represents the current iteration step, and m 15 That is what the measure seeks;

[0131] Reasonable value range: The value range is (-∞,1]. The closer the index is to 1, the better the learning ability of action is.

[0132] 4) Safety of mobility:

[0133] Meaning: Can you make safe action decisions in complex real-world environments?

[0134] Metrics: Accuracy in adversarial environments. Use adversarial attack techniques to generate adversarial samples, and then count the prediction accuracy of the intelligent system on the adversarial sample set. The higher the accuracy, the higher the security of the intelligent system's action capability.

[0135] Calculation formula:

[0136]

[0137] Where N is the total number of samples in the adversarial sample set, y i is the true label of the i-th sample, is the predicted label of the ith sample, It means when y i equal 1 when it is, otherwise 0, m 16 That is what the measure seeks;

[0138] Reasonable value range: The value range is [0,1]. The closer the indicator is to 1, the better the safety of the action ability.

[0139] As a further improvement of the present invention, step 2, indicator weight allocation, is as follows:

[0140] During the evaluation process, the importance of each indicator varies depending on the situation. By using three weighting algorithms, namely CRITIC weighting method, entropy weight method and combined weighting method, the weight of each indicator is reasonably determined to ensure the accuracy and fairness of the evaluation.

[0141] As a further improvement of the present invention, the CRITIC weighting method is specifically as follows:

[0142] Assume that there are n intelligent systems, and each system to be tested has p evaluation indicators, as shown in the following formula. ij Represents the value of the jth evaluation index of the i-th sample;

[0143]

[0144] 1) Index normalization processing:

[0145] In order to avoid inaccurate results due to dimension differences, each indicator needs to be dimensionless so that different indicators are comparable in the same evaluation system. For extremely large indicators, the standard deviation method is usually used for dimensionless processing. This method converts the indicator into a dimensionless value by calculating the deviation of each indicator value from the average value, so that it can be compared under a unified standard. The specific calculation formula is as follows;

[0146]

[0147] Where: K1 is the dimensionless value of the extremely large index; x i is the actual value of the indicator; x max The maximum value for this indicator; x min This is the minimum value of the indicator;

[0148] For extremely small indicators, that is, the smaller the indicator value, the better, the processing method is as follows;

[0149]

[0150] Where: K2 is the dimensionless value of the extremely small index;

[0151] 2) Indicator variability:

[0152] The greater the variability of an indicator, the greater its volatility, and the more important it is. It can be expressed by standard deviation, and the calculation formula is as follows;

[0153]

[0154] Where: S j represents the standard deviation of the i-th indicator; x ij The value of the jth evaluation index for the i-th sample; is the average value of all samples of this indicator;

[0155] 3) Conflict of indicators:

[0156] Indicator conflict measures the degree of correlation between an indicator and other indicators. If the correlation is strong, it means that the weight of the indicator should be lower because other indicators have covered its information. The conflict value is calculated by the correlation coefficient between indicators. The specific formula is as follows;

[0157]

[0158] Where: T j is the conflict of the jth index, r ij Represents the correlation coefficient between evaluation indicators i and j;

[0159] 4) Information volume and weight calculation:

[0160] The amount of information is the product of the variability and conflict of the indicator. The larger the value, the greater the amount of information carried by the corresponding indicator. The more important the indicator is, the greater the weight is. The specific formula is as follows;

[0161]

[0162] Where: C j is the information content of the jth indicator;

[0163] The information value of each indicator is calculated using the above formula, and the objective weight of the corresponding indicator can be obtained by normalizing the value. The normalization formula is shown as follows:

[0164]

[0165] Where: μ j is the objective weight of the jth indicator;

[0166] As a further improvement of the present invention, the entropy weight method is specifically as follows:

[0167] The amount of information is determined by quantifying the uncertainty of entropy, and the entropy value is used to measure the degree of confusion or uncertainty of information. A higher entropy value indicates greater instability and uncertainty of information. This indicator has a greater impact on the evaluation results and should be assigned a higher weight.

[0168] Assume that the system under test has n evaluation indicators, m systems under test, and x ij , i = 1, 2, 3, ... n; j = 1, 2, 3, ... m represents the data of the i-th indicator of the j-th sample. The specific steps of this method are as follows:

[0169] 1) Normalize the data of each indicator;

[0170] 2) The proportion of the characteristics of the jth evaluated indicator under the i-th indicator is P ij , as shown below:

[0171]

[0172] 3) The information entropy calculation formula of the i-th indicator is as follows:

[0173]

[0174] 4) For the evaluation index i, x of a certain system to be tested ij The smaller the fluctuation between the values, the i The larger the value of x ij The greater the change in e i The smaller the value of , the higher the degree of differentiation of the index i on the object being evaluated. The coefficient of difference is expressed as g i =1-e i It means that the higher the difference coefficient, the more important the indicator is and the greater the weight is. The final weight formula is as follows:

[0175]

[0176] As a further improvement of the present invention, the combined weighting method is specifically as follows:

[0177] In order to reduce the bias of subjective and objective weighting, the least square method was used;

[0178] According to the basic principle of least squares method, as shown in the following formula, the weight of the combined weight is θ i ;

[0179] h(θ i )=(w i -θ i ) 2 +(μ i -θ i ) 2

[0180] For h(θ i ) is derived as shown below;

[0181] h′(θ i )=4θ i -2w i -2μ i

[0182] Let h(θ i )=0, and we get θ i The value is as follows:

[0183]

[0184] w i represents the weight vector of AHP, ∑w i =1;

[0185] μ i represents the weight vector of CRITIC weighting method, ∑μ i =1;

[0186] θ i represents the combined weight vector, ∑θ i =1.

[0187] As a further improvement of the present invention, the step three is based on the intelligent level evaluation of multi-index fusion, and the specific steps are as follows:

[0188] (1) Quantitative evaluation method;

[0189] The TOPSIS comprehensive evaluation method is used as a quantitative evaluation method. TOPSIS is a method for ranking a limited number of evaluation objects according to their proximity to an idealized goal. It is used in multi-objective decision analysis to determine their order of merit by comparing and analyzing multiple evaluation objects.

[0190] (2) Qualitative evaluation method;

[0191] Use the multi-dimensional spider web model comprehensive evaluation method as a qualitative evaluation method to analyze the intelligence capabilities of different intelligent systems;

[0192] The spider web model is used to dynamically analyze the impact of price fluctuations on the output of the next cycle. For autonomous evaluation, the multi-dimensional "spider web" model is a new evaluation model based on the spider web model.

[0193] As a further improvement of the present invention, the detailed implementation steps of the TOPSIS method are:

[0194] 1) Construct a decision matrix;

[0195] List all alternatives and evaluation indicators to form a decision matrix X. Assuming there are m alternatives and n evaluation indicators, the matrix is ​​in the form of:

[0196]

[0197] Among them, x ij It represents the value of the i-th alternative under the j-th evaluation index;

[0198] 2) Standardized decision matrix;

[0199] Since the dimensions of the evaluation indicators are different, the decision matrix needs to be standardized to ensure the comparability of different evaluation indicators. The standardization method is vector normalization:

[0200]

[0201] 3) construct a weighted normalization matrix;

[0202] The weight of each evaluation index w j Applied to the normalized matrix, we get the weighted normalized matrix V:

[0203] v ij =w j ·r ij

[0204] where w j is the weight of the jth indicator;

[0205] 4) Determine the ideal solution and the negative ideal solution;

[0206] Positive ideal solution A + :The optimal value of each indicator is recorded as:

[0207] A + =(maxv ij |j∈J1,minv ij |j∈J2)

[0208] Among them, J1 represents the benefit index set, the larger the value, the better, J2 represents the cost index set, the smaller the value, the better; negative ideal solution A - : The worst value of each indicator is recorded as:

[0209] A - =(minv ij |j∈J1,maxv ij |j∈J2)

[0210] 5) Calculate the Euclidean distance between each solution and the ideal solution and the negative ideal solution;

[0211] Calculate the difference between each solution i and the positive ideal solution A + Distance:

[0212]

[0213] Calculate the difference between each solution i and the negative ideal solution A - Distance:

[0214]

[0215] 6) Calculate relative proximity:

[0216] Calculate the relative closeness C of each solution i , that is, the degree to which solution i is close to the positive ideal solution and far away from the negative ideal solution at the same time:

[0217]

[0218] Among them, C i The value range of is [0,1], and the closer the value is to 1, the better the solution is;

[0219] 7) Sort and select the best solution;

[0220] According to the relative proximity C i Sort all alternatives in descending order, C i The solution with the largest value is the optimal solution.

[0221] As a further improvement of the present invention, the steps of constructing the multidimensional spider web model are:

[0222] 1) Establish the center point;

[0223] 2) Extend the lines representing the capabilities of the intelligent system outward from the center point;

[0224] 3) Determine the corresponding level of each line;

[0225] 4) Connect the points. The area of ​​the figure enclosed by the dotted line represents the degree of intelligence of the evaluated system. The larger the enclosed area, the higher the intelligence.

[0226] Compared with the prior art, the present invention has the following beneficial effects:

[0227] The present invention proposes a method for evaluating the intelligence level of intelligent systems based on the fusion of multiple capability indicators. Based on the existing research on unmanned intelligent systems, with the help of the OODA behavior model (i.e., the four links of perception, analysis and judgment, decision-making and action), an indicator system for evaluating the intelligence level of intelligent systems is constructed. In general, in order to intuitively and comprehensively evaluate the intelligence level of intelligent systems, this patent distinguishes from the capability dimension (including perception ability, cognitive ability, decision-making ability and action ability) and the technical dimension (including autonomy, collaboration, learning and security), reflecting the differences in the intelligence levels of intelligent systems in different capabilities. Finally, based on the capability dimension and technical dimension of the intelligent system, a set of graded evaluation methods that use the spider web model to comprehensively evaluate the overall intelligent capabilities and technologies of the intelligent system are designed. BRIEF DESCRIPTION OF THE DRAWINGS

[0228] Figure 1 It is the design process of the intelligent grade evaluation technical solution of the intelligent system of the present invention;

[0229] Figure 2 It is the OODA behavior model of the present invention;

[0230] Figure 3 It is a structure diagram of the intelligent system based on OODA theory of the present invention;

[0231] Figure 4 It is the multi-dimensional "spider web" model of the present invention. DETAILED DESCRIPTION

[0232] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 The design process of the intelligent level evaluation technical solution of the intelligent system of the present invention is as follows:

[0233] Step 1: Construction of indicator system based on OODA theory;

[0234] 1.OODA modeling theory;

[0235] like Figure 2 As shown in the figure, OODA consists of four parts: perception (Observe), analysis and judgment (Orient), decision (Decide) and action (Act), among which (1) Perception (Observe): extract data features. (2) Cognition (Orient): recognize data features. (3) Decision (Decide): make decisions in different environments. (4) Action (Act): act in a complex actual environment. Figure 3 It shows an intelligent system structure based on OODA. Therefore, this patent decomposes the capabilities of the intelligent system into four links: intelligent perception, intelligent cognition, intelligent decision-making and intelligent action.

[0236] 2. Construction of indicator evaluation system;

[0237] Based on the four index system principles of core focus, taking into account characteristics, comprehensiveness and scalability, this patent subdivides each capability layer from the four dimensions of autonomy, collaboration, learning and security, and divides the technical layer according to relevant documents, thereby obtaining an index system for evaluating the intelligent capabilities of intelligent systems. As shown in the following table, the technical layer is divided into autonomy, collaboration, learning and security, and each layer has corresponding underlying indicators to reflect the intelligent capabilities of the intelligent system. The meaning, measurement, calculation formula and reasonable value range of each underlying indicator are given below.

[0238] Table 1 Index evaluation system

[0239]

[0240]

[0241] (1) Perception capability technology layer

[0242] By decomposing the intelligent system's intelligence dimension, the perception capability can be divided into autonomy, collaboration, learning, and security. The technical layer corresponds to four underlying indicators: feature vector clustering, feature vector similarity, training convergence speed, and adversarial training performance improvement.

[0243] 1) Autonomy of perception:

[0244] Meaning: Whether the data characteristics can be perceived autonomously.

[0245] Measurement: Clustering evaluation of feature vectors. During the training process of the intelligent system, the intelligent system will extract the features of the training data to form a feature vector. By performing cluster analysis on the feature vectors and evaluating the clustering effect, the quality of the feature vectors perceived by the intelligent system can be evaluated, further illustrating the autonomy of the system's perception ability. Use a dimensionality reduction algorithm to reduce the dimensionality of the feature vector, then use a clustering algorithm to cluster it, and finally use a clustering evaluation algorithm to evaluate the clustering effect. The dimensionality reduction algorithm used in this patent is TSNE, the clustering algorithm used is the KMeans algorithm, and the clustering effect evaluation method used is the silhouette coefficient.

[0246] Calculation formula:

[0247]

[0248] Where a(i) represents the average distance from sample i to other samples in its cluster, i.e., intra-cluster dissimilarity. b(i) represents the average distance from sample i to the nearest cluster (nearest neighbor cluster), i.e., inter-cluster dissimilarity. N represents the total number of training data samples. m1 is what this metric seeks.

[0249] Reasonable value range: the value range of the silhouette coefficient. Silhouette is a clustering algorithm effect evaluation indicator used to evaluate the compactness and separation of clustering results. Its value range is [-1,1]. The closer the index is to 1, the better the perception ability autonomy.

[0250] 2) Synergy of Perception Abilities

[0251] Meaning: Whether it can collaborate with other intelligent systems to perceive data features.

[0252] Metric: Similarity of feature vectors. The similarity between the feature vectors extracted from the system under test and the feature vectors extracted from the baseline system is used to characterize the degree of coordination between the system under test and the baseline system. The higher the similarity, the better the coordination of the system under test. This patent uses the cosine similarity value to calculate the similarity between feature vectors.

[0253] Calculation formula:

[0254]

[0255] Where F1 and F2 represent the feature vector perceived by the model under test and the feature vector perceived by the baseline model, respectively, i represents the i-th training data, and N represents the number of samples in the training set. m2 is what the metric seeks.

[0256] Reasonable value range: the value range of cosine similarity. The value range of cosine similarity is [-1,1]. The closer the value of cosine similarity is to 1, the more consistent the directions of the two vectors are, and the more similar they are. Therefore, the closer this indicator is to 1, the better the synergy of perception ability is.

[0257] 3) Learnability of Perception Ability:

[0258] Meaning: Whether the perceived data features can be learned quickly.

[0259] Metric: Training convergence speed. Intelligent systems usually improve their ability to perceive data features by learning input data, so the convergence speed of the intelligent system during training can characterize the learnability of the perception ability. This patent uses the convergence speed of the loss value during training to measure the learnability of the perception ability of the intelligent system.

[0260] Calculation formula:

[0261]

[0262] Where L(t) represents the loss function value, T represents the total number of iterations, and t represents the current iteration step. m3 is what the metric is looking for.

[0263] Reasonable value range: the value range of the convergence speed of the loss value. The value range is (-∞,1]. The closer the index is to 1, the better the learning ability of perception is.

[0264] 4) Security of perception ability:

[0265] Meaning: Whether the characteristics of the data can be safely perceived.

[0266] Metric: Adversarial training performance improvement. This patent uses adversarial samples to retrain the intelligent system, and statistically analyzes the performance improvement of the intelligent system on adversarial samples before and after training to reflect the security of the system's perception capabilities.

[0267] Calculation formula:

[0268] m4=ACC adv -ACC origin

[0269] ACC adv ACC represents the accuracy of the model on adversarial samples after adversarial training. origin It represents the accuracy of the original model in adversarial samples. m4 is what this metric is looking for.

[0270] Reasonable value range: the value range is [0,ACC adv ], the closer this indicator is to ACC advThis means that the security of the perception ability is better.

[0271] (2) Cognitive Ability Technology Layer

[0272] 1) Autonomy of cognitive abilities:

[0273] Meaning: Whether the perceived data features can be recognized autonomously.

[0274] Metric: Feature contribution. By using statistics, we can get the degree of influence of the features of the data itself on the prediction results when the intelligent system predicts the data. The greater the performance difference, the greater the contribution of the features of the data itself to the prediction results of the intelligent system, which means that the intelligent system is too dependent on the appearance of the data and its cognitive ability is weaker.

[0275] Calculation formula:

[0276] m5=ACC origin -ACC masked

[0277] ACC origin Indicates the accuracy of the original model in the test set, ACC masked It represents the accuracy of the original model on the added masked test set. m5 is what this metric seeks.

[0278] Reasonable value range: the value range is [0,ACC origin ] The closer this indicator is to ACC origin This indicates that the greater the autonomy of cognitive ability, the better.

[0279] 2) Synergy of cognitive abilities:

[0280] Meaning: Whether the system's ability to recognize data features can be improved with the collaborative assistance of other intelligent systems.

[0281] Metric: Performance improvement with assistance from other models.

[0282] Calculation formula:

[0283] m6=ACC Ch atGLM -ACC origin

[0284] ACC Ch atGLM Indicates the accuracy of the model on the enhanced data after the data is enhanced by ChaGLM, ACC origin Indicates the accuracy of the model on the original data. m6 is what this metric is looking for.

[0285] Reasonable value range: the value range is [0,ACC Ch atGLM ] The closer this indicator is to ACC Ch atGLM This indicates that the coordination of cognitive abilities is better.

[0286] 3) Learnability of cognitive abilities:

[0287] Meaning: Whether the cognitive ability of the system can be steadily improved during the model training process.

[0288] Measurement: Smoothness of loss decline. The intelligent system improves its ability to recognize data features by learning from input data. This patent uses the smoothness of the loss curve during the training process to measure the learning ability of the intelligent system's cognitive ability. That is, the faster the smoothness, the stronger the learning ability of the cognitive ability.

[0289] Calculation formula:

[0290]

[0291] where L(t) is the change of loss function over time (or training steps), is the first derivative of the loss function, Var represents the variance. m7 is what this metric seeks.

[0292] Reasonable value range: The value range is [0, +∞]. The closer this indicator is to 0, the better the cognitive ability and learning ability.

[0293] 4) Cognitive safety:

[0294] Meaning: Whether the characteristics of the data can be recognized safely.

[0295] Metric: Tolerance to adversarial attacks. Use the performance improvement of adversarial training to characterize the security of cognitive ability. This patent sets a threshold for the success rate of attacks on intelligent systems, and adjusts the perturbation value in the process of generating adversarial samples for adversarial attacks until the success rate of adversarial attacks exceeds or reaches the threshold. The perturbation value is the metric of tolerance to adversarial attacks. The greater the tolerance to adversarial attacks, the higher the security of the system's cognitive ability.

[0296] Calculation formula:

[0297]

[0298] Where x is the original input, δ is the perturbation, and f is the output function of the model. m8 is what the metric requires.

[0299] Reasonable value range: the value range of δ. The range of δ set in this patent is [0.1, 0.9]. The closer this indicator is to 0.9, the better the cognitive ability security is.

[0300] (3) Decision-making capability technology layer

[0301] 1) Autonomy of decision-making ability:

[0302] Meaning: Whether the perceived data features can be recognized autonomously.

[0303] Measurement: Decision accuracy. The autonomy of decision-making ability is measured by the accuracy of the statistical intelligent system on the test set. That is, the higher the accuracy of the decision, the stronger the autonomy of the decision-making ability.

[0304] Calculation formula:

[0305]

[0306] Where N is the total number of test set samples, y i is the true label of the i-th sample, is the predicted label of the i-th sample. It means when y i equal 1 when , otherwise 0. m9 is the desired metric.

[0307] Reasonable value range: The value range is [0,1]. The closer this indicator is to 1, the better the decision-making ability is.

[0308] 2) Synergy of decision-making ability:

[0309] Meaning: Whether it can make decisions in collaboration with other intelligent systems in a decision-making environment.

[0310] Metric: Similarity of feature vectors in a decision-making environment. This patent uses the similarity between the feature vectors extracted from the system under test and the feature vectors extracted from the baseline system in the test set to characterize the degree of coordination between the system under test and the baseline system, and uses the cosine similarity value to calculate the similarity between feature vectors.

[0311] Calculation formula:

[0312]

[0313] Where F1 and F2 represent the feature vector perceived by the model to be tested and the feature vector perceived by the baseline model, respectively, i represents the i-th test data, and N represents the number of samples in the test set. 10 This is what the measurement seeks.

[0314] Reasonable value range: the value range of cosine similarity. Therefore, the closer this index is to 1, the better the decision-making ability coordination is.

[0315] 3) Learnability of decision-making ability:

[0316] Meaning: Whether the decision-making ability of the system can be quickly improved during the model training process.

[0317] Metric: How fast the accuracy converges in the training environment.

[0318] Calculation formula:

[0319]

[0320] Where A(t) represents the accuracy function value, T represents the total number of iterations, and t represents the current iteration step. 11 This is what the measurement seeks.

[0321] Reasonable value range: The value range is (-∞,1]. The closer the indicator is to 1, the better the decision-making ability is.

[0322] 4) Security of decision-making ability:

[0323] Meaning: Whether it is safe to make decisions on input data.

[0324] Metric: Accuracy at a specific confidence level. The higher the accuracy at a specific confidence level, the safer the decision-making of the intelligent system.

[0325] Calculation formula:

[0326] wh en confidence=th resh old#(2.12)

[0327] Where N is the total number of test set samples, y i is the true label of the i-th sample, is the predicted label of the i-th sample. It means when y i equal 1 when the value is 0, otherwise, th resh old is the confidence threshold that needs to be specified to characterize the accuracy under a specific confidence level. 12 This is what the measurement seeks.

[0328] Reasonable value range: The value range is [0,1]. The closer this indicator is to 1, the better the decision-making ability security is.

[0329] (4) Action Capability Technology Layer

[0330] 1) Autonomy of action:

[0331] Meaning: Whether the system can autonomously complete actions in a complex real-world action environment.

[0332] Calculation formula:

[0333]

[0334] Where N is the total number of test set samples with added noise, y i is the true label of the i-th sample, is the predicted label of the i-th sample. It means when y i equal 1 when , otherwise 0, to measure the accuracy of the model to be tested on the noise data set. 13 This is what the measurement seeks.

[0335] Reasonable value range: The value range is [0,1]. The closer the index is to 1, the better the autonomy of action ability.

[0336] 2) Synergy of action planning capabilities:

[0337] Meaning: Can it collaborate with other intelligent systems to make predictions in action?

[0338] Metric: Jaccard correlation coefficient of the decision of the system under test and the baseline system. The intelligent system will predict the input data and generate prediction labels. It is characterized by the correlation coefficient between the prediction labels generated by the system under test and the baseline system. This patent uses the Jaccard correlation coefficient between the prediction labels to measure. The larger the Jaccard value, the stronger the synergy of the action capability.

[0339] Calculation formula:

[0340]

[0341] Among them, M o ∩M i Indicates that the model to be tested and the baseline model predict the same label for the sample. o ∪M i Indicates that both models have at least one predicted sample. 14 This is what the measurement seeks.

[0342] Reasonable value range: the value range of the Jaccard correlation coefficient. The value range of the Jaccard correlation coefficient is [0,1]. The closer the index is to 1, the better the coordination of action capabilities.

[0343] 3) Learning of action ability:

[0344] Meaning: Whether the system's action error can be quickly reduced during the model training process using a real and complex training set.

[0345] Measurement: Convergence speed of training loss in a noisy environment. The intelligent system is trained on a noisy training set, and the convergence speed of the training loss curve during the training process is statistically analyzed to measure the learnability of the action ability. That is, the faster the convergence speed, the stronger the learnability of the action ability.

[0346] Calculation formula:

[0347]

[0348] Where L(t) represents the loss function value of the model to be tested when trained under the noise dataset, T represents the total number of iterations, and t represents the current iteration step. 15 This is what the measurement seeks.

[0349] Reasonable value range: The value range is (-∞,1]. The closer the indicator is to 1, the better the learning ability of action is.

[0350] 4) Safety of mobility:

[0351] Meaning: Whether safe action decisions can be made in complex real-world environments.

[0352] Metric: Accuracy in adversarial environment. Use adversarial attack technology to generate adversarial samples, and then count the prediction accuracy of the intelligent system on the adversarial sample set. The higher the accuracy, the higher the security of the intelligent system's action capability.

[0353] Calculation formula:

[0354]

[0355] Where N is the total number of samples in the adversarial sample set, y i is the true label of the i-th sample, is the predicted label of the i-th sample. It means when y i equal 1 when the value is 1, otherwise 0. 16 This is what the measurement seeks.

[0356] Reasonable value range: The value range is [0,1]. The closer the indicator is to 1, the better the safety of the action ability.

[0357] Step 2: Indicator weight allocation

[0358] During the evaluation process, the importance of each indicator varies depending on the situation. This patent uses three weighting algorithms, namely CRITIC weighting method, entropy weighting method and combined weighting method, to reasonably determine the weight of each indicator to ensure the accuracy and fairness of the evaluation.

[0359] (1) CRITIC Empowerment Law

[0360] Assume there are n intelligent systems, and each system to be tested has p evaluation indicators. As shown in the following formula. ij Represents the value of the jth evaluation index of the ith sample.

[0361]

[0362] 1) Normalization of indicators: In order to avoid inaccurate results due to dimension differences, each indicator needs to be dimensionless so that different indicators are comparable in the same evaluation system. For extremely large indicators, the standard deviation method is usually used for dimensionless processing. This method converts the indicator into a dimensionless value by calculating the deviation of each indicator value from the average value, so that it can be compared under a unified standard. The specific calculation formula is as follows.

[0363]

[0364] Where: K1 is the dimensionless value of the extremely large index; x i is the actual value of the indicator; x max The maximum value for this indicator; x min This is the minimum value for this indicator.

[0365] For extremely small indicators, that is, the smaller the indicator value, the better. The processing method is as shown in the following formula.

[0366]

[0367] Where: K2 is the dimensionless value of the extremely small index.

[0368] 2) Indicator variability: The greater the variability of an indicator, the greater its volatility and the more important it is. It can be represented by standard deviation, and the calculation formula is shown below.

[0369]

[0370] Where: S j represents the standard deviation of the i-th indicator; x ij The value of the jth evaluation index for the i-th sample; It is the average value of all samples of this indicator.

[0371] 3) Indicator conflict: Indicator conflict measures the degree of correlation between an indicator and other indicators. If the correlation is strong, it means that the weight of the indicator should be lower because other indicators have covered its information. The conflict value can be calculated by the correlation coefficient between indicators. The specific formula is as follows.

[0372]

[0373] Where: T j is the conflict of the jth index, r ij Represents the correlation coefficient between evaluation indicators i and j.

[0374] 4) Calculation of information volume and weight: The information volume is the product of indicator variability and conflict. The larger the value, the greater the amount of information carried by the corresponding indicator. The more important the indicator is, the greater the weight is. The specific formula is as follows.

[0375]

[0376] Where: C j is the information content of the jth indicator.

[0377] Use the above formula to calculate the information value of each indicator, and normalize the value to get the objective weight of the corresponding indicator. The normalization formula is shown below.

[0378]

[0379] Where: μ j is the objective weight of the jth indicator.

[0380] (2) Entropy Weight Method

[0381] The entropy weight method is an objective weighting method that determines the amount of information by quantifying the uncertainty of entropy, and the entropy value is used to measure the degree of confusion or uncertainty of information. A higher entropy value indicates greater instability and uncertainty of information. This indicator has a greater impact on the evaluation results and should be assigned a higher weight.

[0382] Assume that the system under test has n evaluation indicators, m systems under test, and x ij (i=1,2,3,…n;j=1,2,3,…m) represents the data of the i-th indicator of the j-th sample. The specific steps of this method are as follows:

[0383] 1) Normalize the data of each indicator (same as CRITIC method)

[0384] 2) The proportion of the characteristics of the jth evaluated indicator under the i-th indicator is P ij , as shown below:

[0385]

[0386] 3) The information entropy calculation formula of the i-th indicator is as follows:

[0387]

[0388] 4) For the evaluation index i, x of a certain system to be tested ij The smaller the fluctuation of the value between i The larger the value of x ij The greater the change in e i The smaller the value of , the higher the degree of differentiation of the index i on the object being evaluated. The coefficient of difference is expressed as g i =1-e i It means that the higher the coefficient of difference, the more important the indicator is and the greater its weight is. The final weight formula is as follows:

[0389]

[0390] (3) Combination weighting method

[0391] In order to reduce the deviation of subjective and objective weighting, this patent adopts the least squares method.

[0392] According to the basic principle of least squares method, as shown in the following formula, the weight of the combined weight is θ i

[0393] h(θ i )=(w i -θ i ) 2 +(μ i -θ i ) 2

[0394] For h(θ i ) is derived as shown below.

[0395] h′(θ i )=4θ i -2w i -2μ i

[0396] Let h(θ i )=0, and we get θ i The value is as follows:

[0397]

[0398] w i represents the weight vector of AHP, ∑w i =1;

[0399] μ i represents the weight vector of CRITIC weighting method, ∑μ i =1;

[0400] θ i represents the combined weight vector, ∑θ i =1;

[0401] Step 3: Intelligent level evaluation based on multi-index fusion

[0402] (1) Quantitative evaluation method

[0403] This patent uses the TOPSIS comprehensive evaluation method as a quantitative evaluation method. TOPSIS (Technique for Order of Preference by Similarity to Ideal Solution) is a method for ranking based on the degree of proximity between a finite number of evaluation objects and an idealized goal. It is mainly used in multi-objective decision analysis to determine their order of merit by comparing and analyzing multiple evaluation objects. The following are the detailed implementation steps of the TOPSIS method:

[0404] 1) Build a decision matrix

[0405] List all alternatives and evaluation indicators to form a decision matrix X. Assuming there are m alternatives and n evaluation indicators, the matrix is ​​in the form of:

[0406]

[0407] Among them, x ij Represents the value of the i-th alternative under the j-th evaluation index.

[0408] 2) Standardized decision matrix

[0409] Since the dimensions of each evaluation index are different, the decision matrix needs to be standardized to ensure the comparability between different evaluation indexes. The commonly used standardization method is vector normalization:

[0410]

[0411] 3) Construct a weighted normalization matrix

[0412] The weight of each evaluation index w j Applied to the normalized matrix, we get the weighted normalized matrix V:

[0413] v ij =w j ·r ij

[0414] where w j is the weight of the jth indicator.

[0415] 4) Determine the ideal solution and negative ideal solution

[0416] Positive ideal solution (A + ): The optimal value of each indicator, denoted as:

[0417] A + =(maxv ij |j∈J1,minv ij |j∈J2)

[0418] Among them, J1 represents the benefit-type indicator set (the larger the value, the better), and J2 represents the cost-type indicator set (the smaller the value, the better).

[0419] Negative ideal solution (A - ): The worst value of each indicator, recorded as:

[0420] A - =(minv ij |j∈J1,maxv ij |j∈J2)

[0421] 5) Calculate the Euclidean distance between each solution and the ideal solution and the negative ideal solution

[0422] Calculate the difference between each solution i and the positive ideal solution A + Distance:

[0423]

[0424] Calculate the difference between each solution i and the negative ideal solution A - Distance:

[0425]

[0426] 6) Calculate relative proximity

[0427] Calculate the relative closeness C of each solution i , that is, the degree to which solution i is close to the positive ideal solution and far away from the negative ideal solution at the same time:

[0428]

[0429] Among them, C i The value range of is [0,1]. The closer the value is to 1, the better the solution is.

[0430] 7) Sort and select the best solution

[0431] According to the relative proximity C i Sort all alternatives in descending order, C i The solution with the largest value is the optimal solution.

[0432] (2) Qualitative evaluation method

[0433] This patent uses a multi-dimensional "spider web" model comprehensive evaluation method as a qualitative evaluation method to analyze the intelligent capabilities of different intelligent systems.

[0434] The spider web model is mainly used to dynamically analyze the impact of price fluctuations on the output of the next cycle, such as Figure 4 As shown in the figure, for autonomy assessment, the multidimensional "spider web" model is a new assessment model based on the spider web model. The steps for building the model are:

[0435] 1) Establish a center point.

[0436] 2) Extend outward from the center point to represent the capabilities of the intelligent system.

[0437] 3) Determine the corresponding level of each line.

[0438] 4) Connect the points, as shown by the dotted lines in the figure. The area enclosed by the dotted lines in the figure represents the degree of intelligence of the evaluated system, and the larger the enclosed area, the higher the intelligence.

[0439] In general, in order to intuitively and comprehensively evaluate the intelligence level of intelligent systems, this patent distinguishes from the capability dimension (including perception, cognition, decision-making and action) and the technical dimension (including autonomy, collaboration, learning and security), reflecting the differences in the intelligence level of intelligent systems in different capabilities. Finally, based on the capability dimension and technical dimension of the intelligent system, a set of graded evaluation methods that use the spider web model to comprehensively evaluate the overall intelligent capability and technology of the intelligent system are designed.

[0440] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A method for evaluating the intelligence level of an intelligent system based on the fusion of multiple capability indicators, characterized by: It includes the following steps: Step 1: Construction of indicator system based on OODA theory; Step 2: indicator weight allocation; Step three: Intelligent level evaluation based on multi-indicator fusion.

2. According to claim 1, a method for evaluating the intelligent level of an intelligent system based on the fusion of multiple capability indicators is characterized by: The step 1 is to construct an indicator system based on the OODA theory, as follows: 1.OODA modeling theory; OODA consists of four parts: perception, analysis and judgment, decision-making and action, among which: (1) Perception: extracting data features; (2) Cognition: cognition of data characteristics; (3) Decision-making: making decisions in different environments; (4) Action: Acting in a realistic and complex environment; Decompose the capabilities of intelligent systems into four links: intelligent perception, intelligent cognition, intelligent decision-making and intelligent action; 2. Construction of indicator evaluation system; Based on the four indicator system principles of core focus, taking into account characteristics, comprehensiveness and scalability, each capability layer is subdivided from four dimensions: autonomy, collaboration, learning and security. At the same time, the technical layer is divided according to relevant documents, and then the indicator system for evaluating the intelligent capabilities of intelligent systems is obtained. The technical layer is divided into autonomy, collaboration, learning and security. Each layer has corresponding underlying indicators to reflect the intelligent capabilities of the intelligent system.

3. The intelligent system intelligence level evaluation method based on multi-capability index fusion according to claim 2 is characterized in that : The perception technology capability layer is as follows: (1) Perception capability technology layer; By decomposing the intelligent system's intelligence dimension, the perception capability is divided into autonomy, collaboration, learning, and security. The technical layer corresponds to four underlying indicators: feature vector clustering, feature vector similarity, training convergence speed, and adversarial training performance improvement. 1) Autonomy of perception: Meaning: Whether it is possible to perceive data features autonomously; Measurement: Clustering evaluation of feature vectors. During the training process of the intelligent system, the intelligent system extracts the features of the training data to form feature vectors. By clustering the feature vectors and evaluating the clustering effect, the quality of the feature vectors perceived by the intelligent system can be evaluated. Use the dimensionality reduction algorithm to reduce the dimension of the feature vector, then use the clustering algorithm to cluster, and finally use the clustering evaluation algorithm to evaluate the clustering effect. The dimensionality reduction algorithm used is TSNE, the clustering algorithm used is KMeans algorithm, and the clustering effect evaluation method used is silhouette coefficient; Calculation formula: Where a(i) represents the average distance from sample i to other samples in its cluster, i.e., intra-cluster dissimilarity, b(i) represents the average distance from sample i to the nearest cluster, i.e., inter-cluster dissimilarity, N represents the total number of training data samples, and m1 is what this metric seeks; Reasonable value range: the value range of the silhouette coefficient. Silhouette is a clustering algorithm effect evaluation indicator used to evaluate the compactness and separation of clustering results. Its value range is [-1, 1]. The closer the index is to 1, the better the perception ability autonomy. 2) Synergy of perception capabilities; Meaning: Whether it can collaborate with other intelligent systems to perceive data features; Measurement: Similarity of feature vectors. The similarity between the feature vectors extracted from the system under test and the baseline system is used to characterize the degree of coordination between the system under test and the baseline system. The higher the similarity, the better the coordination between the systems under test. The cosine similarity value is used to calculate the similarity between feature vectors. Calculation formula: Where F1 and F2 represent the feature vector perceived by the model to be tested and the feature vector perceived by the baseline model, respectively, i represents the i-th training data, N represents the number of samples in the training set, and m2 is what the metric requires; Reasonable value range: the value range of cosine similarity. The value range of cosine similarity is [-1,1]. The closer the value of cosine similarity is to 1, the more consistent the directions of the two vectors are, and the more similar they are. Therefore, the closer this indicator is to 1, the better the synergy of perception ability is. 3) Learnability of Perception Ability: Meaning: Whether the perceived data features can be learned quickly; Metrics: Training convergence speed. Intelligent systems usually improve their ability to perceive data features by learning from input data. Therefore, the convergence speed of the intelligent system during training can characterize the learning of the perception ability. The convergence speed of the loss value during training is used to measure the learning of the intelligent system's perception ability. Calculation formula: Where L(t) represents the loss function value, T represents the total number of iterations, t represents the current iteration step, and m3 is the metric required; Reasonable value range: the value range of the convergence speed of the loss value, the value range is (-∞,1], the closer the index is to 1, the better the perceptual learning ability is; 4) Security of perception ability: Meaning: Whether the characteristics of the data can be sensed safely; Measurement: Adversarial training performance improvement. Use adversarial samples to retrain the intelligent system. Statistical analysis of the performance improvement of the intelligent system on adversarial samples before and after training to reflect the security of the system's perception capabilities. Calculation formula: m4=ACC adv -ACC origin ACC adv ACC represents the accuracy of the model on adversarial samples after adversarial training. origin It represents the accuracy of the original model in adversarial samples, and m4 is what this metric requires; Reasonable value range: the value range is [0,ACC adv ], the closer this indicator is to ACC adv This indicates that the better the perceived safety is; (2) Cognitive capability technology layer; 1) Autonomy of cognitive abilities: Meaning: Whether the perceived data features can be recognized autonomously; measure : Feature contribution, through statistics, the influence of the characteristics of the data itself on the prediction results when the intelligent system predicts the data is obtained. The greater the performance difference, the greater the contribution of the characteristics of the data itself to the prediction results of the intelligent system, which means that the intelligent system is too dependent on the appearance characteristics of the data, and its cognitive ability is weaker. Calculation formula: n5=ACC origin -ACC masked ACC origin Indicates the accuracy of the original model in the test set, ACC origin It represents the accuracy of the original model on the test set with mask added, and m5 is what this metric seeks; Reasonable value range: the value range is [0,ACC origin ], the closer this indicator is to ACC origin This indicates that the greater the autonomy of cognitive ability, the better; 2) Synergy of cognitive abilities: Meaning: Can the system's ability to recognize data features be improved with the assistance of other intelligent systems? Metrics: Performance improvement with the assistance of other models; Calculation formula: m6=ACC ChatGLM -ACC origin ACC ChatGLM Indicates the accuracy of the model on the enhanced data after the data is enhanced by ChaGLM, ACC origin It represents the accuracy of the model on the original data, and m6 is what this metric seeks; Reasonable value range: the value range is [0,ACC ChatGLM ], the closer this indicator is to ACC ChatGLM This indicates that the better the synergy of cognitive abilities; 3) Learnability of cognitive abilities: Meaning: Whether the cognitive ability of the system can be steadily improved during the model training process; Measurement: Smoothness of loss decline. The intelligent system improves its ability to recognize data features by learning from input data. The smoothness of the loss curve during training is used to measure the learning ability of the intelligent system's cognitive ability. That is, the faster the smoothness, the stronger the learning ability of the cognitive ability. Calculation formula: where L(t) is the change in loss function over time or training steps, is the first-order derivative of the loss function, Var represents the variance, and m7 is what the metric seeks; Reasonable value range: The value range is [0, +∞]. The closer the index is to 0, the better the cognitive ability and learning ability. 4) Cognitive safety: Meaning: Whether the characteristics of the data can be recognized safely; Measurement: Adversarial attack tolerance. Use the performance improvement of adversarial training to characterize the security of cognitive ability. Set a threshold for the success rate of attacks on intelligent systems. Adjust the perturbation value in the process of generating adversarial samples by adversarial attacks until the success rate of adversarial attacks exceeds or reaches the threshold. The perturbation value is the measurement indicator of adversarial attack tolerance. The greater the adversarial attack tolerance, the higher the security of the system's cognitive ability. Calculation formula: Where x is the original input, δ is the perturbation, f is the output function of the model, and m8 is the metric required; Reasonable value range: the value range of δ. The range of δ is set to [0.1, 0.9]. The closer this index is to 0.9, the better the cognitive ability security is; (3) Decision-making capability technical layer; 1) Autonomy of decision-making ability: Meaning: Whether the perceived data features can be recognized autonomously; Measurement: Decision accuracy. The autonomy of decision-making ability is measured by the accuracy of the statistical intelligent system on the test set. That is, the higher the accuracy of the decision, the stronger the autonomy of the decision-making ability. Calculation formula: Where N is the total number of test set samples, y i is the true label of the i-th sample, is the predicted label of the ith sample, It means when y i equal It is 1 when , otherwise it is 0, and m9 is what the metric requires; Reasonable value range: The value range is [0,1]. The closer the index is to 1, the better the decision-making autonomy is. 2) Synergy of decision-making ability: Meaning: Can it make decisions in collaboration with other intelligent systems in a decision-making environment? Measurement: Similarity of feature vectors in the decision environment. The similarity between the feature vectors extracted from the test system and the baseline system in the test set is used to characterize the degree of coordination between the test system and the baseline system, and the cosine similarity value is used to calculate the similarity between feature vectors. Calculation formula: Where F1 and F2 represent the feature vector perceived by the model to be tested and the feature vector perceived by the baseline model, respectively, i represents the i-th test data, N represents the number of samples in the test set, and m 10 That is what the measure seeks; Reasonable value range: the value range of cosine similarity. Therefore, the closer this index is to 1, the better the decision-making ability coordination is. 3) Learnability of decision-making ability: Meaning: Whether the decision-making ability of the system can be quickly improved during the model training process; Metrics: Accuracy convergence speed in training environment; Calculation formula: Where A(t) represents the accuracy function value, T represents the total number of iterations, t represents the current iteration step, and m 11 This is what the metric seeks; Reasonable value range: The value range is (-∞,1], the closer the index is to 1, the better the decision-making ability is; 4) Security of decision-making ability: Meaning: Can decisions be made safely on input data? Metric: Accuracy at a specific confidence level. The higher the accuracy at a specific confidence level, the higher the decision safety of the intelligent system. Calculation formula: Where N is the total number of test set samples, y i is the true label of the i-th sample, is the predicted label of the ith sample, It means when y i equal 1 when the confidence level is 0, otherwise, th resh old is the confidence threshold to be specified, which is used to characterize the accuracy under a specific confidence level. 12 That is what the measure seeks; Reasonable value range: The value range is [0,1]. The closer the index is to 1, the better the decision-making ability security is. (4) Action capability technology layer; 1) Autonomy of action: Meaning: Whether the person can autonomously complete actions in a complex real-life action environment; Calculation formula: Where N is the total number of test set samples with added noise, y i is the true label of the i-th sample, is the predicted label of the ith sample, It means when y i equal 1 when it is, otherwise 0, in order to measure the accuracy of the model to be tested on the noise data set, m 13 That is what the measure seeks; Reasonable value range: The value range is [0,1]. The closer the index is to 1, the better the autonomy of action ability. 2) Synergy of action planning capabilities: Meaning: Can it collaborate with other intelligent systems to make predictions in action? Measurement: Jaccard correlation coefficient of the decision of the system under test and the baseline system. The intelligent system will predict the input data and generate prediction labels. The correlation coefficient between the prediction labels generated by the system under test and the baseline system is used to represent it. The Jaccard correlation coefficient between the prediction labels is used to measure it. The larger the Jaccard value, the stronger the coordination of action capabilities. Calculation formula: Among them, M o ∩M i Indicates that the model to be tested and the baseline model predict the same label for the sample, M o ∪M i Indicates that both models have at least one predicted sample, m 14 That is what the measure seeks; Reasonable value range: the value range of the Jaccard correlation coefficient. The value range of the Jaccard correlation coefficient is [0,1]. The closer the index is to 1, the better the coordination of action capabilities. 3) Learning of action ability: Meaning: Can the system's action error be quickly reduced during the model training process using a real and complex training set? Measurement: The convergence speed of training loss in a noisy environment. The intelligent system is trained on a noisy training set. The convergence speed of the training loss curve during the training process is used to measure the learning ability of the action ability. That is, the faster the convergence speed, the stronger the learning ability of the action ability. Calculation formula: Where L(t) represents the loss function value of the model to be tested under the noisy data set, T represents the total number of iterations, t represents the current iteration step, and m 15 That is what the measure seeks; Reasonable value range: The value range is (-∞,1]. The closer the index is to 1, the better the learning ability of action is. 4) Safety of mobility: Meaning: Can you make safe action decisions in complex real-world environments? Metrics: Accuracy in adversarial environments. Use adversarial attack techniques to generate adversarial samples, and then count the prediction accuracy of the intelligent system on the adversarial sample set. The higher the accuracy, the higher the security of the intelligent system's action capability. Calculation formula: Where N is the total number of samples in the adversarial sample set, y i is the true label of the i-th sample, is the predicted label of the ith sample, It means when y i equal 1 when it is, otherwise 0, m 16 That is what the measure seeks; Reasonable value range: The value range is [0,1]. The closer the indicator is to 1, the better the safety of the action ability.

4. According to claim 1, a method for evaluating the intelligent level of an intelligent system based on the fusion of multiple capability indicators is characterized by: Step 2: indicator weight allocation, as follows; During the evaluation process, the importance of each indicator varies depending on the situation. By using three weighting algorithms, namely CRITIC weighting method, entropy weight method and combined weighting method, the weight of each indicator is reasonably determined to ensure the accuracy and fairness of the evaluation.

5. According to claim 4, a method for evaluating the intelligent level of an intelligent system based on the fusion of multiple capability indicators is characterized by: The CRITIC empowerment method is as follows: Assume that there are n intelligent systems, and each system to be tested has p evaluation indicators, as shown in the following formula, where x ij Represents the value of the jth evaluation index of the i-th sample; 1) Index normalization processing: In order to avoid inaccurate results due to dimension differences, each indicator needs to be dimensionless so that different indicators are comparable in the same evaluation system. For extremely large indicators, the standard deviation method is usually used for dimensionless processing. This method converts the indicator into a dimensionless value by calculating the deviation of each indicator value from the average value, so that it can be compared under a unified standard. The specific calculation formula is as follows; Where: K1 is the dimensionless value of the extremely large index; x i is the actual value of the indicator; x max The maximum value for this indicator; x min This is the minimum value of the indicator; For extremely small indicators, that is, the smaller the indicator value, the better, the processing method is as follows; Where: K2 is the dimensionless value of the extremely small index; 2) Indicator variability: The greater the variability of an indicator, the greater its volatility, and the more important it is. It can be expressed by standard deviation, and the calculation formula is as follows; Where: S j represents the standard deviation of the ith indicator; x ij The value of the jth evaluation index for the i-th sample; is the average value of all samples of this indicator; 3) Conflict of indicators: Indicator conflict measures the degree of correlation between an indicator and other indicators. If the correlation is strong, it means that the weight of the indicator should be lower because other indicators have covered its information. The conflict value is calculated by the correlation coefficient between indicators. The specific formula is as follows; Where: T j is the conflict of the jth index, r ij Represents the correlation coefficient between evaluation indicators i and j; 4) Information volume and weight calculation: The amount of information is the product of the variability and conflict of the indicator. The larger the value, the greater the amount of information carried by the corresponding indicator. The more important the indicator is, the greater the weight is. The specific formula is as follows; Where: C j is the information content of the jth indicator; The information value of each indicator is calculated using the above formula, and the objective weight of the corresponding indicator can be obtained by normalizing the value. The normalization formula is shown as follows: Where: μ j is the objective weight of the jth indicator.

6. According to claim 4, a method for evaluating the intelligent level of an intelligent system based on the fusion of multiple capability indicators is characterized by: The entropy weight method is specifically as follows: The amount of information is determined by quantifying the uncertainty of entropy, and the entropy value is used to measure the degree of confusion or uncertainty of information. A higher entropy value indicates greater instability and uncertainty of information. This indicator has a greater impact on the evaluation results and should be assigned a higher weight. Assume that the system under test has n evaluation indicators, m systems under test, and x ij , i = 1, 2, 3, ... n; j = 1, 2, 3, ... m represents the data of the i-th indicator of the j-th sample. The specific steps of this method are as follows: 1) Normalize the data of each indicator; 2) The proportion of the characteristics of the jth evaluated indicator under the i-th indicator is P ij , as shown below: 3) The information entropy calculation formula of the i-th indicator is as follows: 4) For the evaluation index i, x of a certain system to be tested ij The smaller the fluctuation of the value between i The larger the value of x ij The greater the change in e i The smaller the value, the higher the degree of distinction of the index i on the evaluated object. The coefficient of difference is expressed as g i =1-e i It means that the higher the difference coefficient, the more important the indicator is and the greater the weight is. The final weight formula is as follows:

7. The method for evaluating the intelligent level of an intelligent system based on the fusion of multiple capability indicators according to claim 4 is characterized by: The combined weighting method is specifically as follows: In order to reduce the bias of subjective and objective weighting, the least square method was used; According to the basic principle of least squares method, as shown in the following formula, the weight of the combined weight is θ i ; h(θ i )=(w i -θ i ) 2 +(m i -θ i ) 2 For h(θ i ) is derived as shown below; h′(θ i )=4θ i -2w i -2m i Let h(θ i )=0, and we get θ i The value is as follows: w i represents the weight vector of AHP, ∑w i =1; μ i represents the weight vector of CRITIC weighting method, ∑μ i =1; θ i represents the combined weight vector, ∑θ i =1.

8. The method for evaluating the intelligent level of an intelligent system based on the fusion of multiple capability indicators according to claim 1 is characterized by: The step 3 is based on the intelligent level evaluation of multi-index fusion, and the specific steps are as follows: (1) Quantitative evaluation method; The TOPSIS comprehensive evaluation method is used as a quantitative evaluation method. TOPSIS is a method for ranking a limited number of evaluation objects according to their proximity to an idealized goal. It is used in multi-objective decision analysis to determine their order of merit by comparing and analyzing multiple evaluation objects. (2) Qualitative evaluation method; Use the multi-dimensional spider web model comprehensive evaluation method as a qualitative evaluation method to analyze the intelligence capabilities of different intelligent systems; The spider web model is used to dynamically analyze the impact of price fluctuations on the output of the next cycle. For autonomous evaluation, the multi-dimensional "spider web" model is a new evaluation model based on the spider web model.

9. The method for evaluating the intelligent level of an intelligent system based on the fusion of multiple capability indicators according to claim 8 is characterized by: The detailed implementation steps of the TOPSIS method are: 1) Construct a decision matrix; List all alternatives and evaluation indicators to form a decision matrix X. Assuming there are m alternatives and n evaluation indicators, the matrix is ​​in the form of: Among them, x ij It represents the value of the i-th alternative under the j-th evaluation index; 2) Standardized decision matrix; Since the dimensions of the evaluation indicators are different, the decision matrix needs to be standardized to ensure the comparability of different evaluation indicators. The standardization method is vector normalization: 3) construct a weighted normalization matrix; The weight of each evaluation index w j Applied to the normalized matrix, we get the weighted normalized matrix V: v ij =w j ·r ij where w j is the weight of the jth indicator; 4) Determine the ideal solution and the negative ideal solution; Positive ideal solution A + :The optimal value of each indicator is recorded as: A + =(maxv ij |j∈J1,minv ij |j∈j2) Among them, j1 represents the benefit-based indicator set, the larger the value, the better; j2 represents the cost-based indicator set, the smaller the value, the better; Negative ideal solution A - : The worst value of each indicator is recorded as: A - =(minv ij |j∈j1,maxv ij |j∈J2) 5) Calculate the Euclidean distance between each solution and the ideal solution and the negative ideal solution; Calculate the difference between each solution i and the positive ideal solution A + Distance: Calculate the difference between each solution i and the negative ideal solution A - Distance: 6) Calculate relative proximity: Calculate the relative closeness C of each solution i , that is, the degree to which solution i is close to the positive ideal solution and far away from the negative ideal solution at the same time: Among them, C i The value range of is [0,1], and the closer the value is to 1, the better the solution is; 7) Sort and select the best solution; According to the relative proximity C i Sort all alternatives in descending order, C i The solution with the largest value is the optimal solution.

10. The method for evaluating the intelligent level of an intelligent system based on the fusion of multiple capability indicators according to claim 8, characterized in that: The steps of constructing the multidimensional spider web model are as follows: 1) Establish the center point; 2) Extend the lines representing the capabilities of the intelligent system outward from the center point; 3) Determine the corresponding level of each line; 4) Connect the points. The area of ​​the figure enclosed by the dotted line represents the degree of intelligence of the evaluated system. The larger the enclosed area, the higher the intelligence.

Images