System autonomy evaluation model design method
By constructing an autonomy evaluation index system and combining subjective and objective weighting methods, the problems of subjectivity and accuracy in the autonomy assessment of intelligent systems were solved, and the quantitative assessment and optimized design of the system's autonomy level were realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-15
- Publication Date
- 2026-04-10
AI Technical Summary
Existing methods for assessing the autonomy level of intelligent systems suffer from problems such as strong subjectivity, inconsistent standards, and difficulty in verifying accuracy, resulting in insufficient stability and credibility of the assessment results.
An autonomous evaluation index system was constructed, and a weighting method combining subjective and objective weighting methods was adopted. Through index homogenization and standardization, combined with expert experience and data characteristics, a weighted comprehensive method was designed for quantitative evaluation.
It improves the accuracy and reliability of the autonomous assessment of intelligent systems, provides quantitative evaluation results and improvement suggestions, and supports the autonomous optimization design of the system.
Smart Images
Figure CN121834263A_ABST
Abstract
Description
Technical Field
[0001] This invention mainly relates to the field of integrated system support technology, and in particular to a design method for a system autonomy assessment model. Background Technology
[0002] Quantitatively assessing the autonomy level of intelligent systems is fundamental to achieving autonomous optimization design and capability enhancement. The core of this process lies in mining and synthesizing autonomy indicator information to generate centralized, cohesive, and intuitive autonomy assessment results. This allows users to grasp the overall autonomy status and weaknesses, facilitating comparison and optimization. Such results not only help users comprehensively understand the overall development status of system autonomy but also clearly identify weaknesses hindering performance improvement, thus providing a reliable basis for comparing autonomy levels between different systems or versions and for targeted optimization decisions.
[0003] However, existing autonomous assessment technologies still rely on traditional methods such as expert surveys and qualitative scoring, which have obvious limitations: First, they are highly subjective, and the assessment results are easily affected by expert experience and personal judgment, resulting in insufficient stability; second, the standards are not uniform, and the evaluation systems of different institutions or projects differ greatly, making it difficult to compare assessment conclusions horizontally; third, the accuracy is difficult to verify, lacking objective data support, and the credibility and repeatability of the assessment results are questionable.
[0004] Therefore, it is urgent to study new quantitative evaluation models to achieve uniformity and standardization of indicator evaluation, improve the credibility of evaluation results, and provide a complete, scientific, and engineering-feasible scheme design method for the quantitative and accurate evaluation of the autonomy of intelligent systems. Summary of the Invention
[0005] To address the issues of strong subjectivity, inconsistent standards, and unknown accuracy in existing technologies, this invention aims to provide a design method for a system autonomy assessment model. This method constructs an autonomy evaluation index system by combining the characteristics of the intelligent system itself; designs standardization methods for index types; then quantitatively determines the importance of the indicators and establishes comprehensive weights by comprehensively utilizing expert experience; after determining the index set, weight set, and comment set, a weighted synthesis method and a multiplicative synthesis method are used to synthesize the results layer by layer from low to high levels to obtain the final quantitative evaluation results. The invention also provides interpretation conclusions and result analysis of the evaluation results, supporting subsequent autonomous optimization design and improving the accuracy of autonomy assessment.
[0006] To achieve the above objectives, this invention provides a method for designing a system autonomy assessment model, comprising the following steps: Step 1: Establish a comprehensive evaluation index system for the system's autonomous performance; Step 1.1: Classify the system autonomy level: Use the level of cooperation between humans and intelligent systems as the basis for classifying the system autonomy level, and set the system autonomy evaluation results into several levels; Step 1.2: Construct an indicator tree: Construct a three-level security evaluation indicator system, namely the target layer, the subsystem layer, and the indicator layer. The target layer represents the comprehensive autonomy of the intelligent system, the subsystem layer includes each subsystem in the intelligent system, and the indicator layer is specifically designed according to the functional requirements and autonomy characteristics of different subsystems. Step 1.3: Complete the autonomy evaluation process design: Using subsystems as the descriptive dimension, establish indicator requirements based on different autonomy evaluation results as the explicit basis for subsystem evaluation, form an autonomy grading characteristic description table, construct a weight set and a quantitative parameter set, and combine the grade evaluation results with the scores under that grade as the final system autonomy evaluation results; Step 1.4: Indicator alignment and standardization: After obtaining the parameter values of the final system autonomy evaluation results, the parameter values are aligned and standardized to eliminate the influence of outliers and dimensions, which facilitates subsequent quantitative evaluation. Step 2: Determine the importance weights of the indicators using a combination of subjective and objective weighting methods, and then perform weight adjustment and synthesis. Step 3: Complete the quantitative evaluation and interpretation of autonomy based on weighted addition.
[0007] Furthermore, in step 1.1, the system autonomy evaluation results are set into three levels: machine-assisted A1, human-machine collaboration A2, and machine autonomy A3, with machine-assisted A1 being the lowest level and machine autonomy A3 being the highest level.
[0008] Furthermore, the indicators in the security evaluation index system in step 1.2 include combat capability effectiveness indicators, human-computer interaction capability indicators, and learning optimization capability indicators.
[0009] Furthermore, in step 1.3, the evaluation of the autonomy level of the intelligent system adopts a sequential testing method from low level to high level. If the requirements of the autonomy level description table indicators are not met, the testing subjects of the higher level are stopped, and the corresponding level quantitative scoring is carried out at the same time.
[0010] Furthermore, step 1.4 involves standardizing and aligning the parameter values, including the following steps: Step 1.4.1: Perform homogenization and divide the autonomous evaluation parameters of the intelligent system into four categories: extremely large parameters, extremely small parameters, intermediate parameters, and interval parameters; Step 1.4.2: Classify the safety evaluation indicators according to the intelligent system autonomy evaluation parameters, and then perform homogenization processing on each.
[0011] Furthermore, step 2, the subjective weighting method weight calculation, includes the following steps: Step 2.1: First, perform pairwise comparisons between the safety evaluation indicators to construct a judgment matrix; Step 2.2: Perform a consistency check. Use the negative average of the remaining eigenvalues of the judgment matrix (excluding the largest eigenvalue) as an indicator of the matrix's deviation from consistency. The expression for the indicator of the matrix's deviation from consistency is: ,in To determine the largest eigenvalue of matrix C, n is the order of the matrix. Step 2.3: Compare the consistency index of the judgment matrix and the random matrix to quantitatively determine the metric that satisfies consistency, expressed as follows: , where RI is the consistency index of the random matrix. When the CR value is less than 0.1, the judgment matrix is considered to be consistent. Step 2.4: Calculate the comprehensive weight of the index using the square root method through the judgment matrix C, including: Step 2.4.1: Calculate the product of the elements in each row of the judgment matrix C; Step 2.4.2: For the product of the elements in each row, calculate its nth root; Step 2.4.3: Form a vector from the nth root and normalize it.
[0012] Furthermore, step 2, the objective weighting method for weight calculation, includes the following steps: Step 2.5: Adjust the weights using the coefficient of variation method. The internal contrast strength of index x is measured using the coefficient of variation S. The expression for the coefficient of variation S is: ,in, The standard deviation of the representative indicator data, The mean of the representative indicator data; Step 2.6: After normalizing the coefficient of variation S of each indicator data, the comprehensive weight value of the indicator obtained in the objective weighting method is obtained.
[0013] Furthermore, in step 2, the weight adjustment and synthesis are carried out using... The final comprehensive weight is obtained by synthesis, where W is the final comprehensive weight, and W1 and W2 are the weights obtained by the subjective weighting method and the objective weighting method, respectively. This is the proportionality coefficient.
[0014] Furthermore, step 3 completes the quantitative evaluation and interpretation of autonomy based on weighted addition, including the following steps: Step 3.1: Subsystem Autonomy Score Synthesis: The subsystem evaluation model is as follows: Wherein, Score is the autonomy score of the subsystem. This indicates the task completion status of the subsystem. This indicates the level of human-computer interaction in the subsystem. This indicates the subsystem's learning and optimization capabilities. The weighting coefficients represent the degree of human-computer interaction at different levels of autonomy. The weighting coefficients represent the learning optimization capabilities at different levels of autonomy. Step 3.2: System Autonomy Score Synthesis: After obtaining the autonomy level and score of each subsystem, a weighted comprehensive evaluation method is used to assess the system's autonomy level. Weights are assigned according to the importance of each subsystem, and the sum of the weighted products of the autonomy scores of each subsystem is calculated to obtain the system's autonomy score. The expression is as follows: ,in, The system's autonomy is scored, where n is the number of subsystems. Let i be the weight of subsystem i. Score the autonomy of subsystem i; Step 3.3: After obtaining the system's autonomy score, based on the established correspondence between autonomy levels and autonomy score ranges, the system's autonomy level is obtained; Step 3.4: Complete the system autonomy result synthesis and analysis: Use the results of each indicator and their weights to perform weighted summation to obtain the final score, and analyze the results.
[0015] Furthermore, the results are analyzed, including plotting the weights of each subsystem or its corresponding indicator on the X-axis and the scores on the Y-axis in a coordinate system, and assigning four labels—optimization, attention, assurance, and maintenance—according to the region to guide the direction of improving the system's autonomy.
[0016] Beneficial effects: This invention provides a system autonomy assessment model design method with the following beneficial effects: (1) It combines expert knowledge and data characteristics to solve the problems of strong subjectivity and difficulty in grasping accuracy in traditional assessment methods; (2) It establishes a comprehensive evaluation index system for the autonomous performance of intelligent systems, including classifying autonomy levels, constructing an intelligent system autonomous performance index set, designing an evaluation process, and providing effective parameter index processing operation methods; (3) It designs a weighting method that combines subjective and objective approaches, avoids the problem of data lack in the design stage, and uses the subjective weighting method to determine the weights using expert experience; after accumulating sufficient data in the test and operation stages, it uses the data's own characteristics to correct the comprehensive weights using the objective weighting method, thereby improving the accuracy of the comprehensive evaluation; (4) It provides a quantitative evaluation and result interpretation method for autonomous performance based on weighted addition. After determining the index set, weight set, and comment set, it designs a method for synthesizing the autonomy scores of specific subsystems and a method for calculating the weighted comprehensive system indexes, integrating them layer by layer from low to high levels to obtain the final quantitative evaluation results, and provides interpretation conclusions and improvement suggestions for the evaluation results. Attached Figure Description
[0017] Figure 1 This is a schematic diagram illustrating the principle of autonomous quantitative assessment; Figure 2 This is a flowchart for assessing the autonomy level of intelligent systems; Figure 3 It is a mapping function for index standardization; Figure 4 This is a diagram illustrating the weighted survey form and the 9-point scale method; Figure 5 It is a flowchart of subjective empowerment; Figure 6 This is the logic design diagram for indicator synthesis; Figure 7 This is a diagram illustrating the improvement of autonomy. Detailed Implementation
[0018] The preferred mechanisms and implementation methods of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0019] like Figures 1-7 As shown, this invention provides a system autonomy assessment model design method, aiming to solve the problems of strong subjectivity and difficulty in ensuring accuracy in traditional autonomy assessment processes. This invention breaks through fundamental technologies such as indicator systems, synthesis methods, quantification functions, and weight design to quantitatively assess the autonomy level of a system. Its general principle is as follows: First, describe the indicator system and synthesis tree to be constructed; second, design the indicator quantification function; third, confirm the indicator weights; and finally, conduct application verification. Its assessment principle is as follows: Figure 1 As shown.
[0020] Example 1: This invention provides a method for designing a system autonomy assessment model, comprising the following steps: Step 1: Establish a comprehensive evaluation index system for the system's autonomous performance; Step 2: Determine the importance weights of the indicators using a combination of subjective and objective weighting methods, and then perform weight adjustment and synthesis. Step 3: Complete the quantitative evaluation and interpretation of autonomy based on weighted addition.
[0021] I. Establish a comprehensive evaluation index system for the autonomous performance of the system, which mainly includes the following steps: (1) Autonomy level classification Humans and intelligent systems cooperate in different ways when performing tasks. These collaborations can be proactive, negotiated, or passive. The distinction between these modes primarily lies in the degree of human dominance and the division of labor between humans and intelligent systems. A lower degree of human dominance indicates a higher level of intelligence in the intelligent system, enabling it to complete most tasks independently and demonstrating a higher level of autonomy. Conversely, a higher degree of human dominance indicates a lower level of intelligence in the intelligent system, requiring greater external assistance and resulting in a lower level of autonomy. Therefore, this embodiment uses the level of human-intelligent system cooperation as the basis for classifying the level of autonomy in the intelligent system, setting the evaluation results of autonomy capabilities into three levels: machine-assisted (A1), human-machine collaboration (A2), and machine autonomy (A3).
[0022] (2) Construction of indicator tree Based on the relevant standards' definition and requirements for autonomy, the testing of the autonomous capabilities of intelligent systems mainly focuses on setting indicators related to their "combat capability effectiveness," "degree of human manipulation / intervention (human-computer interaction capability)," and "learning and optimization capability." Through literature review, research, expert consultation, and consideration of the characteristics of intelligent systems, this embodiment constructs a security evaluation indicator system at three levels: the target layer, the subsystem layer, and the indicator layer. The target layer represents the overall autonomy of the intelligent system, denoted as U; the subsystem layer includes each subsystem within the system; and the indicator layer is specifically designed according to the functional requirements and autonomous characteristics of different subsystems, such as the anti-interference analysis system including success rate and the proportion of human assistance.
[0023] (3) Design of the autonomy evaluation process After determining the indicator tree, it is necessary to combine it with the autonomy level classification, using subsystems as the descriptive dimension, to establish indicator requirements at different autonomy levels, serving as explicit basis for subsystem evaluation, and forming an autonomy level characteristic description table, the table format of which is shown in Table 1: .
[0024] The system's autonomy level assessment process adopts a tiered judgment system. Under the autonomy capability test subject of each subsystem function, the autonomy level is determined based on the test results of the autonomy indicators and the satisfaction of each indicator with respect to the tiered characteristic table in Table 1. Simultaneously, to support the improvement of autonomy capabilities and the comparison of autonomy among different systems, a quantitative score is calculated based on the comprehensive autonomy indicator test results under the corresponding autonomy level.
[0025] The system's autonomy level assessment adopts a sequential testing approach from lower to higher levels. If the requirements of the grading characteristic table for a certain level are not met, the testing of the higher-level subjects is stopped, and a quantitative score for the corresponding level is carried out simultaneously. The process is as follows: Figure 2 As shown. Figure 2 A0 in the middle represents a lack of autonomy.
[0026] As described above, the quantitative assessment of autonomy adopts a three-level indicator (A1, A2, A3) evaluation scheme. For each level, such as the autonomy classification characteristic description table, a separate set of weights and quantitative parameters are constructed.
[0027] The determination of the autonomy level is based on the "Intelligent Manufacturing Capability Maturity Assessment Standard". First, the level is judged from low to high according to the grading characteristic table. Only after meeting the requirements of the lower level can the evaluation parameters of the higher level be used for judgment. If the judgment criteria are not met, the results of each indicator are combined with the weight of the lower level to obtain the final score.
[0028] The rating results are then combined with the scores under that rating to obtain the final system autonomy rating result.
[0029] (4) Indicator alignment and standardization After obtaining the parameter values, they need to be normalized and standardized to eliminate the influence of outliers, dimensions, etc., to facilitate subsequent quantitative evaluation. The method for normalizing and standardizing the design indicators in this patent is as follows: First, there's the issue of alignment. Based on performance evaluation relationships, autonomy evaluation parameters can generally be categorized into four types: "Extremely large" parameter: The larger the value, the better; "Extremely small" parameter: the smaller the value, the better; "Center-aligned" parameter: The more centered the value, the better; "Interval type" parameter: The best value is one that falls within a certain interval.
[0030] The indicators are categorized according to the above types, and then subjected to homogenization processing, as follows: For the "extremely large" parameter x, it remains unchanged; For the "minimal" parameter x, let: , where M is an allowed upper bound for x; For the "centering" parameter x, let: , where m is an allowed lower bound for the parameter; M is an allowed upper bound for the parameter; For the "interval type" parameter x, let: ,in[ , Let ] be the optimal stable interval of parameter x, and M and m be the allowable upper and lower bounds of parameter x, respectively.
[0031] The above operations transform the parameters into extremely large indicators. Next, to eliminate the influence of differences in the dimensions and magnitudes of the various indicators and to avoid irrational phenomena, the evaluation indicators are standardized (dimensionless), specifically for "extremely large" indicators: , where M and m are the maximum and minimum values of the indicator value sequence, respectively, corresponding to different membership functions, expressing the strictness of the indicator requirements at this level.
[0032] The underlying performance indicators of a system autonomy evaluation index system often have different dimensions and units, making direct calculation impossible. Therefore, quantization functions are needed for preprocessing to address the comparability issues between indicator data. Quantization functions include... Figure 3 The diagram illustrates the mapping between actual indicator values and quantification results, with its shape determined by the indicator quantification parameters (q1, q2, m, M). Different indicator quantification parameters need to be designed for different evaluation indicators to achieve standardization and dimensionlessness. For example, in lower-level autonomy assessments, where autonomous target recognition does not place high demands on the intelligent system's learning capabilities, a lower m value can be set to ensure a higher score even at lower levels; conversely, in higher-level assessments, a relatively higher m value can be designed to impose stricter requirements on target recognition accuracy.
[0033] Outliers often appear during the acquisition of autonomous parameters, and these outliers can significantly affect the stability of the standardized indicators. Therefore, an extreme value handling method is used to improve this process. This involves pre-setting upper and lower limits x_high and x_low, causing the original "extremely large" indicator x to undergo a limit preprocessing step. The standardized indicator results can be obtained through the above operations for subsequent performance evaluation.
[0034] II. A weighting method combining subjective and objective weighting is used to determine the importance weights of indicators, followed by weight adjustment and synthesis. This includes the following steps: (1) Calculation of weights using the subjective weighting method In this embodiment, when performing subjective weighting, pairwise comparisons are first performed between indicators to construct a judgment matrix. The judgment matrix represents a comparison of the relative importance of an evaluation target (such as comprehensive autonomy) to its subsystems, assigning a specific numerical value to each "importance," according to... Figure 4 The weighted survey table is constructed using the 9-level scaling method shown. Experts were consulted to complete the judgment matrix according to the judgment scale shown in the right figure. The table shown is for illustrative purposes only and does not represent the actual indicators. The subjective weighting process is as follows: Figure 5 As shown.
[0035] After completing the survey form, a pairwise comparison judgment matrix can be obtained. ,in This represents the relative importance of indicator i and indicator j to the evaluation target, where n is the number of indicators.
[0036] Due to factors such as subjective bias, it is difficult for experts to ensure consistency among their judgments on the importance of indicators; that is, contradictions often exist between the assessments of indicator importance, necessitating consistency checks. The negative average of the remaining eigenvalues of the judgment matrix (excluding the largest eigenvalue) is used as an indicator to measure the consistency of the judgment matrix, i.e.: ,in Let n be the largest eigenvalue of the judgment matrix C, and n be the order of the judgment matrix.
[0037] Since maintaining perfect consistency is difficult in practice, a slight inconsistency in the judgment matrix can be considered acceptable. The consistency index of the judgment matrix is quantitatively determined by comparing it with that of the random matrix, using the following criteria: Where RI is the consistency index of the random matrix, which can be obtained from Table 2. When the RI value is less than 0.1, the judgment matrix is considered to have good consistency and can be used for subsequent comprehensive weight calculation. Table 2 is as follows: .
[0038] Finally, the root method is used to calculate the comprehensive weight of the index through the judgment matrix C, including the following steps: First, calculate the product of the elements in each row of the matrix: ; For the product of the elements in each row, calculate its nth root: ; Form a vector from the nth root and normalize it: ; but This refers to the comprehensive weighting result of the indicators obtained by the subjective weighting method of individual experts. Multiple experts can simply average the obtained weights.
[0039] (2) Calculation of weights using the objective weighting method Once a certain amount of data has been accumulated, the weights are adjusted using the coefficient of variation method to improve accuracy. The coefficient of variation method uses the strength of the comparison within an indicator's data to characterize its importance. The strength of the comparison within an indicator x (after normalization) is typically measured using the coefficient of variation S. ,in The standard deviation of the representative indicator data, The mean of the representative indicator data is then normalized using the coefficient of variation S for each indicator data to obtain the comprehensive weight of the indicator obtained in the objective weighting method.
[0040] (3) Weight Adjustment Synthesis After accumulating sufficient data, adopt The final comprehensive weight is obtained by synthesis, where W is the final comprehensive weight, and W1 and W2 are the weights obtained by the subjective weighting method and the objective weighting method, respectively. This is a proportionality coefficient, determined based on expert experience and data quality.
[0041] III. Complete the quantitative evaluation and interpretation of autonomy based on weighted addition. The quantitative evaluation and interpretation of autonomy based on weighted addition includes the following steps: (1) Subsystem autonomy score synthesis The subsystem evaluation model can be described using the following formula: Wherein, Score is the autonomy score of the subsystem. This indicates the task completion status of the subsystem, which is obtained based on whether its operational effectiveness indicators meet the requirements. Specifically, it refers to the key indicators of each subsystem. If the requirements are met, the value is 1; otherwise, it is 0, indicating that the autonomy evaluation needs to be based on the completion of the task. This indicates the level of human-computer interaction in the subsystem. This indicates the subsystem's learning and optimization capabilities. The weighting coefficients represent the degree of human-computer interaction at different levels of autonomy. The weighting coefficients representing the learning optimization capabilities at different levels of autonomy are used to calculate the autonomy score of the subsystem.
[0042] Some indicators in the indicator system, such as performance indicators, are fundamental to intelligent systems and possess a veto power. Therefore, they can be set as multiplicative composite terms, multiplied by the preliminary evaluation result: when the requirement is not met, the indicator is set to 0, resulting in a score of 0; when the requirement is met, the indicator is set to 1, and the final output is the composite result of other indicators. Other indicators are set as weighted additive composite terms to obtain the preliminary evaluation result. The specific composite model is as follows: Figure 6 As shown.
[0043] (2) Synthesis of system autonomy score After obtaining the autonomy level and score of each subsystem, a weighted comprehensive evaluation method is used to assess the system's autonomy level. Weights are assigned according to the importance of each subsystem, and the sum of the weighted products of the autonomy scores of each subsystem is calculated to obtain the system's autonomy score, expressed as: ,in, The system's autonomy is scored, where n is the number of subsystems. Let i be the weight of subsystem i. The autonomy score is given to subsystem i.
[0044] After obtaining the system's autonomy score, the system's autonomy level is obtained based on the established correspondence between autonomy levels and autonomy score ranges.
[0045] (3) Synthesis and analysis of system autonomy results The autonomy level assessment of a subsystem should be conducted under the autonomous capability test subjects of that subsystem's functions. Based on the test results of the autonomy indicators, the autonomy score for each level is obtained according to the satisfaction of each indicator with the grading characteristic table. A sequential testing approach from lower to higher levels is adopted. If the score requirements of a certain level are not met, the testing subject for the higher level is stopped. As described above, the quantitative assessment of autonomy adopts a three-level indicator (A1, A2, A3) evaluation scheme, and a separate set of weights and quantitative parameters is constructed for each level. The determination of the autonomy level refers to the "Intelligent Manufacturing Capability Maturity Assessment Standard". First, calculations are performed from low to high according to the low-level evaluation model. The results of each indicator are combined with the weights to obtain a weighted sum to obtain the final score.
[0046] After calculating the autonomy evaluation results, the results need to be analyzed. For each subsystem or sub-system, its weight and score are plotted on the X and Y axes respectively in a coordinate system. Based on its region, it is assigned one of four labels: "Optimize," "Pay Attention," "Guarantee," or "Maintain," thereby guiding the direction for improving autonomy. Figure 7 As shown.
[0047] This invention provides a method for designing a system autonomy assessment model. Through expert consultation and practical research, and considering the characteristics of the intelligent system itself, an autonomy evaluation index system is constructed. Standardization methods are designed for different index types. Then, expert experience is comprehensively utilized to quantitatively determine the importance of each index and establish its comprehensive weight. After the index set, weight set, and comment set are determined, a weighted synthesis method and a multiplicative synthesis method are used to synthesize the results layer by layer from low to high levels, obtaining the final quantitative evaluation result. The invention also provides interpretations and analysis of the evaluation results, supporting subsequent autonomous optimization design and improving the accuracy of autonomy assessment.
[0048] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. However, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for designing a system autonomy assessment model, characterized in that, Includes the following steps: Step 1: Establish a comprehensive evaluation index system for the system's autonomous performance; Step 1.1: Classify the system autonomy level: Use the level of cooperation between humans and intelligent systems as the basis for classifying the system autonomy level, and set the system autonomy evaluation results into several levels; Step 1.2: Construct an indicator tree: Construct a three-level security evaluation indicator system, namely the target layer, the subsystem layer, and the indicator layer. The target layer represents the comprehensive autonomy of the intelligent system, the subsystem layer includes each subsystem in the intelligent system, and the indicator layer is specifically designed according to the functional requirements and autonomy characteristics of different subsystems. Step 1.3: Complete the autonomy evaluation process design: Using subsystems as the descriptive dimension, establish indicator requirements based on different autonomy evaluation results as the explicit basis for subsystem evaluation, form an autonomy grading characteristic description table, construct a weight set and a quantitative parameter set, and combine the grade evaluation results with the scores under that grade as the final system autonomy evaluation results; Step 1.4: Indicator alignment and standardization: After obtaining the parameter values of the final system autonomy evaluation results, the parameter values are aligned and standardized to eliminate the influence of outliers and dimensions, which facilitates subsequent quantitative evaluation. Step 2: Determine the importance weights of the indicators using a combination of subjective and objective weighting methods, and then perform weight adjustment and synthesis. Step 3: Complete the quantitative evaluation and interpretation of autonomy based on weighted addition.
2. The system autonomy assessment model design method according to claim 1, characterized in that, In step 1.1, the system autonomy evaluation results are set to three levels: machine assistance A1, human-machine collaboration A2, and machine autonomy A3. Among them, machine assistance A1 is the lowest level and machine autonomy A3 is the highest level.
3. The system autonomy assessment model design method according to claim 1, characterized in that, The indicators in the security evaluation index system in step 1.2 include combat capability effectiveness indicators, human-computer interaction capability indicators, and learning optimization capability indicators.
4. The system autonomy assessment model design method according to claim 1, characterized in that, In step 1.3, the evaluation of the autonomy level of the intelligent system adopts a sequential testing method from low level to high level. If the requirements of the autonomy level description table indicators are not met, the testing subjects of the higher level are stopped, and the corresponding level quantitative scoring is carried out at the same time.
5. The system autonomy assessment model design method according to claim 1, characterized in that, Step 1.4 involves standardizing and aligning the parameter values, including the following steps: Step 1.4.1: Perform homogenization and divide the autonomous evaluation parameters of the intelligent system into four categories: extremely large parameters, extremely small parameters, intermediate parameters, and interval parameters; Step 1.4.2: Classify the safety evaluation indicators according to the intelligent system autonomy evaluation parameters, and then perform homogenization processing on each.
6. The system autonomy assessment model design method according to claim 1, characterized in that, Step 2, the subjective weighting method for weight calculation, includes the following steps: Step 2.1: First, perform pairwise comparisons between the safety evaluation indicators to construct a judgment matrix; Step 2.2: Perform a consistency check. Use the negative average of the remaining eigenvalues of the judgment matrix (excluding the largest eigenvalue) as an indicator of the matrix's deviation from consistency. The expression for the indicator of the matrix's deviation from consistency is: ,in To determine the largest eigenvalue of matrix C, n is the order of the matrix. Step 2.3: Compare the consistency index of the judgment matrix and the random matrix to quantitatively determine the metric that satisfies consistency, expressed as follows: , where RI is the consistency index of the random matrix. When the CR value is less than 0.1, the judgment matrix is considered to be consistent. Step 2.4: Calculate the comprehensive weight of the index using the square root method through the judgment matrix C, including: Step 2.4.1: Calculate the product of the elements in each row of the judgment matrix C; Step 2.4.2: For the product of the elements in each row, calculate its nth root; Step 2.4.3: Form a vector from the nth root and normalize it.
7. The system autonomy assessment model design method according to claim 6, characterized in that, Step 2, the objective weighting method for weight calculation, includes the following steps: Step 2.5: Adjust the weights using the coefficient of variation method. The internal contrast strength of index x is measured using the coefficient of variation S. The expression for the coefficient of variation S is: ,in, The standard deviation of the representative indicator data, The mean of the representative indicator data; Step 2.6: After normalizing the coefficient of variation S of each indicator data, the comprehensive weight value of the indicator obtained in the objective weighting method is obtained.
8. The system autonomy assessment model design method according to claim 7, characterized in that, In step 2, the weight adjustment synthesis is adopted. The final comprehensive weight is obtained by synthesis, where W is the final comprehensive weight, and W1 and W2 are the weights obtained by the subjective weighting method and the objective weighting method, respectively. This is the proportionality coefficient.
9. The system autonomy assessment model design method according to claim 3, characterized in that, Step 3 involves completing the quantitative evaluation and interpretation of autonomy based on weighted addition, including the following steps: Step 3.1: Subsystem Autonomy Score Synthesis: The subsystem evaluation model is as follows: Wherein, Score is the autonomy score of the subsystem. This indicates the task completion status of the subsystem. This indicates the level of human-computer interaction in the subsystem. This indicates the subsystem's learning and optimization capabilities. The weighting coefficients represent the degree of human-computer interaction at different levels of autonomy. The weighting coefficients represent the learning optimization capabilities at different levels of autonomy. Step 3.2: System Autonomy Score Synthesis: After obtaining the autonomy level and score of each subsystem, a weighted comprehensive evaluation method is used to assess the system's autonomy level. Weights are assigned according to the importance of each subsystem, and the sum of the weighted products of the autonomy scores of each subsystem is calculated to obtain the system's autonomy score. The expression is as follows: ,in, The system's autonomy is scored, where n is the number of subsystems. Let i be the weight of subsystem i. Score the autonomy of subsystem i; Step 3.3: After obtaining the system's autonomy score, based on the established correspondence between autonomy levels and autonomy score ranges, the system's autonomy level is obtained; Step 3.4: Complete the system autonomy result synthesis and analysis: Use the results of each indicator and their weights to perform weighted summation to obtain the final score, and analyze the results.
10. The system autonomy assessment model design method according to claim 9, characterized in that, The results are analyzed, including plotting the weights of each subsystem or its corresponding indicator on the X-axis and the scores on the Y-axis in a coordinate system, and assigning four labels—optimization, attention, assurance, and maintenance—according to the region to guide the direction of improving the system's autonomy.