Efficiency evaluation method for man-machine hybrid decision-making system based on hierarchical component extraction

Abstract

The invention belongs to the technical field of artificial intelligence and decision science crossing, and discloses a man-machine hybrid decision system efficiency evaluation method based on hierarchical component extraction, comprising the following steps: acquiring physiological signal data and task performance data when an operator executes a task; performing normalization preprocessing on the physiological signal data and the task performance data; using an analytic hierarchy process to judge the matrix and calculate the weight, constructing an index importance judgment matrix, calculating a feature vector and the weight, and performing consistency check; carrying out weighted summation on the normalized data and the weight, and calculating a comprehensive efficiency score; according to the comprehensive efficiency score, the efficiency level of the man-machine hybrid decision-making system is judged; and performing man-machine hybrid decision system optimization according to the efficiency score. According to the man-machine hybrid decision-making system efficiency evaluation method based on hierarchical component extraction, the efficiency of the man-machine hybrid decision-making system is effectively evaluated by comprehensively considering the physiological data of the operator and the task performance level.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence and decision science, and in particular to a method for evaluating the performance of a human-machine hybrid decision-making system based on hierarchical component extraction. Background Technology

[0002] With the deep integration of artificial intelligence and automation technologies, human-machine hybrid decision-making systems are being used more and more widely in fields such as military, transportation, and finance. However, the technical shortcomings of existing performance evaluation methods severely restrict the reliability and adaptability of these systems in complex scenarios.

[0003] Most existing performance evaluation methods use single-dimensional evaluation indicators, such as classification accuracy and response time. This evaluation approach has obvious limitations. Single-dimensional evaluation indicators only focus on one aspect of the decision-making outcome, neglecting key quality attributes such as the logical traceability and deception resistance of the decision-making process.

[0004] In practical applications, the logical traceability of the decision-making process is crucial for analyzing the rationality of decisions, identifying potential problems, and optimizing decisions; while anti-fraud capability is a key factor in ensuring the safe and reliable operation of the system in complex environments.

[0005] Therefore, traditional methods cannot effectively evaluate the performance of human-machine hybrid decision-making systems. Based on this, this invention proposes a performance evaluation method for human-machine hybrid decision-making systems based on hierarchical component extraction, applicable to the quantitative evaluation and optimization of human-machine collaborative decision-making in complex environments. Summary of the Invention

[0006] The purpose of this invention is to provide a method for evaluating the effectiveness of human-machine hybrid decision-making systems based on hierarchical component extraction. This method comprehensively considers both the operator's physiological data and task performance level to effectively evaluate the effectiveness of human-machine hybrid decision-making systems and provides multi-dimensional data support for human-machine collaborative design.

[0007] To achieve the above objectives, this invention provides a method for evaluating the performance of a human-machine hybrid decision-making system based on hierarchical component extraction, comprising the following steps:

[0008] Step S1: Obtain physiological signal data and task performance data of the operator during task execution;

[0009] Step S2: Normalize and preprocess the physiological signal data and task performance data;

[0010] Step S3: Use the analytic hierarchy process (AHP) to determine the matrix and calculate the weights, construct the indicator importance determination matrix, calculate the eigenvectors and weights, and perform a consistency check.

[0011] Step S4: Calculate the overall performance score by summing the normalized data with the weights.

[0012] Step S5: Determine the performance level of the human-machine hybrid decision-making system based on the comprehensive performance score;

[0013] Step S6: Optimize the human-machine hybrid decision-making system based on the performance score.

[0014] Preferably, in step S1, physiological signal data and task performance data of the operator when performing the route planning task are acquired, and the specific process is as follows:

[0015] By monitoring and quantifying the operator's physiological signals and task execution results, physiological signal data and task performance data are obtained.

[0016] Physiological signal data is collected in real time through wearable sensors to reflect the operator's cognitive load; among them, physiological signal data includes heart rate (HR) and number of fixations (NFP); heart rate reflects the level of psychological tension and is a negative indicator; the number of fixations characterizes the efficiency of attention allocation and is a negative indicator.

[0017] Task performance data is automatically recorded by the system, specifically including task completion time (TCT) and navigation success rate (NSR). Task completion time measures decision-making efficiency and is a negative indicator; navigation success rate assesses planning accuracy and is a positive indicator.

[0018] In addition, questionnaire data was obtained through operators' subjective feedback on the task execution process.

[0019] Preferably, in step S2, the physiological signal data and task performance data undergo normalization preprocessing, the specific process of which is as follows:

[0020] Different normalization formulas are used to address the differences in the directionality of indicators;

[0021] The normalization formula for the negative index is:

[0022]

[0023] in, This represents the normalized observation data of the i-th group of negative indicators; This represents the maximum value of the negative index in the i-th group among all samples; This represents the observation data of the i-th group of original negative indicators; This represents the minimum value of the negative index in the i-th group among all samples;

[0024] The normalization formula for positive indicators is:

[0025]

[0026] in, This represents the normalized observation data of the i-th group of positive indicators; This represents the maximum value of the positive index in the i-th group among all samples; This represents the observation data of the i-th group of original positive indicators; This represents the minimum value of the positive index in the i-th group among all samples;

[0027] Based on the above calculation process, the normalized data range is [0,1].

[0028] Preferably, in step S3, the analytic hierarchy process (AHP) is used to determine the matrix and calculate the weights, constructing an indicator importance judgment matrix, calculating the eigenvectors and weights, and performing a consistency check. The specific process is as follows:

[0029] Step S31: Compare the indicators pairwise using the expert evaluation method to construct an indicator importance judgment matrix;

[0030] Step S32: Solve for the largest eigenvalue of the matrix using MATLAB, and calculate the eigenvector and weights to obtain the normalized weight vector;

[0031] Step S33: Perform a consistency check by calculating the consistency ratio to verify the logical rationality, as shown below:

[0032]

[0033] Where CI is the consistency index, representing the degree of consistency of the judgment matrix; n represents the order of the judgment matrix; RI represents the random consistency index; CR is the consistency ratio, used to determine the logical rationality; λ max This represents the largest eigenvalue.

[0034] Preferably, in step S4, the normalized data is summed with weights to calculate the overall performance score, as shown below:

[0035]

[0036] Where E represents the overall performance score; w j This represents the weight of the j-th indicator.

[0037] Preferably, in step S5, the performance level of the human-machine hybrid decision-making system is determined based on the comprehensive performance score, specifically including:

[0038] (1) When the performance evaluation score is greater than or equal to the first threshold, the performance of the human-machine hybrid decision-making system is excellent.

[0039] (2) When the performance evaluation score is greater than or equal to the second threshold and less than the first threshold, the performance of the human-machine hybrid decision-making system is good.

[0040] (3) When the performance evaluation score is less than the second threshold, the performance of the human-machine hybrid decision-making system is poor.

[0041] Preferably, in step S6, the human-machine hybrid decision-making system is optimized based on the performance score, as follows:

[0042] When the performance of the human-machine hybrid decision-making system is excellent, the current system parameters are maintained without adjustment, and the system continues to be used during flight.

[0043] When the performance of the human-machine hybrid decision-making system is good, adjust the human-machine interface to reduce gaze dispersion.

[0044] When the performance of the human-machine hybrid decision-making system is poor, optimize the threat avoidance algorithm and add decision-making assistance prompts.

[0045] Therefore, the present invention employs the above-mentioned human-machine hybrid decision-making system performance evaluation method based on hierarchical component extraction, and the beneficial effects are as follows:

[0046] (1) The present invention can accurately and dynamically analyze the operator's physiological data and task performance data, capture key information through feature extraction technology, reduce the misjudgment rate, and improve the reliability of the evaluation. This efficient evaluation capability is crucial for improving the safety and efficiency of the human-machine hybrid decision-making system, because it can monitor the driver's physiological state in real time, provide timely auxiliary decision support, and ensure the reliability of the human-machine hybrid decision-making system.

[0047] (2) The present invention fully considers the scalability of the system during the design process, allowing for customized adjustments based on different tasks and environments to adapt to the ever-changing human-machine hybrid decision-making. This scalability ensures that the system can maintain its advanced nature in the long term.

[0048] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. Attached Figure Description

[0049] Figure 1 This is a flowchart of a human-machine hybrid decision-making system performance evaluation method based on hierarchical component extraction according to the present invention.

[0050] Figure 2 This is a flowchart illustrating the calculation process of the analytic hierarchy process in an embodiment of the present invention. Detailed Implementation

[0051] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0052] like Figure 1 As shown, the present invention provides a method for evaluating the performance of a human-machine hybrid decision-making system based on hierarchical component extraction, comprising the following steps:

[0053] Step S1: Obtain physiological signal data and task performance data of the operator during task execution;

[0054] Step S2: Normalize and preprocess the physiological signal data and task performance data;

[0055] Step S3: Use the Analytic Hierarchy Process (AHP) to determine the matrix and calculate the weights, construct the indicator importance determination matrix, calculate the eigenvectors and weights, and perform a consistency check.

[0056] Step S4: Calculate the overall performance score by summing the normalized data with the weights.

[0057] Step S5: Determine the performance level of the human-machine hybrid decision-making system based on the comprehensive performance score;

[0058] Step S6: Optimize the human-machine hybrid decision-making system based on the performance score.

[0059] Example

[0060] Step S1: Obtain physiological signal data and task performance data of the operator when performing route planning tasks.

[0061] By monitoring and quantifying the operator's physiological signals and task execution results, physiological signal data and task performance data are obtained.

[0062] Physiological signal data is collected in real time through wearable sensors to reflect the operator's cognitive load. The physiological signal data includes heart rate (HR) and number of fixations (NFP). Heart rate reflects the level of psychological tension and is a negative indicator (the lower the value, the better). The number of fixations represents the efficiency of attention allocation and is a negative indicator.

[0063] Task performance data is automatically recorded by the system, specifically including Task Completion Time (TCT) and Navigation Success Rate (NSR). Task completion time measures decision-making efficiency and is a negative indicator; navigation success rate assesses planning accuracy and is a positive indicator (the higher the value, the better).

[0064] In addition, questionnaire data was obtained through operators' subjective feedback on the task execution process.

[0065] Step S2: Normalize and preprocess the physiological signal data and task performance data.

[0066] Different normalization formulas are used to address the differences in the directionality of the indicators.

[0067] The normalization formula for negative indicators (HR, NFP, TCT) is as follows:

[0068]

[0069] in, This represents the normalized observation data of the i-th group of negative indicators; This represents the maximum value of the negative index in the i-th group among all samples; This represents the observation data of the i-th group of original negative indicators; This represents the minimum value of the negative index in the i-th group among all samples;

[0070] The normalized formula for the positive index (NSR) is:

[0071]

[0072] in, This represents the normalized observation data of the i-th group of positive indicators; This represents the maximum value of the positive index in the i-th group among all samples; This represents the observation data of the i-th group of original positive indicators; This represents the minimum value of the positive index in the i-th group among all samples;

[0073] Based on the above calculation process, the normalized data range is [0,1], and the larger the value, the better the performance.

[0074] Step S3: Use the Analytic Hierarchy Process (AHP) to determine the importance of the indicators and calculate the weights. Construct the indicator importance determination matrix, calculate the eigenvectors and weights, and perform a consistency check. Figure 2 As shown.

[0075] Step S31: Construct the indicator importance judgment matrix.

[0076] By comparing each indicator pairwise using expert evaluation, a 4×4 judgment matrix A is constructed, as shown below:

[0077]

[0078] Step S32: Calculate the feature vector and weights.

[0079] Find the largest eigenvalue λ of matrix A using MATLAB. max =4.0604 and the corresponding eigenvector, after normalization, yield the weight vector, as shown below:

[0080] w = [0.21, 0.18, 0.24, 0.36] T ;

[0081] Where w represents the normalized weight vector; T represents the transpose operation.

[0082] Step S33: Consistency check.

[0083] The consistency ratio (CR) is calculated to verify the logical rationality, as shown below:

[0084]

[0085] Where CI is the consistency index, representing the degree of consistency of the judgment matrix; n represents the order of the judgment matrix; RI represents the random consistency index; CR is the consistency ratio, used to determine the logical rationality; λ max This represents the largest eigenvalue.

[0086] Based on the above calculation results, the weight allocation is verified to be effective.

[0087] Step S4: Calculate the overall performance score by summing the normalized data with the weights, as shown below:

[0088]

[0089] Where E represents the overall performance score; w j This represents the weight of the j-th indicator.

[0090] Based on the above calculation process, the comprehensive score results are shown in Table 1.

[0091] Table 1 Scoring Results

[0092] Test number Overall performance score 1 0.93 2 0.33 3 0.33

[0093] Step S5: Determine the performance level of the human-machine hybrid decision-making system based on the comprehensive performance score.

[0094] (1) When the performance evaluation score is greater than or equal to the first threshold, the performance of the human-machine hybrid decision-making system is excellent.

[0095] (2) When the performance evaluation score is greater than or equal to the second threshold and less than the first threshold, the performance of the human-machine hybrid decision-making system is good.

[0096] (3) When the performance evaluation score is less than the second threshold, the performance of the human-machine hybrid decision-making system is poor; optimize the threat avoidance algorithm and add decision-making assistance prompts.

[0097] Step S6: Optimize the human-machine hybrid decision-making system based on the performance score.

[0098] Table 1 shows that the human-machine hybrid decision-making system has the highest performance value under the first set of flight path tasks, indicating stable pilot heart rate, fewer fixation points, shorter time, and a high mission success rate. This demonstrates that the Analytic Hierarchy Process (AHP) can comprehensively and quantitatively evaluate mission performance during human-machine collaboration, providing objective and easily understandable assessment results. The obtained performance value can be used to adjust the human-machine hybrid decision-making system, thereby optimizing it.

[0099] In this embodiment, the performance value is calculated using the analytic hierarchy process combined with physiological data and task performance data, and the landing operation performance level value is obtained based on the operator's completion of the set tasks. The performance of the automatic landing auxiliary decision-making system is effectively evaluated by comprehensively using physiological evaluation methods and performance evaluation methods.

[0100] Based on the above analysis, the human-machine hybrid decision-making system was adjusted and improved according to the results of the final performance evaluation, as follows:

[0101] When the performance of the human-machine hybrid decision-making system is excellent, the current system parameters are maintained without adjustment, and the system continues to be used during flight to output the best flight path.

[0102] When the performance of the human-machine hybrid decision-making system is rated as good or poor, further improvements and optimizations are needed to the sensors, decision logic, intelligent algorithms, and human-machine permissions within the system, including:

[0103] When the performance of the human-machine hybrid decision-making system is good, adjust the human-machine interface to reduce gaze dispersion.

[0104] When the performance of the human-machine hybrid decision-making system is poor, optimize the threat avoidance algorithm and add decision-making assistance prompts.

[0105] Therefore, this invention adopts the above-mentioned human-machine hybrid decision-making system performance evaluation method based on hierarchical component extraction. It uses the analytic hierarchy process to quantify objective data (physiological signals, task performance), and uses judgment matrices and consistency checks to ensure the rationality of weight allocation. This solves the problem of fusion of heterogeneous data with multiple indicators, integrates subjective and objective evaluation results, reflects the task execution efficiency and safety under system assistance, and provides multi-dimensional data support for human-machine collaborative design.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can still be further explored.

[0107] Modifications or equivalent substitutions are made, but these modifications or equivalent substitutions cannot improve the modified technology.

[0108] The proposed solution deviates from the spirit and scope of the technical solution of this invention.

Claims

1. A method for evaluating the performance of a human-machine hybrid decision-making system based on hierarchical component extraction, characterized in that, Includes the following steps: Step S1: Obtain physiological signal data and task performance data of the operator during task execution; Step S2: Normalize and preprocess the physiological signal data and task performance data; Step S3: Use the analytic hierarchy process (AHP) to determine the matrix and calculate the weights, construct the indicator importance determination matrix, calculate the eigenvectors and weights, and perform a consistency check. Step S4: Calculate the overall performance score by summing the normalized data with the weights. Step S5: Determine the performance level of the human-machine hybrid decision-making system based on the comprehensive performance score; Step S6: Optimize the human-machine hybrid decision-making system based on the performance score.

2. The method for evaluating the performance of a human-machine hybrid decision-making system based on hierarchical component extraction according to claim 1, characterized in that, In step S1, physiological signal data and task performance data of the operator during the execution of the route planning task are acquired. The specific process is as follows: By monitoring and quantifying the operator's physiological signals and task execution results, physiological signal data and task performance data are obtained. Physiological signal data is collected in real time through wearable sensors to reflect the operator's cognitive load; among them, physiological signal data includes heart rate (HR) and number of fixations (NFP); heart rate reflects the level of psychological tension and is a negative indicator; the number of fixations characterizes the efficiency of attention allocation and is a negative indicator. Task performance data is automatically recorded by the system, specifically including task completion time (TCT) and navigation success rate (NSR). Task completion time measures decision-making efficiency and is a negative indicator; navigation success rate assesses planning accuracy and is a positive indicator. In addition, questionnaire data was obtained through operators' subjective feedback on the task execution process.

3. The method for evaluating the performance of a human-machine hybrid decision-making system based on hierarchical component extraction according to claim 1, characterized in that, In step S2, the physiological signal data and task performance data are preprocessed for normalization. The specific process is as follows: Different normalization formulas are used to address the differences in the directionality of indicators; The normalization formula for the negative index is: in, This represents the normalized observation data of the i-th group of negative indicators; This represents the maximum value of the negative index in the i-th group among all samples; This represents the observation data of the i-th group of original negative indicators; This represents the minimum value of the negative index in the i-th group among all samples; The normalization formula for positive indicators is: in, This represents the normalized observation data of the i-th group of positive indicators; This represents the maximum value of the positive index in the i-th group among all samples; This represents the observation data of the i-th group of original positive indicators; This represents the minimum value of the positive index in the i-th group among all samples; Based on the above calculation process, the normalized data range is [0,1].

4. The method for evaluating the performance of a human-machine hybrid decision-making system based on hierarchical component extraction according to claim 1, characterized in that, In step S3, the analytic hierarchy process (AHP) is used to determine the matrix and calculate the weights, constructing the indicator importance judgment matrix, calculating the eigenvectors and weights, and performing a consistency check. The specific process is as follows: Step S31: Compare the indicators pairwise using the expert evaluation method to construct an indicator importance judgment matrix; Step S32: Solve for the largest eigenvalue of the matrix using MATLAB, and calculate the eigenvector and weights to obtain the normalized weight vector; Step S33: Perform a consistency check by calculating the consistency ratio to verify the logical rationality, as shown below: Where CI is the consistency index, representing the degree of consistency of the judgment matrix; n represents the order of the judgment matrix; RI represents the random consistency index; CR is the consistency ratio, used to determine the logical rationality; λ max This represents the largest eigenvalue.

5. The method for evaluating the performance of a human-machine hybrid decision-making system based on hierarchical component extraction according to claim 1, characterized in that, In step S4, the normalized data is summed with weights to calculate the overall performance score, as shown below: Where E represents the overall performance score; w j This represents the weight of the j-th indicator.

6. The method for evaluating the performance of a human-machine hybrid decision-making system based on hierarchical component extraction according to claim 1, characterized in that, In step S5, the performance level of the human-machine hybrid decision-making system is determined based on the comprehensive performance score, specifically including: (1) When the performance evaluation score is greater than or equal to the first threshold, the performance of the human-machine hybrid decision-making system is excellent. (2) When the performance evaluation score is greater than or equal to the second threshold and less than the first threshold, the performance of the human-machine hybrid decision-making system is good. (3) When the performance evaluation score is less than the second threshold, the performance of the human-machine hybrid decision-making system is poor.

7. The method for evaluating the performance of a human-machine hybrid decision-making system based on hierarchical component extraction according to claim 1, characterized in that, In step S6, the human-machine hybrid decision-making system is optimized based on the performance score, as follows: When the performance of the human-machine hybrid decision-making system is excellent, the current system parameters are maintained without adjustment, and the system continues to be used during flight. When the performance of the human-machine hybrid decision-making system is good, adjust the human-machine interface to reduce gaze dispersion. When the performance of the human-machine hybrid decision-making system is poor, optimize the threat avoidance algorithm and add decision-making assistance prompts.

Images