Human-computer interaction behavior multi-attribute comprehensive test and evaluation method
By constructing a human-computer interaction behavior evaluation index system based on the NASA-TLX scale method that combines subjective and objective evaluations, and combining it with a multi-dimensional feature hierarchical fusion method, the shortcomings of the traditional evaluation system are solved, and a comprehensive and detailed evaluation of human-computer interaction behavior is achieved, thereby improving the design and use efficiency of aviation equipment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINESE FLIGHT TEST ESTAB
- Filing Date
- 2025-12-27
- Publication Date
- 2026-05-08
AI Technical Summary
Traditional human-computer interaction evaluation systems lack comprehensiveness and detail, failing to describe and evaluate the detailed processes of human-computer interaction behavior, and lacking comprehensive consideration based on mission tasks.
Design a comprehensive testing and evaluation method for human-computer interaction behavior with multiple attributes. Use the NASA-TLX scale method to construct an evaluation index system that combines subjective and objective evaluations. Use a multi-dimensional feature hierarchical fusion evaluation method and combine typical task profiles for testing and evaluation.
It enables a comprehensive and detailed evaluation of human-computer interaction behavior, provides a basis for design improvement, and enhances the effectiveness of aviation equipment systems.
Smart Images
Figure CN121998480A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of human-computer interaction evaluation technology, and in particular relates to a method for comprehensive testing and evaluation of multiple attributes of human-computer interaction behavior. Background Technology
[0002] Effective human-machine interaction plays a crucial role in the normal operation of aircraft and is an important component of aircraft design. During the performance qualification testing phase of aviation equipment, testing and evaluating human-machine interaction behavior helps improve human work performance, reduce operational errors, and achieve safe operation.
[0003] Traditional human-computer interaction (HCI) evaluation systems are static, function-oriented, and primarily subjective, failing to describe and evaluate the detailed processes of HCI behavior. Furthermore, these systems typically analyze and evaluate in isolation at a specific design stage, lacking a comprehensive and holistic consideration of HCI behavior based on mission objectives. Therefore, there is an urgent need to construct a human-centered HCI behavior testing and evaluation system that combines subjective evaluation with objective measurement.
[0004] Currently, determining the evaluation indicators and models for human-computer interaction behavior, as well as developing testing plans for human-computer interaction behavior, are all urgent issues that need to be addressed in carrying out human-computer interaction behavior testing and evaluation. Summary of the Invention
[0005] The purpose of this invention is to design a comprehensive testing and evaluation method for human-computer interaction behavior with multiple attributes, to guide the testing and evaluation of human-computer interaction behavior, to evaluate whether the human-computer interaction level of aviation equipment meets the development requirements, and to identify problems and deficiencies in the design, use and maintenance of aviation equipment, providing a basis for design changes and improving the system efficiency of aviation equipment.
[0006] This application provides a method for comprehensive testing and evaluation of multiple attributes of human-computer interaction behavior, the method comprising: S101. Based on the principles for determining the influencing factors and indicators of human-computer interaction behavior, provide evaluation indicators for human-computer interaction behavior. S102. Based on the multi-attribute characteristics of the measured values of the human-computer interaction behavior evaluation indicators, determine the weight coefficients of each evaluation indicator; S103. Based on the multi-attribute characteristics of the measured values of the human-computer interaction behavior evaluation index, establish a multi-attribute comprehensive evaluation mathematical model, and evaluate the human-computer interaction behavior according to the weight coefficients of each evaluation index and the multi-attribute comprehensive evaluation mathematical model.
[0007] Preferably, the method further includes: S104. Based on the human-computer interaction process and typical operations, determine the test task profile for human-computer interaction behavior. S105. Test the human-computer interaction behavior based on the human-computer interaction behavior test task profile and the human-computer interaction behavior evaluation index.
[0008] Preferably, the human-computer interaction behavior evaluation indicators determined in S101 are based on the NASA-TLX scale method, taking into account the influencing factors of human-computer interaction behavior and the indicator determination principles, and are two-level evaluation indicators.
[0009] Preferably, the first-level evaluation indicators for human-computer interaction behavior determined in S101 include: mental workload, physical workload, time performance, operational performance, effort level, and confusion level.
[0010] Preferably, the weight coefficients of each evaluation indicator in S102 are determined by a weighting method based on integrated features, which combines subjective weighting with objective weighting to provide a method for determining the weight coefficients of each evaluation indicator.
[0011] Preferably, the evaluation mathematical model in S103 adopts a multi-dimensional feature hierarchical fusion evaluation method, that is, it integrates feature indicators related to the corresponding dimension through multivariate statistical analysis methods, and extracts the main common trends to achieve evaluation and analysis of human-computer interaction behavior in the corresponding dimension.
[0012] Preferably, the human-computer interaction behavior test task in S104 includes normal operating procedure flight tasks and emergency operating procedure flight tasks.
[0013] Preferably, the human-computer interaction behavior testing procedure in S105 includes testing methods, testing procedures, testing process design, and testing criteria.
[0014] The beneficial technical effects of this application are as follows: The advantages of this invention are that it proposes a comprehensive testing and evaluation method for human-computer interaction behavior based on multiple attributes; it establishes an evaluation index system for human-computer interaction behavior that combines subjective and objective data based on the NASA-TLX scale method; it presents a comprehensive evaluation method for human-computer interaction behavior based on hierarchical fusion of multi-dimensional features; and it forms a testing method for human-computer interaction behavior based on typical task profiles. This effectively solves the problems of traditional human-computer interaction behavior evaluation systems that are primarily static, function-oriented, and subjectively focused, resulting in incomplete evaluations and an inability to meticulously describe and evaluate the details of human-computer interaction behavior. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a flowchart illustrating the technical implementation of the present invention; Figure 2 This is a flowchart of the multi-attribute comprehensive evaluation of human-computer interaction behavior provided in Embodiment 1 of the present invention; Figure 3 This is a flowchart of the multi-attribute comprehensive test of human-computer interaction behavior provided in Embodiment 2 of the present invention. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] The features and illustrative embodiments of various aspects of the present invention will now be described in detail. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention. The invention is by no means limited to any specific setups and methods set forth below, but covers any improvements, substitutions, and modifications to structures, methods, and devices without departing from the spirit of the invention. Well-known structures and techniques are not shown in the drawings and the following description to avoid unnecessarily obscuring the invention.
[0019] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing and simplifying the invention, and should not be construed as limiting the invention. Furthermore, the use of ordinal numbers (e.g., "first and second," etc.) is for distinguishing objects and is not limited to this order, and should not be construed as indicating or implying relative importance.
[0020] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly, encompassing both direct connection and indirect connection via an intermediate medium. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.
[0021] It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other, and the various embodiments can be referenced and cited in each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0022] The following describes the embodiments and appendices. Figure 1 -Appendix Figure 3 The present invention will be described in further detail, but the embodiments of the present invention are not limited thereto.
[0023] This application provides a method for comprehensive testing and evaluation of multiple attributes of human-computer interaction behavior, the method comprising the following steps: Step 1: Study the influencing factors and indicator determination principles of human-computer interaction behavior, and provide evaluation indicators for human-computer interaction behavior.
[0024] Step 2: Study the multi-attribute characteristics of the human-computer interaction behavior evaluation index measurement values and determine the weight coefficients of each evaluation index.
[0025] Step 3: Study the multi-attribute characteristics of the human-computer interaction behavior evaluation index measurement values and establish a mathematical model for comprehensive evaluation of multi-attribute human-computer interaction behavior.
[0026] Step 4: Based on the human-computer interaction process and typical operations, determine the human-computer interaction behavior test task profile.
[0027] Step 5: Determine the human-computer interaction behavior test procedure based on the human-computer interaction behavior task profile and evaluation indicators.
[0028] Example 1 This application provides a multi-attribute comprehensive evaluation method for human-computer interaction behavior, the method comprising the following steps: S101. Based on the influencing factors and index construction principles of human-computer interaction behavior, provide evaluation indicators for human-computer interaction behavior.
[0029] The general principle for determining evaluation indicators for human-computer interaction behavior is to use as few indicators as possible for actual evaluation. However, in the initial establishment, it is necessary to ensure the completeness, rationality, effectiveness, and operability of the evaluation indicators as comprehensively as possible according to certain principles. In constructing evaluation indicators for human-computer interaction behavior, this invention, based on the NASA-TLX scale method, innovatively considers six aspects: mental workload, physical workload, time performance, operational performance, effort level, and confusion level. It comprehensively considers the influencing factors of human-computer interaction behavior and the principles for indicator determination, and provides a two-level evaluation indicator system. Specifically, it includes the following steps: a) Define the principles for constructing evaluation indicators for human-computer interaction behavior: 1) Scientific rigor and objectivity; 2) Comprehensiveness and integration; 3) Hierarchy and independence; 4) Operability.
[0030] b) Preliminary construction of human-computer interaction behavior evaluation indicators: Using human-machine-environment theory and system analysis, and comprehensively considering the influencing factors of human-computer interaction behavior, based on the NASA-TLX scale method, we have preliminarily constructed human-computer interaction behavior evaluation indicators from six aspects: mental workload, physical workload, time performance, operational performance, effort level and confusion level. c) Determination of human-computer interaction behavior evaluation indicators: Based on the Delphi method, opinions from the expert group were solicited and refined, and the final evaluation indicators were selected from the initially constructed human-computer interaction behavior evaluation indicators to establish an indicator system.
[0031] S102. Based on the multi-attribute characteristics of the measured values of the human-computer interaction behavior evaluation indicators, determine the weight coefficients of each evaluation indicator.
[0032] Determining the weighting coefficients of the indicators is a core issue in the evaluation of human-computer interaction behavior. Conventional methods for determining weighting coefficients can be divided into two main categories: weighting methods based on the "function-driven" principle and weighting methods based on the "difference-driven" principle. Specifically: a) The weighting method based on the "function-driven" principle essentially determines the weight coefficients of evaluation indicators based on their relative importance, generally through subjective means. Subjective weighting methods include "direct-push" subjective weighting and "reverse-push" subjective weighting. The former involves the subjective evaluator directly comparing the importance of each indicator to obtain the weight coefficients. The latter involves the evaluator first comparing and judging the merits of the evaluated objects, and then working backwards to obtain the weight coefficients based on the comparison information. b) The weighting method based on the "difference-driven" principle is essentially a measure of the degree of variation of each indicator in the overall indicator population and the degree of influence on other indicators. The original information for weighting should come directly from the objective environment, and the weight coefficient of the corresponding indicator can be determined according to the amount of information provided by each indicator.
[0033] Since the evaluation index system for human-computer interaction behavior has a hierarchical structure, the methods for determining the weight coefficients of different sub-objectives will vary depending on the needs. Therefore, this invention, based on a comprehensive consideration of the advantages and disadvantages of the two weighting methods mentioned above, adopts a comprehensive integrated feature weighting method, which combines subjective and objective weighting, to determine the weight coefficients of each evaluation index.
[0034] S103. Based on the multi-attribute characteristics of the measured values of the human-computer interaction behavior evaluation index, establish a multi-attribute comprehensive evaluation mathematical model.
[0035] Multi-attribute comprehensive evaluation involves using a mathematical model (or comprehensive evaluation function, aggregation model, aggregation unit) to "synthesize" multiple evaluation index values into a single overall comprehensive evaluation value. Taking into account the multi-attribute characteristics of human-computer interaction behavior evaluation index measurements, a nonlinear weighted synthesis method was used to establish a mathematical model for multi-attribute comprehensive evaluation of human-computer interaction behavior.
[0036] Example 2 This application provides a comprehensive testing method for multiple attributes of human-computer interaction behavior, the method comprising the following steps: S101. Based on the principles for determining the influencing factors and indicators of human-computer interaction behavior, evaluation indicators for human-computer interaction behavior are given; the specific construction process is shown in Example 1, S101.
[0037] S104. Based on the human-computer interaction process and typical operations, determine the test task profile for human-computer interaction behavior. Specifically, based on the requirements of human-computer interaction behavior testing, a human-computer interaction behavior testing task profile is constructed, primarily based on flight missions, including a normal operation procedure task profile and an emergency operation procedure task profile. Among these: a) The normal operating procedure mission profile includes the aircraft completing the aircraft start-up, taxiing, take-off, climb, level flight, landing, landing taxiing, and parking according to the normal operating procedure; b) The emergency operating procedure mission profile includes the aircraft completing landing, landing roll, and parking according to the emergency operating procedure.
[0038] S105. Test the human-computer interaction behavior based on the human-computer interaction behavior test task profile and the human-computer interaction behavior evaluation index.
[0039] The human-computer interaction behavior testing program is primarily based on the human-computer interaction behavior task profile and evaluation indicators to determine the human-computer interaction behavior testing plan design, including tester selection, test condition preparation, test method selection, test process design, and test criterion formulation. Specifically, it includes: a) Selection of Test Personnel: A sufficient number of personnel should participate in the human-computer interaction behavior evaluation, primarily composed of pilots. Ideally, pilots with different body dimensions and flight experience on different aircraft types, or those with considerable flight experience and piloting skills, should be selected. If female pilots will be among the users of the aircraft being evaluated, additional female personnel should be selected. Knowing the anthropometric dimensions of the evaluators is crucial for the evaluation; their anthropometric dimensions must be measured before the evaluation. Furthermore, factors such as percentile, designed eye position, and seat reference point should be carefully considered. b) Test preparation: On the one hand, when conducting human-computer interaction behavior assessments, evaluators should wear flight suits and be equipped with appropriate gear, similar to their state during flight, and consider conducting assessments while wearing winter or summer clothing and clothing of different colors (dark / light). On the other hand, it is necessary to prepare in advance the necessary human-computer interaction behavior measurement equipment, including eye trackers, electroencephalograms, motion capture systems, and physiological feedback recorders. c) Selection of testing methods: There are many methods for evaluating human-computer interaction behavior. Taking into account the characteristics of aviation equipment performance appraisal tests, a measurement method that combines subjective evaluation and objective measurement is selected, including subjective rating scales, performance measurement (which can be further divided into main task measurement and secondary task measurement), and psychophysiological measurement.
[0040] d) Test process design: including defining task types, preparing for testing, evaluating operations in different areas, collecting test data, conducting test evaluation, and providing evaluation results; e) Test Criterion Formulation: To ensure the validity of the evaluation results, clear test criteria are needed to facilitate the final evaluation results. The test criteria are formulated as follows: 1) Low workload: There are fewer and simpler tasks, and the crew has ample time to handle various tasks. The work pace is slow and there is almost no pressure. 2) Moderate workload: The workload is moderate, with certain time constraints, but the crew can complete the work in an orderly manner and needs to maintain normal concentration; 3) High workload: The tasks are complex and intensive, and the time is tight. The crew members need to concentrate highly, which may lead to a high level of work pressure. 4) Overload: The workload far exceeds the normal capacity, the time is seriously insufficient, the crew members are under great pressure, which may affect work efficiency and safety.
[0041] This invention discloses a method for comprehensive testing and evaluation of human-computer interaction behavior across multiple attributes. The method includes: determining human-computer interaction behavior evaluation indicators based on influencing factors and indicator determination principles; determining the weight coefficients of each evaluation indicator based on the multi-attribute characteristics of the measured values of the human-computer interaction behavior evaluation indicators, and establishing a multi-attribute comprehensive evaluation mathematical model; determining the human-computer interaction behavior test task profile based on the human-computer interaction process and typical operations; and determining the human-computer interaction behavior test procedure based on the human-computer interaction behavior task profile and evaluation indicators.
[0042] The embodiments of the present invention can form a comprehensive testing and evaluation technology for multiple attributes of human-computer interaction behavior, which can be used for testing and evaluating human-computer interaction behavior.
[0043] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.
Claims
1. A method for comprehensive testing and evaluation of multi-attribute human-computer interaction behavior, characterized in that, The method includes: S101. Based on the principles for determining the influencing factors and indicators of human-computer interaction behavior, provide evaluation indicators for human-computer interaction behavior. S102. Based on the multi-attribute characteristics of the measured values of the human-computer interaction behavior evaluation indicators, determine the weight coefficients of each evaluation indicator; S103. Based on the multi-attribute characteristics of the measured values of the human-computer interaction behavior evaluation index, establish a multi-attribute comprehensive evaluation mathematical model, and evaluate the human-computer interaction behavior according to the weight coefficients of each evaluation index and the multi-attribute comprehensive evaluation mathematical model.
2. The method according to claim 1, characterized in that, The method further includes: S104. Based on the human-computer interaction process and typical operations, determine the test task profile for human-computer interaction behavior. S105. Test the human-computer interaction behavior based on the human-computer interaction behavior test task profile and the human-computer interaction behavior evaluation index.
3. The method according to claim 2, characterized in that, The human-computer interaction behavior evaluation indicators determined in S101 are based on the NASA-TLX scale method, taking into account the influencing factors of human-computer interaction behavior and the principle of indicator determination, and are two-level evaluation indicators.
4. The method according to claim 3, characterized in that, The first-level evaluation indicators for human-computer interaction behavior defined in S101 include: mental workload, physical workload, time performance, operational performance, effort level, and confusion level.
5. The method according to claim 4, characterized in that, In S102, the weight coefficients of each evaluation indicator are determined using a comprehensive integrated feature-based weighting method, which combines subjective and objective weighting to provide a method for determining the weight coefficients of each evaluation indicator.
6. The method according to claim 5, characterized in that, The evaluation mathematical model in S103 adopts a multi-dimensional feature hierarchical fusion evaluation method, that is, it integrates feature indicators related to the corresponding dimensions through multivariate statistical analysis methods, extracts the main common trends, and realizes the evaluation and analysis of human-computer interaction behavior in the corresponding dimensions.
7. The method according to claim 6, characterized in that, The human-computer interaction behavior test tasks in S104 include normal operating procedure flight tasks and emergency operating procedure flight tasks.
8. The method according to claim 7, characterized in that, The human-computer interaction behavior test program in S105 includes test methods, test procedures, test process design, and test criteria.