Interface interaction cognitive load evaluation method based on pilot behavior data

Abstract

The invention relates to the technical field of pilot working state evaluation, and discloses an interface interaction cognitive load evaluation method based on pilot behavior data, which comprises the following steps: real-time data acquisition; discrete information processing; evaluating weights of perception type and mental movement type factors; and deducing and evaluating a cognitive factor weight and outputting a cognitive load value. The technical problems that the degree of association between the evaluated cognitive load and the interface is not high, and optimization design of an aviation interface cannot be effectively guided are solved, and by collecting observable visual, auditory and mental movement data, evaluation of the load level of an unobservable cognitive factor is achieved. And the accuracy of pilot real-time cognitive load quantitative evaluation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of pilot work status evaluation, and in particular to an interface interaction cognitive load evaluation method based on pilot behavior data. Background Art

[0002] The aviation interface is an important bridge for the interaction between pilots and flight systems. The design quality of the aviation interface is directly related to the pilot's cognitive load, that is, the psychological pressure of the pilot when completing tasks under specific environmental and operating conditions. Too high or too low cognitive load will affect the pilot's operational performance and increase the probability of accidents. Only by maintaining cognitive load within an appropriate level can the pilot perform the best task performance.

[0003] In order to better understand and evaluate the pilot's cognitive load in specific mission situations and provide a scientific basis for building a reasonable cockpit human-machine interface design, a pilot cognitive load assessment method with real-time collection and evaluation capabilities is urgently needed.

[0004] At present, the commonly used cognitive load assessment methods include behavioral performance evaluation method, physiological index evaluation method, subjective scale evaluation method and evaluation method based on cognitive model. Among them, the real-time performance of behavioral performance evaluation method is poor. The protective compensatory behavior stimulated by humans when facing excessive task pressure will maintain their task performance from significantly deteriorating in a short period of time; the cognitive load degree reflected by the physiological index evaluation method is poorly interpretable. The cognitive load evaluation model based on physiological parameters is mostly obtained by training in the laboratory using ideal tasks. It is difficult to trace the cause of overload in application; the subjective scale evaluation method has poor objectivity and is mostly used in qualitative evaluation scenarios; the evaluation method based on cognitive model has the advantages of high reliability and high efficiency, and can directly reflect the load situation of users in the interface environment or real scene, and is widely used to study human task performance issues.

[0005] The idea of ​​cognitive load evaluation method based on VACP (Visual, Auditory, Cognitive, Psychomotor) model is to establish a specific task network in the four major channel resources of vision, hearing, cognition and psychomotor, which can provide quantitative cognitive load indicators about when and how often users use which resources. Its applicability has been verified by a large number of studies. However, the existing related methods are mostly used for post-load evaluation, and the feasibility of such methods in real-time application is not high; on the other hand, the existing methods are not highly correlated with interface elements, and the correlation between the assessed cognitive load and the interface is not high, which cannot be effectively used to guide the optimization design of aviation interface. Summary of the invention

[0006] The purpose of the present invention is to establish a network coupling relationship of cognitive load factors based on the VACP cognitive model on the basis of combining the real-time performance of physiological indicators and behavioral data, and propose an interface interaction cognitive load evaluation method based on pilot behavioral data.

[0007] The technical solution of the present invention is achieved in this way:

[0008] A method for evaluating cognitive load of interface interaction based on pilot behavior data comprises the following steps:

[0009] Real-time data collection: with the help of eye tracker, rudder and cockpit master control system, real-time data including the pilot's gaze screen coordinates, interface element information, audio information and rudder control information in aviation interface tasks can be collected and recorded in real time;

[0010] Discrete information processing, combined with the master control system script to process real-time data, extract discrete information required for evaluation, including two types of information: feature and time series; feature information includes the gaze interest area obtained from the gaze coordinates, interface element type, auditory signal type and rudder operation signal type; time series information is the alternating change time node information corresponding to the feature information;

[0011] The weight evaluation of perceptual and psychomotor factors is to classify and evaluate the obtained discrete information according to the determined factor descriptor definition standard and timing judgment standard, combined with the master control system script, and define the factor weight and corresponding start time for the perceptual and psychomotor discrete data;

[0012] The cognitive factor weight inference assessment summarizes the evaluation weights of the perceptual and psychomotor factors, and infers the cognitive evaluation load level and corresponding weights based on the coupling relationship network of cognitive load factors;

[0013] Output of cognitive load value, annotate and visualize the above information according to the time series relationship, accumulate cognitive load according to the time series, and sum it up every time the perception, cognition and psychomotor processes are carried out to achieve real-time output of the pilot's cognitive load quantification curve.

[0014] As a further solution of the present invention: the real-time data includes:

[0015] Eye movement data, using high-precision eye tracker equipment to capture and record in real time the coordinates of the pilot's gaze point when operating the aviation interface;

[0016] Interface data, obtain the interface UI element characteristics and interface element mapping information through the central control system;

[0017] Audio data, collecting audio data in the cockpit;

[0018] The stick and rudder data uses the flight stick and rudder and script data acquisition system to record and analyze the pilot's control behavior in aviation interface tasks in real time.

[0019] As a further solution of the present invention: the real-time data discrete processing includes:

[0020] Eye movement data processing, the coordinates of the gaze point can accurately reflect the pilot's visual focus point, and can be analyzed to obtain the gaze interest area at each moment, as well as the switching time node of the interest area;

[0021] Interface data processing, mapping information can be retrieved in real time, and can further feedback the associated interface data information based on the visual interest area, auditory signal type and rudder operation information type obtained by evaluation;

[0022] Audio data processing: obtaining the type of warning signal the pilot is exposed to through audio data analysis;

[0023] Stick and rudder data processing, stick and rudder data includes key information such as the pilot's operation method of the joystick and pedals, control type and frequency.

[0024] As a further solution of the present invention: the weight evaluation of the perceptual and psychomotor factors includes:

[0025] The visual perception assessment factor consists of 7 auditory descriptors to describe the visual behavior and the degree of mobilization of visual information processing resources;

[0026] The auditory perception assessment factor consists of 7 auditory descriptors, which are used to describe the auditory behavior and the degree of mobilization of auditory information processing resources;

[0027] The psychomotor assessment factor is mainly composed of 7 descriptors, which are used to describe the degree of psychological load caused by the behavior of effectors such as hands and feet.

[0028] As a further solution of the present invention: the visual perception evaluation factor includes:

[0029] The descriptor “V1-Visual Start” is defined as the visual action that starts the task in the cockpit interaction environment from scratch, with a weight of 1.0;

[0030] The descriptor “V2-visual discrimination”, defined as the visual action of discerning whether the signal symbol composed of the basic visual stimulus appears different, has a weight of 3.7;

[0031] The descriptor “V3-Visual Inspection” is defined as the visual action that determines whether the state signal composed of semantically-based basic visual stimuli is the target state, with a weight of 4.0;

[0032] Descriptor “V4-Visual localization / alignment”, defined as the visual action to determine whether the state signal composed of semantically based basic visual stimuli is the target state, with a weight of 5.0;

[0033] Descriptor “V5-Visual pursuit / tracking”, defined as the visual action of continuously observing a dynamic target undergoing continuous displacement, with a weight of 5.4;

[0034] Descriptor “V6-Visual Reading”, defined as the visual behavior of reading text-like elements, with a weight of 5.9;

[0035] The descriptor “V7-Visual scanning / searching / monitoring” is defined as the visual action of continuous observation of multiple targets or multiple types of information under the region of interest with high information density, with a weight of 7.0;

[0036] The auditory perception evaluation factors include:

[0037] The descriptor “A1-Detect / record sound” is defined as identifying whether sound exists, that is, judging whether sound occurs in the environment, with a weight of 1.0;

[0038] The descriptor “A2-sound direction localization”, defined as the approximate directional sound position in space, has a weight of 2.0;

[0039] The descriptor “A3-sound orientation localization”, defined as selecting a sound stimulus in space and orienting the sound position, has a weight of 4.2;

[0040] The descriptor “A4-Verify auditory feedback” is defined as detecting whether the expected auditory stimulus feedback occurs after the operation is performed, with a weight of 4.3;

[0041] The descriptor “A5-Interpretation of semantic content”, defined as the semantic interpretation during communication, has a weight of 4.9;

[0042] The descriptor “A6-discrimination of sound features” is defined as the ability to distinguish the differences between different sound stimuli in the same space, with a weight of 6.6;

[0043] The descriptor “A7-interpretation of sound pattern” is defined as identifying sound stimuli with specific semantics and judging the corresponding state, with a weight of 7.0;

[0044] The psychomotor assessment factors include:

[0045] The descriptor “P1-speech”, defined as communication through spoken language, has a weight of 1.0;

[0046] The descriptor “P2-discrete actuation”, defined as operating discrete control devices such as buttons, switches, or triggers, has a weight of 2.2;

[0047] The descriptor “P3-Continuous Adjustment”, defined as continuous adjustment of the control device through smooth, continuous actions, has a weight of 2.6;

[0048] The descriptor “P4-manipulativeness,” defined as the use of fine motor movements of the hands for precise control of tools or equipment, had a weight of 4.6;

[0049] The descriptor "P5-Separate Adjustment", defined as separate adjustment of the device or instrument by rotation, vertical sliding, or lever adjustment, has a weight of 5.8;

[0050] The descriptor “P6-Symbolic production”, defined as the production and recording of symbols or words through writing tools, has a weight of 6.5;

[0051] The descriptor "P7-Serial discrete operation" is defined as a series of continuous discrete key operations through the keyboard to input data or instructions, with a weight of 7.0.

[0052] As a further solution of the present invention: the cognitive evaluation factor is composed of 7 descriptors, which are used to describe the degree of invocation of cognitive devices in the process of brain information processing and understanding;

[0053] The descriptor “C1-automatic”, defined as response behavior that is unconscious or simply stimulus-associated and does not require operant behavior, has a weight of 1.0;

[0054] Descriptor “C2-alternative choice”, defined as identifying the current state and determining whether the next action is needed, with a weight of 1.2;

[0055] Descriptor “C3-Sign / Signal Recognition”, defined as identifying the state or signal icon and determining the corresponding operation, with a weight of 3.7;

[0056] Descriptor “C4-Evaluation / Judgment (Single Consideration)”, defined as judging the current state and next action based on a single basis, with a weight of 4.6;

[0057] The descriptor “C5-Encoding / Decoding, Memory” is defined as involving the interpretation of the semantics of information in a fixed format, with a weight of 5.3;

[0058] Descriptor “C6-Evaluation / Judgment (Comprehensive Consideration)”, defined as judging the current state and next action based on multiple criteria, with a weight of 6.8;

[0059] The descriptor “C7-estimation, calculation, conversion”, defined as involving cognitive behaviors such as numerical calculation and percentage judgment, has a weight of 7.0.

[0060] As a further solution of the present invention: the coupling relationship between the perception, psychomotor factors and cognitive assessment factors includes:

[0061] V1 behavior usually refers to the process from nothing to something in an interactive environment. When there is no clear instruction on the next operation, the pilot's cognition of the scene is a simple stimulus response, which is coupled with C1.

[0062] For V2 behavior, it is necessary to distinguish whether there are differences in the signal symbols composed of the basic visual stimuli, and then identify the signal symbols and determine the corresponding operations, which has a coupling relationship with C3;

[0063] For V4 behavior, visual adjustment needs to be made towards a static target at a relative distance. In interactive scenarios, the more common static targets are interactive buttons or sliders. For interactive buttons, the pilot needs to identify the current state of the button and click it, which is usually coupled with C2. For sliders, after sliding to the expected position, the pilot needs to determine whether the sliding degree is in compliance, which is coupled with C7.

[0064] For V5 behavior, it is required to continuously pay attention to the dynamic target of continuous displacement; under general mission requirements, the pilot may be required to make a state judgment, or even combine other information to make an overall state judgment of the working area. Therefore, it has a coupling relationship with C4 and C6 respectively, and the coupling situation depends on the mission requirements and operational response;

[0065] For V7 behavior, it corresponds to multi-target information, and the scanning range is usually a large working area. The operational response is not completely clear, so it has a coupling relationship with C3 and C6 respectively;

[0066] For V6 behavior, the corresponding interactive elements are mainly composed of pictures and texts, so they are coupled with C4, C5 and C7 respectively.

[0067] As a further solution of the present invention: the coupling relationship between the auditory perception and cognitive evaluation factors includes:

[0068] For A1 behavior, it usually means the process of auditory tasks from scratch in an interactive environment, which can be used as a baseline for auditory-related mental load; therefore, it is generally coupled with C1;

[0069] For A2 behavior, the pilot is required to perform directional sound localization in space based on auditory clues. The pilot may not need to react, but only need to understand the source of the sound, or may need to locate the sound to make a one-step reaction behavior. Therefore, it has a coupling relationship with C1 and C2 respectively.

[0070] For A3 behavior, the pilot is required to locate the sound of orientation and direction in space based on auditory cues, which means that this type of sound cues requires the pilot to identify the current state and determine whether the next operation is required. Therefore, it has a coupling relationship with C3 and C4 respectively;

[0071] For A4 behavior, the pilot is required to verify the feedback of the auditory cues; if the verification is successful, it has a coupling relationship with C1; if the verification is unsuccessful, it has a coupling relationship with C4 and C6 respectively according to the difficulty set by the task operation steps;

[0072] For A5 behavior, it is required to interpret various types of information semantics based on the content referred to by the auditory clues, that is, it has a coupling relationship with C5 and C7 respectively;

[0073] For A6 behavior, the pilot is required to distinguish the differences between different sound stimuli, which is similar to V2 behavior in the perceptual visual layer in terms of task type, and therefore has a coupling relationship with C2;

[0074] For A7 behavior, pilots are required not only to distinguish voice features and specific semantic behaviors, but also to judge the state mode corresponding to the sound clues; that is, to be able to interpret the semantics of information, make judgments based on the basis, and may involve data calculation or state judgment based on data, etc., and therefore have a coupling relationship with C5 and C6 respectively.

[0075] Beneficial effects of the present invention:

[0076] 1. The present invention builds a cognitive load factor coupling relationship network based on aviation tasks. With the help of this relationship network, the evaluation weights of unobservable cognitive factors can be inferred based on observable perception and operation factors.

[0077] 2. Apply the method provided by the present invention to build an evaluation system, input observable pilot eye movement behavior data, stick and rudder control data and interface information data, infer the unobservable cognitive channel factor weights, and then count and output the pilot's real-time cognitive load evaluation value.

[0078] 3. Based on the VACP model, the method of the present invention combines the characteristics of the flight cockpit mission to establish a cognitive load factor coupling network. By collecting observable visual, auditory and psychomotor data, the evaluation of the load level of unobservable cognitive factors is realized, which improves the accuracy of the quantitative evaluation of the pilot's real-time cognitive load and provides an effective technical path for better understanding and evaluating the pilot's cognitive load in specific mission scenarios.

[0079] 4. Operability. The method of the present invention has significant advantages in operability. With the help of modern eye trackers, rudders and cockpit master control systems, this method can realize real-time collection and processing of key data of pilots in aviation interface tasks, thereby evaluating the pilot's cognitive load. These devices have not only been widely used in the aviation field, but also are relatively simple to operate and can be effectively integrated into existing pilot training and task execution processes. By using the master control system script to perform discrete information processing on real-time data, the key perceptual and psychomotor factor weights can be quickly and accurately extracted, providing a scientific basis for subsequent cognitive factor evaluation. Therefore, this method is not only mature in theory, but also has reliable operability in practice.

[0080] 5. Non-interference. The design of the method described in the present invention takes into account the non-interference during the data collection process to ensure that it does not affect the normal operation and reaction of the pilot. The use of the eye tracker and the rudder control system will not interfere with the pilot's line of sight and control operation, ensuring the safety and accuracy of the flight mission. Through the fine regulation of the master control system, the real-time data collection process is almost imperceptible to the pilot's operation, effectively avoiding the additional cognitive load caused by the use of equipment. This design not only ensures the accuracy and practicality of the data, but also minimizes the impact on the pilot, allowing him to focus on mission execution.

[0081] 6. Quasi-real-time performance. The method of the present invention has good quasi-real-time performance. Through real-time data collection of the eye tracker, rudder and cockpit master control system, combined with the real-time processing and analysis of the master control system script, the method can quickly and accurately assess the weights of the perceptual and psychomotor factors, and infer the evaluation load level and corresponding weights of the cognitive factors. In the real-time data collection stage, the system extracts features and timing information based on real-time data such as the pilot's gaze screen coordinates, interface element information, audio information, and rudder control information, and classifies and evaluates them in combination with pre-defined factor descriptors and timing judgment criteria. Subsequently, these evaluation results are summarized and analyzed, and the pilot's cognitive load quantitative evaluation value is inferred based on the coupling relationship network of cognitive load factors. This process not only ensures the real-time and accuracy of the data, but also can output the pilot's cognitive load status in a short time, providing an important real-time reference for operational adjustment and flight mission management.

[0082] 7. The correlation of interface elements and direct guidance on design. The method described in the present invention directly focuses on the correlation between interface elements and pilot cognitive load, providing a direct guiding basis for interface design. By analyzing and evaluating the pilot's eye movement behavior, stick and rudder operation, and the use of interface elements in aviation interface tasks, the degree of influence of different interface elements on cognitive load can be revealed in detail. These data can not only quantify the impact of different interface design schemes on cognitive load, but also help designers optimize interface layout and element design, and improve pilots' operational efficiency and safety in specific mission scenarios. Therefore, this method is not only a theoretical cognitive load assessment tool, but also a practical engineering method that directly guides the optimization of aviation interface design.

[0083] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a schematic diagram of the process of the method of the present invention;

[0085] Figure 2 It is a schematic diagram of the basic idea framework of inferring and evaluating cognitive evaluation factors in the method of the present invention;

[0086] Figure 3 An example diagram of a sequence of cognitive load values ​​output by the method of the present invention;

[0087] Figure 4 It is the cognitive load factor coupling relationship network in the method of the present invention. DETAILED DESCRIPTION

[0088] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0089] See attached Figure 1-4 The present invention provides an interface interaction cognitive load evaluation method based on pilot behavior data, which specifically comprises the following steps:

[0090] (1) Real-time data acquisition and discrete information processing

[0091] ① Eye movement data

[0092] The present invention uses high-precision eye tracker equipment to capture and record the pilot's gaze point coordinates in real time when operating the aviation interface. These gaze point coordinates can accurately reflect the pilot's visual focus point, and can be analyzed to obtain the gaze area of ​​interest (AOI) at each moment, as well as the switching time node of the interest area.

[0093] ②Interface data

[0094] The method of the present invention can obtain the interface UI element characteristics and interface element mapping information through the central control system. This information can be retrieved in real time, and can further feedback the interface data information associated with it based on the visual interest area, auditory signal type and rudder operation information type obtained by evaluation.

[0095] ③Audio data

[0096] The method of the present invention acquires the type of warning signal to which the pilot is exposed by collecting audio data in the cockpit and analyzing the data.

[0097] ④ Rod and rudder data

[0098] The present invention utilizes the flight stick and rudder and script data acquisition system to record and analyze the pilot's control behavior in the aviation interface task in real time. These data include key information such as the pilot's operation mode of the joystick and pedals, control type and frequency.

[0099] (2) Weight assessment of perceptual and psychomotor factors

[0100] According to the characteristics of the aviation interface mission scenario, the method described in the present invention defines the descriptor standards of the four types of evaluation factors of the VACP model in more detail. By combining the evaluation annotations to classify and evaluate the obtained discrete information, the weight evaluation of the perceptual and psychomotor factors can be obtained. The specific contents are as follows:

[0101] ① Visual perception evaluation factors

[0102] The visual perception assessment factor is mainly composed of 7 auditory descriptors, which are used to describe the visual behavior and the degree of mobilization of visual information processing resources.

[0103] For the descriptor “visual start”, it is defined as the visual action to start the task in the cockpit interaction environment from scratch.

[0104] The descriptor “visual discrimination” is defined as the visual action of distinguishing whether the signal symbols composed of basic visual stimuli are different. For example, a pilot stares at the “altitude warning” symbol to understand and distinguish it, so as to determine whether to take corresponding flight adjustment actions.

[0105] For the descriptor “visual inspection”, it is defined as the visual action of judging whether the state signal composed of the semantically-laden basic visual stimulus is the target state.

[0106] For the descriptor “visual positioning / alignment”, it is defined as the visual action of judging whether the state signal composed of semantically based basic visual stimuli is the target state. For example, the pilot observes the speedometer reading on the instrument panel, judges whether the current speed is within the predetermined range, and makes necessary adjustments.

[0107] For the descriptor “visual pursuit / tracking”, it is defined as the visual action of continuously observing a dynamic target that is continuously moving. For example, the pilot continuously observes and tracks the position and dynamics of the target displayed on the radar interface in order to take appropriate tactical responses.

[0108] For the descriptor “visual reading”, it is defined as the visual behavior of reading textual elements, for example, the action of a pilot reading the flight plan information in the cockpit to obtain key navigation instructions.

[0109] For the descriptor “visual scanning / search / surveillance”, it is defined as the visual action of continuously observing multiple targets or multiple types of information in an area of ​​interest with high information density. For example, a pilot continuously scans and monitors the airspace on the radar screen to detect and respond to possible air targets or threats in a timely manner.

[0110] Table 1 Descriptors and definitions of visual perception category evaluation factors

[0111]

[0112] The temporal judgment criteria of visual perception factor descriptors are as follows:

[0113] Table 2 Timing criteria for visual perception evaluation factors

[0114]

[0115] According to the input gaze screen coordinates (Gaze_X and Gaze_Y), the gaze AOI is analyzed and combined with the corresponding UI element characteristics in the interactive interface to determine the evaluation factor type Vi at that moment and the corresponding start time t sVi and t eVi .

[0116] ② Auditory perception evaluation factors

[0117] The auditory perception assessment factors are mainly composed of 7 auditory descriptors, which are used to describe the auditory behavior and the degree of mobilization of auditory information processing resources.

[0118] For the descriptor "detect / record sound", it is defined as recognizing the presence of sound, that is, judging whether there is sound in the environment. For example, the action of a pilot in the cockpit continuously paying attention to possible alarm sounds or other sounds in the environment during flight.

[0119] For the descriptor “sound direction localization”, it is defined as the approximate directional location of a sound in space, such as the action of a pilot identifying the direction of the source of a warning sound.

[0120] For the descriptor “sound direction localization”, it is defined as the action of selecting a sound stimulus in space and orienting the sound location, such as a pilot hearing an unusual sound behind the engine during flight and quickly confirming the source of the problem through sound direction localization.

[0121] The descriptor "verify auditory feedback" is defined as detecting whether the expected auditory stimulus feedback occurs after the operation is performed. For example, after adjusting the communication equipment in the cockpit, the pilot listens for a confirmation reply from the ground dispatcher to verify whether the communication equipment is working properly.

[0122] For the descriptor “interpret semantic content”, it is defined as semantic interpretation during communication. For example, when a pilot communicates with an air traffic controller, he or she understands and interprets their instructions and prompts to ensure the correct execution of the flight mission and the flight plan.

[0123] For the descriptor “distinguishing sound features”, it is defined as distinguishing the difference between different sound stimuli in the same space. For example, a pilot can distinguish the sound from the radar warning system and the communication system during flight, and distinguish these two different alarms or prompts through sound features.

[0124] For the descriptor "interpretation of sound pattern", it is defined as identifying sound stimuli with specific semantics and judging the corresponding status. For example, the ground controller sends a sound pattern of "route adjustment" in the communication, and the pilot interprets the sound pattern and understands the instruction to adjust the current flight route.

[0125] Table 3 Descriptors and definitions of auditory perception evaluation factors

[0126]

[0127] Descriptor start time t of auditory perception class factor descriptor sAi All are "alarm sound appearance + auditory reaction delay", end time t eAi All are when the alarm sound ends.

[0128] ③Psychomotor assessment factors

[0129] The psychomotor assessment factor is mainly composed of 7 descriptors, which are used to describe the degree of psychological load caused by the actions of effectors such as hands and feet. In this section, the psychomotor descriptors are defined and described, and the final descriptor scale of the psychomotor assessment factor is summarized in Table 4.

[0130] The descriptor "speech" is defined as communication through spoken words. For example, a pilot may speak to an air traffic controller to convey a flight plan, request routing instructions, or report flight status.

[0131] For the descriptor "discrete actuation", the definition is to operate a discrete control device such as a button, switch, or trigger. For example, a pilot uses a discrete actuation device in the cockpit, such as a landing gear switch or a missile launch button, to perform corresponding flight operations or weapon release actions during flight.

[0132] The descriptor “continuous adjustment” is defined as continuous adjustment of controls through smooth, continuous movements, for example, a pilot continuously fine-tuning the aircraft's attitude or flight parameters during flight by smoothly moving a flight stick or pedals to ensure smooth and precise flight.

[0133] For the descriptor "manipulability", it is defined as the use of fine hand movements to perform precise control of tools or equipment. For example, a pilot operates the flight control stick in the cockpit and uses fine hand movements to adjust the aircraft's attitude, thrust, and speed to ensure smooth and safe flight.

[0134] The descriptor "discrete adjustment" is defined as the separate adjustment of a device or instrument by rotation, vertical sliding, or lever adjustment. For example, when a pilot operates the altitude adjustment knob in the flight cockpit, he adjusts the set vertical altitude of the aircraft by turning the knob.

[0135] The descriptor "symbolic production" is defined as the production and recording of symbols or words by writing tools. For example, a pilot uses an aviation map and pen in the flight cockpit to mark and record the aircraft's current position, flight route, and important navigation points for navigation and flight path planning and execution.

[0136] The descriptor "serial discrete operation" is defined as a series of consecutive discrete key presses through the keyboard to enter data or instructions. For example, a pilot enters the coordinates of a navigation destination or executes specific instructions for a flight plan by pressing a series of discrete keys in flight to ensure that the aircraft navigates according to the predetermined route and target.

[0137] Table 4 Descriptors and definitions of psychomotor assessment factors

[0138]

[0139]

[0140] The temporal criteria for psychomotor factor descriptors are as follows:

[0141] Table 5 Timing criteria for psychomotor assessment factors

[0142]

[0143] According to the input stick and rudder control signal and the superposition of the pilot's reaction process delay, combined with the corresponding UI element characteristics in the interactive interface, the evaluation factor type Pi at that moment and the corresponding starting time t can be determined. sPi and t ePi .

[0144] (3) Evaluation of cognitive factor weights

[0145] The theoretical basis for the inference and evaluation of cognitive factor weights is as follows: Figure 2 As shown in the figure, it is the evaluation weight of the perceptual and psychomotor factors, based on Figure 4 The coupling relationship network of cognitive load factors shown in the figure is inferred. The definition of cognitive factors, coupling relationship and timing judgment criteria are as follows:

[0146] ① Descriptors and definitions of cognitive assessment factors

[0147] The cognitive assessment factors are mainly composed of 7 descriptors, which are used to describe the degree of activation of cognitive organs in the process of brain information processing and understanding.

[0148] The descriptor "automatic" is defined as unconscious or simple stimulus-related response behavior, and no operational behavior is required. For example, when the pilot sees the green indicator light in the cockpit, he automatically understands that the flight system is in normal working condition without additional operation or cognitive processing.

[0149] For the descriptor “alternative choice”, it is defined as identifying the current state and judging whether the next action is needed. For example, when a pilot is performing a tactical maneuver, he needs to make a quick choice whether to turn left to avoid enemy fire or turn right to maintain the flight direction.

[0150] The descriptor "sign / signal recognition" is defined as recognizing a state or signal icon and determining the corresponding operation. For example, when a pilot recognizes that the "throttle lever unlocked" indicator light is on in the cockpit during flight, he immediately determines that he needs to press the throttle lever to unlock the state in order to take off or land.

[0151] For the descriptor “Evaluation / Judgment (single consideration)”, it is defined as judging the current state and the next action based on a single basis. For example, the pilot notices that the temperature of the aircraft engine is close to the maximum limit during flight. Based on this single basis, he decides to reduce the throttle to reduce the engine load to ensure safe operation of the aircraft.

[0152] For the descriptor "encoding / decoding, memory", it is defined as involving the interpretation of the semantics of information in a fixed format. For example, when a pilot receives a flight instruction from air traffic control, he needs to decode the semantics of the instruction according to the standard aviation communication format (such as ICAO standard phrases), remember and understand the key information in the instruction to ensure the correct execution of the flight plan.

[0153] For the descriptor "Evaluation / Judgment (Comprehensive Consideration)", it is defined as judging the current state and next action based on multiple bases. For example, the pilot receives data from multiple navigation systems during flight, including GPS, inertial navigation system, and ground radar information. By comprehensively considering this data, the pilot evaluates the current position of the aircraft and the flight path, and then decides whether the heading or altitude needs to be adjusted to maintain the predetermined flight plan.

[0154] For the descriptor “estimate, calculate, convert”, it is defined as cognitive behaviors involving numerical calculations and percentage judgments. For example, during flight, a pilot needs to estimate the remaining fuel required for the aircraft to reach the destination, calculate the current fuel consumption rate, and convert it into the estimated flight time and distance.

[0155] Table 6 Descriptors and definitions of cognitive assessment factors

[0156]

[0157] ② Coupling relationship between perceptual, psychomotor and cognitive evaluation factors

[0158] The various perceptual stimulus characteristic factors in the perception evaluation factors will affect the type of cognitive evaluation factors. For example, for V3 behavior, the pilot needs to judge the current state and the next corresponding operation based on the semantics of the visual stimulus. The considerations in this process are single, so it has a coupling relationship with C4. Similarly, the coupling relationship between the visual perception and cognitive evaluation factors is analyzed as follows.

[0159] V1 behavior usually refers to the process of visual tasks from scratch in an interactive environment. When there is no clear instruction on the next operation, the pilot's cognition of the scene is manifested as a simple stimulus response, which is coupled with C1.

[0160] For V2 behavior, it is necessary to distinguish whether there are differences in the signal symbols composed of the basic visual stimuli, and then identify the signal symbols and determine the corresponding operations, which has a coupling relationship with C3.

[0161] For V4 behavior, visual adjustment is required towards a static target at a relative distance. In interactive scenarios, common static targets are interactive buttons or sliders. For interactive buttons, pilots need to identify the current state of the button and click it, which is usually coupled with C2. For sliders, after sliding to the expected position, pilots need to determine whether the degree of sliding is in compliance, which is coupled with C7.

[0162] For V5 behavior, it is required to continuously focus on the dynamic target of continuous displacement. Under general mission requirements, the pilot may be required to make state judgments, or even combine other information to make overall state judgments in the work area. Therefore, they are coupled with C4 and C6 respectively, and the coupling situation depends on the mission requirements and operational response.

[0163] Similarly, for V7 behavior, it itself corresponds to multi-target information, the scanning range is usually a larger working area, and the operational response made cannot be completely clear, so it has a coupling relationship with C3 and C6 respectively.

[0164] For V6 behavior, the corresponding interactive elements are mainly composed of pictures and texts, so they are coupled with C4, C5 and C7 respectively.

[0165] The coupling relationship between visual perception and cognitive evaluation factors is summarized in the following table.

[0166] Table 7 Coupling relationship between visual perception and cognitive evaluation factors

[0167]

[0168]

[0169] The coupling relationship analysis between auditory perception and cognitive evaluation factors is as follows.

[0170] A1 behavior usually means the process of performing an auditory task from scratch in an interactive environment, which can be used as a baseline for auditory-related mental load. Therefore, it is generally coupled with C1.

[0171] For A2 behavior, the pilot is required to perform directional sound localization in space based on auditory clues. The pilot may not need to react, but only need to understand the source of the sound, or may need to locate the sound to make a one-step reaction behavior. Therefore, they are coupled with C1 and C2 respectively.

[0172] For A3 behavior, the pilot is required to locate the sound of direction and orientation in space based on auditory cues, which means that this type of sound clues requires the pilot to identify the current state and determine whether the next operation is required. Therefore, it has a coupling relationship with C3 and C4 respectively.

[0173] For A4 behavior, the pilot is required to verify the feedback of the auditory cues. If the verification is successful, it will be coupled with C1. If the verification is unsuccessful, it will be coupled with C4 and C6 respectively according to the difficulty set by the task operation steps.

[0174] For A5 behavior, various types of information semantics are required to be interpreted according to the content referred to by the auditory clues, that is, it has a coupling relationship with C5 and C7 respectively.

[0175] For A6 behavior, pilots are required to distinguish the differences between different sound stimuli, which is similar to V2 behavior in the perceptual visual layer in terms of task type, and therefore has a coupling relationship with C2.

[0176] For A7 behavior, pilots are required to identify speech features and specific semantic behaviors, as well as determine the state mode corresponding to the sound clues. That is, they are able to interpret the semantics of information, make judgments based on the evidence, and may involve data calculation or state judgment based on data, so they are coupled with C5 and C6 respectively.

[0177] The coupling relationship between auditory perception and cognitive evaluation factors is summarized in the following table.

[0178] Table 8 Coupling relationship between visual perception and cognitive evaluation factors

[0179]

[0180]

[0181] The psychomotor evaluation factor type can reflect the use of cognitive evaluation factors through behavior. For example, when P1 behavior is performed, C1 can be inferred, so it has a coupling relationship with C1. Similarly, the coupling relationship between the 7 psychomotor evaluation factors and cognitive evaluation factors is summarized in Table 7.

[0182] Table 9 Coupling relationship between cognitive and psychomotor evaluation factors

[0183]

[0184] ③ Timing judgment criteria for cognitive assessment factors

[0185] Table 10 Timing criteria for cognitive evaluation factors

[0186]

[0187]

[0188] Therefore, the weights of cognitive evaluation factors and the corresponding time series parameters can be inferred.

[0189] (4) Output of cognitive load value

[0190] According to the above process, the corresponding weights and start time data of the four channel factors of VACP can be obtained based on real-time data.

[0191] After the data processing is completed, the system built by the method of the present invention performs real-time accumulation and calculation according to the start time (ts) and end time (te) of each VACP weight according to the preset time interval or event trigger condition. Using efficient graphics processing technology, the system converts these data into intuitive and dynamic curve charts to clearly show the changes in the pilot's cognitive load status at different time points, such as Figure 3 shown.

[0192] The generated curve chart can not only show the changes in the pilot's cognitive load status, but also be used to deeply analyze key information such as the peak time of cognitive load, the association with specific tasks and interface elements, etc. Through the curve chart, operators can accurately locate the peak of cognitive load, understand the specific time points of these peaks and the corresponding task activities and interface situations behind them.

[0193] This real-time presentation process not only accurately reflects the changes in the pilot's cognitive state, but also helps operators instantly understand the cognitive demands and stress conditions of the current mission phase, providing important support for real-time decision-making.

[0194] So far, the purpose of the present invention has been achieved.

[0195] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for evaluating cognitive load of interface interaction based on pilot behavior data, characterized in that: The steps include: Real-time data collection: with the help of eye tracker, rudder and cockpit master control system, real-time data including the pilot's gaze screen coordinates, interface element information, audio information and rudder control information in aviation interface tasks can be collected and recorded in real time; Discrete information processing, combined with the master control system script to process real-time data, extract discrete information required for evaluation, including two types of information: feature and time series; feature information includes the gaze interest area obtained from the gaze coordinates, interface element type, auditory signal type and rudder operation signal type; time series information is the alternating change time node information corresponding to the feature information; The weight evaluation of perceptual and psychomotor factors is to classify and evaluate the obtained discrete information according to the determined factor descriptor definition standard and timing judgment standard, combined with the master control system script, and define the factor weight and corresponding start time for the perceptual and psychomotor discrete data; The cognitive factor weight inference assessment summarizes the evaluation weights of the perceptual and psychomotor factors, and infers the cognitive evaluation load level and corresponding weights based on the coupling relationship network of cognitive load factors; Output of cognitive load value, annotate and visualize the above information according to the time series relationship, accumulate cognitive load according to the time series, and sum it up every time the perception, cognition and psychomotor processes are carried out to achieve real-time output of the pilot's cognitive load quantification curve.

2. The method for evaluating the interface interaction cognitive load based on pilot behavior data according to claim 1, characterized in that: The real-time data includes: Eye movement data, using high-precision eye tracker equipment to capture and record in real time the coordinates of the pilot's gaze point when operating the aviation interface; Interface data, obtain the interface UI element characteristics and interface element mapping information through the central control system; Audio data, collecting audio data in the cockpit; The stick and rudder data uses the flight stick and rudder and script data acquisition system to record and analyze the pilot's control behavior in aviation interface tasks in real time.

3. The method for evaluating the interface interaction cognitive load based on pilot behavior data according to claim 2, characterized in that: The real-time data discrete processing includes: Eye movement data processing, the coordinates of the gaze point can accurately reflect the pilot's visual focus point, and can be analyzed to obtain the gaze interest area at each moment, as well as the switching time node of the interest area; Interface data processing, mapping information can be retrieved in real time, and can further feedback the associated interface data information based on the visual interest area, auditory signal type and rudder operation information type obtained by evaluation; Audio data processing: obtaining the type of warning signal the pilot is exposed to through audio data analysis; Stick and rudder data processing, stick and rudder data includes key information such as the pilot's operation method of the joystick and pedals, control type and frequency.

4. The method for evaluating the interface interaction cognitive load based on pilot behavior data according to claim 3, characterized in that: The weight evaluation of the perception and psychomotor factors includes: The visual perception assessment factor consists of 7 auditory descriptors to describe the visual behavior and the degree of mobilization of visual information processing resources; The auditory perception assessment factor consists of 7 auditory descriptors, which are used to describe the auditory behavior and the degree of mobilization of auditory information processing resources; The psychomotor assessment factor is mainly composed of 7 descriptors, which are used to describe the degree of psychological load caused by the behavior of effectors such as hands and feet.

5. The method for evaluating the interface interaction cognitive load based on pilot behavior data according to claim 4, characterized in that: The visual perception evaluation factors include: Descriptor "V1-Visual Start", defined as the visual action that starts the task in the cockpit interaction environment from scratch, with a weight of 1.0; The descriptor "V2-Visual Discrimination", defined as the visual action of distinguishing whether the signal symbols composed of the basic visual stimuli appear different, has a weight of 3.7; Descriptor "V3-Visual Inspection", defined as the visual action of determining whether the state signal composed of semantically based basic visual stimuli is the target state, with a weight of 4.0; Descriptor "V4-Visual localization / alignment", defined as the visual action that determines whether the state signal composed of semantically based basic visual stimuli is the target state, with a weight of 5.0; Descriptor "V5-Visual pursuit / tracking", defined as the visual action of continuously observing a dynamic target with continuous displacement, with a weight of 5.4; Descriptor "V6-Visual Reading", defined as the visual behavior of reading text-like elements, with a weight of 5.9; Descriptor "V7-Visual scanning / searching / monitoring", defined as visual actions that continuously observe multiple targets or multiple types of information in a region of interest with high information density, with a weight of 7.0; The auditory perception evaluation factors include: Descriptor "A1-Detect / record sound", defined as identifying whether sound exists, that is, judging whether sound occurs in the environment, with a weight of 1.0; Descriptor "A2 - Sound direction localization", defined as the approximate directional sound position in space, with a weight of 2.0; Descriptor "A3-sound orientation localization", defined as selecting a sound stimulus and orienting the sound position in space, with a weight of 4.2; Descriptor "A4-Verify auditory feedback", defined as detecting whether the expected auditory stimulus feedback occurs after the operation, with a weight of 4.3; Descriptor "A5-Interpretation of semantic content", defined as semantic interpretation during communication, with a weight of 4.9; Descriptor "A6-Discrimination of sound features", defined as the ability to distinguish the differences between different sound stimuli in the same space, with a weight of 6.6; Descriptor "A7-Interpretation of sound mode", defined as identifying sound stimuli with specific semantics and judging the corresponding state, with a weight of 7.0; The psychomotor assessment factors include: The descriptor "P1-speech", defined as communication through spoken language, has a weight of 1.0; The descriptor "P2-Discrete Actuation", defined as operating discrete control devices such as buttons, switches, or triggers, has a weight of 2.2; Descriptor "P3-Continuous Adjustment", defined as continuous adjustment of the control device through smooth, continuous action, with a weight of 2.6; The descriptor "P4-manipulativeness", defined as the use of fine motor movements for precise control of tools or equipment, had a weight of 4.6; The descriptor "P5-Discrete Adjustment", defined as discrete adjustment of the device or instrument by rotation, vertical slide, or lever adjustment, has a weight of 5.8; The descriptor "P6-Symbolic production", defined as the production and recording of symbols or words through writing tools, has a weight of 6.5; The descriptor "P7-Serial discrete operation" is defined as a series of continuous discrete key operations through the keyboard to input data or instructions, with a weight of 7.

0.

6. The method for evaluating the interface interaction cognitive load based on pilot behavior data according to claim 5, characterized in that: The cognitive assessment factor consists of 7 descriptors, which are used to describe the degree of invocation of cognitive devices in the brain's information processing and understanding process; The descriptor "C1-Automatic", defined as unconscious or simple stimulus-associated response behavior, and no operator behavior is required, has a weight of 1.0; Descriptor "C2-Alternative Choice", defined as identifying the current state and determining whether the next step is needed, with a weight of 1.2; Descriptor "C3-Signal / signal recognition", defined as recognizing the state or signal icon and determining the corresponding operation, with a weight of 3.7; Descriptor "C4-Evaluation / Judgment (Single Consideration)", defined as judging the current state and next action based on a single basis, with a weight of 4.6; Descriptor "C5-Encoding / Decoding, Memory", defined as involving the interpretation of the semantics of information in a fixed format, with a weight of 5.3; Descriptor "C6-Evaluation / Judgment (Comprehensive Consideration)", defined as judging the current state and next action based on multiple criteria, with a weight of 6.8; The descriptor "C7-Estimation, Calculation, Conversion", defined as cognitive behaviors involving numerical calculations and percentage judgments, has a weight of 7.

0.

7. The method for evaluating the interface interaction cognitive load based on pilot behavior data according to claim 6, characterized in that: The coupling relationship between the perception, psychomotor factors and cognitive assessment factors includes: V1 behavior usually refers to the process from nothing to something in an interactive environment. When there is no clear instruction on the next operation, the pilot's cognition of the scene is a simple stimulus response, which is coupled with C1. For V2 behavior, it is necessary to distinguish whether there are differences in the signal symbols composed of the basic visual stimuli, and then identify the signal symbols and determine the corresponding operations, which has a coupling relationship with C3; For V4 behavior, visual adjustment needs to be made towards a static target at a relative distance. In interactive scenarios, the more common static targets are interactive buttons or sliders. For interactive buttons, the pilot needs to identify the current state of the button and click it, which is usually coupled with C2. For sliders, after sliding to the expected position, the pilot needs to determine whether the sliding degree is in compliance, which is coupled with C7. For V5 behavior, it is required to continuously pay attention to the dynamic target of continuous displacement; under general mission requirements, the pilot may be required to make a state judgment, or even combine other information to make an overall state judgment of the working area. Therefore, it has a coupling relationship with C4 and C6 respectively, and the coupling situation depends on the mission requirements and operational response; For V7 behavior, it corresponds to multi-target information, and the scanning range is usually a large working area. The operational response is not completely clear, so it has a coupling relationship with C3 and C6 respectively; For V6 behavior, the corresponding interactive elements are mainly composed of pictures and texts, so they are coupled with C4, C5 and C7 respectively.

8. The method for evaluating the interface interaction cognitive load based on pilot behavior data according to claim 7, characterized in that: The coupling relationship between the auditory perception and cognitive evaluation factors includes: For A1 behavior, it usually means the process of auditory tasks from scratch in an interactive environment, which can be used as a baseline for auditory-related mental load; therefore, it is generally coupled with C1; For A2 behavior, the pilot is required to perform directional sound localization in space based on auditory clues. The pilot may not need to react, but only need to understand the source of the sound, or may need to locate the sound to make a one-step reaction behavior. Therefore, it has a coupling relationship with C1 and C2 respectively. For A3 behavior, the pilot is required to locate the sound of orientation and direction in space based on auditory cues, which means that this type of sound cues requires the pilot to identify the current state and determine whether the next operation is required. Therefore, it has a coupling relationship with C3 and C4 respectively; For A4 behavior, the pilot is required to verify the feedback of the auditory cues; if the verification is successful, it has a coupling relationship with C1; if the verification is unsuccessful, it has a coupling relationship with C4 and C6 respectively according to the difficulty set by the task operation steps; For A5 behavior, it is required to interpret various types of information semantics based on the content referred to by the auditory clues, that is, it has a coupling relationship with C5 and C7 respectively; For A6 behavior, the pilot is required to distinguish the differences between different sound stimuli, which is similar to V2 behavior in the perceptual visual layer in terms of task type, and therefore has a coupling relationship with C2; For A7 behavior, pilots are required not only to distinguish voice features and specific semantic behaviors, but also to judge the state mode corresponding to the sound clues; that is, to be able to interpret the semantics of information, make judgments based on the basis, and may involve data calculation or state judgment based on data, etc., and therefore have a coupling relationship with C5 and C6 respectively.

Images