Operator situation awareness state determination system and method based on eye movement and electroencephalographic features

Abstract

The application provides an operator situation awareness state judgment system and method based on eye movement and electroencephalogram features. The system can realize simulation of a situation awareness standard task containing multiple subtask information, and provides an online operator situation awareness state judgment method based on eye movement and electroencephalogram features, which has the characteristics of strong objectivity, good real-time performance, high accuracy and the like, can provide technical support for situation awareness design of an operator human-machine interaction system, and thus guarantees work performance and human factor safety.

Description

Technical Field

[0001] This invention relates to a system and method for determining the situational awareness state of an operator based on eye movement and electroencephalogram (EEG) characteristics. Background Technology

[0002] In safety-critical fields, over 70% of safety accidents and incidents are caused by human error, and 88% of these errors are considered attributable to situation awareness (SA) issues. Numerous studies have shown that the higher an operator's level of situation awareness, the more quickly and effectively they can perform tasks, thus contributing to better operational performance and safety.

[0003] Current operator situational awareness assessment systems still have some shortcomings. First, most current measurements of operator situational awareness are task-based, lacking a standardized assessment system. Second, current methods primarily focus on objective Situation Awareness Global Assessment Technique (SAGAT) and subjective Situation Awareness Rating Technique (SART) scales, lacking consideration of real-time physiological indicators (especially eye-tracking and EEG indicators). Finally, current situational awareness discrimination algorithms based on eye-tracking and EEG indicators are still immature and require further research.

[0004] Based on the above, a system and method for determining the situational awareness state of operators based on eye movement and EEG characteristics are designed. This system can solve the above-mentioned shortcomings and can be applied to the determination of the situational awareness state of operators in typical work scenarios, thereby providing a certain reference for the design of human-computer interaction and function allocation in operator work. Summary of the Invention

[0005] The purpose of this invention is to provide a system and method for determining the situational awareness state of an operator based on eye movement and electroencephalogram (EEG) characteristics. This system and method provide a subsystem for setting up standard task scenarios based on situational awareness and propose a method for determining the situational awareness state of an operator based on eye movement and EEG characteristics.

[0006] According to one aspect of the present invention, an operator situational awareness state determination system based on eye movement and electroencephalogram (EEG) features is provided, comprising: a situational awareness standard task setting subsystem for setting subtask presentation information of the situational awareness standard task; and an operator situational awareness state determination subsystem for determining the operator's situational awareness state under the situational awareness standard task.

[0007] According to a further aspect of the present invention, the situational awareness state determination subsystem includes an operator eye movement index measurement module, an operator electroencephalogram (EEG) index measurement module, and an operator situational awareness state determination module. The operator eye movement index measurement module is used to record and analyze the changes in the operator's fixation time and the nearest neighbor index (NNI) during the operator's work. The operator electroencephalogram (EEG) index measurement module is used to record and analyze the operator's relative power at electrode β_r_P3 (P3 electrode), relative power at electrode β_r_P4 (P4 electrode), θ / β_CZ (CZ electrode), θ / β_C4 (C4 electrode), θ / β_P3 (P3 electrode), θ / β_PZ (PZ electrode), θ / β_P4 (P4 electrode), and α / β_PZ (P4 electrode). P4 is the α / β, (θ+α) / β of the P4 electrode point._CZ is the (θ+α) / β, (θ+α) / β of the CZ electrode point._C4 is the (θ+α) / β, (θ+α) / β of the C4 electrode point. P3 is the (θ+α) / β, (θ+α) / β_PZ of the P3 electrode point. The (θ+α) / β, (θ+α) / β_P4 of the P4 electrode point is the (θ+α) / β, (θ+α) / (α+β)_P3 of the P3 electrode point.

[0008] According to another aspect of the present invention, a method for determining the situational awareness state of an operator based on eye movement and electroencephalogram (EEG) characteristics is provided, characterized by comprising: a situational awareness standard task setting sub-step for setting sub-task presentation information of the situational awareness standard task; and an operator situational awareness level determination sub-step for measuring the operator's situational awareness state under the situational awareness standard task. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of an operator situational awareness state determination system based on eye movement and electroencephalogram (EEG) features according to an embodiment of the present invention.

[0010] Figure 2 An interface layout diagram for a context-aware standard task setting subsystem according to an embodiment of the present invention.

[0011] Figure 3 This is a flowchart illustrating the overall operation of an operator situational awareness state determination system based on eye movement and electroencephalogram (EEG) features according to an embodiment of the present invention.

[0012] Figures 4A-4BThis is a flowchart illustrating the operation of the operator eye movement index measurement module and the operator EEG index measurement module in an operator situational awareness state determination subsystem based on eye movement and EEG characteristics according to an embodiment of the present invention.

[0013] Figure 5 This is a flowchart of an operator situational awareness state determination module based on eye movement and EEG features according to an embodiment of the present invention.

[0014] Figures 6A-6C This is an example diagram of the interface of an operator situational awareness state determination subsystem based on eye movement and electroencephalogram (EEG) features according to an embodiment of the present invention. Detailed Implementation

[0015] like Figure 1 As shown, the operator situational awareness state determination system based on eye movement and EEG characteristics according to the present invention includes a situational awareness standard task setting subsystem and an operator situational awareness state determination subsystem. The operator situational awareness state determination subsystem includes an operator eye movement index measurement module, an operator EEG index measurement module, and an operator situational awareness state determination module.

[0016] like Figure 2 As shown, a context-aware standard task setting subsystem according to an embodiment of the present invention includes:

[0017] The subtask category settings module is used to set the presentation category of subtask information; Figure 2 In the illustrated embodiment, the subtask information can be set to a category range of 0-6, including: tracking task, communication task, instrument anomaly monitoring task, alarm monitoring task, communication monitoring task, and resource management task, for a total of 6 subtask categories;

[0018] The subtask count setting module is used to set the number of times each type of subtask is presented, in "times";

[0019] The subtask duration setting module is used to set the presentation duration of each type of subtask, in seconds.

[0020] The subtask interval setting module is used to set the interval time for each type of subtask, in seconds.

[0021] In such Figure 3In the embodiment of the operator situational awareness state determination system based on eye movement and EEG characteristics shown, the situational awareness standard task setting subsystem is first used to set the subtask presentation information of the situational awareness standard task during the operator's operation; then, the operator eye movement index measurement module in the operator situational awareness state determination subsystem is used to record and analyze the changes in the operator's eye movement data during the operation, the operator EEG index measurement module is used to record and analyze the changes in the operator's EEG data during the operation, and the operator situational awareness state determination module is used to compare and determine the operator's situational awareness state in the current situation, and finally, the operator's situational awareness state determination result is given.

[0022] like Figures 4A to 4BAs shown, after the context-aware standard task setting subsystem sets the sub-task information of the context-aware standard task presented during the operator's operation, the operator's eye movement index measurement module records and analyzes the fixation time and NNI of the operator's eyes during the operation. Fixation time is defined as the total fixation time on the current page plus the time between different fixation points, rounded to two decimal places. NNI is dimensionless data, calculated based on the ratio of nearest neighbor distance to average random distance. First, an eye tracker is used to collect operator eye movement data. Then, computer programming is used to calculate the fixation time and NNI respectively, obtaining the operator's eye movement index measurement results. Simultaneously, the operator's EEG index measurement module records and analyzes the following parameters during the operator's work: β_r_P3 (β relative power of P3 electrode), β_r_P4 (β relative power of P4 electrode), θ / β_CZ (θ / β of CZ electrode), θ / β_C4 (θ / β of C4 electrode), θ / β_P3 (θ / β of P3 electrode), θ / β_PZ (θ / β of PZ electrode), θ / β_P4 (θ / β of P4 electrode), α / β_P4 (α / β of P4 electrode), (θ+α) / β_CZ ((θ+α) / β of CZ electrode), (θ+α) / β_C4 ((θ+α) / β of C4 electrode), (θ+α) / β_P3 ((θ+α) / β of P3 electrode), (θ+α) / β_P3 ((θ+α) / β of P3 electrode). PZ refers to the (θ+α) / β index at electrode PZ, (θ+α) / β_P4 refers to the (θ+α) / β index at electrode P4, (θ+α) / (α+β)_P3 refers to the (θ+α) / (α+β) index at electrode P3, and (θ+α) / (α+β)_P4 refers to the (θ+α) / (α+β) index at electrode P4. First, the segmented EEG data undergoes a Fast Fourier Transform, and after averaging, it is divided into four different frequency bands: δ (1-4Hz), θ (4-8Hz), α (8-12Hz), and β (13-30Hz). The power percentage and the ratio between these bands and the typical frequency band are then calculated. β_r_P3 and β_r_P4 represent the power percentage. All other EEG indicators are ratios, and all indicators are retained to four significant figures. First, EEG data of the operator is collected using an EEG analyzer. Then, the above indicators are calculated using computer programming to obtain the operator's EEG indicator measurement results.

[0023] like Figure 5 The diagram illustrates the operation process of an operator situational awareness state determination module according to an embodiment of the present invention. The workflow is as follows: after completing the measurement tasks in the operator eye-tracking index measurement module and the operator EEG index measurement module, the obtained operator eye-tracking index measurement results and operator EEG index measurement results are first substituted into the following three equations to obtain situational awareness discrimination feature indicators constructed based on principal component analysis on the basis of previous experimental data:

[0024]

[0025]

[0026]

[0027] In equations (1) to (3), F1, F2, and F3 are the first to third situational awareness discriminant feature indicators, x1 is the fixation dwell time, x2 is the NNI, and x3 to x 17 These are β_r_P3, β_r_P4, θ / β_CZ, θ / β_C4, θ / β_P3, θ / β_PZ, θ / β_P4, α / β_P4, (θ+α) / β_CZ, (θ+α) / β_C4, (θ+α) / β_P3, (θ+α) / β_PZ, (θ+α) / β_P4, (θ+α) / (α+β)_P3, and (θ+α) / (α+β)_P4, respectively. Furthermore, the above three situational awareness discrimination feature indicators are input into the following two equations, which are operator situational awareness state discrimination equations established based on a large amount of experimental data and the Bayes discrimination method.

[0028] Y1=-1.062F1-0.006F2+0.053F3-1.033 (4),

[0029] Y2=1.062F1+0.006F2-0.053F3+1.033 (5);

[0030] In equations (4) to (5), Y1 and Y2 are the discriminant function values ​​for low and high situational awareness states, respectively, and F1, F2, and F3 are the situational awareness discrimination feature index values ​​in equations (1) to (3), respectively. Substituting F1 to F3 into equations (4) to (5) and comparing the values ​​of Y1 and Y2, if Y1 is greater than Y2, the operator is judged to be in a low situational awareness state; if Y1 is less than Y2, the operator is judged to be in a high situational awareness state. If Y1 equals Y2, the operator needs to return to the situational awareness standard task setting subsystem for operation. Finally, the judgment is made according to the above situational awareness state judgment criteria, and the judgment result is output.

[0031] like Figures 6A to 6C The diagram shown is a user interface illustration of an operator situational awareness state determination system based on eye movement and EEG characteristics, according to a specific example of an embodiment of the present invention. After the system enters the operator situational awareness state determination system based on eye movement and EEG characteristics, its main interface is as follows: Figure 6A As shown in the image. This interface contains four buttons: "Instructions for Use," "Situational Awareness Status Determination," and "Exit System." After entering this interface, the user first clicks the "Instructions for Use" button to access the instructions interface, as shown below. Figure 6BAs shown. Users read the system's user manual to understand how to use the system to determine the operator's situational awareness state. After reading the manual, click the "Return to Homepage" button to return to the main interface. When performing a situational awareness state determination, the user clicks the "Situational Awareness State Determination" button to enter the determination system interface, as shown. Figure 6C As shown. In the input section, the user only needs to input the eye movement index results (including fixation time and NNI) collected during the operator's work, as well as the EEG index measurement results (β_r_P3, β_r_P4, etc.). In the output section, the system will automatically provide the corresponding discriminant function value based on the input and give the final result of the operator's situational awareness state.

[0032] The advantages and beneficial effects of the present invention include

[0033] (1) A situation-aware standard task setting subsystem is provided. A situation-aware standard task setting subsystem is connected to a typical job task interaction device. It can simulate the type, number of times, presentation duration and interval of sub-tasks of situation-aware standard tasks in typical job tasks.

[0034] (2) A situational awareness level measurement system based on situational awareness standard measurement task is provided. With the help of situational awareness standard task setting subsystem device, after simple settings, the situational awareness state of the human-computer interaction system under situational awareness standard task can be objectively determined.

[0035] (3) An online method for determining the situational awareness state of an operator based on eye movement and EEG features is provided. By simply inputting objective indicators such as the fixation time of the operator's eye movement and NNI, and β_r_P3 and β_r_P4 of the EEG, the situational awareness state of the operator can be directly determined.

[0036] (4) Compared with traditional methods for determining the situational awareness state of operators, the present invention can discover relevant situational awareness design problems in the early stage of human-computer interaction system development. It can also combine eye-tracking and EEG data to determine the high and low situational awareness states in real time. It can be used to directly guide the situational awareness design of human-computer interaction systems for operators, ensuring work performance and human safety.

Claims

1. A system for determining the situational awareness state of an operator based on eye movement and electroencephalogram (EEG) characteristics, characterized in that... include: The Situational Awareness Standard Task Setting Subsystem is used to set the subtask presentation information for the Situational Awareness Standard Measurement Task; The operator situational awareness level determination subsystem is used to determine the operator's situational awareness state under the standard situational awareness task. in: The context-aware standard task setting subsystem includes: The subtask category setting module is used to set the presentation category of subtask information; The subtask count setting module is used to set the number of times each type of subtask is displayed, in "times"; The subtask duration setting module is used to set the presentation duration of each type of subtask, in seconds; The subtask interval setting module is used to set the interval time for each type of subtask, in seconds. The operator situational awareness level determination subsystem includes: The operator eye movement index measurement module is used to record and analyze the changes in the operator's fixation time and the nearest neighbor index (NNI) during the operation. The operator's electroencephalogram (EEG) index measurement module is used to record and analyze the operator's EEG during the operation process, including the β_r_P3 (β relative power at P3 electrode), β_r_P4 (β relative power at P4 electrode), θ / β_CZ (θ / β at CZ electrode), θ / β_C4 (θ / β at C4 electrode), θ / β_P3 (θ / β at P3 electrode), θ / β_PZ (θ / β at PZ electrode), θ / β_P4 (θ / β at P4 electrode), and α / β_P4 (α / β at P4 electrode). The changes in the indices of (θ+α) / β_CZ (CZ electrode point), (θ+α) / β_C4 (C4 electrode point), (θ+α) / β_P3 (P3 electrode point), (θ+α) / β_PZ (PZ electrode point), (θ+α) / β_P4 (P4 electrode point), (θ+α) / (α+β)_P3 (P3 electrode point), and (θ+α) / (α+β)_P4 (P4 electrode point) are described. The operator situational awareness state determination module is used to determine the operator's situational awareness state. Includes modules that perform the following processing: Substituting the obtained operator eye-tracking index and operator electroencephalogram (EEG) index results into the following three equations, we obtain the situational awareness discrimination feature index constructed based on principal component analysis on the basis of previous experimental data: （1）， （2）， （3）， In the formula, F1, F2, and F3 are the first to third contextual awareness discriminant feature indicators, respectively; x1 is the fixation dwell time; x2 is the NNI; and x3~x 17 They are β_r_P3, β_r_P4, θ / β_CZ, θ / β_C4, θ / β_P3, θ / β_PZ, θ / β_P4, α / β_P4, (θ+α) / β_CZ, (θ +α) / β_C4, (θ+α) / β_P3, (θ+α) / β_PZ, (θ+α) / β_P4, (θ+α) / (α+β)_P3 and (θ+α) / (α+β)_P4, Input the above-mentioned first to third situational awareness discrimination feature indicators into the following two equations. These two equations are the operator situational awareness state discrimination equations established based on the Bayes discrimination method on the basis of a large amount of experimental data: （4）， （5）， In the formula, Y1 and Y2 are the discriminant function values ​​for low and high situational awareness states, respectively. Substitute F1~F3 into formulas (4)~(5) and compare the values ​​of Y1 and Y2. If Y1 is greater than Y2, the operator is judged to be in a low situational awareness state. If Y1 is less than Y2, the operator is judged to be in a high situational awareness state. If Y1 is equal to Y2, return to the situational awareness standard task setting subsystem to perform the operation. In this way, the judgment is made according to the above situational awareness state judgment criteria, and the judgment result is output.

2. The operator situational awareness state determination system based on eye movement and EEG characteristics according to claim 1, characterized in that: The presentation categories include: tracking tasks, communication tasks, instrument anomaly monitoring tasks, alarm monitoring tasks, communication monitoring tasks, and resource management tasks.

3. The operator situational awareness state determination system based on eye movement and EEG characteristics according to claim 1, characterized in that: The operator eye movement index measurement module uses a desktop or glasses-type eye tracker to collect raw eye movement fixation point data, and outputs fixation duration and NNI index through eye movement analysis software; The operator's EEG index measurement module uses a 32-, 64-, or 128-channel conductive cap to collect raw data on the distribution of corresponding brain electrode points. The EEG analysis software or EEGlab toolkit is used to calculate the indices β_r_P3, β_r_P4, θ / β_CZ, θ / β_C4, θ / β_P3, θ / β_PZ, θ / β_P4, α / β_P4, (θ+α) / β_CZ, (θ+α) / β_C4, (θ+α) / β_P3, (θ+α) / β_PZ, (θ+α) / β_P4, (θ+α) / (α+β)_P3, and (θ+α) / (α+β)_P4.

4. The operator situational awareness state determination system based on eye movement and EEG characteristics according to claim 1, characterized in that: The operator situational awareness level determination subsystem supports specific storage location settings and can export result lists and / or situational awareness state discrimination diagrams in Excel, TSV, and EPS formats.

5. A method for determining the situational awareness state of an operator based on eye movement and electroencephalogram (EEG) characteristics, characterized in that... include: The contextual awareness standard task setting sub-step is used to set the sub-task presentation information for the contextual awareness standard measurement task; The operator situational awareness level determination sub-step is used to determine the operator's situational awareness state under the standard situational awareness task. in: The sub-steps for setting up a situational awareness standard task include: The steps for setting subtask categories are used to set the presentation category of subtask information. The subtask count setting step is used to set the number of times each type of subtask is presented, in "times"; The subtask duration setting step is used to set the presentation duration for each type of subtask, in seconds; The subtask interval setting step is used to set the interval time for each type of subtask, in seconds. The sub-steps for determining the operator's situational awareness level include: The operator eye movement index measurement procedure is used to record and analyze the changes in the operator's fixation time and the nearest neighbor index (NNI) during the operator's work process; The operator's electroencephalogram (EEG) measurement procedure is used to record and analyze the operator's EEG parameters during the operation process, including: β_r_P3 (β relative power at P3 electrode), β_r_P4 (β relative power at P4 electrode), θ / β_CZ (θ / β at CZ electrode), θ / β_C4 (θ / β at C4 electrode), θ / β_P3 (θ / β at P3 electrode), θ / β_PZ (θ / β at PZ electrode), θ / β_P4 (θ / β at P4 electrode), and α / β_P4 (α / β at P4 electrode). The changes in the indices of (θ+α) / β_CZ (CZ electrode point), (θ+α) / β_C4 (C4 electrode point), (θ+α) / β_P3 (P3 electrode point), (θ+α) / β_PZ (PZ electrode point), (θ+α) / β_P4 (P4 electrode point), (θ+α) / (α+β)_P3 (P3 electrode point), and (θ+α) / (α+β)_P4 (P4 electrode point) are described. The operator situational awareness state determination step is used to assess the operator's situational awareness state. include: Substituting the obtained operator eye movement index measurement results and operator EEG index measurement results into the following three equations, we obtain the situational awareness discrimination feature index constructed based on the principal component analysis method on the basis of the previous experimental data. （1）， （2）， （3）， In the formula, F1, F2, and F3 are the first to third contextual awareness discrimination feature indicators, respectively; x1 is the fixation dwell time; x2 is the NNI; and x3~x 17 They are β_r_P3, β_r_P4, θ / β_CZ, θ / β_C4, θ / β_P3, θ / β_PZ, θ / β_P4, α / β_P4, (θ+α) / β_CZ, (θ +α) / β_C4, (θ+α) / β_P3, (θ+α) / β_PZ, (θ+α) / β_P4, (θ+α) / (α+β)_P3 and (θ+α) / (α+β)_P4; Input the first to third situational awareness discriminant feature indices into the following two equations. These two equations are operator situational awareness state discriminant equations established based on the Bayes discriminant method and a large amount of experimental data: （4）， （5）， In the formula, Y1 and Y2 are the discriminant function values ​​for low and high situational awareness states, respectively, and F1, F2 and F3 are the first to third situational awareness discriminant feature index values, respectively. Substitute F1~F3 into formula (4)~(5) and compare the values ​​of Y1 and Y2. If Y1 is greater than Y2, the operator is judged to be in a low situational awareness state. If Y1 is less than Y2, the operator is judged to be in a high situational awareness state. If Y1 is equal to Y2, return to the situational awareness standard task setting sub-step to perform the operation, and then make a judgment based on the above situational awareness state judgment criteria and output the judgment result.

6. The method for determining the operator's situational awareness state based on eye movement and electroencephalogram (EEG) characteristics according to claim 5, characterized in that: The presentation categories include: tracking tasks, communication tasks, instrument anomaly monitoring tasks, alarm monitoring tasks, communication monitoring tasks, and resource management tasks.

7. The method for determining the operator's situational awareness state based on eye movement and EEG characteristics according to claim 5, characterized in that: The operator eye movement index measurement sub-step uses a desktop or glasses-type eye tracker to collect raw eye movement fixation point data, and outputs fixation duration and NNI index through eye movement analysis software; The operator's EEG index measurement sub-step uses a 32-, 64-, or 128-channel conductive cap to collect raw data on the distribution of corresponding brain electrode points. The EEG analysis software or EEGlab toolkit is used to calculate the β_r_P3, β_r_P4, θ / β_CZ, θ / β_C4, θ / β_P3, θ / β_PZ, θ / β_P4, α / β_P4, (θ+α) / β_CZ, (θ+α) / β_C4, (θ+α) / β_P3, (θ+α) / β_PZ, (θ+α) / β_P4, (θ+α) / (α+β)_P3, and (θ+α) / (α+β)_P4 indices.

8. The method for determining the operator's situational awareness state based on eye movement and electroencephalogram (EEG) characteristics according to claim 5, characterized in that: The operator situational awareness level determination sub-step supports specific storage location settings and can export result lists and / or situational awareness state discrimination diagrams in Excel, TSV, and EPS formats.

9. A computer-readable storage medium storing a computer program that enables a processor to perform the method according to any one of claims 5-8.

Images