Real-time monitoring system of pilot attention stability based on EEG and eye movement data

Abstract

This invention discloses a real-time monitoring system for pilot attentional stability based on EEG and eye-tracking data, comprising an EEG and eye-tracking data acquisition module, a neurophysiological data processing module, a neurophysiological feature calibration module, and an attentional stability calculation module. The attentional stability model in the attentional stability calculation module is optimized for portable EEG and eye-tracking devices, solving the feasibility problem of applying existing attentional stability monitoring methods in aircraft cockpits. This invention achieves personalized real-time monitoring of attentional stability by real-time acquisition, processing, and analysis of EEG and eye-tracking data, and by integrating a personalized EEG calibration module to perform baseline acquisition and data calibration for specific individuals based on gradual continuous operation tasks. Furthermore, by using gradual continuous operation tasks validated by psychological research for algorithm modeling and data calibration, it solves the previous problem of not being able to monitor attentional stability across different scenarios.

Description

Technical Field

[0001] This invention relates to the field of human-machine-environment systems technology, and to a real-time monitoring system for the stability of a pilot's attention (including normal attention and excessive attention) during a monitoring mission. Background Technology

[0002] Real-time monitoring of pilot attention and stability is a type of human state recognition technology within human-machine interface (HMI) systems. The purpose of HMI human state recognition is to evaluate human capabilities, monitor system safety, and improve human-machine interaction efficiency by identifying human behavior, cognitive states, and emotional states in real time. Attention, emotion, fatigue, workload, and operational intent are among the most closely watched and valuable human states.

[0003] In combat missions, attentional stability can sometimes be a critical factor in determining victory or defeat. For example, pilots often become overly focused when locking onto an enemy aircraft, experiencing an attention tunneling effect, and fail to notice that their aircraft is being tailed and attacked, leading to being shot down. Therefore, if the cockpit can monitor the pilot's attentional stability in real time and provide early warnings and interventions when the pilot is overly focused, it can improve the pilot's combat effectiveness.

[0004] There are three main methods for monitoring attentional stability. First, by detecting changes in performance behavior. For example, in medicine, a person's ability to maintain attentional stability can be measured by observing changes in their reaction time during an alertness task over a period of time. Second, by detecting changes in eye movement patterns. For example, whether the eyes are focused on the primary task area can indicate whether a person's attention is being diverted. Third, by detecting changes in brain activity characteristics. For example, by collecting brainwave data using an electroencephalogram (EEG) and analyzing its patterns, it is possible to determine whether a person is in a state of focused attention or distraction.

[0005] Each of the three methods described above has its limitations. Methods that identify stable attentional states by observing changes in operational behavior are generally only applicable to assessing a person's ability to maintain attention. When used for real-time monitoring, they suffer from a time delay (attentional distraction can only be detected after it has led to a decline in performance). Methods that identify stable attentional states by observing whether the eyes are focused on the primary task area are only applicable to task scenarios where different task areas can be spatially distinguished. Methods that detect stable attentional states by observing changes in brain activity characteristics either require multi-channel EEG acquisition equipment, which is difficult to use in an aircraft cockpit, to collect multi-channel EEG signals as input, or they are modeled for specific usage scenarios (such as students listening to lectures) and cannot be transferred to flight scenarios. In summary, existing methods for monitoring stable attentional states have not solved the problem of effectively monitoring pilots' stable attentional states in the real-time within an aircraft cockpit. Summary of the Invention

[0006] The purpose of this invention is to provide a real-time monitoring system for pilot attentional stability based on EEG and eye-tracking data. By utilizing data collected from portable EEG and eye-tracking devices, a feasible and effective real-time monitoring system for pilot attentional stability can be achieved in the aircraft cockpit.

[0007] A real-time monitoring system for pilot attentional stability based on EEG and eye-tracking data includes an EEG and eye-tracking data acquisition module, a neurophysiological data processing module, a neurophysiological feature calibration module, and an attentional stability calculation module, wherein:

[0008] The EEG and eye-tracking data acquisition module connects to the portable EEG and eye tracker worn by the user and collects the EEG and eye-tracking data acquired by the EEG and eye tracker.

[0009] The neurophysiological data processing module is used to filter, remove artifacts, calibrate, and extract features from the collected EEG data to obtain EEG data features; it also extracts and calibrates the number of abnormal pupil size changes from the eye movement data to obtain eye movement data features; specifically, it uses EEG baseline feature parameters to calibrate the artifact-removed EEG data, and uses eye movement baseline feature parameters to calibrate the features extracted from the eye movement data.

[0010] The neurophysiological feature calibration module uses the EEG and eye movement data acquisition module to acquire the user's EEG baseline data and eye movement baseline data in a specific state, and outputs them to the neurophysiological data processing module to filter and remove artifacts from the EEG baseline data, and extract the number of abnormal pupil size changes from the eye movement baseline data to obtain EEG baseline feature parameters and eye movement baseline feature parameters.

[0011] The attention stability calculation module takes the EEG data features and eye movement data features output by the neurophysiological data processing module as input, and outputs attention stability state labels through a trained attention stability state model; the attention stability state model adopts a nonlinear support vector machine model.

[0012] Furthermore, the filtering process for the EEG data is as follows:

[0013] For each segment of real-time acquired EEG data, a fourth-order Butterworth filter is used to obtain signals from 0.5Hz to 100Hz to remove slow signals and high-frequency noise, thereby obtaining the filtered raw EEG signal.

[0014] Furthermore, the artifact removal process for the EEG data is as follows:

[0015] First, wavelet packet analysis was used to identify the time intervals in which artifacts were located in the filtered raw EEG signal:

[0016] Wavelet packet analysis using Daubechies 4 as the mother wavelet decomposes the raw EEG signal into 7 layers. Wavelet packet coefficients for layers 1-4 are set to 0, while soft-threshold filtering is applied to the wavelet packet coefficients for layers 5-7. Specifically, the set of wavelet packet coefficients for each layer from 5-7 is extracted, and a threshold is calculated using the soft-threshold formula. Wavelet packet coefficients exceeding the soft threshold are then set to 0, while others remain unchanged. The soft-threshold formula is as follows:

[0017]

[0018] In the above formula, Thres is the soft threshold of wavelet packet coefficients of a certain layer, median() is the median, abs() is the absolute value, and coef is an array consisting of the set of wavelet packet coefficients of a certain layer.

[0019] The reconstructed signal after wavelet packet analysis is the signal containing only artifacts; in the signal containing only artifacts, find points with a variance greater than 3 times, these points are the artifacts that need to be corrected; according to the characteristics of blink artifact signals, the time interval where the artifacts that need to be corrected are located needs to be extended forward and backward by a preset time period to cover the electrical signal brought by blinking; the time interval obtained in this way is the artifact interval.

[0020] Secondly, signal replacement processing is performed on the artifact regions:

[0021] The filtered original EEG signal is then processed by Savitzky-Golay filtering; the data in the artifact intervals are replaced with the data of the original EEG signal processed by Savitzky-Golay filtering, thus completing the artifact removal process.

[0022] Furthermore, the calibration process for the EEG data is as follows:

[0023] The baseline EEG characteristic parameters of the user in a specific state are obtained from the neurophysiological characteristic calibration module, including the average value M of the EEG data. EEG With variance SD EEG The filtered signal is then transformed as follows:

[0024]

[0025] Among them, EEG Ind To calibrate the processed EEG data, Filtered EEG data that has undergone artifact removal processing.

[0026] Furthermore, the characteristics of the EEG data include:

[0027] The mean of the entire signal; the variance of the entire signal; the skewness of the entire signal; the kurtosis of the entire signal; the Hjorth mobility of the entire signal; the Hjorth complexity of the entire signal; the entropy of the entire signal; the Higuchi fractal dimension of the entire signal; the intensity of the delta wave (0.5-4Hz) based on wavelet packet analysis; the skewness of the delta wave (0.5-4Hz) based on wavelet packet analysis; the kurtosis of the delta wave (0.5-4Hz) based on wavelet packet analysis; the intensity of the theta wave (4-8Hz) based on wavelet packet analysis; the skewness of the theta wave (4-8Hz) based on wavelet packet analysis; the theta wave (4-8Hz) based on wavelet packet analysis. Kurtosis; intensity of alpha wave (8-12Hz) based on wavelet packet analysis; skewness of alpha wave (8-12Hz) based on wavelet packet analysis; kurtosis of alpha wave (8-12Hz) based on wavelet packet analysis; intensity of beta wave (12-30Hz) based on wavelet packet analysis; skewness of beta wave (12-30Hz) based on wavelet packet analysis; kurtosis of beta wave (12-30Hz) based on wavelet packet analysis; intensity of beta wave (30-100Hz) based on wavelet packet analysis; skewness of beta wave (30-100Hz) based on wavelet packet analysis; kurtosis of beta wave (30-100Hz) based on wavelet packet analysis.

[0028] Furthermore, the method for determining the number of abnormal pupil size changes in the eye movement data is as follows:

[0029] For each segment of real-time acquired eye-tracking data containing pupil size data, firstly, wavelet packet analysis is performed on the data. Daubechies 8 is used as the mother wavelet, decomposed to the second layer, and soft threshold filtering is applied to the wavelet packet coefficients of the second layer. The number of non-zero wavelet packet coefficients after filtering is the number of abnormal pupil size changes.

[0030] Furthermore, the calibration and feature extraction process for the eye-tracking data is as follows:

[0031] The number of abnormal pupil size changes is subtracted from the average number of abnormal pupil size changes obtained from the neurophysiological feature calibration module, which is the baseline eye movement feature parameter of the user in a specific state. The calibrated number of abnormal pupil size changes is obtained as the unique eye movement data feature extracted from the eye movement data.

[0032] Furthermore, for each user, it is necessary to collect EEG and eye movement data for T seconds while the user is in a resting state as EEG baseline data and eye movement baseline data;

[0033] For baseline EEG data, using t seconds as a window, the average value m of the data in each window is calculated after filtering and artifact removal operations in the neurophysiological data processing module. EEG With variance sd EEG And take the average value M of all windows EEG With variance SD EEG Stored as baseline characteristics of EEG;

[0034] For eye-tracking baseline data, with t seconds as a window, the number of abnormal pupil size changes in each window is calculated using the pupil size abnormality number determination method in the neurophysiological data processing module, and the average value of all windows is calculated. The average value of the number of abnormal pupil size changes is used as the eye-tracking baseline feature parameter.

[0035] A method for real-time monitoring of pilot attentional stability based on electroencephalogram (EEG) and eye-tracking data includes:

[0036] Step 1: Turn on the portable EEG and eye tracker, confirm that the EEG and eye tracker are working properly, and have the driver put on the EEG and eye tracker.

[0037] Step 2: If this is the driver's first time being monitored, calibration data needs to be collected; otherwise, proceed to Step 3. The specific steps for collecting calibration data are as follows:

[0038] Enter the driver's ID, name, gender, and other basic information; instruct the driver to maintain a stable sitting posture, relax their body, and keep their eyes focused on the display screen in the cockpit. Once the driver is in a resting state, enter the calibration data acquisition mode. Utilize the neurophysiological characteristic calibration module, EEG and eye movement data acquisition module to begin acquiring T-second baseline EEG and eye movement data. Then, combine the neurophysiological data processing module to calculate the baseline characteristic parameters of the EEG and eye movement as calibration data. After the calibration data acquisition is completed, the system enters standby mode and proceeds to step 4.

[0039] Step 3: If the driver has already had calibration data collected, enter the driver's number in the system to retrieve the driver's calibration data; if recalibration is required, follow the steps in Step 2 to collect the data again; otherwise, proceed to Step 4.

[0040] Step 4: Begin monitoring stability in the system:

[0041] After the detection begins, the EEG and eye-tracking data acquisition module will continuously receive EEG and eye-tracking data from the EEG and eye tracker, and send them to the neurophysiological data processing module. The neurophysiological data processing module will use t seconds as the moving window, and combine the driver's calibration data from the neurophysiological feature calibration module to filter, remove artifacts, calibrate, and extract features from the EEG data. It will also extract the number of abnormal pupil size changes from the eye-tracking data and perform calibration. Finally, it will extract the EEG and eye-tracking data features from the t seconds of data and output them to the attention stability calculation module. Based on the EEG and eye-tracking data features, the attention stability calculation module will output two attention state labels: "normal attention" or "excessive attention".

[0042] Compared with the prior art, the present invention has the following technical features:

[0043] 1. The real-time algorithm model for attentional stability state in the attentional stability calculation module of this invention is optimized for portable EEG and eye trackers. It can effectively output the attentional stability state with only single-channel EEG data and pupil size data, thus solving the feasibility problem of applying existing attentional stability state monitoring methods in aircraft cockpits.

[0044] 2. This invention achieves personalized real-time monitoring of attentional stability by real-time acquisition, processing and analysis of EEG and eye movements, and by integrating a personalized EEG calibration module to perform baseline acquisition and data calibration for specific individuals based on gradual continuous operation tasks.

[0045] 3. This invention uses a gradual, continuous operation task that has been verified by psychological research for algorithm modeling and data calibration, which to some extent deviates from the monitoring of attentional stability in specific task scenarios, thus solving the problem that it is previously impossible to monitor attentional stability across scenarios. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0047] Figure 2 This is a schematic diagram of the usage process of the present invention;

[0048] Figure 3 This is a schematic diagram of a portable EEG acquisition device.

[0049] Figure 4 This is a schematic diagram of a portable eye tracker;

[0050] Figure 5 A schematic diagram of experimental materials for a gradual continuous operation task experimental paradigm.

[0051] Figure 6 This is a schematic diagram illustrating the state analysis of the experimental paradigm for a gradual continuous operation task. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0053] This invention employs a gradual, continuous operational task experimental paradigm to conduct human factors experiments. It uses portable EEG and eye-tracking devices to collect data, extracts neurophysiological features related to attentional stability, constructs a machine learning model based on support vector machines, and uses the constructed model for real-time monitoring of pilots' attentional stability.

[0054] See Figure 1 As shown, this invention is a real-time monitoring system for pilot attentional stability based on EEG and eye-tracking data, comprising an EEG and eye-tracking data acquisition module, a neurophysiological data processing module, a neurophysiological feature calibration module, and an attentional stability calculation module, wherein:

[0055] 1. EEG and eye-tracking data acquisition module

[0056] The EEG and eye-tracking data acquisition module connects to the portable EEG and eye tracker worn by the user and collects the EEG and eye-tracking data acquired by the EEG and eye tracker.

[0057] The portable EEG device can be a lightweight EEG acquisition device from Beijing Huixinlian Technology, which primarily collects dual-channel EEG data from the left and right prefrontal cortex. The portable eye tracker can be a glasses-type eye tracker from Tobii, which can collect pupil size data. Note that this invention is not limited to the aforementioned brands and models of instruments. The portable EEG device needs to be able to collect single-channel EEG signals from the left prefrontal cortex, with a sampling rate of 250Hz or higher. The eye tracker needs to be able to collect pupil size data in millimeters, with a sampling rate of 30Hz or higher. Both instruments need to support real-time data transmission, i.e., transmitting the collected data to a PC in real time via Bluetooth or Wi-Fi.

[0058] 2. Neurophysiological data processing module

[0059] The neurophysiological data processing module is used to filter, remove artifacts, calibrate, and extract features from the collected EEG data to obtain EEG data features; it also extracts and calibrates the number of abnormal pupil size changes from the eye movement data to obtain eye movement data features; specifically, it uses EEG baseline feature parameters to calibrate the artifact-removed EEG data, and uses eye movement baseline feature parameters to calibrate the features extracted from the eye movement data.

[0060] The neurophysiological data processing module processes EEG and eye-tracking data in a sliding window with 2-second intervals.

[0061] 2.1 Processing of EEG Data

[0062] 2.1.1 Filtering of EEG data

[0063] For each 2-second segment of real-time acquired EEG data, filtering is first performed. The filtering method is a 4th-order Butterworth filter to obtain signals from 0.5Hz to 100Hz, in order to remove slow signals and high-frequency noise, thereby obtaining the original filtered EEG signal.

[0064] 2.1.2 Artifact Removal Processing of EEG Data

[0065] After filtering, artifact removal is performed: the main artifacts in EEG signals are caused by blinking. The artifact removal process is as follows:

[0066] First, wavelet packet analysis was used to identify the time intervals in which artifacts were located in the original EEG signal:

[0067] Wavelet packet analysis using Daubechies 4 as the mother wavelet decomposes the raw EEG signal into 7 layers. Wavelet packet coefficients for layers 1-4 are set to 0, while soft-threshold filtering is applied to the wavelet packet coefficients for layers 5-7. Specifically, the set of wavelet packet coefficients for each layer from 5-7 is extracted, and a threshold is calculated using the soft-threshold formula. Wavelet packet coefficients exceeding the soft threshold are then set to 0, while others remain unchanged. The soft-threshold formula is as follows:

[0068]

[0069] In the above formula, Thres is the soft threshold of wavelet packet coefficients of a certain layer, median() is the median, abs() is the absolute value, and coef is an array consisting of the set of wavelet packet coefficients of a certain layer.

[0070] The reconstructed signal after wavelet packet analysis is a signal containing only artifacts. Among the signal containing only artifacts, points with a variance greater than 3 times are found, and these points are the artifacts that need to be corrected. According to the characteristics of the blink artifact signal, the time interval where the artifacts that need to be corrected are located needs to be extended forward by 0.16 seconds and backward by 0.84 seconds to cover the electrical signal brought by the blink. The 1-second interval obtained in this way is the artifact interval.

[0071] Secondly, signal replacement processing is performed on the artifact regions:

[0072] The filtered original EEG signal is then processed by Savitzky-Golay filtering; the Savitzky-Golay filter is of order 3 and the window width is 41; the data in the artifact interval is replaced with the data of the original EEG signal processed by Savitzky-Golay filtering, thus completing the artifact removal process.

[0073] 2.1.3 Calibration and processing of EEG data

[0074] After artifact removal but before feature extraction, the EEG data needs to be calibrated. This mainly involves obtaining the baseline EEG characteristic parameters of the user in a specific state, namely the resting state, from the neurophysiological feature calibration module, including the average value M of the EEG data. EEG With variance SD EEG The filtered signal is then transformed as follows:

[0075]

[0076] Among them, EEG Ind To calibrate the processed EEG data, Filtered EEG data that has undergone artifact removal processing.

[0077] 2.1.4 Feature Extraction of EEG Data

[0078] After calibration, the following EEG data characteristics were calculated from the EEG signals:

[0079] (a) The average value of the entire signal;

[0080] (b) The variance of the entire signal;

[0081] (c) Skewness of the entire signal;

[0082] (d) Kurtosis of the entire signal;

[0083] (e) The Hjorth mobility of the entire signal, its formula is: Where y(t) is the calibrated EEG signal, and var() is the variance;

[0084] (f) The Hjorth complexity of the entire signal, its formula is: Where y(t) is the calibrated EEG signal, and Mobility() is Hjorth mobility;

[0085] (g) The entropy of the entire signal;

[0086] (h) The Higuchi fractal dimension of the entire signal;

[0087] (i) The intensity of delta wave (0.5-4Hz) based on wavelet packet analysis, that is, the wavelet packet decomposition of the calibrated EEG signal is performed to find the array of wavelet packet coefficients corresponding to the 0.5-4Hz frequency band, and the average value of the array is calculated.

[0088] (j) Skewness of delta wave (0.5-4Hz) based on wavelet packet analysis, that is, performing wavelet packet decomposition on the calibrated EEG signal, finding the array of wavelet packet coefficients corresponding to the 0.5-4Hz frequency band, and calculating the skewness of the array.

[0089] (k) Kurtosis of delta wave (0.5-4Hz) based on wavelet packet analysis, that is, performing wavelet packet decomposition on the calibrated EEG signal, finding the array of wavelet packet coefficients corresponding to the 0.5-4Hz frequency band, and calculating the kurtosis of the array.

[0090] (l) Intensity of theta wave (4-8Hz) based on wavelet packet analysis, referring to the calculation of features (i);

[0091] (m) Skewness of the theta wave (4-8Hz) based on wavelet packet analysis, with reference to the calculation of the (j) feature;

[0092] (n) Kurtosis of the theta wave (4-8Hz) based on wavelet packet analysis, with reference to the calculation of the (k) feature;

[0093] (o) Intensity of alpha wave (8-12Hz) based on wavelet packet analysis, with reference to the calculation of features (i);

[0094] (p) Skewness of alpha wave (8-12Hz) based on wavelet packet analysis, with reference to the calculation of feature (j);

[0095] (q) Kurtosis of the alpha wave (8-12Hz) based on wavelet packet analysis, with reference to the (k) feature calculation;

[0096] (r) Intensity of beta waves (12-30Hz) based on wavelet packet analysis, with reference to the calculation of features (i);

[0097] (s) Skewness of beta wave (12-30Hz) based on wavelet packet analysis, with reference to the calculation of the (j) feature;

[0098] (t) Kuness of beta wave (12-30Hz) based on wavelet packet analysis, with reference to (k) feature calculation;

[0099] (u) Intensity of beta waves (30-100Hz) based on wavelet packet analysis, with reference to the calculation of features (i);

[0100] (v) Skewness of beta waves (30-100Hz) based on wavelet packet analysis, with reference to the calculation of features (j);

[0101] (w) Kuness of beta waves (30-100Hz) based on wavelet packet analysis, with reference to (k) feature calculation.

[0102] 2.1.5 Determination of the number of abnormal pupil size changes in eye-tracking data

[0103] For eye-tracking data, it is necessary to utilize the pupil size data within the eye-tracking data to extract the feature of the frequency of abnormal pupil size changes.

[0104] For each 2-second segment of real-time acquired pupil size eye movement data, wavelet packet analysis is first performed on the data. Daubechies 8 is used as the mother wavelet, decomposed to the second layer, and a soft thresholding filter is applied to the wavelet packet coefficients of the second layer. The processing method is the same as the soft thresholding filter for EEG data, and will not be described in detail here. The number of non-zero wavelet packet coefficients after filtering is the number of abnormal pupil size changes.

[0105] 2.1.6 Calibration and Feature Extraction of Eye-Tracking Data

[0106] The number of abnormal pupil size changes is subtracted from the average number of abnormal pupil size changes obtained from the neurophysiological feature calibration module when the user is in a specific state, namely the resting state. The calibrated pupil size abnormality change feature is obtained and used as the unique eye movement data feature extracted from the eye movement data.

[0107] Finally, a total of 24 features were extracted from the 2-second EEG and eye movement signals (23 features from EEG data and 1 feature from eye movement data); these 24 features were input into the attention stability calculation module in the form of an array.

[0108] 3. Neurophysiological Characteristic Calibration Module

[0109] The neurophysiological feature calibration module utilizes the EEG and eye-tracking data acquisition module to acquire baseline EEG and eye-tracking data of the user in a specific state. This data is then output to the neurophysiological data processing module for filtering and artifact removal of the EEG baseline data, and for extracting features of abnormal pupil size changes from the eye-tracking baseline data, thus obtaining baseline EEG and eye-tracking feature parameters. This module also records basic user information, including user ID, name, and gender.

[0110] For each user, 30 seconds of EEG and eye movement data need to be collected when the user is in a specific state, namely a resting state, to serve as EEG baseline data and eye movement baseline data, in order to improve the accuracy of attentional stable state recognition.

[0111] For EEG baseline data, the 30-second baseline data needs to be divided into 2-second units (fixed window). After filtering and artifact removal operations in the "Neurophysiological Data Processing Module" mentioned above, the average value m of the data in each window is calculated. EEG With variance sd EEG And take the average value M of all windows EEG With variance SD EEG The data is stored as baseline EEG parameters and input into the "Neurophysiological Data Processing Module" in real time for EEG data calibration.

[0112] For eye-tracking baseline data, the 30-second baseline data needs to be divided into 2-second units (fixed windows). The number of abnormal pupil size changes in each window is calculated using the pupil size abnormal change count determination method in the "Neurophysiological Data Processing Module" mentioned above. The average value of all windows is then calculated, and the average value of the number of abnormal pupil size changes is used as the eye-tracking baseline feature parameter. This value is then input into the "Neurophysiological Data Processing Module" in real time for eye-tracking data calibration.

[0113] 4. Pay attention to the stability calculation module.

[0114] The attention stability calculation module takes the EEG and eye movement data features output by the neurophysiological data processing module as input, and outputs attention stability state labels through a trained attention stability state model.

[0115] In this scheme, the attention stability state model adopts a nonlinear support vector machine model.

[0116] Note that after the stable state model training is complete, it receives EEG data features and eye movement data features output by the "Neurophysiological Data Processing Module", specifically in the format {f1,f2,…,f 24}, where f1-f 24 The model extracts 24 features from EEG and eye-tracking data within a 2-second window; the output of the model is either "normal attention" or "excessive attention".

[0117] Example:

[0118] Note that the training of the stability state model is mainly carried out by collecting data and modeling through experiments on gradual continuous operation tasks; in this embodiment, the modeling data includes the 30 subjects who participated in the experiment.

[0119] The experimental materials for the gradient continuous operation task consisted of two types of circular grayscale images: urban landscapes and mountain scenery (10 images each). Figure 1 The images represent "city" and "mountain." In the experimental task, "city" and "mountain" are presented randomly with a probability of 90% and 10%, respectively, and the same image is not presented repeatedly in adjacent trials. The transition from one image to the next is achieved through a gradual change in transparency, with a transition period of 0.8 seconds between adjacent images. For example, in the first 0.8 seconds, the transparency of the first image gradually changes from 0% (completely clear) to 100% (completely transparent), while the transparency of the second image gradually changes from 100% to 0%. In the next cycle, the transparency of the second image gradually changes from 0% to 100%, while the transparency of the third image gradually changes from 100% to 0%, and so on. Except for the first image in each formal experiment, the transparency of each other image gradually changes from 100% to 0% and then back to 100%, spanning two cycles, for a total of 1.6 seconds. The task requires the participants to press the space bar when they see "city" (90%) and not to press any key when they see "mountain" (10%).

[0120] Upon arrival at the laboratory, participants were required to read and sign an informed consent form. Before the experiment officially began, participants sat in front of the screen, adjusting the chair height and chin rest (to stabilize the chin and maintain the required distance between the eyes and the center of the screen), followed by eye calibration. After calibration, experimental instructions were presented, requiring participants to understand the experimental procedure and familiarize themselves with all 20 images used in the experiment (labeled "city" and "mountain"). Participants then had two minutes to practice and master the task requirements before the experiment officially began. The experiment consisted of three formal sets, each lasting eight minutes, with a two-minute break between sets.

[0121] The participants' attentional states were calculated from their reaction times during the task. The dataset contained reaction time data with an average value every 0.8 seconds, corresponding to the presentation duration of each image. The reaction time data was processed as follows: First, the time-varying variability (TVA) was calculated from the reaction time data. TVA is an indicator of reaction time variability. For each participant, all their reaction times were converted into corresponding z-scores (i.e., standard scores), and the absolute deviation of each trial from the overall average reaction time is the TVA. When classifying different attentional states, the TVA time series within each experimental group was first smoothed using a Gaussian kernel. The 75th percentile of the smoothed TVA (within each group) was used as the dividing line; states below the 75th percentile were classified as "normal attention," and states above the 75th percentile were classified as "excessive attention." Thus, each 0.8-second time window in the dataset corresponds to a label for an attentional state.

[0122] When constructing the training dataset, since the data labels are marked in 0.8-second increments, while EEG and eye-tracking feature extraction is performed in 2-second increments, the following processing is required:

[0123] First, EEG and eye movement feature extraction is performed with a 2-second moving window and a 0.5-second step size. Within each window, the aforementioned 24 features are extracted. The data for these 24 features, along with the start and end times of the corresponding moving window, are recorded as a single row of data: {f1, f2, ..., f...} 24 ,ts,te}, where f1-f 24The 24 extracted features are denoted as ts, where ts is the start time of the time window and te is the end time. Then, all attention state labels within the time window are examined. If all labels within the window are "Normal Attention," then that row of data becomes a sample with the label "Normal Attention." If all labels within the window are "Excessive Attention," then that row of data becomes a sample with the label "Excessive Attention." If the window contains both "Normal Attention" and "Excessive Attention," then the data within the window is discarded. Finally, the sample format used for training is {f1, f2, ..., f...} 24 ,label}, where f1-f 24 The 24 features are extracted, and the label is the tag.

[0124] After the system is established according to this plan, in practical application, for a driver to be monitored, the following steps are required:

[0125] Step 1: Turn on the portable EEG and eye tracker, confirm that the EEG and eye tracker are working properly, and have the driver put on the EEG and eye tracker.

[0126] Step 2: If this is the driver's first time being monitored, calibration data needs to be collected; otherwise, proceed to Step 3. The specific steps for collecting calibration data are as follows:

[0127] Enter the driver's ID, name, gender, and other basic information; instruct the driver to maintain a stable sitting posture, relax their body, and keep their eyes focused on the display screen in the cockpit. Once the driver is in a resting state, enter the calibration data acquisition mode. Use the neurophysiological characteristic calibration module, EEG and eye movement data acquisition module to start collecting 30 seconds of EEG baseline data and eye movement baseline data. Then, combine the neurophysiological data processing module to calculate the EEG baseline characteristic parameters and eye movement baseline characteristic parameters as calibration data. After the calibration data acquisition is completed, the system enters standby mode and proceeds to step 4.

[0128] Step 3: If the driver has already had calibration data collected, enter the driver's number in the system to retrieve the driver's calibration data; if recalibration is required, follow the steps in Step 2 to collect the data again; otherwise, proceed to Step 4.

[0129] Step 4: Begin monitoring stability in the system:

[0130] After the detection begins, the EEG and eye-tracking data acquisition module will continuously receive EEG and eye-tracking data from the EEG and eye tracker, and send them to the neurophysiological data processing module. The neurophysiological data processing module will use a 2-second moving window and combine it with the driver's calibration data from the neurophysiological feature calibration module to filter, remove artifacts, calibrate, and extract features from the EEG data. It will also extract the number of abnormal pupil size changes from the eye-tracking data and perform calibration. Finally, it will extract 24 EEG and eye-tracking data features from the 2-second data and output them to the attention stability calculation module. The attention stability calculation module will then output two attention status labels: "normal attention" or "excessive attention" based on the EEG and eye-tracking data features.

[0131] It is worth noting that the various units included in the above system embodiments are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of the present invention.

[0132] Furthermore, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the corresponding program can be stored in a computer-readable storage medium, such as ROM / RAM, disk, or optical disk.

[0133] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A real-time monitoring system for pilot attentional stability based on electroencephalogram (EEG) and eye-tracking data, characterized in that, The pilot's attentional stability includes normal attention and excessive attentional concentration; the real-time monitoring system includes an EEG and eye-tracking data acquisition module, a neurophysiological data processing module, a neurophysiological characteristic calibration module, and an attentional stability calculation module, wherein: The EEG and eye-tracking data acquisition module connects to the portable EEG and eye tracker worn by the user and collects the EEG and eye-tracking data acquired by the EEG and eye tracker. The neurophysiological data processing module is used to filter, remove artifacts, calibrate, and extract features from the collected EEG data to obtain EEG data features; it also extracts and calibrates the number of abnormal pupil size changes from the eye movement data to obtain eye movement data features; specifically, it uses EEG baseline feature parameters to calibrate the artifact-removed EEG data, and uses eye movement baseline feature parameters to calibrate the features extracted from the eye movement data. The neurophysiological feature calibration module uses the EEG and eye movement data acquisition module to acquire the user's EEG baseline data and eye movement baseline data in a specific state, and outputs them to the neurophysiological data processing module to filter and remove artifacts from the EEG baseline data, and extract the number of abnormal pupil size changes from the eye movement baseline data to obtain EEG baseline feature parameters and eye movement baseline feature parameters. The attention stability calculation module takes the EEG data features and eye movement data features output by the neurophysiological data processing module as input, and outputs attention stability state labels through a trained attention stability state model; the attention stability state model adopts a nonlinear support vector machine model. The method for determining the number of abnormal pupil size changes in the eye movement data is as follows: For each segment of real-time acquired eye-tracking data containing pupil size eye-tracking data, firstly, wavelet packet analysis is performed on the data. Daubechies 8 is used as the mother wavelet, decomposed to the second layer, and soft threshold filtering is applied to the wavelet packet coefficients of the second layer. The number of non-zero wavelet packet coefficients after filtering is the number of abnormal pupil size changes. The calibration and feature extraction process for the eye-tracking data is as follows: The number of abnormal pupil size changes is subtracted from the average number of abnormal pupil size changes obtained from the neurophysiological feature calibration module, which is the baseline eye movement feature parameter of the user in a specific state. The calibrated number of abnormal pupil size changes is obtained as the unique eye movement data feature extracted from the eye movement data.

2. The real-time monitoring system for pilot attentional stability based on EEG and eye-tracking data according to claim 1, characterized in that, The filtering process for the EEG data is as follows: For each segment of real-time acquired EEG data, a fourth-order Butterworth filter is used to obtain signals from 0.5Hz to 100Hz to remove slow signals and high-frequency noise, thereby obtaining the filtered raw EEG signal.

3. The real-time monitoring system for pilot attentional stability based on EEG and eye-tracking data according to claim 1, characterized in that, The artifact removal process for the EEG data is as follows: First, wavelet packet analysis was used to identify the time intervals in which artifacts were located in the filtered raw EEG signal: Wavelet packet analysis using Daubechies 4 as the mother wavelet decomposes the raw EEG signal into 7 layers. Wavelet packet coefficients for layers 1-4 are set to 0, while soft-threshold filtering is applied to the wavelet packet coefficients for layers 5-7. Specifically, the set of wavelet packet coefficients for each layer from 5-7 is extracted, and a threshold is calculated using the soft-threshold formula. Wavelet packet coefficients exceeding the soft threshold are then set to 0, while others remain unchanged. The soft-threshold formula is as follows: In the above formula, Thres The soft threshold for wavelet packet coefficients at a certain level. median () represents the median. abs () represents the absolute value. coef It is an array consisting of the set of wavelet packet coefficients of a certain layer; The reconstructed signal after wavelet packet analysis is the signal containing only artifacts; in the signal containing only artifacts, find points with a variance greater than 3 times, these points are the artifacts that need to be corrected; according to the characteristics of blink artifact signals, the time interval where the artifacts that need to be corrected are located needs to be extended forward and backward by a preset time period to cover the electrical signal brought by blinking; the time interval obtained in this way is the artifact interval. Secondly, signal replacement processing is performed on the artifact regions: The filtered original EEG signal is then processed by Savitzky-Golay filtering; the data in the artifact intervals are replaced with the data of the original EEG signal processed by Savitzky-Golay filtering, thus completing the artifact removal process.

4. The real-time monitoring system for pilot attentional stability based on EEG and eye-tracking data according to claim 1, characterized in that, The calibration process for the EEG data is as follows: The baseline EEG characteristic parameters of the user in a specific state are obtained from the neurophysiological characteristic calibration module, including the average value of the EEG data. M EEG With variance SD EEG The filtered signal is then transformed as follows: in, EEG Ind To calibrate the processed EEG data, EEG Filtered EEG data that has undergone artifact removal processing.

5. The real-time monitoring system for pilot attentional stability based on EEG and eye-tracking data according to claim 1, characterized in that, The characteristics of the EEG data include: The mean of the entire signal; the variance of the entire signal; the skewness of the entire signal; the kurtosis of the entire signal; the Hjorth mobility of the entire signal; the Hjorth complexity of the entire signal; the entropy of the entire signal; the Higuchi fractal dimension of the entire signal; the intensity of the delta wave (0.5-4Hz) based on wavelet packet analysis; the skewness of the delta wave (0.5-4Hz) based on wavelet packet analysis; the kurtosis of the delta wave (0.5-4Hz) based on wavelet packet analysis; the intensity of the theta wave (4-8Hz) based on wavelet packet analysis; the skewness of the theta wave (4-8Hz) based on wavelet packet analysis; the theta wave (4-8Hz) based on wavelet packet analysis. Kurtosis; intensity of alpha wave (8-12Hz) based on wavelet packet analysis; skewness of alpha wave (8-12Hz) based on wavelet packet analysis; kurtosis of alpha wave (8-12Hz) based on wavelet packet analysis; intensity of beta wave (12-30Hz) based on wavelet packet analysis; skewness of beta wave (12-30Hz) based on wavelet packet analysis; kurtosis of beta wave (12-30Hz) based on wavelet packet analysis; intensity of beta wave (30-100Hz) based on wavelet packet analysis; skewness of beta wave (30-100Hz) based on wavelet packet analysis; kurtosis of beta wave (30-100Hz) based on wavelet packet analysis.

6. The real-time monitoring system for pilot attentional stability based on EEG and eye-tracking data according to claim 1, characterized in that, For each user, EEG and eye movement data need to be collected for T seconds while the user is in a resting state as EEG baseline data and eye movement baseline data; For baseline EEG data, using t seconds as a window, the average value m of the data in each window is calculated after filtering and artifact removal operations in the neurophysiological data processing module. EEG With variance sd EEG And take the average value M of all windows EEG With variance SD EEG Stored as baseline characteristics of EEG; For eye-tracking baseline data, with t seconds as a window, the number of abnormal pupil size changes in each window is calculated using the pupil size abnormality number determination method in the neurophysiological data processing module, and the average value of all windows is calculated. The average value of the number of abnormal pupil size changes is used as the eye-tracking baseline feature parameter.

7. A method for real-time monitoring of pilot attentional stability based on electroencephalogram (EEG) and eye-tracking data, characterized in that, The real-time monitoring method is performed using the pilot attention stability real-time monitoring system based on EEG and eye-tracking data as described in any one of claims 1 to 6, comprising: Step 1: Turn on the portable EEG and eye tracker, confirm that the EEG and eye tracker are working properly, and have the driver put on the EEG and eye tracker. Step 2: If this is the driver's first time being monitored, calibration data needs to be collected; otherwise, proceed to Step 3. The specific steps for collecting calibration data are as follows: Enter the driver's ID, name, gender, and other basic information; instruct the driver to maintain a stable sitting posture, relax their body, and keep their eyes focused on the display screen in the cockpit. Once the driver is in a resting state, enter the calibration data acquisition mode. Utilize the neurophysiological characteristic calibration module, EEG and eye movement data acquisition module to begin acquiring T-second baseline EEG and eye movement data. Then, combine the neurophysiological data processing module to calculate the baseline characteristic parameters of the EEG and eye movement as calibration data. After the calibration data acquisition is completed, the system enters standby mode and proceeds to step 4. Step 3: If the driver has already had calibration data collected, enter the driver's number in the system to retrieve the driver's calibration data; if recalibration is required, follow the steps in Step 2 to collect the data again; otherwise, proceed to Step 4. Step 4: Begin monitoring stability in the system: After the detection begins, the EEG and eye-tracking data acquisition module will continuously receive EEG and eye-tracking data from the EEG and eye tracker, and send them to the neurophysiological data processing module. The neurophysiological data processing module will use t seconds as the moving window, and combine the driver's calibration data from the neurophysiological feature calibration module to filter, remove artifacts, calibrate, and extract features from the EEG data. It will also extract the number of abnormal pupil size changes from the eye-tracking data and perform calibration. Finally, it will extract the EEG data features and eye-tracking data features from the t seconds of data and output them to the attention stability calculation module. The attention stability calculation module will output two attention state labels: "normal attention" or "excessive attention" based on the EEG data features and eye-tracking data features.

Images