Eye tracking system, method and device integrating EEG signals and non-contact eye tracker

Abstract

The present invention belongs to the field of eye tracking, and specifically relates to an eye tracking system, method and device that integrates EEG signals and non-contact eye trackers. It aims to solve the problems of limited acquisition range of existing non-contact eye trackers and low spatial resolution of EEG signals inferring eye movement behavior. The present invention includes: synchronously acquiring eye movement data and EEG signals, and after calibration mapping and timing alignment preprocessing, using eye movement fixation point labels to supervise the training of EEG modal behavior recognition models to establish a correlation between the two modalities. A dynamic fusion strategy is adopted in the online stage. When the eye movement data is valid, the model prediction results are fused with the eye movement data to output the fixation point. When the eye movement fails, the fixation point is calculated by combining the model prediction with historical data, thereby realizing enhanced compensation of EEG signals for eye tracking, effectively making up for the defect of low spatial resolution of a single EEG signal, and improving the robustness and tracking accuracy of the system in complex scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of eye tracking, and in particular relates to an eye tracking system, method and device that integrates electroencephalogram (EEG) signals with a non-contact eye tracker. Background Art

[0002] Non-contact eye tracking technology uses sensors such as cameras to capture eye movements remotely, eliminating the need for physical equipment on the subject. It can accurately capture visual attention information, such as gaze position and saccade trajectory, and offers advantages such as intuitiveness and high spatial resolution. However, traditional eye trackers also have limitations: First, their acquisition relies on the visibility and stable posture of the subject's eyes, and tracking may fail in strong light, under occlusion, or when the line of sight strays from the camera's effective range.

[0003] Electroencephalogram (EEG) signals record changes in electrical activity in the cerebral cortex. EEG signals often contain changes in electrical signals caused by eye movements, and can be used to infer eye movement behavior to a certain extent. When visual conditions are poor or the eye tracker fails, EEG can still be collected continuously, regardless of whether the line of sight is seen by the camera. EEG itself also has shortcomings: low spatial resolution, making it difficult to accurately decode the gaze point directly from brain signals. Past studies have shown that the mean square error of eye tracking based solely on EEG is still several centimeters higher than that of camera eye trackers, making it difficult to achieve practical accuracy. In addition, EEG signals are noisy and have significant individual differences, and model training requires a large amount of data correction. This makes it challenging to rely solely on EEG to locate the gaze.

[0004] Based on this, the present invention proposes an eye tracking system, method and device that integrates EEG signals with a non-contact eye tracker. Summary of the Invention

[0005] In order to solve the above-mentioned problems in the prior art, namely, the limited acquisition range of the prior art non-contact eye tracker and the low spatial resolution of eye movement behavior inferred from EEG signals, the present invention provides an eye tracking system, method and device that integrates EEG signals with a non-contact eye tracker.

[0006] In a first aspect, the present invention provides an eye tracking system that integrates EEG signals with a non-contact eye tracker. The system comprises:

[0007] Data acquisition module, used to synchronously collect eye movement data and EEG signals from a non-contact eye tracker;

[0008] A calibration and preprocessing module calibrates and maps the eye movement data to the screen coordinate system, and performs filtering, denoising, and time synchronization preprocessing on the EEG signal to generate time-aligned multimodal data;

[0009] An offline model training module, based on the multimodal data and using the gaze point labels provided by the eye tracker as supervision signals, trains a pre-built EEG modality eye movement behavior recognition model; the EEG modality eye movement behavior recognition model is used to learn the association between EEG signals and eye movement data;

[0010] An online multimodal fusion and inference module is used to obtain eye movement data and EEG signals in real time and determine whether the calibrated eye movement data is valid;

[0011] If it is valid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain a prediction result, and the prediction result and the calibrated eye movement data are fused using a strategy to output the fused gaze point;

[0012] If it is invalid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain the prediction result, and the fixation point is calculated in combination with the historical data.

[0013] Furthermore, the eye movement behavior recognition model of the EEG modality includes:

[0014] The temporal convolutional network layer is used to extract local temporal features from the preprocessed EEG signals. Through causal convolution and dilated convolution operations, it captures the transient EEG waveform changes caused by eye movement behavior, aligns them with the eye movement data in the time window, and establishes a preliminary association between the EEG signals and the eye movement data.

[0015] The bidirectional long short-term memory network layer receives the temporal features output by the temporal convolutional network layer and uses forward and backward LSTM units to model the historical data and future predicted data of the EEG signal on the time axis, respectively, to generate a feature sequence containing temporal context information; the feature sequence is associated with the saccade direction and gaze duration information in the eye movement label;

[0016] An attention mechanism layer is used to perform weighted aggregation on the feature sequence output by the bidirectional long short-term memory network layer, calculate the contribution weight of the features at each time point to the current eye movement behavior classification or gaze point prediction, and focus on the EEG waveform segments corresponding to eye movement events; wherein the eye movement events include saccade initiation and blinking;

[0017] A multimodal feature fusion layer, which obtains the eye movement data and extracts the eye movement feature vector, and merges it with the weighted aggregated feature vector output by the attention mechanism layer in the fusion layer to obtain a fused feature vector;

[0018] The output layer outputs the probability distribution of eye movement behavior categories or the regression value of the gaze point coordinates based on the fused feature vector, and the supervision signal of the output layer is directly generated by the label provided by the eye tracker, including at least one of the gaze point position, the saccade direction and the amplitude.

[0019] Furthermore, the EEG modality eye movement behavior recognition model also includes an online adaptation module, which updates model parameters using newly added data through incremental fine-tuning and few-sample transfer learning.

[0020] Furthermore, the method for determining whether the eye movement data is valid is as follows:

[0021] When the number of frames in which the eye tracker fails to detect the pupil exceeds a threshold, or the eye tracker confidence index is lower than a preset value, the eye movement data is deemed invalid.

[0022] Furthermore, the gaze point is calculated by combining historical data. The specific calculation method is:

[0023] The eye movement behavior type, direction, and amplitude parameters predicted by the EEG modality eye movement behavior recognition model are used to obtain the new fixation point by superimposing the movement increments with the last valid eye tracker fixation point as the benchmark.

[0024] During the period of continuous failure, recursive forward calculation is used, and the gaze point predicted by the eye movement behavior recognition model of the previous EEG modality is used as the current calculation benchmark, and the position is updated in combination with the movement increment predicted by the eye movement behavior recognition model of the EEG modality.

[0025] Furthermore, the system determines whether the user is in an effective visual attention state by analyzing attention-related features in the EEG signal; when distraction or eye closure is detected, the gaze point prediction update is suspended.

[0026] Furthermore, when the eye movement data recovers from failure, a transition fusion strategy is adopted:

[0027] In the initial recovery phase, the average of the eye movement data obtained by the eye tracker and the results predicted by the EEG eye movement behavior recognition model is used as the output;

[0028] Within the set time window, gradually increase the weight of the eye tracker data until the tracking mode based on the eye tracker is fully restored.

[0029] Furthermore, the fusion strategy includes residual fusion, and the fusion method is:

[0030] Superimposing the gaze point offset predicted by the eye movement behavior recognition model of the EEG modality with the gaze point coordinates corresponding to the eye movement data to obtain a final gaze point;

[0031] The offset is generated by the eye movement behavior recognition model of the EEG modality according to the eye movement direction and amplitude predicted by the current EEG sequence.

[0032] In a second aspect, the present invention provides an eye tracking method that integrates EEG signals and a non-contact eye tracker. The method is based on an eye tracking system that integrates EEG signals and a non-contact eye tracker, and includes:

[0033] Synchronously collect eye movement data and EEG signals from a non-contact eye tracker;

[0034] Calibrate and map the eye movement data to a screen coordinate system, and perform filtering, denoising, and time synchronization preprocessing on the EEG signal to generate time-aligned multimodal data;

[0035] Based on the multimodal data, a pre-built EEG modality eye movement behavior recognition model is trained using the gaze point labels provided by the eye tracker as a supervisory signal; the EEG modality eye movement behavior recognition model is used to learn the association between the EEG signal and the eye movement data;

[0036] Acquire eye movement data and EEG signals in real time; determine whether the calibrated eye movement data is valid;

[0037] If it is valid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain a prediction result, and the prediction result and the calibrated eye movement data are fused using a strategy to output the fused gaze point;

[0038] If it is invalid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain the prediction result, and the fixation point is calculated in combination with the historical data.

[0039] A third aspect of the present invention provides an electronic device, comprising:

[0040] at least one processor; and

[0041] a memory communicatively connected to at least one of the processors; wherein,

[0042] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned eye tracking method that integrates EEG signals and a non-contact eye tracker.

[0043] Beneficial effects of the present invention:

[0044] Improve tracking robustness and continuity: Through the complementary fusion of EEG signals and eye tracker data, when the eye tracker fails due to occlusion, strong light or line of sight deviation, EEG signals are used to maintain the continuity of eye tracking, significantly reducing the risk of tracking interruption caused by environmental or posture interference, and adapting to the needs of complex scenarios.

[0045] Enhanced spatial resolution and accuracy: Using high-precision eye tracker data as a supervisory signal to train the EEG model, combined with an online multimodal fusion strategy, effectively compensates for the low spatial resolution of a single EEG signal and improves the accuracy of gaze point positioning, especially achieving more precise gaze point inference within the effective range of the eye tracker.

[0046] Reduce individual differences and data dependence: Through joint modeling of multimodal data with time alignment, and using eye tracker calibration information to optimize the feature expression of EEG signals, the problems of high EEG signal noise and significant individual differences are alleviated, the model's dependence on a large amount of annotated data from a single modality is reduced, and generalization ability is improved.

[0047] Dynamic adaptive scene switching: The online module dynamically selects fusion or pure EEG reasoning mode based on the validity of eye movement data. This not only takes advantage of the high precision of the eye tracker, but also seamlessly switches to EEG-dominated tracking mode when it fails, ensuring the flexibility and reliability of the system in different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0049] Figure 1 This is a schematic diagram of the working process of an eye tracking system that integrates EEG signals and a non-contact eye tracker according to the present invention;

[0050] Figure 2 This is a schematic diagram of the training process of an eye movement behavior recognition model of EEG modality in an eye tracking system that integrates EEG signals and a non-contact eye tracker of the present invention. DETAILED DESCRIPTION

[0051] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0052] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0053] The first embodiment of the present invention provides an eye tracking system that integrates EEG signals with a non-contact eye tracker. The system includes:

[0054] Data acquisition module, used to synchronously collect eye movement data and EEG signals from a non-contact eye tracker;

[0055] A calibration and preprocessing module calibrates and maps the eye movement data to the screen coordinate system, and performs filtering, denoising, and time synchronization preprocessing on the EEG signal to generate time-aligned multimodal data;

[0056] An offline model training module, based on the multimodal data and using the gaze point labels provided by the eye tracker as supervision signals, trains a pre-built EEG modality eye movement behavior recognition model; the EEG modality eye movement behavior recognition model is used to learn the association between EEG signals and eye movement data;

[0057] An online multimodal fusion and inference module is used to obtain eye movement data and EEG signals in real time and determine whether the calibrated eye movement data is valid;

[0058] If it is valid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain a prediction result, and the prediction result and the calibrated eye movement data are fused using a strategy to output the fused gaze point;

[0059] If it is invalid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain the prediction result, and the fixation point is calculated in combination with the historical data.

[0060] In order to more clearly explain the eye tracking system of the present invention that integrates EEG signals and non-contact eye tracker, Figure 1 and Figure 2 Each module in the embodiment of the present invention is described in detail as follows:

[0061] Data acquisition module, used to synchronously collect eye movement data and EEG signals from a non-contact eye tracker;

[0062] In this embodiment, the EEG signal is acquired based on an EEG acquisition device.

[0063] The eye movement data is specifically the user's gaze point trajectory.

[0064] A calibration and preprocessing module calibrates and maps the eye movement data to the screen coordinate system, and performs filtering, denoising, and time synchronization preprocessing on the EEG signal to generate time-aligned multimodal data;

[0065] The pre-processing step also includes artifact correction.

[0066] An offline model training module, based on the multimodal data and using the gaze point labels provided by the eye tracker as supervision signals, trains a pre-built EEG modality eye movement behavior recognition model; the EEG modality eye movement behavior recognition model is used to learn the association between EEG signals and eye movement data;

[0067] In this embodiment, the EEG modality eye movement behavior recognition model includes:

[0068] The temporal convolutional network layer is used to extract local temporal features from the preprocessed EEG signals. Through causal convolution and dilated convolution operations, it captures the transient EEG waveform changes caused by eye movement behavior, aligns them with the eye movement data in the time window, and establishes a preliminary association between the EEG signals and the eye movement data.

[0069] The bidirectional long short-term memory network layer receives the temporal features output by the temporal convolutional network layer and uses forward and backward LSTM units to model the historical data and future predicted data of the EEG signal on the time axis, respectively, to generate a feature sequence containing temporal context information; the feature sequence is associated with the saccade direction and gaze duration information in the eye movement label;

[0070] An attention mechanism layer is used to perform weighted aggregation on the feature sequence output by the bidirectional long short-term memory network layer, calculate the contribution weight of the features at each time point to the current eye movement behavior classification or gaze point prediction, and focus on the EEG waveform segments corresponding to eye movement events; wherein the eye movement events include saccade initiation and blinking;

[0071] A multimodal feature fusion layer, which obtains the eye movement data and extracts the eye movement feature vector, and merges it with the weighted aggregated feature vector output by the attention mechanism layer in the fusion layer to obtain a fused feature vector;

[0072] The output layer outputs the probability distribution of eye movement behavior categories or the regression value of the gaze point coordinates based on the fused feature vector, and the supervision signal of the output layer is directly generated by the label provided by the eye tracker, including at least one of the gaze point position, the saccade direction and the amplitude.

[0073] The online adaptation module updates model parameters using newly added data through incremental fine-tuning and few-shot transfer learning.

[0074] Specifically, the Temporal Convolutional Network (TCN) layer is responsible for extracting local temporal pattern features from the raw EEG time series. The TCN consists of multiple layers of one-dimensional convolutions, using causal convolutions to ensure that convolutions do not leak future information, while using dilated convolutions to expand the receptive field. This structure enables efficient and parallel processing of long sequences while avoiding the vanishing / exploding gradient issues common in RNNs. The TCN excels at capturing short-term transient features in EEG signals (such as sudden changes in EEG waveforms due to eye movements) and offers faster computation and more stable training than traditional LSTMs.

[0075] Bidirectional LSTM layer: A bidirectional long short-term memory (LSTM) network is connected in series on top of the feature sequence extracted by the TCN. The forward LSTM reads the sequence to capture past dependencies, while the backward LSTM reads it backward to capture future dependencies. The hidden states in both directions are then concatenated. For eye movement recognition, certain patterns (such as saccade initiation and termination) may echo each other in the EEG. This bidirectional structure helps to more fully capture these patterns. The LSTM's LSTM units can also retain information over longer timescales, complementing the local features of the TCN.

[0076] Attention Mechanism: An attention layer is added to the Bi-LSTM output sequence to calculate the weight of the hidden state's relevance to the final task at each moment. This approach employs a self-attention approach: first, the similarity score between the hidden state and a learnable query vector is calculated. A softmax is then applied to obtain normalized weights, and the weighted sum is then used to obtain the context vector for the entire sequence. This mechanism enables the model to "focus" on key time points or feature dimensions (such as a significant eye movement-related EEG waveform change), giving them greater influence in decision-making. The attention mechanism effectively enhances the model's ability to model time series data and has been shown to improve accuracy and interpretability in multiple EEG classification tasks.

[0077] Multimodal feature fusion: To fuse EEG and eye movement features, this paper employs a dual-stream design within the network architecture. The EEG sequence is encoded into an EEG feature vector via the aforementioned TCN, BiLSTM, and attention matrix. Simultaneously, eye tracker data (such as gaze coordinates and movement speed at recent moments) is fed into another neural network branch (which can utilize one or two fully connected or convolutional layers for feature extraction) to generate an eye movement feature vector. The two modal features are then combined in a fusion layer. The fused output then passes through several subsequent fully connected layers to predict the final eye movement state classification or gaze point coordinates.

[0078] Output layer: Depending on the task, the output layer can be either a classification layer or a regression layer. For eye movement recognition (e.g., determining whether a saccade occurs), the output layer is typically a softmax classification layer, outputting a probability distribution over several eye movement categories. For gaze point coordinate regression, the output is either two single values ​​(x and y) or a two-dimensional vector. Cross-entropy loss or mean squared error loss is used during training, respectively.

[0079] Online adaptation module: In order to ensure that the model maintains its performance as users and environments change in actual applications, the present invention introduces an online learning strategy. The first is incremental fine-tuning: During the operation of the system, new EEG and eye movement data pairs are continuously collected (using the output of the eye tracker as real-time labels), and the model parameters are regularly updated in small batches. Using a smaller learning rate and optimization algorithm (such as Adam) for fine-tuning can gradually correct the model's adaptability to the specific brain signal pattern of the current user without forgetting the original ability. The second is few-sample transfer learning: when a new user or a new scene is changed, the existing model can be loaded as pre-trained parameters, and only a small number of calibration samples (such as a few minutes of synchronized data) can be used for rapid training to adjust the high-level parameters of the model.

[0080] An online multimodal fusion and inference module is used to obtain eye movement data and EEG signals in real time and determine whether the calibrated eye movement data is valid;

[0081] If it is valid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain a prediction result, and the prediction result and the calibrated eye movement data are fused using a strategy to output the fused gaze point;

[0082] If it is invalid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain the prediction result, and the fixation point is calculated in combination with the historical data.

[0083] In this embodiment, the method for determining whether the eye movement data is valid is as follows:

[0084] When the number of frames in which the eye tracker fails to detect the pupil exceeds a threshold, or the eye tracker confidence index is lower than a preset value, the eye movement data is deemed invalid.

[0085] In this embodiment, when the eye movement data is invalid, the gaze point is calculated by combining historical data. The specific calculation method is:

[0086] The eye movement behavior type, direction, and amplitude parameters predicted by the EEG modality eye movement behavior recognition model are used to obtain the new fixation point by superimposing the movement increments with the last valid eye tracker fixation point as the benchmark.

[0087] During the period of continuous failure, recursive forward calculation is used, and the gaze point predicted by the eye movement behavior recognition model of the previous EEG modality is used as the current calculation benchmark, and the position is updated in combination with the movement increment predicted by the eye movement behavior recognition model of the EEG modality.

[0088] The system determines whether the user is in an effective visual attention state by analyzing attention-related features in EEG signals; when attention distraction or eye closure is detected, the gaze point prediction update is suspended.

[0089] When the eye movement data recovers from failure, a transition fusion strategy is adopted:

[0090] In the initial recovery phase, the average of the eye movement data obtained by the eye tracker and the results predicted by the EEG eye movement behavior recognition model is used as the output;

[0091] Within the set time window, gradually increase the weight of the eye tracker data until the tracking mode based on the eye tracker is fully restored.

[0092] Specifically, when the user's gaze exceeds the eye tracker's range, the eyes are blocked, or the ambient light causes the eye tracker to temporarily fail, the system will enable the EEG-based gaze point prediction mechanism to maintain continuous tracking of the user's gaze. First, the system determines whether a failure has occurred by monitoring the validity of the eye tracker output: for example, if the pupil is not detected for several consecutive frames, or the eye tracker confidence index is below a threshold, it is considered to have entered a "transition region" or a failure state. At this point, the fusion module automatically switches to EEG unimodal prediction mode, and the EEG model estimates the current gaze direction or gaze point position.

[0093] The EEG model acts like a virtual eye tracker in this scenario. Because it's difficult to infer precise coordinates directly from EEG, a combination of kinematic constraints and sequence prediction improves reliability: the EEG model first predicts the current eye movement based on the most recent EEG sequence, such as whether a saccade has occurred, its direction, and its magnitude. It then uses the user's last known eye tracker gaze point to deduce a new gaze point prediction. For example, if the EEG model identifies a saccade to the upper right and estimates its magnitude to be approximately 15 degrees of visual angle, the last known gaze position before the transition is used as the starting point to infer the corresponding position of the new gaze point in that direction. This uses the prior position plus the EEG-inferred increment to achieve continuous prediction of the gaze point.

[0094] To cope with longer periods of eye tracker failure, our EEG prediction model can use a recursive method to gradually forward-calculate, that is, each time the new gaze point inferred by the EEG is used as the reference starting point for the next moment, and the position is continuously updated in combination with the EEG sequence until the eye tracker recaptures the target. In this process, it is also necessary to consider that the prediction uncertainty accumulates over time, so the system introduces a decaying trust mechanism when predicting EEG: as the duration of the failure increases, the model should be more conservative about the change in position (for example, assuming that the user may be staring at something still) to avoid prediction drift. On the other hand, the EEG signal itself can be used to detect whether the user's visual attention is still on the current task. If the user is distracted or closes his eyes to rest, the system can temporarily suspend updating the gaze point.

[0095] When the user's eyes return to the eye tracker's field of view, the system needs to smoothly transition back to eye tracker tracking mode. To this end, the present invention incorporates a transitional region fusion process: At the moment the eye tracker redetects the eyes, there may still be errors or jitter in its positioning. At this point, continued averaging and fusion based on EEG predictions can avoid sudden output fluctuations. Subsequently, the system gradually returns to eye tracker-based mode within tens of milliseconds. This mechanism ensures that the system consistently estimates the gaze point regardless of the user's gaze movement, significantly improving robustness.

[0096] Within the normal operating range of the eye tracker, this system uses a data fusion strategy to combine eye tracker and EEG information in order to improve the robustness and accuracy of eye tracking. Specifically, when the eye tracker can reliably output the user's gaze point, we use the eye tracker data as the main source of information. On the one hand, EEG can provide supplementary information about the user's cognitive state and attention, helping to determine the nature of the current eye movement. On the other hand, the EEG model can perform redundancy checks and subtle corrections to the eye tracker output: if the EEG prediction result is significantly inconsistent with the eye tracker, it may indicate that the eye tracker has a momentary frame loss or error. At this time, the system can increase the weight of the EEG and smoothly adjust the output gaze point to avoid jumps.

[0097] Therefore, in this embodiment, when the eye movement data is valid, the coordinates are output based on the fusion strategy, and the specific fusion methods include:

[0098] Superimposing the gaze point offset predicted by the eye movement behavior recognition model of the EEG modality with the gaze point coordinates corresponding to the eye movement data to obtain a final gaze point;

[0099] The offset is generated by the eye movement behavior recognition model of the EEG modality according to the eye movement direction and amplitude predicted by the current EEG sequence.

[0100] Specifically, the predictions from the EEG modality-based eye movement behavior recognition model are treated as a residual correction to the eye tracker output. The final gaze point output = eye tracker coordinates + EEG prediction offset. Under normal circumstances, the EEG only fine-tunes the results. However, when the eye tracker is misaligned, the EEG correction automatically increases to compensate.

[0101] This embodiment also provides two other fusion methods, namely feature-level fusion and decision-level fusion. Specifically:

[0102] Feature-level fusion combines features extracted from EEG signals with features from an eye tracker (such as gaze direction vector and pupil size) early in the EEG eye movement recognition model, allowing for joint decision-making through subsequent networks. This requires temporal synchronization and scale normalization of the two modal data, and the use of specialized fusion layers (such as cross-modal attention) to exploit correlations between EEG and eye movement signals.

[0103] Decision-level fusion involves dynamically weighting the fusion results based on their reliability after obtaining gaze point predictions or eye movement event identifications based on both EEG and eye trackers. Since eye trackers are generally more accurate, the system assigns them a greater weight, while the output of the EEG modality's eye movement behavior recognition model serves as an auxiliary correction factor.

[0104] For example, when the eye tracker signal quality is high, the eye tracker's gaze point is weighted 90% and the EEG modality's eye movement behavior recognition model prediction is weighted 10%. When the eye tracker signal deteriorates, the eye tracker weight is reduced and the EEG weight is increased. These weights can be automatically adjusted based on the confidence index provided by the eye tracker, forming an adaptive weighted fusion.

[0105] For both normal and failure cases, we define a unified multimodal fusion prediction function to describe how the system integrates the information from the two sensors to output the final gaze point. This function dynamically adjusts the strategy based on the availability of eye tracker data:

[0106] When the eye movement data is valid, obtain the gaze point coordinates output by the eye tracker in real time Corrected offset predicted by the eye movement behavior recognition model based on EEG modality ;

[0107] Dynamically calculate the fusion weight based on the confidence index of the current frame of the eye tracker α ,in α The value range of is close to 1 (for example, 0.9≤α≤1), and the higher the confidence level, α The closer it is to 1;

[0108] Perform weighted fusion of the eye tracker coordinates and the EEG-corrected coordinates to generate the final fixation point:

[0109] Calculate corrected eye tracker coordinates p eye( t )+Δ p EEG( t );

[0110] The final output is , among which α =1, completely trust the eye tracker data.

[0111] When eye movement data is invalid, the eye movement behavior recognition model based on EEG modality predicts the gaze point offset based on the current and historical EEG sequences ;

[0112] The last valid eye tracker gaze point coordinates before failure As a benchmark, the offset predicted by the eye movement behavior recognition model of the EEG modality is superimposed to generate the current gaze point .

[0113] The prediction logic of the eye movement behavior recognition model of the EEG modality in the failure state further includes:

[0114] Offset of predictions of eye movement behavior recognition models based on EEG modalities Apply kinematic constraints:

[0115] Based on the assumption of the continuity of the user's eye movement behavior, a low-pass filtering algorithm or a Kalman filtering model is used to Smoothing the instantaneous change amplitude of

[0116] If the offset direction or amplitude of the consecutive multi-frame prediction exceeds the preset physiological range (for example, the single saccade angle does not exceed 30 degrees), it is forced to correct to within a reasonable threshold;

[0117] Output smoothed and constrained offsets for calculation .

[0118] When the eye tracker signal is in the transition region, the dynamic weighted fusion strategy is executed according to the following logic:

[0119] Transition zone determination conditions:

[0120] Real-time monitoring of eye tracker pupil detection results and signal quality indicators;

[0121] If the number of consecutive valid frames of the eye tracker is lower than the threshold, or the signal noise level exceeds the preset threshold, it is determined to have entered the transition area.

[0122] Fusion function execution steps:

[0123] Assign dynamic confidence weights to each frame of eye tracker data w ( t ), whose value range is [0,1] and satisfies:

[0124] When the eye tracker signal quality is high, w ( t ) approaches 1;

[0125] When the signal quality degrades or is interrupted, w ( t ) approaches 0;

[0126] Real-time calculation of the fixation point predicted by the eye movement behavior recognition model based on EEG modality ;

[0127] Output fused gaze points: ;

[0128] Smooth transition logic during the recovery phase:

[0129] When the eye tracker recovers from a failed state, it initializes w ( t )=0;

[0130] Within a set transition time window (e.g. 200 milliseconds), gradually increase the value linearly or exponentially. w ( t ) to 1;

[0131] During this period, the eye movement data is continuously integrated with the prediction results of the eye movement behavior recognition model based on the EEG modality to avoid jumps in the gaze point coordinates.

[0132] The confidence weight of the transition region w ( t ) is calculated comprehensively through the following multi-dimensional conditions:

[0133] Condition 1: Eye tracker pupil detection stability

[0134] Count the number of consecutive frames in which pupils are successfully detected within the last N frames;

[0135] If the number of consecutive frames is lower than the threshold, the w ( t ).

[0136] Condition 2: Eye tracker signal noise level

[0137] Extract the signal-to-noise ratio (SNR) or high-frequency noise energy of the eye tracker raw signal;

[0138] If the SNR is lower than the preset threshold or the noise energy exceeds the threshold, the noise level is reversely mapped to w ( t ).

[0139] Condition 3: Cross-modal short-term consistency check

[0140] Calculate the Euclidean distance error between the eye tracker predicted coordinates and the eye movement behavior recognition model predicted coordinates within the most recent K frames;

[0141] If the error exceeds the allowable range, reduce w ( t ) to improve the weight of the prediction results of the eye movement behavior recognition model.

[0142] Final weight calculation:

[0143] Perform weighted summation or take the minimum value of the above condition output weights to generate w ( t )∈[0,1].

[0144] It should be noted that the eye tracking system that integrates EEG signals and a non-contact eye tracker provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiment can be combined into one module or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not to be regarded as improper limitations of the present invention.

[0145] A second embodiment of the present invention provides an eye tracking method that integrates EEG signals and a non-contact eye tracker. The method is based on an eye tracking system that integrates EEG signals and a non-contact eye tracker. The method includes:

[0146] Synchronously collect eye movement data and EEG signals from a non-contact eye tracker;

[0147] Calibrate and map the eye movement data to a screen coordinate system, and perform filtering, denoising, and time synchronization preprocessing on the EEG signal to generate time-aligned multimodal data;

[0148] Based on the multimodal data, a pre-built EEG modality eye movement behavior recognition model is trained using the gaze point labels provided by the eye tracker as a supervisory signal; the EEG modality eye movement behavior recognition model is used to learn the association between the EEG signal and the eye movement data;

[0149] Acquire eye movement data and EEG signals in real time; determine whether the calibrated eye movement data is valid;

[0150] If it is valid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain a prediction result, and the prediction result and the calibrated eye movement data are fused using a strategy to output the fused gaze point;

[0151] If it is invalid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain the prediction result, and the fixation point is calculated in combination with the historical data.

[0152] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0153] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the above-described method can refer to the corresponding process in the aforementioned system embodiment and will not be repeated here.

[0154] An electronic device according to a third embodiment of the present invention includes:

[0155] at least one processor; and

[0156] a memory communicatively connected to at least one of the processors; wherein,

[0157] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned eye tracking method that integrates EEG signals and a non-contact eye tracker.

[0158] A fourth embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned eye tracking method that integrates EEG signals and a non-contact eye tracker.

[0159] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0160] Those skilled in the art should be able to appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0161] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0162] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0163] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. An eye tracking system that integrates EEG signals and a non-contact eye tracker, characterized in that: The system includes: Data acquisition module, used to synchronously collect eye movement data and EEG signals from a non-contact eye tracker; A calibration and preprocessing module calibrates and maps the eye movement data to the screen coordinate system, and performs filtering, denoising, and time synchronization preprocessing on the EEG signal to generate time-aligned multimodal data; An offline model training module, based on the multimodal data and using the gaze point labels provided by the eye tracker as supervision signals, trains a pre-built EEG modality eye movement behavior recognition model; the EEG modality eye movement behavior recognition model is used to learn the association between EEG signals and eye movement data; The eye movement behavior recognition model of the EEG modality includes: The temporal convolutional network layer is used to extract local temporal features from the preprocessed EEG signals. Through causal convolution and dilated convolution operations, it captures the transient EEG waveform changes caused by eye movement behavior, aligns them with the eye movement data in the time window, and establishes a preliminary association between the EEG signals and the eye movement data. The bidirectional long short-term memory network layer receives the temporal features output by the temporal convolutional network layer and uses forward and backward LSTM units to model the historical data and future predicted data of the EEG signal on the time axis, respectively, to generate a feature sequence containing temporal context information; the feature sequence is associated with the saccade direction and gaze duration information in the eye movement label; An attention mechanism layer is used to perform weighted aggregation on the feature sequence output by the bidirectional long short-term memory network layer, calculate the contribution weight of the features at each time point to the current eye movement behavior classification or gaze point prediction, and focus on the EEG waveform segments corresponding to eye movement events; wherein the eye movement events include saccade initiation and blinking; A multimodal feature fusion layer, which obtains the eye movement data and extracts the eye movement feature vector, and merges it with the weighted aggregated feature vector output by the attention mechanism layer in the fusion layer to obtain a fused feature vector; An output layer, which outputs a probability distribution of eye movement behavior categories or a regression value of gaze point coordinates based on the fused feature vector, wherein the supervision signal of the output layer is directly generated by the label provided by the eye tracker, and includes at least one of the gaze point position, saccade direction, and amplitude; An online multimodal fusion and inference module is used to obtain eye movement data and EEG signals in real time and determine whether the calibrated eye movement data is valid; If it is valid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain a prediction result, and the prediction result and the calibrated eye movement data are fused using a strategy to output the fused gaze point; If it is invalid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain the prediction result, and the fixation point is calculated in combination with the historical data.

2. The eye tracking system integrating EEG signals and non-contact eye tracker according to claim 1, characterized in that: The EEG modality eye movement behavior recognition model also includes an online adaptation module, which updates model parameters using newly added data through incremental fine-tuning and few-sample transfer learning.

3. The eye tracking system integrating EEG signals and non-contact eye tracker according to claim 1, characterized in that: Whether the eye movement data is valid is determined by: When the number of frames in which the eye tracker fails to detect the pupil exceeds a threshold, or the eye tracker confidence index is lower than a preset value, the eye movement data is deemed invalid.

4. The eye tracking system integrating EEG signals and non-contact eye tracker according to claim 1, characterized in that: The gaze point is calculated by combining historical data. The specific calculation method is: The eye movement behavior type, direction, and amplitude parameters predicted by the EEG modality eye movement behavior recognition model are used to obtain the new fixation point by superimposing the movement increments with the last valid eye tracker fixation point as the benchmark. During the period of continuous failure, recursive forward calculation is used, and the gaze point predicted by the eye movement behavior recognition model of the previous EEG modality is used as the current calculation benchmark, and the position is updated in combination with the movement increment predicted by the eye movement behavior recognition model of the EEG modality.

5. The eye tracking system integrating EEG signals and non-contact eye tracker according to claim 4, characterized in that: The system determines whether the user is in an effective visual attention state by analyzing attention-related features in EEG signals; when attention distraction or eye closure is detected, the gaze point prediction update is suspended.

6. The eye tracking system integrating EEG signals and non-contact eye tracker according to claim 1, characterized in that: When the eye movement data recovers from failure, a transition fusion strategy is adopted: In the initial recovery phase, the average of the eye movement data obtained by the eye tracker and the results predicted by the EEG eye movement behavior recognition model is used as the output; Within the set time window, gradually increase the weight of the eye tracker data until the tracking mode based on the eye tracker is fully restored.

7. The eye tracking system integrating EEG signals and non-contact eye tracker according to claim 1, characterized in that: The fusion strategy includes residual fusion, and the fusion method is: Superimposing the gaze point offset predicted by the eye movement behavior recognition model of the EEG modality with the gaze point coordinates corresponding to the eye movement data to obtain a final gaze point; The offset is generated by the eye movement behavior recognition model of the EEG modality according to the eye movement direction and amplitude predicted by the current EEG sequence.

8. An eye tracking method integrating EEG signals and a non-contact eye tracker, based on the eye tracking system integrating EEG signals and a non-contact eye tracker according to any one of claims 1 to 7, characterized in that: The method includes: Synchronously collect eye movement data and EEG signals from a non-contact eye tracker; Calibrate and map the eye movement data to a screen coordinate system, and perform filtering, denoising, and time synchronization preprocessing on the EEG signal to generate time-aligned multimodal data; Based on the multimodal data, a pre-built EEG modality eye movement behavior recognition model is trained using the gaze point labels provided by the eye tracker as a supervisory signal; the EEG modality eye movement behavior recognition model is used to learn the association between the EEG signal and the eye movement data; The eye movement behavior recognition model of the EEG modality includes: The temporal convolutional network layer is used to extract local temporal features from the preprocessed EEG signals. Through causal convolution and dilated convolution operations, it captures the transient EEG waveform changes caused by eye movement behavior, aligns them with the eye movement data in the time window, and establishes a preliminary association between the EEG signals and the eye movement data. The bidirectional long short-term memory network layer receives the temporal features output by the temporal convolutional network layer and uses forward and backward LSTM units to model the historical data and future predicted data of the EEG signal on the time axis, respectively, to generate a feature sequence containing temporal context information; the feature sequence is associated with the saccade direction and gaze duration information in the eye movement label; An attention mechanism layer is used to perform weighted aggregation on the feature sequence output by the bidirectional long short-term memory network layer, calculate the contribution weight of the features at each time point to the current eye movement behavior classification or gaze point prediction, and focus on the EEG waveform segments corresponding to eye movement events; wherein the eye movement events include saccade initiation and blinking; A multimodal feature fusion layer, which obtains the eye movement data and extracts the eye movement feature vector, and merges it with the weighted aggregated feature vector output by the attention mechanism layer in the fusion layer to obtain a fused feature vector; An output layer, which outputs a probability distribution of eye movement behavior categories or a regression value of gaze point coordinates based on the fused feature vector, wherein the supervision signal of the output layer is directly generated by the label provided by the eye tracker, and includes at least one of the gaze point position, saccade direction, and amplitude; Acquire eye movement data and EEG signals in real time; determine whether the calibrated eye movement data is valid; If it is valid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain a prediction result, and the prediction result and the calibrated eye movement data are fused using a strategy to output the fused gaze point; If it is invalid, the preprocessed EEG signal is input into the EEG modality eye movement behavior recognition model to obtain the prediction result, and the fixation point is calculated in combination with the historical data.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the eye tracking method of claim 8 that integrates electroencephalogram signals and a non-contact eye tracker.

Images