Eye movement tracking system, method and equipment integrating electroencephalogram signals and non-contact eye tracker

Abstract

The invention belongs to the field of eye movement tracking, and particularly relates to an eye movement tracking system, method and equipment integrating electroencephalogram signals and a non-contact eye tracker. The objective of the invention is to solve the problems of limited acquisition range of a non-contact eye tracker and low spatial resolution of eye movement behavior deduction by EEG signals in the prior art. The method comprises the steps that eye movement data and electroencephalogram signals are synchronously collected, after calibration mapping and time sequence alignment preprocessing are conducted, an electroencephalogram modal behavior recognition model is supervised and trained through eye movement fixation point labels, and the incidence relation of the two modals is established. A dynamic fusion strategy is adopted in the online stage, when eye movement data are valid, a model prediction result and the eye movement data are fused to output a fixation point, when eye movement fails, the fixation point is calculated in combination with model prediction and historical data, enhanced compensation of eye movement tracking by electroencephalogram signals is achieved, the defect that the spatial resolution of a single EEG signal is low is effectively overcome, and the accuracy of eye movement tracking is improved. And the robustness and the tracking precision of the system in a complex scene are improved.

Description

Background Art

[0002] Non-contact eye tracking technology uses sensors such as cameras to capture eye movements remotely, eliminating the need for physical equipment on the subject. It can accurately capture visual attention information, such as gaze position and saccade trajectory, and offers advantages such as intuitiveness and high spatial resolution. However, traditional eye trackers also have limitations: First, their acquisition relies on the visibility and stable posture of the subject's eyes, and tracking may fail in strong light, under occlusion, or when the line of sight strays from the camera's effective range.

[0003] Electroencephalogram (EEG) signals record changes in electrical activity in the cerebral cortex. EEG signals often contain changes in electrical signals caused by eye movements, and can be used to infer eye movement behavior to a certain extent. When visual conditions are poor or the eye tracker fails, EEG can still be collected continuously, regardless of whether the line of sight is seen by the camera. EEG itself also has shortcomings: low spatial resolution, making it difficult to accurately decode the gaze point directly from brain signals. Past studies have shown that the mean square error of eye tracking based solely on EEG is still several centimeters higher than that of camera eye trackers, making it difficult to achieve practical accuracy. In addition, EEG signals are noisy and have significant individual differences, and model training requires a large amount of data correction. This makes it challenging to rely solely on EEG to locate the gaze.

[0004] Based on this, the present invention proposes an eye tracking system, method and device that integrates EEG signals with a non-contact eye tracker. Summary of the Invention

[0005] In order to solve the above-mentioned problems in the prior art, namely, the limited acquisition range of the prior art non-contact eye tracker and the low spatial resolution of eye movement behavior inferred from EEG signals, the present invention provides an eye tracking system, method and device that integrates EEG signals with a non-contact eye tracker.

[0006] In a first aspect, the present invention provides an eye tracking system that integrates EEG signals with a non-contact eye tracker. The system comprises: Data acquisition module, used to synchronously collect eye movement data and EEG signals from a non-contact eye tracker; A calibration and preprocessing module calibrates and maps the eye movement data to the screen coordinate system, and performs filtering, denoising, and time synchronization preprocessing on the EEG signal to generate time-aligned multimodal data; An offline model training module, based on the multimodal data and using the gaze point labels provided by the eye tracker as supervision signals, trains a pre-built EEG modality eye movement behavior recognition model; the EEG modality eye movement behavior recognition model is used to learn the association between EEG signals and eye movement data; An online multi-modal fusion and inference module for real-time acquisition of eye movement data and EEG signals; determining whether the calibrated eye movement data is valid; If it is valid, the preprocessed EEG signal is input into the eye movement behavior recognition model of the EEG modality to obtain a prediction result. The prediction result and the calibrated eye movement data are adopted with a fusion strategy to output the fused fixation point; If it is invalid, the preprocessed EEG signal is input into the eye movement behavior recognition model of the EEG modality to obtain a prediction result, and the fixation point is calculated in combination with historical data.

[0007] Furthermore, the eye movement behavior recognition model of the EEG modality includes: A temporal convolutional network layer for extracting local temporal features from the preprocessed EEG signal. Through causal convolution and dilated convolution operations, it captures the changes in the EEG transient waveform caused by eye movement behavior and aligns them with the eye movement data in a time window to establish a preliminary association between the EEG signal and the eye movement data; A bidirectional long short-term memory network layer for receiving the temporal features output by the temporal convolutional network layer. By using forward and backward LSTM units to model the historical data and future prediction data of the EEG signal on the time axis respectively, it generates a feature sequence containing temporal context information; the feature sequence is associated with the saccade direction and fixation duration information in the eye movement label; An attention mechanism layer for weighted aggregation of the feature sequence output by the bidirectional long short-term memory network layer, calculating the contribution weights of the features at each time point to the current eye movement behavior classification or fixation point prediction, and focusing on the EEG waveform segments corresponding to the eye movement events; among them, the eye movement events include saccade initiation and blinking; A multi-modal feature fusion layer for obtaining the eye movement data and extracting an eye movement feature vector, and merging it with the weighted aggregated feature vector output by the attention mechanism layer in the fusion layer to obtain a fused feature vector; An output layer for outputting the probability distribution of eye movement behavior categories or the regression value of the fixation point coordinates based on the fused feature vector. The supervision signal of the output layer is directly generated by the label provided by the eye tracker, including at least one of the fixation point position, saccade direction, and amplitude.

[0008] Furthermore, the eye movement behavior recognition model of the EEG modality also includes an online adaptation module, which updates the model parameters by incremental fine-tuning and few-shot transfer learning using new data.

[0009] Furthermore, the method for determining whether the eye movement data is valid is as follows: When it is detected that the number of frames in which the eye tracker continuously fails to detect the pupil exceeds the threshold, or the confidence index of the eye tracker is lower than the preset value, it is determined that the eye movement data is invalid.

[0010] Further, the fixation point is calculated by combining historical data, and the specific calculation method is as follows: Based on the eye movement behavior type, direction, and amplitude parameters predicted by the eye movement behavior recognition model of the EEG modality, taking the last valid eye tracker fixation point as a reference, a new fixation point is obtained by superimposing the movement increment; During the continuous failure period, recursive forward extrapolation is used. The fixation point predicted by the eye movement behavior recognition model of the EEG modality at the previous moment is used as the current extrapolation reference, and the position is updated by combining the movement increment predicted by the eye movement behavior recognition model of the EEG modality.

[0011] Further, the system determines whether the user is in an effective visual attention state by analyzing the attention-related features in the EEG signal; when attention dispersion or eye closure is detected, the prediction update of the fixation point is paused.

[0012] Further, when the eye movement data recovers from failure, a transition fusion strategy is adopted: In the initial stage of recovery, the average value of the eye movement data obtained by the eye tracker and the result predicted by the eye movement behavior recognition model of the EEG modality is used as the output; Within a set time window, the weight of the eye tracker data is gradually increased until the eye tracker-based tracking mode is fully restored.

[0013] Further, the fusion strategy includes residual fusion, and the fusion method is as follows: The fixation point offset predicted by the eye movement behavior recognition model of the EEG modality is superimposed on the fixation point coordinates corresponding to the eye movement data to obtain the final fixation point; Among them, the offset is generated by the eye movement direction and amplitude predicted by the eye movement behavior recognition model of the EEG modality according to the current EEG sequence.

[0014] In the second aspect of the present invention, an eye movement tracking method that fuses EEG signals and a non-contact eye tracker is proposed. Based on an eye movement tracking system that fuses EEG signals and a non-contact eye tracker, the method includes: Synchronously collect the eye movement data of the non-contact eye tracker and the EEG signal; Calibrate and map the eye movement data to the screen coordinate system, and perform preprocessing of filtering, denoising, and time synchronization alignment on the EEG signal to generate multi-modal data with time alignment; Based on the multi-modal data, using the fixation point label provided by the eye tracker as a supervision signal, train the pre-constructed eye movement behavior recognition model of the EEG modality; the eye movement behavior recognition model of the EEG modality is used to learn the association between the EEG signal and the eye movement data; Real-time obtain the eye movement data and the EEG signal; determine whether the calibrated eye movement data is valid; If it is effective, input the preprocessed EEG signal into the eye movement behavior recognition model of the EEG modality to obtain a prediction result, and use a fusion strategy for the prediction result and the calibrated eye movement data to output the fused fixation point; If it is ineffective, input the preprocessed EEG signal into the eye movement behavior recognition model of the EEG modality to obtain a prediction result, and calculate the fixation point in combination with historical data.

[0015] In a third aspect of the present invention, an electronic device is proposed, including: At least one processor; and A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned eye movement tracking method that fuses EEG signals and a non-contact eye tracker.

[0016] Advantages of the present invention: Improve tracking robustness and continuity: Through the complementary fusion of EEG signals and eye tracker data, when the eye tracker fails due to occlusion, strong light, or line-of-sight deviation, use EEG signals to maintain the continuity of eye movement tracking, significantly reducing the risk of tracking interruption caused by environmental or posture interference and meeting the requirements of complex scenarios.

[0017] Enhance spatial resolution and accuracy: Use high-precision eye tracker data as a supervision signal to train the EEG model, and combine an online multi-modal fusion strategy to effectively make up for the defect of low spatial resolution of a single EEG signal, improve the accuracy of fixation point positioning, and especially achieve more refined inference of the line-of-sight landing point within the effective range of the eye tracker.

[0018] Reduce individual differences and data dependence: Through the joint modeling of multi-modal data with time series alignment, use the calibration information of the eye tracker to optimize the feature expression of EEG signals, alleviate the problems of large EEG signal noise and significant individual differences, reduce the dependence of the model on a large amount of labeled data of a single modality, and improve the generalization ability.

[0019] Dynamic adaptive scene switching: The online module dynamically selects the fusion or pure EEG inference mode according to the validity of eye movement data, which can not only give play to the high-precision advantage of the eye tracker but also seamlessly switch to the EEG-dominated tracking mode when it fails, ensuring the flexibility and reliability of the system in different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objectives, and advantages of the present application will become more obvious: Figure 1 It is a schematic diagram of the working process of an eye movement tracking system that fuses EEG signals and a non-contact eye tracker of the present invention; Figure 2 It is a schematic diagram of the training process of the eye movement behavior recognition model in the electroencephalogram modality of the eye movement tracking system that integrates electroencephalogram signals and a non-contact eye tracker according to the present invention. Specific embodiments

[0021] The following further elaborates on the present application in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the relevant invention and do not limit the invention. Additionally, it should be noted that for ease of description, only parts related to the relevant invention are shown in the drawings.

[0022] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The following will elaborate on the present application in detail with reference to the drawings and embodiments.

[0023] In the first embodiment of the present invention, an eye movement tracking system that integrates electroencephalogram signals and a non-contact eye tracker is proposed. The system includes: A data acquisition module for synchronously acquiring eye movement data of the non-contact eye tracker and electroencephalogram signals; A calibration and preprocessing module that calibrates and maps the eye movement data to the screen coordinate system, and preprocesses the electroencephalogram signals by filtering out noise and time synchronization alignment to generate multi-modal data with time alignment; An offline model training module that, based on the multi-modal data, uses the fixation point labels provided by the eye tracker as a supervision signal to train a pre-constructed eye movement behavior recognition model in the electroencephalogram modality; the eye movement behavior recognition model in the electroencephalogram modality is used to learn the association between electroencephalogram signals and eye movement data; An online multi-modal fusion and inference module for real-time acquisition of eye movement data and electroencephalogram signals; determining whether the calibrated eye movement data is valid; If it is valid, the preprocessed electroencephalogram signals are input into the eye movement behavior recognition model in the electroencephalogram modality to obtain a prediction result, and the prediction result and the calibrated eye movement data are fused using a fusion strategy to output a fused fixation point; If it is invalid, the preprocessed electroencephalogram signals are input into the eye movement behavior recognition model in the electroencephalogram modality to obtain a prediction result, and the fixation point is calculated in combination with historical data.

[0024] To more clearly illustrate an eye movement tracking system that integrates electroencephalogram signals and a non-contact eye tracker according to the present invention, the following combines Figure 1 and Figure 2 to elaborate on each module in the embodiments of the present invention in detail, and the detailed description is as follows: A data acquisition module for synchronously acquiring eye movement data of the non-contact eye tracker and electroencephalogram signals; In this embodiment, the electroencephalogram (EEG) signal is acquired based on an EEG acquisition device.

[0025] Among them, the eye movement data is specifically the trajectory of the user's fixation point.

[0026] The calibration and preprocessing module calibrates the eye movement data and maps it to the screen coordinate system, and performs preprocessing on the EEG signal, including filtering for noise reduction and time synchronization alignment, to generate multi-modal data with time alignment. The steps of the preprocessing also include artifact correction.

[0027] The offline model training module, based on the multi-modal data, uses the fixation point labels provided by the eye tracker as the supervision signal to train a pre-constructed eye movement behavior recognition model for the EEG modality; the eye movement behavior recognition model for the EEG modality is used to learn the association between the EEG signal and the eye movement data. In this embodiment, the eye movement behavior recognition model for the EEG modality includes: The temporal convolutional network layer is used to extract local temporal features from the preprocessed EEG signal. Through causal convolution and dilated convolution operations, it captures the changes in the EEG transient waveforms caused by eye movement behaviors and aligns them with the eye movement data in a time window to establish a preliminary association between the EEG signal and the eye movement data. The bidirectional long short-term memory network layer is used to receive the temporal features output by the temporal convolutional network layer. By using forward and backward LSTM units to model the historical data and future prediction data of the EEG signal on the time axis respectively, it generates a feature sequence containing temporal context information; the feature sequence is associated with the saccade direction and fixation duration information in the eye movement label. The attention mechanism layer is used to perform weighted aggregation on the feature sequence output by the bidirectional long short-term memory network layer, calculate the contribution weights of the features at each time point to the current eye movement behavior classification or fixation point prediction, and focus on the EEG waveform segments corresponding to eye movement events; among them, the eye movement events include saccade initiation and blinking. The multi-modal feature fusion layer obtains the eye movement data and extracts the eye movement feature vector, and merges it with the weighted aggregated feature vector output by the attention mechanism layer in the fusion layer to obtain the fused feature vector. The output layer, based on the fused feature vector, outputs the probability distribution of eye movement behavior categories or the regression value of the fixation point coordinates. The supervision signal of the output layer is directly generated by the labels provided by the eye tracker, including at least one of the fixation point position, saccade direction, and amplitude.

[0028] The online adaptation module updates the model parameters by incremental fine-tuning and few-shot transfer learning using new data.

[0029] Specifically, the Temporal Convolutional Network (TCN) layer: is responsible for extracting local temporal pattern features from the original EEG time series. The TCN consists of multiple one-dimensional convolutions. Causal convolutions are used to ensure that the convolution does not leak future information, while dilated convolutions are used to expand the receptive field. This structure can efficiently process long sequences in parallel and avoid the common gradient vanishing / exploding problems of RNNs. The TCN is good at capturing short-term transient features in EEG signals (such as sudden changes in EEG waveforms caused by eye movements), and provides faster computation and more stable training compared to traditional LSTMs.

[0030] Bidirectional LSTM layer: On top of the feature sequence extracted by the TCN, a bidirectional long short-term memory network is concatenated. The forward LSTM reads the sequence to capture past dependencies, and the backward LSTM reads the sequence in reverse to capture future dependencies, and then the hidden states in both directions are concatenated. For eye movement behavior recognition, some patterns (such as saccade start and end) may echo each other in EEG. The bidirectional structure helps to more completely characterize such patterns. The long short-term memory units of the LSTM can also preserve information on a longer time scale, complementing the local features of the TCN.

[0031] Attention mechanism: An attention layer is added to the output sequence of the Bi-LSTM to calculate the relevance weights of the hidden states at each moment for the final task. The Self-Attention method is used: First, the similarity scores between the hidden states and a learnable query vector are calculated, then the normalized weights are obtained through Softmax, and then the weighted sum is used to obtain the context vector of the entire sequence. This mechanism enables the model to "focus" on key time points or feature dimensions (such as a significant EEG waveform change related to an eye movement), giving it higher influence in decision-making. The attention mechanism effectively improves the model's ability to model temporal data and has proven to improve accuracy and interpretability in multiple EEG classification tasks.

[0032] Multi-modal feature fusion: For the fusion of EEG features and eye movement features, the present invention designs a two-stream branch in the network structure. The EEG sequence is encoded into an EEG feature vector through the above TCN + BiLSTM + attention; at the same time, the data of the eye tracker (such as the gaze coordinates and movement speed in recent moments) is input into another branch neural network (one or two layers of fully connected / convolutional layers can be used to extract features) to obtain the eye movement feature vector. Subsequently, the two-modal features are merged in the fusion layer. The fusion output then passes through several subsequent fully connected layers to predict the final eye movement state classification or fixation point coordinates.

[0033] Output layer: Depending on the task, the output layer can be a classification layer or a regression layer. For eye movement behavior recognition (such as identifying whether it is a saccade), the output layer is usually Softmax classification, outputting the probability distribution of several eye movement behavior categories; for fixation point coordinate regression, two single values (x and y) or a two-dimensional vector are output. During training, cross-entropy loss or mean squared error loss is used respectively.

[0034] Online adaptation module: To enable the model to maintain performance as the user and environment change in actual applications, the present invention introduces an online learning strategy. First is incremental fine-tuning: During the operation of the system, new EEG and eye movement data pairs are continuously collected (using the output of the eye tracker as an immediate label), and the model parameters are updated regularly in small batches. A smaller learning rate and optimization algorithm (such as Adam) are used for fine-tuning, which can gradually correct the model's adaptability to the specific brain signal patterns of the current user without forgetting the original capabilities. Second is few-shot transfer learning: When a new user or a new scenario is encountered, the existing model can be loaded as pre-trained parameters, and only a small number of calibration samples (such as several minutes of synchronized data) are used for rapid training to adjust the high-level parameters of the model.

[0035] Online multimodal fusion and inference module, used to obtain eye movement data and EEG signals in real time; determine whether the calibrated eye movement data is valid; If it is valid, the preprocessed EEG signal is input into the eye movement behavior recognition model of the EEG modality to obtain a prediction result, and the prediction result and the calibrated eye movement data are used with a fusion strategy to output the fused fixation point; If it is invalid, the preprocessed EEG signal is input into the eye movement behavior recognition model of the EEG modality to obtain a prediction result, and the fixation point is calculated by combining historical data.

[0036] In this embodiment, the method for determining whether the eye movement data is valid is as follows: When the number of consecutive frames in which the eye tracker fails to detect the pupil exceeds the threshold, or the confidence index of the eye tracker is lower than the preset value, it is determined that the eye movement data is invalid.

[0037] In this embodiment, when the eye movement data is invalid, the fixation point is calculated by combining historical data, and the specific calculation method is as follows: Based on the eye movement behavior type, direction, and amplitude parameters predicted by the eye movement behavior recognition model of the EEG modality, with the last valid eye tracker fixation point as the benchmark, a movement increment is superimposed to obtain a new fixation point; During the continuous invalid period, recursive forward extrapolation is used, taking the fixation point predicted by the eye movement behavior recognition model of the EEG modality at the previous moment as the current extrapolation benchmark, and updating the position by combining the movement increment predicted by the eye movement behavior recognition model of the EEG modality.

[0038] The system determines whether the user is in an effective visual attention state by analyzing attention-related features in the EEG signals; when attention dispersion or eye closure is detected, the fixation prediction update is paused.

[0039] When the eye movement data recovers from failure, a transitional fusion strategy is adopted: In the initial stage of recovery, the average value of the eye movement data obtained by the eye tracker and the result predicted by the eye movement behavior recognition model of the EEG modality is used as the output; Within a set time window, the weight of the eye tracker data is gradually increased until the eye tracker-based tracking mode is fully restored.

[0040] Specifically, when the user's line of sight exceeds the capture range of the eye tracker, the eyes are blocked, or the eye tracker fails temporarily due to ambient light, the system will enable the EEG-based fixation prediction mechanism to maintain continuous tracking of the user's line of sight. First, the system determines the occurrence of failure by monitoring the validity of the eye tracker output: for example, if the pupil is not detected in several consecutive frames, or the eye tracker confidence index is lower than the threshold, it is determined to enter the "transition region" or failure state. At this time, the fusion module automatically switches to the EEG single-modal prediction mode, and the EEG model estimates the current fixation direction or fixation point position.

[0041] The role of the EEG model in this situation is similar to that of a virtual eye tracker. Since it is difficult to directly reverse the exact coordinates from the EEG, a method that comprehensively utilizes kinematic constraints and sequence prediction is used to improve reliability: the EEG model first predicts the current eye movement behavior based on the recent EEG sequence, such as determining whether a saccade has occurred at this time and its direction and amplitude. Then, combined with the last known eye tracker fixation point of the user, a new fixation point prediction is inferred. For example, if the EEG model identifies a saccade to the upper right and estimates its amplitude to be approximately 15 degrees of visual angle, then starting from the last known fixation position before the transition, the new fixation point is inferred at the corresponding position in this direction. In this way, by using the prior position plus the increment inferred by the EEG, continuous prediction of the fixation point is achieved.

[0042] To handle longer eye tracker failures, our EEG prediction model can recursively extrapolate step by step. That is, each time the new fixation point inferred from the EEG is used as the reference starting point for the next moment, continuously updating the position by combining the EEG sequence until the eye tracker recaptures the target. During this process, it is also necessary to consider that the prediction uncertainty accumulates and increases over time. Therefore, the system will introduce a decay trust mechanism during EEG prediction: as the failure duration increases, the model should be more conservative in the position change (for example, assuming that the user may be fixating statically somewhere), to avoid prediction drift. On the other hand, the EEG signal itself can be used to detect whether the user's visual attention is still on the current task. If the user's attention is distracted or the user closes their eyes to rest, the system can suspend updating the fixation point.

[0043] When the user's eyes return to the eye tracker's field of view, the system needs to smoothly transition back to the eye tracker tracking mode. For this purpose, the present invention designs a transition region fusion process: at the moment when the eye tracker just redetects the eyes, there may still be errors or jitters in its positioning. At this time, continuing to perform average fusion with reference to the EEG prediction can avoid sudden changes in the output. Subsequently, it gradually resumes the eye tracker-based mode within dozens of milliseconds. Through the above mechanism, no matter how the user's line of sight moves, this system can continuously give an estimate of the fixation point, significantly improving the robustness.

[0044] Within the normal working range of the eye tracker, this system adopts a data fusion strategy to combine the two types of information of the eye tracker and EEG, in order to improve the robustness and accuracy of eye movement tracking. Specifically, when the eye tracker can reliably output the user's fixation point, we use the eye tracker data as the main information source. On the one hand, EEG can provide supplementary information about the user's cognitive state and attention, helping to judge the nature of the current eye movement. On the other hand, the EEG model can perform redundant verification and fine correction on the output of the eye tracker: if the EEG prediction result is significantly inconsistent with the eye tracker, it may indicate that the eye tracker has an instantaneous frame loss or error. At this time, the system can increase the weight of the EEG and smoothly adjust the output fixation point to avoid jumps.

[0045] Therefore, in this embodiment, when the eye movement data is valid, coordinates are output based on the fusion strategy, and its specific fusion method includes: Superimpose the fixation point offset predicted by the eye movement behavior recognition model of the EEG modality on the fixation point coordinates corresponding to the eye movement data to obtain the final fixation point; Among them, the offset is generated by the eye movement behavior recognition model of the EEG modality according to the predicted eye movement direction and amplitude of the current EEG sequence.

[0046] Specifically, the prediction of the eye movement behavior recognition model in the EEG modality is regarded as a residual correction to the output of the eye tracker. The output of the final fixation point = the coordinates of the eye tracker + the EEG prediction offset. In this way, under normal circumstances, the EEG only fine-tunes the result, and when the eye tracker malfunctions, the correction amount of the EEG will automatically increase to play a compensatory role.

[0047] This embodiment also gives two other fusion methods, namely feature-level fusion and decision-level fusion. Specifically: Feature-level fusion is to merge the features extracted from the EEG signal with the features of the eye tracker (such as the line-of-sight direction vector, pupil size, etc.) early in the eye movement behavior recognition model in the EEG modality, and make a joint decision through the subsequent network. This requires time synchronization and scale normalization processing of the two-modal data, and a special fusion layer (such as cross-modal attention) is used to mine the correlation between the EEG and eye movement signals.

[0048] Decision-level fusion is to perform dynamic weighted fusion of the results according to the reliability after obtaining the respective fixation point predictions or eye movement event discriminations based on the EEG and the eye tracker respectively. Since the eye tracker usually has higher accuracy, the system can assign it a larger weight, and the output of the eye movement behavior recognition model in the EEG modality is used as an auxiliary correction term.

[0049] For example, when the signal quality of the eye tracker is high, 90% of the weight is used for the fixation point of the eye tracker, and 10% of the weight is used for the prediction of the eye movement behavior recognition model in the EEG modality; when the signal of the eye tracker deteriorates, the weight of the eye tracker is correspondingly reduced and the EEG weight is increased. Such weights can be automatically adjusted according to the confidence index provided by the eye tracker to form an adaptive weighted fusion.

[0050] For the above two cases of normal and failure, we define a unified multi-modal fusion prediction function to describe how the system synthesizes the information of the two sensors to output the final fixation point. This function dynamically adjusts the strategy according to the availability of the eye tracker data: When the eye movement data is valid, obtain the fixation point coordinates output by the eye tracker in real time and the correction offset predicted by the eye movement behavior recognition model in the EEG modality ; Dynamically calculate the fusion weight according to the confidence index of the current frame of the eye tracker α , where α takes values in the range of numbers close to 1 (for example, 0.9 ≤ α ≤ 1), and the higher the confidence, α the closer it is to 1; Perform weighted fusion on the coordinates of the eye tracker and the coordinates corrected by the EEG to generate the final fixation point: Calculate the corrected coordinates of the eye tracker p eye( t )+Δ pEEG( t ); The final output is , where when α = 1, the eye tracker data is fully trusted.

[0051] When the eye movement data fails, the eye movement behavior recognition model based on the EEG modality predicts the fixation point offset according to the current and historical EEG sequences ; Taking the coordinates of the fixation point of the last valid frame of the eye tracker before failure as a reference, the offset predicted by the eye movement behavior recognition model of the EEG modality is superimposed to generate the current fixation point .

[0052] The prediction logic of the eye movement behavior recognition model of the EEG modality in the failure state further includes: Applying kinematic constraints to the offset predicted by the eye movement behavior recognition model of the EEG modality : Based on the assumption of the movement continuity of the user's eye movement behavior, a low-pass filtering algorithm or a Kalman filter model is used to smooth the instantaneous change amplitude of; If the direction or amplitude of the offset predicted in multiple consecutive frames exceeds the preset physiological range (for example, the single saccade angle does not exceed 30 degrees), then forcefully correct to within a reasonable threshold; Output the smoothed and constrained offset for calculation .

[0053] When the eye tracker signal is in the transition region, the dynamic weighted fusion strategy is executed according to the following logic: Transition region determination condition: Real-time monitoring of the eye tracker pupil detection results and signal quality indicators; If the number of consecutive valid frames of the eye tracker is lower than the threshold, or the signal noise level exceeds the preset threshold, it is determined to enter the transition region.

[0054] Steps for executing the fusion function: Assign dynamic confidence weights to each frame of eye tracker data w ( t ), whose value range is [0, 1] and satisfies: When the signal quality of the eye tracker is high, w ( t ) approaches 1; When the signal quality deteriorates or is interrupted, w ( t ) approaches 0; Real-time calculation of the fixation point predicted by the eye movement behavior recognition model of the EEG modality ; Output the fused fixation points: ; Smoothing transition logic in the recovery phase: When the eye tracker recovers from the failure state, initialize w ( t ) = 0; Within the set transition time window (e.g., 200 milliseconds), gradually increase w ( t ) to 1 according to a linear or exponential function; During this period, continuously fuse the eye movement data with the prediction results of the eye movement behavior recognition model in the EEG modality to avoid jumps in the fixation point coordinates.

[0055] The confidence weight of the transition region w ( t ) is comprehensively calculated through the following multi-dimensional conditions: Condition 1: Eye tracker pupil detection stability Count the number of consecutive frames in which the pupil is successfully detected in the most recent N frames; If the number of consecutive frames is below the threshold, reduce w ( t ) proportionally.

[0056] Condition 2: Eye tracker signal noise level Extract the signal-to-noise ratio (SNR) or high-frequency noise energy of the eye tracker raw signal; If the SNR is below the preset threshold or the noise energy exceeds the threshold, inversely map according to the noise level to w ( t )

[0057] Condition 3: Cross-modal short-term consistency check Calculate the Euclidean distance error between the predicted coordinates of the eye tracker and the predicted coordinates of the eye movement behavior recognition model in the most recent K frames; If the error exceeds the allowable range, reduce w ( t ) to increase the weight of the prediction results of the eye movement behavior recognition model.

[0058] Final weight calculation: Perform weighted summation or take the minimum value of the weights output by the above conditions to generate w ( t ) ∈ [0, 1].

[0059] It should be noted that, for the eye movement tracking system integrating electroencephalogram (EEG) signals and a non-contact eye tracker provided in the above embodiments, only the division of the above functional modules is used for illustration. In practical applications, the above functions can be allocated to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further split into multiple sub-modules to complete all or part of the functions described above. For the names of the modules and steps involved in the embodiments of the present invention, they are only used to distinguish each module or step, and are not regarded as improper limitations on the present invention.

[0060] In the second embodiment of the present invention, an eye movement tracking method integrating electroencephalogram (EEG) signals and a non-contact eye tracker is proposed. Based on an eye movement tracking system integrating electroencephalogram (EEG) signals and a non-contact eye tracker, the method includes: Synchronously collect eye movement data of the non-contact eye tracker and EEG signals; Calibrate and map the eye movement data to the screen coordinate system, and perform preprocessing on the EEG signals including filtering, denoising, and time synchronization alignment to generate multi-modal data with time alignment; Based on the multi-modal data, use the fixation point label provided by the eye tracker as a supervision signal to train a pre-constructed eye movement behavior recognition model in the EEG modality; the eye movement behavior recognition model in the EEG modality is used to learn the association between EEG signals and eye movement data; Obtain eye movement data and EEG signals in real time; determine whether the calibrated eye movement data is valid; [[ID=X]] If it is valid, input the preprocessed EEG signals into the eye movement behavior recognition model in the EEG modality to obtain a prediction result, and use a fusion strategy for the prediction result and the calibrated eye movement data to output a fused fixation point; If it is invalid, input the preprocessed EEG signals into the eye movement behavior recognition model in the EEG modality to obtain a prediction result, and calculate the fixation point in combination with historical data.

[0061] Although the above steps are described in the above sequential order in the above embodiments, those skilled in the art can understand that, in order to achieve the effects of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reversed order, and these simple changes are all within the protection scope of the present invention.

[0062] Those skilled in the art to which the present invention pertains can clearly understand that, for the sake of convenience and brevity of description, the specific working processes and related descriptions of the above-described method can refer to the corresponding processes in the foregoing system embodiments, and will not be elaborated herein.

[0063] An electronic device according to a third embodiment of the present invention includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned eye movement tracking method that fuses electroencephalogram signals and a non-contact eye tracker.

[0064] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, and the computer instructions are used to be executed by the computer to implement the above-mentioned eye movement tracking method that fuses electroencephalogram signals and a non-contact eye tracker.

[0065] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes and related descriptions of the above-described storage device and processing device can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0066] Those skilled in the art should be able to realize that the modules and method steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. The programs corresponding to the software modules and method steps can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in the form of electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0067] Terms such as "first" and "second" are used to distinguish similar objects, rather than to describe or represent a specific order or sequence.

[0068] The term "including" or any other similar term is intended to cover non-exclusive inclusion, so that a process, method, article, or device / equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent in these processes, methods, articles, or devices / equipment.

[0069] Thus far, the technical solution of the present invention has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easily understood by those skilled in the art that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims

1. An eye movement tracking system integrating electroencephalogram signals and a non-contact eye tracker, characterized in that, The system includes: A data acquisition module, which is used to synchronously acquire the eye movement data and electroencephalogram signals of a non-contact eye tracker; A calibration and preprocessing module, which calibrates the eye movement data and maps it to the screen coordinate system, and performs preprocessing on the electroencephalogram signals, including filtering, denoising, and time synchronization alignment, to generate temporally aligned multimodal data; An offline model training module, which, based on the multimodal data and using the fixation point labels provided by the eye tracker as the supervision signal, trains a pre-constructed eye movement behavior recognition model for the electroencephalogram modality; the eye movement behavior recognition model for the electroencephalogram modality is used to learn the correlation between electroencephalogram signals and eye movement data; An online multimodal fusion and inference module, which is used to obtain eye movement data and electroencephalogram signals in real time; and determine whether the calibrated eye movement data is valid; If it is valid, the preprocessed electroencephalogram signals are input into the eye movement behavior recognition model for the electroencephalogram modality to obtain a prediction result, and the prediction result and the calibrated eye movement data are combined using a fusion strategy to output the fused fixation point; If it is invalid, the preprocessed electroencephalogram signals are input into the eye movement behavior recognition model for the electroencephalogram modality to obtain a prediction result, and the fixation point is calculated by combining historical data.

2. The eye movement tracking system integrating electroencephalogram signals and a non-contact eye tracker according to claim 1, characterized in that, The eye movement behavior recognition model for the electroencephalogram modality includes: A temporal convolutional network layer, which is used to extract local temporal features from the preprocessed electroencephalogram signals. Through causal convolution and dilated convolution operations, it captures the electroencephalogram transient waveform changes caused by eye movement behaviors, aligns them with the eye movement data in a time window, and establishes a preliminary correlation between electroencephalogram signals and eye movement data; A bidirectional long short-term memory network layer, which is used to receive the temporal features output by the temporal convolutional network layer. By using forward and backward LSTM units to model the historical data and future prediction data of electroencephalogram signals on the time axis respectively, it generates a feature sequence containing temporal context information; the feature sequence is associated with the saccade direction and fixation duration information in the eye movement label; An attention mechanism layer, which is used to perform weighted aggregation on the feature sequence output by the bidirectional long short-term memory network layer, calculate the contribution weights of the features at each time point to the current eye movement behavior classification or fixation point prediction, and focus on the electroencephalogram waveform segments corresponding to eye movement events; among them, the eye movement events include saccade initiation and blinking; A multimodal feature fusion layer, which obtains the eye movement data and extracts eye movement feature vectors, and merges them with the weighted aggregated feature vectors output by the attention mechanism layer in the fusion layer to obtain fused feature vectors; An output layer, which, based on the fused feature vectors, outputs the probability distribution of eye movement behavior categories or the regression value of the fixation point coordinates. The supervision signal of the output layer is directly generated by the labels provided by the eye tracker, including at least one of the fixation point position, saccade direction, and amplitude.

3. The eye movement tracking system integrating electroencephalogram signals and a non-contact eye tracker according to claim 2, characterized in that The eye movement behavior recognition model for the electroencephalogram modality further includes an online adaptation module, which updates the model parameters using new data through incremental fine-tuning and few-shot transfer learning.

4. An eye movement tracking system integrating electroencephalogram signals and a non-contact eye tracker according to claim 1, characterized in that, The method for determining whether the eye movement data is valid is as follows: When the number of consecutive frames in which the eye tracker fails to detect the pupil exceeds a threshold, or the confidence index of the eye tracker is lower than a preset value, it is determined that the eye movement data is invalid.

5. The eye movement tracking system integrating electroencephalogram signals and a non-contact eye tracker according to claim 1, wherein The fixation points are calculated by combining historical data, and the specific calculation method is as follows: Based on the eye movement behavior type, direction, and amplitude parameters predicted by the eye movement behavior recognition model of the EEG modality, taking the last valid eye tracker fixation point as a reference, a new fixation point is obtained by superimposing the movement increment; During the continuous failure period, recursive forward calculation is used. The fixation point predicted by the eye movement behavior recognition model of the EEG modality at the previous moment is used as the current calculation reference, and the position is updated by combining the movement increment predicted by the eye movement behavior recognition model of the EEG modality.

6. The eye movement tracking system integrating electroencephalogram signals and a non-contact eye tracker according to claim 5, characterized in that, The system determines whether the user is in an effective visual attention state by analyzing the attention-related features in the EEG signal; when attention dispersion or eye closure is detected, the fixation point prediction update is paused.

7. The eye movement tracking system integrating electroencephalogram signals and a non-contact eye tracker according to claim 1, characterized in that, When the eye movement data recovers from failure, a transition fusion strategy is adopted: In the initial stage of recovery, the average value of the eye movement data obtained by the eye tracker and the result predicted by the eye movement behavior recognition model of the EEG modality is used as the output; Within a set time window, the weight of the eye tracker data is gradually increased until the eye tracker-based tracking mode is fully restored.

8. The eye movement tracking system integrating electroencephalogram signals and a non-contact eye tracker according to claim 1, wherein, The fusion strategy includes residual fusion, and its fusion method is as follows: The fixation point offset predicted by the eye movement behavior recognition model of the EEG modality is superimposed on the fixation point coordinates corresponding to the eye movement data to obtain the final fixation point; Among them, the offset is generated by the eye movement direction and amplitude predicted by the eye movement behavior recognition model of the EEG modality according to the current EEG sequence.

9. A method for eye movement tracking that integrates electroencephalogram signals and a non-contact eye tracker, based on the eye movement tracking system that integrates electroencephalogram signals and a non-contact eye tracker according to any one of claims 1-8, characterized in that, The method includes: Synchronously collecting eye movement data of a non-contact eye tracker and EEG signals; Calibrating and mapping the eye movement data to the screen coordinate system, and performing preprocessing of filtering, denoising, and time synchronization alignment on the EEG signals to generate multi-modal data with time sequence alignment; Based on the multi-modal data, using the fixation point label provided by the eye tracker as a supervision signal, training a pre-constructed eye movement behavior recognition model of the EEG modality; the eye movement behavior recognition model of the EEG modality is used to learn the association between EEG signals and eye movement data; Real-time obtaining eye movement data and EEG signals; judging whether the calibrated eye movement data is valid; If it is valid, the preprocessed EEG signal is input into the eye movement behavior recognition model of the EEG modality to obtain a prediction result, and the prediction result and the calibrated eye movement data are adopted with a fusion strategy to output the fused fixation point; If it is invalid, the preprocessed EEG signal is input into the eye movement behavior recognition model of the EEG modality to obtain a prediction result, and the fixation point is calculated by combining historical data.

10. An electronic device, characterized in that, Including: At least one processor; And A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement an eye movement tracking method for fusing EEG signals and a non-contact eye tracker as claimed in claim 9.

Images