An eye movement event classification method and device based on time-frequency domain fusion
By employing a time-frequency domain fusion-based eye-tracking event classification method, which utilizes a frequency domain learner and a self-attention mechanism, the algorithm addresses the issue of noise sensitivity in eye-tracking event detection, achieving higher classification accuracy and stability while reducing reliance on expert experience.
Patent Information
- Application Number
- CN202510671591.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-23
- Publication Date
- 2026-04-17
- Estimated Expiration
- 2045-05-23
AI Technical Summary
Existing eye-tracking event detection algorithms are sensitive to noise, have difficulty adapting to noise interference in different individuals or task scenarios, and struggle to capture long-term dependencies between eye-tracking features, resulting in low classification accuracy.
An eye-tracking event classification method based on time-domain-frequency domain fusion is adopted. Spectral features are extracted through a frequency domain learner, and a self-attention mechanism is combined to capture long-term dependencies between variables. An improved iTransormer model is used for training and classification.
It effectively suppresses noise interference, improves the stability and robustness of the model, can more accurately distinguish different types of eye-tracking events, reduces reliance on expert experience, and enhances the development efficiency and application promotion capabilities of the recognition system.
Smart Images

Figure CN120724234B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of eye-tracking technology, specifically to an eye-tracking event classification method and apparatus based on time-domain-frequency domain fusion. Background Technology
[0002] In recent years, eye-tracking technology has been widely used in visual perception and cognitive research, becoming an important tool for exploring the mechanisms of human visual information acquisition. However, the unavoidable noise interference in eye-tracking data poses a significant challenge to the accuracy and robustness of eye-tracking event detection. These noise sources are complex and diverse, persisting throughout all stages of data acquisition, feature extraction, and analysis, significantly impacting the reliability of research results and the performance of classification models. Traditional eye-tracking event detection algorithms are mostly based on rule-based methods, such as using velocity thresholds and discrete thresholds for event detection. For example, the velocity threshold recognition algorithm (I-VT) proposed by one person distinguishes between fixation and saccades based on velocity features: values below the velocity threshold are considered fixation, and values above are considered saccades. However, this rule-based algorithm is highly sensitive to noise, and its detection performance heavily relies on preset thresholds, making it difficult to adapt to noise interference in different individuals or task scenarios. Algorithms based on discrete analysis assume that the spatial distribution of fixation events is relatively concentrated, but require manually setting multiple parameters, resulting in a significant decrease in detection accuracy under complex noise or variable data conditions. Although some methods have improved performance through adaptive threshold selection and principal component analysis, they still perform poorly on highly noisy data. Researchers also proposed a hierarchical ensemble learning method based on Cascade Forest (CF). This method relies on a cascaded structure to extract and transform features layer by layer, utilizing multiple base classifiers (such as Random Forest, Extreme Random Forest, XGBoost, and LightGBM) for multi-level joint learning. The Cascade Forest method combines the advantages of multiple classifiers, enhancing the model's classification performance through hierarchical feature extraction and transformation. However, despite this, these methods still struggle to capture long-term dependencies between eye-tracking features and fail to fully utilize the dynamic characteristics of time-series eye-tracking data. Summary of the Invention
[0003] This invention provides a method and apparatus for classifying eye-movement events based on time-domain-frequency-domain fusion, in order to solve at least one of the above-mentioned technical problems.
[0004] The technical solution of this invention to solve the above-mentioned technical problems is as follows: An eye-tracking event classification method based on time-domain-frequency domain fusion, comprising:
[0005] S1, acquire time-based eye movement data sequences, and divide the eye movement data sequences into training set, test set and validation set;
[0006] S2, Construct an improved iTransormer eye-tracking event classification model based on frequency domain-time domain feature fusion; wherein, the eye-tracking event classification model includes a frequency domain learner, a stacked layer, and a classification layer;
[0007] S3, use the training set to train the frequency domain learner on frequency domain features and output the frequency domain training result; based on the self-attention mechanism, use the frequency domain training result to train the stacked layer on temporal features and output the temporal training result; use the temporal training result to train the classification layer on classification and calculate the cross-entropy loss.
[0008] S4, determine whether the cross-entropy loss has reached the minimum; if not, use the Adam optimizer to adjust the model parameters of the eye-tracking event classification model and return to execute S3; if yes, save the eye-tracking event classification model trained at this time as the model with the best classification effect.
[0009] S5. Use the model with the best classification effect to classify eye-tracking events and obtain the classification results.
[0010] Based on the above technical solution, the present invention can be further improved as follows.
[0011] Furthermore, in S1, the eye movement data sequence includes multiple eye movement data, each of which includes the recording time, the X-axis coordinate value of the eye position, the Y-axis coordinate value of the eye position, the confidence level, the first human expert judgment result of the eye movement event, the second human expert judgment result of the eye movement event, the comprehensive judgment result of the eye movement event, the eye movement velocity, the eye movement acceleration, and the eye movement direction.
[0012] Furthermore, the recording time, the X-axis coordinate value of the eye position, and the Y-axis coordinate value of the eye position are obtained by tracking and measuring with an eye tracker and filtering out interference.
[0013] The interference filtering includes screen anomaly filtering, flicker value filtering, and single-sample peak filtering.
[0014] Furthermore, in the eye movement data sequence, the eye movement velocity, eye movement direction, and eye movement acceleration of the first eye movement data are all 0;
[0015] The eye movement velocity, eye movement direction, and eye movement acceleration of the eye movement data in the eye movement data sequence, excluding the first eye movement data, are respectively represented as follows:
[0016]
[0017] Among them, v i The x represents the eye movement velocity of the i-th eye movement data in the eye movement data sequence, where i ≥ 2; i and yi Let t represent the X-axis coordinate and Y-axis coordinate of the eye position for the i-th eye movement data point, respectively; i θ represents the recording time of the i-th eye-tracking data point; i Indicates the eye movement direction of the i-th eye movement data; a i This represents the eye movement acceleration of the i-th eye movement data.
[0018] Furthermore, in step S3, the frequency domain learner is trained using the training set to perform frequency domain feature training, specifically as follows:
[0019] S301, The training set is transformed from the time domain to the frequency domain using Fourier transform to obtain domain-transformed data;
[0020] S302, the domain transformation data is input into the frequency domain learner for frequency domain learning, so as to iteratively update the weights and biases, and then obtain the real and imaginary parts of the frequency components.
[0021] S303 combines the real and imaginary parts of the frequency components to form a complex number, thus obtaining the initial frequency domain training result;
[0022] S304, use inverse Fourier transform to convert the initial frequency domain training result from the frequency domain to the time domain to obtain the frequency domain training result.
[0023] Furthermore, in step S3, the stacked layer is trained with temporal features using the frequency domain training results, specifically as follows:
[0024] S305, each feature variable's time series in the frequency domain training result is independently encoded into a variable time point;
[0025] S306, Calculate the correlation between different time points of the variables to generate a query, key, and value vector;
[0026] S307, based on the attention mechanism, calculates the similarity between each pair of queries and keys to obtain an attention score, and then obtains the attention score matrix;
[0027] S308, Perform a Softmax operation on the attention score matrix to obtain the attention weight matrix;
[0028] S309, The attention weight matrix is used to perform weighted aggregation on the value vector to obtain attention data;
[0029] S310, Perform layer normalization on the attention data to obtain layer normalized data;
[0030] S311, Perform a nonlinear transformation on the layer normalized data to obtain the initial time-domain training results;
[0031] S312, Perform layer normalization on the initial time-domain training result to obtain the time-domain training result.
[0032] Furthermore, in S311, a feedforward network is specifically used to perform a nonlinear transformation on the attention data. The feedforward network includes a first linear layer, a first activation function, and a second linear layer.
[0033] The attention data is nonlinearly transformed using a feedforward network, specifically as follows:
[0034] The attention data is subjected to a linear transformation that increases its dimensionality using the first linear layer to obtain the first linear transformation result;
[0035] The first linear transformation result is subjected to a nonlinear transformation using the first activation function to obtain a nonlinear transformation result.
[0036] The nonlinear transformation result is subjected to a linear transformation to restore the dimension, thereby obtaining the time-domain training result.
[0037] Furthermore, in step S3, the classification layer is trained using the time-domain training results, and the cross-entropy loss is calculated, specifically as follows:
[0038] The time-domain training results are converted into a probability distribution using the Softmax function, thereby obtaining scores for each category and outputting the category with the highest score.
[0039] The cross-entropy loss is calculated based on the scores of each category.
[0040] Furthermore, during the classification training process using the time-domain training results for the classification layer, after each round of training is completed, the following steps are also included:
[0041] Calculate the training loss;
[0042] The trained frequency domain learner and the stacked layers are tested and validated using the test set and the validation set, and the test loss and validation loss are calculated.
[0043] If the verification loss does not decrease for three consecutive rounds, the early training stop mechanism is triggered, and the eye-tracking event classification model trained at this time is saved as the model with the best classification effect.
[0044] Based on the aforementioned eye-tracking event classification method based on time-domain-frequency domain fusion, this invention also provides an eye-tracking event classification device based on time-domain-frequency domain fusion.
[0045] An eye-tracking event classification device based on time-domain-frequency domain fusion includes a processor, a memory, and a computer program stored in the memory. When the computer program is executed by the processor, it implements the eye-tracking event classification method based on time-domain-frequency domain fusion as described above.
[0046] The beneficial effects of this invention are as follows: In the eye-tracking event classification method and apparatus based on time-domain-frequency-domain fusion, by introducing a frequency-domain learner, potential spectral features can be extracted from eye-tracking signals, effectively suppressing noise interference and improving the stability and robustness of the model in complex experimental scenarios. Simultaneously, this invention integrates time-domain multivariate dynamic features with frequency-domain information and introduces a self-weighted attention mechanism to dynamically weight different types of features, enabling the model to more accurately distinguish eye-tracking event types such as gaze, saccades, and smooth tracking. Compared to traditional methods based on manual feature extraction and threshold rules, this invention can automatically complete frequency-domain and time-domain feature learning, reducing reliance on expert experience and significantly improving the development efficiency and application promotion capabilities of eye-tracking event recognition systems. Attached Figure Description
[0047] Figure 1 This is a flowchart of an eye-tracking event classification method based on time-domain-frequency domain fusion according to the present invention;
[0048] Figure 2 The training flowchart for the frequency domain learner and stacked layers is shown below.
[0049] Figure 3 This is a comparison chart of the detection results and the original eye movement categories. Detailed Implementation
[0050] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.
[0051] like Figure 1 As shown, an eye-tracking event classification method based on time-domain-frequency domain fusion includes:
[0052] S1, acquire time-based eye movement data sequences, and divide the eye movement data sequences into training set, test set and validation set;
[0053] S2, Construct an improved iTransormer eye-tracking event classification model based on frequency domain-time domain feature fusion; wherein, the eye-tracking event classification model includes a frequency domain learner, a stacked layer, and a classification layer;
[0054] S3, use the training set to train the frequency domain learner on frequency domain features and output the frequency domain training result; based on the self-attention mechanism, use the frequency domain training result to train the stacked layer on temporal features and output the temporal training result; use the temporal training result to train the classification layer on classification and calculate the cross-entropy loss.
[0055] S4, determine whether the cross-entropy loss has reached the minimum; if not, use the Adam optimizer to adjust the model parameters of the eye-tracking event classification model and return to execute S3; if yes, save the eye-tracking event classification model trained at this time as the model with the best classification effect.
[0056] S5. Use the model with the best classification effect to classify eye-tracking events and obtain the classification results.
[0057] To address the challenge of existing eye-tracking event classification methods struggling to adapt to noise interference across diverse individuals and task scenarios, this invention proposes a time-domain-frequency domain fusion-based eye-tracking event classification method. This aims to overcome the low classification accuracy caused by significant noise impact in traditional methods. Its main features include: performing spectral analysis on eye-tracking data using a frequency domain learner to extract the frequency-domain differences in various types of eye-tracking events; and then utilizing a multivariate self-attention mechanism to capture long-term dependencies between variables. This enables the model to effectively cope with complex and diverse noise interference, maintain stable robustness, and achieve high classification accuracy. This method can be seamlessly integrated into existing eye-tracking analysis systems and provides a more accurate and effective solution for eye-tracking event classification tasks.
[0058] Based on the above technical solutions, the present invention also has the following preferred solutions:
[0059] Preferably, in step S1, the eye movement data sequence includes multiple eye movement data, each of which includes the recording time, X-axis coordinate value of eye position, Y-axis coordinate value of eye position, confidence level, first human expert judgment result of eye movement event, second human expert judgment result of eye movement event, comprehensive judgment result of eye movement event, eye movement velocity, eye movement acceleration, and eye movement direction.
[0060] The recording time, the X-axis coordinate value of the eye position, and the Y-axis coordinate value of the eye position are obtained by tracking and measuring with an eye tracker and then filtering for interference. The interference filtering includes filtering for screen outliers, filtering for flickering values, and filtering for single-sample peak values.
[0061] Specifically, an eye tracker is used to track and measure the position and movement of the eyeballs. The raw data collected by the eye tracker needs to be processed before training to eliminate three different types of interference: screen outliers, flickering, and single-sample peaks.
[0062] Confidence score is a rating based on the reliability of each frame of eye movement data recorded by the eye tracker; a higher score indicates more accurate and reliable data. It is determined by factors such as pupil detection quality, image sharpness, occlusion, and blinking.
[0063] In the eye movement data sequence, the eye movement velocity, eye movement direction, and eye movement acceleration of the first eye movement data are all 0;
[0064] The eye movement velocity, eye movement direction, and eye movement acceleration of the eye movement data in the eye movement data sequence, excluding the first eye movement data, are respectively represented as follows:
[0065]
[0066] Among them, v i The x represents the eye movement velocity of the i-th eye movement data in the eye movement data sequence, where i ≥ 2; i and y i Let t represent the X-axis coordinate and Y-axis coordinate of the eye position for the i-th eye movement data point, respectively; i θ represents the recording time of the i-th eye-tracking data point; i Indicates the eye movement direction of the i-th eye movement data; a i This represents the eye movement acceleration of the i-th eye movement data.
[0067] The first human expert's judgment result, the second human expert's judgment result, and the comprehensive judgment result of the eye movement event are the results of the two human experts' judgments on their respective eye movement event categories and the comprehensive judgment result.
[0068] In addition, in S1, the ratio of dividing the eye-tracking data sequence into training set, test set and validation set is 7:1:2.
[0069] Preferably, in S2, the frequency domain learner is used to learn the frequency domain features of the eye-tracking data. After learning the frequency domain features, the data is fed into the stacked layer to learn the time domain features. Finally, Sorftmax is used in the classification layer to complete the eye-tracking event classification.
[0070] Preferred, such as Figure 2 As shown, in step S3, the frequency domain learner is trained using the training set to perform frequency domain feature training, specifically as follows:
[0071] S301, The training set is transformed from the time domain to the frequency domain using Fourier transform to obtain domain-transformed data;
[0072] S302, the domain transformation data is input into the frequency domain learner for frequency domain learning, so as to iteratively update the weights and biases, and then obtain the real and imaginary parts of the frequency components.
[0073] S303 combines the real and imaginary parts of the frequency components to form a complex number, thus obtaining the initial frequency domain training result;
[0074] S304, use inverse Fourier transform to convert the initial frequency domain training result from the frequency domain to the time domain to obtain the frequency domain training result.
[0075] Specifically, the input training set is first transformed from the time domain to the frequency domain using Fourier transform. The Fourier transform formula is as follows: Where: H(f) is the frequency domain representation, H(v) is the time domain signal, f is the frequency variable, v is the time channel dimension, and j is the imaginary unit. After domain transformation, H is sent as input to the frequency domain learner, which processes the weights and biases of the complex number by calculating the real and imaginary parts of the frequency components respectively. The frequency domain MLP can be represented as: Y0 = H; where, This is the final output. Indicates the first In the layer, σ is the activation function, and Y0 is the initial output. During training, the weights and bias parameters of the MLP are updated each time. Since H and W are both complex numbers, according to the rules of complex multiplication, the above formula needs to be further extended to:
[0076]
[0077] in: It is a complex weight matrix; This is a complex bias term. According to the above formula, during training, the weights and bias parameters are updated each time to achieve frequency domain learning. These are then combined to form a complex number, yielding the initial frequency domain training result. This initial frequency domain training result needs to be transformed to the time domain via an inverse Fourier transform, becoming the input to the stacked layers. The formula for the inverse Fourier transform is as follows:
[0078] Preferred, such as Figure 2 As shown, in step S3, the stacked layer is trained with temporal features using the frequency domain training results, specifically as follows:
[0079] S305, each feature variable's time series in the frequency domain training result is independently encoded into a variable time point;
[0080] S306, Calculate the correlation between different time points of the variables to generate a query, key, and value vector;
[0081] S307, based on the attention mechanism, calculates the similarity between each pair of queries and keys to obtain an attention score, and then obtains the attention score matrix;
[0082] S308, Perform a Softmax operation on the attention score matrix to obtain the attention weight matrix;
[0083] S309, The attention weight matrix is used to perform weighted aggregation on the value vector to obtain attention data;
[0084] S310, Perform layer normalization on the attention data to obtain layer normalized data;
[0085] S311, Perform a nonlinear transformation on the layer normalized data to obtain the initial time-domain training results;
[0086] S312, Perform layer normalization on the initial time-domain training result to obtain the time-domain training result.
[0087] Furthermore, in S311, a feedforward network is specifically used to perform a nonlinear transformation on the attention data. The feedforward network includes a first linear layer, a first activation function, and a second linear layer.
[0088] The attention data is nonlinearly transformed using a feedforward network, specifically as follows:
[0089] The attention data is subjected to a linear transformation that increases its dimensionality using the first linear layer to obtain the first linear transformation result;
[0090] The first linear transformation result is subjected to a nonlinear transformation using the first activation function to obtain a nonlinear transformation result.
[0091] The nonlinear transformation result is subjected to a linear transformation to restore the dimension, thereby obtaining the time-domain training result.
[0092] Specifically, the stacked layer mainly consists of three parts: a self-attention mechanism, layer normalization, and a feedforward network (FFN). The stacked layer independently encodes the time series of each feature variable into a token, and then uses a multivariate self-attention mechanism to calculate the correlation between different tokens, capturing complex dependencies between variables. Next, the feedforward network is used to perform a non-linear transformation on each token, which helps to capture more complex time series patterns.
[0093] The self-attention mechanism is applied, which helps the model capture the dependencies between different variables by calculating the correlation between each token (i.e., each variable across all time points). The computation of self-attention involves three main steps: generating query (Q), key (K), and value vectors (V), and then calculating attention scores and performing weighted aggregation.
[0094] The attention score A is obtained by calculating the similarity between each pair of queries and keys, denoted as:
[0095]
[0096] Where i and j represent different variables, A i,j This is the attention score between two variables. The attention score measures the correlation between the target variable (eye movement category in this invention) and other variables; a higher score indicates a stronger correlation. k This represents the scaling factor. Then, the score matrix A is... i,j Perform a Softmax operation. Softmax converts the score of each row into a probability value, so that the sum of the scores for each row is 1. The Softmax operation is represented as follows:
[0097]
[0098] Through the softmax operation, the scores are normalized into weights, and each weight represents the degree of attention the current element pays to other elements. The attention score matrix A after the softmax operation is used to weight the value vector V, thereby generating the final attention output. The formula is as follows: Attention(Q,K,V)=softmax(A)V. V is a value vector, usually representing the actual content or information of each element. Through this weighting process, the model weights each value vector V according to the probability values in the attention score matrix. Value vectors with higher weights have a larger proportion in the final output, and the model will pay more attention to these highly correlated parts.
[0099] The weighted output is then used as input to the feedforward network. The feedforward network performs a non-linear transformation on these weighted representations to further process the feature information at each location. A feedforward network typically consists of two linear layers and an activation function. The feedforward network is calculated using the following formula: FFN(h) = ReLU(hW1 + b1)W2 + b2, where W1 and W2 are the weight matrices of the linear transformation, respectively shifting the input from dimension d. modeld_ Transform into a larger dimension d ff And then convert back to the original dimension d modeld h represents the input feature vector; b1 and b2 represent the bias terms of the first and second layers, respectively, used to enhance the model's expressive power and convergence speed. When extended to larger dimensions, the feedforward network uses activation functions to perform non-linear transformations, making each transformation more than just linear. This helps the model capture complex patterns and features, improving its expressive power and thus enhancing its performance.
[0100] Furthermore, the results obtained after attention score calculation and feedback network transformation need to be normalized. Layer normalization reduces changes in the input distribution, avoids gradient instability caused by parameter initialization or activation functions, and maintains the stability of the training process. The layer normalization formula is as follows:
[0101]
[0102] Where x is the input vector, μ is the mean of the input vector x, σ is the standard deviation of the input vector x, and γ and β are trainable parameters used for scaling and translation operations, respectively, to restore the expressive power of the normalized output.
[0103] Preferably, in step S3, the classification layer is trained using the time-domain training results, and the cross-entropy loss is calculated, specifically as follows:
[0104] The time-domain training results are converted into a probability distribution using the Softmax function, thereby obtaining scores for each category and outputting the category with the highest score.
[0105] The cross-entropy loss is calculated based on the scores of each category.
[0106] Specifically, the features processed by the feedforward network are fed into the classification layer, whose task is to map the model's output to the final class labels. The classification layer's output typically uses the Softmax function to calculate the probability distribution for each class, and determines the final predicted label based on these probability values. The Softmax function transforms the feature vector output by the feedforward network into a probability distribution, ensuring that the probability value for each class is within the range [0,1], and that the sum of the probabilities of all classes is 1. For a classification task with C classes, assuming the output of the feedforward network is z = [z1, z2, ..., z...],... C The formula for calculating the Softmax function is:
[0107]
[0108] Among them, z c This represents the score of class c in the feedforward network output. This is an exponential function mapping for the category, enhancing the influence of categories with higher scores. The denominator is the exponential sum of the scores of all categories, ensuring that the sum of the output probabilities is 1. The eye-tracking category with the highest probability value is then selected for output.
[0109] Preferably, in step S4, during training, the Adam optimizer is used to tune the model's parameters to minimize the cross-entropy loss function and improve the model's classification performance. The cross-entropy loss measures the difference between two probability distributions, and its formula is:
[0110]
[0111] Among them, y c It is the one-hot encoding of the real label. If the real label is category c, then y c =1 otherwise 0. During training, the model adjusts the learning rate through backpropagation to minimize the cross-entropy loss. This means the model continuously updates its weights to reduce the gap between the predicted probability distribution and the true label. Ultimately, when the cross-entropy loss function reaches its minimum, the model's parameters are optimized for accurate classification. The Adam update rule is as follows:
[0112] Assume that at step t, the model parameters are θ. t The gradient is g t The Adam optimizer updates its parameters by following these steps: calculating the first moment m of the gradient. t and second moment v t :m t =β1m t-1 +(1-β1)g t , β1 and β2 are the decay rates of the first and second moments, typically set to 0.9 and 0.999, respectively. t The gradient is calculated at step t. Next, bias correction is performed:
[0113]
[0114] These two items are used to eliminate m in the early stages of optimization. t and v t The problem of bias towards zero. t and v t Biasing towards zero affects the learning rate, resulting in smaller parameter update increments and slower optimization. The final step is updating the parameters, i.e., the learning rate:
[0115]
[0116] θ t+1 α is the updated learning rate, α is the initial learning rate, which is a fixed value; ∈ is a small constant.
[0117] Preferably, in the process of training the classification layer using the time-domain training results, after each round of training is completed, the following steps are also included:
[0118] Calculate the training loss;
[0119] The trained frequency domain learner and the stacked layers are tested and validated using the test set and the validation set, and the test loss and validation loss are calculated.
[0120] If the verification loss does not decrease for three consecutive rounds, the early training stop mechanism is triggered, and the eye-tracking event classification model trained at this time is saved as the model with the best classification effect.
[0121] Specifically, after each training round, the model outputs three loss values: training loss, validation loss, and test loss. If the validation loss does not decrease for three consecutive rounds, an early stopping mechanism is triggered, and the optimal model from the training process is saved. The training loss evaluates whether the model has learned the data features; a smaller loss indicates better learning. The test loss evaluates the model's generalization ability; a smaller loss indicates that the model has not merely memorized the data but has understood its features. Accuracy is used to measure the recognition of three eye-movement event categories: fixation, saccades, and smooth tracking. The model's automatic classification results are compared with the original categories as shown below. Figure 3 As shown in the figure; the test results for the three subsets are shown in Table 1 below.
[0122] Table 1: Test Results Table
[0123] Subset / Recognition Rate (%) gaze Scan Smooth Tracking Breite_strasse 0.9976 0.9802 0.9615 Ducks_boat 0.9922 0.9629 0.9338 Doves 0.9987 0.9754 0.9623
[0124] Here, Breite_strasse, Ducks_boat, and Doves represent the names of three noisy subsets selected from the public Gazcom dataset, which has 16 subsets. These three subsets (i.e., three eye-tracking data sequences) have noise percentages of 4.31%, 5.56%, and 6.83%, respectively, and have been preprocessed for direct use in training and testing.
[0125] Based on the aforementioned eye-tracking event classification method based on time-domain-frequency domain fusion, this invention also provides an eye-tracking event classification device based on time-domain-frequency domain fusion.
[0126] An eye-tracking event classification device based on time-domain-frequency domain fusion includes a processor, a memory, and a computer program stored in the memory. When the computer program is executed by the processor, it implements the eye-tracking event classification method based on time-domain-frequency domain fusion as described above.
[0127] In this invention, an eye-tracking event classification method and apparatus based on time-domain-frequency-domain fusion, a frequency-domain learner is introduced to extract potential spectral features from eye-tracking signals, effectively suppressing noise interference and improving the stability and robustness of the model in complex experimental scenarios. Simultaneously, this invention integrates time-domain multivariate dynamic features with frequency-domain information and introduces a self-weighted attention mechanism to dynamically weight different types of features, enabling the model to more accurately distinguish eye-tracking event types such as gaze, saccades, and smooth tracking. Compared to traditional methods based on manual feature extraction and threshold rules, this invention can automatically complete frequency-domain and time-domain feature learning, reducing reliance on expert experience and significantly improving the development efficiency and application promotion capabilities of eye-tracking event recognition systems.
[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for classifying eye-tracking events based on time-domain-frequency domain fusion, characterized in that, include: S1, acquire time-based eye movement data sequences, and divide the eye movement data sequences into training set, test set and validation set; S2, Construct an improved iTransormer eye-tracking event classification model based on frequency domain-time domain feature fusion; wherein, the eye-tracking event classification model includes a frequency domain learner, a stacked layer, and a classification layer; S3, use the training set to train the frequency domain learner on frequency domain features and output the frequency domain training result; based on the self-attention mechanism, use the frequency domain training result to train the stacked layer on temporal features and output the temporal training result; use the temporal training result to train the classification layer on classification and calculate the cross-entropy loss. S4, determine whether the cross-entropy loss has reached the minimum; if not, use the Adam optimizer to adjust the model parameters of the eye-tracking event classification model and return to execute S3; if yes, save the eye-tracking event classification model trained at this time as the model with the best classification effect. S5. Classify the eye-tracking events using the model with the best classification effect to obtain the classification results; In step S3, the frequency domain learner is trained using the training set to perform frequency domain feature training, specifically as follows: S301, Use Fourier transform to transform the training set from the time domain to the frequency domain to obtain domain-transformed data; S302, the domain transformation data is input into the frequency domain learner for frequency domain learning, so as to iteratively update the weights and biases, and then obtain the real and imaginary parts of the frequency components. S303 combines the real and imaginary parts of the frequency components to form a complex number, thus obtaining the initial frequency domain training result; S304, Use inverse Fourier transform to convert the initial frequency domain training result from the frequency domain to the time domain to obtain the frequency domain training result; In step S3, the stacked layer is trained with temporal features using the frequency domain training results, specifically as follows: S305, each feature variable's time series in the frequency domain training result is independently encoded into a variable time point; S306, Calculate the correlation between different time points of the variables to generate a query, key, and value vector; S307, based on the attention mechanism, calculates the similarity between each pair of queries and keys to obtain an attention score, and then obtains the attention score matrix; S308, Perform a Softmax operation on the attention score matrix to obtain the attention weight matrix; S309, The attention weight matrix is used to perform weighted aggregation on the value vector to obtain attention data; S310, Perform layer normalization on the attention data to obtain layer normalized data; S311, Perform a nonlinear transformation on the layer normalized data to obtain the initial time-domain training results; S312, Perform layer normalization on the initial time-domain training result to obtain the time-domain training result; In step S3, the classification layer is trained using the time-domain training results, and the cross-entropy loss is calculated, specifically as follows: The time-domain training results are converted into a probability distribution using the Softmax function, thereby obtaining scores for each category and outputting the category with the highest score. The cross-entropy loss is calculated based on the scores of each category.
2. The eye-tracking event classification method based on time-domain-frequency domain fusion according to claim 1, characterized in that, In S1, the eye movement data sequence includes multiple eye movement data, each of which includes the recording time, X-axis coordinate value of eye position, Y-axis coordinate value of eye position, confidence level, first human expert judgment result of eye movement event, second human expert judgment result of eye movement event, comprehensive judgment result of eye movement event, eye movement velocity, eye movement acceleration, and eye movement direction.
3. The eye-tracking event classification method based on time-domain-frequency domain fusion according to claim 2, characterized in that, The recording time, the X-axis coordinate value of the eye position, and the Y-axis coordinate value of the eye position are obtained by tracking and measuring with an eye tracker and filtering out interference. The interference filtering includes screen anomaly filtering, flicker value filtering, and single-sample peak filtering.
4. The eye-tracking event classification method based on time-domain-frequency domain fusion according to claim 2, characterized in that, The eye movement velocity, eye movement direction, and eye movement acceleration of the first eye movement data in the eye movement data sequence are all 0. The eye movement velocity, eye movement direction, and eye movement acceleration of the eye movement data in the eye movement data sequence, excluding the first eye movement data, are respectively represented as follows: ; ; ; in, Indicates the first... Eye movement speed of each eye movement data point ; and They represent the first The X-axis coordinates and Y-axis coordinates of the eye position for each eye movement data point; Indicates the first Each eye-tracking data recording moment; Indicates the first Eye movement direction of each eye movement data point; Indicates the first Eye movement acceleration data for each eye movement.
5. The eye-tracking event classification method based on time-domain-frequency domain fusion according to claim 1, characterized in that, In S311, a feedforward network is specifically used to perform a nonlinear transformation on the attention data. The feedforward network includes a first linear layer, a first activation function, and a second linear layer. The attention data is nonlinearly transformed using a feedforward network, specifically as follows: The attention data is subjected to a linear transformation that increases its dimensionality using the first linear layer to obtain the first linear transformation result; The first linear transformation result is subjected to a nonlinear transformation using the first activation function to obtain a nonlinear transformation result. The nonlinear transformation result is subjected to a linear transformation to restore the dimension, thereby obtaining the time-domain training result.
6. The eye-tracking event classification method based on time-domain-frequency domain fusion according to claim 1, characterized in that, In the process of training the classification layer using the time-domain training results, after each round of training, the following steps are also included: Calculate the training loss; The trained frequency domain learner and the stacked layers are tested and validated using the test set and the validation set, and the test loss and validation loss are calculated. If the verification loss does not decrease for three consecutive rounds, the early training stop mechanism is triggered, and the eye-tracking event classification model trained at this time is saved as the model with the best classification effect.
7. An eye-tracking event classification device based on time-domain-frequency domain fusion, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory, wherein the computer program, when executed by the processor, implements the eye-tracking event classification method based on time-domain-frequency-domain fusion as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Audio processing with neural networks
CN109074820A
Image three-classification decoding method and device based on brain hole and eye movement characteristics
CN114998641A
