Transformer-Based Eye Movement Event Detection Method
Through the Transformer-based eye movement event detection method, the global features of long sequences are established using CNN and Self-Attention mechanisms, which solves the shortcomings of LSTM in establishing long sequence dependency relationships and parallel processing, and achieves more efficient eye movement event detection.
Patent Information
- Application Number
- CN202210845526.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-18
- Publication Date
- 2025-07-25
- Estimated Expiration
- 2042-07-18
AI Technical Summary
The existing LSTM-based eye movement event detection method is difficult to effectively establish a long-term dependency between observations at different times on long sequences and cannot achieve high parallelization of sequence processing, resulting in limited detection performance.
Using Transformer-based eye movement event detection method, the characteristics of the line of sight position sequence are extracted through the CNN network, combined with position coding and Self-Attention mechanism, the global characteristics of the eye movement sequence are learned, and the dependency between the current observation and observation at different times is established, and classification is carried out through Softmax.
It improves the accuracy and efficiency of eye movement event detection, realizes high parallelization of sequence processing, and improves the performance of detection algorithms.
Smart Images
Figure CN115410266B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of eye movement event detection, and relates to an eye movement event detection method based on Transformer. Background Art
[0002] The purpose of eye movement event detection is to accurately and robustly extract eye movement events such as fixation, saccade, and smooth pursuit from the original observation video of an eye tracker. For a long time, there have been two types of algorithms applied in this direction, one is the velocity-based algorithm and the other is the dispersion-based algorithm. Velocity-Threshold Identification (I-VT) was proposed by Bahill et al. (1981), and its principle can be traced back to the Boyce algorithm in the 1960s. By setting a velocity threshold, those above the threshold are classified as saccades, and those below the threshold are classified as fixations. Dispersion-Threshold Identification (I-DT) was proposed by Salvucci et al. (2000), and its principle is to use a moving window spanning consecutive sample points to detect potential fixations. Since the low-speed characteristic of fixations will make their sample points tend to cluster closely, the events within the window below the dispersion threshold are classified as fixations. These two methods rely on manually designed eye movement features (such as eye position, velocity, acceleration, etc.), and it is difficult to find a universal threshold for different subjects. The detection performance is not only limited by the accuracy of the feature extraction method, but also difficult to handle the simultaneous extraction of multiple eye movement events.
[0003] In recent years, machine learning methods have been preliminarily applied to eye movement event detection based on line-of-sight position. Pekkanen (2017) et al. proposed a naive piecewise linear regression method to denoise and segment the line-of-sight position time series simultaneously, and then used a hidden Markov classifier to classify eye movement events into four categories (fixation, saccade, smooth pursuit, and post-saccadic oscillation). Zemblys (2018) et al. first extracted 14 features from the original line-of-sight position time series (the data lengths used for different features are different), then used a random forest classifier to perform multi-classification on each sampling point, and finally integrated the classification results based on rules to achieve eye movement event detection and classification. The above two methods still use manually designed features, but only use machine learning methods in the classifier design.
[0004] With the rapid development of deep learning methods, Hoppe and Bulling (2016) proposed an end-to-end model based on convolutional neural network to simultaneously detect different eye movement events from continuous eye movement sequences, and its model performance is better than that of I-VT and I-DT algorithms. Startsev et al. (2018) proposed a 1D-CNN-BLSTM network, which was evaluated on the GazeCom dataset and compared with 12 reference algorithms. The experimental results show that its performance is better than other algorithms. Zemblys et al. (2019) further proposed an end-to-end eye movement event detection method based on deep learning (GazeNet). This method first uses two one-dimensional convolutional layers to extract the features of the original line-of-sight position time series, and then uses a three-layer bidirectional long short-term memory network (Long Short Term Memory network, LSTM) plus a fully connected layer to classify each sampling point.
[0005] Currently, the eye movement event detection methods based on deep learning generally use LSTM and its variants as the backbone network. In LSTM, the observation at the current moment only strongly affects the feature representation at the next moment, and its influence quickly disappears after several time steps. Therefore, LSTM cannot effectively establish long-term dependencies between observations at different moments on long sequences; in addition, LSTM is a sequential model, which means that sample points are processed in order and cannot achieve a high degree of parallelization of sequence processing. Summary of the Invention
[0006] The purpose of the present invention is to provide an eye movement event detection method based on Transformer, which solves the limitations of the existing LSTM-based deep eye movement event detection methods on the performance of the detection algorithm, where LSTM cannot effectively establish long-term dependencies between observations at different moments on long sequences and cannot achieve a high degree of parallelization of sequence processing.
[0007] The technical solution adopted by the present invention is as follows:
[0008] An eye movement event detection method based on Transformer, comprising the following steps:
[0009] Step 1: The input data is a public eye movement event detection dataset, and the original line-of-sight position sequence data is preprocessed to obtain a differential line-of-sight position sequence;
[0010] Step 2: The differential line-of-sight position sequence is sent into a CNN network for feature extraction, and the output of the CNN network is used as the word vector Word Embedding;
[0011] Step 3: Perform positional encoding Positional Encoding on the line-of-sight position sequence to represent the front and back position information of the line-of-sight position sequence;
[0012] Step 4: After adding the word vectors and the positional encodings, f is obtained. t Feed it into the Encoder layer of the Transformer. After passing through N layers of Encoders, the information covering the entire sequence is learned. Finally, through a linear fully connected layer and Softmax, each sample point in the eye movement sequence is classified into fixation, saccade, and post-saccadic oscillation, realizing eye movement event detection.
[0013] Step 5: Use event-level Cohen’s Kappa to evaluate the performance of the three classified eye movement events.
[0014] The features of the present invention also lie in:
[0015] The specific preprocessing in Step 1 includes:
[0016] Step 1.1: For the input data set, remove other eye movement events except fixation, saccade, and post-saccadic oscillation.
[0017] Step 1.2: Then convert the line-of-sight position (x, y) represented by pixels into the line-of-sight position (x s , y s ) represented by spatial angles.
[0018] Step 1.3: Perform a difference operation on the input line-of-sight position sequence. Subtract the position of the previous moment from the position of the next moment to obtain the difference sequence of the line-of-sight position as the input of the CNN network.
[0019] The calculation formula for Step 1.2 is as follows:
[0020]
[0021]
[0022] where x is the horizontal coordinate, y is the vertical coordinate, p w is the number of pixels in the width direction of the display in the foreground camera image, p h is the number of pixels in the height direction of the display in the foreground camera image, s w is the screen width, s h is the screen height, and d is the distance between the eyes and the screen.
[0023] The line-of-sight position sequence in Step 1.3 is expressed as: [(x s1 , y s1 ), (x s2 , y s2 ), (x s3 , y s3 ), …, (x sm , ysm ), (x s(m+1) , y s(m+1) )] a total of m + 1 sample points. The sequence after differencing is represented as: A total of m sample points, where the differencing calculation formula is:
[0024]
[0025]
[0026] Among them, x s(m+1) and y s(m+1) represent the coordinate values of the sample point at the (m + 1)-th moment of the original line-of-sight position sequence, and x sm and y sm represent the coordinate values of the sample point at the m-th moment of the original line-of-sight position sequence. and represent the coordinate values of the sample point at the m-th moment of the sequence after differencing.
[0027] The convolutional kernel size adopted by the CNN network in step 2 is 2×11, which is used to extract spatio-temporal local features from the input data and abstract them into high-dimensional features through convolution. The output feature vector serves as the word vector of the eye movement sequence.
[0028] The positional encoding in step 3 provides the position information of each sample point in the sequence to the Transformer to identify the sequential relationship in the sequence, and uses the linear transformation of the sin and cos functions to provide the position information to the model:
[0029]
[0030]
[0031] Among them, pos refers to the position of a certain sample point in the sequence, i refers to the dimension serial number of the word vector, and d model refers to the dimension of the word vector.
[0032] The parameters of the Transformer layer in step 4 are 4 Encoder layers. Each Encoder layer is divided into two sub-layers. The first layer uses the Self-Attention mechanism, and the second layer uses a feed-forward neural network layer. Among them, the dimension d model of the word vector is 64 dimensions, and the dimension d ff of the feed-forward neural network is 256 dimensions;
[0033] Among them, the Self-Attention mechanism defines three matrices W Q , W K , W V, these three matrices are used to perform three linear transformations on all word vectors respectively, and thus three new vectors q are derived from all the word vectors t , k t , v t . All the q t vectors are assembled into a large matrix, denoted as the query matrix Q. All the k t vectors are assembled into a large matrix, denoted as the key matrix K. All the v t vectors are assembled into a large matrix, denoted as the value matrix V; Next, Q and K T are multiplied, and then divided by . After passing through Softmax, it is multiplied by V to obtain the output of Self-Attention. The specific formula implementation is as follows:
[0034]
[0035] where Z is the output of Self-Attention, Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key matrix.
[0036] The beneficial effects of the present invention are:
[0037] This method uses CNN and Transformer to detect eye movement events, integrating the advantages of CNN and Transformer. The CNN layer is used to learn the spatio-temporal local features of the eye movement sequence and abstract them into high-dimensional features through convolution as the input word vectors of the Transformer. The Transformer learns the global features of the eye movement sequence. Compared with the existing method using LSTM, the Transformer realizes a high degree of parallelism in processing with the entire sequence as the input unit, and establishes the dependence relationship between the current observation and the observations at different times in the input sequence through the Self-Attention mechanism, enabling the model to not only focus on the relationship between the current observation and the observations at adjacent times, but also focus on the dependence relationship between the current observation and the observations at different times in the sequence. In addition, the multi-head attention mechanism and position encoding of the Transformer both provide information about the relationship between different sample points. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 is the overall framework flowchart of the present invention;
[0039] Figure 2 is the network structure diagram of the Encoder layer of the Transformer. DETAILED DESCRIPTION OF THE INVENTION
[0040] The present invention will be described in detail below in conjunction with the drawings and specific embodiments.
[0041] As Figure 1 shown, a method for detecting eye movement events based on Transformer provided by an embodiment of the present invention includes the following steps:
[0042] Step 1, preprocessing of the eye movement sequence.
[0043] The input data uses the Lund2013 eye movement dataset, which includes timestamp, horizontal coordinate x, vertical coordinate y, eye movement event type event, screen width s w , screen height s h , number of pixels p in the width direction of the display in the foreground camera image w , number of pixels p in the height direction of the display in the foreground camera image h and the distance d between the eyes and the screen. First, other eye movement events except fixation, saccade, and post-saccadic oscillation need to be removed, so that the training data only contains three events: fixation, saccade, and post-saccadic oscillation for training and testing. The specific method is to first process the eye movement event type event, mark the three events to be detected as True, and mark other events as False. Then, segmentally remove the sample points of the eye movement sequence with the label False, and only retain the segmented eye movement sequence marked as True.
[0044] Then, convert the line-of-sight position (x, y) represented by pixels to the line-of-sight position (x s , y s ) represented by spatial angles. The conversion formula is as follows:
[0045]
[0046]
[0047] where x is the horizontal coordinate, y is the vertical coordinate, p w is the number of pixels in the width direction of the display in the foreground camera image, p h is the number of pixels in the height direction of the display in the foreground camera image, s w is the screen width, s h is the screen height, and d is the distance between the eyes and the screen.
[0048] Finally, perform a difference operation on the input line-of-sight position sequence, subtract the position of the previous moment from the position of the next moment to obtain the difference sequence of the line-of-sight position as the input of the CNN network. The original line-of-sight position sequence is represented as [(x s1 , y s1 ), (x s2 , y s2 ), (x s3 , ys3 ), …, (x sm , y sm ), (x s(m+1) , y s(m+1) ), a total of m + 1 sample points. The sequence after differencing is denoted as a total of m sample points, where the differencing calculation formula is:
[0049]
[0050]
[0051] where x s(m+1) and y s(m+1) represent the coordinate values of the sample point at the (m + 1)-th moment of the original line-of-sight position sequence, and x sm and y sm represent the coordinate values of the sample point at the m-th moment of the original line-of-sight position sequence. and represent the coordinate values of the sample point at the m-th moment of the sequence after differencing.
[0052] Step 2: Use a CNN network to extract features from the line-of-sight position sequence.
[0053] The CNN network adopts a convolutional layer with a convolutional kernel size of 2×11 for 6 layers to extract spatio-temporal local features from the input data, and abstract them into high-dimensional features through convolution. The output feature vector is used as the word vector (WordEmbedding) of the eye movement sequence, and this word vector is used as the input of the Transformer.
[0054] Step 3: Perform positional encoding (Positional Encoding) on the line-of-sight position sequence.
[0055] Positional encoding is used because the Transformer model does not have the iterative operation of the recurrent neural network, so it is necessary to provide the position information of each sample point in the sequence to the Transformer so that it can recognize the sequential relationship in the sequence. Use the linear transformation of the sin and cos functions to provide position information to the model:
[0056]
[0057]
[0058] where PE represents positional encoding, pos refers to the position of a certain sample point in the sequence, f refers to the dimension number of the word vector, and d model refers to the dimension of the word vector.
[0059] Step 4: Use a Transformer to learn information covering the entire sequence.
[0060] After adding the word vectors and the positional encodings, we get f t Feed it into the Encoder layer of the Transformer. The Encoder layer of the Transformer is divided into two sub-layers, as Figure 2 shown. The first layer uses the Self-Attention mechanism, enabling the encoder to focus on the relationships with other sample points when encoding a specific sample point. The second layer uses a feed-forward neural network layer. The Self-Attention mechanism defines three matrices W Q 、W K 、W V . Use these three matrices to perform three linear transformations on all the word vectors respectively. Thus, all the word vectors derive three new vectors q t 、k t 、v t . Concatenate all the q t vectors into a large matrix, denoted as the query matrix Q. Concatenate all the k t vectors into a large matrix, denoted as the key matrix K. Concatenate all the v t vectors into a large matrix, denoted as the value matrix V. Next, multiply Q and K T and then divide by . After passing through Softmax, multiply by V to obtain the output of Self-Attention. The specific formula implementation is as follows:
[0061]
[0062] where Z is the output of Self-Attention, Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the dimension of the key matrix.
[0063] Furthermore, the output of Self-Attention is connected in a residual manner with the input f t . After passing through LayerNorm, it is output to the feed-forward neural network layer. Finally, the output of this layer's Encoder is used as the input to the next Encoder layer. After passing through N layers of Encoders, information covering the entire sequence is learned. Finally, through the fully connected layer and Softmax, the probabilities of three events are output. The event with the highest probability is the predicted type of eye movement event, thus realizing eye movement event detection.
[0064] The Transformer layer parameters adopted in the present invention are 4 layers of Encoders, d model is 64-dimensional, and the dimension d ffIt is 256 - dimensional, and the dimensions of the three matrices Q, K, and V are all 32 - dimensional. The number n of the multi - head attention mechanism heads is 2.
[0065] Step 5: Use event - level Cohen’s Kappa to evaluate the performance of the three classified eye movement events.
[0066] Example 1
[0067] The test set used in this Example 1 is 22 eye movement sequences divided from the Lund2013 dataset. After performing the above Steps 1 to 5, the event - level Cohen’s Kappa scores for fixation, saccade, and post - saccadic oscillation on this test set are 0.923, 0.925, and 0.782 respectively.
[0068] Table 1: Evaluation scores of each eye movement sequence on the test set.
[0069]
[0070] Table 1 shows the evaluation scores of each eye movement sequence on the test set. Ke_Fixation represents the event - level Cohen’s Kappa score for fixation, Ke_Saccade represents the event - level Cohen’s Kappa score for saccade, and Ke_PSOs represents the event - level Cohen’s Kappa score for post - saccadic oscillation.
Claims
1. A Transformer-based eye movement event detection method, characterized in that: It includes the following steps: Step 1: The input data is a publicly available eye movement event detection dataset. The original line-of-sight position sequence data is preprocessed to obtain a differential line-of-sight position sequence; Step 2: The differential line-of-sight position sequence is fed into a CNN network for feature extraction, and the output of the CNN network is used as the word vector Word Embedding; Step 3: Positional Encoding is performed on the line-of-sight position sequence to represent the front and back position information of the line-of-sight position sequence; Step 4: After adding the word vectors and the positional encodings, we obtain and send it into the Encoder layer of the Transformer. After passing through N layers of Encoders, global information covering the entire sequence is learned. Finally, through a linear fully-connected layer and Softmax, each sample point in the eye movement sequence is classified into fixation, saccade, and post-saccadic oscillation, thus realizing eye movement event detection. The Transformer layer parameters described in step 4 are 4 Encoder layers. Each Encoder layer is divided into two sub-layers. The first layer uses the Self-Attention mechanism, and the second layer uses a feed-forward neural network layer. The dimension of the word vector is 64 dimensions, and the dimension of the feed-forward neural network is 256 dimensions; Among them, the Self-Attention mechanism defines three matrices , , . Using these three matrices to perform three linear transformations on all word vectors respectively, thus all word vectors derive three new vectors , , . Concatenate all the vectors into a large matrix, denoted as the query matrix . Concatenate all the vectors into a large matrix, denoted as the key matrix . Concatenate all the vectors into a large matrix, denoted as the value matrix ; Next, multiply and , then divide by , and after Softmax, multiply by to get the output of Self-Attention. The specific formula implementation is as follows: Among them is the output of Self-Attention, is the query matrix, is the key matrix, is the value matrix, is the dimension of the key matrix; Step 5: Event-level Cohen’s Kappa is used to evaluate the performance of the three classified eye movement events.
2. The method for detecting eye movement events based on Transformer according to claim 1, characterized in that: The preprocessing described in Step 1 specifically includes: Step 1.1, for the input dataset, other eye movement events except fixation, saccade, and post-saccadic oscillation are excluded; Step 1.2, and then convert the line-of-sight position represented in pixels to the line-of-sight position represented in spatial angles ; Step 1.3, a differential operation is performed on the input line-of-sight position sequence, and the position of the next moment is subtracted from the position of the previous moment to obtain the differential sequence of the line-of-sight position as the input of the CNN network.
3. The method for detecting eye movement events based on Transformer according to claim 2, wherein: The calculation formula of Step 1.2 is as follows: Among them, is the horizontal coordinate, is the vertical coordinate, is the number of pixels in the width direction of the display in the foreground camera image, is the number of pixels in the height direction of the display in the foreground camera image, is the screen width, is the screen height, is the distance between the eyes and the screen.
4. The method for detecting eye movement events based on Transformer according to claim 2, wherein: The line-of-sight position sequence in step 1.3 is expressed as: A total of sample points, and the sequence after differentiation is expressed as: A total of sample points, where the differentiation calculation formula is: wherein and represent the coordinate values of the sample point at the th moment of the original line-of-sight position sequence, and represent the coordinate values of the sample point at the th moment of the original line-of-sight position sequence, and represent the coordinate values of the sample point at the th moment of the differentiated sequence.
5. The method for detecting eye movement events based on Transformer according to claim 1, wherein: The CNN network described in Step 2 uses a convolution kernel size of 2×11 to extract spatio-temporal local features from the input data, and abstracts them into high-dimensional features through convolution. The output feature vector is used as the word vector of the eye movement sequence.
6. The method for detecting eye movement events based on Transformer according to claim 1, wherein: The positional encoding described in Step 3 provides the position information of each sample point in the sequence to the Transformer to identify the sequential relationship in the sequence, and uses the linear transformation of the sin and cos functions to provide the position information to the model: where refers to the position of a sample point in the sequence, refers to the dimension serial number of the word vector, refers to the dimension of the word vector.