A real-time eye movement signal classification method based on temporal convolutional attention network

Abstract

The present invention belongs to the field of eye movement analysis technology and relates to a real-time eye movement signal classification method based on a temporal convolutional attention network (TCAN). The multi-scale feature extraction method proposed in the present invention uses eye movement feature sequences at multiple different time scales as input to a network model, reflecting the characteristics of eye movement changes at different time scales in the temporal and spatial dimensions, so that eye movement features are clearly distinguished between different eye movement types, thereby increasing the degree of differentiation between different eye movement types. The fixed-length eye movement sequence data is transmitted to a data preprocessing module in real time through a queue cache method, and then the multi-time-scale eye movement feature sequence is extracted and input into a pre-trained eye movement signal classification network model, and the eye movement classification result at the current moment is output in real time, thereby achieving real-time eye movement classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of eye movement analysis and relates to a real-time eye movement signal classification method based on a temporal convolutional attention network (TCAN). Background Art

[0002] The eyes are the primary channel for acquiring visual information. The acquired eye movement signals can reflect a person's eye movement behavior, attention state, and, at a higher level, cognitive state. Common eye movement signals can be divided into three types: fixation, saccade, and smooth pursuit.

[0003] Currently, numerous researchers are applying eye movement research to various fields, such as psychology, advertising design, manufacturing, remote control, and computer interaction. The analysis and application of eye movement is no longer limited to offline data analysis. Compared to analyzing and applying only raw eye movement data, more powerful real-time eye movement signal classification technology can provide a more diverse data type in eye tracking applications and in research on the understanding of perceptual processes. This facilitates the design of more novel interaction methods and improves human understanding of perceptual processes. Therefore, developing a real-time eye movement classification method is extremely important.

[0004] Currently, most eye movement classification methods available on the market are offline algorithms, which fall into two main categories: one is based on traditional threshold classification methods, including those based on velocity thresholds and those based on discreteness thresholds; the other is based on machine learning, which primarily utilizes classic algorithms to train classification models, such as decision trees, hidden Markov models (HMMs), and Bayesian decision tree algorithms (I-BDTs). With the rapid development of deep learning in recent years, a number of models using deep learning for eye movement classification have emerged. For example, the one-dimensional convolutional neural network and bidirectional long short-term memory network (1DCNN+BLSTM) proposed by Startsev et al. have achieved good classification accuracy in offline eye movement classification tasks based on the Gazecom dataset.

[0005] However, with the development of eye movement technology, offline eye movement classification methods have become unable to meet the changing market demand. Some researchers have begun to study online eye movement classification algorithms, but to date, relevant research remains limited. In 2016, Thiago Santini et al. proposed a Bayesian Decision Theory Identification (I-BDT) algorithm for real-time eye movement classification. However, the dataset sample rate used by this method was very low, and the time interval between adjacent sample points was large, resulting in insufficient precision in the classification of different eye movement types. In 2019, Liu Xin of Nanjing University designed a real-time eye movement recognition algorithm based on traditional threshold classification. It can identify three types of eye movement behaviors in real time: gaze, gesture, and conscious blink, and applied it to eye control systems. The proposed method demonstrated that real-time eye movement classification technology can provide a novel interactive method for eye control systems. However, this method treats points within the target area as fixation points and points outside the target area as saccade points. As a result, the classification of eye movement types depends on the size of the target area, and the classification accuracy is limited by the threshold setting, resulting in poor universality. In 2020, Carlos Elmadjian and others proposed an eye movement classification model based on a temporal convolutional neural network (TCN). Their real-time eye movement classification method at the sample point level provides more diverse data types for research on eye movement interaction, intent recognition, and decision-making. However, the algorithm still has the problem of low classification accuracy of smooth trailing, and the method for achieving real-time classification has not been made public. Since smooth trailing is a relatively important type of eye movement behavior, the smooth trailing data can be used to identify and track significant moving targets, and the spatial position of moving targets can also be predicted. Therefore, achieving real-time, high-precision smooth trailing recognition is also of great significance for research on intent recognition, decision-making, etc. In summary, proposing a real-time eye movement classification method with high smooth trailing accuracy has become an important problem to be solved in the field of eye movement. Summary of the Invention

[0006] In order to solve the problems of low smooth-tailing classification accuracy in existing eye movement signal classification algorithms and the lack of real-time classification methods, the present invention proposes a real-time eye movement signal classification method based on the Temporal Convolutional Attention Network (TCAN). The method can classify the user's eye movement signals into three categories in real time: fixation, saccade, and smooth-tailing. The accuracy of the eye movement classification model proposed in the present invention is higher than that of existing eye movement classification algorithms.

[0007] The technical solutions for implementing the present invention are as follows:

[0008] A real-time eye movement signal classification method based on a temporal convolutional attention network, the method comprising the following steps:

[0009] Step 1: Obtain the original eye movement data; obtain the user's gaze point coordinates (x i ,y i ) and the current time ti ;

[0010] Step 2, real-time data transmission; the raw eye movement data acquired in step 1 is transmitted and cached in the message queue, and the length of the message queue is set to l = w + k max , where w represents the size of the fixed sliding window, k max Indicates the maximum time scale; when the message queue is full, the sequence data Q of fixed length l in the message queue is transmitted in real time and preprocessed; then the data at the head of the message queue is removed to leave space for the queue to cache the eye movement data of the next moment;

[0011] Q={(x i-l+1 ,y i-l+1 , t i-l+1 ), (x i-l+2 ,y i-l+2 , t i-l+2 ),...,(x i ,y i , t i )}

[0012] The preprocessing refers to performing a sliding average filtering process on the sequence data Q of length l transmitted in real time from the message queue to remove the random noise of eye movement and obtain filtered eye movement data;

[0013] Step 3, multi-time scale feature extraction: extract multi-time scale eye movement features from the filtered eye movement data according to multiple different time scales k, where the value of k is 1, 2, ..., k max sample points, k max Indicates the maximum time scale, calculate t i The multi-time scale characteristic method of the sample point at time t is: calculate t i-k Sample point data at time (x i-k ,y i-k , t i-k ) and t i Sample point data at time (x i ,y i , t i ), the eye movement features mainly include: displacement dis, direction θ, velocity v, acceleration acc, velocity standard deviation std, and finally the different eye movement features extracted at different time scales are combined as t i The eye movement feature combination at time t is calculated in sequence according to the above method. i-w+1 ~t i The multi-time-scale eye movement feature combination corresponding to the moment, the final data with a length of l will output a multi-time-scale eye movement feature sequence with a length of w after multi-time-scale feature extraction;

[0014] Step 4: Real-time eye movement signal classification: Input the multi-time scale eye movement feature sequence of length w into the pre-trained eye movement signal classification network model, and then output the current time t i Eye movement classification results

[0015] Furthermore, the multi-timescale feature extraction method proposed in this invention calculates eye movement features at different time scales: displacement dis, direction θ, velocity v, acceleration acc, and velocity standard deviation std. Compared to existing feature extraction methods, the eye movement feature proposed in this invention comprehensively considers the temporal and spatial characteristics of eye movement data and calculates eye movement features at multiple time scales, rather than just considering eye movement characteristics at a single time scale. This allows for clear distinctions between different eye movement types, laying a solid foundation for high-precision eye movement classification.

[0016] Calculate t i The calculation method of eye movement features at the time scale k is as follows:

[0017] (1) Calculate the displacement characteristics dis i，k

[0018] Δx=x i -x i-k

[0019] Δy=y i -y i-k

[0020]

[0021] Among them, x i Indicates the horizontal coordinate of the gaze point at the current moment, x i-k Indicates the horizontal coordinate of the gaze point k sample points before the current moment, y i Indicates the vertical coordinate of the gaze point at the current moment, y i-k represents the ordinate of the gaze point at an interval of k sample points before the current moment, Δx represents the horizontal displacement of the gaze point at time scale k, and Δy represents the vertical displacement of the gaze point at time scale k;

[0022] (2) Calculate the directional feature θ i，k

[0023] θ i，k =atan(Δy, Δx)

[0024] (3) Calculate the velocity characteristic v i，k

[0025]

[0026] Where Δt = t i -t i-k , t i-k It represents the moment k sample points before the current moment. The positive or negative sign of the velocity represents its direction, which is determined by the horizontal displacement Δx and the vertical displacement Δy.

[0027] (4) Calculate the acceleration characteristic acc i，k

[0028]

[0029] in,

[0030]

[0031]

[0032] v x(i，k) =v i，k *cos(θ i，k ), v y(i，k) =v i，k *sin(θ i，k )

[0033] v x(i-k，k) =v i-k，k *cos(θ i-k，k ), v y(i-k，k) =v i-k，k *sin(θ i-k，k )

[0034] v i-k，k Indicates t i-k Velocity eye movement characteristics at time scale k, θ i-k，k Indicates t i-k The directional eye movement characteristics at the time scale of k; the positive or negative acceleration represents its direction, which is determined by t i The horizontal component of acceleration acc at time scale k x(i，k) and t i The vertical component of acceleration acc at time scale k y(i，k) Joint decision.

[0035] (5) Calculate the speed standard deviation std i，k

[0036]

[0037] Among them, v i-k+j，k Indicates t i-k+j Velocity eye movement characteristics at time scale k, t i-k+j represents the time interval kj sample points before the current time, Indicates that from t i-k Time to t i The arithmetic mean of the speed eye movement characteristics corresponding to the time scale k is:

[0038] Furthermore, the pre-trained eye movement signal classification network model described in the present invention refers to a pre-trained offline eye movement classification network model; this network model is mainly based on the structure of the Temporal Convolutional Attention Network (TCAN) using causal convolution, which realizes parallel processing of data while only using the input before the current moment for classification decisions; at the same time, compared with LSTM and RNN, it takes up less memory and can better realize lightweight real-time eye movement classification. In addition, the present invention adds an attention mechanism on the basis of each layer of TCN structure, which can capture the relationship between internal sequence features, and the multi-layer TCAN structure can obtain a sufficiently large receptive field, thereby obtaining the global information of the input sequence, and then matching it with a one-dimensional convolutional layer (1DCNN) to identify local details, and finally output the classification results of the sample points at each moment through the fully connected layer and the softmax classifier. High-precision eye movement signal classification can be achieved. The data input of the offline eye movement classification network model is the eye movement feature sequence of different time scales after data preprocessing and multi-scale feature extraction, and the output is the eye movement classification result label of the same length as the input, including fixation, saccade and smooth trailing; when performing real-time eye movement classification, the input network model is the t of length w after real-time processing. i-w+1 ~t i The model outputs the eye movement classification result label of the same length as the input, where the last label is t i The eye movement label corresponding to each moment is the result of real-time classification.

[0039] Furthermore, the TCAN structure described in the present invention is mainly composed of a self-attention structure, two dilated causal convolution layers, and a residual structure using a 1×1 convolution layer. The self-attention structure is located before the first dilated causal convolution to obtain information between input feature sequences of a fixed sliding window size. The global association weight is used to derive the weighted information of a single feature vector in the global context, reflecting the varying degrees of global importance of feature sequences at different times, thereby focusing the network's "attention" on the most important areas. The attention mechanism formula used by the self-attention structure is as follows:

[0040]

[0041] Where Q = XW Q , K=XW K , V=XWy , X represents the input sequence, W Q 、W K 、W V is the weight matrix, d k is the length of the input sequence; mask is used The purpose of this processing is to ensure that the self-attention structure only uses past information when processing sequence tasks. The mask formula is as follows:

[0042]

[0043] Among them, m=i-w+1, i-w+2,...,i, n=i-w+1, i-w+2,...,i.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] First, this invention provides a real-time eye movement signal classification network model. By adopting a causal convolutional temporal convolutional attention network (TCAN) structure, it can parallelize eye movement data processing, accelerate eye movement classification, and complete real-time eye movement classification without relying on future data. In addition, the self-attention structure introduced in each TCN layer can better capture the correlation between data and improve the model's classification accuracy.

[0046] Second, the present invention provides a method for realizing real-time eye movement classification, which transmits fixed-length eye movement sequence data to a data preprocessing module in real time through a queue caching method, then extracts multi-time-scale eye movement feature sequences and inputs them into a pre-trained eye movement signal classification network model, and outputs the eye movement classification results at the current moment in real time, thereby realizing real-time eye movement classification.

[0047] Third, the multi-scale feature extraction method proposed in the present invention calculates eye movement feature sequences at multiple different time scales as input to the network model, reflecting the characteristics of eye movement changes at different time scales in the time and space dimensions, making the eye movement features clearly distinguishable between different eye movement types, thereby increasing the discrimination between different eye movement types.

[0048] Fourth, compared with existing technologies, this invention solves the problems of low smooth-tail classification accuracy and limited real-time classification methods in existing deep learning-based eye movement signal classification methods. Furthermore, the classification results of user eye movement signals collected in real time are superior to existing models that can be used for real-time eye movement classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flow chart for implementing real-time eye movement signal classification in the present invention;

[0050] Figure 2 Schematic diagram of eye movement feature sequence extraction at different time scales in the present invention;

[0051] Figure 3 A network framework diagram for a specific implementation of the present invention;

[0052] Figure 4 This is a comparison table of the accuracy (F1-score) of different models based on the GazeCom dataset;

[0053] Figure 5 Schematic diagram of the internal structure of TCAN in the present invention. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the present application, the following will further describe in detail the implementation of the method of the present invention in conjunction with the accompanying drawings and specific implementations.

[0055] This paper proposes a real-time eye movement signal classification method based on a temporal convolutional attention network (TCAN). This method aims to classify collected eye movement signals into fixations, saccades, and smooth followings in real time using an eye movement classification model. In this example, the hardware required is a Tobbi Eye Tracker 4C eye tracker to collect user eye movement data, and the software environments used are Unity3D and PyCharm.

[0056] The flowchart of the present invention for realizing real-time eye movement signal classification is shown in the attached figure. Figure 1 As shown, the specific implementation steps are:

[0057] Step 1: Obtain raw eye movement data; use an eye tracker to collect the user's gaze point coordinates (x i ,y i ) and the current time t i Since the sampling frequency of the Tobii eye tracker is inconsistent with the data set, the extracted raw eye movement data needs to be linearly interpolated to increase the eye movement details before real-time data transmission.

[0058] Step 2: Real-time data transmission: First, the eye movement data obtained in step 1 needs to be transmitted in real time through the UDP protocol and cached in the message queue. The length of the message queue is set to l = w + k max , where w represents the size of the fixed sliding window, k maxIndicates the maximum time scale; in this embodiment, if the fixed sliding window size is set to 257 sample points and the maximum time scale is 128 sample points, the length of the message queue should be 385 sample points. When the message queue is full, the sequence data Q of fixed length l in the queue is preprocessed; then the data at the head of the message queue is removed to make room for the queue to cache the eye movement data of the next moment;

[0059] Where Q = {(x i-l+1 ,y i-l+1 , t i-l+1 ), (x i-l+2 ,y i-l+2 , t i-l+2 ),...,(x i ,y i , t i )}

[0060] Therefore, in order to achieve high-precision real-time eye movement classification, it is necessary to first collect eye movement data for a certain period of time (queue length) for eye movement classification.

[0061] The preprocessing refers to performing a sliding average filtering process on the sequence data Q of length l transmitted in real time from the message queue to remove the random noise of eye movement and obtain filtered eye movement data;

[0062] Step 3: Multi-time scale feature extraction: Extract multi-time scale eye movement features from the filtered eye movement data according to different time scales k, where the values ​​of k are 1, 2, 4, 8, 16, 32, 64, and 128 sample points. max Indicates the maximum time scale, calculate t i The multi-time scale characteristic method of the sample point at time t is: calculate t i-k Sample point data at time (x i-k ,y i-k , t i-k ) and t i Sample point data at time (x i ,y i , t i ), the eye movement features mainly include: displacement dis, direction θ, velocity v, acceleration acc, velocity standard deviation std, and finally the different eye movement features extracted at different time scales are combined as t i The eye movement feature combination at time t is calculated in sequence according to the above method. i-w+1 ~t i The multi-time-scale eye movement feature combination corresponding to the moment, the final data with a length of l will output a multi-time-scale eye movement feature sequence with a length of w after multi-time-scale feature extraction;

[0063] Step 4: Real-time eye movement signal classification: Input the multi-time scale eye movement feature sequence of length w extracted in real time into the pre-trained eye movement signal classification network model, and the current time t i Eye movement classification results

[0064] Furthermore, the schematic diagram of extracting eye movement feature sequences at different time scales using the multi-time scale feature extraction method proposed in the present invention is shown in the attached figure. Figure 2 As shown, we need to calculate t in sequence i-w+1 ~t i The multi-time scale eye movement feature combination corresponding to the moment; for example, when the time scale size is 128 sample points, to calculate the current time scale t i To calculate the eye movement features at the moment, it is necessary to calculate the eye movement features from the i-128th sample point to the i-th sample point.

[0065] Calculate t i The calculation method of eye movement features at the time scale k is as follows:

[0066] (1) Calculate the displacement characteristics dis i，k

[0067] Δx=x i -x i-k

[0068] Δy=y i -y i-k

[0069]

[0070] Among them, x i Indicates the horizontal coordinate of the gaze point at the current moment, x i-k Indicates the horizontal coordinate of the gaze point k sample points before the current moment, y i Indicates the vertical coordinate of the gaze point at the current moment, y i-k represents the ordinate of the gaze point at an interval of k sample points before the current moment, Δx represents the horizontal displacement of the gaze point at time scale k, and Δy represents the vertical displacement of the gaze point at time scale k;

[0071] (2) Calculate the directional feature θ i，k

[0072] θ i，k =atan(Δy, Δx)

[0073] (3) Calculate the velocity characteristic v i，k

[0074]

[0075] Where Δt = t i -t i-k , t i-k It represents the moment k sample points before the current moment. The positive or negative sign of the velocity represents its direction, which is determined by the horizontal displacement Δx and the vertical displacement Δy.

[0076] (4) Calculate the acceleration characteristic acc i，k

[0077]

[0078] in,

[0079]

[0080]

[0081] v x(i，k) =v i，k *cos(θ i，k ), v y(i，k) =v i，k *sin(θ i，k )

[0082] v x(i-k，k) =v i-k，k *cos(θ i-k，k ), v y(i-k，k) =v i-k，k *sin(θ i-k，k )

[0083] v i-k，k Indicates t i-k Velocity eye movement characteristics at time scale k, θ i-k，k Indicates t i-k The directional eye movement characteristics at the time scale of k; the positive or negative acceleration represents its direction, which is determined by t i The horizontal component of acceleration acc at time scale k x(i，k ) and t i The vertical component of acceleration acc at time scale k y(i，k) Joint decision.

[0084] (5) Calculate the speed standard deviation std i，k

[0085]

[0086] Among them, v i-k+j，k Indicates t i-k+j Velocity eye movement characteristics at time scale k, t i-k+jrepresents the time interval kj sample points before the current time, Indicates that from t i-k Time to t i The arithmetic mean of the speed eye movement characteristics corresponding to the time scale k is:

[0087] Therefore, according to the above feature extraction method, eye movement features at multiple different time scales can be extracted, and the combination of the above eye movement features can be used as the input of the eye movement classification model. Compared with the existing feature extraction methods, the speed, acceleration, and direction features proposed in the present invention all reflect the changes of the original eye movement data x(t) and y(t) over time, among which the speed standard deviation feature further reflects the changes in the discreteness of the speed value at the current time scale. Since the features extracted above have obvious feature changes between different eye movement types, and the sample points used for feature extraction are all at the current time t i The previous sample points lay a good foundation for the present invention to achieve real-time high-precision eye movement classification.

[0088] Furthermore, the pre-trained eye movement signal classification network model described in the present invention refers to a pre-trained offline eye movement classification network model; in an embodiment of the present invention, 8 layers of TCAN layers are constructed to learn the relationship between eye movement feature sequences at different time scales, the convolution kernel size of each layer of dilated causal convolution is 8, and the dilation factor d takes the following values: 1, 2, 4, 8, 16, 32, 64, and 128. After the input features have been processed by the multi-layer TCAN structure to learn relevant features, they are then processed by two one-dimensional convolutional layers (1DCNN) to extract local features, and finally, they are processed by the fully connected layer and the softmax classifier to output the classification results of the sample points at each moment. The specific implementation of the network framework diagram is shown in the attached Figure 3 shown.

[0089] In this embodiment of the present invention, the pre-trained eye movement signal classification network model uses the ReLU activation function and the cross-entropy loss function. Leave-one-out cross-validation is employed to divide the eye movement data from the Gazecom dataset into training, validation, and test sets, with a training-to-validation ratio of 9:1. A fixed-size sliding window is used to capture feature combinations extracted at various time scales as network input. After multiple rounds of training, the network model that performs best on the test set is ultimately obtained for real-time eye movement signal classification.

[0090] After testing, when the sliding window size used is 775 sample points and the multi-scale feature extraction uses 8 different time scale sizes (1, 2, 4, 8, 16, 32, 64, and 128 sample points), the eye movement signal classification network model based on the temporal attention network proposed in this invention has the best performance on the GazeCom dataset. The classification accuracy rate reaches 96.96%, and the F1 value is 0.9846, of which the F1 value of fixation is 0.9818, the F1 value of saccade is 0.9465, and the F1 value of smooth trailing is 0.9257. The accuracy (F1-score) comparison table of different models based on the GazeCom dataset is shown in the attached figure. Figure 4 In actual use, if you want to improve the speed of real-time eye movement signal classification, you can reduce the sliding window and time scale, but this will result in a decrease in classification accuracy.

[0091] Furthermore, the TCAN structure of the present invention mainly consists of a self-attention structure, two dilated causal convolutional layers and a residual structure using a 1×1 convolutional layer, as shown in the attached figure. Figure 5 As shown in the figure, the self-attention structure is placed before the first dilated causal convolution to obtain information between input feature sequences of a fixed sliding window size. The global association weight is used to obtain the weighted information of a single feature vector in the global context, which reflects the different degrees of global importance of feature sequences at different times, thereby focusing the network's "attention" on the most important areas. The attention mechanism formula is as follows:

[0092]

[0093] Where Q = XW Q , K=XW K , V=XW V , X represents the input sequence, W Q 、W K 、W V is the weight matrix, d k is the length of the input sequence; mask is used The purpose of this processing is to ensure that the self-attention structure only uses past information when processing sequence tasks. The mask formula is as follows:

[0094]

[0095] Among them, m=i-w+1, i-w+2,...,i, n=i-w+1, i-w+2,...,i.

[0096] Since then, a real-time eye movement signal classification method based on temporal convolutional attention network has been realized.

[0097] To sum up, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A real-time eye movement signal classification method based on temporal convolutional attention network, characterized by The steps of the method include: Step 1: Obtain raw eye movement data in real time; Step 2: The raw eye movement data acquired in real time in step 1 is transmitted and cached in a message queue. When the message queue is full, the sequence data in the message queue is transmitted in real time and subjected to sliding average filtering to obtain filtered eye movement data. The data at the head of the message queue is then removed. Step 3: extracting different eye movement features at multiple time scales from the eye movement data filtered in step 2, combining the different eye movement features to form an eye movement feature combination, and then combining the eye movement features obtained at different times to form an eye movement feature sequence; Step 4: Input the eye movement feature sequence generated in step 3 into the pre-trained eye movement signal classification network model to obtain real-time eye movement classification results; In step 1, raw eye movement data is obtained through an eye tracking device, where the raw eye movement data includes gaze point coordinates and timestamps; In step 2, the length of the message queue is l, l=w+k max , the sequence data is represented by Q, and the sequence data Q is: Q={(x i-l+1 ,y i-l+1 ,t i-l+1 ),(x i-l+2 ,y i-l+2 ,t i-l+2 ),…,(x i ,y i ,t i )} Among them, w represents the size of the fixed sliding window, k max represents the maximum time scale, t i represents the current time, t i-l+1 Indicates the time interval l-1 sample points before the current time, t i-l+2 represents the time interval l-2 sample points before the current time; x i Indicates the horizontal coordinate of the gaze point at the current moment, y i Indicates the vertical coordinate of the gaze point at the current moment, x i-l+1 Indicates the horizontal coordinate of the gaze point that is l-1 sample points before the current moment, x i-l+2 Indicates the horizontal coordinate of the gaze point that is l-2 sample points before the current moment, y i-l+1 Indicates the vertical coordinate of the gaze point that is l-1 sample points before the current moment, y i-l+2 Indicates the vertical coordinate of the gaze point that is l-2 sample points before the current moment; In step 3, the eye movement features include displacement eye movement features, direction eye movement features, velocity eye movement features, acceleration eye movement features, and velocity standard deviation eye movement features; In step 3, t i The displacement eye movement characteristics dis at the time scale k i,k for: Δx=x i -x i-k Δy=y i -y i-k Among them, x i-k Indicates the horizontal coordinate of the gaze point k sample points before the current moment, y i-k represents the ordinate of the gaze point at the interval of k sample points before the current moment, Δx represents the horizontal displacement of the gaze point under time scale k; Δy represents the vertical displacement of the gaze point under time scale k; t i Directional eye movement feature θ at the time scale k i,k for: θ i,k =atan(Δy,Δx) t i Velocity eye movement characteristics v at time scale k i,k for: Where Δt = t i -t i-k , t i-k Indicates the time that is k sample points before the current time.

2. The real-time eye movement signal classification method based on a temporal convolutional attention network according to claim 1, characterized in that: In step 3, t i Acceleration eye movement feature acc at time scale k i,k for: in, v x(i,k) =v i,k *cos(θ i,k ),v y(i,k) =v i,k *sin(θ i,k ) v x(i-k,k) =v i-k,k *cos(θ i-k,k ),v y(i-k,k) =v i-k,k *sin(θ i-k,k 0 v i-k,k Indicates t i-k Velocity eye movement characteristics at time scale k, θ i-k,k Indicates t i-k Directional eye movement characteristics at the moment time scale k.

3. The real-time eye movement signal classification method based on temporal convolutional attention network according to claim 2 is characterized by: In step 3, t i The standard deviation of the eye movement characteristic std at the time scale k i,k for: Among them, v i-k+j,k Indicates t i-k+j Velocity eye movement characteristics at time scale k, t i-k+j represents the time interval kj sample points before the current time, 4. The real-time eye movement signal classification method based on temporal convolutional attention network according to claim 1, characterized in that: In step 4, the pre-trained eye movement signal classification network model refers to a pre-trained offline eye movement classification network model, including a temporal convolutional attention network structure using causal convolution, a one-dimensional convolutional layer, a fully connected layer and a softmax classifier; Among them, the temporal convolutional attention network structure includes a self-attention structure, two dilated causal convolution layers and a residual structure using a 1×1 convolution layer. The self-attention structure is located before the first dilated causal convolution layer.

5. The real-time eye movement signal classification method based on temporal convolutional attention network according to claim 4 is characterized in that: The attention mechanism formula used by the self-attention structure is as follows: Where Q = XW Q ,K=XW K ,V=XW V , X represents the input sequence, W Q 、W K 、W V is the weight matrix, d k is the length of the input sequence, using mask For processing, the mask formula is as follows: Where m=i-w+1,i-w+2,…,i, n=i-w+1,i-w+2,…,i.

6. The real-time eye movement signal classification method based on temporal convolutional attention network according to claim 5, characterized in that: In step 4, the real-time eye movement classification results include fixation, saccade and smooth follow.

Images