An eye-tracking recognition method based on a hybrid Transformer and LSTM network

By combining a hybrid network structure of Transformer, LSTM and Fourier-CNN, the problem of insufficient feature mining in existing eye-tracking recognition technology is solved, achieving higher recognition accuracy and lower error rate, especially with excellent performance in tests on multiple databases and at different time periods.

CN118230400BActive Publication Date: 2026-11-10CHONGQING TECH & BUSINESS UNIV +2
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410147131.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-31
Publication Date
2026-11-10
Estimated Expiration
2044-01-31

AI Technical Summary

Technical Problem

Existing eye-tracking recognition technologies rely on traditional machine learning and single convolutional neural networks, which make it difficult to fully extract the complex features and related information in eye-tracking data, resulting in low recognition accuracy and a high error rate.

Method used

An eye-tracking recognition method based on a hybrid Transformer and LSTM network is adopted, which combines Siamese CNN, LSTM, Transformer and Fourier-CNN. The feature extraction and classification of eye-tracking data are performed by multiple networks. The model is trained using an adaptive optimization algorithm and a preset loss function to extract eye-tracking features in the time domain and frequency domain.

Benefits of technology

It significantly reduced the equal error rate (EER), demonstrated significant performance improvements on multiple databases, and enhanced the accuracy and generalization ability of eye-tracking recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118230400B_ABST
    Figure CN118230400B_ABST
Patent Text Reader

Abstract

This invention provides an eye-tracking recognition method based on a hybrid Transformer and LSTM network. The method includes: acquiring eye-tracking data; preprocessing the eye-tracking data to obtain preprocessed eye-tracking data; constructing a hybrid network eye-tracking recognition model; training the hybrid network eye-tracking recognition model using the preprocessed eye-tracking data; optimizing the hybrid network eye-tracking recognition model based on the training results; and obtaining eye-tracking recognition results using the optimized hybrid network eye-tracking recognition model based on the preprocessed eye-tracking data. This invention can extract eye-tracking features in both the time and frequency domains. When tested on multiple databases using data from different time periods, it shows significant performance improvements, with a significantly reduced EER compared to the most advanced algorithms currently available.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of eye-tracking image recognition technology, and more particularly to an eye-tracking recognition method based on a hybrid Transformer and LSTM network. Background Technology

[0002] Eye movements are the coordinated contractions and relaxations of six eye muscles controlled by the brainstem during visual processing. They are a crucial mechanism for humans to select information from the visual environment, enabling us to focus our attention on things of interest. Eye movements contain rich information about brain cognitive functions and neural pathways, and are considered behavioral characteristics, along with keystroke characteristics, gait characteristics, and electrocardiograms. Early research demonstrated that the way humans move their eyes in response to specific stimuli is highly individualized, and recent studies have shown that these individual characteristics are reliable over time.

[0003] Eye movement data is obtained by illuminating the eye with an infrared light source and collecting the reflections on the cornea and pupil. This data can be divided into saccades and fixations using the velocity thresholding (I-VT) algorithm. During fixation, the eye remains almost stationary at a fixation point, maintaining high-frequency, low-amplitude tremor movements; this movement has been shown to exhibit individual differences. During saccades, the eye rapidly moves from one fixation point to the next, at speeds reaching 200–400° / s. Both fixation point and movement data can be acquired using eye-tracking devices, and eye movement features can be extracted from this data for biometric identification tasks.

[0004] Current eye-tracking recognition processes primarily rely on traditional machine learning, requiring manual feature extraction and being limited by feature representation capabilities. In recent years, with the rise of deep learning, neural networks can automatically learn feature representations from raw data without manual feature extraction. However, the network structures used in eye-tracking recognition are relatively simple, mainly relying on traditional convolutional neural networks. This simple network structure may not be able to fully extract the complex features and correlations within eye-tracking data. Summary of the Invention

[0005] Therefore, it is necessary to provide an eye-tracking recognition method based on a hybrid network of Transformer and LSTM to address the aforementioned technical problems.

[0006] An eye-tracking recognition method based on a hybrid Transformer and LSTM network includes the following steps:

[0007] Acquire eye-tracking data, and preprocess the eye-tracking data to obtain preprocessed eye-tracking data;

[0008] A hybrid network eye-tracking recognition model is constructed, and the hybrid network eye-tracking recognition model is trained using the preprocessed eye-tracking data. The hybrid network eye-tracking recognition model is then optimized based on the training results.

[0009] Based on the preprocessed eye-tracking data, the optimized hybrid network eye-tracking recognition model is used to obtain the eye-tracking recognition result.

[0010] In one embodiment, training the hybrid network eye-tracking recognition model using the preprocessed eye-tracking data, and optimizing the hybrid network eye-tracking recognition model based on the training results includes:

[0011] Using preprocessed eye-tracking data as the source input of the network and the eye-tracking recognition result corresponding to the preprocessed eye-tracking data as the output, the preprocessed eye-tracking data is applied to train the hybrid network eye-tracking recognition model to obtain the optimized hybrid network eye-tracking recognition model.

[0012] In one embodiment, the hybrid network eye-tracking recognition model includes:

[0013] Siamese CNN and hybrid networks;

[0014] The hybrid network includes:

[0015] LSTM, Transformer and Fourier-CNN.

[0016] In one embodiment, based on the preprocessed eye-tracking data, the eye-tracking recognition result is obtained using an optimized hybrid network eye-tracking recognition model, including:

[0017] The preprocessed eye-tracking data is input into the Siamese CNN as the source of the first network to obtain the first processing result;

[0018] The first processing result is used as the source of the second network and input into the hybrid network to obtain the eye-tracking recognition result;

[0019] The first processing result is used as the source of the second network and input into the hybrid network to obtain eye-tracking recognition results, including:

[0020] The first processing result is input into the LSTM to obtain the recursive classification result;

[0021] The first processing result is input into the Transformer to obtain the attention classification result;

[0022] The first processing result is input into Fourier-CNN to obtain the Fourier classification result;

[0023] The eye-tracking recognition result is obtained based on the recursive classification result, the attention classification result, and the Fourier classification result.

[0024] In one embodiment, eye-tracking data is acquired, and the eye-tracking data is preprocessed to obtain preprocessed eye-tracking data, including:

[0025] Obtaining eye-tracking data:

[0026] X = (x1, ..., x) t Y = (y1, ..., y) t );

[0027] Where X represents the sequence of horizontal coordinates of the eye fixation point, Y represents the sequence of vertical coordinates of the eye fixation point, and x t The x-coordinate value at time t is represented by y. t This represents the value of the ordinate at time t;

[0028] Based on the eye-tracking data, the velocity information at the sampling time is obtained:

[0029]

[0030]

[0031] in, Represents the velocity of the sampling point's x-axis. The vertical axis of the sampling point represents the velocity, x. (i) The x-coordinate of the sampling point is represented by x. (i-1) The x-coordinate value of the sampling point at the previous moment is represented by y. (i) The y-coordinate of the sampling point represents the numerical value. (i-1) The value of the ordinate at the time preceding the sampling point, t (i) Indicates the sampling time, t (i-1) Indicates the moment immediately preceding the sampling;

[0032] Based on the velocity information at the sampling time, fast eye movement data and slow eye movement data are obtained:

[0033] The fast eye movement data and the slow eye movement data are respectively input into two multi-layer one-dimensional convolutional sub-networks with different convolutional kernel parameters to extract features at different scales, thereby obtaining extracted eye movement data.

[0034] The extracted eye-tracking data is then concatenated along the channel dimension to obtain preprocessed eye-tracking data.

[0035] In one embodiment, obtaining fast eye movement data and slow eye movement data based on the velocity information at the sampling time includes:

[0036] Obtain the minimum speed; if the minimum speed corresponding to the speed information at the sampling time is greater than the preset minimum speed, fast data is obtained.

[0037]

[0038] in, This represents the segmented fast eye-tracking data. This indicates that the velocities on the horizontal and vertical axes have been normalized. Indicates that the value is zero, v min This indicates the preset speed division threshold;

[0039] The fast data is Z-score normalized to obtain fast eye-tracking data:

[0040]

[0041] in, Let δ represent the fast eye-tracking data at the i-th sampling point. i Let represent the velocity of the i-th sampling point, μ represent the mean of the velocities of the sampling points, and σ represent the variance of the velocities of the sampling points.

[0042] If the minimum speed responding to the speed information at the sampling time is less than the preset minimum speed, slow data is obtained;

[0043] Based on the slow-motion data, slow-motion eye movement data is obtained:

[0044]

[0045] in, This represents slow eye movement data. This indicates that the velocity on the horizontal and vertical axes is multiplied by a constant c and then mapped using a hyperbolic tangent function, converting the numerical range to (-1, +1).

[0046] In one embodiment, the Siamese CNN includes:

[0047] A first extraction subnet and a second extraction subnet; wherein, the first extraction subnet receives fast eye-tracking data; and the second extraction subnet receives slow eye-tracking data;

[0048] The first extraction subnetwork includes: a first convolutional layer, a first batch of normalization layers, a first ReLU layer, and a first average pooling layer;

[0049] The second extraction subnetwork includes: a second convolutional layer, a second batch normalization layer, a second ReLU layer, and a second average pooling layer.

[0050] In one embodiment, the first processing result is input into an LSTM to obtain recursive classification results, including:

[0051] The recursive classification result is calculated using the following formula:

[0052] f t =σ(W f x′+U f h t-1 +b f )

[0053] i t =σ(W i x′+U i h t-1 +b i )

[0054] o t =σ(W o x′+U o h t-1 +b o )

[0055]

[0056]

[0057] h t =LSTM(W x x t +b x ,h t-1 C t-1 )

[0058] y = W y h t +b y

[0059] Among them, f t This represents the forget gate of LSTM, i t This represents the input gate of the LSTM, o t σ represents the output gate of the LSTM, W represents the softmax function. f Let U represent the forget gate weight matrix, x′ represent the forget gate input, and U represent the forget gate weight matrix. f h represents the recursive connection weight matrix of the forget gate. t-1 b represents the hidden state in the previous time step. f W represents the bias term of the forget gate. i U represents the input gate weight matrix. i b represents the input gate recursive connection weight matrix. i W represents the bias term of the input gate. o U represents the output gate weight matrix. o b represents the output gate recursive connection weight matrix. o This represents the bias term of the output gate. Represents the candidate cell state, tanh() represents the hyperbolic tangent function, W C U represents the cell state weight matrix. C The cell state recursive connection weight matrix, b C C represents the cell state bias term. t C represents the current state of the cell. t-1 Represents the cell state at the previous moment, ⊙ represents element-wise matrix multiplication, h t W represents the hidden state at time t. x Let x represent the input weight matrix. t Represents the input sequence, b x Let W represent the input bias term, y represent the recursive classification result, and W represent the input bias term. y This represents the output weight matrix, b y This indicates the output bias term.

[0060] In one embodiment, the first processing result is input into the Transformer to obtain the attention classification result, including:

[0061] Add location encoding to the first processing result to obtain location information;

[0062] Add the location information to the embedding vector to obtain the embedded location code:

[0063] PE = (pe1, ..., pe) t )

[0064] Where PE represents the embedding position code, pe t The position code represents the t-th first processing result;

[0065] The output of the self-attention mechanism is calculated using the following formula via embedded positional encoding:

[0066]

[0067] Where Attention(Q,K,V) represents the output of the self-attention mechanism, Softmax() represents the normalized exponential function, Q represents the corresponding query obtained by linearly transforming the weight matrix into the first processing result, K represents the corresponding key obtained by linearly transforming the weight matrix into the first processing result, and V represents the corresponding vector obtained by linearly transforming the weight matrix into the first processing result. T Represents the dot product. Indicates the scaling factor;

[0068] The output of the multi-head attention mechanism is calculated using the following formula:

[0069] MultiHead(Q,K,V)=Concat(head1,head2,head3,head4)W o

[0070] In this function, MultiHead(Q,K,V) represents the output of the multi-head attention mechanism, Concat() represents the array concatenation function, head1 represents the output of the first attention head, head2 represents the output of the first and second attention heads, head3 represents the output of the third attention head, head4 represents the output of the fourth attention head, and W... o Represents a linear transformation matrix;

[0071] The attention classification result is obtained using the following formula:

[0072] Z = LayerNorm(X) ’ +MultiHead(LN(X ’ )))

[0073] F = ReLU(Z·W1+b1)·W2+b2

[0074] Where Z represents the output after layer normalization of the multi-head attention mechanism output, LayerNorm() represents the layer normalization function of the feature vector, and X ’ Let f represent the input, LN represent the linear transformation, F represent the output feature vector, ReLU() represent the activation function, and W1, b1, W2, b2 represent the learnable parameters.

[0075] In one embodiment, the first processing result is input into Fourier-CNN to obtain the Fourier transformed feature vector, which includes:

[0076] Based on the first processing result, the frequency domain processing result is obtained using the following formula:

[0077]

[0078] Where X(k) represents the complex value of the k-th frequency component in the frequency domain, T represents the sequence length, n represents the n-th sampling point, x(n) represents the complex value of the n-th sampling point, and j represents the imaginary unit;

[0079] Based on the frequency domain processing results, the amplitude component and phase component are obtained using the following formula:

[0080]

[0081]

[0082] Where A(k) represents the amplitude component, Let X(k) denote the phase component, Im(X(k)) denote the real part of the component, Re(X(k)) denote the imaginary part of the component, and arctan() denote the arctangent function.

[0083] Based on the amplitude and phase components, the Fourier classification result is obtained using the following formula:

[0084]

[0085] Where X(n) represents the time-domain feature vector, Let represent the inverse Fourier transform, Conv1d represent the one-dimensional convolution operation, and A(k) represent the amplitude component. denoted by phase component, and complex is a complex combination function.

[0086] Compared with existing technologies, the advantages and beneficial effects of this invention are as follows: This invention can extract eye movement features in the time domain and frequency domain. When tested on multiple databases using data from different time periods, it shows significant performance improvement and significantly reduces EER compared with the most advanced algorithms currently available. Attached Figure Description

[0087] Figure 1 This is a flowchart illustrating an eye-tracking recognition method based on a hybrid Transformer and LSTM network in one embodiment.

[0088] Figure 2 This is a schematic diagram of the structure of a hybrid network eye-tracking recognition model in one embodiment. Detailed Implementation

[0089] Before describing the specific embodiments of the present invention, the overall concept of the present invention will be explained as follows:

[0090] This invention primarily focuses on the development of eye-tracking recognition processes. Currently, most traditional machine learning methods for eye-tracking recognition divide eye movements into specific events such as saccades and fixations, forming biometric templates from each event as vectors of discrete features. These are then combined with feature extraction techniques and statistical or machine learning algorithms for eye-tracking recognition. However, this method suffers from high false acceptance and false rejection rates, resulting in low recognition accuracy.

[0091] Building upon this, deep learning-based eye-tracking recognition relies on relatively simple network structures, primarily traditional convolutional neural networks. This simplistic approach may not fully capture the complex features and relationships within eye-tracking data. While some work has introduced LSTM (Long Short-Term Memory) into eye-tracking recognition—a variant of recurrent neural networks—it merely involves simple module stacking. When processing longer time-series data, it may face long-term dependency issues, making it difficult to effectively capture long-range relationships within eye-tracking sequences. Therefore, more complex and innovative network architectures are needed to improve model performance and generalization capabilities.

[0092] The inventors, through analysis, discovered that the main reason for the aforementioned problems is the overly simplistic structure of deep learning-based eye-tracking recognition networks. Furthermore, eye-tracking data acquisition in different environments is subject to noise and uncertainty. By using a multi-dimensional eye-tracking data extraction network to extract high-dimensional temporal features from eye-tracking information and improve the overall performance of the model, these problems can be avoided. Therefore, this invention proposes an eye-tracking recognition method based on a hybrid Transformer and LSTM network. This method utilizes a fusion deep learning model of Transformer and LSTM and introduces a Fourier-CNN network to extract frequency domain features from the signal. This method can extract high-dimensional temporal features from eye-tracking information, outperforming network structures such as Dense-LSTM, DEL, and exonentially-dilated CNN.

[0093] After introducing the overall concept of the present invention, in order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0094] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of this specification should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in one or more embodiments of this specification do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0095] For ease of understanding, the terms used in the embodiments of this invention are explained below:

[0096] Transformer Model: The Transformer model is built upon the Self-Attention mechanism. Self-Attention is a method for calculating the relationships between different positions in a sequence. In the Transformer, each input is divided into multiple sub-sequences after passing through the Embedding layer. Each sub-sequence passes through multiple layers of Self-Attention and fully connected layers, and finally, the output is obtained through a linear transformation.

[0097] LSTM: Long Short-Term Memory (LSTM) is a type of temporal recurrent neural network suitable for processing and predicting important events with relatively long intervals and delays in time series.

[0098] EER: Equal Error Rate. The threshold is adjusted so that the False Rejection Rate (FRR) equals the False Acceptance Rate (FAR). The values ​​of FAR and FRR at this point are called equal error rates.

[0099] FAR: False Acceptance Rate. In classification problems, if two samples are of different classes (not the same person) but are mistakenly identified as belonging to the same class (the same person) by the system, this is a false acceptance case. The false acceptance rate is the proportion of false acceptance cases among all out-of-class matching cases.

[0100] FRR: False Rejection Rate. In classification problems, if two samples are of the same class (the same person) but are mistakenly identified as different classes (not the same person) by the system, this is a false rejection. The false rejection rate is the proportion of falsely rejected cases among all matching cases of the same class.

[0101] Siamese CNN: Siamese Convolutional Neural Network.

[0102] Fourier-CNN: Fourier Convolutional Neural Network.

[0103] CNN: Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolutional computation and has a deep structure. It is one of the representative algorithms of deep learning.

[0104] In one embodiment, such as Figure 1 As shown, an eye-tracking recognition method based on a hybrid Transformer and LSTM network is provided, including the following steps:

[0105] Step S101: Acquire eye-tracking data and preprocess the eye-tracking data to obtain preprocessed eye-tracking data.

[0106] Specifically, eye-tracking data is first acquired and then preprocessed to ensure that the step size of the model input samples is consistent with the acquisition frequency, i.e., the time-series data generated by eye movements within 1 second.

[0107] Based on this, eye-tracking data is acquired and preprocessed to obtain preprocessed eye-tracking data, including:

[0108] Obtaining eye-tracking data:

[0109] X = (x1, ..., x) t Y = (y1, ..., y) t );

[0110] Where X represents the sequence of horizontal coordinates of the eye fixation point, Y represents the sequence of vertical coordinates of the eye fixation point, and x t The x-coordinate value at time t is represented by y. t This represents the value of the ordinate at time t;

[0111] Based on the eye-tracking data, the velocity information at the sampling time is obtained:

[0112]

[0113]

[0114] in, Represents the velocity of the sampling point's x-axis. The vertical axis of the sampling point represents the velocity, x. (i) The x-coordinate of the sampling point is represented by x. (i-1) The x-coordinate value of the sampling point at the previous moment is represented by y. (i) The y-coordinate of the sampling point represents the numerical value. (i-1) The value of the ordinate at the time preceding the sampling point, t (i) Indicates the sampling time, t (i-1) Indicates the moment immediately preceding the sampling;

[0115] Based on the velocity information at the sampling time, fast eye movement data and slow eye movement data are obtained;

[0116] The fast eye movement data and the slow eye movement data are respectively input into two multi-layer one-dimensional convolutional sub-networks with different convolutional kernel parameters to extract features at different scales, thereby obtaining extracted eye movement data.

[0117] The extracted eye-tracking data is then concatenated along the channel dimension to obtain preprocessed eye-tracking data.

[0118] Specifically, eye-tracking data is retrieved from the database, specifically the eye fixation point coordinates X = (x1, ..., x...). t Y = (y1, ..., y) t The coordinate information is transformed into velocity information at each sampling time using the one-sample backward difference method.

[0119]

[0120]

[0121] The data was processed using a single-sample backward difference method to remove moments without fixation points, and the remaining Nan values ​​were replaced with 0. The data from each eye constituted the network's two input channels: <δ x > represents the horizontal velocity sequence, <δ y Vertical velocity sequence.

[0122] Based on the velocity information calculated at the sampling time, fast eye movement data and slow eye movement data are obtained.

[0123] The fast eye movement data and the slow eye movement data are respectively input into two multi-layer one-dimensional convolutional sub-networks with different convolutional kernel parameters to extract features at different scales, thereby obtaining extracted eye movement data.

[0124] The extracted eye-tracking data is concatenated along the channel dimension to obtain preprocessed eye-tracking data. This preprocessed data is then input into a hybrid network for further feature extraction and classification.

[0125] Based on this, and according to the velocity information at the sampling time, the fast eye movement data and slow eye movement data are obtained, including:

[0126] To obtain the minimum speed, the minimum speed that corresponds to the speed information at the sampling time is greater than the minimum speed, thus obtaining fast data:

[0127]

[0128] in, This represents the segmented fast eye-tracking data. This indicates that the velocities on the horizontal and vertical axes have been normalized. Indicates that the value is zero, v min This indicates the preset speed division threshold;

[0129] The fast data is Z-score normalized to obtain fast eye-tracking data:

[0130]

[0131] in, Let δ represent the fast eye-tracking data at the i-th sampling point. i Let represent the velocity of the i-th sampling point, μ represent the mean of the velocities of the sampling points, and σ represent the variance of the velocities of the sampling points.

[0132] If the minimum speed in response to the velocity information at the sampling time is greater than the minimum speed, then slow data is obtained.

[0133] Based on the slow-motion data, slow-motion eye movement data is obtained:

[0134]

[0135] in, This represents slow eye movement data. This indicates that the velocity on the horizontal and vertical axes is multiplied by a constant c and then mapped using a hyperbolic tangent function, converting the numerical range to (-1, +1).

[0136] Specifically, the characteristics of saccades and fixations in eye movements are analyzed at different speed scales. A speed thresholding method is used to transform the raw data into fast data. <D fast >and slow data <D slow Two parts. The minimum velocity v is captured. min The above data is from fast eye-tracking, and Z-score normalization has been applied. The processed formula is shown below:

[0137]

[0138]

[0139] Where δ i Let be the velocity of the i-th sampling point, and μ and σ be the mean and variance of the velocities of the sampling points.

[0140] For slow data <D slow The input is compressed to the range (-1, +1) using the hyperbolic tangent function.

[0141]

[0142] The velocity threshold v = 40 and the scaling factor c = 0.02 are both [value missing]. Fixed hyperparameters of a function.

[0143] Step S102: Construct a hybrid network eye-tracking recognition model, train the hybrid network eye-tracking recognition model using the preprocessed eye-tracking data, and optimize the hybrid network eye-tracking recognition model based on the training results.

[0144] Specifically, a hybrid network eye-tracking recognition model, EmMix-Net, which combines Transformer and LSTM networks, is constructed, trained, and optimized based on the training results.

[0145] Based on this, hybrid network eye-tracking recognition models include:

[0146] Siamese CNN and hybrid networks;

[0147] The hybrid network includes:

[0148] LSTM, Transformer and Fourier-CNN.

[0149] Specifically, such as Figure 2 As shown, EmMix-Net, a hybrid network eye-tracking recognition model that combines Transformer and LSTM networks, is constructed, including... Figure 2 (a) shows Siamese CNN and Figure 2 (b) shows the hybrid network.

[0150] Hybrid networks include LSTM, Transformer, and Fourier-CNN. These multiple networks share the output of the Siamese CNN part as input to extract richer and more comprehensive feature representations.

[0151] In this embodiment, eye movement recognition is performed through multiple networks to ensure its accuracy. This allows for the extraction of rich, comprehensive, and accurate feature representations, thereby effectively improving the performance of eye movement recognition and classification.

[0152] Based on this, Siamese CNN includes:

[0153] A first extraction subnet and a second extraction subnet; wherein, the first extraction subnet receives fast eye-tracking data; and the second extraction subnet receives slow eye-tracking data;

[0154] The first extraction subnetwork includes: a first convolutional layer, a first batch of normalization layers, a first ReLU layer, and a first average pooling layer;

[0155] The second extraction subnetwork includes: a second convolutional layer, a second batch normalization layer, a second ReLU layer, and a second average pooling layer.

[0156] In one embodiment, the Siamese CNN includes two sub-networks, each of which stacks four one-dimensional convolutional modules for initial feature extraction. These convolutional modules include convolutional layers, batch normalization layers, ReLU layers, and average pooling layers. Figure 2 As shown in (c), the two sub-networks receive preprocessed fast eye movement information and slow eye movement information, respectively.

[0157] Based on this, the hybrid network eye-tracking recognition model is trained using the preprocessed eye-tracking data, and the hybrid network eye-tracking recognition model is optimized based on the training results, including:

[0158] Using preprocessed eye-tracking data as the source input of the network and the eye-tracking recognition result corresponding to the preprocessed eye-tracking data as the output, the preprocessed eye-tracking data is applied to train the hybrid network eye-tracking recognition model to obtain the optimized hybrid network eye-tracking recognition model.

[0159] Specifically, the network parameters are optimized using the cross-entropy loss function and the Adam optimizer. The cross-entropy loss function is used to measure the difference between the model's output and the actual labels, helping the model learn the correct classification. The Adam optimizer is used to adaptively adjust the network parameters, continuously optimizing the model's performance during training. The learning rate is set to 0.0002, and the size of each input batch is fixed at 64, with the model trained for 1000 epochs.

[0160] Based on this, and using the preprocessed eye-tracking data, the optimized hybrid network eye-tracking recognition model yields the following eye-tracking recognition results:

[0161] The preprocessed eye-tracking data is input into the Siamese CNN as the source of the first network to obtain the first processing result;

[0162] The first processing result is used as the source of the second network and input into the hybrid network to obtain the eye-tracking recognition result;

[0163] The first processing result is used as the source of the second network and input into the hybrid network to obtain eye-tracking recognition results, including:

[0164] The first processing result is input into the LSTM to obtain the recursive classification result;

[0165] The first processing result is input into the Transformer to obtain the attention classification result;

[0166] The first processing result is input into Fourier-CNN to obtain the Fourier classification result;

[0167] The eye-tracking recognition result is obtained based on the recursive classification result, the attention classification result, and the Fourier classification result.

[0168] Specifically, preprocessed eye-tracking data is input into the Siamese CNN as the source of the first network to obtain a first processing result. The first processing result is then input into the three networks of the hybrid network to obtain the eye-tracking recognition result.

[0169] In this embodiment, LSTM and Transformer networks extract long-term and global dependencies in time series data. LSTM (Long Short-Term Memory) networks, through their gating mechanism and memory units, can effectively capture and model long-term dependencies in time series data, such as... Figure 2 As shown in (e). The Transformer model, on the other hand, uses a self-attention mechanism to learn the dependencies between different positions, exhibiting good parallel computing properties and global awareness, such as... Figure 2 As shown in (d), LSTM and Transformer networks can extract important temporal features from eye-tracking data.

[0170] Fourier-CNN networks focus on extracting frequency domain features. By performing convolution operations on eye-tracking data after Fourier transform, this branch can capture frequency-related features, such as... Figure 2 As shown in (d), the Fourier-CNN network can enhance the model's ability to perceive frequency domain information.

[0171] Three different network structures are concatenated along the feature dimension to obtain a comprehensive feature vector. The sigmoid function is used to map the feature values ​​to probability values ​​between 0 and 1, representing the probability that a sample belongs to a certain class. This combination leverages the advantages of each structure to extract rich, comprehensive, and accurate feature representations, thereby effectively improving the performance of time series data classification.

[0172] Based on this, the first processing result is input into the LSTM to obtain the recursive classification results, including:

[0173] The recursive classification result is calculated using the following formula:

[0174] f t =σ(W f x′+U f h t-1 +b f )

[0175] it =σ(W i x′+U i h t-1 +b i )

[0176] o t =σ(W o x′+U o h t-1 +b o )

[0177]

[0178]

[0179] h t =LSTM(W x x t +b x ,h t-1 C t-1 )

[0180] y = W y h t +b y

[0181] Among them, f t This represents the forget gate of LSTM, i t This represents the input gate of the LSTM, o t σ represents the output gate of the LSTM, W represents the softmax function. f Let U represent the forget gate weight matrix, x′ represent the forget gate input, and U represent the forget gate weight matrix. f h represents the recursive connection weight matrix of the forget gate. t-1 b represents the hidden state in the previous time step. f W represents the bias term of the forget gate. i U represents the input gate weight matrix. i b represents the input gate recursive connection weight matrix. i W represents the bias term of the input gate. o U represents the output gate weight matrix. o b represents the output gate recursive connection weight matrix. o This represents the bias term of the output gate. Represents the candidate cell state, tanh() represents the hyperbolic tangent function, W C U represents the cell state weight matrix. C The cell state recursive connection weight matrix, b C C represents the cell state bias term. t C represents the current state of the cell. t-1Represents the cell state at the previous moment, ⊙ represents element-wise matrix multiplication, h t W represents the hidden state at time t. x Let x represent the input weight matrix. t Represents the input sequence, b x Let W represent the input bias term, y represent the recursive classification result, and W represent the input bias term. y This represents the output weight matrix, b y This indicates the output bias term.

[0182] Specifically, LSTM is a special type of recurrent neural network (RNN) that addresses the problem of RNNs only being able to learn past information but not long-term dependencies by introducing memory cells and multiple control gates. It is widely used in sequence processing. In LSTM, the memory cell is similar to a storage unit inside a cell, storing and transferring information, while the control gates are responsible for controlling the flow and processing of information. A standard LSTM unit consists of three main control gates: a forget gate, an input gate, and an output gate. These gates control the flow of information using the sigmoid function as the activation function. The forget gate determines which information should be deleted from its previous state, the input gate determines which new input information should be stored in the unit state, and the output gate controls what information should be output. The specific calculation formula is as follows:

[0183] f t =σ(W f x′+U f h t-1 +b f )

[0184] i t =σ(W i x′+U i h t-1 +b i )

[0185] o t =σ(W o x′+U o h t-1 +b o )

[0186]

[0187]

[0188] Where f t i t and o t These are the forget gate, input gate, and output gate, respectively. C t and h t These represent the candidate cell state and the cell state, respectively, and ⊙ indicates element-wise matrix multiplication.

[0189] In one embodiment, an LSTM is used as a classifier with both input and output dimensions of 256, focusing only on the output of the last time step. During computation, each time the eye-tracking sequence (x1, ..., x...) is processed from the input... t Take one element from the sequence and compute the hidden state sequence (h1, ..., h2). t The formula is shown below:

[0190] h t =LSTM(W x x t +b x ,h t-1 C t-1 )

[0191] y = W y h t +b y

[0192] The initial unit state C0 and hidden state h0 are initialized as zero vectors, W x and W y The LSTM output is a learnable weight matrix. After training, the LSTM output is converted into class scores, i.e., the recursive classification result.

[0193] Based on this, the first processing result is input into the Transformer to obtain the attention classification results, including:

[0194] Add location encoding to the first processing result to obtain location information;

[0195] Add the location information to the embedding vector to obtain the embedded location code:

[0196] PE = (pe1, ..., pe) t )

[0197] Where PE represents the embedding position code, pe t The position code represents the t-th first processing result;

[0198] The output of the self-attention mechanism is calculated using the following formula via embedded positional encoding:

[0199]

[0200] Where Attention(Q,K,V) represents the output of the self-attention mechanism, Softmax() represents the normalized exponential function, Q represents the corresponding query obtained by linearly transforming the weight matrix into the first processing result, K represents the corresponding key obtained by linearly transforming the weight matrix into the first processing result, and V represents the corresponding vector obtained by linearly transforming the weight matrix into the first processing result. T Represents the dot product. Indicates the scaling factor;

[0201] The output of the multi-head attention mechanism is calculated using the following formula:

[0202] MultiHead(Q,K,V)=Concat(head1,head2,head3,head4)W o

[0203] In this function, MultiHead(Q,K,V) represents the output of the multi-head attention mechanism, Concat() represents the array concatenation function, head1 represents the output of the first attention head, head2 represents the output of the first and second attention heads, head3 represents the output of the third attention head, head4 represents the output of the fourth attention head, and W... o Represents a linear transformation matrix;

[0204] The attention classification result is obtained using the following formula:

[0205] Z = LayerNorm(X) ’ +MultiHead(LN(X ’ )))

[0206] F = ReLU(Z·W1+b1)·W2+b2

[0207] Where Z represents the output after layer normalization of the multi-head attention mechanism output, LayerNorm() represents the layer normalization function of the feature vector, and X ’ Let f represent the input, LN represent the linear transformation, F represent the output feature vector, ReLU() represent the activation function, and W1, b1, W2, b2 represent the learnable parameters.

[0208] Specifically, the Transformer was first used in language tasks. Unlike traditional recurrent neural networks (such as LSTM and GRU), the Transformer adopts a completely new architecture, based entirely on the self-attention mechanism. This attention mechanism helps the model establish global dependencies in the sequence without having to process the sequence sequentially like a recurrent neural network.

[0209] For the first processing result, add position encoding to obtain position information, and add the position information to the embedding vector PE = (pe1, ..., pe2). t In ) pe i This represents the positional encoding of the i-th first processing result. A weight matrix is ​​obtained, and a linear transformation is performed on the first processing result to obtain the corresponding query (Q), key (K), and value (V) vectors. An attention weight distribution is obtained by calculating the similarity between Q and K, representing the importance of each position to the current position. The similarity value is expressed as a dot product QK. T Get and divide by a scaling factor To maintain model stability, d represents the dimension of the word vectors. The attention weights are weighted and summed with the value vector V to obtain the output of the self-attention mechanism, as shown in the formula below:

[0210]

[0211] In one embodiment, a multi-head attention mechanism (nhead = 4) encoding layer is set up in the Transformer network, and four layers are stacked to form a complete Transformer encoder, where the dropout rate applied in the attention mechanism is 0.5. The output of this multi-head attention mechanism can be represented as:

[0212] MultiHead(Q,K,V)=Concat(head1,head2,head3,head4)W o

[0213] head i W represents the output of the i-th attention head. o This is the final linear transformation matrix, used to linearly transform the weighted average obtained after attention calculation.

[0214] The obtained results are then subjected to residual join and layer normalization operations with the input first processed result:

[0215] Z=LayerNorm(X+MultiHead(LN(X ’ )))

[0216] Finally, a nonlinear transformation is applied to each of the first processing results using a feed-forward neural network to obtain the attention classification result:

[0217] F = ReLU(Z·W1+b1)·W2+b2

[0218] W1, W2, b1, and b2 are learnable parameters.

[0219] Based on this, the first processing result is input into Fourier-CNN to obtain the Fourier classification results, including:

[0220] Based on the first processing result, the frequency domain processing result is obtained using the following formula:

[0221]

[0222] Where X(k) represents the complex value of the k-th frequency component in the frequency domain, T represents the sequence length, n represents the n-th sampling point, x(n) represents the complex value of the n-th sampling point, and j represents the imaginary unit;

[0223] Based on the frequency domain processing results, the amplitude component and phase component are obtained using the following formula:

[0224]

[0225]

[0226] Where A(k) represents the amplitude component, Let X(k) denote the phase component, Im(X(k)) denote the real part of the component, Re(X(k)) denote the imaginary part of the component, and arctan() denote the arctangent function.

[0227] Based on the frequency domain processing results, the Fourier classification result is obtained using the following formula:

[0228]

[0229] Where X(n) represents the time-domain feature vector, Let represent the inverse Fourier transform, Conv1d represent the one-dimensional convolution operation, and A(k) represent the amplitude component. denoted by phase component, and complex is a complex combination function.

[0230] Specifically, in order to extract the complex frequency domain information from the first processing result, we will extract the elements (x1, ..., x) from the first processing result. t ) is considered as a complex number (x1+0i, ..., x) with an imaginary part of 0. t +0i), using the spectral convolution theorem in Fourier theory, a one-dimensional discrete Fourier transform (DFT) is performed on the sequence dimension of the first processing result, transforming it from the time domain to the frequency domain, to obtain the frequency domain processing result, as shown in the following formula:

[0231]

[0232] Where X(k) represents the complex value of the k-th frequency component in the frequency domain, x(n) represents the complex value of the n-th sampling point in the time domain, and T is the sequence length. This ensures normalization consistency between the DFT and the inverse Fourier transform (IDFT).

[0233] In the frequency domain, the amplitude component A(k) and the phase component can be obtained using the real part Re(X(k)) and the imaginary part Im(X(k)) of X(k). The formula is as follows:

[0234]

[0235]

[0236] Feature extraction of the amplitude component A(k) and phase component φ(k) is performed using two independent one-dimensional convolutions. Convolution operations can capture feature patterns and phase information at different frequencies. Finally, the amplitude component A(k) is taken as the real part, and the phase component... The data is combined using the imaginary part and then transformed back to the time domain using the inverse Fourier transform to obtain the Fourier classification result, as shown in the following formula:

[0237]

[0238] Step S103: Based on the preprocessed eye movement data, the optimized hybrid network eye movement recognition model is used to obtain the eye movement recognition result.

[0239] Specifically, the optimized hybrid network eye-tracking recognition model, obtained through training, can produce eye-tracking recognition results.

[0240] This invention can extract time-domain and frequency-domain eye movement features from a hybrid network eye movement recognition model with multiple network combinations. When tested on multiple databases using data from different time periods, it shows significant performance improvement and a significant reduction in EER compared to the most advanced algorithms currently available.

[0241] It should be noted that the method of this embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this embodiment, and the multiple devices will interact with each other to complete the method described.

[0242] It should be noted that the above description describes some embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0243] The technical solutions of this invention will be clearly and completely described below with reference to the embodiments thereof. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0244] Example 1

[0245] Under the same experimental conditions, comparative experiments used the first round data from the four sub-databases RAN, TEX, FXS, and HSS in GazeBase, as well as the JuDo1000 and EMglasses databases, to evaluate four network structures—EER, DEL, Expansion CNN, and DenseLSTM—using FRR@FAR as indicators. The minimum EER value after 1000 training rounds was selected as the result, as shown in Tables 1 to 6.

[0246] Regardless of the dataset, this invention consistently performs best.

[0247]

[0248] Table 1 Comparison of RAN Database Indicators

[0249]

[0250] Table 2 Comparison of TEX Database Indicators

[0251]

[0252]

[0253] Table 3 Comparison of FXS Database Indicators

[0254]

[0255] Table 4 Comparison of HSS Database Indicators

[0256]

[0257] Table 5 Comparison of JuDo1000 Database Indicators

[0258]

[0259] Table 6 compares and evaluates EMglasses database metrics. Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of this invention, the technical features of the above embodiments or different embodiments can be combined, the steps can be implemented in any order, and many other variations exist regarding different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity.

[0260] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0261] Furthermore, given that specific details have been set forth to describe exemplary embodiments of the invention, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0262] Although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description.

[0263] The embodiments of this invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this invention should be included within the protection scope of this invention.

Claims

1. An eye-tracking recognition method based on a hybrid Transformer and LSTM network, characterized in that, include: Acquire eye-tracking data, and preprocess the eye-tracking data to obtain preprocessed eye-tracking data; A hybrid network eye-tracking recognition model is constructed, trained using the preprocessed eye-tracking data, and optimized based on the training results. The hybrid network eye-tracking recognition model includes Siamese CNN and a hybrid network, which includes LSTM, Transformer, and Fourier-CNN. Based on the preprocessed eye movement data, the optimized hybrid network eye movement recognition model is used to obtain the eye movement recognition result; The step of obtaining eye movement recognition results based on the preprocessed eye movement data and using the optimized hybrid network eye movement recognition model includes: The preprocessed eye-tracking data is input into the Siamese CNN as the source of the first network to obtain the first processing result; The first processing result is used as the source of the second network and input into the hybrid network to obtain the eye-tracking recognition result; The first processing result is used as the source of the second network and input into the hybrid network to obtain eye-tracking recognition results, including: The first processing result is input into the LSTM to obtain the recursive classification result; The first processing result is input into the Transformer to obtain the attention classification result; The first processing result is input into Fourier-CNN to obtain the Fourier classification result; The eye-tracking recognition result is obtained based on the recursive classification result, the attention classification result, and the Fourier classification result.

2. The eye-tracking recognition method based on a hybrid Transformer and LSTM network according to claim 1, characterized in that, The step of training the hybrid network eye-tracking recognition model using the preprocessed eye-tracking data and optimizing the hybrid network eye-tracking recognition model based on the training results includes: Using preprocessed eye-tracking data as the source input of the network and the eye-tracking recognition result corresponding to the preprocessed eye-tracking data as the output, the preprocessed eye-tracking data is applied to train the hybrid network eye-tracking recognition model to obtain the optimized hybrid network eye-tracking recognition model.

3. The eye-tracking recognition method based on a hybrid Transformer and LSTM network according to claim 1, characterized in that, The process of acquiring eye-tracking data and preprocessing the eye-tracking data to obtain preprocessed eye-tracking data includes: Obtaining eye-tracking data: , ; in, Represents the sequence of x-coordinates of the eye's fixation point. Represents the sequence of ordinates of the eye's fixation point. This represents the x-coordinate value at time t. This represents the value of the ordinate at time t; Based on the eye-tracking data, the velocity information at the sampling time is obtained: in, Represents the velocity of the sampling point's x-axis. Represents the velocity of the vertical axis of the sampling point. This represents the x-coordinate value of the sampling point. This represents the x-coordinate value of the sampling point at the previous moment. This represents the ordinate value of the sampling point. This represents the ordinate value of the sampling point at the previous moment. Indicates the sampling time. Indicates the moment immediately preceding the sampling; Based on the velocity information at the sampling time, fast eye movement data and slow eye movement data are obtained; The fast eye movement data and the slow eye movement data are respectively input into two multi-layer one-dimensional convolutional sub-networks with different convolutional kernel parameters to extract features at different scales, thereby obtaining extracted eye movement data. The extracted eye-tracking data is then concatenated along the channel dimension to obtain preprocessed eye-tracking data.

4. The eye-tracking recognition method based on a hybrid Transformer and LSTM network according to claim 3, characterized in that, The process of obtaining fast eye movement data and slow eye movement data based on the velocity information at the sampling time includes: Obtain the minimum speed; if the minimum speed corresponding to the speed information at the sampling time is greater than the preset minimum speed, fast data is obtained. in, This represents the segmented fast eye-tracking data. This indicates that the velocities on the horizontal and vertical axes have been normalized. This indicates that the value has been reduced to zero. This indicates the preset speed division threshold; The fast data is Z-score normalized to obtain fast eye-tracking data: in, Indicates the first Fast eye-tracking data from 1 sampling point Indicates the first The speed of each sampling point This represents the average velocity of the sampling points. The variance of the sampling point velocity; If the minimum speed responding to the speed information at the sampling time is less than the preset minimum speed, slow data is obtained; Based on the slow-motion data, slow-motion eye movement data is obtained: in, This represents slow eye movement data. , This indicates that the velocity on the horizontal and vertical axes is multiplied by a constant c and then mapped using a hyperbolic tangent function, converting the numerical range to (-1, +1).

5. The eye-tracking recognition method based on a hybrid Transformer and LSTM network according to claim 1, characterized in that, The Siamese CNN includes: A first extraction subnet and a second extraction subnet; wherein, the first extraction subnet receives fast eye-tracking data; and the second extraction subnet receives slow eye-tracking data; The first extraction subnetwork includes: a first convolutional layer, a first batch of normalization layers, a first ReLU layer, and a first average pooling layer; The second extraction subnetwork includes: a second convolutional layer, a second batch normalization layer, a second ReLU layer, and a second average pooling layer.

6. The eye-tracking recognition method based on a hybrid Transformer and LSTM network according to claim 1, characterized in that, The step of inputting the first processing result into the LSTM to obtain the recursive classification result includes: The recursive classification result is calculated using the following formula: in, This represents the forget gate of LSTM. This represents the input gate of the LSTM. This represents the output gate of the LSTM. This represents the softmax function. This represents the forget gate weight matrix. Indicates the input for the forget gate. The recursive connection weight matrix of the forget gate is represented. This indicates the hidden state in the previous moment. The bias term representing the forget gate. This represents the input gate weight matrix. This represents the input gate recursive connection weight matrix. This represents the bias term of the input gate. This represents the output gate weight matrix. This represents the output gate recursive connection weight matrix. This represents the bias term of the output gate. Indicates the state of candidate cells. Represents the hyperbolic tangent function. Represents the cell state weight matrix. The recursive connection weight matrix represents the cell state. This represents the cell state bias term. This indicates the current state of the cell. This indicates the cell state at the previous moment. This represents element-wise matrix multiplication. This represents the hidden state at time t. This represents the input weight matrix. Represents the input sequence. Indicates the input bias term. This represents the result of recursive classification. This represents the output weight matrix. This indicates the output bias term.

7. The eye-tracking recognition method based on a hybrid Transformer and LSTM network according to claim 1, characterized in that, The step of inputting the first processing result into the Transformer to obtain the attention classification result includes: Add location encoding to the first processing result to obtain location information; Add the location information to the embedding vector to obtain the embedded location code: in, Indicates the embedding position encoding. Representing the The positional encoding of the first processing result; The output of the self-attention mechanism is calculated using the following formula via embedded positional encoding: V in, This represents the output of the self-attention mechanism. Represents the normalized exponential function, This indicates that the weight matrix performs a linear transformation on the first processing result to obtain the corresponding query. This indicates that the weight matrix is ​​used to perform a linear transformation on the first processing result to obtain the corresponding key. This indicates that the weight matrix is ​​used to perform a linear transformation on the first processing result to obtain the corresponding vector. Represents the dot product. Indicates the scaling factor; The output of the multi-head attention mechanism is calculated using the following formula: in, This represents the output of the multi-head attention mechanism. This represents the array concatenation function. This represents the output of the first attention head. This represents the output of the second attention head. This represents the output of the third attention head. This represents the output of the fourth attention head. Represents a linear transformation matrix; The attention classification result is obtained using the following formula: in, This represents the output after layer normalization of the multi-head attention mechanism output. The layer normalization function represents the eigenvectors. Indicates input, Represents a linear transformation. This represents the output feature vector. This represents the activation function. This represents the learnable parameters.

8. The eye-tracking recognition method based on a hybrid Transformer and LSTM network according to claim 1, characterized in that, The step of inputting the first processing result into Fourier-CNN to obtain the Fourier classification result includes: Based on the first processing result, the frequency domain processing result is obtained using the following formula: in, Indicates the first in the frequency domain The complex values ​​of each frequency component Indicates the sequence length. This represents the nth sampling point. This represents the complex value of the nth sampling point. Represents the imaginary unit; Based on the frequency domain processing results, the amplitude component and phase component are obtained using the following formula: in, Indicates the amplitude component, Represents the phase component, Indicates the real part of the component. Indicates the imaginary part of the component. Represents the arctangent function; Based on the amplitude and phase components, a complex number is formed, and the spatiotemporal domain feature vector output is obtained through the following formula: in, Represents the time-domain eigenvector. Indicates the inverse Fourier transform. This represents a one-dimensional convolution operation. Indicates the amplitude component, denoted by phase component, and complex is a complex combination function.

Citation Information

Patent Citations

  • Space-time Transform action recognition method for sign language recognition

    CN115205966A

  • Transform-based eye movement event detection method

    CN115410266A