Ternary eye movement classification method, system and equipment based on optimization time convolution

Abstract

The invention discloses a ternary eye movement behavior classification method, system and device based on optimization time convolution, and relates to the field of artificial intelligence and human-computer interaction. The method comprises the following steps: receiving eye movement data and extracting characteristics such as speed and direction under multiple time scales as a basic set; constructing an optimized time convolution network integrating SENet channel attention, multi-head attention and residual connection, wherein the optimized time convolution network is used for capturing time sequence features; and training the model by using a focus loss function and a label smoothing technology and outputting a classification result. According to the method, by introducing multi-time scale feature extraction and a double attention mechanism, the accuracy and robustness of eye movement behavior classification are remarkably improved, and the recognition problem caused by data complexity and sample imbalance of special crowds is effectively solved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, specifically to a three-element eye-tracking behavior classification method, system, and device based on optimized temporal convolution. Background Technology

[0002] Eye movement is an important physiological signal reflecting human cognitive state, visual attention, and intention, and it has broad research and application value in psychology, cognitive science, neuromedical diagnosis, and human-computer interaction. By automatically and accurately classifying basic events such as fixation, saccades, and smooth tracking in eye movements, a key foundation can be provided for further behavioral analysis, state assessment, and interactive system development.

[0003] Traditional eye-tracking behavior classification methods primarily rely on manually designed features (such as velocity and acceleration thresholds) combined with machine learning algorithms (such as support vector machines and decision trees). These methods heavily depend on the experience of domain experts to define features and thresholds, and the features have limited representational power, resulting in insufficient model generalization ability and difficulty in adapting to individual differences and changes in eye-tracking patterns in complex scenarios.

[0004] In recent years, deep learning technology has achieved remarkable results in time series data analysis. Temporal Convolutional Networks (TCNs), as a deep learning model capable of capturing long-term dependencies in time series data, have shown great potential in eye-tracking behavior classification tasks. However, traditional TCN models have limitations in simultaneously modeling the dynamic features of eye-tracking signals at different time scales, making it difficult to fully capture the complete temporal context information encompassed by instantaneous changes to continuous tracking behavior. This results in limited recognition capabilities for complex eye-tracking patterns, especially events with continuous and dynamic characteristics such as smooth tracking, thus limiting their performance in ternary eye-tracking behavior classification tasks. Therefore, this invention proposes a ternary eye-tracking behavior classification method, system, and device based on optimized temporal convolution. Summary of the Invention

[0005] The purpose of this invention is to provide a three-element eye-movement behavior classification method, system, and device based on optimized temporal convolution. By optimizing the temporal convolutional network, it is not only applicable to general eye-movement behavior classification, but also focuses on solving the analysis problem of special data of deaf people, providing a foundation for the development of future assistive technologies.

[0006] According to a first aspect of the present invention, in order to achieve the above-mentioned objective, the present invention provides the following technical solution: a three-ary eye-tracking behavior classification method based on optimized temporal convolution, comprising the following steps: Receive eye-tracking behavior data and extract features at multiple time scales, including velocity, direction, acceleration, displacement, and standard deviation, as the basic feature set; An optimized temporal convolutional network model is constructed, which includes an SENet channel attention module, a multi-head attention module, and residual connections, for capturing time series data; The SENet channel attention module compresses, excites, and assigns values ​​to eye-tracking data, dynamically adjusting channel weights to obtain weighted output features. The multi-head attention module extracts different feature space information through parallel independent attention heads, concatenates the outputs of each head to obtain the multi-head attention output features, and fuses the SENet channel attention output features and multi-head attention output features through residual connections to obtain comprehensive features. Based on the basic feature set, the focus loss function combined with label smoothing technique is used to train the optimized temporal convolutional network model, and the trained optimized temporal convolutional network model is obtained. The eye movement data features to be identified are input into the trained optimized temporal convolutional network model, which outputs the eye movement behavior classification results.

[0007] Furthermore, the basic feature set of eye-tracking data, including velocity, direction, acceleration, displacement, and standard deviation, is obtained as follows: A progressively increasing time scale strategy is adopted to calculate the basic feature set at multiple different time steps in order to capture long-term dependencies. Each sample consists of a fixed context window, and the sequences are overlapped and pruned, meaning that there is overlapping data between adjacent sequences. Using Python 3, five features were extracted from each sample based on the time step, and features of eye-tracking data were extracted stepwise across eight time scales to capture behavioral patterns during long-term eye-tracking processes.

[0008] Furthermore, in the compression operation of the SENet channel attention module, global average pooling is performed on all pixels in each channel, reducing the size to [value missing]. The two-dimensional feature map is compressed into a scalar : In the formula, It generates a weight coefficient for each channel. For the input feature map at the th Data on each channel For dimensions.

[0009] Furthermore, in the activation operation of the SENet channel attention module, the activation formula is: in, This represents the process of compressing the input feature map, transforming it into channel weights, and obtaining the channel-wide features after compression. First, use a fully connected layer. Reduce the number of channels and use a nonlinear function. Introducing a non-linear relationship, and then passing it through a fully connected layer. To restore the number of channels, use the Sigmoid function. The attention weights are obtained by compressing the output values ​​into the (0,1) space. ,in This represents the transformation of a two-layer fully connected network.

[0010] Furthermore, the assignment operation of the SENet channel attention module: in This represents the weights generated by the channel attention mechanism, which readjust the values ​​of each channel of the input feature map to transform the original feature map. and the calculated channel attention weights ,Will Take the bus one by one to Finally, the feature map after channel attention weighting is obtained.

[0011] Furthermore, the multi-head attention module extracts different feature space information through parallel independent attention heads, and concatenates the outputs of each head to obtain the output features of the multi-head attention, as follows: (61) First, input the feature matrix It is divided into multiple subspaces, each with a dimension of . ,in It's about the number of heads to focus on. It is the dimension of the input features; (62) Perform linear transformations on the input features to obtain the query matrix. Key matrix Sum matrix ,in , , It is the trainable parameter matrix that needs to be learned to optimize the temporal convolutional network model; (63) Calculate the attention weights independently for each head and perform a weighted sum: in, It is a scaling factor to prevent the gradient from becoming unstable due to excessively large values. (64) Finally, the outputs of all attention points are concatenated, and the final output is obtained through a linear transformation: in, Indicates the first The output of each attention head, It is the parameter matrix of the output transformation. For the vertical concatenation operation of the matrix, Different attention layers The results are concatenated to form a feature vector. .

[0012] Furthermore, the focus loss function combined with label smoothing technique is used to train the optimized temporal convolutional network model, as follows: (71) Focal Loss introduces a modulation factor before cross-entropy loss to reduce the loss weight of easily classified samples. The formula is as follows: in, Focal Loss modulation factor For standard cross-entropy loss, where For the distribution of the true categories, This represents the model's predicted probability for each class, i.e., the Softmax output. To adjust the factor and control the impact of easily classified samples on the loss, The total number of samples; (72) Introduce label smoothing technology, which smooths the distribution of real labels. The loss function for label smoothing is calculated as follows: in, It is a smoothing coefficient, with a value of 0.1 or 0.01. For the number of categories, It is a one-hot encoded tag.

[0013] Furthermore, the classification results include five categories: unlabeled, gaze, saccades, smooth tracking, and noise. The optimized temporal convolutional network model outputs the probabilities of the five categories—unlabeled, gaze, saccades, smooth tracking, and noise—through fully connected layers and the Softmax function.

[0014] According to a second aspect of the present invention, the present invention provides a trigram eye-tracking behavior classification system based on optimized temporal convolution, for implementing the trigram eye-tracking behavior classification method based on optimized temporal convolution described in the first aspect, comprising: The data receiving module is used to receive the velocity, direction, acceleration, displacement, and standard deviation features of eye-tracking data as a basic feature set; The model building module is used to build an optimized temporal convolutional network model, which includes an SENet channel attention module, a multi-head attention module, and residual connections, for capturing time-series data. The SENet channel attention module compresses, excites, and assigns values ​​to eye-tracking data, dynamically adjusting channel weights to obtain weighted output features. The multi-head attention module extracts different feature space information through parallel independent attention heads, concatenates the outputs of each head to obtain the multi-head attention output features, and fuses the SENet channel attention output features and multi-head attention output features through residual connections to obtain comprehensive features. The model training module is used to train the optimized temporal convolutional network model based on the basic feature set, using the focus loss function combined with label smoothing technology, to obtain the trained optimized temporal convolutional network model. The recognition output module is used to input the eye movement data features to be recognized into the trained optimized temporal convolutional network model and output the eye movement behavior classification results.

[0015] According to a third aspect of the present invention, the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor loads and executes the computer program, it employs the three-ary eye-tracking behavior classification method based on optimized temporal convolution described in the first aspect.

[0016] This invention has at least the following beneficial effects: 1. This invention optimizes the complex characteristics of eye-tracking data from the deaf population, such as non-stationarity and high variability. The synergistic effect of multi-scale feature extraction and SENet channel attention and multi-head attention mechanisms enables adaptive capture of richer dynamic features in the eye-tracking behavior of deaf individuals. At the same time, the implementation of anti-interference training strategies significantly enhances the robustness of the model to individual-differentiated data. This method is not only applicable to general eye-tracking behavior analysis, but also demonstrates excellent adaptability in processing data from special populations, providing a reliable technical foundation for cognitive research and human-computer interaction technology development for the deaf.

[0017] 2. This invention employs a progressively increasing multi-timescale feature extraction strategy, simultaneously calculating core eye-tracking features such as velocity, direction, acceleration, displacement, and standard deviation across multiple time dimensions of different granularities. This effectively captures the full-cycle dynamic characteristics of eye-tracking behavior, from instantaneous changes to long-term trends. This multi-scale modeling approach significantly enhances the model's ability to characterize long-term dependencies, making the semantic expression of complex eye-tracking patterns more complete and providing richer temporal contextual information for subsequent classification decisions.

[0018] 3. This invention integrates the SENet channel attention mechanism and the multi-head attention mechanism to construct a hierarchical feature selection system. The SENet channel attention mechanism effectively suppresses redundant information interference by dynamically evaluating the importance weight of each feature channel, significantly enhancing the model's sensitivity to key behavioral features. The multi-head attention mechanism, on the other hand, mines the correlation between global and local features in parallel from multiple independent perspectives, particularly improving the model's accuracy in discriminating subtle eye movement patterns. The two attention mechanisms achieve feature fusion through residual connections, which not only preserves the original information flow but also enhances the effectiveness of feature selection, enabling the model to maintain stable feature extraction performance even in noisy environments.

[0019] 4. This invention introduces an optimized training strategy combining a focus loss function and label smoothing technology. The focus loss function, through an adaptive adjustment mechanism of weight allocation between easy and difficult samples, allows the model training process to focus more on learning deep features of minority class difficult samples such as smooth tracking and saccades. Label smoothing technology effectively mitigates the risk of overfitting caused by labeling errors and reduces the model's excessive reliance on incorrect labels. This strategy significantly improves the classification accuracy of minority class behaviors, enabling the model to possess more balanced recognition capabilities in practical applications.

[0020] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method described in this invention; Figure 2 This is a schematic diagram illustrating the principle of multi-timescale feature extraction in this invention; Figure 3 This is a schematic diagram of the structure of the multi-head attention module of the present invention; Figure 4 This is a schematic diagram of the overall structure of the optimized temporal convolutional network model of the present invention; Figure 5 The confusion matrix comparison diagram of the optimized temporal convolutional network model of the present invention is shown: (a) is the original temporal convolutional network model; (b) is the optimized temporal convolutional network model. Detailed Implementation

[0022] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0023] Example 1: Explanation of related terms: Temporal Convolutional Networks (TCNs) are a deep learning architecture that excels at processing time-series data. They combine causal convolutions, dilated convolutions, and residual connections to effectively capture patterns in time-series data.

[0024] Causal convolution: The key characteristic of causal convolution is that when processing the current time step, the model can only use data from the current and previous time steps, and cannot use future data. This feature ensures the causality of the model, avoids the leakage of future information, and maintains the temporal consistency of the data.

[0025] Dilated convolution extends the functionality of causal convolution by expanding the receptive field through varying convolution strides, thereby enabling the extraction of information over larger temporal scales. For eye-tracking behavior prediction, dilated convolution not only relies on past information but also leverages future information to improve prediction accuracy.

[0026] Residual modules are used to alleviate the vanishing gradient problem in deep networks, ensuring efficient information flow and accelerating the training process. Residual connections allow the network to skip connections between deeper layers, reducing training difficulties and improving model performance.

[0027] Please see Figures 1-5 This invention provides a technical solution: a three-element eye-tracking behavior classification method based on optimized temporal convolution, comprising the following steps: S1. Receive eye-tracking behavior data and extract features at multiple time scales, including velocity, direction, acceleration, displacement, and standard deviation, as a basic feature set; We used Python 3 to extract features from eye-tracking data at different time scales to better capture behavioral patterns during long-term eye-tracking processes. Let the coordinates of the i-th eye-tracking point be ( , ), , The first point, The sampling time and displacement of a point can be expressed as: Speed ​​can be expressed as: Acceleration can be expressed as: Motion can be represented as: The standard deviation can be expressed as: ; Five features are obtained through the above calculations. These features are then calculated on progressively increasing time scales (e.g., [1, 2, 4, 8, 16, 32, 64, 128]) to capture long-term dependencies. Each sample consists of a fixed context window of approximately 1 second (257 samples), and overlapping pruning is used between sequences, meaning that adjacent sequences have overlapping data. For example, when the window size is 257 samples, the number of overlapping samples is 192; when the window size is 385 samples, the number of overlapping samples is 312. Overlap pruning ensures a fixed number of sequences (50,000 sequences), which is achieved by appropriately adjusting the window size and the number of overlapping samples. The feature extraction process extracts five features from each sample according to the time step and progressively extracts them across eight time scales. Figure 2 This is a schematic diagram of the multi-timescale feature extraction module. This method effectively captures the dynamic changes of eye movements at different timescales, which helps to improve the accuracy of eye movement behavior analysis. S2. Construct and optimize the temporal convolutional network model. The overall structure diagram of the optimized temporal convolutional network model is shown below. Figure 4 As shown, it specifically includes: The OTCN (Time-Based Convolutional Network) model is an optimized version of TCN (Time-Based Convolutional Network) that improves upon it to enhance feature extraction capabilities and reduce the number of parameters. The model consists of the following modules: Feature extraction module Five features were calculated and extracted at eight progressively increasing time scales, namely the time step values ​​[1, 2, 4, 8, 16, 32, 64, 128]. SENet Channel Attention Module: Used to emphasize important feature channels, reduce redundant features, and improve the ability to express high-level features; global average pooling and max pooling are performed on eye-tracking data in the time dimension to generate 1D vectors; the activation function is ReLU, and Dropout is set to 0.3; Temporal convolutional layers: extract local features and capture long-term dependencies; use 128 filters with tanh activation function and Dropout of 0.3; residual connections are integrated to solve the problems of gradient vanishing and memory loss. Multi-head attention module: Global dependency modeling is performed on the output of the temporal convolutional layer, focusing on the most informative part of the sequence; the input sequence embedding is divided into 3 heads for parallel processing, and the dropout missing rate is set to 0.3.

[0028] Since two attention mechanisms are introduced, increasing the number of network layers may lead to gradient vanishing. Residual connections are introduced to alleviate this problem. At the same time, the outputs of SENet channel attention and multi-head attention are added to enhance the performance of the model. Output module: It consists of a fully connected layer with 5 units, outputting 5 categories (unlabeled, gaze, saccades, smooth tracking, and noise). The Softmax function is used to predict the probability of each category, and the output size is 257. 5; The SENet channel attention module compresses, excites, and assigns values ​​to eye-tracking data, dynamically adjusting channel weights to obtain weighted output features. The multi-head attention module extracts different feature space information through parallel independent attention heads, concatenates the outputs of each head to obtain the multi-head attention output features, and fuses the SENet channel attention output features and multi-head attention output features through residual connections to obtain comprehensive features. S21.SENet is a common channel attention method that focuses on extracting task-critical features and suppressing redundant features by evaluating the importance of features in different channels. It is particularly suitable for eye-tracking classification tasks. SENet dynamically adjusts the weights of each channel through compression, activation, and assignment operations, thereby strengthening the model's focus on important features, as detailed below: (S21.1) The squeezing formula is: in, It generates a weight coefficient for each channel. For the input feature map at the th Data on each channel, with a size of [size missing]. Perform global average pooling on all pixels in each channel, and... The two-dimensional feature map is compressed into a scalar This indicates the global information for that channel; (S21.2) In the excitation operation of the SENet channel attention module, the excitation formula is: Among them, the channel global features obtained after the squeeze operation , This represents the process of compressing the input feature map, transforming it into channel weights, and obtaining the channel-wide features after compression. First, use a fully connected layer. Reduce the number of channels and use a nonlinear function. Introducing a non-linear relationship, and then passing it through a fully connected layer. To restore the number of channels, use the Sigmoid function. The attention weights are obtained by compressing the output values ​​into the (0,1) space. ,in This represents the transformation of a two-layer fully connected network; (S21.3) In the assignment operation of the SENet channel attention module, the assignment (scale) formula is: in This represents the weights generated by the channel attention mechanism, which readjust the values ​​of each channel of the input feature map to transform the original feature map. and the calculated channel attention weights ,Will Take the bus one by one to Finally, the feature map after channel attention weighting is obtained; S22. Multi-head attention is an improvement on traditional attention mechanisms. By splitting the input data into multiple sub-sequences and processing them separately, it enhances the model's ability to focus on different features. Through multiple independent attention mechanisms operating in parallel, multi-head attention can extract information from different feature spaces, enhancing the model's ability to express input features. Finally, the outputs of each head are concatenated and transformed to generate a comprehensive representation of the input. Figure 3 This is a schematic diagram of the multi-head attention mechanism module structure; The input features are calculated using multiple independent attention mechanisms, and the outputs of each head are concatenated to form the final result. (S22.1) First, input the feature matrix It is divided into multiple subspaces, each with a dimension of . ,in It's about the number of heads to focus on. This refers to the dimension of the input features. Linear transformations are then performed on each input feature to obtain the query matrix. Key matrix Sum matrix ,in , , It is the trainable parameter matrix that the model needs to learn; (S22.2) Calculate the attention weights independently for each head and perform a weighted sum: in, It is a scaling factor to prevent the gradient from becoming unstable due to excessively large values. (S22.3) Finally, the outputs of all attention points are concatenated, and the final output is obtained through a linear transformation: in, Indicates the first The output of each attention head, It is the parameter matrix of the output transformation. For the vertical concatenation operation of the matrix, Different attention layers The results are concatenated to form a feature vector. ; S3. Based on the basic feature set, the focus loss function combined with label smoothing technology is used to train the optimized temporal convolutional network model, and the trained optimized temporal convolutional network model is obtained. S31. To address the issues of class imbalance and excessively high weights on easily classified samples, Focal Loss introduces a modulation factor before cross-entropy loss to reduce the loss weights of easily classified samples. The formula is as follows: The formula is divided into two parts. Focal Loss modulation factor For standard cross-entropy loss, where For the distribution of the true categories, This represents the model's predicted probability for each class, i.e., the Softmax output. To adjust the factor and control the impact of easily classified samples on the loss, The total number of samples; S32. In model training, it's inevitable that the model may become overly reliant on incorrect labels. Therefore, label smoothing is introduced to smooth the distribution of true labels, reducing the model's dependence on incorrect labels and mitigating the risk of overfitting. The formula for calculating the loss function using label smoothing is as follows: in, It is a smoothing coefficient, typically with a value of 0.1 or 0.01. For the number of categories, These are one-hot encoded labels; assuming there are 3 categories of one-hot encoded labels [0,0,1], let... When the value is 0.1, the target label after smoothing becomes [0.033, 0.033, 0.933]. This makes the label less extreme, which helps to alleviate the model's overconfident prediction of certain categories. It should be further noted that during training, the model was trained using leave-one-out cross-validation, running a total of 18 experiments to ensure its robustness with small sample sizes. The Adam optimizer was used with an initial learning rate of 0.0001, a batch size of 128, and 20 epochs. The model was validated and evaluated on the public dataset GazeCom, using the F1 score as the evaluation metric. S4. Input the eye movement data features to be identified into the trained optimized temporal convolutional network model, and output the eye movement behavior classification results.

[0029] The technical effects of the present invention will now be verified with reference to specific embodiments: Experiments were conducted on a deaf eye-tracking behavior dataset. As shown in Table 1, the proposed OTCN model outperformed the comparison models on the GazeCom dataset, particularly excelling in smooth tracking classification. Its F1 score was 2.9% higher than the baseline model TCN, and it also improved by 0.1% and 0.2% in fixation and saccades, respectively. OTCN enhances its ability to focus on key features by optimizing temporal convolution and fusion attention mechanisms, demonstrating significant advantages in handling minority class tasks. Compared to other models such as decision trees, CPC, TCN, and TPE-TCN, OTCN outperforms in minority class learning and feature extraction in smooth tracking.

[0030] Table 1. Algorithm test results for the GazeCom dataset. Table 2 shows the impact of window size on the classification performance of the OTCN model. The results indicate that an appropriate window length significantly improves classification performance. Experiments revealed that a 1.5-second window (385 samples) performed best, especially in smooth tracking, where the F1 score improved by 1.6% compared to a 1-second window (257 samples). Increasing the window size to 2 seconds (514 samples) improved performance by only 0.6% compared to a 1-second window (257 samples). Excessively large windows, such as those with 514 samples, did not provide significant improvement and instead reduced efficiency due to noise or redundant information. Overall, the 1.5-second (385 samples) context window achieved the highest evaluation score. This suggests that an appropriate window length helps capture the complete behavior of gaze and smooth tracking, while excessively large windows introduce interfering factors that negatively impact classification performance.

[0031] Table 2 Test results of different time window algorithms To verify the performance improvement effect of each module, ablation experiments were conducted on the GazeCom dataset. The performance differences between the model and the baseline TCN were compared using focal loss, multi-head attention, and SE modules, as well as the combinations thereof. The experimental results are shown in Table 3. The results indicate that the model integrating focal loss, SE, and multi-head attention (MHA) achieves the best overall performance.

[0032] The focal loss function improves the F1 score of smooth tracking by 0.5%, the SE module by 2.0%, and multi-head attention by 1.1%, but has little impact on fixation and saccade types. The model that integrates focal loss, SE, and multi-head attention has the best overall performance, especially in smooth tracking classification, where the correct classification ratio increases from 0.72 to 0.75, the accuracy improves by 4.2%, and the probability of misclassification is significantly reduced.

[0033] Table 3 Ablation Experiment Results Depend on Figure 5 visible, Figure 5 (a) is the original temporal convolutional network model; Figure 5 In (b), to optimize the temporal convolutional network model, the overall misclassification probability of the optimized temporal convolutional model is reduced, especially in the classification performance of smooth tracking, which is significantly improved, with the proportion increasing from 0.72 to 0.75.

[0034] Example 2: This invention provides a trinomial eye-tracking behavior classification system based on optimized temporal convolution, used to implement the trinomial eye-tracking behavior classification method based on optimized temporal convolution described in Embodiment 1, comprising: The data receiving module is used to receive the velocity, direction, acceleration, displacement, and standard deviation features of eye-tracking data as a basic feature set; The model building module is used to build an optimized temporal convolutional network model, which includes an SENet channel attention module, a multi-head attention module, and residual connections, for capturing time-series data. The SENet channel attention module compresses, excites, and assigns values ​​to eye-tracking data, dynamically adjusting channel weights to obtain weighted output features. The multi-head attention module extracts different feature space information through parallel independent attention heads, concatenates the outputs of each head to obtain the multi-head attention output features, and fuses the SENet channel attention output features and multi-head attention output features through residual connections to obtain comprehensive features. The model training module is used to train the optimized temporal convolutional network model based on the basic feature set, using the focus loss function combined with label smoothing technology, to obtain the trained optimized temporal convolutional network model. The recognition output module is used to input the eye movement data features to be recognized into the trained optimized temporal convolutional network model and output the eye movement behavior classification results.

[0035] Example 3: This embodiment provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor. When the processor loads and executes the computer program, it adopts the three-element eye-tracking behavior classification method based on optimized temporal convolution described in Embodiment 1.

[0036] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server, and the terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0037] Furthermore, the processor can be a central processing unit (CPU). Of course, depending on the actual use, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be used. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it in this regard.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0039] For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances. When an element is referred to as being "assembled on," "mounted on," "fixed to," or "set on" another element, it may be directly on the other element or there may be an intermediate element present. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element present. The terms "vertical," "horizontal," "upper," "lower," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible embodiments.

[0040] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0041] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

Claims

1. A three-ary eye-tracking behavior classification method based on optimized temporal convolution, characterized in that, Includes the following steps: Receive eye-tracking behavior data and extract features at multiple time scales, including velocity, direction, acceleration, displacement, and standard deviation, as the basic feature set; An optimized temporal convolutional network model is constructed, which includes an SENet channel attention module, a multi-head attention module, and residual connections, for capturing time series data; The SENet channel attention module compresses, excites, and assigns values ​​to eye-tracking data, dynamically adjusting channel weights to obtain weighted output features. The multi-head attention module extracts different feature space information through parallel independent attention heads, concatenates the outputs of each head to obtain the multi-head attention output features, and fuses the SENet channel attention output features and multi-head attention output features through residual connections to obtain comprehensive features. Based on the basic feature set, the focus loss function combined with label smoothing technique is used to train the optimized temporal convolutional network model, and the trained optimized temporal convolutional network model is obtained. The eye movement data features to be identified are input into the trained optimized temporal convolutional network model, which outputs the eye movement behavior classification results.

2. The three-element eye-tracking behavior classification method based on optimized temporal convolution according to claim 1, characterized in that: The basic feature set of eye-tracking data includes velocity, direction, acceleration, displacement, and standard deviation features, which are obtained as follows: A progressively increasing time scale strategy is adopted to calculate the basic feature set at multiple different time steps in order to capture long-term dependencies. Each sample consists of a fixed context window, and the sequences are overlapped and pruned, meaning that there is overlapping data between adjacent sequences. Using Python 3, five features were extracted from each sample based on the time step, and features of eye-tracking data were extracted stepwise across eight time scales to capture behavioral patterns during long-term eye-tracking processes.

3. The three-ary eye-tracking behavior classification method based on optimized temporal convolution according to claim 2, characterized in that: In the compression operation of the SENet channel attention module, global average pooling is performed on all pixels in each channel, reducing the size to [value missing]. The two-dimensional feature map is compressed into a scalar : In the formula, It generates a weight coefficient for each channel. For the input feature map at the th Data on each channel For dimensions.

4. The three-element eye-tracking behavior classification method based on optimized temporal convolution according to claim 3, characterized in that: In the activation operation of the SENet channel attention module, the activation formula is: in, This represents the process of compressing the input feature map, transforming it into channel weights, and obtaining the channel-wide features after compression. First, use a fully connected layer. Reduce the number of channels and use a nonlinear function. Introducing a non-linear relationship, and then passing it through a fully connected layer. To restore the number of channels, use the Sigmoid function. The attention weights are obtained by compressing the output values ​​into the (0,1) space. ,in This represents the transformation of a two-layer fully connected network.

5. The three-element eye-tracking behavior classification method based on optimized temporal convolution according to claim 4, characterized in that: Assignment operations for the SENet channel attention module: in This represents the weights generated by the channel attention mechanism, which readjust the values ​​of each channel of the input feature map to transform the original feature map. and the calculated channel attention weights ,Will Take the bus one by one to Finally, the feature map after channel attention weighting is obtained.

6. The three-element eye-tracking behavior classification method based on optimized temporal convolution according to claim 5, characterized in that: The multi-head attention module extracts information from different feature spaces through parallel independent attention heads, and concatenates the outputs of each head to obtain the output features of the multi-head attention, as detailed below: (61) First, input the feature matrix It is divided into multiple subspaces, each with a dimension of . ,in It's about the number of heads to focus on. It is the dimension of the input features; (62) Perform linear transformations on the input features to obtain the query matrix. Key matrix Sum matrix ,in , , It is the trainable parameter matrix that needs to be learned to optimize the temporal convolutional network model; (63) Calculate the attention weights independently for each head and perform a weighted sum: in, It is a scaling factor to prevent the gradient from becoming unstable due to excessively large values. (64) Finally, the outputs of all attention points are concatenated, and the final output is obtained through a linear transformation: in, Indicates the first The output of each attention head, It is the parameter matrix of the output transformation. For the vertical concatenation operation of the matrix, Different attention layers The results are concatenated to form a feature vector. .

7. The three-element eye-tracking behavior classification method based on optimized temporal convolution according to claim 5, characterized in that: The focus loss function combined with label smoothing technique is used to train the optimized temporal convolutional network model, as detailed below: (71) Focal Loss introduces a modulation factor before cross-entropy loss to reduce the loss weight of easily classified samples. The formula is as follows: in, Focal Loss modulation factor For standard cross-entropy loss, where For the distribution of the true categories, This represents the model's predicted probability for each class, i.e., the Softmax output. To adjust the factor and control the impact of easily classified samples on the loss, The total number of samples; (72) Introduce label smoothing technology, which smooths the distribution of real labels. The loss function for label smoothing is calculated as follows: in, It is a smoothing coefficient, with a value of 0.1 or 0.

01. For the number of categories, It is a one-hot encoded tag.

8. The three-element eye-tracking behavior classification method based on optimized temporal convolution according to claim 7, characterized in that: The classification results include five categories: unlabeled, gaze, saccades, smooth tracking, and noise. The optimized temporal convolutional network model outputs the probabilities of the five categories—unlabeled, gaze, saccades, smooth tracking, and noise—through fully connected layers and the Softmax function.

9. A trinomial eye-tracking behavior classification system based on optimized temporal convolution, used to implement the trinomial eye-tracking behavior classification method based on optimized temporal convolution as described in any one of claims 1 to 8, characterized in that, include: The data receiving module is used to receive the velocity, direction, acceleration, displacement, and standard deviation features of eye-tracking data as a basic feature set; The model building module is used to build an optimized temporal convolutional network model, which includes an SENet channel attention module, a multi-head attention module, and residual connections, for capturing time-series data. The SENet channel attention module compresses, excites, and assigns values ​​to eye-tracking data, dynamically adjusting channel weights to obtain weighted output features. The multi-head attention module extracts different feature space information through parallel independent attention heads, concatenates the outputs of each head to obtain the multi-head attention output features, and fuses the SENet channel attention output features and multi-head attention output features through residual connections to obtain comprehensive features. The model training module is used to train the optimized temporal convolutional network model based on the basic feature set, using the focus loss function combined with label smoothing technology, to obtain the trained optimized temporal convolutional network model. The recognition output module is used to input the eye movement data features to be recognized into the trained optimized temporal convolutional network model and output the eye movement behavior classification results.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor loads and executes the computer program, it implements the three-element eye-tracking behavior classification method based on optimized temporal convolution as described in any one of claims 1 to 8.

Images