Eye movement recognition method based on Transform global semantic coding

Abstract

The invention relates to an eye movement recognition method based on Transform global semantic coding, and relates to the field of eye movement recognition. Comprising the steps of collecting corresponding eye movement data according to different task requirements, and constructing an eye movement recognition data set; the eye movement data set comprises an image and eye movement data corresponding to the watched image; according to the fixation point position recorded in the eye movement data, performing fixation point mask processing on the related image, and taking the image subjected to mask processing and the image not subjected to mask processing as the input of a Transform model; the method comprises the following steps: carrying out pre-training on a Transform model; extracting deep features from all fixation point positions of each image by using a pre-trained Transform model, and predicting an image-based category probability based on the deep features; and integrating the category probabilities based on the images to obtain a comprehensive classification result of the subjects. The problems that an existing eye movement recognition method is insufficient in deep semantic information mining and cannot capture the global dependency relationship are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of eye movement recognition, and in particular to an eye movement recognition method based on Transformer global semantic encoding, that is, extracting semantic features at the visual fixation point of an image through a Transformer model, and using these features to classify eye movement data. Background Art

[0002] Eye tracking is a technology used to record and analyze human eye movements. By capturing the movement of the eyeballs in space and time, it can reveal an individual's visual attention and perception process. Eye tracking data describes human visual attention behavior. By analyzing eye movement data, researchers identify differences in visual attention, cognitive processing, and decision-making behavior among different groups, thereby classifying groups of different ages, genders, professional backgrounds, or cognitive states. Eye movement recognition can provide a scientific basis for personalized services, precision education, and precise medical intervention.

[0003] The earliest saliency attention model was proposed by Itti et al. in the article "A Model of Saliency-Based Visual Attention for Rapid Scene Analysis", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1254-1259, 1998. It simulates the processing method of the primate visual system and uses multi-scale image features to generate saliency maps to quickly identify and analyze salient locations in complex scenes. Later, Wang et al. proposed a saliency attention model in the article "Deep visual attention prediction", IEEE Trans.

[0004] Image Process., vol.27, no.5, pp.2368–2378, May 2018. proposed a saliency prediction model based on deep learning, which predicts human visual attention through encoder-decoder architecture and multi-level and multi-scale feature fusion. These saliency attention models mainly focus on the common features reflected in people's visual attention, but often ignore the differences in attention between different groups or individuals.

[0005] In recent years, researchers have gradually noticed that there are differences in eye movement patterns among different individuals and groups, and have tried to reveal how these differences affect people's processing and understanding of visual information. At the same time, eye movement recognition has also become a research direction that has attracted much attention. In the article "Gender-based eye movement differences in passive indoor picture viewing: An eye-tracking study", Physiology & behavior, vol. 206, pp. 43–50, 2019, Vallabh et al. extracted low-level features to train random forest and logistic regression classifiers to classify people of different genders. Traditional methods only rely on low-level features to represent eye movement features, but do not extract high-level semantic information of the image, ignoring the deep features in the visual information.

[0006] Jiang et al. first proposed an autism recognition model based on deep learning in “Learning visual attention to identify people with autism spectrum disorder”, in Proc. IEEE Int. Conf. Comput. Vis., Venice, Italy, 2017, pp. 1–8. The model extracted gaze features using a convolutional neural network (CNN) and trained a SVM classifier to classify autism and typical development groups. Xu et al. proposed a personalized saliency detection computational model in “Y.Xu, S.Gao, J.Wu, N.Li, and J.Yu, “Personalized saliency and its prediction”, IEEE transactions on pattern analysis and machine intelligence, vol.41, no.12, pp.2975–2989, 2018. They used two prediction methods, multi-task CNN and extended CNN with person-specific information encoding filters, to perform saliency estimation. However, when using CNN models to process images, they tend to focus more on local features and cannot effectively capture global information. This becomes a significant disadvantage in complex visual tasks, especially in situations where the overall scene or long-distance contextual relationships need to be understood.

[0007] In addition, existing research has not yet formed a unified and efficient standard for identifying population differences, a challenge that is particularly prominent under diverse populations and dynamically changing environmental conditions. Existing models often rely on training on specific datasets, which may limit their generalization capabilities in unseen new environments or data. Summary of the invention

[0008] The technical problems to be solved by the present invention are:

[0009] Although there have been some research progress in the field of eye movement recognition, the accuracy of the recognition method still needs to be improved due to insufficient mining of eye movement data features. At present, the main problems are as follows: First, existing methods usually only rely on low-level features to represent eye movement features. These methods fail to effectively combine deep features and thus fail to fully mine semantic information; second, existing deep learning models often only focus on the local representation of each fixation area. This method cannot capture the global semantic dependency when processing the coupled input of image and eye movement data; third, few existing eye movement recognition algorithms can be applied to a variety of different scenarios and tasks across multiple fields, and the generalization ability of the model needs to be improved. Therefore, the purpose of the present invention is to overcome the shortcomings of the existing technology, integrate the global semantic information in the observer's visual attention, and propose an eye movement recognition framework based on global semantic encoding, which uses the bidirectional encoder representation of Transformer (BERT) to extract high-level representations from coupled inputs and capture global relationships. Specifically, a gaze point-guided masking strategy is first adopted to enhance the learning of fixation areas, predict the visual markers of masked image blocks, and learn the contextual relationships and global semantic information between image blocks. Then, the pre-trained Transformer encoder is used to extract the features of the fixation area of ​​the image, and the extracted features are used for classification. This can effectively solve the problems of insufficient mining of deep semantic information and inability to capture global dependencies.

[0010] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0011] An eye movement recognition method based on Transformer global semantic encoding, characterized by comprising:

[0012] Collect corresponding eye movement data according to different task requirements and construct an eye movement recognition data set; the eye movement data set includes images and eye movement data corresponding to viewing the images;

[0013] According to the gaze point position recorded in the eye movement data, the relevant images are masked at the gaze point, and the masked images and unmasked images are used as inputs of the Transformer model;

[0014] Pre-train the Transformer model;

[0015] Use the pre-trained Transformer model to extract deep features from all gaze positions in each image, and predict image-based class probabilities based on the deep features;

[0016] The image-based category probabilities are integrated to obtain the comprehensive classification results of the subjects.

[0017] A further technical solution of the present invention is to collect corresponding eye movement data according to different task requirements and construct an eye movement recognition data set, specifically:

[0018] We collected our own datasets for the two tasks of ASD identification and gender identification. Specifically, we recruited subjects from different groups and recorded their eye movement data by having them watch a series of preset images.

[0019] For the two tasks of task identification and identity recognition, two public datasets are directly selected.

[0020] A further technical solution of the present invention is to perform a mask operation at the fixation point on the relevant image according to the fixation point position recorded in the eye movement data, specifically:

[0021] The image is divided into multiple image blocks, and the image block where the gaze point is located is masked.

[0022] A further technical solution of the present invention is to pre-train the Transformer model, specifically:

[0023] Before training begins, a pre-trained discrete variational autoencoder (DVAE) is used to process the entire training set, converting each image patch into a fixed-length visual label vector.

[0024] During the training process, the Transformer model uses its self-attention mechanism to process the mixed information of masked and unmasked image patches, and predicts the content of the masked area by understanding the context of the image; the difference between the model's output and the corresponding visual marker is quantified through the cross-entropy loss, and then this loss is minimized to adjust its parameters and optimize the prediction ability.

[0025] A further technical solution of the present invention is to predict the image-based category probability based on deep features, specifically:

[0026] All gaze point features extracted from the image are integrated through the average pooling method to obtain image-level features, which are then sent to the final classification layer. The coupled input of the image and eye movement data is mapped to a probability distribution to predict the probability that the image belongs to each category.

[0027] A further technical solution of the present invention is to integrate the image-based category probabilities to obtain a comprehensive classification result of the subject, specifically:

[0028] The class probabilities of all images for each subject are averaged to obtain a comprehensive feature vector representing the subject:

[0029]

[0030] Among them, N k represents the number of valid images of the kth subject;

[0031] The final classification of the subject is given by the output vector The class corresponding to the maximum value in is determined.

[0032] A computer system, characterized in that it comprises: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0033] A computer-readable storage medium, characterized in that it stores computer-executable instructions, which are used to implement the above method when executed.

[0034] A computer program product, characterized by comprising computer executable instructions, wherein the instructions are used to implement the above method when executed.

[0035] The beneficial effects of the present invention are:

[0036] The eye movement recognition method based on Transformer global semantic coding provided by the present invention has the following advantages:

[0037] 1. Through large-scale pre-training, the present invention constructs a universal eye tracking and recognition framework, which can effectively extract global semantic features. Compared with traditional methods that rely on the underlying features of eye movement data and lack the integration of global semantic information of the image, the framework can more effectively encode eye movement data, thereby achieving more accurate classification.

[0038] 2. This paper proposes a gaze-guided image masking method that focuses on distinguishing object regions and better models the relationship between image contexts. This method can also be extended to other mask-based applications to utilize eye tracking data to improve the results of other single modalities.

[0039] 3. This invention solves the problem of identity recognition from the perspective of semantic features for the first time, making identity recognition no longer rely solely on traditional biometric features. Identity recognition is performed by analyzing the unique way individuals observe and process visual information, effectively expanding the data dimension and complexity of identity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0041] Figure 1 The figure is a flow chart of the method of the present invention.

[0042] Figure 2 It is the overall framework diagram of the model of the present invention.

[0043] Figure 3 Schematic diagram of model input (coupled image and eye movement data) and output under different tasks in the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0045] The present invention provides an eye movement recognition method based on Transformer global semantic coding, comprising: first, collecting corresponding eye movement data according to different task requirements and constructing an eye movement recognition data set. Then, according to the gaze point position recorded in the eye movement data, a mask operation is performed on the relevant image at the gaze point. In the pre-training stage, the Transformer architecture is used to predict the visual mark of the mask position. In the eye movement recognition stage, the pre-trained Transformer model is used to extract deep features from all gaze point positions of each image, these features are fused to predict the image score, and then multiple image-based weak classifiers are integrated to generate a comprehensive classification result for the subject. The implementation steps include the following:

[0046] (1) Building an eye movement recognition dataset

[0047] For the two tasks of ASD identification and gender identification, we collected data sets by ourselves, recruited subjects from different groups, and asked them to watch a series of preset pictures and record their eye movement data to complete the construction of the data sets. For the two tasks of task identification and identity recognition, we selected two public data sets for experiments.

[0048] (2) Mask the image according to the gaze point position

[0049] The image is divided into multiple image patches, and the image patches where the gaze point is located are masked to generate a set of mask sequences of the image. Then, these masked image patches are replaced with learnable mask embeddings. These mask embedding vectors are passed to subsequent layers for processing along with other unmasked embedding vectors.

[0050] (3) Visual marker prediction at the mask

[0051] Before training begins, a pre-trained discrete variational autoencoder (DVAE) is used to process the entire training set, converting each image patch into a fixed-length visual tag vector. During training, the Transformer model uses its self-attention mechanism to process the mixed information of masked and unmasked image patches, and predicts the content of the masked area by understanding the context of the image. The difference between the model's output and the corresponding visual tag is quantified using the cross-entropy loss, and then this loss is minimized to adjust its parameters and optimize the prediction ability.

[0052] (4) Extracting deep features at the gaze point

[0053] A pre-trained Transformer encoder is used to process the input image and generate a series of feature representations, each of which corresponds to an image patch. Special attention is paid to the image patch where the fixation point is located. These patches are processed by the model's self-attention mechanism to extract key semantic information related to the fixation point. The self-attention mechanism enables the model to calculate the relationship weights between image patches, thereby capturing deep features at the fixation point. These features not only cover the visual content of the fixation point itself, but also include the interaction and semantic association between the fixation point and the image context.

[0054] (5) Classification based on images

[0055] All gaze point features extracted from the image are integrated through the average pooling method, so that the obtained image-level features can reflect the visual information of all gaze points on the image. Then the image-level features are sent to the final classification layer, which maps the coupled input of the image and eye movement data to a probability distribution and predicts the probability of the image belonging to each category.

[0056] (6) Classification based on subjects

[0057] When classifying each subject, a model ensemble-like strategy was adopted to perform classification by integrating the output results produced after viewing all images, averaging the output vectors of all images for each subject, and obtaining a comprehensive feature vector representing the subject.

[0058] In order to enable those skilled in the art to better understand the present invention, the present invention is described in detail below in conjunction with specific embodiments.

[0059] Example:

[0060] Step 1: Build an eye movement recognition dataset

[0061] For the two tasks of ASD identification and gender identification, we collected data sets by ourselves, recruited subjects from different groups, and asked them to watch a series of preset pictures and record their eye movement data to complete the construction of the data sets. For the two tasks of task identification and identity recognition, we selected two public data sets for experiments.

[0062] Step 2: Mask the image based on the gaze point position

[0063] For an input image I∈R H×W×C , where H, W, C are the width, height and number of channels of the image respectively. Image I can be divided into N image blocks, where N = HW / P 2 , (P, P) is the resolution of the image block. At this time, the vector composed of image blocks can be used To represent the input image.

[0064] The mask sequence of the image is obtained by masking the image block where the gaze point is located, which is denoted as M.

[0065] Each image block is mapped into a linear space get The masked image patches are embedded with learnable masks Ex M These masked embeddings are passed to subsequent layers for processing along with other unmasked embeddings.

[0066] Step 3: Visual marker prediction at the mask

[0067] To model the mask image, DVAE is used to first learn a visual vocabulary to encode each image patch into discrete visual tags. The encoder encodes each patch x i Mapped to a discrete latent space z. The output of the encoder is a probability distribution:

[0068] q(z|x i )=softmax(f φ (x i )),

[0069] Among them, f φ (x i ) is the neural network output of the encoder. The decoder then maps the discrete latent variable z back to the original image patch space:

[0070]

[0071] Among them, g θ (z) is the neural network output of the decoder.

[0072] The reconstruction error measured by the cross entropy loss quantifies the difference between the reconstructed data and the original data:

[0073]

[0074] Among them, x is the input data, To reconstruct the data. In addition, the difference between the approximate posterior distribution q(z|x) and the prior distribution p(z) is measured by calculating the KL divergence:

[0075]

[0076] The total loss function L vocabulary Including reconstruction error and KL divergence:

[0077] L vocabulary =-(E q(z|x) [logp(x|z)]-D KL (q(z|x)||p(z))).

[0078] Finally, we can deduce for each patch x i The probability distribution q(z|x i ), which can be used for the pre-training process of mask image modeling.

[0079] The embedded vector after projection mapping in step 2 is input into the Transformer, which generates a probability distribution over the visual vocabulary for each mask position. The predicted probability of the visual token at position i of the mask input is given by . For a single mask token at position i, the cross entropy loss is defined as:

[0080]

[0081] Among them, t i is the true mark at position i, is the masked input sequence, is the probability predicted by the model given the masked input. For each mask position M in the input sequence, the total cross entropy loss of the model is the average of the losses at each mask position:

[0082]

[0083] Where M is the set of these masked positions. By calculating the average log probability of the masked positions, the model can effectively predict the visual markers at the masked positions.

[0084] Step 4: Extract deep features at the gaze point

[0085] The embedding representation obtained by mapping all image patches into linear space Special marker z indicating the result of fixation classification cls and position embeddings representing the order of image patches As the encoder input Z0 of Transfomer:

[0086]

[0087] The Transformer encoder is then used to capture the dependencies between different positions in the sequence and extract higher-level feature representations.

[0088] The core of Transformer encoder is the multi-head self-attention mechanism. Given the lth layer Z l-1 For the input, the head self-attention is calculated as follows:

[0089]

[0090] Among them, Q i =Z l-1 W Qi , K i =Z l-1 W Ki ,V i =Z l-1 W Vi .W Qi , W Ki and W Vi is a learnable weight matrix, and d k is a scaling factor for the attention mechanism. For multi-head attention, we calculate each head connection output according to the above equation and apply a linear transformation:

[0091] MultiHead(Z l )=Concat(h1,h2,...,h h )W O ,

[0092] in, After processing by the residual connection and layer normalization modules, the output is calculated as follows:

[0093] Z l '=LayerNorm(Z l +MultiHead(Zl )),

[0094] Finally, the output is passed through a feed-forward neural network (FFN), followed by another round of residual connections and layer normalization modules:

[0095] FFN(Z l ')=max(0,Z l 'W1+b1)W2+b2,

[0096] Z l =LayerNorm(Z l '+FFN(Z l ')).

[0097] For each fixation point q i , extract the feature f of the image block where the gaze point is located i .

[0098] Step 5: Classify based on image

[0099] All the gaze features {f1,f2,...,f i}Integrate image-level features through average pooling method The image-level features are then fed into the final classification layer, which maps the coupled input of image and eye-tracking data to a probability distribution y∈R over C categories. C superior:

[0100] y=Classifier(f image ).

[0101] In the classifier learning of each classification task, the model is optimized by minimizing the cross entropy loss between the probability distribution of the predicted category and the actual distribution. For the i-th sample, the output of the model is a C-dimensional probability distribution vector y i , indicating the probability that the sample belongs to each category. The true label is represented by a one-hot encoded vector y i true The total loss function is the average difference between the two probability distributions over N samples:

[0102]

[0103] in, represents the true label of the i-th sample in category j, y i,j It indicates the probability that the model predicts that the sample belongs to category j.

[0104] Step 6: Classify based on subjects

[0105] After obtaining the output probability distribution for each image in step 5, the last step is to comprehensively classify each subject based on all the image outputs of the subject. A strategy similar to model integration is adopted here to create a more robust classifier by combining the results of multiple weak classifiers. Specifically, for the kth subject, the average output vector of all the viewed images of the subject is calculated:

[0106]

[0107] Among them, N k represents the number of valid images for the kth subject. The final classification of the subject is given by the output vector The class corresponding to the maximum value in is determined.

[0108] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. An eye movement recognition method based on Transformer global semantic encoding, characterized in that: include: Collect corresponding eye movement data according to different task requirements and construct an eye movement recognition data set; the eye movement data set includes images and eye movement data corresponding to viewing the images; According to the gaze point position recorded in the eye movement data, the relevant images are masked at the gaze point, and the masked images and unmasked images are used as inputs of the Transformer model; Pre-train the Transformer model; Use the pre-trained Transformer model to extract deep features from all gaze positions in each image, and predict image-based class probabilities based on the deep features; The image-based category probabilities are integrated to obtain the comprehensive classification results of the subjects.

2. The eye movement recognition method based on Transformer global semantic encoding according to claim 1, characterized in that: According to different task requirements, corresponding eye movement data are collected to construct an eye movement recognition dataset, specifically: We collected our own datasets for the two tasks of ASD identification and gender identification. Specifically, we recruited subjects from different groups and recorded their eye movement data by having them watch a series of preset images. For the two tasks of task identification and identity recognition, two public datasets are directly selected.

3. The eye movement recognition method based on Transformer global semantic encoding according to claim 1, characterized in that: According to the gaze point position recorded in the eye movement data, the relevant image is masked at the gaze point, specifically: The image is divided into multiple image blocks, and the image block where the gaze point is located is masked.

4. The eye movement recognition method based on Transformer global semantic encoding according to claim 1, characterized in that: Pre-train the Transformer model, specifically: Before training begins, a pre-trained discrete variational autoencoder (DVAE) is used to process the entire training set, converting each image patch into a fixed-length visual label vector. During the training process, the Transformer model uses its self-attention mechanism to process the mixed information of masked and unmasked image patches, and predicts the content of the masked area by understanding the context of the image; the difference between the model's output and the corresponding visual marker is quantified through the cross-entropy loss, and then this loss is minimized to adjust its parameters and optimize the prediction ability.

5. The eye movement recognition method based on Transformer global semantic encoding according to claim 1, characterized in that: Predict image-based category probabilities based on deep features, specifically: All gaze point features extracted from the image are integrated through the average pooling method to obtain image-level features, which are then sent to the final classification layer. The coupled input of the image and eye movement data is mapped to a probability distribution to predict the probability that the image belongs to each category.

6. The eye movement recognition method based on Transformer global semantic encoding according to claim 1, characterized in that: The comprehensive classification results of the subjects are obtained by integrating the class probabilities based on the images, specifically: The class probabilities of all images for each subject are averaged to obtain a comprehensive feature vector representing the subject: Among them, N k represents the number of valid images of the kth subject; The final classification of the subject is given by the output vector The class corresponding to the maximum value in is determined.

7. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

8. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

9. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images