Pre-training method and system for eye movement emotion recognition model

Abstract

The embodiment of the invention provides a pre-training method and system for an eye movement emotion recognition model. The method comprises the following steps: acquiring an eye movement tracking heat map and an eye movement track, and determining an embedded heat map and an embedded track corresponding to the eye movement tracking heat map; inputting the embedded heat map and the embedded track into an eye movement emotion recognition model; obtaining heat map coded output and track coded output based on the first encoder and the second encoder, and obtaining a plurality of predicted eye movement tracking heat maps and predicted eye movement tracks; determining training loss by using the plurality of predicted eye movement tracking heat maps, the predicted eye movement trajectory, a prepared actual eye movement tracking heat map and an actual eye movement trajectory; and pre-training the eye movement emotion recognition model based on the training loss until a preset target is reached. According to the embodiment of the invention, synchronous pre-training is carried out on the eye movement thermodynamic diagram / track data, emotion recognition can be carried out through the eye movement thermodynamic diagram / track, and the limitation of emotion recognition based on a monocular movement mode is effectively overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to a pre-training method and system for an eye movement emotion recognition model. Background Art

[0002] Emotion recognition is an important topic in the field of human-computer interaction and artificial intelligence. By learning from physiological and non-physiological signals, machines can recognize and interpret complex human emotions. Due to the complexity and importance of emotions as a psychophysiological process triggered by multiple factors, researchers in psychology, neuroscience, and computer science have been actively exploring methods for emotion recognition. With the continuous development of technology, emotion recognition methods have become more and more sophisticated, utilizing various signal forms. Researchers use non-physiological signals such as voice, facial expressions, micro-expressions, body postures, eye movements, and gait to enable machines to effectively identify human emotion types. In addition, physiological signals such as electroencephalograms, electrocardiograms, and skin electrodermal activity have also been effectively processed to recognize human emotions.

[0003] In the prior art, eye movements have been shown to be closely related to human emotions. The subject's eyeballs show involuntary movements in response to the stimulation of emotional words. Since various physiological signals with higher accuracy can be collected and applied to emotion recognition, eye movement signals are often used as an auxiliary modality in emotion recognition tasks. In different studies and data sets, eye movement signals are often collected together with EEG (electroencephalogram) and ECG signals. In subsequent studies, eye movement signals were used as an important modality in multimodal emotion recognition algorithms to improve the accuracy of emotion recognition. Eye movement signals have obvious advantages and disadvantages in emotion recognition. Although the accuracy of eye tracking systems is generally lower than that of EEG and facial expression recognition, unlike EEG and ECG signals that require wearable devices, eye tracking systems benefit from non-contact acquisition. In addition, the quality of eye movement data is less affected by head and body movements, which have the greatest impact on bioelectric signals. In addition, compared with traditional facial expression recognition, eye movement signals have the advantages of other physiological signals in better reflecting real human inner emotions. Specifically:

[0004] The method is based on the classification of pupil diameter: pupil diameter changes under different emotional states. In most experiments, the pupil diameter is largest when sad and smallest when calm. The pupil diameter of positive and negative emotions is larger than that of neutral emotions. This is consistent with previous psychological literature. This result shows that there is a great relationship between pupil diameter and emotion. This method can extract emotion-related features from pupil diameter to classify different emotions.

[0005] Emotion classification based on eye movement statistics: This method is trained with six types of eye movement signal features: pupil diameter features after short-time Fourier transform, gaze deviation, saccade time and amplitude, blink time, fixation time, and event features. At the same time, the PSD feature of pupil diameter and other eye movement signal features are combined into a total feature for classification, thus obtaining a series of classification accuracy rates.

[0006] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:

[0007] Although statistics of pupil diameter and eye movement events are effective in emotion recognition, these data fail to reflect some additional information in the raw eye movement data, such as: the distribution of eye fixation areas, changes in eye movement positions over time, etc. In general, the performance of eye movement signals in emotion recognition tasks is inherently limited because they can hardly surpass higher precision signals such as EEG. However, in terms of monocular movement modalities, current methods for emotion recognition using eye movement signals have obvious shortcomings. These methods mainly rely on manually extracted statistical measurements related to eye movements. Although these measurements are combined and filtered to classify relevant emotions, they ignore important information contained in the raw eye movement data, such as visual attention location and temporal order information. Summary of the invention

[0008] In order to at least solve the problem that the performance of eye movement signals in emotion recognition tasks in the prior art is limited, important information contained in the original eye movement data is ignored.

[0009] In a first aspect, an embodiment of the present invention provides a pre-training method for an eye movement emotion recognition model, comprising:

[0010] Obtain an eye tracking heat map and an eye movement trajectory, and determine an embedded heat map corresponding to the eye tracking heat map and an embedded trajectory in a high-dimensional space corresponding to the eye movement trajectory;

[0011] Input the embedded heat map and the embedded trajectory into an eye movement emotion recognition model, wherein the eye movement emotion recognition model includes: a first encoder for heat map encoding and a second encoder for trajectory encoding based on Transformer multi-head self-attention, respectively, a first decoder and a second decoder with corresponding structures, an embedding conversion module for heat map and trajectory structure conversion, and a multi-channel decoder for receiving processing by the embedding conversion module;

[0012] Based on the heat map encoding output and the trajectory encoding output obtained by the first encoder and the second encoder, the heat map encoding output and the trajectory encoding output are processed by the first decoder, the second decoder and the multi-channel decoder processed by the embedded conversion module to obtain a plurality of predicted eye tracking heat maps and predicted eye tracking trajectories;

[0013] Determine a training loss using the plurality of predicted eye tracking heat maps and predicted eye movement trajectories and the prepared actual eye tracking heat maps and actual eye movement trajectories;

[0014] The eye movement emotion recognition model is pre-trained based on the training loss until a preset goal is achieved, wherein the pre-training includes unidirectional pre-training, multidirectional pre-training and cross pre-training.

[0015] In a second aspect, an embodiment of the present invention provides a pre-training system for an eye movement emotion recognition model, comprising:

[0016] An embedding determination module, used to obtain an eye tracking heat map and an eye movement trajectory, and determine an embedded heat map corresponding to the eye tracking heat map, and an embedded trajectory in a high-dimensional space corresponding to the eye movement trajectory;

[0017] An input module, used for inputting the embedded heat map and the embedded trajectory into the eye movement emotion recognition model, wherein the eye movement emotion recognition model comprises: a first encoder for heat map encoding and a second encoder for trajectory encoding based on Transformer multi-head self-attention, respectively, a first decoder and a second decoder with corresponding structures, an embedding conversion module for heat map and trajectory structure conversion, and a multi-channel decoder for receiving processing by the embedding conversion module;

[0018] A prediction module, configured to obtain a heat map encoding output and a trajectory encoding output based on the first encoder and the second encoder, and process the heat map encoding output and the trajectory encoding output using the first decoder, the second decoder, and the multi-channel decoder processed by the embedded conversion module to obtain a plurality of predicted eye tracking heat maps and predicted eye tracking trajectories;

[0019] A loss determination module, configured to determine a training loss using the plurality of predicted eye tracking heat maps and predicted eye movement trajectories and the prepared actual eye tracking heat maps and actual eye movement trajectories;

[0020] A pre-training module is used to pre-train the eye movement emotion recognition model based on the training loss until a preset goal is achieved, wherein the pre-training includes unidirectional pre-training, multidirectional pre-training and cross pre-training.

[0021] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the pre-training method of the eye movement emotion recognition model of any embodiment of the present invention.

[0022] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the pre-training method of the eye movement emotion recognition model of any embodiment of the present invention are implemented.

[0023] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the steps of the pre-training method of the eye movement emotion recognition model of any embodiment of the present invention are implemented.

[0024] The beneficial effect of the embodiment of the present invention is that the method develops a multi-channel pre-training method for eye movement heat map / trajectory: the eye movement heat map / trajectory data is pre-trained synchronously, and the obtained model can realize emotion recognition through eye movement heat map / trajectory after fine-tuning. This enables the method to effectively overcome the limitations of emotion recognition based on monocular movement modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 is a flow chart of a pre-training method for an eye movement emotion recognition model provided by an embodiment of the present invention;

[0027] Figure 2 It is a schematic diagram of the process of data processing of eye movement trajectories of a pre-training method of an eye movement emotion recognition model provided by one embodiment of the present invention;

[0028] Figure 3 It is a schematic diagram of a pre-training model structure of a pre-training method for an eye movement emotion recognition model provided by an embodiment of the present invention;

[0029] Figure 4 It is a schematic diagram of embedding conversion of a pre-training method of an eye movement emotion recognition model provided by an embodiment of the present invention;

[0030] Figure 5 It is a schematic diagram of multi-way cross pre-training of a pre-training method of an eye movement emotion recognition model provided by one embodiment of the present invention;

[0031] Figure 6 It is a schematic diagram of a classifier structure of monocular modality and multi-viewpoint fusion of a pre-training method of an eye movement emotion recognition model provided by an embodiment of the present invention;

[0032] Figure 7 It is a schematic diagram of a confusion matrix of different methods of a pre-training method of an eye movement emotion recognition model and an eye movement pattern provided by an embodiment of the present invention;

[0033] Figure 8 The performance (accuracy and standard deviation, %) of different methods and eye patterns of a pre-training method of an eye movement emotion recognition model provided by an embodiment of the present invention;

[0034] Fig. 9 The performance (accuracy and standard deviation, %) of a fusion model with ERM vector added to a pre-training method for an eye movement emotion recognition model provided by an embodiment of the present invention;

[0035] Fig.10 It is a structural schematic diagram of a pre-training system for an eye movement emotion recognition model provided by an embodiment of the present invention;

[0036] Fig.11 A schematic diagram of the structure of an embodiment of an electronic device for pre-training an eye movement emotion recognition model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] like Figure 1 FIG. 1 is a flow chart of a pre-training method for an eye movement emotion recognition model provided by an embodiment of the present invention, comprising the following steps:

[0039] S11: Obtain an eye tracking heat map and an eye movement trajectory, and determine an embedded heat map corresponding to the eye tracking heat map and an embedded trajectory in a high-dimensional space corresponding to the eye movement trajectory;

[0040] S12: Input the embedded heat map and the embedded trajectory into an eye movement emotion recognition model, wherein the eye movement emotion recognition model includes: a first encoder for heat map encoding and a second encoder for trajectory encoding based on Transformer multi-head self-attention, respectively, a first decoder and a second decoder with corresponding structures, an embedding conversion module for heat map and trajectory structure conversion, and a multi-channel decoder for receiving processing by the embedding conversion module;

[0041] S13: Based on the heat map encoding output and the trajectory encoding output obtained by the first encoder and the second encoder, the heat map encoding output and the trajectory encoding output are processed by the first decoder, the second decoder and the multi-channel decoder processed by the embedded conversion module to obtain a plurality of predicted eye tracking heat maps and predicted eye tracking trajectories;

[0042] S14: Determine a training loss using the multiple predicted eye tracking heat maps and predicted eye movement trajectories and the prepared actual eye tracking heat maps and actual eye movement trajectories;

[0043] S15: Pre-training the eye movement emotion recognition model based on the training loss until a preset goal is reached, wherein the pre-training includes unidirectional pre-training, multidirectional pre-training and cross pre-training.

[0044] In this embodiment, the method develops a model based on Transformer and Vision Transformer and pre-trains it on large-scale unlabeled eye movement data. Through the multi-path prediction task between the two types of eye movement data, the pre-training method is used to enhance the applicability of the emotion recognition model based on heatmaps and trajectories. The method effectively improves the accuracy of emotion recognition using multi-view eye movement data. The method further combines the method with eye-related measurement features to further improve the accuracy of emotion recognition over a longer period of time through single eye movement patterns.

[0045] For step S11, regarding the dataset of eye tracking heat maps and eye movement trajectories obtained by this method, this method uses the SEED-VII dataset of the Emotional Electroencephalogram Dataset (SEED). The SEED-VII dataset is designed to simultaneously record EEG (electroencephalogram) and eye movement signals while inducing seven emotions (happiness, surprise, neutrality, sadness, disgust, fear, and anger). The dataset contains data from 20 subjects (10 males and 10 females). Each subject performed four 20 trials at different times. Each trial lasted between 2 and 5 minutes, and the total duration of all clips was approximately 14097.86 seconds. The SEED-VII dataset is the first dataset in the SEED series to contain raw eye movement data for the entire duration (due to equipment limitations, previous datasets only recorded the start and end points of eye movement events).

[0046] For eye tracking heatmap, it visualizes user attention by tracking and recording the location and duration of eye gaze on a page or screen. In visual analysis, eye tracking heatmaps usually use different colors to represent the frequency and duration of user gaze. However, for data analysis, color information is useless. Therefore, this method removes the color and only retains the numerical value of each pixel to indicate the degree of attention in a given area. The heatmap used for analysis is essentially a two-dimensional matrix. The calculation process is as follows:

[0047] For any subject looking at a computer screen during a period of time, the fixation point for each time sample is recorded as (x, y), where x∈[0,w] and y∈[0,h], w and h represent the width and height resolution of the screen, respectively. All fixations from that period of time are compiled into a list and converted into a 2D histogram. A 2D Gaussian filter is used to generate the values ​​of the heat map matrix from this 2D histogram data. The 2D Gaussian is processed as follows:

[0048]

[0049] Where x and y are the values ​​in the two-dimensional histogram. x and σ y denote the standard deviation along the x and y axes, respectively. To ensure uniform scaling, this method defines σ as the product of the screen width and height resolution multiplied by a factor α, so σ x =αw,σ y = αh. This method sets α to 0.05 and sets the dimension of the 2D histogram to H × W. The Gaussian filtered heat map is then treated as a single-channel image.

[0050] For the eye tracking, it shows the real-time line of sight of the subject on the screen. Each gaze point on the track is represented by a two-dimensional coordinate (x, y). Flickering can cause interruptions in the eye tracking data, so this method proposes a two-dimensional linear gap filling interpolation to process the track data. In addition, the original eye tracking is also affected by noise. To alleviate this situation, a denoising process is used in two steps: (1) a moving average filter with a window of 0.02 seconds is applied to all data; (2) the denoised data is uniformly sampled at a frequency of 10 Hz, as shown in the figure. Figure 2 As shown in the figure, the final eye movement trajectory data size is 2×t, where t is the time step at the new sampling rate.

[0051] It is standard practice to use 4-second non-overlapping eye movement statistics windows in emotion recognition tasks. This method aligns the eye movement data with the EEG data, which also uses a 4-second window. Although suitable for high-frequency EEG signals, this method believes that the 4-second window is too short to capture stable eye movement data. This method compares the classification results of eye movement data features using a 4-second window and an overlapping 20-second window with a 4-second step size. Compared with the 4-second window, the use of a 20-second window significantly improves the accuracy of emotion recognition. Therefore, this method uses an overlapping 20-second window with a 4-second step size as the baseline for all subsequent data analysis. The above steps enable this method to calculate eye tracking heat maps and eye movement trajectories from the raw eye movement data. At the same time, a baseline for training (that is, the actual eye tracking heat map and the actual eye movement trajectory) is also prepared. After the eye tracking heat map and eye movement trajectory are obtained, the corresponding embedded heat map and embedded trajectory are determined.

[0052] Regarding the determination of embedded heatmaps and embedded trajectories, eye tracking heatmaps and eye trajectories are processed by separate embedding modules. For heatmap data, each heatmap matrix obtained by Gaussian filtering is regarded as a single-channel image of dimension [H, W, C], where C = 1. Each heatmap is divided into blocks of size (P, P), resulting in a size of N h ×P 2 A matrix where N h =HW / P 2 The position embedding of each patch is then combined with the patch embedding vector obtained by linear projection to obtain the embedded patch output. This process is expressed as follows:

[0053]

[0054] Regarding trajectory data, the embedded trajectory data is represented as The size of the original eye movement trajectory data is [t, D], where t represents the number of trajectory points and D represents the dimension of each trajectory point, D=2.

[0055] Use linear projection with a temporal embedding layer to convert fixed-length eye tracking data into a high-dimensional representation The encoder maps the input sequence with coordinates in the original space to this high-dimensional space as follows:

[0056]

[0057] Get the embedded heat map E h and the embedded trajectory E t .

[0058] For step S12, the embedded heat map E h and the embedded trajectory E t Input to the eye movement emotion recognition model. For the eye movement emotion recognition task, this method uses a Transformer-based multi-way cross pre-training model as the eye movement emotion recognition model. The model uses the source heat map and trajectory data to perform autoregressive prediction pre-training tasks on the future target heat map and trajectory data. In this process, it learns the relationship between multi-viewpoint eye movement data. Figure 3 As shown in the figure, the structure of the eye movement emotion recognition model includes independent Transformer encoders for heat map encoding and trajectory encoding based on Transformer multi-head self-attention, Transformer decoders with corresponding structures, an embedding conversion module for heat map and trajectory structure conversion, and a multi-channel decoder for receiving processing from the embedding conversion module;

[0059] For step S13, the embedded heat map E h and the embedded trajectory E t In the eye movement emotion recognition model, each independent Transformer encoder is used, where the core component of each Transformer encoder is MHSA (multi-head self-attention). h and the embedded trajectory E t In the encoder, it is transformed into a query Q through three linear layers i , key K i Sum value V i The self-attention is calculated as follows:

[0060]

[0061] Each head of MHSA can be represented by H i =Attention(Q i , K i , V i). The output of multi-head attention is represented by X = Concat(H1, H2, ..., H h )W is given, where W represents the weight. The heat map encoding X of the encoded output is obtained h And trajectory code X t ;

[0062] As an implementation mode, the heat map encoding output and the trajectory encoding output are processed by the first decoder, the second decoder and the multi-channel decoder processed by the embedded conversion module to obtain multiple predicted eye tracking heat maps and predicted eye movement trajectories, including:

[0063] Determine the embedded heat map E using the first encoder h Heatmap encoding of X h ;

[0064] Determine the embedded trajectory E using the second encoder t The trajectory code X t ;

[0065] The heat map is encoded based on the first decoder and the multi-channel decoder cooperated by the embedded conversion module h Decoding is performed to obtain a first predicted eye tracking heat map and a first predicted eye movement trajectory;

[0066] The track encoding X is performed based on the second decoder and the multi-channel decoder cooperated by the embedded conversion module t Decoding is performed to obtain a second predicted eye tracking heat map and a second predicted eye movement trajectory.

[0067] In this embodiment, during the pre-training process, the encoding output X h and X t They are used to predict the heat map data and trajectory data for the next 4 seconds. For example, when predicting the target heat map, the encoding output X is directly sent to the decoder of the same structure to obtain the prediction result Specifically, the encoding is performed by the Transformer encoder in the eye movement emotion recognition model. Specifically, the embedded heat map E is determined by using the first encoder in the upper half of the figure. h Heatmap encoding of X h , the embedded trajectory E is determined by using the second encoder in the lower half of the figure t The trajectory code X t .

[0068] In the eye tracking heat map training process, the first decoder in the upper part is used for decoding to obtain the first predicted eye tracking heat map for predicting the future; on the other hand, due to the embedded conversion module, it is converted to the eye trajectory training line. At this time, the multi-channel decoder receives the conversion result from the embedded conversion module for decoding, and obtains the first predicted eye trajectory for predicting the future. Similarly, in the eye trajectory training process. Similar to the eye tracking heat map processing process, the second predicted eye tracking heat map and the second predicted eye trajectory are obtained in the same way, which will not be repeated here.

[0069] For step S14, the method uses the Euclidean distance to calculate the loss between the predicted eye tracking heat map and the actual eye tracking heat map prepared 4 seconds later. Specifically, determining the training loss includes:

[0070] Determine a first heat map loss L based on the error between the prepared actual eye tracking heat map and the actual eye movement trajectory and the first predicted eye tracking heat map and the first predicted eye movement trajectory. h_h And the first trajectory loss L h_t ;

[0071] Determine a second heat map loss L based on the error between the prepared actual eye tracking heat map and the actual eye movement trajectory and the second predicted eye tracking heat map and the second predicted eye movement trajectory. t_h And the second trajectory loss L t_t .

[0072] In this embodiment, Represents the actual eye movement heat map prepared, and uses the Euclidean distance to calculate the loss between the predicted eye tracking heat map and the actual eye movement heat map prepared 4 seconds later:

[0073]

[0074] Regarding the embedding conversion module, since the output of the encoded heat map and the output of the encoded trajectory have different structures, this method uses another MHSA module to convert E h Converted to a vector of the same length as the target 4-second encoded trajectory and add the temporal embedding. Embedded heatmap output E h and embedded trace output E t All have fixed lengths. This method discards E h The second half of Et (e.g., half of the length, which can also be discarded from back to front based on the length of the trajectory) is used to match the length of Et. This result is then fed into a linear projection module to align the embedding dimension with the trajectory, as Figure 4 The output is passed through the trajectory converter encoder to obtain the predicted embedded trajectory. represents the actual eye movement trajectory of the preparation, and the loss is expressed as:

[0075]

[0076] Similarly, for the source eye movement trajectory input, the eye movement emotion recognition model predicts the second predicted eye movement tracking heat map and the second predicted eye movement trajectory within the next 4 seconds, which is consistent with the above L h_h and L h_t The calculation method is the same as that of the second heat map loss and the second trajectory loss. t_h and L t_t , I will not go into details here.

[0077] For step S15, after obtaining the training loss, use L h_h , L h_t , L t_h and L t_t , the eye movement emotion recognition model is pre-trained. Specifically, the pre-training includes unidirectional pre-training, multi-directional pre-training and cross pre-training.

[0078] Specifically, during one-way pre-training, the method includes:

[0079] Only using the first heatmap loss L h_h Pre-training a first encoder for heat map encoding;

[0080] Only using the second trajectory loss L t_t Pre-train the second encoder for trajectory encoding.

[0081] During multi-directional pre-training, the method includes:

[0082] Based on the first heat map loss L h_h And the first trajectory loss L h_t Pre-training a first encoder for heat map encoding and a multi-channel decoder for heat map encoding using the determined pre-trained heat map loss;

[0083] Based on the second heatmap loss L t_h And the second trajectory loss L t_t The determined pre-trained trajectory loss is used to pre-train a second encoder for trajectory encoding and a multi-channel decoder for trajectory encoding, wherein the pre-training of the first encoder, the second encoder and the multi-channel decoder is synchronized.

[0084] During cross pre-training, the method includes:

[0085] Based on the first heat map loss L h_h , the first trajectory loss L h_t , the second heat map loss L t_hAnd the second trajectory loss L t_t The determined pre-trained cross loss is used to pre-train the first encoder, the second encoder, and the multi-path decoder.

[0086] In this embodiment, the method uses three pre-training methods, namely, unidirectional, multidirectional and cross-training. In order to facilitate a clear comparison, unidirectional pre-training is first introduced as a baseline.

[0087] (1) One-way: For eye tracking heatmaps, the pre-training task only involves predicting the target heatmap, which is similar to the eye trajectory map. It only predicts the target trajectory. This is the conventional training method in the prior art and will not be described in detail here.

[0088] (2) Multi-channel: Multi-channel pre-training tasks simultaneously predict target heatmaps and target trajectories (e.g. Figure 5 The pre-training loss is defined as L = L h_h +L h_t For the trajectory, the loss is L = L t_h +L t_t In multi-channel pre-training, the source heat map data and trajectory data are pre-trained separately.

[0089] (3) Cross-method: In multi-channel pre-training, two independent decoders with the same structure are used: one for source heatmaps to predict future heatmaps, and the other for source trajectories to predict future trajectories. These decoders run independently during training (e.g. Figure 5 In contrast, in cross pre-training, a single heatmap decoder and a single trajectory decoder are trained simultaneously on the input and The relevant loss is defined as L = L h_h +L h_t +L t_h +L t_t .

[0090] The embedded heatmaps and trajectories are processed through a pre-trained encoder to obtain high-dimensional vectors. For the heatmap part, this method uses its class label to extract the heatmap encoding output X h For the trajectory module, use X t In the one-way pre-training stage, this method uses a linear classifier to predict the target heat map and trajectory feature vector (such as Figure 6 Left half). During the multi-path and cross-path pre-training phase, our method uses a concat layer on the encoded output of the heatmap and trajectory data. The concatenated vector is passed through a linear layer for the emotion recognition task (e.g. Figure 6 The eye movement emotion recognition model is pre-trained using the above loss until the preset goal is achieved.

[0091] It can be seen from this implementation that the present method has developed a multi-channel pre-training method for eye movement heatmaps / trajectories: the eye movement heatmap / trajectory data is pre-trained synchronously, and the obtained model can realize emotion recognition through eye movement heatmaps / trajectories after fine-tuning. This enables the present method to effectively overcome the limitations of emotion recognition based on monocular movement modality.

[0092] The method is experimentally illustrated. Regarding the heatmap results, the method first analyzes the case where only eye tracking heatmaps are used. In the results of the method, three traditional computer vision models with randomly initialized parameters: VGG-16, ResNet-18, and ViT are used as baseline models. Then, unidirectional pre-trained ViT and multi-directional pre-trained ViT are tested.

[0093] For the classification of seven emotions, ViT performs best among non-pretrained models. Among the pre-trained ViT models, the multi-way pre-trained model has the highest accuracy of 38.46%. This shows that the accuracy of the model is improved by using unlabeled heatmap data for self-prediction. In addition, using multi-way pre-training, the heatmap predicts future heatmaps and trajectories simultaneously, which effectively improves the emotion classification performance.

[0094] Regarding the trajectory results, for the eye tracking trajectory results, this method uses three time series prediction models with randomly initialized parameters: RNN, LSTM, and Transformer as baseline models. Then, this method tests the unidirectional pre-trained Transformer and the multi-directional pre-trained Transformer on the trajectory data. For the classification of seven emotions using only eye tracking, the Transformer model performs best. Similarly, the accuracy of multi-way pre-training is the highest among all methods, reaching 27.88%. Overall, the accuracy of emotion recognition using only eye tracking is significantly lower than that using heatmaps alone. Regarding fusion, this method compares methods using only heatmaps, trajectories, and fusion. Among the methods using only heatmaps or trajectories, multi-way pre-training achieves the highest accuracy. For pre-training methods using both heatmaps and trajectories, this method tries three methods: unidirectional, multi-directional, and cross pre-training, all of which use the same classifier structure for emotion classification. Among them, multi-way fusion exceeds the results of the other two pre-training methods, and also exceeds the results of using heatmaps or trajectories alone, achieving the highest accuracy of 47.39%.

[0095] Comparing the performance of using heatmaps alone and trajectories alone, trajectories have lower recognition performance for four emotions (happy, surprised, neutral, and disgust), but higher recognition performance for fear than heatmaps. In the results of the multi-way pre-trained fusion model, the recognition performance of all seven categories of emotions is higher than that of using heatmaps or trajectories alone (such as Figure 7This finding suggests that using cross pre-training does not further improve model performance. And its accuracy is actually lower than using heatmaps alone. This result suggests that merging the decoder in the model and training on both types of eye movement data simultaneously does not make the encoder's latent representation perform better in emotion recognition.

[0096] This method further introduces the eye-related measurement (ERM) vector and proposes a simple decision-level fusion method to integrate the fusion result obtained in the previous step with 20 seconds of ERM data. Previous studies on emotion recognition using eye movement signals have mainly focused on statistical ERM vectors, which include physiological information that is crucial for emotion recognition, such as pupil diameter and blinking, and thus achieved good classification results. However, the limited statistical features of eye tracking limit the ability to extract additional emotional information from eye movements. To address this limitation, this method utilizes high-dimensional features extracted from heat maps and trajectories and combines them with ERM vectors to improve the performance of emotion recognition based on eye movement data.

[0097] The emotion classification results of using ERM alone are slightly higher than those of heatmap and trajectory multi-way fusion (e.g. Figure 8 as well as Fig. 9 As shown). Figure 7 It can be seen that ERM performs better in recognizing fear, while heatmap and trajectory perform better in recognizing surprise. Both modes perform poorly in recognizing disgust. The integration of heatmap and trajectory fusion results can improve the overall emotion recognition performance. This method uses a simple decision-level fusion technique to combine the results of the heatmap and trajectory fusion model with the ERM vector. The analysis results of the confusion matrix show that by combining the ERM model with the multi-way pre-trained model, the recognition rate of all emotions except disgust is improved.

[0098] In summary, this method develops a multi-way autoregressive model based on multi-view eye movement data to process eye tracking heatmaps and eye trajectory data and apply them to emotion recognition. This method improves the limitations of traditional research, which only uses statistical measures related to eye movements for emotion recognition. In this method, either heatmaps or trajectories can simultaneously achieve joint predictions of future heatmaps and trajectories, surpassing the accuracy of non-pretrained models and single-modal pre-training methods. In addition, this method combines eye-related measurements with heatmaps and trajectories of the same window. The obtained emotion recognition accuracy exceeds the performance of each independent part of the eye movement data. This enables this method to effectively overcome the limitations of emotion recognition based on single eye movement modality.

[0099] like Fig.10The figure is a schematic diagram of the structure of a pre-training system for an eye movement emotion recognition model provided by an embodiment of the present invention. The system can execute the pre-training method for an eye movement emotion recognition model described in any of the above embodiments and be configured in a terminal.

[0100] The present embodiment provides a pre-training system 10 for an eye movement emotion recognition model, comprising: an embedding determination module 11, an input module 12, a prediction module 13, a loss determination module 14 and a pre-training module 15.

[0101] Among them, the embedding determination module 11 is used to obtain the eye tracking heat map and the eye movement trajectory, determine the embedded heat map corresponding to the eye tracking heat map, and the embedded trajectory in the high-dimensional space corresponding to the eye movement trajectory; the input module 12 is used to input the embedded heat map and the embedded trajectory into the eye movement emotion recognition model, wherein the eye movement emotion recognition model includes: a first encoder for heat map encoding and a second encoder for trajectory encoding based on Transformer multi-head self-attention, each independently, a first decoder and a second decoder corresponding to the structure, an embedding conversion module for heat map and trajectory structure conversion, and a multi-channel decoder for receiving processing from the embedding conversion module; the prediction module 13 is used to The first encoder and the second encoder obtain heat map encoding output and trajectory encoding output, and the heat map encoding output and trajectory encoding output are processed by the first decoder, the second decoder and the multi-channel decoder processed by the embedded conversion module to obtain multiple predicted eye tracking heat maps and predicted eye movement trajectories; the loss determination module 14 is used to determine the training loss using the multiple predicted eye tracking heat maps and predicted eye movement trajectories and the prepared actual eye tracking heat maps and actual eye movement trajectories; the pre-training module 15 is used to pre-train the eye movement emotion recognition model based on the training loss until a preset goal is reached, wherein the pre-training includes unidirectional pre-training, multi-directional pre-training and cross pre-training.

[0102] The embodiment of the present invention further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the pre-training method of the eye movement emotion recognition model in any of the above method embodiments;

[0103] As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows:

[0104] Obtain an eye tracking heat map and an eye movement trajectory, and determine an embedded heat map corresponding to the eye tracking heat map and an embedded trajectory in a high-dimensional space corresponding to the eye movement trajectory;

[0105] Input the embedded heat map and the embedded trajectory into an eye movement emotion recognition model, wherein the eye movement emotion recognition model includes: a first encoder for heat map encoding and a second encoder for trajectory encoding based on Transformer multi-head self-attention, respectively, a first decoder and a second decoder with corresponding structures, an embedding conversion module for heat map and trajectory structure conversion, and a multi-channel decoder for receiving processing by the embedding conversion module;

[0106] Based on the heat map encoding output and the trajectory encoding output obtained by the first encoder and the second encoder, the heat map encoding output and the trajectory encoding output are processed by the first decoder, the second decoder and the multi-channel decoder processed by the embedded conversion module to obtain a plurality of predicted eye tracking heat maps and predicted eye tracking trajectories;

[0107] Determine a training loss using the plurality of predicted eye tracking heat maps and predicted eye movement trajectories and the prepared actual eye tracking heat maps and actual eye movement trajectories;

[0108] The eye movement emotion recognition model is pre-trained based on the training loss until a preset goal is achieved, wherein the pre-training includes unidirectional pre-training, multidirectional pre-training and cross pre-training.

[0109] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by the processor, the pre-training method of the eye movement emotion recognition model in any of the above method embodiments is executed.

[0110] Fig.11 FIG. 1 is a schematic diagram of the hardware structure of an electronic device for a pre-training method of an eye movement emotion recognition model provided by another embodiment of the present application, such as Fig.11 As shown, the device includes:

[0111] One or more processors 1110 and memory 1120, Fig.11 A processor 1110 is taken as an example. The device of the pre-training method of the eye movement emotion recognition model may also include: an input device 1130 and an output device 1140.

[0112] The processor 1110, the memory 1120, the input device 1130 and the output device 1140 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0113] The memory 1120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the pre-training method of the eye movement emotion recognition model in the embodiment of the present application. The processor 1110 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 1120, that is, the pre-training method of the eye movement emotion recognition model in the above method embodiment is implemented.

[0114] The memory 1120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data, etc. In addition, the memory 1120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1120 may optionally include a memory remotely arranged relative to the processor 1110, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The input device 1130 can receive input digital or character information. The output device 1140 can include a display device such as a display screen.

[0116] The one or more modules are stored in the memory 1120, and when executed by the one or more processors 1110, the pre-training method of the eye movement emotion recognition model in any of the above method embodiments is executed.

[0117] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0118] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0119] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the pre-training method of the eye movement emotion recognition model of any embodiment of the present invention.

[0120] The electronic device of the embodiment of the present application exists in various forms, including but not limited to:

[0121] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0122] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDAs, MIDs, and UMPC devices, such as tablet computers.

[0123] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0124] (4) Other electronic devices with data processing functions.

[0125] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0126] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pre-training method for an eye movement emotion recognition model, comprising: Obtain an eye tracking heat map and an eye movement trajectory, and determine an embedded heat map corresponding to the eye tracking heat map and an embedded trajectory in a high-dimensional space corresponding to the eye movement trajectory; Input the embedded heat map and the embedded trajectory into an eye movement emotion recognition model, wherein the eye movement emotion recognition model includes: a first encoder for heat map encoding and a second encoder for trajectory encoding based on Transformer multi-head self-attention, respectively, a first decoder and a second decoder with corresponding structures, an embedding conversion module for heat map and trajectory structure conversion, and a multi-channel decoder for receiving processing by the embedding conversion module; Based on the heat map encoding output and the trajectory encoding output obtained by the first encoder and the second encoder, the heat map encoding output and the trajectory encoding output are processed by the first decoder, the second decoder and the multi-channel decoder processed by the embedded conversion module to obtain a plurality of predicted eye tracking heat maps and predicted eye tracking trajectories; Determine a training loss using the plurality of predicted eye tracking heat maps and predicted eye movement trajectories and the prepared actual eye tracking heat maps and actual eye movement trajectories; The eye movement emotion recognition model is pre-trained based on the training loss until a preset goal is achieved, wherein the pre-training includes unidirectional pre-training, multidirectional pre-training and cross pre-training.

2. The method according to claim 1, wherein: The step of obtaining a heat map encoding output and a trajectory encoding output based on the first encoder and the second encoder, and processing the heat map encoding output and the trajectory encoding output using the first decoder, the second decoder, and the multi-channel decoder processed by the embedded conversion module to obtain a plurality of predicted eye tracking heat maps and predicted eye tracking trajectories includes: Determine the embedded heat map E using the first encoder h Heatmap encoding of X h ; Determine the embedded trajectory E using the second encoder t The trajectory code X t ; The heat map is encoded based on the first decoder and the multi-channel decoder cooperated by the embedded conversion module h Decoding is performed to obtain a first predicted eye tracking heat map and a first predicted eye movement trajectory; The track encoding X is performed based on the second decoder and the multi-channel decoder cooperated by the embedded conversion module t Decoding is performed to obtain a second predicted eye tracking heat map and a second predicted eye movement trajectory.

3. The method according to claim 2, wherein: Determining the training loss using the plurality of predicted eye tracking heat maps and predicted eye movement trajectories and the prepared actual eye tracking heat maps and actual eye movement trajectories includes: Determine a first heat map loss L based on the error between the prepared actual eye tracking heat map and the actual eye movement trajectory and the first predicted eye tracking heat map and the first predicted eye movement trajectory. h_h And the first trajectory loss L h_t ; Determine a second heat map loss L based on the error between the prepared actual eye tracking heat map and the actual eye movement trajectory and the second predicted eye tracking heat map and the second predicted eye movement trajectory. t_h And the second trajectory loss L t_t .

4. The method according to claim 3, wherein: When multi-directional pre-training is performed on the eye movement emotion recognition model based on the training loss, the method includes: Based on the first heat map loss L h_h And the first trajectory loss L h_t Pre-training a first encoder for heat map encoding and a multi-channel decoder for heat map encoding using the determined pre-trained heat map loss; Based on the second heatmap loss L t_h And the second trajectory loss L t_t The determined pre-trained trajectory loss is used to pre-train a second encoder for trajectory encoding and a multi-channel decoder for trajectory encoding, wherein the pre-training of the first encoder, the second encoder and the multi-channel decoder is synchronized.

5. The method according to claim 3, wherein: When cross pre-training the eye movement emotion recognition model based on the training loss, the method includes: Based on the first heat map loss L h_h , the first trajectory loss L h_t , the second heat map loss L t_h And the second trajectory loss L t_t The determined pre-trained cross loss is used to pre-train the first encoder, the second encoder, and the multi-path decoder.

6. The method according to claim 3, wherein: When the eye movement emotion recognition model is pre-trained in one direction based on the training loss, the method includes: Only using the first heatmap loss L h_h Pre-training a first encoder for heat map encoding; Only using the second trajectory loss L t_t Pre-train the second encoder for trajectory encoding.

7. The method according to claim 2, wherein: The method further comprises: Discard the embedded heatmap E h The second half of the projected image is processed and the processed embedded heat map E h With the embedded trajectory E t Embedding dimension alignment.

8. A pre-training system for an eye movement emotion recognition model, comprising: An embedding determination module, used to obtain an eye tracking heat map and an eye movement trajectory, and determine an embedded heat map corresponding to the eye tracking heat map, and an embedded trajectory in a high-dimensional space corresponding to the eye movement trajectory; An input module, used for inputting the embedded heat map and the embedded trajectory into the eye movement emotion recognition model, wherein the eye movement emotion recognition model comprises: a first encoder for heat map encoding and a second encoder for trajectory encoding based on Transformer multi-head self-attention, respectively, a first decoder and a second decoder with corresponding structures, an embedding conversion module for heat map and trajectory structure conversion, and a multi-channel decoder for receiving processing by the embedding conversion module; A prediction module, configured to obtain a heat map encoding output and a trajectory encoding output based on the first encoder and the second encoder, and process the heat map encoding output and the trajectory encoding output using the first decoder, the second decoder, and the multi-channel decoder processed by the embedded conversion module to obtain a plurality of predicted eye tracking heat maps and predicted eye tracking trajectories; A loss determination module, configured to determine a training loss using the plurality of predicted eye tracking heat maps and predicted eye movement trajectories and the prepared actual eye tracking heat maps and actual eye movement trajectories; A pre-training module is used to pre-train the eye movement emotion recognition model based on the training loss until a preset goal is achieved, wherein the pre-training includes unidirectional pre-training, multidirectional pre-training and cross pre-training.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 7.

11. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images