Emotion recognition method and device based on self-supervision and multi-modal comparison and medium

Abstract

The invention discloses an emotion recognition method and device based on self-supervision and multi-modal comparison and a medium. The method comprises the following steps: acquiring multi-modal data, and preprocessing the multi-modal data to obtain a feature matrix; performing self-supervised feature learning pre-training on the feature matrix to obtain a comparative learning multi-mode encoder, and outputting alignment features of each mode through the comparative learning multi-mode encoder; performing multi-modal feature fusion on the alignment features of each modal output by the comparative learning multi-modal encoder to obtain a multi-modal feature vector; and performing emotion recognition based on the multi-modal feature vector to obtain an emotion recognition result. According to the method, self-supervised pre-training is introduced, so that more representative multi-modal features can be effectively extracted; feature fusion is carried out on three modes of micro expression, eye movement and electroencephalogram; and finally, fine changes are captured through a cross-modal attention and interaction mechanism, so that the multi-modal cross-subject emotion recognition method shows higher stability and advancement in a multi-modal cross-subject emotion recognition task.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, specifically to an emotion recognition method, device, and medium based on self-supervised and multimodal comparison. Background Technology

[0002] Mental health is a key component of overall health, and mental health and related issues are important topics of discussion among the public. Emotional cognition and regulation, as core elements of mental health, present significant difficulties in some populations, potentially increasing the risk of mental illnesses such as depression. Furthermore, sub-optimal mental health is also quite common. Therefore, attention to mental health plays a vital role in promoting harmonious social development.

[0003] Traditional emotion intervention methods, such as psychological counseling and drug treatment, are effective to some extent, but they face problems such as high cost and strong individual differences, and cannot effectively complete the emotion recognition task in the present.

[0004] Currently, while using biological characteristics such as facial expressions and EEG signals for emotion recognition is more effective than traditional methods, it still suffers from problems such as strong individual differences, weak resistance to interference, and low recognition accuracy. Most experiments use video or audio to passively induce emotions in subjects, who cannot choose their own emotion regulation methods or coping strategies, lacking autonomy and showing significant differences from their daily emotion regulation behaviors. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an emotion recognition method, device and medium based on self-supervision and multimodal comparison.

[0006] The first aspect of this invention provides an emotion recognition method based on self-supervised and multimodal comparison, comprising the following steps: Acquire multimodal data, preprocess the multimodal data, and obtain the feature matrix; The feature matrix is ​​pre-trained using self-supervised feature learning to obtain a contrastive learning multimodal encoder, which then outputs the alignment features of each modality. Multimodal feature fusion is performed on the alignment features of each modality output by the contrastive learning multimodal encoder to obtain a multimodal feature vector; Emotion recognition is performed based on the multimodal feature vectors to obtain the emotion recognition result.

[0007] Furthermore, the multimodal data includes at least two of the following: EEG signals, eye-tracking signals, and micro-expression data.

[0008] Furthermore, the preprocessing of multimodal data specifically includes preprocessing of EEG signals, preprocessing of eye-tracking signals, and preprocessing of micro-expression data. The preprocessing of EEG signals specifically includes the following steps: Bandpass filtering, artifact removal, and downsampling are performed on the EEG signal; The EEG signal is processed using short-time Fourier transform to obtain frequency domain characteristic signals; The frequency domain feature signal is processed using differential entropy feature calculation to obtain a differential entropy feature sequence; The differential entropy feature sequence is smoothed using a linear dynamic system and combined into an EEG feature matrix; The preprocessing of eye movement signals specifically includes the following steps: Perform missing value processing and outlier processing on eye movement signals; The core features in the eye movement signal are identified, and feature extraction is performed on the core features to obtain multiple core feature values; The core feature values ​​are standardized and combined into an eye-tracking feature matrix. The preprocessing of micro-expression data specifically includes the following steps: Candidate facial windows are generated based on micro-expression data; Bounding box regression calibration is performed on the candidate face windows to eliminate erroneous candidate face windows; The remaining candidate facial windows are localized for facial landmarks, facial landmark features are identified, and combined into a micro-expression feature matrix.

[0009] Furthermore, the self-supervised feature learning pre-training of the feature matrix specifically includes the following steps: Perform time window alignment processing on the feature matrix; Generative self-supervised pre-training is performed on the feature matrix of each modality to obtain the initial encoder for each modality; For each modality's feature matrix, construct positive and negative pairs; where a positive pair represents multiple feature sample combinations that have the same collection time, collection object, and object's emotional state, and a negative pair represents multiple feature sample combinations that have at least one different collection time, collection object, and object's emotional state. Use the initial encoder as the training encoder; The initial encoders for each modality are trained using positive and negative pairs of each modality, and the contrastive loss of the trained encoders for each modality is calculated. The contrastive loss of each modality training encoder is weighted and summed to obtain the total contrastive loss; Update the intermediate parameters of the encoder for each modality, and return to the step of training the initial encoder for each modality using the positive and negative pairs of each modality until the total contrastive loss converges. Once the total contrastive loss converges, the alignment features of each modality are obtained through the final trained encoder.

[0010] Furthermore, the generative self-supervised pre-training of the feature matrix for each modality specifically includes the following steps: The feature matrix of each modality is randomly masked, and the masked feature matrix is ​​used as the encoder training data. The encoder training data is input into the encoder, and the encoder processes the data to obtain the latent representation of each mode. The feature matrix of each modality is reconstructed using a decoder based on the latent representation and positional encoding of each modality; The reconstruction loss for each mode is calculated using the mean squared error function; The reconstruction loss of each mode is summed to calculate the total reconstruction loss; The reconstructed feature matrix is ​​used as the new encoder training data for each modality. The process of inputting the encoder training data into the encoder and processing it to obtain the latent representation of each modality continues until the total reconstruction loss is minimized. Once the total reconstruction loss is minimized, the encoder for each mode is used as the output of the initial encoder.

[0011] Furthermore, the multimodal feature fusion of the alignment features of each modality output by the contrastive learning multimodal encoder specifically includes the following steps: Alignment features of each modality are mapped to the same dimension to obtain projection vectors of multiple modalities; The projection vectors are modally fused pairwise to obtain multiple fused feature vectors; Global feature fusion is performed on multiple fusion feature vectors to obtain multimodal feature vectors.

[0012] Furthermore, in the step of performing pairwise modal fusion on the projection vectors, modal fusion of EEG modality and eye-tracking modality is completed by PSO fusion; modal fusion of eye-tracking modality and micro-expression modality is completed by LSTM fusion; and modal fusion of EEG modality and micro-expression modality is completed by cross-attention fusion. In the step of performing global feature fusion on multiple fusion feature directions, global feature fusion is completed through Transformer fusion.

[0013] Furthermore, the emotion recognition based on the multimodal feature vector specifically includes the following steps: The multimodal feature vectors are subjected to average pooling to obtain pooled feature vectors; The pooling feature vector is mapped to the emotion category space, and then linearly transformed through a fully connected layer to obtain multiple category scores. The category scores are converted into an emotion probability distribution using Softmax classification, and the emotion category and emotion intensity are output as the emotion recognition result based on the emotion probability distribution.

[0014] Another aspect of the present invention discloses an electronic device, including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the aforementioned emotion recognition method based on self-supervised and multimodal comparison.

[0015] In another aspect, the present invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the above-described emotion recognition method based on self-supervised and multimodal comparison.

[0016] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0017] The embodiments of the present invention have the following beneficial effects: The present invention provides an emotion recognition method, device, and medium based on self-supervised and multimodal comparison. By extracting and fusing facial micro-expressions, EEG signals, and eye-tracking signals, a self-supervised pre-training strategy is introduced. Self-supervised pre-training can effectively extract more representative multimodal features, alleviate the dependence of traditional supervised models on large-scale labeled data, and reduce the uncertainty caused by modal and individual differences. At the same time, the feature fusion of micro-expressions, eye-tracking, and EEG modalities captures the key correlations between different modalities, thereby significantly improving the robustness and accuracy of emotion recognition. Finally, by capturing subtle changes through cross-modal attention and interaction mechanisms, the present invention exhibits higher stability and advancement in multimodal cross-subject emotion recognition tasks, and can be widely applied in the field of emotion recognition.

[0018] Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description or may be learned by practice of the invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the basic implementation process of the emotion recognition method based on self-supervision and multimodal comparison of the present invention; Figure 2 This is a time-domain waveform diagram of the EEG signal after preprocessing in this invention; Figure 3 This is a box-shaped diagram of the EEG signal distribution after preprocessing in this invention; Figure 4 This is a schematic diagram of the distribution of eye movement signals after preprocessing in this invention; Figure 5 This is a schematic diagram of the MTCNN network structure used in the micro-expression data preprocessing of this invention; Figure 6 This is a scatter plot visualization of micro-expression data after preprocessing in this invention; Figure 7 This is a schematic diagram of the self-supervised pre-training learning process in this invention; Figure 8 This is a schematic diagram of the initial architecture of the encoder used in the self-supervised learning of this invention; Figure 9 This is a schematic diagram of the contrastive learning architecture in this invention; Figure 10 This is a schematic diagram of the encoder structure for obtaining EEG signals through self-supervised learning training in this invention; Figure 11 This is a schematic diagram of the structure of an electronic device according to the present invention; Figure 12 This is a schematic diagram of a computer-readable storage medium structure according to the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] The first embodiment of this invention provides an emotion recognition method based on self-supervised and multimodal comparison, such as... Figure 1 As shown, it includes the following steps: S1. Acquire multimodal data, preprocess the multimodal data, and obtain the feature matrix; S2. Perform self-supervised feature learning pre-training on the feature matrix to obtain a contrastive learning multimodal encoder, and output the alignment features of each modality through the contrastive learning multimodal encoder; S3. Perform multimodal feature fusion on the alignment features of each modality output by the contrastive learning multimodal encoder to obtain a multimodal feature vector; S4. Emotion recognition is performed based on multimodal feature vectors to obtain the emotion recognition results.

[0023] The emotion recognition method based on self-supervised and multimodal comparison provided in this invention extracts and fuses facial micro-expressions, EEG signals and eye-tracking signals, and introduces a self-supervised pre-training strategy to achieve multimodal emotion recognition. The system captures emotion features from multiple dimensions, significantly improving recognition accuracy and robustness, and enhancing signal anti-interference ability.

[0024] The implementation process of each step of this invention is described in detail below: S1. Acquire multimodal data, preprocess the multimodal data, and obtain the feature matrix.

[0025] The multimodal data used in this embodiment of the invention includes at least two of the following: EEG (Electroencephalogram) signals, eye-tracking signals, and micro-expression data. Among them, EEG signals have high temporal resolution in emotion recognition, but such signals are easily affected by external interference, requiring high signal quality and data preprocessing; moreover, there are large inter-individual differences, resulting in weak cross-individual generalization ability, making it difficult for the model to achieve stable results in independent subject experiments.

[0026] Eye-tracking signals primarily analyze fixation patterns and areas of gaze, revealing visual attention allocation and emotion recognition strategies. However, they cannot directly reflect the brain's deep processing of emotions and have limited ability to recognize latent emotions. EEG signals, on the other hand, can capture real-time brain activity, revealing activation patterns and neural oscillation characteristics in brain regions induced by emotion, making them suitable for analyzing latent emotional responses and rapid changes. Therefore, combining EEG and eye-tracking signals can compensate for the shortcomings of a single mode, providing a more comprehensive emotion recognition analysis.

[0027] Microexpression data refers to the rapid, unconscious, spontaneous facial movements that occur when a person experiences strong emotions. Microexpression data has shown high reliability in emotion recognition and can complement EEG signals and eye-tracking signals effectively.

[0028] It is evident that fusing multimodal signals (including micro-expressions, EEG, eye movements, etc.) for emotion recognition can capture emotional features from multiple angles, reduce interference and limitations caused by single signals, and effectively improve the accuracy and robustness of emotion recognition.

[0029] Step S1 involves preprocessing the multimodal data, specifically including preprocessing the EEG signal, preprocessing the eye movement signal, and preprocessing the micro-expression data.

[0030] In this embodiment of the invention, the preprocessing of the EEG signal is accomplished through short-time Fourier transform and differential entropy feature calculation. Specifically, the EEG signal is first subjected to bandpass filtering, artifact removal, and downsampling. Bandpass filtering, using a digital filter, applies a bandpass filter of 0.2-50Hz to remove low-frequency and high-frequency noise. Then, a threshold algorithm is used to remove blink artifacts; by calculating the standard deviation of the signal, all signal segments with a standard deviation higher than a certain threshold are considered blink artifacts and removed from the data. Finally, a digital filter is used to reduce the processed EEG signal from 1000Hz to 200Hz to reduce data volume and computational complexity. These operations effectively remove noise and artifacts, improve signal quality, and thus facilitate subsequent data analysis and research.

[0031] The short-time Fourier transform operation includes the following steps: Windowing and Parameter Settings: Perform a short-time Fourier transform on the EEG signal. First, divide the EEG signal into windows, with each window length set to 4 seconds and no overlap between adjacent windows. Since the sampling frequency after downsampling is 200Hz, each window contains 800 sample points (4 seconds × 200Hz).

[0032] Window Function Selection and Application: When performing a short-time Fourier transform, the Hanning window function is used to smooth the signal within each window to reduce spectral leakage. The Hanning window function is defined as follows: ; in, This is the window length (800 in this case).

[0033] For the signal within each window (where n is the number of samples in the window, i.e., 800), its short-time Fourier transform formula is: ; in, ( (This refers to the number of sample points, i.e., 800).

[0034] Through this transformation, the signal is successfully converted from time-domain features to frequency-domain features.

[0035] The calculation of differential entropy features includes the following steps: First, the frequency domain signal after short-time Fourier transform is divided into five commonly used frequency ranges: Delta (0.5-4Hz), Theta (4-7Hz), Alpha (8-12Hz), Beta (13-30Hz), and Gamma (31-50Hz). The EEG signal sequence largely follows a Gaussian distribution. And because the DC component has been filtered out, the mean Its variance According to Passevar's theorem, the mean of EEG signal energy is proportional to the energy spectrum, i.e., the variance of different frequency bands. With the energy of this frequency band Proportional.

[0036] Feature calculation process: Therefore, the formula for calculating the differential entropy feature on each frequency band is: ; in, Indicates different frequency bands.

[0037] By calculating the differential entropy value within each frequency band, a DE feature vector sequence is obtained, ultimately forming a... The feature matrix of , where 5 represents the number of windows after being segmented into 4-second intervals, and 5 represents the number of frequency bands in the DE feature vector.

[0038] The effect of this invention on EEG signal preprocessing is as follows: Figure 2 , 3 As shown, by extracting the energy and statistical indicators of EEG signals in different frequency bands, the impact of external interference on subsequent model training is significantly reduced.

[0039] In some embodiments, considering that emotion is a dynamic process, EEG signal features may undergo abrupt changes due to physiological or technical factors. To reduce the impact of such abrupt changes on subsequent analysis, embodiments of the present invention employ a Linear Dynamic System (LDS) to smooth the feature sequence. LDS can better capture the temporal changes of EEG signal features over time, enabling smooth connections of feature curves, thereby more accurately reflecting the emotional change process and improving the accuracy of emotion recognition.

[0040] In this embodiment of the invention, the preprocessing of eye movement signals is mainly performed through time-domain processing. First, the eye movement signals are cleaned, including the following steps: Missing value handling: Carefully check the eye-tracking data for missing values. Depending on the data distribution, if the data is approximately normally distributed, the mean can be used for imputation; if the data distribution is skewed, the median is more suitable; interpolation algorithms, such as linear interpolation, can also be used to imput missing values ​​to ensure data integrity.

[0041] Outlier identification and handling: Statistical analysis is used to calculate the mean and standard deviation of each feature of the eye-tracking data, and reasonable thresholds are set to identify outliers. For identified outliers, they can be deleted to avoid interfering with subsequent analysis, or data correction can be performed to better reflect the overall trend of the data.

[0042] After data cleaning, this embodiment of the invention selects core features from eye-tracking characteristics. For example, changes in pupil diameter are potentially correlated with the level of emotional arousal, and saccade patterns are closely related to attention and emotional state. To determine the most valuable subset of features, this embodiment of the invention uses methods such as analysis of variance to calculate the statistical differences of each feature under different emotional states, filtering out features that show significant differences under different emotions, thereby identifying key eye-tracking features for emotion recognition as core features.

[0043] As a preferred embodiment, the core features selected in the embodiments of the present invention are shown in the following table:

[0044] After selecting the core features, the average value of each selected eye-movement feature is calculated over a specific time window. Taking pupil diameter as an example, the average pupil diameter is calculated during each presentation of an emotional stimulus. This reflects the overall level of pupil size under that emotional state. The formula for calculating the average is: ; in, These are data points in a time series. It represents the number of data points.

[0045] Standard deviation calculation: The standard deviation of eye movement features is calculated to measure the dispersion of the data around the mean. A larger standard deviation indicates a greater range of variation in eye movement features within that emotional state. The formula for standard deviation is: ; Extreme value features: Extracting the maximum and minimum values ​​of eye movement features within a time window, these extreme values ​​can provide relevant information about the range of eye movement variations. For example, the maximum pupil diameter may be closely related to peak emotional responses, while the minimum pupil diameter may reflect physiological characteristics of emotional troughs or calm states.

[0046] Slope and Rate of Change: Calculating the rate of change of eye movement features over time, such as the slope of pupil diameter between adjacent time points, reflects the dynamic characteristics of eye movement changes and helps capture subtle differences in emotional changes. The formula for calculating the slope is: ; in, It refers to a specific point in time.

[0047] After feature extraction, this embodiment of the invention employs a min-max normalization method to map the feature values ​​to the [0,1] interval. The normalization formula is: ; in, It is the minimum value of the feature. It is the maximum value of the feature.

[0048] Data standardization eliminates the influence of different dimensions between features, ensuring that each feature has equal importance in subsequent analysis, thus facilitating model training and evaluation. Finally, the extracted and standardized eye-tracking features are organized into a feature matrix. In this matrix, each row represents a sample, i.e., an eye-tracking data record under a single emotional stimulus; each column represents a feature, such as the mean pupil diameter, standard deviation, extreme values, and slope. This matrix structure facilitates data management and subsequent model input, and the effect of eye-tracking signal preprocessing is as follows: Figure 4 As shown.

[0049] In this embodiment of the invention, preprocessing of micro-expression data is performed using the MTCNN model. For example... Figure 5 As shown, MTCNN is composed of three distributed convolutional neural networks, including P-Net (Proposal Network), R-Net (Refine Network), and O-Net (Output Network).

[0050] The MTCNN model processes facial micro-expression data through the following three stages: Phase 1: This phase uses P-Net (Proposal Network) with fully convolutional networks to obtain candidate face windows and their bounding box regression vectors. The candidates are then calibrated based on the estimated bounding box regression vectors. Highly overlapping candidates are then merged using Non-Maximum Suppression (NMS).

[0051] Phase 1 is formulated as a binary classification problem. For each sample... P-Net uses cross-entropy loss for processing: ; in It is a representation sample generated by the network. The probability of a face. Notation. This indicates the actual label.

[0052] Phase 2: In this phase, all candidates are fed into another R-Net (refined network) with a CNN structure. The R-Net can further eliminate a large number of erroneous candidates, perform bounding box regression calibration, and perform NMS processing.

[0053] For each candidate window, R-Net predicts its offset from the nearest ground truth target. Phase 2 is formulated as a regression problem, where R-Net calculates the offset for each sample. Using Euclidean loss: ; in The regression target is obtained from the network. These are the actual coordinates.

[0054] Phase 3: The processing in Phase 3 is similar to that in Phase 2, but the goal in this phase is to more accurately identify facial regions and output the positions of five facial landmarks, which are combined into a micro-expression feature matrix.

[0055] Phase 3 is formulated as a regression problem, and O-Net is used to minimize the Euclidean loss and output facial landmarks: ; in These are the coordinates of facial landmarks obtained from the internet. These are the true coordinates of the i-th sample.

[0056] The effect of micro-expression data processed by the MTCNN network is as follows: Figure 5 As shown. From Figure 5 As can be seen, the preprocessing algorithm of this invention performs excellently in suppressing noise, removing outliers, and preserving the intrinsic features of the signal, providing high-quality input data for emotion recognition.

[0057] S2. Perform self-supervised feature learning pre-training on the feature matrix to obtain a contrastive learning multimodal encoder, and output the alignment features of each modality through the contrastive learning multimodal encoder.

[0058] When labeled data is insufficient, traditional direct training of multimodal emotion recognition models often leads to overfitting or insufficient generalization ability. To address this issue, self-supervised pre-training on a large amount of unlabeled or weakly labeled data is a highly effective approach. This allows the model to learn more general and robust representations without relying on manual annotation, laying a solid foundation for subsequent fine-tuning of labeled emotion classification. In this embodiment, the invention employs a self-supervised learning method for model pre-training, then freezes or semi-freezes the encoders of each modality for downstream classification or fusion. By designing contrastive learning and generative reconstruction tasks, it fully utilizes massive amounts of unlabeled data to learn deep and robust feature representations, thereby significantly reducing dependence on labeled data and improving the model's generalization ability.

[0059] Specifically, such as Figure 7 As shown, the self-supervised feature learning pre-training of the feature matrix includes the following steps: S2-1. Perform time window alignment processing on the feature matrix; In this embodiment of the invention, the EEG signal, eye movement signal, and micro-expression data are first segmented and aligned without labels; a feature sample can be denoted as x. EEG x EOG x ME .

[0060] S2-2. Generative self-supervised pre-training is performed on the feature matrix of each modality to obtain the initial encoder of each modality.

[0061] Generative self-supervised learning primarily learns features by reconstructing or predicting masked data. It does not require external sentiment labels and can learn a "general representation" that well characterizes the distribution of input data by relying on reconstruction loss. By randomly occluding the input data, the encoder extracts the latent representation, and then the decoder reconstructs the original input, thereby forcing the model to learn the distribution characteristics of each modality.

[0062] In this embodiment of the invention, generative self-supervised pre-training is performed on the feature matrix of each modality, specifically including the following steps: S2-2-1. Randomly mask the feature matrix of each modality to obtain the masked feature matrix as the encoder training data.

[0063] During training, random masking is performed on parts of the input from different modalities or the same modality, such as masking for “EEG occlusion”, which forces the encoder to reconstruct the masked EEG using “EOG + ME” to achieve cross-modal reconstruction.

[0064] For each modal data x m(m represents EEG, EOG, ME) Perform random occlusion to generate the occluded input. For example, randomly select a time step p% and set the corresponding input value to 0: ; in It is the set of time steps that are covered.

[0065] S2-2-2. Input the encoder training data into the encoder, and obtain the latent representation of each mode through encoder processing; like Figure 8 As shown, in this embodiment of the invention, each modality is processed using an LSTM encoder to obtain a latent representation.

[0066] S2-2-3. Reconstruct the feature matrix of each modality using a decoder based on the latent representation and positional encoding of each modality; After obtaining the latent representation, embodiments of the present invention use an RNN decoder. potential representation Using location information (such as time step location encoding) as a condition, reconstruct the original data: .

[0067] S2-2-4. Calculate the reconstruction loss for each mode using the mean squared error function.

[0068] In this embodiment of the invention, mean squared error (MSE) is used as the loss function, and only the reconstruction error of the obscured portion is calculated: .

[0069] S2-2-5. Summarize the reconstruction loss for each mode and calculate the total reconstruction loss: .

[0070] S2-2-6. Use the reconstructed feature matrix as the new encoder training data for each modality, return to the step of inputting the encoder training data into the encoder and obtaining the latent representation of each modality through encoder processing, until the total reconstruction loss is minimized.

[0071] S2-2-7. After the total reconstruction loss is minimized, the encoder of each mode is used as the output of the initial encoder.

[0072] After training, only the RNN encoders for each modality are retained. , used for downstream tasks.

[0073] S2-3. Construct positive and negative pairs for the feature matrix of each mode; This invention employs a strategy combining contrastive learning and generative self-supervised multimodal modeling (MAE): first, MAE is used to learn general low-order feature representations; then, contrastive learning is used to align multimodal features and enhance discriminative power. The contrastive learning architecture is as follows: Figure 9 As shown.

[0074] In EEG / eye-tracking / micro-expression multimodal scenarios: EEG segments, EOG segments, and micro-expression data collected at the same time, with identical subjects and emotional states, are set as positive pairs. For segments from different time periods, different individuals, or different emotional states, negative pairs are constructed.

[0075] S2-4. Use the initial encoder as the training encoder; In this embodiment of the invention, an independent encoder is designed for each mode. , , Each encoder maps the input to a fixed-dimensional representation vector, and subsequent comparative learning is performed to bring multimodal features within the same time window closer together. This embodiment of the invention uses convolutional networks to extract local spatiotemporal features and encode sequences or image frames.

[0076] Taking EEG signals as an example, such as Figure 10 As shown, in this embodiment of the invention, the input is set... It can be viewed as a two-dimensional "image" (time * channels), and features are extracted using convolutional layers: ; Where "*" represents a convolution operation, and BN is batch normalization. The activation function is ReLU, and GAP is global average pooling. The final output is... .

[0077] For eye movement signals and micro-expression data, the embodiments of the present invention also design corresponding CNN encoders.

[0078] S2-5. Train the initial encoder of each modality using the positive and negative pairs of each modality, and calculate the contrastive loss of the trained encoder for each modality.

[0079] Taking EEG and eye-tracking signals as an example, let the sample be the one directly opposite to ( For N samples in a batch, the InfoNCE loss is: ; in For cosine similarity, For temperature parameters. Similarly, corresponding losses are constructed for EEG-microexpression and EOG-microexpression.

[0080] S2-6. Weighted summation of the contrastive loss for each modality training encoder to obtain the total contrastive loss; Finally, the contrastive losses of each modality are weighted and summed: .

[0081] S2-7. Update the intermediate parameters of the encoder for each modality, and return to the step of training the initial encoder for each modality using the positive and negative pairs of each modality until the total contrastive loss converges. In this embodiment of the invention, the optimizer Adam is used to update all encoder parameters until the total contrast loss converges.

[0082] S2-8. Once the total contrastive loss converges, the alignment features of each modality are obtained through the final trained encoder.

[0083] After training, this embodiment of the invention establishes a contrastive learning pre-trained multimodal encoder that can directly output alignment features of the three modalities. .

[0084] In this embodiment of the invention, fine-grained multimodal representations are captured through self-supervised feature learning. Through self-supervised pre-training, the model can automatically extract key semantic information and spatiotemporal dynamic features from the intrinsic structures of different modalities (such as EEG, eye movement, and micro-expressions), enabling it to obtain consistent feature representations even in the face of high noise or uneven data distribution.

[0085] S3. Perform multimodal feature fusion on the alignment features of each modality output by the contrastive learning multimodal encoder to obtain a multimodal feature vector.

[0086] In step S3, the alignment features of each modality output by the contrastive learning multimodal encoder are fused using multimodal features. This specifically includes the following steps: S3-1. Map the alignment features of each modality to the same dimension to obtain the projection vectors of multiple modalities.

[0087] For subsequent PCL-Transformer fusion processing, step S3-1 first maps them to a unified dimension d to obtain the projection vectors of each modality:

[0088]

[0089]

[0090] in All of these are learnable parameters. Meanwhile, in , Add positional and modal encoding to explicitly provide temporal positional and modal differentiation information.

[0091] S3-2. Perform modal fusion on each pair of projection vectors to obtain multiple fused feature vectors; Step S3-2 uses the PCL-Transformer model for deep association learning. The core of PCL-Transformer lies in its self-attention mechanism, which allows different modalities to "attention" each other at the temporal, frequency, and feature levels, and automatically assigns higher weights to key information. Compared to traditional single feature extraction or simple fusion methods, Transformer can simultaneously capture subtle changes in different frequencies of EEG, dynamic features of eye movements, and subtle changes in micro-expressions, greatly improving the accuracy and generalization ability of emotion recognition.

[0092] To reduce interference from multiple noise sources on EEG and eye-tracking data, this embodiment of the invention uses PSO fusion for modal fusion of EEG and eye-tracking modalities. For example... Figure 8 As shown, PSO is a global optimization algorithm based on population search, which has high search efficiency and global convergence. It finds the optimal solution by simulating the cooperative behavior of biological populations, avoiding the problem of single-point search algorithms getting trapped in local optima. It performs particularly well in high-dimensional complex problems (such as multimodal data fusion).

[0093] The PSO fusion algorithm is based on a linear weighted model, which is expressed as follows: ; in F This represents a fused feature vector containing the predicted probabilities of four categories. and These correspond to the correlation vector of the EEG feature and the posterior probability of the eye movement feature in the j-th trial, respectively. and These represent the fusion weights of EEG features and eye-tracking features, respectively. The classification result corresponds to the feature with the highest probability. F The category. As two elements of a particle, and Determined by the PSO method. The specific steps of PSO are as follows: Initialize parameters such as particle swarm size, learning factors (c1, c2), inertia weight w, and number of iterations.

[0094] The PSO (Particle Swarm Optimization) classifier uses features from both individual and unimodal models as input, and then calculates the fitness value using a fitness function. The fitness function includes a penalty term for the prediction error of both individual and overall samples. This makes the optimal particle more robust to the feature distributions of both individual and overall samples. The fitness function is defined as: ; Where M is the number of prediction errors, and It is the penalty parameter of the fitness function. Let represent the true label of the j-th trial, and This represents the predicted label for the j-th trial.

[0095] When the fitness value no longer meets the stopping condition, the particles are updated using the particle update equation. Finally, the best particle generated during the iteration process is used as the fusion weight. The particle update equation is defined as: ; in It is the optimal position for the current particle. It represents the optimal position among all particles, and rand is a random number. It is the particle velocity, and It is the first l The position of each particle.

[0096] The PSO algorithm, through diverse particle exploration and update mechanisms, can maintain diversity and filter out noise interference during the search process, optimizing feature combinations. Furthermore, in multimodal fusion, EEG and eye-tracking typically have different temporal and frequency characteristics. PSO can quickly find the optimal parameters for fusion by dynamically adjusting the position and velocity weights of particles. Its global search characteristic ensures exploration of the entire feature space, while its local search characteristic enhances detail optimization capabilities when approaching the optimal solution, thereby improving overall model performance.

[0097] Eye-tracking signals and micro-expression data are essentially time-series data with temporal correlation. Therefore, this embodiment of the invention employs the LSTM temporal fusion method for modal fusion. LSTM is suitable for processing time-series data and can capture long-term dependencies. The synchronization of eye-tracking signals and micro-expressions is crucial in behavioral analysis, and LSTM can simultaneously process and fuse multiple modal data by stacking different network layers or using multi-channel input. This allows the model to synchronously process different input streams on the timeline, naturally fusing multimodal data while maintaining the temporal correlation between different data sources.

[0098] like Figure 9As shown, the LSTM model is mainly divided into three parts: the input layer (inputting processed eye movement signals and micro-expression data features), the feature extraction layer (using the LSTM network to extract the temporal features of eye movement signals and micro-expressions respectively), and the multimodal fusion layer (fusion of eye movement signals and micro-expression features).

[0099] Input layer: Each input data is a time series matrix. Indicates at time step Input features. For eye-tracking signals and micro-expression data, define respectively: Eye movement signal input: ; in dE This represents the dimension of eye movement features.

[0100] Micro-expression input: ; in dM This represents the dimension of eye movement features.

[0101] LSTM feature extraction layer: This invention designes two independent LSTM networks: LSTM for eye-tracking signal E: Used to extract temporal features of eye-tracking signals.

[0102] LSTM for Microexpressions M: Used to extract temporal features from microexpression data.

[0103] The input to each LSTM network is a time series matrix. : ; in This indicates the hidden state at the current time step. In cellular state, This represents the current input feature.

[0104] Each LSTM network is calculated according to the following recursive formula: Input Gate: ; Input gate controls the current input How does it affect the cell state? It depends on the current input and the previous hidden state. Generate a switch to determine the weight of the input information.

[0105] Forgotten Gate: ; The forgetting gate determines the previous cell state. Which information should be retained or discarded?

[0106] Output gate: ; Output gate controls new cell states How to generate output hidden state .

[0107] Cell status update: ; Cell state updates can generate candidate information using the current input and the previous hidden state.

[0108] Cell state: ; Cellular state is the core of LSTM memory, which combines information from the previous moment with current input.

[0109] Hidden state: ; Hidden states are used for the output of the current time step, for subsequent calculations, or to be passed to the next time step.

[0110] in This represents element-wise multiplication, and tanh represents the Sigmoid activation function.

[0111] Multimodal fusion layer: An attention mechanism is introduced into the multimodal fusion layer to assign different weights to different time steps: ; Calculate the weighted fusion: ; Eye movement signals and micro-expression changes are dynamic in behavior recognition. LSTM can continuously and dynamically adjust weights during model training and prediction to adapt to changes in input. This dynamic modeling capability makes LSTM more effective than traditional methods in capturing subtle differences in behavioral patterns. When fusing eye movement and micro-expression signals, LSTM can automatically learn complex nonlinear relationships from the data, while traditional methods often rely on manual feature extraction, which can easily miss potential related features or lead to performance degradation. Through its neural network structure, LSTM can adaptively extract meaningful features from the input data. Furthermore, in practical applications, eye movement signals and micro-expression data may be affected by noise (such as problems with the acquisition equipment or data loss). The recursive structure and memory units of LSTM can effectively filter out some noisy data and have better tolerance to uncertainty.

[0112] Finally, regarding EEG signals and micro-expression data, each contains a wealth of information. Traditional methods tend to treat this information with equal weight, thus ignoring the different weights of key features. Therefore, this invention uses a cross-attention mechanism for feature fusion to simulate the neural processes involved in emotion recognition in the brain, thereby fusing EEG and micro-expression features. Cross-modal attention fusion mechanisms can capture the correlation information between different modal input sequences, which is commonly used in machine translation or multimedia tasks. Facial expressions are used as visual input patterns, and EEG signals are used as a direct reflection of brain activity. This invention effectively focuses on and integrates information from different modalities, simulating the collaborative work between dynamic neural structures in the brain, which is also key to improving the recognition accuracy of the multimodal model in this invention.

[0113] Specifically, the cross-attention fusion network used in the embodiments of the present invention is as follows: Figure 10 As shown, it includes a dimensionality reduction encoder, a cross-attention layer, and an output layer.

[0114] (1) Dimensional reduction encoder: For size The facial expression feature vector F in the facial expression domain and of size The EEG feature vectors in the cognitive domain have a set of hyperparameters that need to be optimized. These values ​​are crucial for determining the accuracy of recognition. Through multiple tests and comparisons, embodiments of the present invention have found that when… =16, The optimal performance is achieved when the vector dimension E is 64. To meet the cross-attention requirement of vector alignment and minimize the use of additional parameters, this embodiment of the invention designs a projection layer as an alignment encoder to reduce the vector dimension E from 64 to 16. The projection layer includes a normalization layer, a fully connected layer, and a dropout layer, which generate the projection representation. Subsequently and F Both are input into the public channel and the private channel. In the private channel, the projection vector... Assigned to However, in public passageways, and F New feature vectors with mutual information will be generated under the influence of the cross-modal attention layer.

[0115] (2) Cross-attention layer: The cross-attention layer primarily utilizes self-attention and cross-attention mechanisms. Self-attention is an attention mechanism that establishes relationships between different positions in the input sequence. The calculation formula is as follows: ； ； in ， ， Represents three trainable coefficient matrices, with the input matrix... X Multiply by the three matrix parameters to obtain Q , K , V Then, attention scores are obtained according to the formula. First, the attention is sorted before cross-attention, and preliminary extraction is performed. and F .

[0116] The basic form of multi-head attention is shown below: ; ; ; ; ; Wherein, the input matrix X is multiplied by the parameters. , , Get Q , K , V Due to multi-head attention, these values ​​are mapped to , , Under different parameters , , Then, the attention score matrices from different heads are concatenated and used with parameters. Obtain the final multi-head attention score matrix.

[0117] Cross-attention is a variation of multi-head attention. It simply swaps the heads. , Generate A and B in the same head from different input matrices: ; ; Cross-attention allows one modality to receive information from another, thereby creating a feature vector that incorporates information from both modalities. Furthermore, embodiments of the invention deploy another layer of self-attention after cross-attention, with the aim of again... and FAfter information fusion processing, the EEG projection vectors are... and micro-expression feature vectors F Under the influence of the cross-modal attention layer, it is ultimately generated as and .

[0118] Private Channel and Connect to the public channel via residual. and To mitigate the gradient explosion problem, this embodiment of the invention applies layer normalization after the residual connection. After the data passes through the entire cross-modal attention fusion layer, the feature vector... F In the field of facial expressions, it has been transformed into The feature vectors in the cognitive domain are transformed into .

[0119] (3) Output layer and They are concatenated together to generate the final multimodal feature vector: .

[0120] Emotional activity involves not only individual neural events but also the interaction of a distributed network. This distributed structure includes the occipitotemporal neocortex, amygdala, orbitofrontal cortex, and right frontoparietal cortex. Furthermore, the topological organization of this functional network is dynamic, with smaller functional units recombining at different points in time. This results in the brain collecting and processing information from different brain regions during emotional activity.

[0121] Attention mechanisms can assign weights based on the importance of inputs, improving fusion quality and model performance. EEG and micro-expression data have different temporal and spatial characteristics, and traditional fusion methods are limited in capturing the deep relationships between them. Cross-attention mechanisms enhance the connections between different modalities by cross-mapping EEG features with micro-expression features, enabling the model to better capture cross-modal associations. Micro-expressions are subtle and short-lived, and traditional methods easily lose this important information when identifying subtle changes in micro-expressions. Through multi-layer attention mechanisms, the model can capture detailed features of EEG and micro-expressions at different levels, improving the ability to detect subtle changes. At the same time, EEG signals and micro-expressions have significant temporal dimensions. Cross-attention mechanisms can identify the temporal synchronicity or lag between EEG activity and micro-expression changes by associating temporal features, thereby better capturing the dynamic features of emotional or cognitive states. EEG signals are often affected by noise and artifacts. Through attention mechanisms, the model can ignore or reduce the weight of noisy features, making the fusion process more robust, especially maintaining high accuracy in real-time or non-ideal environments.

[0122] S3-3. Perform global feature fusion on multiple fusion feature vectors to obtain multimodal feature vectors.

[0123] In step S3-3, this embodiment of the invention employs a Transformer based on a self-attention mechanism for global feature fusion. The introduction of the attention mechanism is the core design of this model, enabling modalities to "attention" each other, thereby learning deep relationships between modalities and improving the accuracy of emotion recognition. The self-attention mechanism facilitates information exchange between different modalities, allowing the model to automatically assign higher weights to key information in different modalities, thus improving emotion recognition performance. Through multi-head attention, the Transformer can learn multiple types of relationships simultaneously, capturing combinations of different frequency variations in EEG signals, dynamic features of eye-tracking signals, and subtle changes in micro-expressions. This is more advantageous than traditional methods that rely on a single feature extraction mechanism.

[0124] like Figure 11 As shown, the overall framework of the Transformer model includes: 1. Input module: Receives and preprocesses EEG, eye-tracking, and micro-expression data, representing them as input-ready feature vectors.

[0125] The input feature vector for each modality can be represented as: EEG-Eye-tracking fusion data: ; EEG-Microexpression Fusion Data: ; Micro-expression-eye-tracking fusion data: ; in, T Indicates the number of time steps. X This represents the feature dimension. Input data can be mapped to the same dimension through embedding layers for subsequent processing.

[0126] 2. Transformer Encoder: A separate Transformer encoder is designed for each modality. Each encoder consists of multiple layers of self-attention and feedforward networks.

[0127] Multi-head attention calculation: ; in, Q , K , V These are the query matrix, key matrix, and value matrix in the self-attention mechanism, respectively.

[0128] The calculation of the i-th attention head is as follows: ; in, , , , It is the weight matrix for learning.

[0129] Each input feature After linear transformation, we obtain Each projection is a different projection. An attention mechanism is applied to each projection, and then the results are merged and passed through... Linear transformation output.

[0130] Feedforward networks: ; in , , , These are the weight matrix and bias terms in the feedforward network.

[0131] 3. Cross-modal self-attention mechanism: During the fusion process, the interaction between EEG-eye movement, EEG-microexpression, and eye movement-microexpression is calculated.

[0132] Formula for calculating cross-modal self-attention: ; Each of them It is an attention head calculated based on cross-modal features: .

[0133] 4. Fusion Module: Combines the outputs of each modality using a fusion strategy.

[0134] Weighted fusion using trainable weights: ; in For learnable parameters, This is the fused multimodal feature vector.

[0135] S4. Emotion recognition is performed based on multimodal feature vectors to obtain the emotion recognition results.

[0136] Regarding the classification module and loss function, this embodiment of the invention uses average pooling and fully connected layers to achieve classification, which helps reduce model complexity and prevent overfitting. Softmax and cross-entropy loss functions are suitable for multi-class classification tasks and can well adapt to the multi-class requirements in emotion recognition.

[0137] S4-1. Perform average pooling on the multimodal feature vectors to obtain pooled feature vectors.

[0138] Features after fusion Average pooling is performed along the time series dimension to aggregate time series information: ; in T The time step number corresponds to the temporal length of the fused features. The feature vector after average pooling. For the first t Features of each time step.

[0139] This step reduces the dimensionality of the features, integrating the information from each time step into a global vector.

[0140] S4-2. Map the pooling feature vector to the emotion category space, and perform a linear transformation through a fully connected layer to obtain scores for multiple categories.

[0141] Fully connected layers are used to map pooled features to the number of classes, as shown in the following formula: ; in, , These are the weight matrix and the bias terms. y This is the output of the fully connected layer, used for classification.

[0142] S4-3. Convert the category scores into an emotion probability distribution using Softmax classification, and output the emotion category and emotion intensity as the emotion recognition result based on the emotion probability distribution.

[0143] Multi-class classification is performed using Softmax and cross-entropy loss functions.

[0144] The Softmax function is shown below. It calculates the probability value for each category, ensuring that the sum of the probabilities is 1: ; Obtain the probability distribution vector .

[0145] After obtaining the probability distribution vector, select the category with the highest probability. p max As the final emotion label, and based on the highest probability value mapped to the intensity level, such as p max >0.6 indicates high strength, 0.6> p max >0.4 indicates medium intensity. p max <0.4 indicates low intensity. The emotion category and intensity are output as the emotion recognition result based on the emotion probability distribution.

[0146] This invention introduces a self-supervised pre-training strategy into the multimodal emotion recognition process. Compared to training solely based on labeled data, this project fully utilizes massive or additional unlabeled multimodal data (including physiological signals such as micro-expressions, eye movements, and EEG) to mine latent feature representations through self-supervised learning, thereby achieving better generalization ability and robustness in subsequent cross-subject emotion recognition tasks. For high-noise, small-sample, or cross-subject emotional brain-computer interface data, self-supervised pre-training can effectively extract more representative multimodal features, alleviating the dependence of traditional supervised models on large-scale labeled data, while reducing the uncertainty caused by modal and individual differences.

[0147] Secondly, this invention innovatively fuses features from three modalities: micro-expressions, eye movements, and electroencephalography (EEG). Multimodal signals contain more comprehensive emotional information: EEG and eye movements can reflect the user's internal emotional state, while micro-expressions supplement visual emotional expression. Unlike most approaches that only use a single modality or directly splice three modalities, this invention employs a strategy of "first fusing two modalities, then performing trimodal self-attention fusion," progressively enhancing feature interactions between modalities, reducing redundancy and noise, and helping to better capture key correlations between different modalities, thereby significantly improving the robustness and accuracy of emotion recognition.

[0148] Finally, based on the fusion of self-supervised pre-training and multimodal feature representations, a self-attention mechanism (Transformer architecture) is introduced for deep associative learning. The core of PCL-Transformer lies in utilizing the self-attention mechanism to allow different modalities to "attention" each other at the temporal, frequency, and feature levels, automatically assigning higher weights to key information. Compared to traditional single feature extraction or simple fusion methods, Transformer can simultaneously capture subtle changes in different frequencies of EEG, dynamic features of eye movements, and subtle changes in micro-expressions, greatly improving the accuracy and generalization ability of emotion recognition. The multimodal prior features obtained through self-supervised pre-training, combined with the cross-modal attention and interaction mechanism of PCL-Transformer, enable this project to demonstrate higher stability and state-of-the-art performance on multimodal cross-subject emotion recognition tasks.

[0149] Figure 11 This is a schematic diagram of the electronic device proposed in the second embodiment of the present invention. In this embodiment, the memory stores program instructions for implementing the emotion recognition method based on self-supervised and multimodal comparison of any of the above embodiments. The processor executes the program instructions stored in the memory to perform emotion recognition based on self-supervised and multimodal comparison. The processor can also be called a CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0150] The methods described in the first embodiment of the present invention are applicable to the embodiments of the present electronic device. The specific functions implemented by the embodiments of the present electronic device are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0151] Figure 12This is a schematic diagram of the structure of a computer-readable storage medium according to the third embodiment of the present invention. The computer-readable storage medium of the fourth embodiment of the present invention stores program instructions capable of implementing the above-described emotion recognition based on self-supervision and multimodal comparison. These program instructions can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned computer-readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0152] The methods described in the first embodiment of the present invention are applicable to the computer-readable storage medium embodiment. The specific functions implemented by the computer-readable storage medium embodiment are the same as those in the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method.

[0153] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to achieve the emotion recognition based on self-supervision and multimodal comparison provided in the above embodiment.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0155] In embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of the present invention may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0156] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention. Other embodiments of the present invention will readily conceive of by considering the specification and practicing the invention. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

Claims

1. An emotion recognition method based on self-supervised and multimodal comparison, characterized in that, Includes the following steps: Acquire multimodal data, preprocess the multimodal data, and obtain the feature matrix; The feature matrix is ​​pre-trained using self-supervised feature learning to obtain a contrastive learning multimodal encoder, which then outputs the alignment features of each modality. Multimodal feature fusion is performed on the alignment features of each modality output by the contrastive learning multimodal encoder to obtain a multimodal feature vector; Emotion recognition is performed based on the multimodal feature vectors to obtain the emotion recognition result.

2. The emotion recognition method based on self-supervised and multimodal comparison according to claim 1, characterized in that, The multimodal data includes at least two of the following: EEG signals, eye-tracking signals, and micro-expression data.

3. The emotion recognition method based on self-supervised and multimodal comparison according to claim 2, characterized in that, The preprocessing of multimodal data specifically includes preprocessing of EEG signals, preprocessing of eye-tracking signals, and preprocessing of micro-expression data. The preprocessing of EEG signals specifically includes the following steps: Bandpass filtering, artifact removal, and downsampling are performed on the EEG signal; The EEG signal is processed using short-time Fourier transform to obtain frequency domain characteristic signals; The frequency domain feature signal is processed using differential entropy feature calculation to obtain a differential entropy feature sequence; The differential entropy feature sequence is smoothed using a linear dynamic system and combined into an EEG feature matrix; The preprocessing of eye movement signals specifically includes the following steps: Perform missing value processing and outlier processing on eye movement signals; The core features in the eye movement signal are identified, and feature extraction is performed on the core features to obtain multiple core feature values; The core feature values ​​are standardized and combined into an eye-tracking feature matrix. The preprocessing of micro-expression data specifically includes the following steps: Candidate facial windows are generated based on micro-expression data; Bounding box regression calibration is performed on the candidate face windows to eliminate erroneous candidate face windows; The remaining candidate facial windows are localized for facial landmarks, facial landmark features are identified, and combined into a micro-expression feature matrix.

4. The emotion recognition method based on self-supervised and multimodal comparison according to claim 1, characterized in that, The self-supervised feature learning pre-training of the feature matrix specifically includes the following steps: Perform time window alignment processing on the feature matrix; Generative self-supervised pre-training is performed on the feature matrix of each modality to obtain the initial encoder for each modality; For each modality's feature matrix, construct positive and negative pairs; where a positive pair represents multiple feature sample combinations that have the same collection time, collection object, and object's emotional state, and a negative pair represents multiple feature sample combinations that have at least one different collection time, collection object, and object's emotional state. Use the initial encoder as the training encoder; The initial encoder for each modality is trained using positive and negative pairs of each modality, and the contrastive loss of the trained encoder for each modality is calculated. The contrastive loss of each modality training encoder is weighted and summed to obtain the total contrastive loss; Update the intermediate parameters of the encoder for each modality, and return to the step of training the initial encoder for each modality using the positive and negative pairs of each modality until the total contrastive loss converges. Once the total contrastive loss converges, the alignment features of each modality are obtained through the final trained encoder.

5. The emotion recognition method based on self-supervised and multimodal comparison according to claim 4, characterized in that, The process of performing generative self-supervised pre-training on the feature matrix of each modality includes the following steps: The feature matrix of each modality is randomly masked, and the masked feature matrix is ​​used as the encoder training data. The encoder training data is input into the encoder, and the encoder processes the data to obtain the latent representation of each mode. The feature matrix of each modality is reconstructed using a decoder based on the latent representation and positional encoding of each modality; The reconstruction loss for each mode is calculated using the mean squared error function; The reconstruction loss of each mode is summed to calculate the total reconstruction loss; The reconstructed feature matrix is ​​used as the new encoder training data for each modality. The process of inputting the encoder training data into the encoder and processing it to obtain the latent representation of each modality continues until the total reconstruction loss is minimized. Once the total reconstruction loss is minimized, the encoder for each mode is used as the output of the initial encoder.

6. The emotion recognition method based on self-supervised and multimodal comparison according to claim 1, characterized in that, The step of fusing multimodal features of the alignment features of each modality output by the contrastive learning multimodal encoder specifically includes the following steps: Alignment features of each modality are mapped to the same dimension to obtain projection vectors of multiple modalities; The projection vectors are modally fused pairwise to obtain multiple fused feature vectors; Global feature fusion is performed on multiple fusion feature vectors to obtain multimodal feature vectors.

7. The emotion recognition method based on self-supervised and multimodal comparison according to claim 6, characterized in that, In the step of performing pairwise modal fusion on the projection vectors, modal fusion of EEG modality and eye-tracking modality is completed by PSO fusion; modal fusion of eye-tracking modality and micro-expression modality is completed by LSTM fusion; and modal fusion of EEG modality and micro-expression modality is completed by cross-attention fusion. In the step of performing global feature fusion on multiple fusion feature directions, global feature fusion is completed through Transformer fusion.

8. The emotion recognition method based on self-supervised and multimodal comparison according to claim 1, characterized in that, The emotion recognition based on the multimodal feature vector specifically includes the following steps: The multimodal feature vectors are subjected to average pooling to obtain pooled feature vectors; The pooling feature vector is mapped to the emotion category space, and then linearly transformed through a fully connected layer to obtain multiple category scores. The category scores are converted into an emotion probability distribution using Softmax classification, and the emotion category and emotion intensity are output as the emotion recognition result based on the emotion probability distribution.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement an emotion recognition method based on self-supervised and multimodal comparison as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program, which is executed by a processor to implement an emotion recognition method based on self-supervision and multimodal comparison as described in any one of claims 1-8.

Images