A method for emotion recognition based on EEG and synchronized physiological signals based on cross-modal contrastive learning and multi-scale representation

Abstract

This invention relates to the field of emotion recognition technology and discloses an emotion recognition method based on cross-modal contrastive learning and multi-scale representation using EEG and synchronous physiological signals. In the pre-training stage, physiological signals from different modalities are used to extract features through encoders of the corresponding modalities to learn modal representations related to emotional stimuli. These modal representations are then mapped to a latent space through a mapping layer to calculate similarity, and subsequently, the contrastive loss for each modality is calculated for intra-modal contrastive learning. The encoder-extracted representations based on each modality undergo cross-modal contrastive learning, aligning the representations of different modalities to a shared latent space. Furthermore, this invention utilizes short-term and long-term emotional changes in each modality to pre-train encoders at different time scales. In the fine-tuning stage, the pre-trained encoder, an initialized modality fusion module, and a classifier are used for emotion recognition training, and the model weights are optimized based on the emotion labels of the physiological signals.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, specifically to an emotion recognition method based on cross-modal contrastive learning and multi-scale representation of EEG and synchronous physiological signals. Background Technology

[0002] Emotion recognition technology is gaining increasing attention due to its ability to provide crucial support for perception and decision-making in human-computer interaction (HCI) systems. Emotions are psychophysiological processes triggered by multiple factors, and their complexity and importance cannot be ignored. Researchers in psychology, neuroscience, and computer science have been dedicated to exploring the mechanisms and methods of emotion recognition for many years, but many challenges in detecting and analyzing human emotions remain largely unresolved and understudied.

[0003] Extensive research has demonstrated that electroencephalography (EEG) provides a more direct and objective measurement of human emotional responses. Unlike recorded behavioral signals (such as facial expressions, body posture, and voice) and peripheral physiological signals such as electrocardiogram (ECG), eye movement signals, skin conductance response (GSR), and electromyography (EMG), the EEG signals generated by emotional responses cannot be easily suppressed or faked. Furthermore, EEG signals offer advantages such as non-invasiveness, low cost, and high temporal resolution. While EEG possesses these advantages, the signals themselves are noisy, frequently affected by noise and physiological artifacts (such as eye movements and muscle activity), and exhibit significant individual variability. Fortunately, research has shown that the physiological signals from different modalities generated by the same emotional stimulus are complementary in information; that is, different physiological signals may contain both common and complementary attributes when representing emotional states. The former refers to the consistency in the characteristics of different physiological signals in representing emotional states, while the latter indicates that the characteristics of different physiological signals can represent different aspects of emotional states. To leverage this characteristic, previous methods have used EEG signals as a basis and other peripheral physiological signals to enhance EEG data, thereby improving the robustness and discriminative ability of emotion recognition decisions from different perspectives. For example, restricted Boltzmann machines (RBMs) are used to capture higher-order dependencies between EEG and peripheral physiological features. Another approach employs a cross-modal guided neural network that simultaneously extracts features from both EEG and eye-tracking signals, using EEG features to guide the extraction of eye-tracking features and mitigating the influence of subjective factors. Similarly, attention mechanisms are used to align modalities in the temporal domain, capturing accurate multimodal feature representations.

[0004] However, while these studies incorporate physiological signals from other modalities, they often overlook the consistency between multimodal physiological signals. Intuitively, physiological signals generated by the same emotional stimulus in different ways should exhibit correlation, i.e., physiological synchrony. Emotions typically affect multiple physiological systems, such as the brain, heart, and skin conductance, which are interconnected rather than isolated. Physiological synchrony can capture the relationships between these systems, thus providing a more comprehensive representation of emotional states. For example, convolutional neural networks (CNNs) are used to learn intersubjective aligned spatiotemporal representations from EEG time series through contrastive learning, achieving physiological synchrony among subjects. However, this method typically performs contrastive learning at a single temporal resolution; in reality, emotional fluctuations are not only temporally consistent but also exhibit short-term and long-term correlations. Other studies employ cross-modal Transformer fusion models, aiming to eliminate inconsistencies between different physiological signals and reduce information redundancy to some extent, but they do not completely solve the problem of cross-modal alignment or physiological synchrony.

[0005] In addition, various modality fusion methods have been proposed for multimodal physiological signal fusion, including feature-level fusion, decision-level fusion, adaptive weighting, attention mechanisms, and cross-attention. Although multimodal fusion usually improves emotion recognition by complementing information between modalities, inconsistencies in decisions between different modalities may affect the fusion results. Therefore, accurately evaluating the prediction reliability of each modality is also crucial for effectively combining information from multiple modalities. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an EEG and synchronous physiological signal emotion recognition method. This method is based on intramodal contrastive learning, cross-modal contrastive learning, and multi-scale representation. It effectively solves the problem of individual differences in feature extraction among different subjects, extracts the temporal information of physiological signals at different time scales, achieves cross-modal alignment and physiological synchronization, and effectively combines information from multiple modalities.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] An emotion recognition method based on cross-modal contrastive learning and multi-scale representation of EEG and synchronous physiological signals, including a pre-training stage and a fine-tuning stage;

[0009] Pre-training phase: Physiological signals from different modalities at a specific time scale are used to extract features through encoders of the corresponding modalities to learn modal representations related to emotional stimuli. These modal representations are then mapped to a latent space through a mapping layer to calculate similarity, and the contrastive loss for each modality is calculated. Intramodal contrastive learning is then performed to ensure that the physiological signals of different modalities under the same emotional stimulus from different subjects are similar, while the physiological signals of different modalities under different emotional stimuli remain different. The modal-based representations extracted by the encoders will also undergo cross-modal contrastive learning to align the representations of different modalities to a shared latent space. By optimizing the contrastive loss of each modality and the cross-modal contrastive learning loss, encoders corresponding to each modality at a specific time scale are pre-trained.

[0010] In the fine-tuning phase, emotion recognition training is performed using a pre-trained encoder, an initialized modality fusion module, and a classifier. The model weights are optimized based on the emotion labels of physiological signals to obtain an emotion recognition model that can output the emotion categories corresponding to different modal physiological signals.

[0011] Furthermore, the different modalities of physiological signals include electroencephalogram (EEG) signals and peripheral physiological signals.

[0012] Furthermore, during the comparative learning process, the physiological signals of different modalities of the subjects are segmented in chronological order to obtain physiological signal segments. Physiological signal segments generated by the same video stimulus form a positive sample pair, and physiological signal segments generated by different video stimuli form a negative sample pair.

[0013] Furthermore, it also includes the process of expanding physiological signal data through scaling and noise addition operations; the scaling operation generates original data variants without changing the time pattern of physiological signals by applying different scaling factors; at the same time, Gaussian noise with zero mean and fixed signal-to-noise ratio is added to the original physiological signal to simulate environmental noise.

[0014] Furthermore, the contrastive learning strategy in the pre-training phase enables encoders of each modality to identify whether different physiological signals correspond to the same emotional stimulus, thereby achieving physiological synchronization within the modality and physiological alignment across modalities.

[0015] Furthermore, the realization of intramodal physiological synchronization specifically includes:

[0016] Using mini-batch physiological signals as input to the model, physiological signals from subjects A and B are selected from a multimodal physiological signal dataset and obtained through sampling and expansion. Where m represents electroencephalogram (EEG) signal or peripheral physiological signal; small batches of physiological signals Inputting z parallel linear layers yields the feature vector E. iPhysiological signals are input into a single linear layer and a sigmoid function to obtain activation feature vectors. E i and Element-wise multiplication is performed, and aggregation is performed on the z-dimensional embedding to obtain the embedding vector e = (e1, ..., e2). z ):

[0017]

[0018] Where σ represents the sigmoid function, ⊙ represents the Hadamard product, and e i Let be the i-th element in e; BN represents batch normalization, and ReLU represents the activation function.

[0019] An embedding vector e is preceded by a learnable class label E. cls And embed the learned location information into E pos and modal information embedding E mod Adding it to e yields a labeled embedding vector.

[0020]

[0021] Embedded vector This is transformed into a query Q, key K, and value V through three linear layers:

[0022]

[0023] Where, d e This is the dimension of the embedding vector; h heads are used for self-attention, and the representation of each head is denoted as H = Attention(Q,K,V); the output of multi-head attention is Concat(H1,H2,…,H). h The data is processed by a feedforward neural network, where W is the weight matrix. This process is repeated twice to obtain features related to individual physiological stimuli.

[0024] A non-linear mapping layer is used after the encoder; the mapping layer consists of three linear layers, a non-linear activation function ReLU, a Dropout layer, and a batch normalization layer. The mapping layer is used to process the features extracted by the encoder.

[0025]

[0026] Where W1, W2, and W3 are the weight matrices of the three linear layers, and b1, b2, and b3 are the bias vectors. The three linear layers represent the output characteristics respectively; i represents the number of physiological signal segments, S represents the subject, S∈{A,B}, A represents subject A, B represents subject B, and m represents EEG signal or peripheral physiological signal.

[0027] Intramodal temporal contrastive learning is performed, with the contrastive loss for each modality calculated separately; physiological signals are transformed into a latent representation G after passing through the basic encoder and mapping layer. latent ={z i,S |i=1,2,…,5K;S∈{A,B}}, where 5K is the number of samples in the training dataset; then, the similarity between samples is calculated using the following formula, and then the contrastive loss within each modality is calculated:

[0028]

[0029] Wherein, indicator function I [j≠i] It equals 1 when j≠i, and 0 in other cases;

[0030]

[0031] By minimizing the loss function L m This increases the similarity of positive sample pairs while decreasing the similarity of negative sample pairs.

[0032] Furthermore, the achievement of cross-modal physiological alignment specifically includes:

[0033] Physiological signal segments induced by the same stimulus in different modalities are considered positive pairs, while physiological signal segments produced by different stimuli are considered negative pairs.

[0034] Features extracted by the encoder First, it is passed through a linear layer to obtain... Then, following the calculation process of equations (14) to (16), the cross-modal contrast loss L is obtained. cc ;

[0035] During the pre-training phase, the total loss L is:

[0036] L=α·L eeg +β·L m1 +γ·L cc ;

[0037] Among them, L eeg For contrast loss within EEG modalities, L m1 As a contrast loss within a peripheral physiological signal m1, L cc This is the contrast loss across modalities.

[0038] Furthermore, in the fine-tuning phase, emotion recognition training is performed using a pre-trained encoder, an initialized modality fusion module, and a classifier. The model weights are optimized based on emotion labels derived from physiological signals. Specifically, this includes:

[0039] During the pre-training phase, encoders for different durations are obtained by pre-training physiological signal segments of different lengths.

[0040] During the fine-tuning phase, physiological signal segments of different durations and modalities are input into the corresponding encoders. Then, the long-term and short-term features of each modality are fused to obtain the final features of each modality, which are then input into the modality fusion module. The final features from the two modalities are integrated using the maximum class probability, and the confidence of the modality is calculated and appropriate weights are assigned.

[0041]

[0042] Where w represents the set of network parameters, The first step is to predict the emotion category based on physiological signals. Next, features from the two modalities are concatenated, and finally, these combined features are passed to a classifier for final classification, using cross-entropy loss. Optimize:

[0043]

[0044] Among them, P(H joint ) represents the probability that the joint feature vector belongs to a certain category, and y is the emotion label of the physiological signal.

[0045] Compared with the prior art, the beneficial technical effects of the present invention are:

[0046] 1. This invention proposes a self-supervised contrastive learning framework for emotion-stimuli alignment. It utilizes temporal contrastive learning within each modality to minimize the differences in physiological signals generated by different subjects under the same emotional stimulus due to environmental, personality, and other factors, while reducing reliance on emotion labels. Furthermore, this invention incorporates cross-modal contrastive learning, enhancing intermodal interaction while bringing similar samples from different modalities closer together and distancing irrelevant samples, thus promoting the learning of a shared semantic space for representations across different modalities.

[0047] 2. In addition to using EEG modalities, this invention also simultaneously incorporates peripheral physiological signals as a supplement to current emotional information, obtaining more complete emotional information and achieving better recognition results. Furthermore, data augmentation is performed on physiological signals from different modalities through scaling, noise addition, and combination methods. This increases the amount of pre-training data, introduces more positive correlations, enhances the effect of contrastive learning, and improves the model's generalization ability.

[0048] 3. This invention pre-trains the representations of EEG and peripheral physiological signals across different time windows (e.g., 1-second and 5-second windows), effectively simulating short-term and long-term emotional dynamics. During fine-tuning, fusing features across temporal resolution and patterns significantly improves the accuracy of emotion recognition.

[0049] 4. To mitigate the uncertainty in the modality fusion process, this invention designs a modality fusion module that uses maximum class probability (MCP) to predict the confidence level of physiological signals from different modalities and assign corresponding weights. This ensures that in multimodal emotion recognition tasks, the contribution of each modality to the final prediction is dynamic and adjustable, flexibly adjusting the information transmission in the modality fusion process and ensuring more reliable modality fusion. Attached Figure Description

[0050] Figure 1 This is a general structural block diagram of the present invention;

[0051] Figure 2 This is a schematic diagram of the encoder and mapping layer of the present invention;

[0052] Figure 3 This is a schematic diagram of the modal fusion module of the present invention. Detailed Implementation

[0053] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0054] like Figure 1As shown, the framework proposed in this invention consists of two stages: contrastive learning pre-training and emotion recognition fine-tuning. In the pre-training stage, data augmentation techniques (such as time scaling, translation, and noise addition) are first used to enhance physiological signals from different modalities, providing the model with more trainable data and introducing more positive samples for contrastive learning, thus improving the model's robustness and generalization. Then, by extracting features from physiological signals at different time scales using corresponding encoders, modal representations related to emotional stimuli are learned. These representations are then mapped to a latent space through a mapping layer (projector) to calculate similarity, thereby calculating the contrastive loss for each modality. This ensures that the physiological signal representations of different subjects under the same emotional stimulus are similar, while maintaining differences under different emotional stimuli. Furthermore, the representations extracted by the encoders of each modality are used for cross-modal contrastive learning, increasing interaction between different modalities and aligning the representations of different modalities to a shared latent space, reducing heterogeneity between modalities. Finally, by optimizing the two contrastive losses, the encoders for each modality are pre-trained. During the fine-tuning phase, the pre-trained encoder, the initialization modal fusion module, and the classifier are used to train the emotion recognition system. The model weights are optimized based on the emotion labels of the labeled data to further improve the accuracy of emotion classification.

[0055] The following are the specific procedures and model details:

[0056] The contrastive learning strategy in the pre-training phase aims to enable encoders of each modality in the model to accurately identify whether different physiological signal segments correspond to the same emotional stimulus, while simultaneously achieving intra-modal and inter-modal alignment and physiological synchronization. Specifically, this invention first samples a small batch as the model input. In the multimodal physiological signal dataset, taking the EEG signals of subjects A and B as an example, the EEG signals obtained from each experiment of subject A are first divided into segments of t seconds according to time sequence. Assuming a total of N segments of EEG signals are obtained (t is 1 second or 5 seconds, aiming to pre-train encoders for different time windows), these segments are labeled as follows: Where C represents the number of channels in the EEG signal and S represents the number of sampling points. For subject B, a similar processing method was used, and the resulting EEG signal segment is represented as follows: EEG signal segments generated from the same video stimulus and Forming a positive sample pair, in contrast, EEG signal segments generated by different video stimuli and (Where j = 1, 2, 3, ..., N, j ≠ i, S ∈ {A, B}) forms a negative sample pair. Similar segmentation is performed for all subjects. To obtain a mini-batch, the sampler randomly samples EEG signal segments corresponding to K video stimuli from 2 subjects each time. This forms a mini-batch. This serves as the input for pre-training. In this mini-batch, there are a total of K(2K-1) pairs of EEG signal segments, of which K pairs are positive samples. Similar processing methods are used to obtain mini-batches for other physiological signal modalities. Where 'm' represents either an electroencephalogram (EEG) or a peripheral physiological signal.

[0057] Publicly available physiological signal datasets are typically limited in terms of the number of participants and the amount of data, which may restrict the training effect of the model. To address this issue, in a preferred embodiment of the present invention, a data augmentation technique is provided to improve the generalization ability of the model by artificially expanding the dataset. Furthermore, data augmentation can add more positive samples, thereby enhancing the effect of contrastive learning. The present invention employs two common data augmentation methods: scaling and noise addition. Specifically, the scaling operation generates original data variants without altering the signal temporal pattern by applying different scaling factors; simultaneously, the present invention adds Gaussian noise with zero mean and a fixed signal-to-noise ratio to the original physiological signal to simulate the effects of environmental noise, etc. Finally, these two augmentation methods can be combined to further improve the diversity of training data and the robustness of the model.

[0058] Scaling (G) m )=α·G m (1)

[0059]

[0060] Where α (obtained by uniform sampling at (0.7, 0.8) and (1.1, 1.2)) is the scaling factor. This indicates that the mean is zero and the variance is σ. 2 Gaussian noise.

[0061] The enhanced data can be represented as:

[0062]

[0063] The data obtained will be enhanced next. After passing through the encoder, the encoder has the following components. For example... Figure 2 As shown, the physiological signal is first passed through z parallel linear layers, encouraging the model to focus on different feature views. Another linear layer is followed by an activation function that gates the embedding with information useful for emotion recognition. Subsequently, E...i and Element-wise multiplication and aggregation over the z-dimensional embeddings yields e. i =(e1,…,e z The formula is as follows:

[0064]

[0065] Here, σ represents the sigmoid function, which restricts the output value between 0 and 1, ⊙ represents the Hadamard product (i.e., element-wise product), BN represents batch normalization, and ReLU represents the activation function. In this way, the input features are transformed into sequences from different perspectives, which can then be further processed by subsequent layers.

[0066] Before being passed to the next layer, a learnable class label E is added before the embedding vector e. cls This tag is used to aggregate information from the entire sequence. To integrate positional and modal information, the learned positional information is embedded in E. pos and modal information embedding E mod It will be added to the input embedding, represented as:

[0067]

[0068] Then After passing through a multi-head self-attention layer, the core component of the Transformer is the multi-head self-attention mechanism. (Embedded vectors) The query is transformed into a key Q and a value V through three linear layers, and the calculation formula is as follows:

[0069]

[0070] Where, d e It is the dimension of the embedded vector.

[0071] Then, this invention uses h heads for self-attention, with each head represented as H = Attention(Q,K,V). The output of the multi-head attention is Concat(H1,H2,…,H). h The encoder process involves a weight matrix W, which is then further processed by a feedforward neural network (FFN). This process is repeated twice to obtain the features associated with the individual physiological stimulus. The entire encoder process can be summarized as follows:

[0072]

[0073] Inspired by the SimCLR framework, a nonlinear mapping layer is used between the base encoder and the final contrastive loss. For example... Figure 2As shown, the mapping layer plays a crucial role in mapping features to a new feature space, which better captures the underlying structure of the data and helps optimize the contrastive learning objective. In the module of this invention, the mapping layer consists of multiple linear layers, the non-linear activation function ReLU, and Dropout and BN layers, used to process the features extracted by the encoder, as follows:

[0074]

[0075] Where W1, W2, and W3 are the weight matrices of the three linear layers, and b1, b2, and b3 are the bias vectors. These represent the output characteristics of each layer. Here, i represents the number of physiological signal segments, S represents the subject, and m represents the electroencephalogram or peripheral physiological signal.

[0076] Figure 2 The encoder is on the left, and the mapping layer is on the right.

[0077] First, intra-modal temporal contrastive learning is performed, with the contrastive loss for each modality calculated separately. After passing through the basic encoder and mapping layer, the physiological signal is transformed into a latent representation G. latent ={z i,S |i=1,2,…,5K;S∈{A,B}}. Then, the similarity between samples is calculated using the following formula, and subsequently, the contrastive loss within each modality is calculated:

[0078]

[0079] Wherein, indicator function I [j≠i] The loss function equals 1 when j ≠ i, and 0 otherwise. By minimizing this loss function, the model increases the similarity of positive sample pairs while decreasing the similarity of negative sample pairs. The total loss for the entire mini-batch can be expressed as:

[0080]

[0081] The above process calculates intramodal temporal contrastive learning loss to achieve physiological synchronization among different subjects. Furthermore, to enhance intermodal interactions, this invention performs cross-modal contrastive learning, aligning physiological signals from different modalities into a shared semantic space, thereby promoting physiological synchronization between different physiological signals. During cross-modal contrastive learning, physiological signal fragments from different modalities induced by the same stimulus are considered positive pairs, while physiological signal fragments generated by different stimuli are considered negative pairs. Specifically, the calculation process involves using features extracted by the encoder... First, it is passed through a linear layer to obtain... Then, following the calculation process of equations (14) to (16), the cross-modal contrast loss L is obtained.cc During the pre-training phase, the total loss L can be expressed as:

[0082] L=α·L eeg +β·L m1 +γ·L cc (17)

[0083] Where L eeg For contrast loss within EEG modalities, and L m1 For another type of contrast loss within peripheral physiological signal m1, L cc This is the contrast loss across modalities.

[0084] like Figure 1 As shown, in the fine-tuning stage, this invention further fine-tunes the pre-trained base encoder using labeled samples. During pre-training, this invention pre-trains the representations of EEG and peripheral signals in different time windows (e.g., 1-second and 5-second windows), thereby pre-training the encoder for each respective time window. This method effectively simulates short-term and long-term emotional dynamics. Specifically, this invention decomposes the physiological signal segments of the long-term window into 1-second intervals and passes them through their respective pre-trained encoders. This allows the extraction of long-term and short-term temporal features for each modality, which are then fused together. By combining these long-term and short-term features, this invention captures the dynamic changes in emotional states, providing a comprehensive perspective for emotion recognition. In actual experiments, this invention uses t=5 seconds as the long-term window and divides it into five 1-second segments. After passing these segments through their respective encoders, the temporal features are obtained. Finally, the long-term and short-term temporal features of each modality are fused to generate the final feature representation H. e and H m1 Then the results are input into the modal fusion module.

[0085] In the modality fusion module, this invention integrates features from two modalities by using Maximum Class Probability (MCP), calculating the confidence level of each modality, and assigning appropriate weights. The formula is as follows:

[0086]

[0087] Where w represents the set of network parameters, This is the predicted category for the sample.

[0088] Next, the features from the two modalities are concatenated, and finally, these combined features are passed to the classifier for final classification, optimized using the cross-entropy loss function. The formula for cross-entropy loss is as follows:

[0089]

[0090] Among them, P(H joint ) represents the probability that the joint feature vector belongs to a certain category, and y is the label of the sample.

[0091] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0092] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for emotion recognition based on cross-modal contrastive learning and multi-scale representation of EEG and synchronous physiological signals, characterized in that, Includes a pre-training phase and a fine-tuning phase; Pre-training phase: Physiological signals from different modalities at a specific time scale are used to extract features through encoders of the corresponding modalities to learn modal representations related to emotional stimuli. These modal representations are then mapped to a latent space through a mapping layer to calculate similarity, and the contrastive loss for each modality is calculated. Intramodal contrastive learning is then performed to ensure that the physiological signals of different modalities under the same emotional stimulus from different subjects are similar, while the physiological signals of different modalities under different emotional stimuli remain different. The modal-based representations extracted by the encoders will also undergo cross-modal contrastive learning to align the representations of different modalities to a shared latent space. By optimizing the contrastive loss of each modality and the cross-modal contrastive learning loss, encoders corresponding to each modality at a specific time scale are pre-trained. In the fine-tuning phase, emotion recognition training is performed using a pre-trained encoder, an initialized modality fusion module, and a classifier. Model weights are optimized based on emotion labels derived from physiological signals to obtain an emotion recognition model capable of outputting emotion categories corresponding to different modal physiological signals. Specifically, this includes: In the pre-training phase, encoders for different durations of physiological signal segments are pre-trained. In the fine-tuning phase, physiological signal segments of different durations and modalities are input into the corresponding encoders. Then, the long-term and short-term features of each modality are fused to obtain the final features of each modality, which are then input into the modality fusion module. The final features from the two modalities are integrated using the maximum class probability, and the confidence of the modality is calculated and appropriate weights are assigned. ;in, Represents the set of network parameters. The first step is to predict the emotion category based on physiological signals. Next, features from the two modalities are concatenated, and finally, these combined features are passed to a classifier for final classification, using cross-entropy loss. Optimize: ;in, This represents the probability that the joint feature vector belongs to a certain category. These are emotional labels for physiological signals; The different modalities of physiological signals include electroencephalogram (EEG) signals and peripheral physiological signals; The contrastive learning strategy in the pre-training phase is to enable encoders of different modalities to identify whether different physiological signals correspond to the same emotional stimulus, thereby achieving physiological synchronization within the modality and physiological alignment across modalities.

2. The method according to claim 1, characterized in that, During the comparative learning process, the physiological signals of different modalities of the subjects are segmented in chronological order to obtain physiological signal segments. Physiological signal segments generated by the same video stimulus form a positive sample pair, and physiological signal segments generated by different video stimuli form a negative sample pair.

3. The method according to claim 1, characterized in that, It also includes the process of expanding physiological signal data through scaling and noise addition operations; the scaling operation generates original data variants without changing the time pattern of physiological signals by applying different scaling factors; at the same time, Gaussian noise with zero mean and fixed signal-to-noise ratio is added to the original physiological signal to simulate environmental noise.

4. The method according to claim 1, characterized in that, The achievement of intramodal physiological synchronization specifically includes: Using a small batch of physiological signals as input to the model, subjects were selected from a multimodal physiological signal dataset. and The physiological signals are sampled and expanded to obtain small batches of physiological signals. Where m represents the electroencephalogram (EEG) signal or peripheral physiological signal; small batches of physiological signals Input z parallel linear layers to obtain feature vectors Physiological signals are input into a single linear layer and a sigmoid function to obtain activation feature vectors. ;Will and Multiply by elements, and Aggregation is performed on the dimensional embeddings to obtain the embedding vector. : ； ； ； in, This represents the sigmoid function. Represents the Hadama product. for The i-th element in the array; BN represents batch normalization, and ReLU represents the activation function; Embedded vector A learnable class tag has been added in front. And embed the learned location information and modal information embedding Add to In this process, we obtain labeled embedding vectors. : ； Embedded vector This is transformed into a query Q, key K, and value V through three linear layers: ； in, It is the dimension of the embedding vector; self-attention is performed using h heads, and the representation of each head is denoted as h. The output of multi-head attention is Where W is the weight matrix; subsequently, it is further processed by a feedforward neural network, and the above process is repeated twice to finally obtain the features related to individual physiological signals. ; A non-linear mapping layer is used after the encoder; the mapping layer consists of three linear layers, a non-linear activation function ReLU, a Dropout layer, and a batch normalization layer. The mapping layer is used to process the features extracted by the encoder. ； ； ； in, , , It is the weight matrix of a three-layer linear layer. , , It is a bias vector. , , These represent the output features of the three linear layers respectively; Indicates the number of physiological signal segments. Indicates the subject, , Subject A, Indicates subject B, This indicates electroencephalogram (EEG) signals or peripheral physiological signals; Intramodal temporal contrastive learning is performed, with the contrastive loss for each modality calculated separately; physiological signals are transformed into latent representations after passing through the basic encoder and mapping layer. , The number of samples in the training dataset is given; then, the similarity between samples is calculated using the following formula, and then the contrastive loss within each modality is calculated: ；（14） ；（15） Among them, indicator function exist The value is 1 in the case of time, and 0 in other cases. ；（16） By minimizing the loss function This increases the similarity of positive sample pairs while decreasing the similarity of negative sample pairs.

5. The method according to claim 4, characterized in that, The achievement of cross-modal physiological alignment specifically includes: Physiological signal segments induced by the same stimulus in different modalities are considered positive pairs, while physiological signal segments produced by different stimuli are considered negative pairs. Features extracted by the encoder First, it is passed through a linear layer to obtain... Then, following the calculation process of equations (14) to (16), the cross-modal contrast loss is obtained. ; During the pre-training phase, the total loss for: ； in, For contrast loss within EEG modalities, This represents a contrast loss within a peripheral physiological signal m1. This is the contrast loss across modalities.

Images