Electroencephalogram and synchronous physiological signal emotion recognition method based on cross-modal comparative learning and multi-scale representation

Abstract

The invention relates to the technical field of emotion recognition, and discloses an electroencephalogram and synchronous physiological signal emotion recognition method based on cross-modal comparative learning and multi-scale characterization, in a pre-training stage, features are extracted from physiological signals of different modalities through encoders corresponding to the modalities, and modal characterization related to emotion stimulation is learned; the modal representations are mapped to a potential space through a mapping layer to calculate the similarity, then the contrast loss of each modal is calculated, and contrast learning in the modal is carried out; performing cross-modal comparative learning on the representation based on each modal extracted by an encoder, and aligning the representation of different modals to a shared potential space; according to the invention, the encoder is pre-trained at different time scales by using short-term and long-term emotion changes in each mode; in the fine tuning stage, emotion recognition training is carried out by using a pre-trained encoder, an initialized modal fusion module and a classifier, and the model weight is optimized based on the emotion label of the physiological signal.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to an emotion recognition method based on electroencephalogram and synchronous physiological signals based on cross-modal contrast learning and multi-scale representation. Background Art

[0002] Emotion recognition technology is gaining increasing attention because it can provide key support for perception and decision-making in human-computer interaction (HCI) systems. Emotion is a psychophysiological process triggered by multiple factors, and its complexity and importance cannot be ignored. Researchers in psychology, neuroscience, and computer science have been working on exploring the mechanisms and methods of emotion recognition for many years, but many challenges in detecting and analyzing human emotions remain largely unstudied and unsolved.

[0003] Extensive research has demonstrated that electroencephalography (EEG) provides a more direct and objective measurement of human emotional responses. Unlike recorded behavioral signals (such as facial expressions, body postures, sounds, etc.), and peripheral physiological signals such as electrocardiogram (ECG), eye movement signals, galvanic skin response (GSR), and electromyography (EMG), the EEG signals generated by such emotional responses cannot be easily suppressed or disguised, and EEG signals have the advantages of non-invasiveness, low cost, and high temporal resolution. Although EEG has the above advantages, the signal itself is noisy, often plagued by noise and physiological artifacts (such as eye movements and muscle activity), and shows obvious individual differences. Fortunately, research has shown that physiological signals of different modalities generated by the same emotional stimulus are complementary in information, that is, different physiological signals may contain common and complementary attributes when representing emotional states. The former means that the characteristics of different physiological signals are consistent in representing emotional states, while the latter indicates that the characteristics of different physiological signals can represent different aspects of emotional states. In order to take advantage of this feature, some previous methods have used EEG signals as a basis and combined them with other peripheral physiological signals to enhance EEG data, thereby enhancing the robustness and discrimination ability of emotion recognition decisions from different angles. For example, a restricted Boltzmann machine is used to capture the high-order dependency between EEG and peripheral physiological features. In addition, a cross-modal guided neural network is used, which extracts features of EEG and eye movement signals at the same time, and uses EEG features to guide the extraction of eye movement features to reduce the influence of subjective factors. Similarly, an attention mechanism is used to align modalities in the time domain to capture accurate multimodal feature representations.

[0004] However, although these studies combine physiological signals from other modalities, they often ignore the consistency between multimodal physiological signals. Intuitively, physiological signals of different modalities generated by the same emotional stimulus should show correlation, that is, physiological synchronization. Emotions usually affect multiple physiological systems, such as the brain, heart, and skin conductance, which are interconnected rather than isolated. Physiological synchronization can capture the relationship between these systems, thereby providing a more comprehensive representation of emotional states. For example, convolutional neural networks (CNNs) are applied to learn inter-subject aligned spatiotemporal representations from EEG time series through contrastive learning to achieve physiological synchronization between subjects. However, this method usually performs contrastive learning at a single time resolution. In fact, emotional fluctuations are not only temporally consistent, but also show short-term and long-term correlations. In addition, there are studies that use cross-modal Transformer fusion models to eliminate the inconsistency between different physiological signals and reduce information redundancy to a certain extent, but it does not completely solve the problem of cross-modal alignment or physiological synchronization.

[0005] In addition, when fusing multimodal physiological signals, people have also proposed various modal fusion methods, including feature-level fusion, decision-level fusion, adaptive weighting, attention mechanism, and cross-attention. Although multimodal fusion usually improves the effect of emotion recognition through information complementarity between modalities, the inconsistency of decisions between different modalities may affect the fusion results. In order to effectively combine information from multiple modalities, it is also very critical to accurately evaluate the prediction reliability of each modality. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a method for emotion recognition based on EEG and synchronized physiological signals. The method is based on intra-modal contrastive learning, cross-modal contrastive learning and multi-scale representation, which effectively solves the problem of individual differences in feature extraction between different subjects, extracts the time information of physiological signals at different time scales, realizes cross-modal alignment and physiological synchronization, and effectively combines information from multiple modalities.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A method for emotion recognition based on EEG and synchronous physiological signals based on cross-modal contrastive learning and multi-scale representation, including a pre-training stage and a fine-tuning stage;

[0009] Pre-training stage: The physiological signals of different modalities at a specific time scale are extracted with features through the encoder of the corresponding modality, and the modal representation related to the emotional stimulus is learned. The modal representation is then mapped to the latent space through the mapping layer to calculate the similarity, and then the contrast loss of each modality is calculated to perform intra-modal contrast learning, so that the physiological signals of different modalities of different subjects under the same emotional stimulus are close, while the physiological signals of different modalities under different emotional stimuli remain different; the representations extracted by the encoder based on each modality will also be subjected to cross-modal contrast learning to align the representations of different modalities to the shared latent space; by optimizing the contrast loss of each modality and the cross-modal contrast learning loss, the encoders corresponding to each modality at a specific time scale are pre-trained;

[0010] In the fine-tuning stage, the pre-trained encoder, initialized modal fusion module and classifier are used for emotion recognition training, and the model weights are optimized based on the emotion labels of physiological signals to obtain an emotion recognition model that can output emotion categories corresponding to physiological signals of different modalities.

[0011] Furthermore, the physiological signals of different modalities include electroencephalogram signals and peripheral physiological signals.

[0012] Furthermore, during the comparative learning process, the physiological signals of different modalities of the subjects are segmented in chronological order to obtain physiological signal segments. The physiological signal segments generated by the same video stimulation form a positive sample pair, and the physiological signal segments generated by different video stimulation form a negative sample pair.

[0013] Furthermore, it also includes a process of expanding physiological signal data through scaling operations and noise addition operations; the scaling operation generates original data variants that do not change the temporal pattern of the physiological signal by applying different scaling factors; at the same time, Gaussian noise with zero mean and fixed signal-to-noise ratio is added to the original physiological signal to simulate environmental noise.

[0014] Furthermore, the contrastive learning strategy in the pre-training stage is to let the encoders of each modality identify whether different physiological signals correspond to the same emotional stimulus, achieve physiological synchronization within the modality, and achieve physiological alignment across modalities.

[0015] Furthermore, the achieving intra-modality physiological synchronization specifically includes:

[0016] Taking small batches of physiological signals as the input of the model, the physiological signals of subjects A and B are selected from the multimodal physiological signal dataset, and small batches of physiological signals are obtained through sampling and expansion. Where m represents EEG signal or peripheral physiological signal; small batch physiological signal Input z parallel linear layers to get the feature vector E i; Input the physiological signal into a single linear layer and sigmoid function to obtain the activation feature vector E i and Multiply element-wise and aggregate over the z-dimensional embedding to obtain the embedding vector e = (e1,…,e z ):

[0017]

[0018] Among them, σ represents the sigmoid function, ⊙ represents the Hadamard product, e i is the i-th element in e; BN stands for batch normalization, and RELU represents the activation function;

[0019] The embedding vector e is preceded by a learnable class label E cls , and embed the learned location information into E pos and modal information embedding E mod Add to e to get the labeled embedding vector

[0020]

[0021] Embedding vector Transformed into query Q, key K and value V through three linear layers:

[0022]

[0023] Among them, d e is the dimension of the embedding vector; h heads are used for self-attention, and the representation of each head is recorded as H = Attention(Q, K, V); the output of multi-head attention is Concat(H1, H2, ..., H h )W, where W is the weight matrix; then, it is further processed by the feedforward neural network and the above process is repeated twice to finally obtain the features related to the individual stimulation of the physiological signal

[0024] A nonlinear mapping layer is used after the encoder; the mapping layer consists of three linear layers, a nonlinear activation function ReLU, a Dropout layer, and a batch normalization layer. The mapping layer is used to process the features extracted by the encoder:

[0025]

[0026] Among them, W1, W2, W3 are the weight matrices of the three linear layers, b1, b2, b3 are the bias vectors, Represent the output features of the three linear layers respectively; i represents the number of physiological signal segments, S represents the subject, S∈{A,B}, A represents subject A, B represents subject B, and m represents EEG signal or peripheral physiological signal;

[0027] Temporal contrast learning is performed within the modality, and the contrast loss of each modality is calculated separately; after passing through the basic encoder and mapping layer, the physiological signal is converted into a potential representation G latent ={z i,S |i=1,2,…,5K; S∈{A,B}}, 5K is the number of samples in the training dataset; then, the similarity between samples is calculated using the following formula, and then the contrast loss within each modality is calculated:

[0028]

[0029] Among them, the indicator function I [j≠i] It is equal to 1 when j≠i, and 0 otherwise;

[0030]

[0031] By minimizing the loss function L m To increase the similarity of positive sample pairs and reduce the similarity of negative sample pairs.

[0032] Furthermore, the achieving cross-modal physiological alignment specifically includes:

[0033] Physiological signal segments of different modalities induced by the same stimulus were considered as positive pairs, while physiological signal segments produced by different stimuli were considered as negative pairs;

[0034] The features extracted by the encoder First pass it through a linear layer to get Then, according to the calculation process from equation (14) to equation (16), the cross-modal contrast loss L is obtained cc ;

[0035] During the pre-training phase, the total loss L is:

[0036] L=α·L eeg +β·L m1 +γ·L cc ;

[0037] Among them, L eeg is the contrast loss within the EEG modality, L m1 is a contrast loss within the peripheral physiological signal m1, L cc is the cross-modal contrast loss.

[0038] Furthermore, in the fine-tuning stage, the emotion recognition training is performed using the pre-trained encoder, the initialized modal fusion module and the classifier, and the model weights are optimized based on the emotion labels of the physiological signals, specifically including:

[0039] In the pre-training stage, encoders for different time lengths are pre-trained by using physiological signal segments of different time lengths;

[0040] In the fine-tuning stage, physiological signal segments of different time lengths and different modalities are input into the corresponding encoders, and then the long-term features and short-term features of each modality are fused to obtain the final features of each modality, which are input into the modality fusion module. The final features from the two modalities are integrated by using the maximum class probability, and the confidence of the modality is calculated and the appropriate weight is assigned:

[0041]

[0042] Among them, w represents the network parameter set, is the predicted emotion category of the physiological signal; next, the features from the two modalities are concatenated, and finally these combined features are passed to the classifier for final classification, and the cross entropy loss is used. To optimize:

[0043]

[0044] Among them, P(H joint ) represents the probability that the joint feature vector belongs to a certain category, and y is the emotion label of the physiological signal.

[0045] Compared with the prior art, the beneficial technical effects of the present invention are:

[0046] 1. The present invention proposes a self-supervised contrastive learning framework for emotion stimulus alignment, which uses temporal contrastive learning within each modality to minimize the differences in physiological signals generated by different subjects under the same emotional stimulus due to environmental, personality, etc., while reducing dependence on emotion labels. In addition, the present invention sets up cross-modal contrastive learning, which enhances the interaction between modalities while bringing similar samples from different modalities closer and pushing away irrelevant samples, promoting the representation learning of different modalities to a shared semantic space.

[0047] 2. In addition to using EEG modalities, the present invention also simultaneously introduces peripheral physiological signals as a supplement to current emotional information, obtains more complete emotional information, and achieves better recognition effects. At the same time, data enhancement is performed on physiological signals of different modalities by means of scale transformation, noise addition, and combination. While increasing the amount of pre-training data, more positive pairs are introduced to enhance the effect of contrastive learning and improve the generalization ability of the model.

[0048] 3. The present invention pre-trains the representation of EEG and peripheral physiological signals on different time windows (e.g., 1 second and 5 second windows), effectively simulating short-term and long-term emotional dynamics. During the fine-tuning process, the fusion of features across time resolutions and patterns significantly improves the accuracy of emotion recognition.

[0049] 4. In order to reduce the uncertainty in the modal fusion process, the present invention designs a modal fusion module, which uses the maximum class probability (MCP) to predict the confidence of physiological signals of different modalities and assign corresponding weights to ensure that in the multimodal emotion recognition task, the contribution of each modality to the final prediction is dynamic and adjustable, and the information transmission in the modal fusion process is flexibly adjusted to ensure more reliable modal fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is the overall structural diagram of the present invention;

[0051] Figure 2 It is a schematic diagram of the structure of the encoder and mapping layer of the present invention;

[0052] Figure 3 It is a structural schematic diagram of the modal fusion module of the present invention. DETAILED DESCRIPTION

[0053] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0054] like Figure 1As shown, the framework proposed by the present invention consists of two stages: contrastive learning pre-training and emotion recognition fine-tuning. In the pre-training stage, the physiological signals of different modalities are first enhanced through data enhancement techniques (such as time scaling, translation, noise addition, etc.) to provide more trainable data for the model, and at the same time introduce more positive samples for contrastive learning to improve the robustness and generalization of the model. Then, by extracting features from physiological signals of different time scales through corresponding encoders, the modal representations related to emotional stimuli are learned, and these representations are mapped to the potential space through the mapping layer (Projector) to calculate the similarity, and then the contrast loss of each modality is calculated, so that the physiological signal representations of different subjects under the same emotional stimulus are close, while the representations under different emotional stimuli remain different. In addition, the representations extracted by the encoders of each modality will also be used for cross-modal contrast learning, increase the interaction between different modalities, and align the representations of different modalities to the shared potential space to reduce the heterogeneity between modalities. Finally, by optimizing the two contrast losses, the encoders of each modality are pre-trained. In the fine-tuning stage, the pre-trained encoder, initialized Modal Fusion Module and classifier are used for emotion recognition training, and the model weights are optimized based on the emotion labels of the annotated data to further improve the accuracy of emotion classification.

[0055] The following are the specific process and model details:

[0056] The strategy of contrastive learning in the pre-training stage is to let the encoders of each modality in the model accurately identify whether different physiological signal segments correspond to the same emotional stimulus, and at the same time achieve intra-modal and inter-modal alignment and physiological synchronization. Specifically, the present invention first samples a small batch as the input of the model. In the multimodal physiological signal data set, taking the EEG signals of subjects A and B as an example, the EEG signals obtained from each experiment of subject A are first divided into t seconds in chronological order. Assuming that a total of N EEG signals are obtained (t is 1s or 5s, aiming to pre-train encoders of different time windows respectively), they are marked as Where C represents the number of channels of the EEG signal, and S represents the number of sampling points. For subject B, a similar processing method is used, and the obtained EEG signal segment is expressed as Segments of EEG signals generated from the same video stimulus and Form a positive sample pair, compare the EEG signal segments generated by different video stimuli and (where j = 1, 2, 3, ..., N, j ≠ i, S ∈ {A, B}) forms a negative sample pair. For all subjects, a similar segmentation process is performed. To obtain a mini-batch, the sampler randomly samples EEG signal segments corresponding to K video stimuli from 2 subjects each time. This forms a mini-batch As the input for pre-training. In this mini-batch, there are a total of K(2K-1) pairs of EEG signal segments, of which there are K pairs of positive sample pairs. For other physiological signal modalities, similar processing methods are used to obtain mini-batches of specific modalities. Where m represents electroencephalogram or peripheral physiological signal.

[0057] Public physiological signal datasets are usually limited in the number of subjects and the amount of data, which may impose certain restrictions on the training effect of the model. In order to solve this problem, in the preferred embodiment of the present invention, a data enhancement technology is also provided to improve the generalization ability of the model by artificially expanding the dataset. In addition, data enhancement can also add more positive samples, thereby enhancing the effect of contrastive learning. The present invention adopts two common data enhancement methods: scaling and noise addition. Specifically, the scaling operation generates variants of the original data that do not change the temporal pattern of the signal by applying different scaling factors; at the same time, the present invention also adds Gaussian noise with a mean of zero and a fixed signal-to-noise ratio to the original physiological signal to simulate the effects of environmental noise, etc. Finally, these two enhancement methods can be used in combination to further improve the diversity of training data and the robustness of the model:

[0058] Scaling(G m )=α·G m ; (1)

[0059]

[0060] where α (uniformly sampled at (0.7, 0.8) and (1.1, 1.2)) is the scaling factor, It means the mean is zero and the variance is σ 2 Gaussian noise.

[0061] The enhanced data can be expressed as:

[0062]

[0063] Next, we will enhance the data After the encoder, the encoder has the following components. Figure 2 As shown in Figure 1, the physiological signal first passes through z parallel linear layers to encourage the model to focus on different feature views. Another linear layer is followed by an activation function to gate the embedding with information useful for emotion recognition.i and Multiply element-wise and aggregate over the z-dimensional embedding to obtain e i =(e1,…,e z ), the formula is as follows:

[0064]

[0065] Among them, σ represents the sigmoid function, which limits the output value between 0 and 1, ⊙ represents the Hadamard product (i.e. element-wise product). BN stands for batch normalization, and RELU represents the activation function. In this way, the input features are transformed into sequences from different perspectives, which can be further processed by subsequent layers.

[0066] Before being passed to the next layer, the embedding vector e is prepended with a learnable class label E cls , which is used to aggregate information from the entire sequence. In order to integrate position information and modality information, the learned position information embedding E pos and modal information embedding E mod will be added to the input embedding, expressed as:

[0067]

[0068] Then After the multi-head self-attention layer, the core component of Transformer is the multi-head self-attention mechanism. Embedding vector It is converted into query Q, key K and value V through three linear layers, and its calculation formula is as follows:

[0069]

[0070] Among them, d e is the dimension of the embedding vector.

[0071] Then, the present invention uses h heads for self-attention, and each head is represented by H = Attention(Q, K, V). The output of multi-head attention is Concat(H1, H2, ..., H h )W, where W is the weight matrix. Subsequently, after further processing by the feedforward neural network (FFN), the above process is repeated twice, and finally the features related to the individual stimulus of the physiological signal are obtained. The entire encoder process can be briefly summarized as follows:

[0072]

[0073] Inspired by the SimCLR framework, a nonlinear mapping layer is used between the base encoder and the final contrastive loss. Figure 2As shown, the mapping layer plays an important role in mapping features to a new feature space, which better captures the potential structure of the data and helps optimize the contrastive learning objective. In the module of the present invention, the mapping layer is composed of multiple Linear layers, nonlinear activation function ReLU, and Dropout and BN layers, which are used to process the features extracted by the encoder, as follows:

[0074]

[0075] Among them, W1, W2, W3 are the weight matrices of the three linear layers, b1, b2, b3 are the bias vectors, Here, i represents the number of physiological signal segments, S represents the subject, and m represents the EEG or peripheral physiological signal.

[0076] Figure 2 The encoder is on the left and the mapping layer is on the right.

[0077] First, temporal contrast learning within the modality is performed, and the contrast loss of each modality is calculated separately. After passing through the basic encoder and mapping layer, the physiological signal is converted into a potential representation G latent ={z i,S |i=1,2,…,5K;S∈{A,B}}. Then, the similarity between samples is calculated using the following formula, and then the contrast loss within each modality is calculated:

[0078]

[0079] Among them, the indicator function I [j≠i] It is equal to 1 when j≠i and 0 otherwise. By minimizing this loss function, the model increases the similarity of positive sample pairs while reducing the similarity of negative sample pairs. The total loss of the entire mini-batch can be expressed as:

[0080]

[0081] The above process calculates the intra-modal temporal contrastive learning loss, aiming to achieve physiological synchronization of different subjects. In addition, in order to enhance the interaction between modalities, the present invention performs cross-modal contrastive learning to align physiological signals from different modalities into a shared semantic space, thereby promoting physiological synchronization between different physiological signals. In the cross-modal contrastive learning process, physiological signal segments of different modalities induced by the same stimulus are regarded as positive pairs, while physiological signal segments generated by different stimuli are regarded as negative pairs. The specific calculation process is to extract the features from the encoder. First pass it through a linear layer to get Then, according to the calculation process from equation (14) to equation (16), the cross-modal contrast loss L is obtainedcc . In the pre-training stage, the total loss L can be expressed as:

[0082] L=α·L eeg +β·L m1 +γ·L cc ; (17)

[0083] Where L eeg is the contrast loss within the EEG modality, and L m1 is another type of contrast loss in peripheral physiological signal m1, L cc is the cross-modal contrast loss.

[0084] like Figure 1 As shown, in the fine-tuning stage, the present invention further fine-tunes the pre-trained basic encoder through labeled samples. During pre-training, the present invention pre-trains the representation of EEG and peripheral signals in different time windows (for example, 1 second and 5 second windows), thereby pre-training the encoder for each respective time window. This method effectively simulates short-term and long-term emotional dynamics. Specifically, the present invention decomposes the physiological signal segments of the long time window into 1 second intervals and passes them through the respective pre-trained encoders. In this way, the long-term and short-term time features of each modality can be extracted and then fused together. By combining these long-term and short-term features, the present invention captures the dynamic changes of emotional states and provides a comprehensive perspective for emotion recognition. In actual experiments, the present invention uses t=5 seconds as a long time window and divides it into 5 1-second segments. After passing these segments through their respective encoders, the time features are obtained. Finally, the long-term and short-term temporal features of each modality are fused to generate the final feature representation H e and H m1 , and then input the result into the modal fusion module.

[0085] In the modality fusion module, the present invention integrates the features from the two modalities by using the Maximum Class Probability (MCP), calculates the confidence of the modality and assigns appropriate weights. The formula is as follows:

[0086]

[0087] Among them, w represents the network parameter set, is the predicted category of the sample.

[0088] Next, the features from the two modalities are concatenated, and finally these combined features are passed to the classifier for final classification and optimized using the cross entropy loss function. The formula for cross entropy loss is as follows:

[0089]

[0090] Among them, P(H joint ) represents the probability that the joint feature vector belongs to a certain category, and y is the label of the sample.

[0091] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0092] In addition, it should be understood that although the present specification is described in terms of implementation modes, not every implementation mode contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for emotion recognition based on EEG and synchronous physiological signals based on cross-modal contrastive learning and multi-scale representation, characterized in that: Includes pre-training stage and fine-tuning stage; Pre-training stage: The physiological signals of different modalities at a specific time scale are extracted with features through the encoder of the corresponding modality, and the modal representation related to the emotional stimulus is learned. The modal representation is then mapped to the latent space through the mapping layer to calculate the similarity, and then the contrast loss of each modality is calculated to perform intra-modal contrast learning, so that the physiological signals of different modalities of different subjects under the same emotional stimulus are close, while the physiological signals of different modalities under different emotional stimuli remain different; the representations extracted by the encoder based on each modality will also be subjected to cross-modal contrast learning to align the representations of different modalities to the shared latent space; by optimizing the contrast loss of each modality and the cross-modal contrast learning loss, the encoders corresponding to each modality at a specific time scale are pre-trained; In the fine-tuning stage, the pre-trained encoder, initialized modal fusion module and classifier are used for emotion recognition training, and the model weights are optimized based on the emotion labels of physiological signals to obtain an emotion recognition model that can output emotion categories corresponding to physiological signals of different modalities.

2. The method according to claim 1, characterized in that The physiological signals of different modalities include electroencephalogram signals and peripheral physiological signals.

3. The method according to claim 1, characterized in that During the contrastive learning process, the physiological signals of different modalities of the subjects are segmented in chronological order to obtain physiological signal segments. The physiological signal segments generated by the same video stimulation form a positive sample pair, and the physiological signal segments generated by different video stimulation form a negative sample pair.

4. The method according to claim 1, characterized in that: It also includes a process of expanding physiological signal data through scaling operations and noise addition operations; the scaling operation generates original data variants that do not change the temporal pattern of the physiological signal by applying different scaling factors; at the same time, Gaussian noise with zero mean and fixed signal-to-noise ratio is added to the original physiological signal to simulate environmental noise.

5. The method according to claim 1, characterized in that The contrastive learning strategy in the pre-training stage is to let the encoders of each modality identify whether different physiological signals correspond to the same emotional stimulus, achieve physiological synchronization within the modality, and achieve physiological alignment across modalities.

6. The method according to claim 5, characterized in that The achieving intra-modal physiological synchronization specifically includes: Taking small batches of physiological signals as the input of the model, the physiological signals of subjects A and B are selected from the multimodal physiological signal dataset, and small batches of physiological signals are obtained through sampling and expansion. Where m represents EEG signal or peripheral physiological signal; small batch physiological signal Input z parallel linear layers to get the feature vector E i ; Input the physiological signal into a single linear layer and sigmoid function to obtain the activation feature vector E i and Multiply element-wise and aggregate over the z-dimensional embedding to obtain the embedding vector e = (e1,…,e z ): Among them, σ represents the sigmoid function, ⊙ represents the Hadamard product, e i is the i-th element in e; BN stands for batch normalization, and RELU represents the activation function; The embedding vector e is preceded by a learnable class label E cls , and embed the learned location information into E pos and modal information embedding E mod Add to e to get the labeled embedding vector Embedding vector Transformed into query Q, key K and value V through three linear layers: Among them, d e is the dimension of the embedding vector; h heads are used for self-attention, and the representation of each head is recorded as H = Attention(Q, K, V); the output of multi-head attention is Concat(H1, H2, ..., H h )W, where W is the weight matrix; then, it is further processed by the feedforward neural network and the above process is repeated twice to finally obtain the features related to the individual stimulation of the physiological signal A nonlinear mapping layer is used after the encoder; the mapping layer consists of three linear layers, a nonlinear activation function ReLU, a Dropout layer, and a batch normalization layer. The mapping layer is used to process the features extracted by the encoder: Among them, W1, W2, W3 are the weight matrices of the three linear layers, b1, b2, b3 are the bias vectors, Represent the output features of the three linear layers respectively; i represents the number of physiological signal segments, S represents the subject, S∈{A,B}, A represents subject A, B represents subject B, and m represents EEG signal or peripheral physiological signal; Temporal contrast learning is performed within the modality, and the contrast loss of each modality is calculated separately; after passing through the basic encoder and mapping layer, the physiological signal is converted into a potential representation G latent ={z i,S |i=1,2,…,5K; S∈{A,B}}, 5K is the number of samples in the training dataset; then, the similarity between samples is calculated using the following formula, and then the contrast loss within each modality is calculated: Among them, the indicator function I [j≠i] It is equal to 1 when j≠i, and 0 otherwise; By minimizing the loss function L m To increase the similarity of positive sample pairs and reduce the similarity of negative sample pairs.

7. The method according to claim 6, characterized in that The achieving of cross-modal physiological alignment specifically includes: Physiological signal segments of different modalities induced by the same stimulus were considered as positive pairs, while physiological signal segments produced by different stimuli were considered as negative pairs; The features extracted by the encoder First pass it through a linear layer to get Then, according to the calculation process from equation (14) to equation (16), the cross-modal contrast loss L is obtained cc ; During the pre-training phase, the total loss L is: L=α·L eeg +β·L m1 +γ·L cc ; Among them, L eeg is the contrast loss within the EEG modality, L m1 is a contrast loss within the peripheral physiological signal m1, L cc is the cross-modal contrast loss.

8. The method according to claim 1, characterized in that In the fine-tuning stage, the pre-trained encoder, the initialized modal fusion module and the classifier are used to perform emotion recognition training, and the model weights are optimized based on the emotion labels of the physiological signals, specifically including: In the pre-training stage, encoders for different time lengths are pre-trained by using physiological signal segments of different time lengths; In the fine-tuning stage, physiological signal segments of different time lengths and different modalities are input into the corresponding encoders, and then the long-term features and short-term features of each modality are fused to obtain the final features of each modality, which are input into the modality fusion module. The final features from the two modalities are integrated by using the maximum class probability, and the confidence of the modality is calculated and the appropriate weight is assigned: Among them, w represents the network parameter set, is the predicted emotion category of the physiological signal; next, the features from the two modalities are concatenated, and finally these combined features are passed to the classifier for final classification, and the cross entropy loss is used. To optimize: Among them, P(H joint ) represents the probability that the joint feature vector belongs to a certain category, and y is the emotion label of the physiological signal.

Images