Single-source multi-mode emotion recognition method based on EEG signal

Abstract

The invention provides a single-source multi-mode emotion recognition method based on EEG (electroencephalogram) signals. A multi-modal emotion recognition network is constructed, the multi-modal emotion recognition network mainly comprises a feature extraction module, a cross-modal feature mapping module and a feature fusion and emotion classification module, the feature extraction module extracts feature vectors of signal data, the cross-modal feature mapping module performs cross-modal data feature mapping in combination with a diffusion model, and the feature fusion and emotion classification module performs feature fusion. The feature fusion and emotion classification module fuses three modal data features and performs emotion classification identification; by performing staged training on the network, high-accuracy emotion recognition can be performed by effectively utilizing EEG signals, meanwhile, many problems caused by multi-modal data fusion are avoided, and lightweight of a network model is realized.

Description

Technical Field

[0001] This invention belongs to the field of biomedical and artificial intelligence interdisciplinary technology, specifically relating to a single-source multimodal emotion recognition method based on EEG signals, which can improve the accuracy of emotion recognition based on EEG signals. Background Technology

[0002] With the continuous development of artificial intelligence technology, emotion recognition has demonstrated tremendous application value in areas such as human-computer interaction and mental health monitoring. Accurately identifying and understanding human emotional states is crucial for improving the intelligence level of these fields. Currently, emotion recognition research based on electroencephalogram (EEG) signals is gradually attracting attention. EEG signals directly reflect brain activity and can provide objective physiological information about emotions, playing a key role in deeply exploring the neural mechanisms of emotional states and promoting the advancement of affective computing and artificial intelligence technologies.

[0003] However, using EEG signals for emotion recognition faces numerous challenges. On the one hand, the brain mechanisms are not yet fully understood; emotions involve complex neural connections across multiple brain regions, making it difficult to accurately identify emotional states using only EEG signals, resulting in low accuracy. On the other hand, EEG signals are inherently weak and susceptible to interference, further increasing the difficulty of emotion recognition. To address these issues, some studies (Yin K., Shin H.-B., Li D., Lee S.-W. EEG-based Multimodal Representation Learning for Emotion Recognition[J]. arXiv:2411.00822v1,2024.) have introduced multimodal data for emotion recognition, improving accuracy by fusing EEG signals with other modal data (such as audio and video). However, this approach faces challenges such as difficulty in aligning multimodal information, a significant increase in data volume and model parameters, increased model complexity, and higher computational resource requirements. The introduction of multimodal data has significantly increased the number of model parameters and the amount of GPU memory used, making the model training and inference process more complex and limiting its promotion and use in practical applications.

[0004] In recent years, diffusion models, as an emerging generative model, have achieved remarkable results in tasks such as image generation and cross-modal data generation. For example, the denoising probabilistic model proposed by Jonathan Ho and Ajay Jain in 2020 (Ho J., Jain A., Abbeel P. Denoising diffusion probabilistic models[J].arXiv:2006.11239,2020.) is modeled by simulating the steps of adding noise to data. The model first approximates the data distribution using a Gaussian perturbation model, and then samples it through its corresponding reverse denoising process. A neural network is used to derive the reverse denoising process step by step and express this process as a multi-step sampling. This model generates data by gradually adding noise to the data and then learning how to reverse this process, including forward noise addition and reverse denoising, providing a new approach to solving problems such as data generation and feature mapping. However, in the field of emotion recognition based on EEG signals, how to effectively utilize diffusion models to help improve the accuracy of emotion recognition while avoiding many problems caused by multimodal data fusion still lacks an effective solution.

[0005] In summary, existing technologies lack an effective method that can efficiently utilize EEG signals for high-accuracy emotion recognition, avoid the many problems caused by multimodal data fusion, fully leverage the advantages of diffusion models, and is lightweight and efficient. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a single-source multimodal emotion recognition method based on EEG signals. A multimodal emotion recognition network is constructed, mainly including a feature extraction module, a cross-modal feature mapping module, and a feature fusion and emotion classification module. The feature extraction module extracts feature vectors from the signal data; the cross-modal feature mapping module combines a diffusion model to map cross-modal data features; and the feature fusion and emotion classification module fuses features from three modalities and performs emotion classification and recognition. By training the network in stages, high-accuracy emotion recognition can be effectively achieved using EEG signals, while avoiding many problems caused by multimodal data fusion and realizing a lightweight network model.

[0007] A single-source multimodal emotion recognition method based on EEG signals, characterized by the following steps:

[0008] S1: Input EEG, video and audio signals, and preprocess the three signal data;

[0009] S2: Construct a multimodal emotion recognition network, mainly including a feature extraction module, a cross-modal feature mapping module, and a feature fusion and emotion classification module. The feature extraction module performs single-modal emotion recognition on the preprocessed EEG signal, video signal, and audio signal respectively, extracting the corresponding feature vectors. The cross-modal feature mapping module uses the EEG feature vector as a condition and a diffusion model to generate corresponding video and audio feature vectors. The feature fusion and emotion classification module concatenates and fuses the EEG features, video features, and audio features, and performs emotion classification through a multi-head attention mechanism and an MLP layer, ultimately outputting the emotion recognition result.

[0010] S3: Input the preprocessed EEG, video and audio data into the multimodal emotion recognition network constructed in step S2, and train the network in stages to obtain the trained network;

[0011] S4: Input the EEG signal to be identified into the trained multimodal emotion recognition network and output the emotion recognition result.

[0012] Specifically, the preprocessing described in step S1 includes processing EEG data using downsampling, bandpass filtering, and slicing methods; processing video data using downsampling and face detection using OpenCV's MTCNN library; resampling audio data; and adjusting the dimensions to meet the input requirements of the AST model.

[0013] Specifically, the feature extraction module performs single-modal emotion recognition on EEG signals as follows: First, a one-dimensional convolutional layer is used to perform preliminary feature extraction on the EEG data. Then, global features of the EEG data are extracted through 6 Transformer layers, where each Transformer layer includes a self-attention mechanism, a dropout layer, residual connections, and layer normalization.

[0014] Specifically, the feature extraction module uses pre-trained ViT and AST models as single-modal emotion recognition models for video and audio data, respectively, and then removes the final classifier layer to extract the corresponding feature vectors.

[0015] Specifically, the processing procedure of the cross-modal feature mapping module is as follows: conditional diffusion is performed using the DDPM probabilistic denoising model, the EEG feature vector and time step embedding are mapped to the same dimension as each Resblock, the EEG feature vector and time step embedding are summed into the Resblock to achieve conditional control and time step control, and the U-Net architecture is used for denoising and feature generation to obtain the mapped video and audio features.

[0016] Specifically, the feature fusion and emotion classification module uses an 8-head attention mechanism for feature fusion and an MLP layer for emotion classification. The MLP layer includes a linear layer, a ReLU activation function, and a dropout layer, and finally outputs the emotion classification result through a linear layer.

[0017] Specifically, the phased network training includes three stages. The first stage is training the single-modal emotion recognition model of the feature extraction module, using the following cross-entropy loss function.

[0018]

[0019] Where y represents the true emotion label. Let i represent the predicted sentiment label, and i represent the i-th sample.

[0020] After training is complete, the final MLP layer is replaced with an nn.Identity layer as the final feature extractor.

[0021] The second stage involves training the conditional diffusion model for the cross-modal feature mapping module. During training, a random time step t is sampled, and noise is added to the video or audio features to obtain intermediate noise. The U-Net network is then used to predict the noise at time step t, and the mean squared error loss function L(θ) is used to minimize the mean squared error between the predicted noise and the added noise.

[0022]

[0023] Where E[·] represents the mathematical expectation, ∈ represents random noise, and ∈ θ This represents the model's prediction noise. α i Here, x0 represents the original clean sample, and c represents the conditional embedding;

[0024] The third stage is the training feature fusion and emotion classification module, which uses the cross-entropy loss function during training.

[0025] Specifically, the U-Net model structure includes a downsampling module, a bottleneck layer, and an upsampling module, while also embedding a temporal embedding module and a conditional embedding module.

[0026] The downsampling module consists of multiple cascaded downsampling steps, each containing two residual blocks. At the end of each step, a convolutional layer is used for spatial downsampling and to increase the number of channels. The input to the downsampling module is an intermediate noise signal or a Gaussian noise signal, and its output is fed into the next downsampling module or bottleneck layer, as well as the corresponding upsampling module, to achieve skip connections. The residual block contains two identical sub-layers, which in turn have a normalization layer, a convolutional layer, and an activation function. The output is the sum of the output and input of the sub-layer. GroupNorm is used for normalization, and the convolutional layer is used for feature extraction. It also receives information from the temporal embedding and conditional embedding modules.

[0027] The bottleneck layer contains two residual blocks. The input is the signal features obtained after processing by all downsampling modules. The residual blocks using the reference attention mechanism further extract features, and the output features are sent to the upsampling module.

[0028] The upsampling module consists of multiple cascaded upsampling steps. Each step contains the same residual block as the corresponding downsampling step. At the beginning of each step, upsampling is performed through linear interpolation to restore spatial resolution and reduce the number of channels. The input to the upsampling module is the output of the bottleneck layer or the previous downsampling module, as well as the features obtained by splicing the feature maps of the corresponding downsampling steps on the channels through skip connections. After passing through the upsampling module, the intermediate noise signal is obtained as the input to the next U-Net downsampling module, until denoising is completed and the corresponding video or audio features are obtained.

[0029] The time embedding module uses sine and cosine position coding to generate an embedding vector for each time step t. This vector is then subjected to dimensionality and nonlinear transformation through two linear layers and the Swish activation function. Finally, the processed time step information is added to the residual blocks of the downsampling and upsampling modules.

[0030] The conditional embedding module transforms the external conditional information and EEG signal features through two linear layers and the Swish activation function in terms of dimensionality and nonlinearity, and then adds them to each residual block to guide the denoising process.

[0031] The attention mechanism is introduced after the residual block, normalized using GroupNorm, and generates queries, keys, and values ​​through a linear layer. Attention weights are calculated using the Softmax function to model global dependencies between different locations in the feature map.

[0032] Specifically, the Pearson correlation coefficient r is used as an evaluation metric for the cross-modal feature mapping module to assess the linear correlation between generated features and real features. The specific formula is as follows:

[0033]

[0034] Where, x i Let y represent the feature vector of the original i-th sample. i This represents the feature vector generated for the i-th sample. This represents the average value of the original feature vectors. This represents the average value of the generated feature vector.

[0035] Specifically, during the inference phase, the cross-modal feature mapping module uses the DDPM model for denoising, randomly samples a Gaussian noise signal, and gradually recovers the video and audio features through several iterations.

[0036] The beneficial effects of this invention are as follows: By using EEG signal feature vectors as conditional inputs and employing a diffusion model for cross-modal mapping, accurate mapping from EEG signals to video and audio features can be achieved, effectively solving the problem of multimodal data alignment and significantly improving the accuracy of emotion recognition based on EEG signals. Simultaneously, the use of feature vector mapping avoids the significant increase in data volume and model parameters caused by multimodal data fusion, achieving lightweight and efficient models. The efficient generation capability of the diffusion model accelerates feature generation and emotion recognition, improving the overall efficiency of the method. The use of a multi-head attention mechanism for feature fusion fully utilizes the feature information of different modalities, further improving the accuracy of emotion recognition. The fusion of cross-modal feature mapping and multi-head attention significantly improves the accuracy of emotion recognition, giving the method high accuracy and robustness. Preprocessing and feature extraction steps enhance the model's robustness to EEG signal noise and interference, improving the stability of emotion recognition. Through staged model training, high-accuracy emotion recognition can be effectively achieved using EEG signals, while avoiding many problems caused by multimodal data fusion. Attached Figure Description

[0037] Figure 1 This is a flowchart of a single-source multimodal emotion recognition method based on EEG signals according to the present invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments. The present invention includes, but is not limited to, the following embodiments.

[0039] Traditional unimodal methods for emotion recognition based on EEG signals have several limitations, such as the complexity and susceptibility to interference in EEG signals, leading to low accuracy in emotion recognition. While multimodal methods can improve accuracy, they face challenges such as difficulty in aligning multimodal information, a significant increase in data volume and model parameters, increased model complexity, and higher computational resource requirements. To address these challenges, this invention proposes a unimodal emotion recognition method based on EEG signals. Figure 1 As shown, the specific implementation process is as follows:

[0040] S1: Input EEG, video, and audio signals, and preprocess the three signal data. This includes operations such as downsampling, bandpass filtering, and slicing of the EEG signal to extract EEG feature vectors. Typically, the EEG signal can be downsampled to 100Hz, filtered using a 5th-order Butterworth bandpass filter with a passband frequency range of 0.5Hz to 45Hz, and the 20 seconds of EEG data can be divided into four 5-second data segments. For the video signal, operations such as frame extraction and face detection are performed to extract video feature vectors. Specifically, OpenCV functions can be used for frame extraction, processing one frame every six frames, and the MTCNN model is used for face detection. The video data is divided into four 5-second video samples. For the audio signal, operations such as resampling and spectrogram transformation are performed to extract audio feature vectors. Specifically, the signal is resampled to 16kHz, and the 20 seconds of audio data is divided into four 5-second data segments.

[0041] S2: Construct a multimodal emotion recognition network, which mainly includes a feature extraction module, a cross-modal feature mapping module, and a feature fusion and emotion classification module.

[0042] The feature extraction module performs single-modal emotion recognition on the preprocessed EEG, video, and audio signals respectively, extracting corresponding feature vectors. For EEG signals, local and global features are extracted using a one-dimensional convolutional layer and a 6-layer Transformer layer, respectively. The process is as follows: first, a one-dimensional convolutional layer is used to perform preliminary feature extraction on the EEG data; then, global features of the EEG data are extracted through a 6-layer Transformer layer. Each Transformer layer includes a self-attention mechanism, a dropout layer, residual connections, and layer normalization, resulting in a feature vector with a dimension of 1*60. For video signals, a pre-trained ViT model is used to extract features, with a feature vector dimension of 1*768. For audio signals, a pre-trained AST model is used to extract features, with a feature vector dimension of 1*768.

[0043] The cross-modal feature mapping module uses EEG feature vectors as conditions and a diffusion model to generate corresponding video and audio feature vectors, achieving feature mapping between different modalities. Specifically, the DDPM probabilistic denoising model is used for conditional diffusion, mapping the EEG feature vectors and time step embeddings to the same dimension as each Resblock. The EEG feature vectors and time step embeddings are summed into the Resblock to achieve conditional and time step control. The U-Net architecture is used for denoising and feature generation, and the generated video and audio feature dimensions are both 1*768.

[0044] The U-Net model structure includes a downsampling module, a bottleneck layer, and an upsampling module, while also embedding a temporal embedding module and a conditional embedding module.

[0045] The downsampling module consists of multiple cascaded downsampling steps, each containing two residual blocks. At the end of each step, a convolutional layer is used for spatial downsampling and to increase the number of channels. The input to the downsampling module is an intermediate noise signal or a Gaussian noise signal, and its output is fed into the next downsampling module or bottleneck layer, as well as the corresponding upsampling module, to achieve skip connections. The residual block contains two identical sub-layers, which in turn have a normalization layer, a convolutional layer, and an activation function. The output is the sum of the output and input of the sub-layer. GroupNorm is used for normalization, and the convolutional layer is used for feature extraction. It also receives information from the temporal embedding and conditional embedding modules.

[0046] The bottleneck layer contains two residual blocks. The input is the signal features obtained after processing by all downsampling modules. The residual blocks using the reference attention mechanism further extract features, and the output features are sent to the upsampling module.

[0047] The upsampling module consists of multiple cascaded upsampling steps. Each step contains the same residual block as the corresponding downsampling step. At the beginning of each step, upsampling is performed through linear interpolation to restore spatial resolution and reduce the number of channels. The input to the upsampling module is the output of the bottleneck layer or the previous downsampling module, as well as the features obtained by splicing the feature maps of the corresponding downsampling steps on the channels through skip connections. After passing through the upsampling module, the intermediate noise signal is obtained as the input to the next U-Net downsampling module, until denoising is completed and the corresponding video or audio features are obtained.

[0048] The time embedding module uses sine and cosine position coding to generate an embedding vector for each time step t. This vector is then subjected to dimensionality and nonlinear transformation through two linear layers and the Swish activation function. Finally, the processed time step information is added to the residual blocks of the downsampling and upsampling modules.

[0049] The conditional embedding module transforms the external conditional information and EEG signal features through two linear layers and the Swish activation function in terms of dimensionality and nonlinearity, and then adds them to each residual block to guide the denoising process.

[0050] The attention mechanism is introduced after the residual block, normalized using GroupNorm, and generates queries, keys, and values ​​through a linear layer. Attention weights are calculated using the Softmax function to model global dependencies between different locations in the feature map.

[0051] The feature fusion and emotion classification module concatenates and fuses EEG features, video features, and audio features, and performs emotion classification using a multi-head attention mechanism and an MLP layer, ultimately outputting the emotion recognition result. Specifically, the feature fusion network uses an 8-head attention mechanism for feature fusion. Emotion classification is performed using an MLP layer, including a linear layer, a ReLU activation function, and a dropout layer, with the final emotion classification result output through a linear layer. These feature fusion and emotion classification operations fully utilize the feature information from the three modalities, improving the accuracy of emotion recognition.

[0052] S3: Input the preprocessed EEG, video and audio data into the multimodal emotion recognition network constructed in step S2, and train the network in stages to obtain the trained network.

[0053] The phased network training described includes three stages. The first stage is training the single-modal emotion recognition model of the feature extraction module, using the following cross-entropy loss function.

[0054]

[0055] Where y represents the true emotion label. Let i represent the predicted sentiment label, and i represent the i-th sample.

[0056] After training, the final MLP layer is replaced with an nn.Identity layer as the final feature extractor to extract video and audio features respectively. The extracted feature vectors have a dimension of 1*768.

[0057] The second stage involves training the conditional diffusion model for the cross-modal feature mapping module. During training, a random time step t is sampled, and noise is added to the video or audio features to obtain intermediate noise. The U-Net network is then used to predict the noise at time step t, and the mean squared error loss function is applied. That is, minimizing the mean square error (MSE) of the predicted noise and the added noise:

[0058]

[0059] Where E[·] represents the mathematical expectation, ∈ represents random noise, and ∈ θ This represents the model's prediction noise. α i Here, x0 represents the original clean sample, and c represents the conditional embedding.

[0060] During the testing (inference) phase, the cross-modal feature mapping module uses the DDPM model for denoising. It randomly samples a Gaussian noise signal and gradually denoises it through multiple denoising processes to recover the original video or audio features.

[0061] The Pearson correlation coefficient r is used as an evaluation index to assess the linear correlation between generated features and real features. The specific formula is as follows:

[0062]

[0063] Where, x i Let y represent the feature vector of the original i-th sample. i This represents the feature vector generated for the i-th sample. This represents the average value of the original feature vectors. This represents the average value of the generated feature vector.

[0064] The third stage is the training feature fusion and emotion classification module, which uses the cross-entropy loss function during training. To optimize the classification performance of the model.

[0065] S4: Input the EEG signal to be identified into the trained multimodal emotion recognition network and output the emotion recognition result.

[0066] The single-source multimodal emotion recognition method based on EEG signals provided in this embodiment effectively improves the accuracy of emotion recognition based on EEG signals by using EEG signals as conditions for cross-modal feature mapping. This cross-modal feature mapping mechanism achieves accurate mapping from EEG signals to video and audio features by using EEG signal feature vectors as conditional inputs to the diffusion model. This not only solves the problem of difficult alignment of multimodal data but also avoids the significant increase in data volume and model parameters caused by multimodal data fusion, achieving lightweight and efficient model design. Furthermore, feature fusion through a multi-head attention mechanism fully utilizes the feature information of different modalities, further improving the accuracy of emotion recognition.

[0067] This invention offers the following significant advantages: First, its innovative cross-modal mapping mechanism, by using EEG signal feature vectors as conditional input to the diffusion model, achieves precise mapping from EEG signals to video and audio features. This innovative mechanism effectively solves the challenge of multimodal data alignment, providing a novel technical path for emotion recognition based on EEG signals. Second, its lightweight and efficient model, employing feature vector mapping, significantly reduces the number of model parameters and GPU memory usage, lowering the computational resource requirements. Simultaneously, leveraging the efficient generation capability of the diffusion model accelerates feature generation and emotion recognition, improving the overall system efficiency. Third, its high accuracy and robustness, achieved through the fusion of cross-modal feature mapping and multi-head attention mechanisms, significantly improves the accuracy of emotion recognition. Furthermore, preprocessing and feature extraction steps enhance the model's robustness to EEG signal noise and interference, improving the stability of emotion recognition. Finally, by training the model in stages, it effectively utilizes EEG signals for high-accuracy emotion recognition while avoiding many problems associated with multimodal data fusion, fully leveraging the advantages of the diffusion model to achieve lightweight and efficient performance. Compared to EEG unimodal emotion recognition, the method of this invention improves accuracy by 11%, demonstrating the effectiveness of the model. Furthermore, the method of this invention performs exceptionally well in single-source multimodal emotion recognition tasks, achieving an average accuracy of 0.612, indicating that the model can effectively integrate features from EEG signals and other modal data, and that the data from different modalities complement each other.

[0068] Therefore, this invention is not only technically innovative, but also has broad prospects in practical applications. It is applicable to fields such as mental health monitoring and human-computer interaction, and can provide an efficient and accurate emotion recognition solution for related fields, which has important social application value and practical significance.

Claims

1. A single-source multimodal emotion recognition method based on EEG signals, characterized in that... The steps are as follows: S1: Input EEG, video and audio signals, and preprocess the three signal data; S2: Construct a multimodal emotion recognition network, which mainly includes a feature extraction module, a cross-modal feature mapping module, and a feature fusion and emotion classification module. The feature extraction module performs single-modal emotion recognition on the preprocessed EEG signal, video signal, and audio signal respectively to extract the corresponding feature vectors. The cross-modal feature mapping module uses the EEG feature vector as a condition and uses a diffusion model to generate the corresponding video and audio feature vectors. The feature fusion and emotion classification module concatenates and fuses EEG features, video features, and audio features, and performs emotion classification through a multi-head attention mechanism and an MLP layer, ultimately outputting the emotion recognition result. S3: Input the preprocessed EEG, video and audio data into the multimodal emotion recognition network constructed in step S2, and train the network in stages to obtain the trained network; S4: Input the EEG signal to be identified into the trained multimodal emotion recognition network and output the emotion recognition result.

2. The single-source multimodal emotion recognition method based on EEG signals as described in claim 1, characterized in that: The preprocessing described in step S1 includes processing EEG data using downsampling, bandpass filtering, and slicing methods; processing video data using downsampling and face detection using OpenCV's MTCNN library; and resampling audio data and adjusting the dimensions to meet the input of the AST model.

3. The single-source multimodal emotion recognition method based on EEG signals as described in claim 1, characterized in that: The feature extraction module performs single-modal emotion recognition on EEG signals as follows: First, a one-dimensional convolutional layer is used to perform preliminary feature extraction on the EEG data. Then, global features of the EEG data are extracted through 6 Transformer layers. Each Transformer layer includes a self-attention mechanism, a dropout layer, residual connections, and layer normalization.

4. The single-source multimodal emotion recognition method based on EEG signals as described in claim 1, characterized in that: The feature extraction module uses pre-trained ViT and AST models as unimodal emotion recognition models for video and audio data, respectively. Then, the final classifier layer is removed to extract the corresponding feature vectors.

5. The single-source multimodal emotion recognition method based on EEG signals as described in claim 1, characterized in that: The cross-modal feature mapping module processes as follows: it uses the DDPM probabilistic denoising model for conditional diffusion, maps the EEG feature vectors and time step embeddings to the same dimension as each Resblock, sums the EEG feature vectors and time step embeddings to the Resblock to achieve conditional control and time step control, and uses the U-Net architecture for denoising and feature generation to obtain the mapped video and audio features.

6. The single-source multimodal emotion recognition method based on EEG signals as described in claim 1, characterized in that: The feature fusion and emotion classification module uses an 8-head attention mechanism for feature fusion and an MLP layer for emotion classification. The MLP layer includes a linear layer, a ReLU activation function, and a dropout layer. Finally, the emotion classification result is output through a linear layer.

7. The single-source multimodal emotion recognition method based on EEG signals as described in claim 1, characterized in that: The phased network training described includes three stages. The first stage is training the single-modal emotion recognition model of the feature extraction module, using the following cross-entropy loss function. Where y represents the true emotion label. Let i represent the predicted sentiment label, and i represent the i-th sample. After training is complete, the final MLP layer is replaced with an nn.Identity layer as the final feature extractor. The second stage involves training the conditional diffusion model for the cross-modal feature mapping module. During training, a random time step t is sampled, and noise is added to the video or audio features to obtain intermediate noise. The U-Net network is then used to predict the noise at time step t, and the mean squared error loss function L(θ) is used to minimize the mean squared error between the predicted noise and the added noise. Where E[·] represents the mathematical expectation, ∈ represents random noise, and ∈ θ This represents the model's prediction noise. α i Here, x0 represents the original clean sample, and c represents the conditional embedding; The third stage is the training feature fusion and emotion classification module, which uses the cross-entropy loss function during training.

8. The single-source multimodal emotion recognition method based on EEG signals as described in claim 5, characterized in that: The U-Net model structure includes a downsampling module, a bottleneck layer, and an upsampling module, and also embeds a temporal embedding module and a conditional embedding module. The downsampling module consists of multiple cascaded downsampling steps, each containing two residual blocks. At the end of each step, a convolutional layer is used for spatial downsampling and to increase the number of channels. The input to the downsampling module is an intermediate noise signal or a Gaussian noise signal, and its output is fed into the next downsampling module or bottleneck layer, as well as the corresponding upsampling module, to achieve skip connections. The residual block contains two identical sub-layers, which in turn have a normalization layer, a convolutional layer, and an activation function. The output is the sum of the output and input of the sub-layer. GroupNorm is used for normalization, and the convolutional layer is used for feature extraction. It also receives information from the temporal embedding and conditional embedding modules. The bottleneck layer contains two residual blocks. The input is the signal features obtained after processing by all downsampling modules. The residual blocks using the reference attention mechanism further extract features, and the output features are sent to the upsampling module. The upsampling module consists of multiple cascaded upsampling steps. Each step contains the same residual block as the corresponding downsampling step. At the beginning of each step, upsampling is performed through linear interpolation to restore spatial resolution and reduce the number of channels. The input to the upsampling module is the output of the bottleneck layer or the previous downsampling module, as well as the features obtained by splicing the feature maps of the corresponding downsampling steps on the channels through skip connections. After passing through the upsampling module, the intermediate noise signal is obtained as the input to the next U-Net downsampling module, until denoising is completed and the corresponding video or audio features are obtained. The time embedding module uses sine and cosine position coding to generate an embedding vector for each time step t. This vector is then subjected to dimensionality and nonlinear transformation through two linear layers and the Swish activation function. Finally, the processed time step information is added to the residual blocks of the downsampling and upsampling modules. The conditional embedding module transforms the external conditional information and EEG signal features through two linear layers and the Swish activation function in terms of dimensionality and nonlinearity, and then adds them to each residual block to guide the denoising process. The attention mechanism is introduced after the residual block, normalized using GroupNorm, and generates queries, keys, and values ​​through a linear layer. Attention weights are calculated using the Softmax function to model global dependencies between different locations in the feature map.

9. The single-source multimodal emotion recognition method based on EEG signals as described in claim 5, characterized in that: The Pearson correlation coefficient r is used as an evaluation metric for the cross-modal feature mapping module to assess the linear correlation between generated features and real features. The specific formula is as follows: Where, x i Let y represent the feature vector of the original i-th sample. i This represents the feature vector generated for the i-th sample. This represents the average value of the original feature vectors. This represents the average value of the generated feature vector.

10. The single-source multimodal emotion recognition method based on EEG signals as described in claim 5, characterized in that: During the inference phase, the cross-modal feature mapping module uses the DDPM model for denoising, randomly samples a Gaussian noise signal, and gradually recovers the video and audio features through several iterations.

Images