Multi-modal emotion recognition method and system

Through the multimodal fusion of spatial and temporal context encoder and modal attention mechanism, the problem of failure to fully utilize modal context information in the prior art is solved, and more efficient emotion recognition performance and stability are achieved to adapt to emotions. Identify the complexity and diversity of technology.

CN119066559AActive Publication Date: 2024-12-03SOUTH CHINA NORMAL UNIV

Patent Information

Application Number
CN202410962672.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-12-03
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

The existing multimodal emotion recognition technology fails to fully utilize the spatial context information of EEG signals and facial expression data, resulting in low recognition accuracy and efficiency. The traditional method has limited ability to capture long-distance dependencies when processing long sequences, and cannot be effective. Capture nonlinear complex relationships between modals.

Method used

The modal features are extracted using spatial context encoder and temporal context encoder, and the weighted fusion is performed through a multimodal fusion module based on the modal attention mechanism, combining SwinTransformer and Transformer structures to ensure that high-resolution information is not lost and the modal adjustment is dynamically adjusted. state weights to adapt to the dominance of different emotions.

Benefits of technology

It improves the accuracy and stability of emotion recognition, enhances the generalization ability of the model, can capture the complexity and multidimensionality of emotions more accurately, adapt to different emotional expression scenarios, and improves the performance of emotion recognition technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066559A_ABST
    Figure CN119066559A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-modal emotion recognition method and system, and the method specifically comprises the following steps: 1, data preprocessing and feature extraction: for emotion electroencephalogram, processing electroencephalogram data through an EEGLAB tool, carrying out the band-pass filtering of the EEG data through a zero-phase-shift FIR filter at a 0.5-50 Hz wave band, eliminating the electrical interference in 50 Hz line noise, and carrying out the feature extraction; interpolation is carried out on the positions of the removed bad electrodes through a spherical interpolation method, and ICA processing is carried out to eliminate non-brain-related artifacts; according to the method provided by the invention, different modes, namely space and time context information of electroencephalogram signals and facial expressions, are fully considered; by introducing a spatio-temporal context encoder, key information at different time points and spatial positions can be captured, so that the dynamic change and spatial distribution characteristics of emotions can be deeply understood; the comprehensive context information utilization is crucial for improving the accuracy of emotion recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence, computer vision, and emotional computing technology, and specifically relates to a multimodal emotion recognition method and system. Background Art

[0002] Emotion recognition technology has shown its great potential and value in many practical application scenarios. For example, in human-computer interaction, this technology can enable machines to understand human needs more deeply and provide intelligent services. In medical auxiliary diagnosis, emotion recognition is of great help in the diagnosis and treatment of neurological diseases such as sleep disorders, schizophrenia, Parkinson's disease, etc., and can also monitor the patient's physiological conditions, such as fatigue, sleepiness, depression and pain. In addition, emotion recognition is also of great significance to the research of psychological diseases such as autism, ADHD and panic disorder.

[0003] In the past, research results on emotion recognition mainly focused on the analysis and application of a single data source (such as facial expressions or voice signals), such as text sentiment analysis (TSA), speech emotion recognition (SER) and facial expression recognition (FER). Although these technologies have been able to accurately recognize emotions through machine learning and deep learning methods, each single-modal recognition method has certain limitations, such as insufficient recognition accuracy, difficulty in balancing accuracy and time efficiency, and lack of interpretability.

[0004] In recent years, researchers in the field of emotion recognition have gradually realized that emotional expression is a complex and multidimensional process. It does not rely solely on a single perceptual channel, but requires the integration of information from multiple perceptual channels to form a more comprehensive and accurate emotional expression. Therefore, multimodal emotion recognition technology has emerged, which combines information from multiple perceptual sources to identify and understand human emotional states. Compared with unimodal emotion recognition, multimodal emotion recognition can fully explore the consistency and complementary features between different modal emotion data.

[0005] Facial expression is one of the modalities that can accurately represent a variety of emotions. As a natural feature with non-contact, human face is widely used in various scenarios due to its high distinguishability, easy acquisition and stability. EEG signals, as a kind of physiological signal, can reflect the electrical activity of nerve cells in the cerebral cortex, thereby revealing the true emotional state. Many psychophysiological studies have pointed out that emotions are related to the aggregated electrical activity of nerve cells in the cerebral cortex. Therefore, combining EEG signals with facial expression data and using the information of the two perception channels to complement each other can overcome the limitations of a single data source in emotion recognition, thereby improving the accuracy and stability of emotion recognition.

[0006] The current patents involving multimodal emotion recognition of EEG signals and facial expressions have the following problems:

[0007] Patent application number CN117591967A is a multimodal emotion recognition method and system based on confidence fusion. This patent fails to fully consider the spatial context information inherent in each modal signal (EEG signal and facial expression data), and ignores the use of this information to improve recognition accuracy. The LSTM (Long Short-Term Memory Network) method based on it limits the model parallelization capability due to its element-by-element processing characteristics, resulting in inefficient training process. In addition, LSTM has limited ability to capture long-distance dependencies when processing long sequences, which further affects the accuracy of emotion recognition;

[0008] Patent application number CN115359576A is a multimodal emotion recognition method, device, electronic device and storage medium. This patent fails to fully utilize the spatial context information of each modal signal. In the multimodal fusion stage, it uses a fixed weight allocation method, which ignores the dominant differences of different emotions in different modalities and cannot capture the nonlinear complex relationship that may exist between modalities. Therefore, this fusion strategy lacks rationality in theory and methodology;

[0009] Patent application number CN117520826A is a multimodal emotion recognition method and system based on wearable devices. The patent fails to effectively utilize the spatial context information of each modal signal. The SVM (support vector machine) method based on it has limitations in processing high-dimensional and complex multimodal data. It is sensitive to data noise and outliers, and is more suitable for small-scale, low-dimensional and high-quality data scenarios. In addition, the method does not mention a clear multimodal fusion solution, which limits its effectiveness in practical applications to a certain extent.

[0010] Patent application number CN117409396A is a multimodal emotion recognition method, system, device and medium. It also attempts to combine EEG signals with facial expressions for multimodal emotion recognition and directly splices EEG feature vectors and facial feature vectors. However, due to the inherent heterogeneity gap between the two modalities of facial images and EEG signals, splicing these two features for emotion classification may lose the semantic information related to emotions in each modality, thereby reducing classification performance. In addition, this method only supports binary emotion recognition, which has certain limitations for deeper research and practical application scenarios in the field of emotion recognition. Summary of the invention

[0011] The purpose of the present invention is to provide a multimodal emotion recognition method and system to solve the problem raised in the above background technology that the current emotion recognition method has the limitation of a single data source in emotion recognition, which makes the computer unable to effectively recognize and understand the user's emotional state.

[0012] To achieve the above object, the present invention provides the following technical solution: a multimodal emotion recognition method, specifically comprising the following steps:

[0013] Step 1: Data preprocessing and feature extraction. For emotional EEG, the EEGLAB tool is used to process the EEG data. First, the EEG data is bandpass filtered in the 0.5-50Hz band using a zero-phase-shift FIR filter to eliminate the electrical interference in the 50Hz line noise. The spherical interpolation method is used to interpolate the removed bad electrode positions, and ICA processing is performed to eliminate non-brain-related artifacts to ensure the accuracy of subsequent analysis. The EEG data is downsampled using a sampling frequency of 256 Hz to speed up the calculation speed. After processing the EEG data, the three-dimensional transformation process of the multi-channel EEG will be performed;

[0014] Step 2: Extract spatial context features from the preprocessed emotional EEG features and facial expression features using a spatial context encoder;

[0015] Step 3: Use the temporal context encoder to further extract temporal context features;

[0016] Step 4: Use a multimodal fusion module based on the modal attention mechanism to fuse the spatiotemporal feature representations of different modalities, and finally perform multimodal feature emotion classification. Using the same network structure to process data of different modalities is not only conducive to the uniformity and efficiency of multimodal model training, but also helps to complement and integrate different modal information in subsequent stages, thereby improving the performance of emotion recognition.

[0017] As a preferred technical solution in the present invention, in step one, the three-dimensional transformation process involves converting the signals recorded by multiple EEG electrodes into a topographic map form, wherein the power distribution of each electrode constitutes a specific point on the topographic map.

[0018] As a preferred technical solution in the present invention, in the step 2, the spatial context encoder adopts SwinTransformer as the backbone architecture to learn the spatial context information of EEG features and facial features, and ensures that high-resolution data information is not lost during the processing process, which is crucial for capturing subtle changes in the spatial distribution of data. The spatial context features are extracted from the preprocessed EEG features or facial features to obtain a fused spatial context information feature map.

[0019] As a preferred technical solution in the present invention, in the step three, the temporal context encoder is designed based on the Transformer structure to alleviate problems such as face occlusion and abnormal EEG peaks in the time dimension and capture the dynamic characteristics of EEG signals and facial expressions changing over time.

[0020] As a preferred technical solution in the present invention, when discussing multimodal emotion recognition, the present invention observed that there are differences in the dominance of different emotions in different modalities. Some emotions may be mainly dominated by electroencephalogram (EEG), while other emotions may be more reflected through facial expressions. In order to adapt the backbone network of the present invention and make full use of the information of these two modalities, a multimodal attention mechanism is designed. This mechanism is based on the calculated attention weights, that is, in the step four, the multimodal fusion module based on the modal attention mechanism performs weighted fusion on the encoding features of EEG and facial expressions, allowing each modality to dynamically adjust its weight according to its contribution to a specific emotion. For emotions dominated by EEG, EEG features will be given a higher weight; and for emotions dominated by facial expressions, facial expression features will be given a greater weight.

[0021] As a preferred technical solution in the present invention, the calculation method of the multimodal fusion module based on the modal attention mechanism is as follows:

[0022] First, given the output feature X∈Rpre of the EEG or face encoder output by the backbone network, pre represents the original dimension of feature X; this feature X is used as the input of this module and is further processed to obtain multimodal feature representation; the following composite operation is used to obtain a feature group Xn∈Rpre*n with a number of n, where n represents the number of modalities; in the task, n is set to 2, indicating that bimodal data is processed; the composite operation includes applying the dimension alignment convolution operation AlignCov (aligning high-dimensional features to low-dimensional features), the rectified linear unit ReLU and the batch normalization BN, which are helpful for the dimension alignment of feature maps of different modalities and the stability of model training convergence;

[0023] Xn=ReLU(B(AlignCov(X))) (1)

[0024] Subsequently, in order to calculate the modality attention tensor, a fully connected operator Fc is introduced; the fully connected layer can learn the complex relationship between different modality features and output a modality attention weight vector; the weight vector is normalized by applying the softmax operator to obtain the attention score of each modality; these scores reflect the importance of different modalities in the current task; the calculation process of the modality attention tensor can be expressed as:

[0025]

[0026] Finally, the modal attention weighted feature Xout is obtained by element-by-element multiplication (⊙) of the modal attention tensor and the original feature group Xn. This operation weights the features of different modalities according to their attention scores, thereby achieving effective fusion of multimodal information. The mathematical expression is:

[0027] Xout=Sn⊙Xn (4)

[0028] In order to learn cross-modal associations and classify them, Xout is input into a multi-layer perceptron (MLP). The constructed MLP is a three-layer stacked fully connected network, which uses its nonlinear fitting ability to further extract and integrate deep features in multimodal information. Through the layer-by-layer transmission and processing of MLP, the potential associations between different modalities can be captured, and emotion classification tasks can be performed based on these features. Finally, it is processed by the softmax layer to obtain the normalized output probability of each emotion category. The role of the softmax layer is to convert the original output score of the MLP into a probability distribution, ensuring that the probability value of each emotion category is between 0 and 1, and the sum of the probabilities of all categories is strictly equal to 1. This This probability distribution form is helpful for the subsequent loss calculation and interpretation of emotion recognition results; subsequently, the cross entropy loss function is used as the optimization target in the training process. The cross entropy loss function is an effective tool to measure the difference between the probability distribution predicted by the model and the true emotion label distribution; its calculation formula is based on the probability distribution output by the softmax layer and the true emotion label distribution, and the difference is quantified by calculating the negative log-likelihood value between the two; during the training process, the model continuously adjusts its internal parameters to minimize the cross entropy loss function, so that the predicted probability distribution gradually approaches the true emotion label distribution, thereby improving the accuracy and performance of emotion recognition.

[0029] The present invention also discloses a multimodal emotion recognition system, which is implemented by combining B / S (browser / server) and C / S (client / server) architectures. The system is divided into three core components: front-end, back-end and database, to improve the maintainability and scalability of the system.

[0030] The front-end component, as a portal for user interaction, uses C++ combined with the QT framework to build the client interface. At the same time, the Vue.js framework is introduced to implement the back-end administrator data management interface, providing rich operation interfaces and intuitive emotion recognition result display for these two types of users; this component is responsible for receiving user operation instructions and sending task requests to the back-end through the API interface; at the same time, the front-end component is also responsible for receiving the task execution results returned by the back-end and displaying them to the user in a visual way;

[0031] The backend component, as the core of the business logic, uses Python programming language and Django RESTframework to build RESTful API to realize the interaction of front-end and back-end data; the jinja2 template engine is used for page rendering to improve user experience; the backend component receives the task request sent by the frontend, executes the corresponding system management or emotion recognition logic, and calls the database component for data storage and retrieval;

[0032] Database components include MySQL, Redis and SQLite. MySQL, as a relational database management system, is responsible for storing structured data, such as user information, network models, etc. Redis, as an in-memory database, is used to cache hot data and improve system response speed. The database component provides a data access interface to support data operation requests from back-end components. SQLite is used for encrypted storage and management of client local data.

[0033] As a preferred technical solution of the present invention, it also includes

[0034] At the algorithm level, MATLAB EEGLab and MNE-Python libraries are used to preprocess EEG signals and extract features, and Pillow library is used to preprocess expression images and extract key information;

[0035] Build and manage deep learning models based on Pytorch and Openmmlab deep learning frameworks;

[0036] For service deployment, Nginx is used as the reverse proxy server and uWSGI server is used as the web server.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The method proposed in the present invention fully considers the spatial and temporal context information of different modalities, namely EEG signals and facial expressions; by introducing a spatiotemporal context encoder, it can capture key information at different time points and spatial positions, so as to deeply understand the dynamic changes and spatial distribution characteristics of emotions; this comprehensive use of context information is crucial to improving the accuracy of emotion recognition; by simultaneously analyzing the neural dynamics in EEG signals and the subtle changes in facial expressions, it can more accurately capture the complexity and multidimensionality of emotions, bringing significant performance improvements to emotion recognition technology;

[0039] 2. The present invention proposes an innovative multimodal fusion method, which can adaptively adjust the attention to EEG signals and facial expressions according to specific circumstances; by introducing an attention mechanism or a dynamic weight allocation strategy, the system can focus on the key information in the two modalities in real time to optimize the accuracy and robustness of emotion recognition; this adaptive fusion strategy not only improves the generalization ability of the model, but also provides new ideas for the development of emotion recognition technology; in practical applications, it can cope with the diversity of emotional expressions in different scenarios, and provide strong support for the research and application of the field of emotional computing;

[0040] 3. The present invention proposes a solution to reduce the heterogeneity between EEG signals and facial expression data, which can reduce the difference between EEG signals and facial expression data and realize the effective fusion of two modal information; this processing scheme not only improves the accuracy of multimodal emotion recognition, but also enhances the robustness and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a roadmap for extracting EEG features and facial expression features in the present invention;

[0042] Figure 2 A flow chart of emotion classification according to the present invention;

[0043] Figure 3 Schematic diagram of the processing flow of the spatial context encoder of the present invention;

[0044] Figure 4 Schematic diagram of the processing flow of the temporal context encoder of the present invention;

[0045] Figure 5 It is a processing flow chart of the multimodal fusion module based on the modal attention mechanism of the present invention;

[0046] Figure 6 It is a system diagram of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] See also Figures 1 to 6 The present invention provides a multi-technical solution: a multimodal emotion recognition method, which specifically includes the following steps:

[0049] Step 1: Data preprocessing and feature extraction. For emotional EEG, the EEGLAB tool is used to process the EEG data. First, the EEG data is bandpass filtered in the 0.5-50Hz band using a zero-phase-shift FIR filter to eliminate the electrical interference in the 50Hz line noise. The spherical interpolation method is used to interpolate the removed bad electrode positions, and ICA processing is performed to eliminate non-brain-related artifacts to ensure the accuracy of subsequent analysis. The EEG data is downsampled using a sampling frequency of 256 Hz to speed up the calculation speed. After processing the EEG data, the three-dimensional transformation process of the multi-channel EEG will be performed;

[0050] Step 2: Extract spatial context features from the preprocessed emotional EEG features and facial expression features using a spatial context encoder;

[0051] Step 3: Use the temporal context encoder to further extract temporal context features;

[0052] Step 4: Use a multimodal fusion module based on the modal attention mechanism to fuse the spatiotemporal feature representations of different modalities, and finally perform multimodal feature emotion classification. Using the same network structure to process data of different modalities is not only conducive to the uniformity and efficiency of multimodal model training, but also helps to complement and integrate different modal information in subsequent stages, thereby improving the performance of emotion recognition.

[0053] In this embodiment, in step 1, the three-dimensional transformation process involves converting the signals recorded by multiple EEG electrodes into a topographic map form, wherein the power distribution of each electrode constitutes a specific point on the topographic map. The present invention extracts the power distribution rectangle of the specified frequency band. This measure is intended to make the EEG data consistent with the facial expression data in form, so that the same network architecture can be used for training. Taking the 32-electrode channel as an example, firstly, the time-frequency decomposition tool of EEGLAB is used, combined with the Morlet wavelet (whose period is set to 1 and the window size winsize is 256), to perform time-frequency analysis on the EEG data to draw the event-related spectral perturbation (Event-Related Spectral Perturbations ERSP) images; for each EEG sample, through this processing, an ERSP image can be generated from each channel; based on the 32 channel data of a sample, 32 ERSP images were successfully drawn, which fully demonstrated the power changes of different channels at different frequencies and time points; secondly, in order to accurately analyze the impact of visual stimulation on EEG activity, the moment of visual stimulation was marked as 0ms on the time axis; considering that the fluctuation of event-related potential (ERP) usually occurs within 1000 milliseconds after stimulation, it was decided to extract ERSP image data from the time range of 0 milliseconds to 1000 milliseconds with a step size of 50 milliseconds; through this In one operation, a total of 20 frames of ERSP image data were obtained; the data size of each frame was 32×18, which means the power distribution of 32 EEG channels in 18 different frequency bands (θ band, from 14Hz to 31Hz) at a specific moment; further, each row in each frame of data (i.e., 32×1 data) can be plotted as a 32×32 square topographic map of a specific frequency band by using spline interpolation technology; these topographic maps can intuitively show the power distribution differences between various EEG channels at a specific frequency band and moment, thereby obtaining an EEG data representation form that is easy to analyze and adapts to the network model proposed by the present invention, providing a solid foundation for subsequent emotion recognition tasks;

[0054] For dynamic facial expression data, in order to uniformly process input videos of different lengths, interpolation cropping operations are used to ensure that the number of frames of each video is fixed at 200 frames, and the sampling interval is synchronized with the EEG signal; this processing method ensures that no matter what the length of the original video is, video clips of fixed length and number can be obtained according to the above scheme, providing standardized input data for subsequent emotion recognition; on this basis, image batch processing is performed on facial expression data, and the Retinaface algorithm is used to detect the face area of ​​each frame, and the detected face area is adjusted to a uniform size of 224×224 pixels to meet the input requirements of the deep learning model;

[0055] Due to the visual noise interference within the pixels, directly applying the emotion recognition model of the present invention on the original video frame may not be the best choice, because this will make it difficult for the model to obtain a stable and robust expression feature representation; in the present invention, a deep convolutional neural network ResNet-18 model is used to extract features from each frame of the image and generate corresponding feature maps; these feature maps not only remove the visual noise in the original pixels, but also retain key information related to emotional expression.

[0056] In this embodiment, in step 2, the spatial context encoder uses SwinTransformer as the backbone architecture to learn the spatial context information of EEG features and facial features, and ensures that high-resolution data information is not lost during the processing process, which is crucial for capturing subtle changes in the spatial distribution of data. The spatial context features are extracted from the preprocessed EEG features or facial features to obtain a fused spatial context information feature map;

[0057] Considering the low resolution of EEG feature map, the input EEG feature map is divided into non-overlapping patches (small blocks), and the size of each patch is 2×2 pixels; the input face feature map is divided into non-overlapping patches of 4×4 pixels; the patches of EEG feature map and face feature map are converted into feature vectors respectively through linear embedding (LinearEmbedding); specifically, the EEG feature map is convolved by Conv2d(18, 18, kernel_size=(2, 2), stride=(2, 2)) to obtain the EEG feature vector, and the feature map with a window size of 4×4 and a shape of (16, 4, 4, 18) is obtained through reshape operation; the face feature map is convolved by Conv2d(3, 96, kernel_size=(4, 4), stride=(4, 4)) to obtain the face feature vector, and the window size of 7×7 and the shape of (64 , 7, 7, 96) feature map; then the feature map is input into the SwinTransformerBlock of Stage1 for window attention calculation, firstly W-MSA (Multi-headSelf-Attention based on Windows) calculates the self-attention score of each window of the feature map, and then SW-MSA (Multi-headSelf-Attention based on ShiftedWindows) calculates the attention score in the sliding window after the feature map window is offset to capture the information between adjacent windows; then through the PatchMerging module, the output feature map of the previous Stage is downsampled and the number of channels is expanded (the size is reduced to 1 / 2 of the original, and the number of channels is expanded to 2 times the original) to be input into the next Stage; through N Stages, the context information of the feature map is gradually extracted to obtain the EEG and face feature maps that integrate the spatial context information.

[0058] In this embodiment, in step 3, the temporal context encoder is designed based on the Transformer structure to alleviate problems such as face occlusion and abnormal EEG peaks in the time dimension and capture the dynamic characteristics of EEG signals and facial expressions changing over time; in a set of feature map sequences, if the positions of these features are exchanged, the overall meaning and contextual relationship of the sequence may change significantly, resulting in the model being unable to correctly understand and process these sequences; in order to add the position information of the feature sequence, relative position encoding is introduced on the basis of the self-attention mechanism;

[0059] Then, layer normalization (LN) is performed; pre-LN (when LN is applied before the activation function) training is more stable, while post-LN (when LN is applied after the activation function) may face the risk of divergence in the early stage of training, and strategies such as learning rate hot start are needed to ensure the smooth progress of training;

[0060] In order to prevent multiple heads from learning the same attention, a regularization method is added to the multi-head attention mechanism to diversify the multi-head attention in the same layer. On the other hand, the dropout technology is used to randomly suppress the output of some heads, which helps prevent the formation of too similar attention patterns between heads.

[0061] Position-wise FFN (Feed-Forward Network) consists of two fully connected layers and acts on the last dimension of the input sequence. Since the state of each position in the sequence is updated separately, it is called position-wise. FFN accepts the output of self-attention as input to further extract and transform features, thereby enhancing the expressiveness of the model. Regarding the specific implementation and mathematical details of position-wise FFN, it is usually expressed as FFN(x)=max(0,xW1+b1)W2+b2, where W1 and W2 are weight matrices, and b1 and b2 are bias terms. Here, max(0,x) represents the ReLU activation function, which is used to introduce nonlinearity.

[0062] By extracting temporal context features from the EEG feature map or face feature map fused with spatial context information, a fused spatiotemporal context information feature map will be obtained.

[0063] In this embodiment, when discussing multimodal emotion recognition, the present invention observed that there are differences in the dominance of different emotions in different modalities. Some emotions may be mainly dominated by electroencephalogram (EEG), while other emotions may be more reflected through facial expressions. In order to adapt the backbone network of the present invention and make full use of the information of these two modalities, a multimodal attention mechanism is designed. This mechanism is based on the calculated attention weights, that is, in step four, the multimodal fusion module based on the modal attention mechanism performs weighted fusion on the encoding features of EEG and facial expressions, allowing each modality to dynamically adjust its weight according to its contribution to a specific emotion. For emotions dominated by EEG, EEG features will be given a higher weight; and for emotions dominated by facial expressions, facial expression features will be given a greater weight.

[0064] In this embodiment, the calculation method of the multimodal fusion module based on the modal attention mechanism is as follows:

[0065] First, given the output feature X∈Rpre of the EEG or face encoder output by the backbone network, pre represents the original dimension of feature X; this feature X is used as the input of this module and is further processed to obtain multimodal feature representation; the following composite operation is used to obtain a feature group Xn∈Rpre*n with a number of n, where n represents the number of modalities; in the task, n is set to 2, indicating that bimodal data is processed; the composite operation includes applying the dimension alignment convolution operation AlignCov (aligning high-dimensional features to low-dimensional features), the rectified linear unit ReLU and the batch normalization BN, which are helpful for the dimension alignment of feature maps of different modalities and the stability of model training convergence;

[0066] Xn=ReLU(B(AlignCov(X))) (1)

[0067] Subsequently, in order to calculate the modality attention tensor, a fully connected operator Fc is introduced; the fully connected layer can learn the complex relationship between different modality features and output a modality attention weight vector; the weight vector is normalized by applying the softmax operator to obtain the attention score of each modality; these scores reflect the importance of different modalities in the current task; the calculation process of the modality attention tensor can be expressed as:

[0068]

[0069] Finally, the modal attention weighted feature Xout is obtained by element-by-element multiplication (⊙) of the modal attention tensor and the original feature group Xn. This operation weights the features of different modalities according to their attention scores, thereby achieving effective fusion of multimodal information. The mathematical expression is:

[0070] Xout=Sn⊙Xn (4)

[0071] In order to learn cross-modal associations and classify them, Xout is input into a multi-layer perceptron (MLP). The constructed MLP is a three-layer stacked fully connected network, which uses its nonlinear fitting ability to further extract and integrate deep features in multimodal information. Through the layer-by-layer transmission and processing of MLP, the potential associations between different modalities can be captured, and emotion classification tasks can be performed based on these features. Finally, it is processed by the softmax layer to obtain the normalized output probability of each emotion category. The role of the softmax layer is to convert the original output score of the MLP into a probability distribution, ensuring that the probability value of each emotion category is between 0 and 1, and the sum of the probabilities of all categories is strictly equal to 1. This This probability distribution form is helpful for the subsequent loss calculation and interpretation of emotion recognition results; subsequently, the cross entropy loss function is used as the optimization target in the training process. The cross entropy loss function is an effective tool to measure the difference between the probability distribution predicted by the model and the true emotion label distribution; its calculation formula is based on the probability distribution output by the softmax layer and the true emotion label distribution, and the difference is quantified by calculating the negative log-likelihood value between the two; during the training process, the model continuously adjusts its internal parameters to minimize the cross entropy loss function, so that the predicted probability distribution gradually approaches the true emotion label distribution, thereby improving the accuracy and performance of emotion recognition.

[0072] The present invention also discloses a multimodal emotion recognition system, which is implemented by combining B / S (browser / server) and C / S (client / server) architectures. The system is divided into three core components: front-end, back-end and database, to improve the maintainability and scalability of the system.

[0073] The front-end component, as a portal for user interaction, uses C++ combined with the QT framework to build the client interface. At the same time, the Vue.js framework is introduced to implement the back-end administrator data management interface, providing rich operation interfaces and intuitive emotion recognition result display for these two types of users; this component is responsible for receiving user operation instructions and sending task requests to the back-end through the API interface; at the same time, the front-end component is also responsible for receiving the task execution results returned by the back-end and displaying them to the user in a visual way;

[0074] The backend component, as the core of the business logic, uses Python programming language and Django RESTframework to build RESTful API to realize the interaction of front-end and back-end data; the jinja2 template engine is used for page rendering to improve user experience; the backend component receives the task request sent by the frontend, executes the corresponding system management or emotion recognition logic, and calls the database component for data storage and retrieval;

[0075] Database components include MySQL, Redis and SQLite. MySQL, as a relational database management system, is responsible for storing structured data, such as user information, network models, etc. Redis, as an in-memory database, is used to cache hot data and improve system response speed. The database component provides a data access interface to support data operation requests from back-end components. SQLite is used for encrypted storage and management of client local data.

[0076] In this embodiment, it also includes

[0077] At the algorithm level, MATLAB EEGLab and MNE-Python libraries are used to preprocess EEG signals and extract features, and Pillow library is used to preprocess expression images and extract key information;

[0078] Build and manage deep learning models based on Pytorch and Openmmlab deep learning frameworks;

[0079] For service deployment, Nginx is used as the reverse proxy server and uWSGI server is used as the web server.

[0080] Although embodiments of the present invention have been shown and described (see the above detailed description for details), it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal emotion recognition method, characterized in that: The specific steps include: Step 1: Data preprocessing and feature extraction. For emotional EEG, the EEGLAB tool is used to process the EEG data. First, the EEG data is bandpass filtered in the 0.5-50Hz band using a zero-phase-shift FIR filter to eliminate the electrical interference in the 50Hz line noise. The spherical interpolation method is used to interpolate the removed bad electrode positions, and ICA processing is performed to eliminate non-brain-related artifacts to ensure the accuracy of subsequent analysis. The EEG data is downsampled using a sampling frequency of 256 Hz to speed up the calculation speed. After processing the EEG data, the three-dimensional transformation process of the multi-channel EEG will be performed; Step 2: Extract spatial context features from the preprocessed emotional EEG features and facial expression features using a spatial context encoder; Step 3: Use the temporal context encoder to further extract temporal context features; Step 4: Use a multimodal fusion module based on the modal attention mechanism to fuse the spatiotemporal feature representations of different modalities, and finally perform multimodal feature emotion classification.

2. A multimodal emotion recognition method according to claim 1, characterized in that: In the step 1, the three-dimensional transformation process involves converting the signals recorded by multiple EEG electrodes into a topographic map form, wherein the power distribution of each electrode constitutes a specific point on the topographic map.

3. The multimodal emotion recognition method according to claim 1, characterized in that: In the step 2, the spatial context encoder uses SwinTransformer as the backbone architecture to learn the spatial context information of EEG features and facial features, and ensures that high-resolution data information is not lost during the processing process. The spatial context features are extracted from the preprocessed EEG features or facial features to obtain a fused spatial context information feature map.

4. The multimodal emotion recognition method according to claim 1, characterized in that: In the step three, the temporal context encoder is designed based on the Transformer structure to alleviate the problems of face occlusion and abnormal EEG peaks in the time dimension and capture the dynamic characteristics of EEG signals and facial expressions changing over time.

5. The multimodal emotion recognition method according to claim 1, characterized in that: In the step 4, the multimodal fusion module based on the modal attention mechanism performs weighted fusion on the encoding features of EEG and facial expressions, allowing the weight of each modality to be dynamically adjusted according to the contribution of each modality to a specific emotion. For emotions dominated by EEG, EEG features will be given a higher weight; and for emotions dominated by facial expressions, facial expression features will be given a greater weight.

6. A multimodal emotion recognition method according to claim 5, characterized in that: The calculation method of the multimodal fusion module based on the modal attention mechanism is as follows: First, given the output feature X∈Rpre of the EEG or face encoder output by the backbone network, pre represents the original dimension of feature X; this feature X is used as the input of this module and is further processed to obtain multimodal feature representation; the following composite operation is used to obtain a feature group Xn∈Rpre*n with a number of n, where n represents the number of modalities; in the task, n is set to 2, indicating that bimodal data is processed; the composite operation includes applying the dimension alignment convolution operation AlignCov (aligning high-dimensional features to low-dimensional features), rectified linear units ReLU and batch normalization BN, which are helpful for dimensional alignment of feature maps of different modalities and stability of model training convergence; Xn = ReLU(B(AlignCov(X)))(1) Subsequently, in order to calculate the modality attention tensor, a fully connected operator Fc is introduced; the softmax operator is applied to normalize the weight vector to obtain the attention score of each modality; these scores reflect the importance of different modalities in the current task; the calculation process of the modality attention tensor can be expressed as: Sn=softmax(Fc(Xn)),Sn∈R pre*n (2) Finally, the modal attention weighted feature Xout is obtained by element-wise multiplication (⊙) of the modal attention tensor and the original feature group Xn; the mathematical expression is: Xout=Sn(⊙Xn (4) In order to learn cross-modal associations and classify them, Xout is input into a multi-layer perceptron (MLP). The constructed MLP is a three-layer stacked fully connected network, which uses its nonlinear fitting ability to further extract and integrate deep features in multimodal information. Through the layer-by-layer transmission and processing of MLP, the potential associations between different modalities can be captured, and emotion classification tasks can be performed based on these features. Finally, it is processed by the softmax layer to obtain the normalized output probability of each emotion category. The role of the softmax layer is to convert the original output score of the MLP into a probability distribution, ensuring that the probability value of each emotion category is between 0 and 1, and the sum of the probabilities of all categories is strictly equal to 1. Subsequently, the cross-entropy loss function is used as the optimization target in the training process. Its calculation formula is based on the probability distribution of the softmax layer output and the true emotion label distribution. The difference is quantified by calculating the negative log-likelihood value between the two. During the training process, the model continuously adjusts its internal parameters to minimize the cross-entropy loss function, so that the predicted probability distribution gradually approaches the true emotion label distribution, thereby improving the accuracy and performance of emotion recognition.

7. A multimodal emotion recognition system, characterized in that: This system is implemented by combining B / S (browser / server) and C / S (client / server) architectures. The system is divided into three core components: front-end, back-end, and database; The front-end component, as a portal for user interaction, uses C++ combined with the QT framework to build the client interface. At the same time, the Vue.js framework is introduced to implement the back-end administrator data management interface, providing rich operation interfaces and intuitive emotion recognition result display for these two types of users; this component is responsible for receiving user operation instructions and sending task requests to the back-end through the API interface; at the same time, the front-end component is also responsible for receiving the task execution results returned by the back-end and displaying them to the user in a visual way; The backend component, as the core of the business logic, uses Python programming language and Django RESTframework to build RESTful API to realize the interaction of front-end and back-end data; the jinja2 template engine is used for page rendering to improve user experience; the backend component receives the task request sent by the frontend, executes the corresponding system management or emotion recognition logic, and calls the database component for data storage and retrieval; Database components include MySQL, Redis, and SQLite. MySQL, as a relational database management system, is responsible for storing structured data. Redis, as an in-memory database, is used to cache hot data and improve system response speed. The database component provides a data access interface to support data operation requests from back-end components. SQLite is used for encrypted storage and management of local client data.

8. A multimodal emotion recognition system according to claim 7, characterized in that: Also includes At the algorithm level, MATLAB EEGLab and MNE-Python libraries are used to preprocess EEG signals and extract features, and Pillow library is used to preprocess expression images and extract key information; Build and manage deep learning models based on Pytorch and Openmmlab deep learning frameworks; For service deployment, Nginx is used as the reverse proxy server and uWSGI server is used as the web server.

Citation Information

Patent Citations

  • Emotion recognition method based on space-time convolution kernel block

    CN111297380A

  • Emotion recognition method and system based on multi-modal physiological information and deep learning

    CN114041795A

  • Emotion recognition method and device, consciousness detection method and device, equipment and storage medium

    CN115337014A

  • Multi-modal emotion recognition method and device, electronic equipment and storage medium

    CN115359576A

  • Method for multimodal emotion classification based on modal space assimilation and contrastive learning

    US20240119716A1

Cited By

  • Emotion recognition method based on multi-mode collaborative optimization

    CN120892820A

  • A multi-modal collaborative optimization based emotion recognition method

    CN120892820B

  • Emotion recognition method and device based on multi-modal physiological signals

    CN122796675A