Multi-modal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression
By combining multimodal emotion recognition methods with EEG and micro-expression, using the SwinTransformer model and the space-time attention mechanism, the problem of high misdiagnosis rate in the diagnosis of patients with consciousness disorders is solved, and more accurate and comprehensive emotional state assessment is achieved, improving the improvement of The therapeutic effect.
Patent Information
- Application Number
- CN202410970432.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-19
AI Technical Summary
The prior art has a high misdiagnosis rate and lacks sensitivity in the diagnosis of patients with consciousness disorders. Traditional methods cannot accurately capture the complex interactions of emotional states, resulting in inaccurate evaluation results, especially in coma, micro-conscious state and non-responsive awakening syndrome. Diagnosis is more challenging.
Using multimodal emotion recognition method based on EEG and micro-expression, through feature extraction and feature fusion, the SwinTransformer model combines the spatiotemporal attention mechanism, fuses EEG and micro-expression data to form multimodal features to improve emotions Accuracy and comprehensiveness of identification.
It significantly improves the ability to detect and classify emotional states in patients with awareness disorders, provides more accurate and comprehensive emotional state descriptions, reduces the rate of misdiagnosis, helps clinical evaluation and personalized treatment decisions, and improves patients' quality of life and treatment results.
Smart Images

Figure CN118902458A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical fields of artificial intelligence, brain-computer interface and emotion recognition, and specifically relates to a multimodal emotion recognition and consciousness detection method based on electroencephalogram (EEG) and micro-expressions. Background Art
[0002] Emotion recognition is to understand a person's emotional state by analyzing his or her physiological behavior and activities, and involves fields such as psychology and medicine. This technology can not only help understand an individual's emotional experience, but also become an important tool for assessing the state of consciousness. Consciousness detection is a technology that accurately monitors and assesses the state of consciousness of patients with consciousness disorders. It is related to the choice of treatment methods and prognosis of patients. Accurate assessment of the state of consciousness is particularly important for such patients. In recent years, people have conducted extensive research on expressing emotions through facial expressions, gestures, and language. Physiological signals such as electromyography, electrocardiography, and electroencephalography are also objective manifestations of emotions that are difficult to hide. Changes in facial muscles can reflect changes in emotions, and changes in electrical signals of normal brain activity can also reflect changes in various physiological electrical signals generated by brain nerve cells in the cerebral cortex, thereby analyzing changes in emotions.
[0003] At present, in the field of emotion recognition, researchers mainly use single modality data, such as electroencephalogram (EEG), electrocardiogram, electromyogram or facial expressions, gestures and language for emotion recognition. However, there are limitations of single modality methods in emotion detection of patients with consciousness disorders. These methods cannot capture the complex interactions of emotional states, resulting in inaccurate and incomplete evaluation results, making the evaluation results inaccurate. Therefore, more accurate and comprehensive evaluation tools are needed. In the field of consciousness disorders, there are challenges in diagnosing the consciousness state of patients with consciousness disorders, especially in the three types of consciousness disorders, coma, minimally conscious state (MCS) and unresponsive wakefulness syndrome (UWS). The diagnostic challenges are becoming increasingly prominent. Consciousness detection methods for patients with consciousness disorders mainly include clinical assessment methods based on motor responses and functional magnetic resonance imaging (fMRI) technology. These methods also have some shortcomings. Traditional clinical assessment methods, such as the Coma Recovery Scale-Revised (CRS-R) based on motor responses, are not suitable for patients with consciousness disorders. This clinical assessment method has the problems of high misdiagnosis rate and lack of sensitivity. The misdiagnosis rate is 37% to 43% when diagnosing consciousness state, which may lead to some patients failing to obtain accurate consciousness state assessment and appropriate treatment. Therefore, more accurate and comprehensive assessment tools are needed. Functional magnetic resonance imaging (fMRI) can show the brain activity patterns of patients with consciousness disorders in specific activity tasks, but its equipment is expensive and complicated to operate. It requires professional operation and analysis, and its application in clinical settings is limited. It has high technical requirements for operators and analysts. In order to reduce the misdiagnosis rate and improve the accuracy of the assessment, it is necessary to introduce simpler and more effective methods to supplement the consciousness state assessment, so that it is easy to implement in the clinical environment and can provide more accurate consciousness state information.
[0004] In order to reduce the risk of misdiagnosis and improve the accuracy of assessment, it is necessary to introduce more simplified and sensitive methods to supplement consciousness assessment. For patients with unresponsive wakefulness syndrome (UWS), they have no awareness of their surroundings and themselves, while patients in minimally conscious state (MCS) have weaker consciousness. Therefore, it can be assumed that patients who experience emotional changes in a passive paradigm have a better chance of recovery. In recent years, due to the rapid development of computer vision and brain-computer interfaces (BCIs), the accuracy of emotion recognition based on facial images and electroencephalograms has improved. Compared with motor response-based assessment methods and fMRI, multimodal emotion recognition based on electroencephalograms and micro-expressions can more simply diagnose the residual consciousness state of patients with consciousness disorders and serve as a tool for auxiliary diagnosis. Summary of the invention
[0005] The purpose of the present invention is to provide a multimodal emotion recognition and consciousness detection method based on electroencephalogram (EEG) and micro-expressions, so as to solve the problems of high misdiagnosis rate and lack of sensitivity of traditional diagnostic methods in the field of consciousness disorders mentioned in the above background technology.
[0006] To achieve the above object, the present invention provides the following technical solution: a multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression, which specifically comprises the following steps:
[0007] Step 1: Feature extraction, focusing on the correlation of multi-channel EEG signals and extracting micro-expression features, and fusing EEG and micro-expression features through the feature layer to form multimodal features;
[0008] For EEG features, the processing steps are as follows:
[0009] a. Preprocess the original EEG signal, including removing artifacts, filtering and component removal;
[0010] b. After preprocessing, the EEG data of each channel is segmented into non-overlapping 1-second intervals according to the output size to extract EEG features;
[0011] Step 2: Build an emotion recognition model and select SwinTransformer as the backbone network of the model. When customizing SwinTransformer, the model optimizes the patch size and network depth to make them consistent with the dimensions of the feature tensor; this ensures that the STST model meets the requirements of the emotion recognition task and can fully utilize the potential of the SwinTransformer architecture;
[0012] Step 3: Sentiment comparison index, using accuracy as a benchmark to compare the performance of its model with other existing methods, providing a quantitative measure of its effectiveness in sentiment classification;
[0013] Step 4: Experimental dataset: test the emotion recognition accuracy on the public dataset and the multimodal dataset MAHNOB-HCI;
[0014] Step 5: Experimental data preprocessing: public data sets and self-collected data sets were used, and a consistent data preprocessing process was adopted for the two data sets to ensure the comparability and robustness of the results;
[0015] Step 6: Experimental parameter setting, extract DE as the input of EEG modality, extract continuous facial image sequence as the input of micro-expression modality; their input sizes are 56×56×5 respectively; EEG modality represents 5 EEG frequency bands, and micro-expression modality represents 5 facial images that change uniformly and continuously within 0.2 seconds; the feasibility of this method is verified on a public dataset, and then pre-trained on the data of healthy subjects and fine-tuned on the data of MCS patients; finally, the evaluation indicators are calculated on the data of UWS patients.
[0016] As a preferred technical solution of the present invention, in step 1, the EEG signal after passing through the bandpass filter obeys the Gaussian distribution N(μ, σ 2 ), the DE feature of EEG is calculated as follows:
[0017]
[0018] where f(x) is the probability density function of x, σ 2 is the variance of this EEG segment; DE features belong to frequency domain features, and EEG signals usually used for emotion recognition are divided into five different frequency bands: δ (0.1-4 Hz), θ (4-8 Hz), α (8-12 Hz), β (12-30 Hz), and γ (30-45 Hz); in this method, the DE features of each frequency band are calculated separately; the final feature vector of each sample contains 56 channels (or 28 channels × 2) × 56 DE features × 5 frequency bands to ensure that it matches the size of the micro-expression features.
[0019] As a preferred technical solution of the present invention, in the step 1, for the multimodal feature fusion of EEG and micro-expression, a spatiotemporal attention mechanism multimodal feature fusion architecture for EEG and micro-expression data is used to separate the significant features of emotion from the multimodal data stream;
[0020] In the temporal attention module, the input micro-expression feature X me Initially, a 1×1 convolutional layer (denoted as W v ) to generate a transformed feature; then, these features are reshaped to a dimension of C me ×2HW, where H and W represent the height and width of the feature map respectively; another 1×1 convolutional layer W q It is used to generate query features, and then the attention distribution is calculated through the softmax layer; the attention distribution map is multiplied by the transposed feature element to obtain the temporal attention feature map Z ta ; The temporal attention mechanism aims to capture the temporal information in the micro-expression image sequence; it uses attention weights to capture the changes in the same spatial position over time to describe the characteristics of micro-expressions; the formula for this process is as follows:
[0021]
[0022] Among them, W q , W v and W z is a 1×1 convolutional layer, σ1 and σ2 are two tensor reshaping operators, is the matrix dot product operation, ⊙ is the channel multiplication; F sm is the Softmax operator; Including Wz Convolution operation followed by layer regularization and Sigmoid operator;
[0023] At the same time, the spatial attention mechanism processes the input EEG features X in a similar way eeg ; The mechanism calculates the spatial attention distribution through global pooling and reshaping operations; then, this distribution passes through softmax and sigmoid layers to generate the spatial attention feature Z sa ; The spatial attention mechanism is good at capturing the spatial attention information of different channels in the same frequency band of EEG signals; The formula for this process is as follows:
[0024]
[0025] Among them, W q and W v is a 1×1 convolutional layer, σ1 and σ2 are two tensor reshaping operators, is the matrix dot product operation, ⊙ is the channel multiplication, G is the global pooling operator; F sm is the Softmax operator; σ s Consists of a tensor reshape operator and a sigmoid operator.
[0026] As a preferred technical solution in the present invention, in the fusion model of EEG and micro-expression features, the attention mechanism is integrated to adapt to the single modality input X;
[0027] In the model, EEG and micro-expression features are first combined; in order to strengthen feature representation, multiple layers of unimodal input are superimposed on the spatiotemporal attention mechanism; in the domain coordination mechanism, the temporal attention feature Z ta and the spatial attention feature Z sa The nonlinear transformation is performed through the PReLU activation function; PReLU introduces learnable parameters to adjust the slope of the negative part of the input, thereby enhancing the model's ability to learn complex patterns from the data; then, the respective results are multiplied by the added fusion features, and the summation result is processed by the Sigmoid function to output the final fusion feature Z dc ; The calculation formula of the domain coordination mechanism is as follows:
[0028] Z dc_1 =x⊙F sg ((F prl (Z ta )+F prl (Z sa ))⊙(Z ta +Z sa ))
[0029] Z dc_2 =Fsg ((F prl (Z ta )+F prl (Z sa ))⊙(Z ta +Z sa ))
[0030] Among them, Z dc_1 and Z dc_2 are the outputs obtained when inputting multimodal data or two dimensions of EEG and micro-expression; F sg represents the sigmoid operator, F prl represents the PReLU operator;
[0031] The feedforward network design used in the spatiotemporal attention module is based on SwinTransformer, which is suitable for processing EEG signal temporal data and micro-expression spatial data. It can capture the local spatial features in micro-expression data and adapt to the temporal characteristics of EEG signals, thereby realizing multimodal data fusion; in addition, the sliding window mechanism changes the calculation method of self-attention.
[0032] As a preferred technical solution in the present invention, in the step 2, the input of the backbone network is a multimodal feature tensor of size 56×56×n, and then these tensors are segmented and linearly embedded to convert the features into a size suitable for STST input; each module is composed of a multi-layer self-attention mechanism and a multi-layer perceptron (MLP), each module is layer normalized, and then connected by residual connections; the self-attention within the block operates on a local window that moves between layers, so that the model can effectively capture local and global context information.
[0033] As a preferred technical solution of the present invention, in step 3, for the self-collected data set, the accuracy of the model is first evaluated using the data of 10 healthy subjects, which establishes a baseline for the expected emotional response; then, the model is applied to the emotional classification of 21 collected patients; the emotional classification probability of the model is mapped to a continuous emotional score interval from -1 to 1, where -1 represents negative, 0 represents neutral, and 1 represents positive; in order to calculate the overall emotional state score, a weighted method is used, which includes multiplying the predicted probability of each category by its associated label value to obtain a continuous score; the formula representing this process is as follows:
[0034] Score = C neg ×C neg +C neu ×P neu +C pos ×P pos
[0035] Among them, C neg, C neu and C pos Represent the contribution values associated with negative, neutral, and positive emotional states, with values of -1, 0, and 1, respectively; P neg , P neu and P pos Represent the probabilities of negative, neutral, and positive emotions predicted by the deep learning model, respectively; this formula aims to convert the probability output of emotion classification into a continuous score from -1 to 1, providing a more accurate comprehensive assessment of emotional tendencies; in order to evaluate the output results of the model, Euclidean distance and cosine similarity are used as indicators for distance calculation and similarity assessment, respectively; these indicators measure the degree of similarity between the emotional patterns of patients with impaired consciousness and those of healthy people;
[0036] Accuracy is used to evaluate the classification performance of emotion recognition; accuracy is a general measure of the model's ability to correctly classify positive and negative instances in different emotion categories. The definition of the metric is as follows:
[0037]
[0038] Among them, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative; the accuracy score allows quantification of the overall classification accuracy of the model on the entire dataset, providing a global performance indicator; when performing emotion recognition on patients with impaired consciousness, the emotional labels of patients with impaired consciousness cannot be used directly; it is impossible to ensure that the patient has been induced with the corresponding emotions, so traditional accuracy evaluation cannot be performed, and Euclidean distance and cosine similarity are used as measurement criteria;
[0039] First, the Euclidean distance is used to measure the spatial distance between emotional expressions. For each sample, its emotional score is represented as a vector, and the Euclidean distance is calculated to measure the difference between samples. The smaller the Euclidean distance, the closer the emotional intensity.
[0040] Secondly, cosine similarity is used to measure the angular relationship between emotion vectors. Specifically, the closer the cosine similarity value is to 1, the more similar the emotion change trends between the two are. On the contrary, the closer the value is to -1, the more opposite the emotion change trends between the two are. For the emotion score vector A = (a1, a2, ..., a1) of healthy subjects when the induced video is played, n ) and the emotion score vector B = (b1, b2, ..., b n ), the Euclidean distance d and cosine similarity s between them are calculated according to the following formula:
[0041]
[0042] The Euclidean distance is the square root of the sum of the square differences of the corresponding components of two vectors. The value range of the Euclidean distance is [0,2]. The value range of the cosine similarity is [-1,1]. The closer the value is to 1, the more similar the two vectors are, and the closer the value is to -1, the more opposite the two vectors are.
[0043] As a preferred technical solution in the present invention, in the step 4, the data in the self-collected data set includes EEG signals and facial expression images; in order to collect high-quality scalp EEG signals, the scalp EEG signals of 32 electrodes are recorded using a NuAmps amplifier and an EEG cap, and the impedance of the 32 electrodes is kept below 30kΩ; the EEG signals are collected at a sampling rate of 1000Hz and then band-pass filtered (0.1-50Hz); the facial expression images are derived from the video recorded by the camera at 1080p 30fps; the subjects are in a controlled environment with minimal external stimulation; the experimental design includes a passive paradigm of watching videos to induce emotions, including images and audio clips.
[0044] As a preferred technical solution of the present invention, in the step 5, when preprocessing the facial image, several steps are taken to ensure the quality and adaptability of the data; the face image is loaded from the data set and a grayscale image is generated to reduce the computational complexity; histogram equalization is applied to enhance the contrast of the image, thereby improving the image quality, and then the MTCNN model is used to detect the face in the image, and the detected faces are aligned to ensure that they have similar positions and sizes; image normalization is performed to make the mean of the pixel value 0 and the variance 1;
[0045] Preprocessing of EEG signals:
[0046] First, the signal was filtered using a 0.1 to 60 Hz bandpass filter and a 50 Hz notch filter to remove noise. It was also checked whether the variance of the lead was too large and whether it contained too many zeros. It was also checked whether the data acquisition of each lead was normal. If there were leads with abnormal acquisition, the average value of the channels around the bad lead was used for interpolation. Independent component analysis (ICA) was used to identify and remove noise from the electrooculogram and electromyography signals. The EEG signals were also standardized to ensure that the amplitude of each channel was consistent.
[0047] As a preferred technical solution in the present invention, in the experiment of step six, the training set and the test set are divided into a ratio of 4:1, and the emotion recognition model is trained using a five-fold cross-validation strategy; the model is implemented using the PyTorch framework, and trained using the AdamW optimizer with a learning rate of 3e-4, and the learning rate decay uses the cosine annealing method; the batch size used for model training is 32, and the model is trained 100 times.
[0048] Compared with the prior art, the present invention has the following beneficial effects:
[0049] Emotion recognition is performed through the multimodal fusion features of EEG and micro-expressions, and the distance is used to measure the gap between the emotional expressions of patients with impaired consciousness and normal people to distinguish the degree of consciousness, so as to assist the detection of consciousness. For multimodal emotion recognition, the present invention proposes a multimodal feature fusion method, which fuses EEG and micro-expression data through an innovative spatiotemporal attention mechanism, and introduces emotion recognition technology into the assessment of consciousness state. This method uses deep learning technology to fuse the differential entropy features of EEG and the time domain features of micro-expressions, which significantly improves the detection and classification capabilities of emotional states. The multimodal method combining EEG and micro-expressions provides a more comprehensive and in-depth understanding of the emotional state of patients with impaired consciousness. EEG is a direct reflection of brain activity, and micro-expressions provide a more comprehensive and accurate description of the emotional state of patients with impaired consciousness through tiny facial change information. This method not only reaches the level of mainstream models in terms of accuracy, but also demonstrates its potential for wide application in medical diagnosis, provides a basis for personalized patient care, and opens up new avenues for clinical practice research. The results of practical applications showed that patients whose mood swings were more similar to those of normal people had higher brain activity levels, which indicated better treatment outcomes, which was consistent with the clinical results one month later. This method effectively merges data from multiple modalities and establishes a new paradigm for emotion recognition in assisted detection of impaired consciousness, which helps to provide more accurate clinical assessments, timely adjust treatment plans for patients, and improve patients' quality of life and treatment outcomes.
[0050] (1) In this paper, an innovative multimodal fusion framework is proposed to combine EEG and micro-expression data to achieve the detection and classification of emotional states through deep learning technology. The key to multimodal fusion in the framework is to propose a spatiotemporal attention module for multimodal fusion, which can independently process the spatial and temporal data from EEG and micro-expressions, and fuse EEG and micro-expression data at the feature level through a spatiotemporal attention mechanism based on polarized attention and domain fusion to improve the accuracy of emotion recognition.
[0051] (2) This method combines low-dimensional artificial features of EEG and micro-expressions with high-dimensional deep learning features to improve the accuracy of emotion recognition. The low-dimensional artificial features extract more relevant information through the spatiotemporal attention module and generate high-dimensional features through the neural network. This hybrid method can combine specific low-dimensional features with complex high-dimensional features to improve the accuracy of emotion recognition.
[0052] (3) The present invention proposes a new multimodal method combining EEG and micro-expression to assess the state of consciousness, providing a more comprehensive and accurate description of the emotional state of patients with impaired consciousness, and providing a basis for personalized medical and nursing decisions for health care professionals. Through multimodal feature fusion and deep learning methods, an evaluation index for assessing the emotional expression of patients with impaired consciousness is established, providing a new quantitative means for assessing the state of consciousness, and also providing a new method and concept for introducing multimodal emotion recognition into clinical practice in the field of consciousness detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 It is a framework diagram of the multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression of the present invention;
[0054] Figure 2 This is a diagram of the multimodal feature fusion architecture of the present invention using a spatiotemporal attention mechanism for EEG and micro-expression data;
[0055] Figure 3 This is a feature fusion architecture diagram of the present invention using a spatiotemporal attention mechanism for multimodal features. DETAILED DESCRIPTION
[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0057] See also Figures 1 to 3 The present invention provides a technical solution: a multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression, which specifically includes the following steps:
[0058] Step 1: Feature extraction, focusing on the correlation of multi-channel EEG signals and extracting micro-expression features, and fusing EEG and micro-expression features through the feature layer to form multimodal features;
[0059] For EEG features, the processing steps are as follows:
[0060] a. Preprocess the original EEG signal, including removing artifacts, filtering and component removal;
[0061] b. After preprocessing, the EEG data of each channel is segmented into non-overlapping 1-second intervals according to the output size to extract EEG features. Previous studies have confirmed that differential entropy (DE) is better than other artificial features in emotion recognition tasks;
[0062] Step 2: Build an emotion recognition model. Swin Transformer is selected as the backbone network of the model because it has shown excellent performance, adaptability and scalability in various vision-related tasks when processing multimodal data. It innovatively uses shifted windows to simulate local and global interactions, and has obvious advantages in processing the complexity of EEG features and micro-expression data, which is crucial in the research. When customizing Swin Transformer, the model optimizes the patch size and network depth to make it consistent with the dimension of the feature tensor, ensuring that the STST model meets the requirements of the emotion recognition task and can give full play to the potential of the Swin Transformer architecture.
[0063] Step 3: Sentiment comparison index, using accuracy as a benchmark to compare the performance of its model with other existing methods, providing a quantitative measure of its effectiveness in sentiment classification;
[0064] Step 4: Experimental dataset. On the public dataset, the emotion recognition accuracy is tested on the multimodal dataset MAHNOB-HCI. MAHNOB-HCI is a dataset focused on multimodal emotion recognition. The dataset consists of 30 healthy subjects, including head-mounted camera videos of different subjects and corresponding 32-channel EEG signals. The dataset provides detailed facial expressions and EEG data, which can deeply explore multimodal emotion recognition.
[0065] Step 5: Experimental data preprocessing: public data sets and self-collected data sets were used, and a consistent data preprocessing process was adopted for the two data sets to ensure the comparability and robustness of the results;
[0066] Step 6: Experimental parameter setting, extract DE as the input of EEG modality, extract continuous facial image sequence as the input of micro-expression modality; their input sizes are 56×56×5 respectively; EEG modality represents 5 EEG frequency bands, and micro-expression modality represents 5 facial images that change uniformly and continuously within 0.2 seconds; the feasibility of the method was verified on a public dataset, and then pre-trained on the data of healthy subjects and fine-tuned on the data of MCS patients; finally, the evaluation indicators were calculated on the data of UWS patients.
[0067] In this embodiment, in step 1, the EEG signal after passing through the bandpass filter obeys the Gaussian distribution N(μ, σ 2 ), the DE feature of EEG is calculated as follows:
[0068]
[0069] where f(x) is the probability density function of x, σ 2is the variance of this EEG segment; DE features belong to frequency domain features. EEG signals usually used for emotion recognition are divided into five different frequency bands: δ (0.1-4 Hz), θ (4-8 Hz), α (8-12 Hz), β (12-30 Hz) and γ (30-45 Hz); in the method, the DE features of each frequency band are calculated separately; the final feature vector of each sample contains 56 channels (or 28 channels × 2) × 56 DE features × 5 frequency bands to ensure matching with the size of micro-expression features;
[0070] A micro-expression is a brief, involuntary facial expression that usually lasts from 1 / 25 second to 1 / 5 second. For micro-expression features, a new method for extracting micro-expression features is used. After exporting the raw images of the video frames, the MTCNN model is used for face detection and alignment. To ensure efficient processing, five facial images are selected with a time window of 1 / 5 second, evenly distributed throughout the duration of the micro-expression, so that the basic dynamics of the facial expression can be captured within the limited time of the micro-expression. Each image is resized to a consistent 56×56 pixel size and then stacked to form a 5×56×56 feature tensor. This tensor reduces computational requirements while retaining key details required for effective emotion recognition. By generating micro-expression features in the feature fusion stage, key information about micro-expressions is ensured to be encapsulated, which helps to improve the accuracy of the analysis and computational efficiency.
[0071] In this embodiment, in step 1, for the multimodal feature fusion of EEG and micro-expression, a spatiotemporal attention mechanism multimodal feature fusion architecture for EEG and micro-expression data is used to separate the significant features of emotion from the multimodal data stream;
[0072] In the temporal attention module, the input micro-expression feature X me Initially, a 1×1 convolutional layer (denoted as W v ) to generate a transformed feature; then, these features are reshaped to a dimension of C me ×2HW, where H and W represent the height and width of the feature map respectively; another 1×1 convolutional layer W q It is used to generate query features, and then the attention distribution is calculated through the softmax layer; the attention distribution map is multiplied by the transposed feature element to obtain the temporal attention feature map Z ta ; The temporal attention mechanism aims to capture the temporal information in the micro-expression image sequence; it uses attention weights to capture the changes in the same spatial position over time to describe the characteristics of micro-expressions; the formula for this process is as follows:
[0073]
[0074] Among them, W q , Wv and W z is a 1×1 convolutional layer, σ1 and σ2 are two tensor reshaping operators, is the matrix dot product operation, ⊙ is the channel multiplication; F sm is the Softmax operator; Including W z Convolution operation followed by layer regularization and Sigmoid operator;
[0075] At the same time, the spatial attention mechanism processes the input EEG features X in a similar way eeg ; The mechanism calculates the spatial attention distribution through global pooling and reshaping operations; then, this distribution passes through softmax and sigmoid layers to generate the spatial attention feature Z sa ; The spatial attention mechanism is good at capturing the spatial attention information of different channels in the same frequency band of EEG signals; The formula for this process is as follows:
[0076]
[0077] Among them, W q and W v is a 1×1 convolutional layer, σ1 and σ2 are two tensor reshaping operators, is the matrix dot product operation, ⊙ is the channel multiplication, G is the global pooling operator; F sm is the Softmax operator; σ s Consists of a tensor reshape operator and a sigmoid operator.
[0078] In this embodiment, in the fusion model of EEG and micro-expression features, the attention mechanism is integrated to adapt to the single modality input X;
[0079] In the model, EEG and micro-expression features are first combined; in order to strengthen feature representation, multiple layers of unimodal input are superimposed on the spatiotemporal attention mechanism; in the domain coordination mechanism, the temporal attention feature Z ta and the spatial attention feature Z sa The nonlinear transformation is performed through the PReLU activation function; PReLU introduces learnable parameters to adjust the slope of the negative part of the input, thereby enhancing the model's ability to learn complex patterns from the data; then, the respective results are multiplied by the added fusion features, and the summation result is processed by the Sigmoid function to output the final fusion feature Z dc ; The calculation formula of the domain coordination mechanism is as follows:
[0080] Z dc_1 =x⊙F sg ((F prl (Z ta)+F prl (Z sa ))⊙(Z ta +Z sa ))
[0081] Z dc_2 =F sg ((F prl (Z ta )+F prl (Z sa ))⊙(Z ta +Z sa ))
[0082] Among them, Z dc_1 and Z dc_2 are the outputs obtained when inputting multimodal data or two dimensions of EEG and micro-expression; F sg represents the sigmoid operator, F prl represents the PReLU operator;
[0083] The feedforward network design used in the spatiotemporal attention module is based on SwinTransformer, which is suitable for processing EEG signal temporal data and micro-expression spatial data. It can capture local spatial features in micro-expression data and adapt to the temporal features of EEG signals, thereby realizing multimodal data fusion. In addition, the sliding window mechanism changes the calculation method of self-attention and improves the calculation efficiency, which is particularly important when processing multimodal data with complex features. This method enables the spatiotemporal attention mechanism to focus on the spatial and temporal information in the EEG and micro-expression signals respectively, and can capture the interactions and associations between different modalities. It can effectively extract shared features between different modalities and extract features that best express emotional states, thereby improving the accuracy of emotion recognition.
[0084] In this embodiment, in step 2, the input of the backbone network is a multimodal feature tensor of size 56×56×n, and then these tensors are split and linearly embedded to convert the features into a size suitable for STST input; each module consists of a multi-layer self-attention mechanism and a multi-layer perceptron (MLP), each module is layer-normalized, and then all are connected by residual connections; the self-attention within the block operates on a local window that moves between layers, so that the model can effectively capture local and global contextual information; this hierarchical processing enables the network to effectively integrate spatiotemporal information from facial images and EEG data, thereby generating a semantically rich feature set with high discrimination for emotion recognition; MLP integrates the features into probability scores corresponding to positive, neutral and negative emotion categories respectively; this end-to-end trainable system ensures that the classification process can utilize both spatiotemporal patterns and refined features obtained from the Swin Transformer module, ultimately forming a highly accurate and robust emotion recognition model.
[0085] In this embodiment, in step 3, for the self-collected data set, the accuracy of the model is first evaluated using the data of 10 healthy subjects, which establishes a baseline for the expected emotional response; then, the model is applied to the emotional classification of 21 patients collected; the emotional classification probability of the model is mapped to a continuous emotional score interval from -1 to 1, where -1 represents negative, 0 represents neutral, and 1 represents positive; in order to calculate the overall emotional state score, a weighted method is used, which includes multiplying the predicted probability of each category by its associated label value to obtain a continuous score; the formula representing this process is as follows:
[0086] Score = C neg ×P neg +C neu ×P neu +C pos ×P pos
[0087] Among them, C neg , C neu and C pos Represent the contribution values associated with negative, neutral, and positive emotional states, with values of -1, 0, and 1, respectively; P neg , P neu and P pos Represent the probabilities of negative, neutral, and positive emotions predicted by the deep learning model, respectively; this formula aims to convert the probability output of emotion classification into a continuous score from -1 to 1, providing a more accurate comprehensive assessment of emotional tendencies; in order to evaluate the output results of the model, Euclidean distance and cosine similarity are used as indicators for distance calculation and similarity assessment, respectively; these indicators measure the degree of similarity between the emotional patterns of patients with impaired consciousness and those of healthy people;
[0088] Accuracy is used to evaluate the classification performance of emotion recognition; accuracy is a general measure of the model's ability to correctly classify positive and negative instances in different emotion categories. The definition of the metric is as follows:
[0089]
[0090] Among them, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative; the accuracy score allows quantification of the overall classification accuracy of the model on the entire dataset, providing a global performance indicator; when performing emotion recognition on patients with impaired consciousness, the emotional labels of patients with impaired consciousness cannot be used directly; it is impossible to ensure that the patient has been induced with the corresponding emotions, so traditional accuracy evaluation cannot be performed, and Euclidean distance and cosine similarity are used as measurement criteria; these two indicators can quantify the similarity of emotional expressions between healthy subjects and patients with impaired consciousness, which provides a feasible performance evaluation method;
[0091] First, the Euclidean distance is used to measure the spatial distance between emotional expressions. For each sample, its emotional score is represented as a vector, and the Euclidean distance is calculated to measure the difference between samples. The smaller the Euclidean distance, the closer the emotional intensity.
[0092] Secondly, cosine similarity is used to measure the angular relationship between emotion vectors. Specifically, the closer the cosine similarity value is to 1, the more similar the emotion change trends between the two are. On the contrary, the closer the value is to -1, the more opposite the emotion change trends between the two are. For the emotion score vector A = (a1, a2, ..., a1) of healthy subjects when the induced video is played, n ) and the emotion score vector B = (b1, b2, ..., b n ), the Euclidean distance d and cosine similarity s between them are calculated according to the following formula:
[0093]
[0094] The Euclidean distance is the square root of the sum of the square differences of the corresponding components of two vectors. The value range of the Euclidean distance is [0,2]. The value range of the cosine similarity is [-1,1]. The closer the value is to 1, the more similar the two vectors are, and the closer the value is to -1, the more opposite the two vectors are.
[0095] In this embodiment, in step 4, the data in the self-collected data set include EEG signals and facial expression images; in order to collect high-quality scalp EEG signals, scalp EEG signals of 32 electrodes were recorded using NuAmps amplifiers and EEG caps, and the impedance of the 32 electrodes was kept below 30kΩ; EEG signals were collected at a sampling rate of 1000Hz and then band-pass filtered (0.1-50Hz); facial expression images were derived from videos recorded by cameras at 1080p 30fps; the subjects were in a controlled environment with minimal external stimulation; the experimental design included a passive paradigm of watching videos to induce emotions, including images and audio clips; there were 31 subjects in the data set, including 10 healthy subjects from South China Normal University and 21 patients with impaired consciousness (including 13 MCS patients and 8 UWS patients) from Zhujiang Hospital of Southern Medical University; each participant or their family signed an informed consent form before participating in the study to ensure that they understood the nature, purpose and potential risks of the experiment; the study was approved by the hospital ethics committee, and all experimental procedures met ethical standards;
[0096] Table 1 Comparison of emotion recognition accuracy on the MAHNOB-HCI dataset
[0097]
[0098]
[0099] In the experiment, participants completed a series of emotion induction tasks; the experiment involved watching 6 video clips that evoked 3 emotions: positive, neutral, and negative; the video clips used in the experiment are shown in Table 1. During the experiment, the subjects' EEG and facial images were recorded; in order to protect the privacy of the participants, the dataset was de-identified; all personally identifiable information was deleted to ensure the privacy and confidentiality of the data.
[0100] In this embodiment, in step five, when preprocessing the facial image, several steps are taken to ensure the quality and adaptability of the data; the face image is loaded from the data set and a grayscale image is generated to reduce the computational complexity; histogram equalization is applied to enhance the contrast of the image to improve the image quality, and then the MTCNN model is used to detect the face in the image, and the detected faces are aligned to ensure that they have similar positions and sizes; image normalization is performed to make the mean of the pixel value 0 and the variance 1;
[0101] Preprocessing of EEG signals:
[0102] First, the signal is filtered using a 0.1 to 60 Hz bandpass filter and a 50 Hz notch filter to remove noise. It is also checked whether the variance of the lead is too large and contains too many zeros, and whether the data acquisition of each lead is normal. If there are leads with abnormal acquisition, the average value of the channels around the bad lead is used for interpolation. Independent component analysis (ICA) is used to identify and remove noise in the electrooculogram and electromyography signals. The EEG signals are also standardized to ensure that the amplitude of each channel is consistent. These preprocessing steps lay a solid foundation for manual feature extraction and emotion recognition.
[0103] In this embodiment, in the experiment of step six, the training set and the test set are divided into a ratio of 4:1, and the emotion recognition model is trained using a five-fold cross-validation strategy; the model is implemented using the PyTorch framework, and is trained using the AdamW optimizer with a learning rate of 3e-4, and the learning rate decay uses the cosine annealing method; the batch size used for model training is 32, and the model is trained 100 times.
[0104] exist Figure 1 In the paper, we start with feature extraction. The raw EEG data is processed through an EEG preprocessing step to extract relevant features, and the facial images are cropped using the MTCNN model. The features extracted from both modalities are then fed into an emotion recognition model that includes a spatiotemporal attention encoder (STAE) to fuse multimodal features into a cohesive representation. Swin-T is used as the backbone network. Emotions are classified by an MLP layer that outputs the probability of positive, neutral, or negative emotions. When applying emotion recognition to consciousness detection, the emotions obtained from the classification of patients with impaired consciousness are used to calculate distance and similarity, which helps detect impaired consciousness by comparing the gap between impaired consciousness patients and healthy people.
[0105] Although embodiments of the present invention have been shown and described (see the above detailed description for details), it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression, characterized in that: The specific steps include: Step 1: Feature extraction, focusing on the correlation of multi-channel EEG signals and extracting micro-expression features, and fusing EEG and micro-expression features through the feature layer to form multimodal features; For EEG features, the processing steps are as follows: a. Preprocess the original EEG signal, including removing artifacts, filtering and component removal; b. After preprocessing, the EEG data of each channel is segmented into non-overlapping 1-second intervals according to the output size to extract EEG features; Step 2: Build an emotion recognition model and select Swin Transformer as the backbone network of the model. When customizing Swin Transformer, the model optimizes the patch size and network depth to make them consistent with the dimensions of the feature tensor; this ensures that the STST model meets the requirements of the emotion recognition task and can fully utilize the potential of the Swin Transformer architecture; Step 3: Sentiment comparison index, using accuracy as a benchmark to compare the performance of its model with other existing methods, providing a quantitative measure of its effectiveness in sentiment classification; Step 4: Experimental dataset: test the emotion recognition accuracy on the public dataset and the multimodal dataset MAHNOB-HCI; Step 5: Experimental data preprocessing: public data sets and self-collected data sets were used, and a consistent data preprocessing process was adopted for the two data sets to ensure the comparability and robustness of the results; Step 6: Experimental parameter setting, extract DE as the input of EEG modality, extract continuous facial image sequence as the input of micro-expression modality; their input sizes are 56×56×5 respectively; EEG modality represents 5 EEG frequency bands, and micro-expression modality represents 5 facial images that change uniformly and continuously within 0.2 seconds; the feasibility of this method is verified on a public dataset, and then pre-trained on the data of healthy subjects and fine-tuned on the data of MCS patients; finally, the evaluation indicators are calculated on the data of UWS patients.
2. The multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression according to claim 1, characterized in that: In step 1, the EEG signal after the bandpass filter follows the Gaussian distribution N(μ, σ 2 ), the DE feature of EEG is calculated as follows: where f(x) is the probability density function of x, σ 2 is the variance of this EEG segment; DE features belong to frequency domain features. EEG signals usually used for emotion recognition are divided into five different frequency bands: δ (0.1-4 Hz), θ (4-8 Hz), α (8-12 Hz), β (12-30 Hz) and γ (30-45 Hz). In this method, the DE features of each frequency band are calculated separately. The final feature vector of each sample contains 56 channels (or 28 channels × 2) × 56 DE features × 5 frequency bands to ensure that it matches the size of the micro-expression features.
3. The multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression according to claim 1, characterized in that: In the step 1, for the multimodal feature fusion of EEG and micro-expression, a spatiotemporal attention mechanism multimodal feature fusion architecture for EEG and micro-expression data is used to separate the significant features of emotion from the multimodal data stream; In the temporal attention module, the input micro-expression feature X me Initially, a 1×1 convolutional layer (denoted as W v ) to generate a transformed feature; then, these features are reshaped to a dimension of C me ×2HW, where H and W represent the height and width of the feature map respectively; another 1×1 convolutional layer W q It is used to generate query features, and then the attention distribution is calculated through the softmax layer; the attention distribution map is multiplied by the transposed feature element to obtain the temporal attention feature map Z ta ; The temporal attention mechanism aims to capture the temporal information in the micro-expression image sequence; it uses attention weights to capture the changes in the same spatial position over time to describe the characteristics of micro-expressions; the formula for this process is as follows: Among them, W q , W v and W z is a 1×1 convolutional layer, σ1 and σ2 are two tensor reshaping operators, is the matrix dot product operation, ⊙ is the channel multiplication; F sm is the Softmax operator; Including W z Convolution operation followed by layer regularization and Sigmoid operator; At the same time, the spatial attention mechanism processes the input EEG features X in a similar way eeg ; The mechanism calculates the spatial attention distribution through global pooling and reshaping operations; then, this distribution passes through softmax and sigmoid layers to generate the spatial attention feature Z sa ; The spatial attention mechanism is good at capturing the spatial attention information of different channels in the same frequency band of EEG signals; The formula for this process is as follows: Among them, W q and W v is a 1×1 convolutional layer, σ1 and σ2 are two tensor reshaping operators, is the matrix dot product operation, ⊙ is the channel multiplication, G is the global pooling operator; F sm is the Softmax operator; σ s Consists of a tensor reshape operator and a sigmoid operator.
4. The multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression according to claim 3, characterized in that: In the fusion model of EEG and micro-expression features, the attention mechanism is integrated to adapt to the single modality input X; In the model, EEG and micro-expression features are first combined; in order to strengthen feature representation, multiple layers of unimodal input are superimposed on the spatiotemporal attention mechanism; in the domain coordination mechanism, the temporal attention feature Z ta and the spatial attention feature Z sa Nonlinear transformation is performed through PReLU activation function; PReLU enhances the model’s ability to learn complex patterns from data by introducing learnable parameters to adjust the slope of the negative part of the input. Then, the respective results are multiplied by the added fusion features, and the summation result is processed by the Sigmoid function to output the final fusion feature Z dc ; The calculation formula of the domain coordination mechanism is as follows: WITH dc_1 =x⊙F sg ((F prl (WITH ta )+F prl (WITH sa ))⊙(Z ta +Z sa )) Z dc_2 =F sg ((F prl (Z ta )+F prl (Z sa ))⊙(Z ta +Z sa )) Among them, Z dc_1 and Z dc_2 are the outputs obtained when inputting multimodal data or two dimensions of EEG and micro-expression; F sg represents the sigmoid operator, F prl represents the PReLU operator; The feedforward network design used in the spatiotemporal attention module is based on SwinTransformer, which is suitable for processing EEG signal temporal data and micro-expression spatial data. It can capture the local spatial features in micro-expression data and adapt to the temporal characteristics of EEG signals, thereby realizing multimodal data fusion; in addition, the sliding window mechanism changes the calculation method of self-attention.
5. The multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression according to claim 1, characterized in that: In the step 2, the input of the backbone network is a multimodal feature tensor of size 56×56×n, and then these tensors are split and linearly embedded to convert the features into a size suitable for STST input; each module consists of a multi-layer self-attention mechanism and a multi-layer perceptron (MLP), each module is layer normalized, and then connected by residual connections; the self-attention within the block operates on a local window that moves between layers, enabling the model to effectively capture local and global contextual information.
6. The multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression according to claim 1, characterized in that: In step 3, for the self-collected data set, the accuracy of the model was first evaluated using data from 10 healthy subjects, which established a baseline for the expected emotional response. Subsequently, the model was applied to the emotional classification of 21 patients collected. The emotional classification probability of the model was mapped to a continuous emotional score interval from -1 to 1, where -1 represents negative, 0 represents neutral, and 1 represents positive. In order to calculate the overall emotional state score, a weighted method was used, which included multiplying the predicted probability of each category by its associated label value to obtain a continuous score. The formula representing this process is as follows: Score=C neg ×P neg +C neu ×P neu +C pos ×P pos Among them, C neg , C neu and C pos Represent the contribution values associated with negative, neutral, and positive emotional states, with values of -1, 0, and 1, respectively; P neg , P neu and P pos Represent the probabilities of negative, neutral, and positive emotions predicted by the deep learning model, respectively; this formula aims to convert the probability output of emotion classification into a continuous score from -1 to 1, providing a more accurate comprehensive assessment of emotional tendencies; in order to evaluate the output results of the model, Euclidean distance and cosine similarity are used as indicators for distance calculation and similarity assessment, respectively; these indicators measure the degree of similarity between the emotional patterns of patients with impaired consciousness and those of healthy people; Accuracy is used to evaluate the classification performance of emotion recognition; accuracy is a general measure of the model's ability to correctly classify positive and negative instances in different emotion categories. The definition of the metric is as follows: Among them, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false negative; the accuracy score allows quantification of the overall classification accuracy of the model on the entire dataset, providing a global performance indicator; when performing emotion recognition on patients with impaired consciousness, the emotional labels of patients with impaired consciousness cannot be used directly; it is impossible to ensure that the patient has been induced with the corresponding emotions, so traditional accuracy evaluation cannot be performed, and Euclidean distance and cosine similarity are used as measurement criteria; First, the Euclidean distance is used to measure the spatial distance between emotional expressions. For each sample, its emotional score is represented as a vector, and the Euclidean distance is calculated to measure the difference between samples. The smaller the Euclidean distance, the closer the emotional intensity. Secondly, cosine similarity is used to measure the angular relationship between emotion vectors. Specifically, the closer the cosine similarity value is to 1, the more similar the emotion change trends between the two are. On the contrary, the closer the value is to -1, the more opposite the emotion change trends between the two are. For the emotion score vector A = (a1, a2, ..., a1) of healthy subjects when the induced video is played, n ) and the emotion score vector B = (b1, b2, ..., b n ), the Euclidean distance d and cosine similarity s between them are calculated according to the following formula: The Euclidean distance is the square root of the sum of the square differences of the corresponding components of two vectors. The value range of the Euclidean distance is [0,2]. The value range of the cosine similarity is [-1,1]. The closer the value is to 1, the more similar the two vectors are, and the closer the value is to -1, the more opposite the two vectors are.
7. The multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression according to claim 1, characterized in that: In the step 4, the data in the self-collected data set include EEG signals and facial expression images; in order to collect high-quality scalp EEG signals, the scalp EEG signals of 32 electrodes were recorded using a NuAmps amplifier and an EEG cap, and the impedance of the 32 electrodes was kept below 30 kΩ; the EEG signals were collected at a sampling rate of 1000 Hz and then band-pass filtered (0.1-50 Hz); the facial expression images were derived from the video recorded by the camera at 1080p 30fps; the subjects were in a controlled environment with minimal external stimulation; the experimental design included a passive paradigm of watching videos to induce emotions, including images and audio clips.
8. The multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression according to claim 1, characterized in that: In the step 5, several steps are taken to ensure the quality and adaptability of the data when preprocessing the facial image; the face image is loaded from the data set and a grayscale image is generated to reduce the computational complexity; histogram equalization is applied to enhance the contrast of the image to improve the image quality, and then the MTCNN model is used to detect the face in the image, and the detected faces are aligned to ensure that they have similar positions and sizes; image normalization is performed to make the mean of the pixel value 0 and the variance 1; Preprocessing of EEG signals: First, the signal was filtered using a 0.1 to 60 Hz bandpass filter and a 50 Hz notch filter to remove noise. It was also checked whether the variance of the lead was too large and whether it contained too many zeros. It was also checked whether the data acquisition of each lead was normal. If there were leads with abnormal acquisition, the average value of the channels around the bad lead was used for interpolation. Independent component analysis (ICA) was used to identify and remove noise from the electrooculogram and electromyography signals. The EEG signals were also standardized to ensure that the amplitude of each channel was consistent.
9. The multimodal emotion recognition and consciousness detection method based on electroencephalogram and micro-expression according to claim 1, characterized in that: In the experiment of step six, the training set and the test set are divided into a ratio of 4:1, and the emotion recognition model is trained using a five-fold cross-validation strategy; the model is implemented using the PyTorch framework and trained using the AdamW optimizer with a learning rate of 3e-4, and the learning rate decay uses the cosine annealing method; the batch size used for model training is 32, and the model is trained 100 times.
Citation Information
Patent Citations
Face emotion recognition method and device, compute device and storage medium
CN109190487A
Electroencephalogram emotion recognition method based on attention mechanism
CN110610168A
Potential emotion recognition method based on EEG, BVP and micro-expression
CN111797747A
Multi-modal emotion analysis method and system based on deep learning for acupuncture
CN111881812A
Horse movement monitoring method and system based on electroencephalogram
CN117204366A
Cited By
Teacher anxiety understanding reasoning method based on brain vision multi-mode large model representation
CN119170281A
Induction control glasses for human body signal recognition and induction control method
CN119738967A
Electroencephalogram dynamic continuous emotion recognition method, system and equipment based on channel selection
CN119837531A
Electroencephalogram emotion recognition method based on deep learning
CN120514387A
A deep learning-based EEG emotion recognition method
CN120514387B
