Multi-view fusion emotion recognition method under sleep deprivation condition

Abstract

The invention discloses a multi-view fusion emotion recognition method under a sleep deprivation condition, and the method comprises the steps: constructing and training a recognition model which comprises a Transform network based on multi-view fusion in an offline stage, carrying out the multi-view fusion of an input signal through employing the trained recognition model in an online stage, and achieving the real-time detection. According to the method, the accuracy of emotion recognition under a single mode (EEG signal) is improved by introducing the multi-view fusion Transform model, the dependence on multi-mode data is reduced, and the influence of the sleep state on the emotion is evaluated more accurately by improving the experimental design and eliminating the interference of material fixation on the result.

Description

Technical Field

[0001] The present invention relates to a technology in the field of electroencephalogram signal processing, specifically to an emotion recognition method based on multi-view fusion under sleep deprivation conditions. Background Art

[0002] In the field of emotion recognition, traditional technologies mostly rely on multimodal data, such as combining facial expressions, voice signals and physiological signals, to improve the accuracy of emotion recognition. Typical methods include convolutional neural networks (CNN), long short-term memory networks (LSTM) and multimodal fusion models, which have been widely used in emotion recognition and classification. Among them, there are few studies on single modality, such as using only electroencephalogram (EEG) signals for emotion recognition. Although the method based on multimodal data has superior performance, it faces problems such as complex data collection and expensive equipment. Summary of the invention

[0003] In view of the uncontrollable training accuracy caused by the complexity of multimodal data collection in the prior art and the insufficient mining and utilization of single modal data, the present invention proposes a multi-perspective fusion emotion recognition method under sleep deprivation conditions. By introducing a multi-perspective fusion Transformer model, the accuracy of emotion recognition under a single modality (EEG signal) is improved, and the dependence on multimodal data is reduced. By improving the experimental design, the interference of material fixation on the results is eliminated, thereby more accurately evaluating the impact of sleep state on emotion.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a multi-view fusion emotion recognition method under sleep deprivation conditions. A recognition model based on a multi-view fusion Transformer network is constructed and trained in an offline stage, and the trained recognition model is used to perform multi-view fusion on input signals in an online stage to achieve real-time detection.

[0006] The recognition model includes: a Transformer encoder and a classifier, wherein: the Transformer encoder learns the key features of each perspective through a self-attention mechanism, and obtains the final sentiment classification result through multi-perspective weighted fusion; the classifier inputs the fused features into a fully connected layer for sentiment classification, and uses a Softmax function to output the probability distribution of each sentiment category.

[0007] The training described above uses EEG signals and obtains differential entropy features as a sample set after preprocessing.

[0008] The preprocessing refers to: after bandpass filtering and noise reduction processing of the EEG signal, using short-time Fourier transform to extract differential entropy features (DE) from different frequency bands of each electrode.

[0009] The multi-view fusion refers to rearranging and embedding the input EEG data in three dimensions: time, spectrum and space, calculating the attention weights of different viewpoints respectively, and fusing the features of these viewpoints through a multi-head attention mechanism. Technical Effects

[0010] The present invention significantly improves the efficiency of emotion recognition through multi-view fusion technology based on unimodal EEG signals, fits the effect of multimodal data with unimodal data with lower acquisition difficulty, and simplifies the data acquisition process. Compared with traditional multimodal emotion recognition methods (relying on facial expressions, voice signals, etc.), the present invention only requires EEG signals, and comprehensively extracts and processes multidimensional features in signals through the decomposition of time, spectrum and space perspectives, thereby improving the effective utilization rate of unimodal data. Under the same model, only unimodal data is used to reach the level of multimodal models. The self-attention mechanism is used to fuse the features of each perspective, and the weight is dynamically adjusted according to the importance of the contribution of each perspective, effectively integrating multidimensional information under a single modality, and significantly improving the accuracy and stability of emotion recognition.

[0011] In addition, the present invention uses randomized video materials to replace traditional fixed materials. By introducing randomized videos, the present invention effectively avoids this deviation, making the experimental results more reliable and better reflecting the real differences in emotional responses under different sleep states. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of the present invention;

[0013] Figure 2 is a flow chart of an embodiment;

[0014] Figure 3 is a flow chart of an embodiment;

[0015] Figure 4 Schematic diagram of the results of the embodiment. DETAILED DESCRIPTION

[0016] like Figure 1 and Figure 2 As shown, this embodiment relates to a multi-view fusion emotion recognition method under sleep deprivation conditions, including:

[0017] Step 1: After the original unimodal EEG is converted into data of three different dimensions of time, space and frequency by data reshaping, the cosine function is used to positionally encode different dimensions. Finally, the reshaped data is unified through linear layers of different shapes to achieve preprocessing, specifically: X c =X eeg (n,c,t,f),Xf =X eeg (n, f, c, t), X t =X eeg , Where: X C , X f , X t , which correspond to the data in the spatial perspective, frequency domain perspective and time domain perspective after the original data is reshaped. The right side of the equal sign is the shape of the reshaped data, and n represents the number of sample points. The difference between the last two formulas and the position embedding in the general Trasnformer method is that the data under different perspectives must be embedded in the corresponding dimensions, that is, X C , X f , X t Although it is the result of reshaping the same original data, the position embedding is performed in different dimensions.

[0018] Step 2: Input each dimension of the data processed in step 1 as a separate modality into the transformer module for training, adopt an eight-head attention mechanism, and normalize the output of each modality using the softmax function;

[0019] Step 3: The single-modal data from different perspectives obtained in step 2 are fused using learnable weight coefficients, specifically: Among them: H represents the output of the Transformer after different perspectives; w represents a learnable iterative weight vector, α represents the calculated attention weight; the subscripts c, f, and t represent the corresponding spatial, frequency domain, and time domain perspectives respectively.

[0020] After specific practical experiments, under the international standard ESI NeuroScan EEG acquisition system, 62 effective electrodes were configured with a sampling frequency of 1000Hz. 20 subjects were collected in three sleep states: sleep deprivation (SD), sleep recovery (SR), and normal sleep (NS) to watch randomly selected video materials, stimulating 4 emotions: happiness, sadness, fear and neutral emotions to conduct a four-category emotion experiment. The differential entropy (DE) features of 5 frequency bands were extracted in the short-time Fourier transform (STFT) within a 4-second Hanning window for data preprocessing. The Adam optimizer was used, with a learning rate range of [1e-4, 1e-2] and a weight decay parameter range of [1e-6, 1e-2]. The training rounds were 150 rounds. The model uses an 8-head self-attention mechanism to process the parameters of 5 time windows and 5 frequency bands. The experimental data obtained by starting the above method are as follows:

[0021] Table 1 Results of cross-subject and subject-dependent experiments (mean / standard deviation, %) method SD (Sleep Deprivation) SR (Sleep Recovery) NS (normal sleep) Subject dependence MLP 41.74 / 14.77 48.98 / 12.21 48.26 / 14.87 SVM 50.65 / 12.21 54.17 / 13.51 55.31 / 14.73 KNN 55.76 / 13.58 58.28 / 10.69 59.52 / 11.91 ASGC 65.14 / 12.44 69.47 / 11.26 70.29 / 13.64 Transformer 68.24 / 9.20 77.12 / 10.46 76.94 / 9.87 MVFT 74.54 / 9.28 79.26 / 8.86 80.18 / 8.57 Across subjects MLP 36.77 / 13.09 42.15 / 12.11 43.42 / 12.65 SVM 45.43 / 11.54 54.28 / 11.73 54.35 / 12.36 KNN 47.56 / 14.28 55.44 / 13.61 56.37 / 14.52 ASGC 51.45 / 11.26 60.34 / 9.87 61.19 / 11.75 Transformer 57.32 / 12.09 65.43 / 11.37 68.51 / 12.63 The present invention 60.26 / 9.81 68.67 / 10.06 70.22 / 11.74

[0022] As shown in Table 1, the accuracy of emotion recognition of the present invention is significantly improved compared with traditional models (such as MLP, KNN, SVM), especially under sleep deprivation conditions, the accuracy is improved by more than 6%, and under sleep recovery and normal sleep conditions, it is improved by 2.14% and 3.24%, respectively. Confusion matrix analysis: Under different sleep states, the recognition accuracy of happy emotions drops significantly when sleep is deprived, while the classification performance of neutral, sad and fearful emotions does not change much. EEG topology analysis: The experiment also provides EEG energy topology maps of different emotions in 5 frequency bands (Delta, Theta, Alpha, Beta, Gamma), showing the energy distribution characteristics of the prefrontal and lateral temporal lobe regions under different emotional states, further proving the impact of sleep deprivation on emotion processing.

[0023] In summary, the present invention significantly simplifies the requirements for acquisition equipment through unimodal processing of EEG data, and is suitable for realizing emotion recognition in environments outside the laboratory, such as sleep monitoring or remote health management. Combined with multi-perspective fusion technology, the present invention also achieves similar results as multimodal methods in terms of emotion recognition accuracy, and reduces the reliance on complex data collection. It ensures the effective integration of multi-dimensional information under a single modality, and improves the accuracy and stability of emotion recognition. Through the application of these technical means, the present invention provides a more efficient and practical solution for the application of EEG signals in emotion recognition.

[0024] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A multi-view fusion emotion recognition method under sleep deprivation conditions, characterized in that: By constructing and training a recognition model based on a multi-view fusion Transformer network in the offline stage, the trained recognition model is used to perform multi-view fusion on the input signal in the online stage to achieve real-time detection.

2. The method for multi-view fusion emotion recognition under sleep deprivation conditions according to claim 1, characterized in that: The recognition model includes: a Transformer encoder and a classifier, wherein: the Transformer encoder learns the key features of each perspective through a self-attention mechanism, and obtains the final sentiment classification result through multi-perspective weighted fusion; the classifier inputs the fused features into a fully connected layer for sentiment classification, and uses a Softmax function to output the probability distribution of each sentiment category.

3. The method for multi-view fusion emotion recognition under sleep deprivation conditions according to claim 1, characterized in that: The training described above uses EEG signals and obtains differential entropy features as a sample set after preprocessing.

4. The method for multi-view fusion emotion recognition under sleep deprivation conditions according to claim 1, characterized in that: The preprocessing refers to: after bandpass filtering and noise reduction processing of the EEG signal, using short-time Fourier transform to extract differential entropy features (DE) from different frequency bands of each electrode.

5. The method for emotion recognition based on multi-view fusion under sleep deprivation conditions according to claim 1, characterized in that: The multi-view fusion refers to rearranging and embedding the input EEG data in three dimensions: time, spectrum and space, calculating the attention weights of different viewpoints respectively, and fusing the features of these viewpoints through a multi-head attention mechanism.

6. The method for multi-view fusion emotion recognition under sleep deprivation conditions according to any one of claims 1 to 5, characterized in that: include: Step 1: After the original unimodal EEG is converted into data of three different dimensions of time, space and frequency by data reshaping, the cosine function is used to positionally encode different dimensions. Finally, the reshaped data is unified through linear layers of different shapes to achieve preprocessing, specifically: X c =X eeg (n,c,t,f),X f =X eeg (n,f,c,t),X t =X eeg , Where: X C , X f , X t , which correspond to the data in the spatial perspective, frequency domain perspective and time domain perspective after the original data is reshaped. The right side of the equal sign is the shape of the reshaped data, and n represents the number of sample points. The difference between the last two formulas and the position embedding in the general Trasnformer method is that the data under different perspectives must be embedded in the corresponding dimensions, that is, X C , X f , X t Although it is the result of reshaping the same original data, the position embedding is performed in different dimensions; Step 2: Input each dimension of the data processed in step 1 as a separate modality into the transformer module for training, adopt an eight-head attention mechanism, and normalize the output of each modality using the softmax function; Step 3: The single-modal data from different perspectives obtained in step 2 are fused using learnable weight coefficients, specifically: Among them: H represents the output of the Transformer after different perspectives; w represents a learnable iterative weight vector, α represents the calculated attention weight; the subscripts c, f, and t represent the corresponding spatial, frequency domain, and time domain perspectives respectively.

Images