An electroencephalogram signal emotion detection method, system, device and medium

Abstract

The application belongs to the technical field of electroencephalogram signal analysis, and discloses an electroencephalogram signal emotion detection method, system, device and medium, comprising: acquiring electroencephalogram signal data; inputting the electroencephalogram signal data into an emotion detection model for prediction classification to obtain an emotion classification result; wherein the emotion detection model comprises a time-frequency feature extraction module, an attention mechanism model, a self-supervised contrast learning module, a pooling module and a full connection prediction module connected in sequence. The technical solution disclosed by the application effectively overcomes the problems of single feature, sample imbalance and small sample in the prior art, and significantly improves the accuracy of emotion detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of electroencephalogram signal analysis, and particularly relates to an electroencephalogram signal emotion detection method, system, device and medium. BACKGROUND

[0002] Emotion recognition is an important research direction in the fields of human-computer interaction, mental health monitoring, intelligent education, and affective computing. Traditional emotion recognition methods rely heavily on facial expressions, speech, and text inputs. However, electroencephalogram (EEG) signals, which directly reflect brain activity, have become a powerful tool for emotion recognition. EEG signals have high temporal efficiency, low invasiveness, and high spatial resolution, making them effective in reflecting individual emotional states and psychological changes. Therefore, emotion recognition research based on EEG signals has attracted widespread attention.

[0003] In recent years, significant progress has been made in the field of EEG signal emotion recognition. Traditional methods mainly rely on classic machine learning algorithms such as support vector machines (SVM) and k-nearest neighbors (KNN), combined with manually extracted features such as spectral features and waveform features. With the rapid development of deep learning technology, emotion recognition methods based on convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformer have gradually become the mainstream of research. Methods based on convolutional neural networks (CNN) usually perform time-frequency transformation on EEG signals to generate a spectrogram, and then apply a convolutional neural network on the spectrogram for emotion recognition. This method has the advantages of automatic feature extraction, avoiding human intervention, and high classification accuracy. However, it cannot fully exploit the temporal characteristics of EEG signals. Methods based on recurrent neural networks (RNN) can effectively handle time series information in EEG signals by capturing temporal dependencies, recognizing long-term dependencies. However, this method has the disadvantages of long training process, high computational resource consumption, and sensitivity to noise. EEG signal classification methods based on Transformer capture long-range dependencies through self-attention mechanisms, efficiently processing temporal characteristics of EEG signals. This method has the advantages of strong parallel computing capability and suitability for long sequence data. However, the model is relatively complex, requiring a large amount of training data and high computational overhead. Therefore, there is an urgent need for an emotion recognition method that comprehensively considers the overall characteristics of EEG signals, has efficient detection capability, and provides accurate classification results. SUMMARY

[0004] The purpose of the present application is to provide an electroencephalogram signal emotion detection method, system, device and medium to solve the problems existing in the prior art.

[0005] To achieve the above purpose, the present application provides an electroencephalogram signal emotion detection method, comprising:

[0006] Acquire electroencephalogram (EEG) signal data;

[0007] The EEG signal data is input into the emotion detection model for prediction and classification to obtain the emotion classification result; wherein, the emotion detection model includes a time-frequency feature extraction module, an attention mechanism model, a self-supervised contrastive learning module, a pooling module, and a fully connected prediction module connected in sequence.

[0008] Optionally, the training process of the emotion detection model specifically includes:

[0009] Acquire training data, which includes EEG signal training data and corresponding emotion classification results;

[0010] An initial emotion detection model is constructed. The training data is input into the initial emotion detection model for prediction and classification. The model is trained with the goal of minimizing the loss between the initial training result after prediction and classification and the emotion classification result corresponding to the EEG signal training data, and the trained emotion detection model is obtained.

[0011] Optionally, the processing procedure of the emotion detection model specifically includes:

[0012] The EEG signal data is input into the time-frequency feature extraction module, where wavelet transform is performed on the EEG signal data to obtain the corresponding frequency domain signal; the EEG signal data is encoded using a parallel multi-scale one-dimensional convolutional layer to obtain time domain feature encoding; the frequency domain signal is encoded using a three-layer one-dimensional convolutional layer to obtain frequency domain feature encoding; and one-dimensional convolution is performed on the EEG signal data and the frequency domain signal respectively to obtain initial time domain features and initial frequency domain features.

[0013] By applying a self-attention mechanism, the initial time-domain features and frequency-domain features are encoded and fused to obtain the first time-frequency feature, and the initial frequency-domain features and time-domain features are encoded and fused to obtain the second time-frequency feature;

[0014] The first time-frequency feature and the second time-frequency feature are concatenated and fused to obtain the fused feature vector;

[0015] Global average pooling is performed on the feature vector to obtain the compressed feature vector;

[0016] The compressed feature vector is input into the fully connected prediction module, and the classification result is output after passing through the softmax layer.

[0017] Optionally, performing wavelet transform on the EEG signal data specifically includes:

[0018] A continuous wavelet transform is applied to each channel of each sample in the EEG signal data to convert the EEG signal from the time domain to the frequency domain and generate a spectrogram; during the continuous wavelet transform process, the Morlet wavelet is used as the mother wavelet.

[0019] Optionally, it also includes inputting each time-frequency feature into a self-supervised contrastive learning module, calculating a contrastive loss based on cosine similarity, and using the contrastive loss to optimize the sentiment detection model.

[0020] A brainwave signal emotion detection system, comprising:

[0021] The data acquisition module is used to acquire electroencephalogram (EEG) signal data;

[0022] The emotion detection module is used to input the EEG signal data into the emotion detection model for prediction and classification to obtain the emotion classification result; wherein, the emotion detection model includes a time-frequency feature extraction module, an attention mechanism model, a self-supervised contrastive learning module, a pooling module, and a fully connected prediction module connected in sequence.

[0023] An electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the aforementioned brainwave signal emotion detection method.

[0024] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for detecting emotions via electroencephalogram (EEG) signals.

[0025] The technical effects of this invention are as follows:

[0026] This invention proposes a method for emotion detection based on time-frequency feature fusion and self-supervised contrastive learning of EEG signals. This method achieves efficient emotion detection by fusing time-frequency features of EEG signals with self-supervised learning techniques. This approach effectively overcomes the problems of single feature sets, imbalanced samples, and small sample sizes in existing technologies, while significantly improving the accuracy of emotion detection. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0029] Figure 1 This is a schematic diagram of the model structure in an embodiment of the present invention. Detailed Implementation

[0030] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0031] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0032] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0033] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0034] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] like Figure 1 As shown, this embodiment provides a method for emotion detection using electroencephalogram (EEG) signals, including: acquiring EEG signal data; inputting the EEG signal data into an emotion detection model for prediction and classification to obtain an emotion classification result; wherein, the emotion detection model includes a time-frequency feature extraction module, an attention mechanism model, a self-supervised contrastive learning module, a pooling module, and a fully connected prediction module connected in sequence.

[0036] This embodiment first converts the EEG signal from the time domain to the frequency domain using wavelet transform to generate a spectrogram; then, it uses one-dimensional convolutional neural networks (1DCNN) to encode the EEG signal in both the time and frequency domains; next, it fuses the time and frequency codes and trains a time-frequency consistent EEG signal code through self-supervised contrastive learning; finally, it connects and performs global average pooling on the fused time and frequency codes, passes them through a fully connected layer and a softmax layer, and outputs the emotion detection result.

[0037] This embodiment specifically includes the following steps:

[0038] Step 1: Use wavelet transform to convert the EEG signal from the time domain to the frequency domain and generate a spectrogram.

[0039] Step 2: Use parallel multi-scale one-dimensional convolution (1DCNN) to encode the EEG signal in the time domain and extract the time domain features.

[0040] Step 3: Encode the EEG signal in the frequency domain using a 3-layer one-dimensional convolutional network (1DCNN) to extract frequency domain features.

[0041] Step 4: After performing a one-dimensional convolution (1DCNN) transformation on the frequency domain EEG signal, fuse it with the temporal feature encoding in Step 2; after performing a one-dimensional convolution (1DCNN) transformation on the temporal EEG signal, fuse it with the frequency domain feature encoding in Step 3.

[0042] Step 5: Use self-supervised contrastive learning to train the feature vectors fused in Step 4 and Step 5 to learn the time-frequency consistent EEG signal encoding.

[0043] Step 6: Concatenate and perform global average pooling on the encoding from Step 5, then pass it through a fully connected layer and a softmax layer to output the sentiment detection result.

[0044] This invention proposes a method for emotion detection based on time-frequency feature fusion and self-supervised contrastive learning of EEG signals. This method achieves efficient emotion detection by fusing time-frequency features of EEG signals with self-supervised learning techniques. This approach effectively overcomes the problems of single feature set, imbalanced samples, and small sample sizes in existing technologies, while significantly improving the accuracy of emotion detection.

[0045] The specific implementation process of this embodiment includes:

[0046] Step 1: Let the size of the EEG signal X be K×C×N, where K is the number of sampling points, C is the number of channels, and N is the number of samples. For the j-th channel of the i-th sample of X, perform continuous wavelet transform using formula (1) to obtain the spectrum I(i,j).

[0047]

[0048] Where ψ(t) is the Morlet wavelet, a is the scaling factor, and b is the translation factor. The EEG signal X is transformed by wavelet to obtain an S×C×N spectrum I.

[0049] Step 2, Temporal Encoding of EEG Signals. Since the original EEG signal data is a one-dimensional time series, the data is only correlated with time in the horizontal direction, but not in the vertical direction. Therefore, a one-dimensional convolutional neural network (CNN) is used to extract and encode features from the EEG signals in the time domain. The specific process is as follows:

[0050] (1) By performing three parallel sets of one-dimensional dilated convolution operations on the EEG signals in the time domain, multi-scale temporal features of the EEG signals can be captured without increasing the parameters. The three sets of one-dimensional dilated convolution structures are as follows:

[0051] The first group of one-dimensional dilated convolutions has two layers: the first layer uses 16 filters, a kernel size of 3, a stride of 1, a dilation factor of 1, uses GeLU as the activation function, and employs L2 regularization to prevent overfitting. The second layer is exactly the same as the first layer.

[0052] The second group of one-dimensional dilated convolutions has two layers: the first layer uses 16 filters, a kernel size of 3, a stride of 1, and GeLU as the activation function, with L2 regularization to prevent overfitting. The difference between the second and first layers is that the dilation coefficient is 2; everything else is the same.

[0053] The third group of one-dimensional dilated convolutions has two layers: the first layer uses 16 filters, a kernel size of 3, a stride of 1, and GeLU as the activation function, with L2 regularization to prevent overfitting. The second layer differs from the first layer in that its dilation coefficient is 3; everything else is the same.

[0054] The results of the three parallel convolutions are then merged and concatenated together, and then input into the next two one-dimensional convolutional layers.

[0055] (2) Apply two one-dimensional convolutional layer operations to the output of (1): The first convolution uses 32 filters, a kernel size of 3, a stride of 1, uses the GeLU activation function, and uses L2 regularization to prevent overfitting; the second convolution uses 64 filters, a kernel size of 3, a stride of 1, also uses the ReLU activation function, and uses L2 regularization to prevent overfitting.

[0056] Step 3, Frequency Domain Encoding of EEG Signals: This is a network for frequency domain feature extraction. Its purpose is to extract frequency domain features from the spectrogram I obtained in Step 1 using a four-layer one-dimensional convolutional neural network (CNN). The specific convolutional operations for each layer are as follows:

[0057] (1) The first convolutional layer uses 16 filters, with a kernel size of 3 and a stride of 1. The activation function is GeLU, which can better capture the nonlinear characteristics of the frequency domain signal. L2 regularization helps to prevent overfitting, especially when dealing with complex frequency domain data.

[0058] (2) The second convolutional layer uses 32 filters, with a kernel size of 3 and a stride of 1. The activation function is GeLU, which can better capture the nonlinear characteristics of the frequency domain signal. L2 regularization helps to prevent overfitting, especially when dealing with complex frequency domain data.

[0059] (3) The third convolutional layer uses 32 filters, with a kernel size of 3 and a stride of 1. The activation function is GeLU, which can better capture the nonlinear characteristics of the frequency domain signal. L2 regularization helps to prevent overfitting, especially when dealing with complex frequency domain data.

[0060] (4) The fourth convolutional layer uses 64 filters, with a kernel size of 3 and a stride of 1. The activation function is GeLU, which can better capture the nonlinear characteristics of the frequency domain signal. L2 regularization helps to prevent overfitting, especially when dealing with complex frequency domain data.

[0061] Step 4, Time-Frequency Feature Fusion: Time-frequency feature fusion is the process of combining information from both the time and frequency domains to enhance the model's ability to understand and model different features in a signal.

[0062] In electroencephalogram (EEG) signal analysis, time-frequency feature fusion can simultaneously capture the temporal variations and frequency patterns of signals, which is highly effective for tasks such as identifying complex brain activity or emotional states. The specific process is as follows:

[0063] (1) The original input time-domain EEG signal was processed by one-dimensional convolution to extract time-domain features. The convolution used 64 filters, with a kernel size of 3 and a stride of 1. The activation function used was GeLU, which can better capture the nonlinear features of the frequency domain signal. L2 regularization helps to prevent overfitting.

[0064] (2) Perform one-dimensional convolution processing on the frequency domain EEG signal from step 1 to extract frequency domain features. This step performs further nonlinear mapping on the frequency domain features to help extract deeper features in the frequency domain. The convolution uses 64 filters, a kernel size of 3, a stride of 1, and the activation function GeLU, which can better capture the nonlinear features of the frequency domain signal. L2 regularization helps prevent overfitting.

[0065] (3) Time-Frequency Feature Fusion Based on Attention Mechanism. Time-Frequency Attention Fusion enhances the model's performance by introducing a self-attention mechanism to automatically select and weight features of different times and frequencies. The specific steps are as follows:

[0066] ① Let the time-domain encoding obtained in step 2 be T X ∈R N×DThe frequency domain code obtained in step 3 is F X ∈R N×D The feature obtained in step 4 (1) is T T ∈R N×D The feature obtained in step 4(2) is F F ∈R N×D Using self-attention mechanisms to study T X and F F and F X and T T Feature fusion will be performed, and T will be used as an example below. X and F F Take fusion as an example.

[0067] ② The time-frequency features Query(Q), Key(K), and Value(V) are obtained by linear transformation, and the specific formulas are as follows:

[0068]

[0069] in

[0070] ③ Calculate attention scores in the time and frequency domains: Based on the self-attention mechanism, attention scores in the time and frequency domains are calculated using the similarity between the query and the key. A common method is to calculate the dot product and then perform scaling.

[0071] Temporal attention score:

[0072]

[0073] Attention score in the frequency domain:

[0074]

[0075] Where d is the dimension of Query and Key, which is usually a preset hyperparameter.

[0076] ④ Use Softmax to calculate the probability distribution of the scores and obtain the attention weights:

[0077] Temporal attention weights:

[0078]

[0079] Attention weights in the frequency domain:

[0080]

[0081] ⑤ Use attention weights plus value:

[0082] Weighted Value Output in the Time Domain:

[0083]

[0084] Weighted Value output in the frequency domain:

[0085]

[0086] ⑥ Weighted output combining time and frequency domains:

[0087] FO TF =α T ×Output T +α F ×Output F (14)

[0088] Where, α T and α F These are the learned weighting coefficients.

[0089] Similarly, F X and T T The above process is used to merge the components, and the final output is FO. FT .

[0090] Step 5, calculate FO TF and FO FT Self-contradictory loss. Self-contradictory loss generates positive sample pairs by applying different enhancements or perturbations to the same input, while simultaneously generating negative sample pairs from different inputs. It then learns through contrastive analysis to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. This effectively improves the quality of feature representations, enabling the model to achieve good results in unsupervised or semi-supervised tasks. TF and FO FT These are sample pairs generated from different inputs, and their similarity is determined through comparative learning. The specific calculation process is as follows:

[0091] (1) For FO TF and FO FT Perform L2 regularization on each vector. L2 regularization scales each vector to a unit length, ensuring that its norm is 1.

[0092] (2) Calculate FO TF and FO FT Cosine similarity between them:

[0093]

[0094] (3) Calculate the contrast loss:

[0095]

[0096] Step 6, Emotion Detection. (The text abruptly ends here, likely due to an incomplete sentence or a formatting error.)TF and FO FT After fusion, the result undergoes global average pooling, compressing the temporal information into a fixed-length feature vector. Finally, a fully connected layer is used to generate the final output y_pred, which is used for sentiment detection tasks. The specific process is as follows:

[0097] (1) FO TF and FO FT Direct connection and fusion are performed, as shown in the following formula:

[0098] FO = concatenate(FO) TF O FT ) = [FO FT O FT (17)

[0099] (2) Perform average pooling on the FO vector to obtain the vector AFO.

[0100] (3) Use the fully connected layer to output the prediction result y_pred, as shown in the following formula:

[0101] y_pred=σ(AFO*W+b) (18)

[0102] Where σ is the Sigmoid function.

[0103] (4) Prediction Loss Function: The loss function between the predicted classification result y_pred and the true classification result y_true is calculated using cross-entropy, as shown in the following formula:

[0104]

[0105] Where N is the number of samples, cnum is the number of sentiment categories, and y_pred i,m y_true is the predicted probability that the i-th sample belongs to the m-th class. i,m It is the one-hot encoded value (0 or 1) of the true label of the i-th sample in the m-th class.

[0106] A brainwave signal emotion detection system, comprising:

[0107] The data acquisition module is used to acquire electroencephalogram (EEG) signal data;

[0108] The emotion detection module is used to input the EEG signal data into the emotion detection model for prediction and classification to obtain the emotion classification result; wherein, the emotion detection model includes a time-frequency feature extraction module, an attention mechanism model, a self-supervised contrastive learning module, a pooling module, and a fully connected prediction module connected in sequence.

[0109] An electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the aforementioned brainwave signal emotion detection method.

[0110] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for detecting emotions via electroencephalogram (EEG) signals.

[0111] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting emotion using electroencephalogram (EEG) signals, characterized in that, include: Acquire electroencephalogram (EEG) signal data; The EEG signal data is input into the emotion detection model for prediction and classification to obtain the emotion classification result; wherein, the emotion detection model includes a time-frequency feature extraction module, an attention mechanism model, a self-supervised contrastive learning module, a pooling module, and a fully connected prediction module connected in sequence; The processing procedure of the emotion detection model specifically includes: The EEG signal data is input into the time-frequency feature extraction module, where wavelet transform is performed on the EEG signal data to obtain the corresponding frequency domain signal; the EEG signal data is encoded using a parallel multi-scale one-dimensional convolutional layer to obtain time domain feature encoding; the frequency domain signal is encoded using a three-layer one-dimensional convolutional layer to obtain frequency domain feature encoding; and one-dimensional convolution is performed on the EEG signal data and the frequency domain signal respectively to obtain initial time domain features and initial frequency domain features. By applying a self-attention mechanism, the initial time-domain features and frequency-domain features are encoded and fused to obtain the first time-frequency feature, and the initial frequency-domain features and time-domain features are encoded and fused to obtain the second time-frequency feature; The first time-frequency feature and the second time-frequency feature are concatenated and fused to obtain the fused feature vector; Global average pooling is performed on the feature vector to obtain the compressed feature vector; The compressed feature vector is input into the fully connected prediction module, and the classification result is output after passing through the softmax layer.

2. The method for detecting emotion using electroencephalogram (EEG) signals according to claim 1, characterized in that, The training process of the emotion detection model specifically includes: Acquire training data, which includes EEG signal training data and corresponding emotion classification results; An initial emotion detection model is constructed. The training data is input into the initial emotion detection model for prediction and classification. The model is trained with the goal of minimizing the loss between the initial training result after prediction and classification and the emotion classification result corresponding to the EEG signal training data, and the trained emotion detection model is obtained.

3. The method for detecting emotion using electroencephalogram (EEG) signals according to claim 1, characterized in that, The wavelet transform of the EEG signal data specifically includes: A continuous wavelet transform is applied to each channel of each sample in the EEG signal data to convert the EEG signal from the time domain to the frequency domain and generate a spectrogram; during the continuous wavelet transform process, the Morlet wavelet is used as the mother wavelet.

4. The method for detecting emotion using electroencephalogram (EEG) signals according to claim 1, characterized in that, It also includes inputting each time-frequency feature into a self-supervised contrastive learning module, calculating the contrastive loss based on cosine similarity, and using the contrastive loss to optimize the sentiment detection model.

5. A brainwave signal emotion detection system, using the method as described in any one of claims 1-4, characterized in that, include: The data acquisition module is used to acquire electroencephalogram (EEG) signal data; The emotion detection module is used to input the EEG signal data into the emotion detection model for prediction and classification to obtain the emotion classification result; wherein, the emotion detection model includes a time-frequency feature extraction module, an attention mechanism model, a self-supervised contrastive learning module, a pooling module, and a fully connected prediction module connected in sequence.

6. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform a brainwave signal emotion detection method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements a brainwave signal emotion detection method as described in any one of claims 1-4.

Images