Cross-subject electroencephalogram emotion recognition method based on time-frequency comparative learning
Through the comparative learning model of time-frequency comparison learning and convolutional structure, combined with the automatic loss mechanism, the inter-individual differences and label noise problems in EEG emotional recognition are solved, and high accuracy is achieved under label-free data pre-training and small amount of data fine-tuning Sex and stability are suitable for cross-subject emotional recognition.
Patent Information
- Application Number
- CN202411153627.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-03
AI Technical Summary
There are inter-individual differences and label noise problems in EEG emotional recognition, which makes it difficult to ensure the accuracy and stability of emotional recognition, especially in the case of a small number of labels, which is difficult to effectively identify emotional states.
Using the cross-participant EEG sentiment recognition method based on time-frequency comparison learning, the time-domain and frequency-domain features are extracted through pre-training and fine-tuning technology of labelless data, and combined with the contrast learning model of the convolutional structure and the automatic loss mechanism, the Emotional classification training ensures the adaptability and accuracy of the model on new subjects.
Improve the accuracy and stability of emotional recognition, effectively solve the label noise problem, can maintain the recognition effect in a small number of labels, expand to new subjects, and extract more effectively through ConvNeXt pre-trained model and adaptive loss mechanism. Discriminant domain unchanged underlying features.
Smart Images

Figure CN119066493A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of electroencephalogram (EEG) signal processing, and in particular to a method for cross-subject EEG emotion recognition. Background Art
[0002] EEG emotion recognition is a technology based on EEG (Electroencephalographic, EEG) signals, which aims to identify and infer the emotional state of an individual.
[0003] This technology combines knowledge from fields such as neuroscience, signal processing, and machine learning. By analyzing the spectrum, time domain, and spatial characteristics of EEG signals, as well as the neural activity patterns associated with emotions, the EEG emotion recognition system is able to identify an individual's emotional state, such as pleasure, sadness, anxiety, etc. Compared with other signals, EEG signals cannot be disguised, which makes them more objective in reflecting human emotional states.
[0004] In addition, EEG signals have high sensitivity, which helps to capture subtle changes in EEG activity and show good real-time response to changes in emotional states. In recent years, with the widespread application of deep learning methods, end-to-end models have been developed to extract EEG emotional features. However, these studies focus on individual subject tasks and aim to classify the emotional states of individual participants. Due to inter-individual differences, extending this ability to classify the emotional states of different individuals faces significant challenges.
[0005] Therefore, researchers have adopted the Domain Adaptation (DA) method to solve this problem, which can effectively transfer knowledge from the source domain to the target domain, and is particularly suitable for handling subject-independent emotion recognition tasks with changes in data distribution. As an example of the DA method, the Domain-Adversarial Neural Network (DANN) constructs a domain discriminator that works adversarially with the feature extractor to align the EEG representations between the source and target domains.
[0006] Another issue that is not easily overlooked in EEG emotion recognition is the label noise problem. During the EEG signal acquisition process, the individual's inherent subjectivity may interfere with the generation of emotional states. If the emotion is different from what the researcher expects, it may lead to inaccurate labeling of the EEG signal.
[0007] In other research areas, commonly used learning methods with noisy labels are based on the noisy transfer matrix and a robust loss function. The noisy transfer matrix specifies the transition probability from each precise label to each noisy label. It is often used to modify the loss to address the label noise problem.
[0008] Inspired by methods from other fields, some methods have also emerged in the field of EEG emotion recognition to address the problem of label noise.
[0009] Some researchers have proposed the EmotionDL method, a method for learning emotion perception distribution, which combines a graph neural network to effectively handle changes in EEG signals and label noise between subjects. The researchers proposed a joint optimization strategy based on capsule networks (JO-CapsNet) to update pseudo labels by predicting the possibility of the existence of category labels to correct noisy labels. Many studies on label noise aim to identify and correct noisy labels, often ignoring methods to avoid them by directly extracting information from the data. Summary of the invention
[0010] The purpose of the present invention is to provide a cross-subject EEG emotion recognition method based on time-frequency contrast learning, which improves the accuracy of emotion recognition by pre-training with unlabeled data and adapts to new objects by fine-tuning a small amount of data.
[0011] To this end, the present invention provides a cross-subject EEG emotion recognition method based on time-frequency contrast learning, comprising the following steps: S1, converting the EEG signal of the subject's emotional feedback from a time domain signal to a frequency domain signal, then enhancing the time domain signal to obtain an enhanced signal in the time domain, enhancing the frequency domain signal to obtain an enhanced signal in the frequency domain, and then dividing the EEG signal into source domain data and target domain data based on the subject; S2, constructing a contrast learning model based on a convolution structure, inputting unlabeled source domain data into the model for contrast pre-training, and obtaining a time domain feature extractor and a frequency domain feature extractor respectively; S3, inputting the labeled target domain data into the pre-trained time domain feature extractor and frequency domain feature extractor, extracting time domain and frequency domain features, splicing the obtained time domain and frequency domain features, and then inputting them into a classifier for emotion classification training; S4, inputting a small amount of labeled unenhanced EEG signals of a new subject into the model, fine-tuning the model, and then using it for emotion recognition of the subject's unlabeled EEG signals.
[0012] Compared with the prior art, the present invention can not only achieve leading emotion recognition accuracy, but also ensure the stability of recognition effect with a small number of labels, effectively solving the label noise problem. The specific beneficial effects are reflected in:
[0013] 1. The ConvNeXt pre-training model used in the present invention can effectively extract potential features related to emotion recognition tasks from EEG signals of a large amount of unlabeled data.
[0014] 2. The automatic loss mechanism proposed in the present invention can effectively enable the model to be pre-trained for comparison to extract more discriminative domain-invariant underlying features.
[0015] 3. The pre-trained model proposed in the present invention can be extended to new subjects during fine-tuning.
[0016] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0018] Figure 1 A schematic diagram of a TFCL model provided in an embodiment of the present invention;
[0019] Figure 2 A schematic diagram of a feature extractor provided in an embodiment of the present invention;
[0020] Figure 3 A schematic diagram of an olfactory emotion experimental paradigm provided by an embodiment of the present invention;
[0021] Figure 4 A schematic diagram showing the comparison of the accuracy of different model variants provided by an embodiment of the present invention;
[0022] Figure 5 A schematic diagram of visualizing the distribution of features at different stages provided by an embodiment of the present invention;
[0023] Figure 6 A schematic diagram showing a comparison of the emotion recognition accuracy of the proposed method when the calibration data provided in the embodiment of the present invention is taken in different proportions. DETAILED DESCRIPTION
[0024] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.
[0025] This paper proposes a new method, Time-Frequency Contrastive Learning (TFCL) with cross-subject domain adaptation, which improves the accuracy of emotion recognition through pre-training on unlabeled data. Specifically, we enable the model to learn the time-frequency information of EEG signals through a contrastive learning method, and adapt to new objects by fine-tuning with a small amount of data. Experimental results show that the proposed method performs well on two public datasets and one self-built dataset, including DEAP, DREAMER, and OVPD-II. Our work contributes to the development of EEG-based emotion recognition systems and promotes the improvement of the quality and efficiency of human-computer interaction.
[0026] The cross-subject EEG emotion recognition method of the present invention based on time-frequency contrast learning comprises the following steps:
[0027] S1. Use EEG equipment to collect EEG signals of subjects when they watch the film, and collect emotional feedback from the subjects to give emotional labels.
[0028] In two public datasets (including DEAP and DREAMER), we define emotional labels based on the two basic dimensions of emotional states provided by the datasets: arousal value and valence value. We define labels for these two dimensions separately. Specifically, when the arousal value is greater than 5, we define the label of the corresponding EEG signal segment as 1, which is regarded as a high-activity emotion; when the arousal value is less than or equal to 5, the label of the corresponding EEG signal segment is 0, which is regarded as a low-activity emotion; when the valence value is greater than 5, the label of the corresponding EEG signal segment is 1, which is regarded as a positive emotion; when the valence value is less than or equal to 5, the label of the corresponding EEG signal segment is 0, which is regarded as a negative emotion.
[0029] In the OVPD-II dataset, in order to obtain the EEG signals of the subjects’ emotional feedback, Figure 3 The olfactory emotion experimental paradigm shown in the figure asks the subjects to watch those film clips that are likely to cause a specific single emotion, and then obtain the subjects' EEG signals. Before watching, the subjects first accept emotional cues for 3 seconds, and then watch the video for 40 seconds. After watching, the subjects are asked to give feedback for 5 seconds, and then rest for 15 seconds before continuing to the next round. Emotional labels are divided into three types: positive emotions, neutral emotions, and negative emotions.
[0030] After collecting the EEG signals, we perform a series of preprocessing to obtain pure EEG signals and define the EEG signals of all subjects as E.
[0031] For a single subject, we regard the preprocessed EEG signal as a time domain representation and define it as E T ={X1,X2,...,X t}∈R N×T ,t∈{1,2,...,T}, where N is the number of EEG signal channels and T is the number of time points in the EEG signal sample. For the signal X at the t-th time sample point t It is composed of signals from N channels and is a vector with a length of N.
[0032] Afterwards, we use a transformation operator, such as the Fast Fourier Transform, to convert the time domain representation of the EEG signal into the frequency domain representation E F ={S1,S2,...,S T}∈R N×T .
[0033] In addition, in order to enrich the time-frequency information during training, we use a time series enhancement library customized for the provided EEG time series samples to generate enhanced signals, denoted as Specifically, EEG temporal signal enhancement methods include jittering, scaling, time shifting, and neighborhood segmentation. By applying these enhancement methods to obtain different temporal signals, we expose the model to complex temporal dynamics, resulting in more robust time-based embeddings.
[0034] In terms of frequency domain signals, we introduce perturbations in both low- and high-frequency domains: the high-frequency component captures the fast-changing part of the time series (e.g., sharp changes), while the low-frequency component reflects the slower changes in the time domain signal. Perturbations in the low-frequency band correspond to changes in the first half of the spectrum.
[0035] Specifically, since each frequency component in the spectrum represents a basis function with a specific frequency and amplitude (e.g., a Fourier transformed sine function), we introduce or remove frequency components to perturb the spectrum. After enhancing the frequency domain signal, the corresponding frequency domain local enhancement representation is obtained, which is defined as
[0036] In addition, we divide the EEG signals into E corresponding to the source domain subjects according to the subjects. s E corresponding to the target domain subjects t , which will be used in S2 and S3 respectively.
[0037] S2. Build a contrastive learning model based on convolutional structure and use unlabeled source domain data for pre-training.
[0038] The learning comparison model includes a time domain feature extractor Gt, a frequency domain feature extractor Gf, a projector Rt, and a projector Rf.
[0039] The feature extractors Gt and Gf have the same structure, which consists of a convolutional layer, a normalization layer, four ConvNeXt modules, an average pooling layer, a normalization layer, a linear layer, and a convolutional attention module.
[0040] A ConvNeXt module consists of a convolutional layer, a normalization layer, a convolutional layer, an activation function layer, a convolutional layer, a scaling and translation layer, a residual connection layer, and a downsampling layer.
[0041] The convolutional attention module contains channel attention and spatial attention.
[0042] Projectors Rt and Rf have the same structure, consisting of two linear layers.
[0043] After that, we will train the source domain data E sThe input model is pre-trained, and the time domain contrast loss is calculated according to formula (1), the frequency domain contrast loss is calculated according to formula (2), and the time-frequency contrast loss is calculated according to formula (3). Then, the calculated loss is calculated according to the automatic loss mechanism such as formula (4) to calculate the overall loss and optimize the model.
[0044]
[0045] L total =λ1L T +λ2L F +λ3L TF --------------------------------- (4)
[0046] In formula (1), we calculate the time domain contrast loss L T . Where M is the number of time samples, and are the features extracted from the i-th time domain original signal and the time domain enhanced signal respectively, τ represents the temperature coefficient of the adjustment scale, sim(v1,v2) represents the cosine similarity calculation of v1 and v2, and exp represents the exponential operation.
[0047] In formula (2), we calculate the frequency domain contrast loss L F Most of the parameters and expressions are the same as those in formula (1). In addition, and are the features extracted from the jth original signal in frequency domain and the jth enhanced signal in frequency domain respectively.
[0048] In formula (3), we calculate the time-frequency contrast loss L TF,i Most of the parameters and expressions are the same as those in formulas (1) and (2). In addition, and They respectively represent the underlying feature representations obtained by projecting the time domain components and the frequency domain components. Indicates from and A two-by-two similarity operation permutation is started, including and Four operations. 1 [j≠i] This means that i and j will not be the same in a single operation.
[0049] In formula (4), we calculate the overall loss L based on the results of formulas (1), (2), and (3): total . Where [λ1,λ2,λ3] is the weight corresponding to each loss calculated by the automatic loss mechanism.
[0050] S3. In the fine-tuning stage, we use a small amount of labeled data to fine-tune the pre-trained model so that it can be transferred to the target domain subjects.
[0051] Specifically, we use t Fine-tune the pre-trained model obtained in the previous stage with a small portion of target domain data.
[0052] During fine-tuning, only the parameters of the pre-trained feature extractor are used. The goal is to use the model parameters learned from the source domain participants as a solid starting point for fine-tuning, facilitating the extraction of more discriminative latent EEG features. Figure 1 The lower part of the paper describes the specific steps of fine-tuning.
[0053] The purpose of fine-tuning is to train a classifier with excellent classification performance and complete the migration of feature modules. First, the unenhanced EEG signal is input into the pre-trained feature extractor to extract time domain and frequency domain features; then, the obtained time domain and frequency domain features are concatenated and input into the classifier for emotion classification. Specifically, we use three fully connected layers to form a classifier, and the classification process is shown in formula (5):
[0054]
[0055] In formula (5), we calculate the final classification result out through the classifier. and Respectively represent the underlying feature representations obtained by projecting the time domain component and the frequency domain component. concatenate represents concatenation. Classifier represents input into the classifier for classification.
[0056] S4. In the test phase, for the given target domain calibration data, we feed it into the trained model to get the classification score. We determine the target domain subjects according to the leave-one-subject method, and then shuffle the target domain subjects' data and divide them into ten parts. Then, we use the ten-fold cross-validation method to use one part as calibration data each time, and the remaining nine parts as test data, so as to evaluate the generalization ability of our model from source domain subjects to target domain subjects.
[0057] In order to quantitatively evaluate the effect of the present invention, the present invention is compared with multiple self-supervised methods and domain adaptation methods through three datasets (DEAP, DREAMER and OVPDⅡ).
[0058] Figure 4To verify the effectiveness of different components in the proposed method, we conducted extensive ablation experiments on three datasets. Specifically, we designed three variants of the model: ①W / CLP: This model eliminates the comparison pre-training segment and directly classifies individuals; ②W / CBAM: This model removes the time-frequency attention module from the feature extraction segment; ③W / AL: This model excludes the adaptive loss mechanism from the overall loss calculation.
[0059] On the DEAP dataset, we observe that the entire model achieves a 12.6% classification improvement compared to the W / CLP variant model. The above experimental results highlight the key role of the underlying time-frequency information extracted from the source domain data in guiding the classification of the target domain model.
[0060] Furthermore, we observed a 5.8% drop in classification accuracy after removing the time-frequency attention module in the variant model W / CBAM. The reduced impact on W / CBAM suggests that this module enhances the model’s perception of key features in the EEG time-frequency signal. Furthermore, the degradation caused by excluding the adaptive loss mechanism in the W / AL variant model highlights the importance of properly weighting each component in the overall loss calculation. This weight is critical for the effective backpropagation of the model.
[0061] To demonstrate the effectiveness of our proposed method, we reduced and visualized the original features of the three subjects and the features extracted by the model using the T-SNE method. T-SNE is an effective nonlinear dimensionality reduction technique that can visualize high-dimensional data in a low-dimensional space, thereby revealing the intrinsic structure and characteristics of the data. Figure 5 Three data distributions extracted from the data of three subjects in the DEAP dataset are shown.
[0062] first, Figure 5 (a) and (d) depict the original data distribution, showing a mixture of various classes and indistinguishable classifications. Figure 5 (b) and (e) show the distribution of features extracted by the network that has not been pre-trained. Finally, Figure 5 (c) and (f) show the distribution of features extracted by the network using the compared pre-trained models.
[0063] By comparing the original feature distribution with the feature distribution of the unpretrained model, the proposed deep learning network features can distinguish samples of different categories. By comparing the feature distribution of the untrained model and the pretrained model, we can see that the time-frequency contrast pretraining improves the classification performance of the model. The visualization results provide a more intuitive description, showing the better representation ability of the model and the time-frequency contrast learning method for the underlying EEG features.
[0064] In addition, in order to explore the impact of the number of labeled data in the training data on the results, Figure 6 The classification performance of the model is discussed using seven proportions of labeled training data: 2%, 4%, 6%, 8%, 10%, 15%, and 20%. The results show that even with only 2% of the target domain data being calibrated, our model produces exceptional classification results. This highlights the importance of our method being extended to new subjects.
[0065] Since only a small amount of EEG data was extracted from the new subjects for calibration, our model has the potential to provide more accurate emotion classification results for them. However, when using a small amount of data above 10%, the experimental effect tends to be stable, which shows that our model pays more attention to data-level features and has certain applicability to the case of few labels.
[0066] In summary, the present invention proposes a new learning model based on adaptive time-frequency contrast in the EEG domain. By introducing contrastive learning methods, the model effectively extracts the potential time-frequency features between subjects, thereby obtaining a pre-trained model that can be transferred to new subjects without labels.
[0067] In addition, the feature extraction module we constructed shows better feature extraction capabilities and performs better in capturing global dependencies, thereby facilitating the extraction of high-quality time-frequency features. At the same time, we introduced an adaptive loss mechanism in the loss calculation process to ensure that our model can balance the time-frequency information of EEG signals and enhance emotion classification in different states.
[0068] Extensive experiments on two public datasets and one self-built dataset demonstrate the effectiveness and generalization ability of our proposed model in cross-subject EEG emotion classification tasks. Importantly, our model represents a general contrastive learning framework that can be extended to new topics and is expected to be applied to emotion-BCI systems in future research.
[0069] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A cross-subject EEG emotion recognition method based on time-frequency contrast learning, characterized in that: The following steps are involved: S1. Convert the EEG signals of the subjects during emotional feedback from time domain signals to frequency domain signals, then enhance the time domain signals to obtain enhanced signals in the time domain, enhance the frequency domain signals to obtain enhanced signals in the frequency domain, and then divide the EEG signals into source domain data and target domain data based on the subjects; S2. Build a contrastive learning model based on convolutional structure, input unlabeled source domain data into the model for contrastive pre-training, and obtain time domain feature extractor and frequency domain feature extractor respectively; S3, input the labeled target domain data into the pre-trained time domain feature extractor and frequency domain feature extractor, extract the time domain and frequency domain features, concatenate the obtained time domain and frequency domain features, and then input them into the classifier for emotion classification training; S4. Input a small amount of unenhanced labeled EEG signals of new subjects into the model, fine-tune the model, and then use it for emotion recognition of the subjects' unlabeled EEG signals.
2. The cross-subject EEG emotion recognition method based on time-frequency contrast learning according to claim 1 is characterized in that: Methods for enhancing time domain signals include jittering, scaling, time shifting and neighborhood segmentation, and methods for enhancing frequency domain signals include introducing or removing frequency components.
3. The cross-subject EEG emotion recognition method based on time-frequency contrast learning according to claim 1 is characterized in that: The time domain feature extractor has the same structure as the frequency domain feature extractor, which is composed of a convolution layer, a normalization layer, four ConvNeXt modules, an average pooling layer, a normalization layer, a linear layer, and a convolutional attention module, wherein the ConvNeXt module is composed of a convolution layer, a normalization layer, a convolution layer, a function activation layer, a convolution layer, a scaling and translation layer, a residual connection layer, and a downsampling layer, and the convolutional attention module includes channel attention and spatial attention.
4. The cross-subject EEG emotion recognition method based on time-frequency contrast learning according to claim 1 is characterized in that: The source domain data used for training is input into the model for pre-training, and the time domain contrast loss, frequency domain contrast loss, and time-frequency contrast loss are calculated. The three losses are used to calculate the overall loss according to the automatic loss mechanism, and then the model is optimized and updated.
5. The cross-subject EEG emotion recognition method based on time-frequency contrast learning according to claim 1 is characterized in that: The calculation methods of the three losses are as follows: L total =λ1L T +λ2L F +λ3L TF ---------------------------- (4) Formula (1) calculates the time domain contrast loss L T , where M is the number of time samples, and are the features extracted from the i-th time domain original signal and the time domain enhanced signal, τ represents the temperature coefficient of the adjustment scale, sim(v1,v2) represents the cosine similarity calculation of v1 and v2, and exp represents the exponential operation. Formula (2) calculates the frequency domain contrast loss L F , and are the features extracted from the jth original signal in the frequency domain and the enhanced signal in the frequency domain, Formula (3) calculates the time-frequency contrast loss L TF,i , and Respectively represent the underlying feature representations obtained by projecting the time domain components and the frequency domain components, Indicates from and Starting with a two-by-two similarity operation permutation, 1 [j≠i] Indicates that i and j will not be the same in a single operation. Formula (4) calculates the overall loss L total , where [λ1,λ2,λ3] are the weights corresponding to each loss calculated by the automatic loss mechanism.
6. The cross-subject EEG emotion recognition method based on time-frequency contrast learning according to claim 1 is characterized in that: The classifier consists of three fully connected layers.
Citation Information
Patent Citations
Behavior recognition model training method based on time-frequency fusion enhancement
CN116756571A
ConvNeXt-based multi-feature fusion picture emotion recognition method
CN117953253A
Cited By
Cross-individual EEG emotion recognition method based on comparative learning
CN119848630A
Construction method for classifying fish sound spectrum characteristic model and sound spectrum acquisition device
CN119964579A
