Cross-individual EEG emotion recognition method based on comparative learning

Abstract

The invention discloses a cross-individual EEG emotion recognition method based on comparative learning, and the method can efficiently extract and fuse cross-time scale spatio-temporal features through a time sequence multi-scale convolution and self-attention fusion model in combination with spatial feature coding, multi-scale time sequence convolution, a dimension projector and a spatio-temporal attention enhancement module. And the capability of capturing short-term and long-term characteristics of the EEG signal is enhanced. In order to solve challenges in cross-subject emotion recognition, a trial consistency contrast learning pre-training strategy is adopted, a signal segment of a multi-scale time window is input, positive and negative sample pairs are defined according to trial numbers, features are mapped to a contrast space through a trainable projection head, feature discrimination is optimized based on a contrast loss function, and the cross-subject emotion recognition method is implemented. And finally, through task adaptability fine tuning, the performance of cross-subject EEG emotion recognition is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of EEG emotion recognition, and in particular relates to a cross-individual EEG emotion recognition method based on contrastive learning. Background Art

[0002] Emotion recognition is one of the key technologies in the fields of human-computer interaction and intelligent health monitoring. Electroencephalogram (EEG), as a physiological signal reflecting the electrical activity of the brain, has become a research focus in the field of affective computing in recent years due to its high temporal resolution and sensitivity to emotional states. Compared with emotion recognition methods based on non-physiological signals such as facial expressions, voice or eye movements, EEG can provide more objective emotional information and effectively avoid interference from deliberate disguise. However, due to the high degree of individualization of EEG signals, cross-subject emotion recognition faces severe challenges in practical applications.

[0003] At present, EEG emotion recognition methods are mainly divided into two categories: traditional machine learning and deep learning. Traditional machine learning methods rely on manual feature extraction and shallow classifiers (such as support vector machines, random forests, etc.), but it is difficult to fully capture the complex nonlinear patterns in EEG signals. Deep learning methods can automatically extract emotion-related features from raw EEG data through end-to-end learning, and have made significant progress in emotion classification tasks. For example, convolutional neural networks (CNNs) are good at extracting local temporal and spatial features, recurrent neural networks (RNNs) can model the temporal dependencies of EEG signals, and Transformer architectures are good at capturing long-distance dependencies. However, most of these methods rely on within-subject training, that is, training and testing the model on the training data of the same subject, which leads to a significant decrease in their performance in cross-subject scenarios.

[0004] The main challenge of cross-subject EEG emotion recognition lies in the individual differences of EEG signals, such as changes in EEG waveform distribution and differences in neurophysiological characteristics between different subjects. These individual differences seriously restrict the model's generalization ability on new subjects. To solve this problem, researchers have proposed some methods such as domain adaptation and transfer learning. For example, in the literature [Li Yang, Zheng Wenming, Zong Yuan, Cui Zhen, Zhang Tong, Zhou Xiaoyan. EEG emotion recognition based on dual-hemispheric domain adversarial neural network [J]. IEEE Transactions on Affective Computing, 2018], a method based on adversarial domain adaptation was proposed. By designing a dual-hemispheric neural network model and using domain adversarial training to reduce the differences in EEG signal distribution between different subjects, this method improved the generalization ability of the model in the cross-subject emotion recognition task. In another example, in the literature [Li Jinpeng, Qiu Tianshuang, Shen Yingying, Liu Congli, He Huiguang. Application of multi-source transfer learning in cross-subject EEG emotion recognition [J]. IEEE Transactions on Cybernetics, 2020, 50(7): 3281-3293], a multi-source transfer learning framework was proposed. By integrating the EEG data of multiple source subjects to train shared feature representations, the model was fine-tuned with a small amount of target subject data to improve the recognition performance on new subjects. However, these methods usually still rely on the data of target subjects for model adjustment, which leads to problems of difficulty in data acquisition and high computational overhead in practical applications.

[0005] In addition, most existing EEG emotion recognition research focuses on instantaneous emotion classification within a short time window, and rarely involves modeling long-term emotion changes. However, long-term emotion recognition is crucial for applications such as mental health monitoring and emotion trend analysis; due to the temporal dynamic changes of EEG signals and the influence of external environment and physiological state, traditional methods have great difficulties in capturing the long-term evolution of emotions. Summary of the invention

[0006] In view of the above, the present invention provides a cross-individual EEG emotion recognition method based on contrastive learning, which utilizes a trial consistency strategy to construct contrastive learning samples and improves the generalization ability of the model between different subjects by strengthening the feature representation related to emotion during the training process.

[0007] A cross-individual EEG emotion recognition method based on contrastive learning includes the following steps: (1) Obtain an emotional EEG dataset, which stimulates the emotional state of the subject through emotion-inducing stimulation and synchronously collects the subject's EEG signal and its corresponding emotion label; (2) Standardize and preprocess the EEG signals in the dataset, and then divide the dataset into a training set and a test set; (3) Construct a temporal multi-scale convolution-self-attention fusion model, which includes: The feature extraction backbone network is used to extract multi-resolution features of the input EEG signal in the spatiotemporal domain and map the extracted low-dimensional features into high-dimensional features; A spatiotemporal attention enhancement module, used to construct contextual dependencies of spatiotemporal information in the high-dimensional features and generate context-aware enhanced features; (4) Pre-training the above fusion model with trial-to-trial consistency contrastive learning; (5) Connect the classification module to the output of the pre-trained fusion model, and use the training set to fine-tune the task adaptability of the fusion model connected to the classification module, thereby obtaining a cross-individual EEG emotion recognition model; (6) The EEG signals in the test set are input into the cross-individual EEG emotion recognition model, which can identify the output and obtain the corresponding emotion category.

[0008] Furthermore, the emotional EEG datasets obtained in step (1) come from two different research institutions, namely, SEED and FACED, two public datasets. Each dataset contains EEG signals from multiple different subjects and their corresponding emotional labels. The EEG signal in the SEED dataset has 62 channels, and the EEG signal in the FACED dataset has 32 channels.

[0009] Furthermore, the specific implementation method of the standardized preprocessing in step (2) is as follows: for the EEG signal in the SEED data set, the signal is first downsampled to 200 Hz, and then a bandpass filter is used to limit the signal frequency range to 0.3~49 Hz, and then the ICA (Independent Component Analysis) algorithm is used to remove artifact interference including eye movement and muscle movement in the signal, and finally the CAR (Common Average Reference) method is applied to complete the standardized preprocessing of the signal; for the EEG signal in the FACED data set, the signal is downsampled to 250 Hz and then subjected to preliminary filtering and artifact removal to complete the standardized preprocessing.

[0010] Furthermore, the specific implementation method of dividing the data set in step (2) is as follows: for the SEED data set, a leave-one-out cross-validation strategy is adopted: the EEG signals of the 15 subjects are divided into 15 independent groups, one group is selected as the test set each time, and the remaining 14 groups are used as training sets; for the FACED data set, a 10-fold cross-validation strategy is adopted: the EEG signals of all subjects are equally divided into 10 groups, one group is selected as the test set each time, and the remaining 9 groups are used as training sets, and it is ensured that the EEG signal of the same subject does not appear in both the training set and the test set.

[0011] Furthermore, the feature extraction backbone network includes: The spatial feature encoding module is used to extract spatial domain features of the input EEG signal along the electrode dimension and capture the spatial dependencies between different electrodes; The multi-scale temporal convolution module is used to extract multi-resolution features along the time dimension from the output of the spatial feature encoding module; The high-dimensional feature projection module is used to project the low-dimensional feature tensor output by the multi-scale temporal convolution module into a high-dimensional latent space to enhance the discriminative expression ability of the features.

[0012] Furthermore, the spatial feature encoding module sequentially passes the input EEG signal through a two-dimensional convolutional neural network and a group normalization layer and then outputs it; the multi-scale temporal convolution module inputs the output of the spatial feature encoding module into multiple parallel temporal convolutional neural networks respectively, and then splices the output results of each temporal convolutional neural network and then processes them through an ELU (Exponential Linear Unit) activation function and then outputs it; the high-dimensional feature projection module first averages the low-dimensional feature tensor output by the multi-scale temporal convolution module, and then uses a three-dimensional convolution kernel with a scale of 1 to construct a nonlinear mapping function to map the pooled features into high-dimensional features, and finally processes them through the ELU activation function and then outputs them.

[0013] Furthermore, the spatiotemporal attention enhancement module is composed of layer normalization L1, multi-head self-attention mechanism layer, layer normalization L2, and MLP (Multilayer Perceptron) connection from input to output, wherein the input of L1 is added to the output of the multi-head self-attention mechanism layer as the input of L2, and the input of L2 is added to the output of MLP as the output of the spatiotemporal attention enhancement module.

[0014] Furthermore, the specific implementation of step (4) is as follows: 4.1 For the EEG signals from different emotion-induced trials of different subjects in the training set, the EEG signals are intercepted by sliding time windows at multiple different window scales to obtain several EEG segments, and all EEG segment combinations in the training set are traversed to generate a large number of sample pairs, wherein the sample pairs contain two EEG segments of the same window scale. If the two EEG segments come from the same emotion-induced trial, the sample pair is a positive sample pair, otherwise it is a negative sample pair; 4.2 Connect the output of the temporal multi-scale convolution-self-attention fusion model to a nonlinear projection network consisting of two MLP connections; 4.3 The sample pairs are input into the fusion model connected to the nonlinear projection network in batches for pre-training, and the model parameters are iteratively optimized by the gradient descent method using the InfoNCE (Information Noise Contrastive Estimation) loss function.

[0015] Furthermore, the classification module connected in the step (5) is composed of a fully connected network and a Softmax activation function. In the process of fine-tuning the task adaptability of the fusion model, a cross entropy loss function and an adaptive moment estimation optimizer are used to perform end-to-end training on the classification module at a fixed learning rate.

[0016] Based on the above technical solutions, the present invention innovatively solves the two core challenges of cross-subject EEG emotion recognition through a trial-consistency contrast learning framework and a multi-scale spatiotemporal feature fusion mechanism: First, the trial-consistency contrast pre-training strategy effectively eliminates the differences between subjects and constructs a unified feature space with cross-subject generalization capabilities; second, a hybrid architecture of multi-scale temporal convolution and self-attention model is adopted, combined with the synergy of local receptive fields and temporal global attention mechanisms, to achieve continuous evolution modeling of emotional states across time scales. The present invention not only significantly improves the cross-subject robustness of emotion recognition, but also provides a scalable solution for the actual application scenarios of emotional brain-computer interfaces through an end-to-end deep learning architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the cross-individual EEG emotion recognition method based on contrastive learning of the present invention.

[0018] Figure 2 It is a structural schematic diagram of the temporal multi-scale convolution-self-attention fusion model of the present invention.

[0019] Figure 3 Schematic diagram of the model pre-training, fine-tuning and testing process based on trial consistency contrastive learning of the present invention. DETAILED DESCRIPTION

[0020] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0021] like Figure 1 As shown, the cross-individual EEG emotion recognition method based on contrastive learning of the present invention includes the following steps: (1) Obtain a standardized emotional EEG dataset, which stimulates the emotional state of the subjects through emotion-inducing stimuli (such as video or audio) and simultaneously collects multi-lead EEG signals and corresponding emotion labels.

[0022] In this embodiment, the data sets come from two different research institutions, SEED and FACED, and each data set contains multi-lead EEG signals from different subjects and corresponding emotional label information. The SEED data set was developed by the BCMI Laboratory of Shanghai Jiao Tong University. It uses 15 standardized movie clips as standardized stimulus materials, recruits 15 healthy subjects and collects signals through a 62-lead EEG system, and collects labeled data containing three emotional dimensions: positive, neutral, and negative. The FACED data set was released by the Department of Psychology of Tsinghua University. It is based on 28 standardized video stimuli to induce 9 types of emotions, covering 32-lead EEG records of 123 subjects, and its labels include 4 types of positive emotions (pleasure, motivation, happiness, tenderness), 4 types of negative emotions (anger, disgust, fear, sadness) and neutral state.

[0023] (2) The original EEG signals in the dataset are standardized and preprocessed to generate normalized time series data suitable for deep neural network input.

[0024] In this embodiment, for the SEED dataset, data containing 62 standard EEG recording channels are used. The signal is first downsampled to 200 Hz, and then a bandpass filter is used to limit the signal frequency range to 0.3-49 Hz. Then, an independent component analysis algorithm is used to remove artifacts such as eye movements and muscle movements, and finally the independent component value weighted average reference method is applied to complete the standardized preprocessing. For the FACED dataset, data containing 32 standard EEG recording channels are used, including 250 Hz EEG signals that have been preliminarily filtered and artifacts removed.

[0025] Next, the training set and test set are divided to ensure that the same training set is used in the pre-training and fine-tuning stages, while keeping the test set invisible. For the SEED dataset, a leave-one-subject cross-validation strategy is adopted to divide the 15 subjects into 14 independent groups. One group is selected as the test set for each experiment, and the remaining 14 groups are used as training sets. For the FACED dataset, a 10-fold cross-validation strategy is adopted to divide the subjects equally into 10 groups. One group is selected as the test set for each experiment, and the remaining 9 groups are used as training sets. The EEG data of the same subject will not appear in both the training set and the test set, and the amount of EEG data for each group must be balanced.

[0026] (3) Construct a temporal multi-scale convolution-self-attention fusion model, such as Figure 2 As shown, its architecture includes: The spatial feature encoding module uses a two-dimensional convolutional neural network to extract spatial domain features along the electrode dimension to capture the spatial dependencies between different electrodes. The convolution scale is the number of channels of the input signal. The specific expression is as follows:

[0027] in: X represents the input signal, Conv S represents a spatial convolutional network, GN represents the group normalization layer; The multi-scale temporal convolution module is composed of multiple parallel temporal convolution branches. Each branch is configured with a convolution kernel of a different time scale for multi-resolution feature extraction. The specific expression is as follows:

[0028] in: Represents the first i A convolutional network, Concat represents parallel splicing, E Represents the ELU activation function.

[0029] The high-dimensional feature projection module uses a three-dimensional convolution kernel with a scale of 1 to construct a nonlinear mapping function, projects the low-dimensional feature tensor output by the previous module into a high-dimensional latent space, and enhances the discriminative expression ability of the feature. The specific expression is as follows:

[0030] in: Conv P represents the 3D convolutional feature projector, Avgpool Represents average pooling.

[0031] The spatiotemporal attention enhancement module realizes the dynamic modeling of the global spatiotemporal feature relationship by calculating the spatiotemporal joint attention weight matrix based on the multi-head self-attention mechanism. The relative position encoding strategy is introduced to retain the temporal dependency characteristics of the bioelectric signal. The specific expression is as follows:

[0032] in: Attention represents the multi-head self-attention mechanism, Q , K and V denote query, key, and value matrices respectively, d K is the dimension of the key.

[0033] (4) Design trial-consistency contrast learning pre-training, such as Figure 3 As shown, the features are mapped to the contrastive space through a trainable projection head, and the feature discriminability is optimized based on the contrastive loss function.

[0034] First, for each emotion evoked trial, several EEG segments were captured by sliding the time window. The time windows corresponding to the three spans are Len1, Len2, and Len3 respectively; the sample pairs are defined according to the trial number to which the samples belong ( X , X' ), samples from the same trial are positive sample pairs, and samples from different trials are negative sample pairs.

[0035] Next, we build a trainable nonlinear projection head network consisting of two layers of MLP:

[0036] in: H Represents the feature vector obtained by the input signal after the fusion model, W 1 and W 2 is the learnable parameter matrix, b 1 and b 2 is the bias term, σ is the RELU activation function, outputting the projection feature Z .

[0037] Then, the multi-scale convolution-self-attention fusion model is combined with the nonlinear projection head to obtain:

[0038] in: M represents the fusion model, gIt is a non-linear projection head.

[0039] Finally, the samples are Input into the combined model in batches to obtain the corresponding projection features , the InfoNCE loss function is used to optimize the representation of the projection features output by the model in the feature space. The loss function is averaged under the input of different time window lengths to obtain the joint loss L con :

[0040]

[0041]

[0042] in: l represents time windows of different lengths, is the cosine similarity of the sample pair, The numerator is the similarity of the positive sample pairs in the batch, and the denominator is the sum of the similarities of all negative sample pairs in the batch. is the temperature hyperparameter, N is the number of sample pairs in a batch.

[0043] (5) Fine-tune the pre-trained model to make it more suitable for the emotion classification task.

[0044] Based on the pre-trained fusion model, a learnable classification head is connected to fine-tune the emotion classification task; in this implementation, the classification head is implemented by connecting a fully connected network and a Softmax activation function. The fully connected network consists of three layers of MLP and can be expressed by the following formula:

[0045] in: h represents the feature vector obtained from the pre-trained fusion model, W 1 , W 2 , W 3 is the learnable weight matrix, b 1 , b 2 , b 3 is the bias term, It is the probability distribution of the emotion category output by the model, and finally the corresponding category is selected by the maximum value.

[0046] In the parameter optimization stage, this implementation adopts the cross entropy loss function Lcls The classification layer is trained end-to-end with an adaptive moment estimation optimizer with a fixed learning rate, and a leave-one-subject cross-validation strategy is adopted to optimize the emotion category decision boundary through supervised learning.

[0047] (6) Deploy an emotion reasoning system across subjects.

[0048] The above-trained cross-subject emotion recognition model is used to predict the sleep stage on the test set data corresponding to each fold, and the prediction accuracy (Accuracy) and Macro-F1 score are calculated. The average value of the results of all folds is statistically calculated and compared with other existing methods. The comparison results are shown in Table 1. It can be seen from the table that the method of the present invention has a significant improvement in recognition effect compared with other existing recognition methods.

[0049] Table 1

[0050] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention should be within the scope of protection of the present invention.

Claims

1. A cross-individual EEG emotion recognition method based on contrastive learning, comprising the following steps: (1) Obtain an emotional EEG dataset, which stimulates the emotional state of the subject through emotion-inducing stimulation and synchronously collects the subject's EEG signal and its corresponding emotion label; (2) Standardize and preprocess the EEG signals in the dataset, and then divide the dataset into a training set and a test set; (3) Construct a temporal multi-scale convolution-self-attention fusion model, which includes: The feature extraction backbone network is used to extract multi-resolution features of the input EEG signal in the spatiotemporal domain and map the extracted low-dimensional features into high-dimensional features; A spatiotemporal attention enhancement module, used to construct contextual dependencies of spatiotemporal information in the high-dimensional features and generate context-aware enhanced features; (4) Pre-training the above fusion model with trial-to-trial consistency contrastive learning; (5) Connect the classification module to the output of the pre-trained fusion model, and use the training set to fine-tune the task adaptability of the fusion model connected to the classification module, thereby obtaining a cross-individual EEG emotion recognition model; (6) The EEG signals in the test set are input into the cross-individual EEG emotion recognition model, which can identify the output and obtain the corresponding emotion category.

2. The method for cross-individual EEG emotion recognition based on contrastive learning according to claim 1, characterized in that: The emotional EEG datasets obtained in step (1) come from two different research institutions, namely, SEED and FACED, two public datasets. Each dataset contains EEG signals from multiple different subjects and their corresponding emotional labels. The EEG signal in the SEED dataset has 62 channels, and the EEG signal in the FACED dataset has 32 channels.

3. The method for cross-individual EEG emotion recognition based on contrastive learning according to claim 2, characterized in that: The specific implementation method of the standardized preprocessing in step (2) is as follows: for the EEG signal in the SEED data set, the signal is first downsampled to 200 Hz, and then a bandpass filter is used to limit the signal frequency range to 0.3-49 Hz, and then the ICA algorithm is used to remove artifact interference including eye movement and muscle movement in the signal, and finally the common mean reference method is used to complete the standardized preprocessing of the signal; for the EEG signal in the FACED data set, the signal is downsampled to 250 Hz and then subjected to preliminary filtering and artifact removal to complete the standardized preprocessing.

4. The method for cross-individual EEG emotion recognition based on contrastive learning according to claim 2, characterized in that: The specific implementation method of dividing the data set in step (2) is as follows: for the SEED data set, a leave-one-out cross-validation strategy is adopted: the EEG signals of the 15 subjects are divided into 15 independent groups, one group is selected as the test set each time, and the remaining 14 groups are used as training sets; for the FACED data set, a 10-fold cross-validation strategy is adopted: the EEG signals of all subjects are equally divided into 10 groups, one group is selected as the test set each time, and the remaining 9 groups are used as training sets, and it is ensured that the EEG signal of the same subject does not appear in both the training set and the test set.

5. The method for cross-individual EEG emotion recognition based on contrastive learning according to claim 1, characterized in that: The feature extraction backbone network comprises: The spatial feature encoding module is used to extract spatial domain features of the input EEG signal along the electrode dimension and capture the spatial dependencies between different electrodes; The multi-scale temporal convolution module is used to extract multi-resolution features along the time dimension from the output of the spatial feature encoding module; The high-dimensional feature projection module is used to project the low-dimensional feature tensor output by the multi-scale temporal convolution module into a high-dimensional latent space to enhance the discriminative expression ability of the features.

6. The method for cross-individual EEG emotion recognition based on contrastive learning according to claim 5, characterized in that: The spatial feature encoding module sequentially passes the input EEG signal through a two-dimensional convolutional neural network and a group normalization layer and then outputs it; the multi-scale temporal convolution module inputs the output of the spatial feature encoding module into multiple parallel temporal convolutional neural networks respectively, and then splices the output results of each temporal convolutional neural network and then processes them through the ELU activation function before outputting them; the high-dimensional feature projection module first performs average pooling on the low-dimensional feature tensor output by the multi-scale temporal convolution module, and then uses a three-dimensional convolution kernel with a scale of 1 to construct a nonlinear mapping function to map the pooled features into high-dimensional features, and finally processes them through the ELU activation function before outputting them.

7. The method for cross-individual EEG emotion recognition based on contrastive learning according to claim 1, characterized in that: The spatiotemporal attention enhancement module is composed of layer normalization L1, multi-head self-attention mechanism layer, layer normalization L2, and MLP connection from input to output, wherein the input of L1 is added to the output of the multi-head self-attention mechanism layer as the input of L2, and the input of L2 is added to the output of MLP as the output of the spatiotemporal attention enhancement module.

8. The method for cross-individual EEG emotion recognition based on contrastive learning according to claim 1, characterized in that: The specific implementation of step (4) is as follows: 4.1 For the EEG signals from different emotion-induced trials of different subjects in the training set, the EEG signals are intercepted by sliding time windows at multiple different window scales to obtain several EEG segments, and all EEG segment combinations in the training set are traversed to generate a large number of sample pairs, wherein the sample pairs contain two EEG segments of the same window scale. If the two EEG segments come from the same emotion-induced trial, the sample pair is a positive sample pair, otherwise it is a negative sample pair; 4.2 Connect the output of the temporal multi-scale convolution-self-attention fusion model to a nonlinear projection network consisting of two MLP connections; 4.3 The sample pairs are input into the fusion model connected to the nonlinear projection network in batches for pre-training, and the model parameters are iteratively optimized by the gradient descent method using the InfoNCE loss function.

9. The method for cross-individual EEG emotion recognition based on contrastive learning according to claim 1, characterized in that: The classification module connected in the step (5) is composed of a fully connected network and a Softmax activation function. In the process of fine-tuning the task adaptability of the fusion model, a cross entropy loss function and an adaptive moment estimation optimizer are used to perform end-to-end training on the classification module at a fixed learning rate.

Images