Cross-subject electroencephalogram-based emotion recognition method

Abstract

A cross-subject electroencephalogram-based emotion recognition method, belonging to the technical field of deep learning. The method comprises the following steps: S1, using a positive and negative sample generator to construct extracted differential entropy features into positive and negative samples; S2, feeding differential entropy features of an anchor and the positive and negative samples into an encoder for encoding, mapping the encoded differential entropy features to a latent space, using an autoregressive model to perform regression prediction on an encoded anchor sample in the latent space, using a probabilistic supervised contrastive loss function to train the encoder, training, by means of reducing the distance between positive sample pairs and increasing the distance between negative sample pairs, the encoder to complete representation learning, and discarding the regression model after the representation learning is completed; and S3, connecting the trained encoder to a classifier for fine tuning, and training the classifier by means of a cross-entropy loss function, during which the encoder does not perform gradient propagation, so as to complete cross-subject emotion recognition. The present invention has a higher recognition accuracy and a smaller standard deviation.

Description

A method for cross-subject electroencephalogram signal emotion recognition TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning, and particularly relates to a method for cross-subject electroencephalogram signal emotion recognition. BACKGROUND

[0002] Emotion recognition is a key technology for realizing advanced human-computer interaction, and is widely used in the fields of psychology, artificial intelligence, medical treatment, entertainment services and the like, and helps to improve the humanization level of machines and improve the experience of human-computer interaction. Compared with non-physiological signals such as facial expressions, body postures and voices, electroencephalogram (EEG) directly reflects the activity of the brain, is not easily affected by the subjective consciousness and will of individuals, and has a high time resolution, and thus can provide objective and true emotion state information. The recognition rate of electroencephalogram is usually high, and different emotion states can be accurately distinguished. Therefore, emotion recognition by means of electroencephalogram is of great significance for the development of human-computer interaction. With the development of deep learning, more and more deep learning classification and recognition models have been applied to EEG emotion recognition. They use artificially designed features such as power spectral density (PSD) and differential entropy (DE), or encoded image features such as time-frequency graphs and frequency spectrum graphs as input forms, and combine various advanced networks and learning frameworks of deep learning such as convolutional neural networks, graph neural networks and attention mechanism-based transformers. These methods have achieved very high accuracy in the field of emotion recognition. At the same time, research shows that compared with image features, selecting artificially designed DE features as the input of a deep model will achieve a stable and high recognition rate. With the continuous progress of electroencephalogram acquisition technology and signal processing technology, emotion recognition using electroencephalogram has achieved many widely recognized research results.

[0003] Traditional emotion recognition models usually need to be personalized for each subject, which requires a large number of experiments and data labeling for each subject. In this case, the accuracy of intra-subject emotion recognition has reached more than 97% since 2022. However, in real-world applications, we often face new subjects whose emotional characteristics and expressions may be different from those in the training set. This is because different subjects have different skull shapes and different sensitivities to stimuli in the same task, resulting in individual differences in physiological activity between subjects. Therefore, compared to subject-independent, establishing a universal recognition method for all subjects and improving the accuracy of cross-subject emotion recognition is a greater challenge. Traditional machine learning algorithms usually rely on the assumption that training data and test data are independent and identically distributed when dealing with cross-subject tasks. However, this assumption often leads to a sharp decline in the performance of the trained traditional classifier in cross-subject tasks. In the past two years, cross-subject emotion recognition has mainly used deep learning methods to solve the problem, mainly including transfer learning, which includes domain adaptation (DA) and domain generalization technology (DG). DA uses the samples of the training set as the source domain and the samples of the test set as the target domain. The model migrates the knowledge obtained in the source domain to the target domain to minimize the difference in data distribution between the two domains. Although this technology can improve the accuracy by 20% compared to machine learning, the model still needs to use part of the test set to measure the difference between the two domains, so it still needs to retrain the model for subjects that the network has not seen before. DG, like DA, also divides into two domains. Its purpose is to find domain-invariant features in the source domain without accessing the test set data. It has shown outstanding performance in cross-subject tasks, so it has attracted more attention from researchers. In addition to transfer learning, Xin ke, Shen, etc. first applied contrastive learning to cross-subject emotion recognition. Using the contrastive learning method, they maximized the similarity of features for positive samples of the same emotional stimulus and minimized the similarity of features for negative samples of different stimuli. Their accuracy reached 86%, surpassing the transfer learning that has been developed for many years. This shows the great potential of contrastive learning in cross-subject emotion recognition. However, in this study, like most self-supervised contrastive learning, there is only one pair of positive samples. EEG-based emotion recognition is different from the recognition task in computer vision, with fewer emotion categories. Therefore, during training, a large number of false negative samples are pushed away as negative samples and anchor samples, affecting the final recognition accuracy. The above previous work shows that there are large differences in emotional expression between subjects, and there must also be subject-invariant representations. However, it is a challenge to have a high recognition accuracy without accessing the test set data and in the cross-subject case during model training. Therefore, exploring cross-subject emotion recognition methods is feasible and meaningful. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a method for emotion recognition across subject EEG signals, comprising the following steps:

[0005] S1. Use a positive and negative sample generator to construct positive and negative samples from the extracted differential entropy features;

[0006] S2. Input the differential entropy features of the anchor points and positive and negative samples into the encoder for encoding, and map them into the latent space. Use an autoregressive model to perform regression prediction on the encoded anchor point samples in the latent space. Use the probabilistic supervised contrastive loss function to train the encoder. By bringing the positive sample pairs closer and the negative sample pairs further apart, the encoder is trained to complete representation learning. After the representation learning is completed, the regression model will be discarded.

[0007] S3. Connect the trained encoder to the classifier for fine-tuning. Train the classifier using the cross-entropy loss function. During this process, the encoder does not perform gradient propagation to complete cross-subject emotion recognition.

[0008] Furthermore, in the constructed positive and negative samples, a positive and negative sample setting strategy combining supervised contrastive loss is adopted, incorporating sample label information into the design of the positive and negative samples. The positive and negative sample generator generates mini-batches as input to the contrastive learning encoder. I = {+, -, ×…} represents the set of emotions; taking the SEED dataset as an example, these represent happiness, sadness, and neutral emotions respectively. S = {1, 2, 3, …, n} represents the set of n subjects. All samples can be labeled as… q∈S, k∈I, C represents the number of channels, D represents the feature dimension extracted within a certain time period, and H represents the set of all samples in this dataset.

[0009] In a batch, a fixed emotion sample for subject 1 is first determined. Then, in each experiment, samples of the same emotion from the participants were taken as positive samples, i.e. f + The number is n*p + p + The number of experimental segments in the dataset that evoked a positive emotion; all participants whose emotions differed from the positive samples in each experiment were taken as negative samples. f - The number is n*p k (k∈I, k≠+);

[0010] To fully capture the characteristics of the sample in a batch, expand the mini batch, a sample 2s, take 6 consecutive sample sequences, define, fix the subject during the process of an experiment, that is, the emotion caused by a stimulus, such as in the SEED dataset, the average duration of the stimulus is 4min, containing 3 emotion categories, then 4*60 / (2*6) = 20 anchor samples will be generated, each anchor will correspond to N positive samples, 2N negative samples, their set e = {f,f + ,f -} as a batch, in the next batch, the anchor and positive and negative samples are reselected, until all samples are used as anchors, and an epoch of training is completed.

[0011] Further, by comparing the prediction coding design to build the feature extraction network, the positive sample pair is close, and the negative sample pair is far away;

[0012] First, the nonlinear encoder g enc maps the input sequence x(t) to the latent representation sequence z(t) = g enc (x t ), an autoregressive model g ar summarizes all z≤t in the latent space, and predicts a latent representation c(t) = g ar (z(t)); in contrast to predictive coding learning, a residual structure is used as the encoder g enc to avoid overfitting, and anchor samples and positive and negative samples are entered into the encoder in batches to obtain z k (t), and the anchor samples are then entered into the LSTM regression prediction network g ar to obtain the prediction result c(t), and LSTM is added as a regression model to improve the time resolution of the features, in the prediction process, the network learns the underlying features of the anchor emotion latent, and the prediction result c t is a feature representation with fixed point emotion, the prediction result and the feature z + (t) obtained by encoding the positive sample form a positive sample pair; and the feature z - (t) formed by encoding the negative sample is a negative sample pair, and finally the supervised contrast loss function is used to narrow the distance of the positive sample pair and to widen the distance of the negative sample pair, and the contrast prediction coding is completed.

[0013] The NCE loss function distinguishes between correct samples and a set of noise samples, and the model is trained by maximizing the probability of correct samples and minimizing the probability of noise samples; in contrast learning, it trains the model by contrasting positive and negative samples, as shown in equation (1):

[0014] Where m, p are the representation vectors of the samples obtained by f(·) network, m·p+ is the dot product similarity between anchor and positive sample i is the dot product similarity between anchor and other samples, K represents the number of negative samples, and τ is the temperature parameter.

[0015] Combining the idea of contrastive predictive coding (CPC), the training of the encoder and the regression model is also included in this loss function. Both the encoder and the autoregressive model are trained to optimize the NCE-based loss together, as shown in equation (2).

[0016] c h is the predicted vector of anchor sample h after g ar (x enc ) t ) q is the representation vector of positive sample of subject q after g enc (x t ) a is the representation vector of samples other than the anchor in a batch after g enc (x t ) Unlike CPC, which only uses samples from a fixed point as positive samples, the label information is combined with the CPC loss. Each fixed point can have multiple positive samples, i.e., samples with the same label are positive samples, so that the contrastive learning is adapted to the fully supervised case, as shown in equation (3).

[0017] q(h) represents the number of positive samples in the anchor, i.e., the number of subjects. The label information produces an embedding space, which is more compact than in the self-supervised case, helping the positive samples to have a more compact distribution in the embedding space.

[0018] Further, after contrastive learning, the encoder has learned to recognize the underlying logical features. The trained encoder is extracted to be used in the next step of classification. The input is no longer in the form of positive and negative sample pairs, but is a random and unordered sample of subjects. The encoder parameters are determined by the previous stage, and the encoder parameters are frozen in this stage. Only the classification head composed of fully connected layers and activation functions is trained through the cross-entropy loss function.

[0019] The beneficial effects of the present application are: the experimental results show that the method proposed in the present application has higher recognition accuracy and smaller standard deviation compared with most advanced methods at present, it can be seen that the performance of all methods on the SEED dataset is better than SEED IV, because under the same experimental paradigm, the SEED IV dataset belongs to four categories and has less data, compared with other methods, especially on the SEED IV dataset with greater challenge, our results are improved by at least 5% compared with the existing method, which shows that our method is less affected by the recognition category and has better generalization ability. And through the confusion matrix, we make a more detailed analysis of the recognition of each emotion, and our model has better performance for the category with strong emotional performance, which is consistent with the research of neurocognition: that is, strong emotions compared with calm emotions, there will be more obvious characteristics and similarities between subjects. In our model, LSTM is used to capture the temporal feature correlation and predict the related emotion, so the length and data volume of the sample will affect the experimental effect, therefore we make a comparative analysis on the number of samples, which shows that under the condition of using 6 samples as a minibatch, the optimal effect is obtained. At the same time, we also make an ablation analysis on the proposed loss function (S-Info NCE), compared with the ordinary cross-entropy loss function, our loss function can maximize the correlation and difference between samples, so it has better recognition effect. BRIEF DESCRIPTION OF DRAWINGS

[0020] Fig. 1 is a model framework diagram based on contrastive prediction of the present application (different shapes represent different subjects, and the same color and filling style represent the same emotion);

[0021] Fig. 2 is a framework diagram of positive and negative sample design of the present application;

[0022] Fig. 3 is a framework diagram of sample feature extraction of the present application;

[0023] Fig. 4 is a structure diagram of a supervised contrastive prediction encoding model of the present application (where the parameters of the encoder genc are shared);

[0024] Fig. 5 is a framework diagram of classifier training of the present application;

[0025] Fig. 6 is a schematic diagram of the experimental process of the SEED dataset of the present application;

[0026] Fig. 7 is a comparison diagram of the performance of each method on two datasets of the present application;

[0027] Fig. 8 is a confusion matrix diagram of each emotion under each dataset of the present application. DETAILED DESCRIPTION

[0028] In order to make the technical means and purposes of the present application easy to understand, the present application is further described below in combination with specific embodiments, a method for cross-subject electroencephalogram signal emotion recognition, the present application proposes a supervised contrastive predictive coding-based electroencephalogram signal emotion recognition which consists of three parts: positive and negative sample generator, supervised contrastive coding representation and fine-tuning classification, as shown in FIG. 1, specifically: first, the differential entropy features extracted are constructed into positive and negative samples by using the positive and negative sample generator; then, the differential entropy features of the anchor points and the positive and negative samples are sent into the encoder for coding, which is mapped into the latent space, an autoregressive model is used to predict the coded anchor point samples in the latent space, the encoder is trained by using the probability supervised contrastive loss function, the distance of the positive sample pairs is shortened and the distance of the negative sample pairs is lengthened, the encoder is trained to complete representation learning, and the regression model will be discarded after the completion of representation learning; finally, the trained encoder is connected to the classifier for fine-tuning, the classifier is trained by using the cross-entropy loss function, and the encoder does not propagate the gradient in this process, and cross-subject emotion recognition is completed.

[0029] 1. Positive and negative sample generator

[0030] In the positive and negative samples we constructed, the positive and negative sample setting strategy combined with the supervised contrastive loss is used, the label information of the samples is included in the design of the positive and negative samples, and the purpose of the positive and negative sample generator design is to generate a mini batch as the input of the contrastive learning encoder. In the positive and negative samples constructed by the present application, the positive and negative sample setting strategy combined with the supervised contrastive loss is used, and the label information of the samples is included in the design of the positive and negative samples. Define I={+,-,×…} to represent the set of emotions, and take the SEED data set as an example, which respectively represents three emotions of happy, sad and neutral. S={1,2,3,…,n} represents the set of n subjects, and all samples can be marked as (q∈S, k∈I, C represents the number of channels, D represents the feature dimension extracted within a certain time, and H represents the set of all samples under this data set). In a batch, first determine the fixed emotion samples of subject 1 Then, in each experiment, the samples of the same emotion of the subjects are taken as positive samples, that is f + The number of f is n*p + (p + represents the number of all experimental segments that trigger the + emotion of the data set). All samples of different emotions of the subjects in each experiment are taken as negative samples, that is f - The number of f is n*p k (k∈I, k≠+). As shown in FIG. 2, the design of the positive and negative samples.

[0031] To fully capture the characteristics of the sample in a batch, we will expand the mini batch. Instead of taking one sample (one sample 2s) in each experiment, we take 6 consecutive sample sequences. As shown in Figure 3. Define, fix the subject during an experiment (i.e. the emotion caused by a stimulus), as in the SEED dataset, the average duration of the stimulus is 4min, containing 3 emotion categories, then 4*60 / (2*6) = 20 anchor samples will be generated, each anchor will correspond to N positive samples, 2N negative samples, their set e = {f, f + , f -} as a batch. In the next batch, the anchor and positive and negative samples are reselected, until all samples are used as anchors, and an epoch of training is completed.

[0032] 2. Supervised contrastive predictive coding representation

[0033] The purpose of contrastive predictive coding design is to construct a feature extraction network so that the positive sample pair is close and the negative sample pair is far apart. First, a nonlinear encoder g enc maps the input sequence x(t) to the latent representation sequence z(t) = g enc (x t ), then an autoregressive model g ar summarizes all z ar ≤t in the latent space and predicts a latent representation c(t) = g ar (z(t)). In contrastive predictive coding learning, since the brain electrical signal dataset belongs to a small-scale dataset, in order to avoid overfitting, we use a residual structure as the encoder g enc , the anchor sample and the positive and negative samples enter the encoder in batches to obtain z k (t), and the anchor sample enters the LSTM regression prediction network g ar to obtain the prediction result c(t). Considering that the features obtained by the anchor point only through the encoder will have a lower time resolution, LSTM is added as a regression model. In the prediction process, the network learns the underlying features of the anchor point emotion, so the prediction result c t is a feature representation with fixed point emotion. The prediction result forms a positive sample pair with the feature z + (t) obtained by encoding the positive sample; and the feature z - (t) formed by encoding the negative sample is a negative sample pair. Finally, through the supervised contrastive loss function, the distance between the positive sample pair is narrowed, and the distance between the negative sample pair is far away, completing the contrastive predictive coding. The framework of supervised contrastive predictive coding is shown in Figure 4, taking 1s as an example.

[0034] The basic idea of the NCE loss function is to distinguish the correct samples from a set of noise samples, and train the model by maximizing the probability of correct samples and minimizing the probability of noise samples. In contrastive learning, it trains the model by contrasting positive samples and negative samples, as shown in equation (1):

[0035] where m, p are the representation vectors of the samples obtained by the f(·) network, m·p + is the dot product similarity between the anchor point and the positive sample, m·p i is the dot product similarity between the anchor point and other samples, K represents the number of negative samples, and τ is the temperature parameter.

[0036] Combining the idea of contrastive predictive coding (CPC), we also include the training of the encoder and the regression model in this loss function. Both the encoder and the autoregressive model are trained to optimize the NCE-based loss, as shown in equation (2).

[0037] c h is the predicted vector obtained by g ar (x enc ) t ) for the anchor sample h, z q is the representation vector obtained by g enc (x t ) for the positive sample of the subject q, A(h) = e\h, z a is the representation vector obtained by g enc (x t ) for the samples in a batch except the anchor point. Unlike CPC, which only considers samples from a fixed point as positive samples, we use label information combined with CPC loss. Each fixed point can have multiple positive samples, i.e., samples with the same label are considered as positive samples, so that contrastive learning can be adapted to the fully supervised case, as shown in equation (3).

[0038] q(h) represents the number of positive samples in the anchor point, i.e., the number of subjects. Since the label information produces an embedding space, its tightness will be higher than that in the self-supervised case, which helps the positive samples have a tighter distribution in the embedding space.

[0039] 3. Fine-tuning classification

[0040] After contrastive learning, the encoder has learned to recognize the underlying logical features. The trained encoder is used for the next step of classification, as shown in FIG. 5. At this time, the input will no longer be input in pairs of positive and negative samples as a batch, and the input is a random and unordered subject sample. The encoder parameters are determined by the previous stage, and the encoder parameters are frozen in this stage. Only the classification head composed of a fully connected layer and an activation function is trained by the cross-entropy loss function.

[0041] Experiment:

[0042] In our invention, SEED and SEED-IV are used as training data sets. First, the data set and preprocessing process are introduced. Second, the experimental design and implementation details are introduced. Then the experimental results are shown. Finally, the results are discussed.

[0043] The SEED data set is a publicly available electroencephalogram data set widely used in emotion recognition. The data set contains EEG data from 15 subjects (8 females, average age 23.27 years, age standard deviation 2.37 years). Each subject was asked to perform three different experiments at different times, with a gap of more than a week between sessions. The experimental team prepared movie clips that could evoke positive, neutral, and negative emotions in the subjects. There were five clips for each emotion, with an average duration of 226 seconds, for a total of 15 movie clips. Each subject watched 15 movie clips during each session. The experimental process included a 5-second prompt, 4 minutes of movie clips, 45 seconds of self-evaluation, and 15 seconds of rest, as shown in FIG. 6. The corresponding emotions were evoked through immersive viewing.

[0044] The experimental team used the 10-20 system to collect 62-channel EEG signals at a sampling frequency of 1000 Hz. The public SEED data set has been downsampled to 200 Hz and filtered between 0 and 75 Hz. The public data set also provides preprocessed experimental data. We selected differential entropy features smoothed by the linear dynamic system (LDS) method as experimental samples. We extracted 5 frequency band EEG signals (delta: 1-3 Hz, theta: 4-7 Hz, alpha: 8-13 Hz, beta: 14-30 Hz, gamma: 31-50 Hz) from each sample with a non-overlapping 1-second time window. Studies have shown that differential entropy (DE) features have stronger discriminative ability in emotion recognition. We concatenated the 62 DE values of the 5 frequency bands. To provide more stable DE features and reduce the impact of noise randomness, while ensuring the number of samples, this paper, like most literature, selected adjacent 2s DE features for splicing, resulting in a feature vector dimension of 62*10 samples. To ensure the processability and consistency of all data sets, we unified the input to 64*10.

[0045] SEED-IV dataset is also a widely used EEG dataset for emotion recognition. It contains 15 subjects' EEG signals, in each session, the subjects are set to watch movie clips that will cause four kinds of emotions: happiness, sadness, neutral, and fear, each emotion corresponds to four different clips, each subject watches 24 movie clips in each session.

[0046] The SEED-IV dataset is consistent with the preprocessing method provided by the SEED dataset, we also choose the same preprocessing method. The only difference is that the SEED-IV dataset is a non-overlapping 4-second time window as a sample, and the differential entropy features of different frequency bands are calculated respectively. Compared with the SEED dataset, the DE features extracted from the SEED-IV dataset contain sufficient time features, and in order to ensure sufficient data to train the model, the input is not spliced. The feature dimension of a sample is 62*5, we fill it to 64*10. Table 1 shows the feature size and feature dimension description of the two datasets.

[0047] Table 1 Dataset Description

[0048] In the cross-subject experiment, we conducted a cross-subject leave-one-subject-out experiment on the SEED and SEED-IV datasets. In order to compare with the most advanced experiment, we kept the same experimental setting as theirs. Specifically, our experimental test set is the sample of a subject, the training set and the validation set are all the samples except the test subject. If the dataset sample has N, then we need to conduct N times of leave-one-out experiment. In order to verify the rationality of the parameter design of our framework, we designed a comparative experiment. The number of samples in each minibatch will affect the experimental results in our designed framework. Too many numbers in a minibatch will lead to insufficient total training data, and too few numbers are not conducive to the network to capture potential emotional features, so we need to consider the number of samples and the amount of total training data. Experiments have shown that about 10s can represent a person's emotion, and in the SEED dataset, we have a sample length of 2s. Therefore, we designed a batch of positive (negative) sample numbers of 4, 5, 6, 7, and 8 to find the optimal number for our dataset. At the same time, in order to verify the effectiveness of the loss function, we did an ablation experiment on the loss function mentioned in the invention.

[0049] The model training in this paper uses Python 3.8 programming language and keras development framework to realize the model. For the encoder, according to the dimension of our input data and the size of the dataset, we designed a network model containing 16 layers of residual structure, and the network model parameters are shown in Table 2. In each residual block, the residual connection connects two convolutional layers. It should be noted that in Resnet_layer2, Resnet_layer3 and Resnet_layer4, the first layer of the residual module needs to pass through a convolutional layer with a size of 1x1, M, where M is the same as the channel number of the output F(x) of the current residual structure, and at the same time, it needs to pass through a down-sampling operation with a step of 2, so that the residual F(x) and the identity mapping x can be added. The hidden dimension of our regression model LSTM is set to 128, Adam is used as the optimizer, the learning rate is set to 0.001, and when the loss does not decrease for three times in succession, the learning rate is halved. The batch size is 64, and the first encoding prediction stage is trained for 80 epochs, and the second classification stage is trained for 40 epochs.

[0050] Table 2 Encoder network parameter structure

[0051] In order to prove the effectiveness of our invention, we compared the results of the SEED dataset across subjects experiment with the following six methods of each method, SVM (Support Vector Machines, SVM), TANN (novel Transferability Attention Neural Network, TANN), CLISA (Contrastive Learning Inter-Subject Alignment, CLISA), MSMRA (Multi-origin and Multi-presentation Adjustment, MSMRA), MS-ADA (Multisource Associate Domain Adaptation, MS-ADA), MGFKD (Multi-domain Geodesic Flow Kernel Dynamic Distribution Alignment, MGFKD).

[0052] Table 3 lists the accuracy of some methods in the SEED dataset (the data listed in Table 3 is from the accuracy of each subject method given in the corresponding paper). Tables 4 and 5 list the average accuracy and standard deviation across subjects on the SEED and SEED IV datasets.

[0053] Table 3 Accuracy of each subject for each method across subjects for the SEED dataset

[0054] Table 4 Mean and standard deviation of accuracy for each method across subjects for the SEED dataset

[0055] Table 5 Mean and standard deviation of accuracy for each method across subjects for the SEED IV dataset

[0056] In Table 3, it can be seen that our method has more subjects with higher accuracy than the other two methods. Due to the differences in models, the potential space found by the features extracted by the models is also different, so the performance of the subjects on different models is not consistent. In Tables 4 and 5, it can be seen that the average accuracy of this paper on the SEED dataset is 87.82%, with a standard deviation of 5.05%; the average accuracy on the SEED IV dataset is 74.62%, with a standard deviation of 5.31%. It is better than the machine learning, pre-adaptation, contrastive learning, domain transfer, and semi-supervised methods listed in the article, and the standard deviation is also lower. Compared with the domain adaptation method MS-ADA, which accesses the test set during the training of the network, our method improves by about 1.6% on the SEED dataset and by about 15% on the SEED IV dataset. Compared with CLISA using contrastive learning, the supervised prediction contrastive learning precision we use also improves by about 1.4%, and it reaches the same conclusion as it does that the number of classes in the dataset has a small effect on the model. Therefore, compared with the advanced cross-subject methods in recent years, our model has higher recognition accuracy.

[0057] [Corrected according to Rule 91 on 21.01.2025] Especially on the SEED IV dataset, our method's accuracy exceeds the suboptimal accuracy by nearly 5%, as shown in Figure 7, which shows the performance differences of each method on the two datasets. Compared with the SEED III classification set, the SEED IV has one more class, which may be the reason for the decrease in accuracy of all methods. However, due to the excellent performance of contrastive learning in multi-class tasks, this can explain why the number of classes has a lower impact on our model than other methods. The left-leaning hatch in Figure 7 represents the data points of the SEED dataset, the grid lines represent the data points of the SEED IV dataset, and the right-leaning hatch represents the difference values of the data points of the SEED dataset and the data points of the SEED IV dataset under each emotion.

[0058] In addition, we also analyzed the recognition rate of each emotion in the form of confusion matrix. In Figure 8, we give the confusion matrix of our method in the SEED, SEED IV dataset, from which we can get the average recognition accuracy under each emotion. It can be seen that in the SEED dataset, the recognition accuracy of positive emotion is as high as 90.8%, and 7.61% and 1.59% of positive emotions are misclassified as neutral and negative. In the SEED dataset, the model has strong classification effect for positive and negative emotions, and weak effect for neutral emotion. This phenomenon can be attributed to the complexity of neutral emotion, which is not as strong as positive and negative emotions, and does not have too obvious neural patterns. In the SEED IV dataset, the recognition effect of happy and sad is also better than that of neutral emotion.

[0059] At the same time, we made a comparative experiment on the number of samples in minibatch. In the SEED dataset, we randomly selected three subjects 2, 6, 9, and set different sample sizes 4, 5, 6, 7, 8 to find the optimal sample size. As shown in Table 6, the three subjects have the optimal recognition accuracy when the sample size is 6, which shows that appropriate sample size and sufficient data volume can help the network capture signal features and improve recognition accuracy.

[0060] Table 6 Recognition results of different sample sizes of random subjects (%)

[0061] In order to verify the effectiveness of the supervised contrastive prediction loss function (S-Info NCE), we also conducted an ablation experiment on the loss function of the contrastive prediction learning stage. We randomly selected three subjects 2, 6, 9, and calculated their average accuracy under two kinds of loss functions, as shown in Table 7, the use of supervised contrastive prediction loss function has higher accuracy than basic cross entropy loss function. This shows that the supervised contrastive prediction loss function can promote the diversity and discriminability of feature learning. By maximizing mutual information, it encourages the model to learn to capture the correlation and difference between samples, so as to produce more informative feature representation and improve the accuracy of cross-subject recognition.

[0062] Table 7 Recognition results of different loss functions

[0063] The experimental results show that the proposed method has higher recognition accuracy and smaller standard deviation compared with most advanced methods. It can be seen that the performance of all methods on the SEED dataset is better than SEED IV, because under the same experimental paradigm, the SEED IV dataset belongs to four categories and has less data. Compared with other methods, our results are at least 5% higher than existing methods, especially on the more challenging SEED IV dataset. This shows that our method is less affected by the recognition category and has better generalization ability. And through the confusion matrix, we make a more detailed analysis of the recognition of each emotion. Our model has better performance for categories with strong emotional expression, which is consistent with the research of neurocognition: that is, strong emotions compared to calm emotions, there will be more obvious characteristics and similarities between subjects. In our model, LSTM is used to capture the temporal correlation of features and predict related emotions, so the length and amount of data will affect the experimental results. Therefore, we make a comparative analysis of the number of samples, which shows that the optimal effect is obtained when using 6 samples as a minibatch. At the same time, we also conduct ablation analysis on the proposed loss function (S-Info NCE). Compared with the ordinary cross-entropy loss function, our loss function can maximize the correlation and difference between samples, so it has better recognition effect.

[0064] The above merely illustrates the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes to the technical solutions and concepts of the present application within the disclosed technical scope, which should be covered within the protection scope of the present application.

Claims

1. A method for emotion recognition across brain electrical signals of subjects, characterized in that, The method comprises the following steps: S1, using positive and negative sample generator to construct the extracted differential entropy features into positive and negative samples; S2, sending the anchor point and the differential entropy features of the positive and negative samples into an encoder for encoding, mapping them into a latent space, using an autoregressive model to make regression prediction on the encoded anchor point samples in the latent space, training the encoder using a probability supervised contrast loss function, training the encoder to complete representation learning by shortening the distance between positive sample pairs and lengthening the distance between negative sample pairs, and discarding the regression model after completing the representation learning; S3, connecting the trained encoder to a classifier for fine-tuning, training the classifier through a cross-entropy loss function, and completing cross-subject emotion recognition without gradient propagation of the encoder.

2. The method of emotion recognition from brain electrical signals across subjects of claim 1, wherein, In the constructed positive and negative samples, the positive and negative sample setting strategy combined with the supervised contrast loss is used, the label information of the sample is included in the design of the positive and negative samples, and the sample generator generates a minibatch as the input of the contrast learning encoder; define I={+, -, ×...} to represent the set of emotions, take the SEED dataset as an example, represent three emotions of happiness, sadness and neutrality respectively, S={1, 2, 3,..., n} represents the set of n subjects, and all samples can be marked as q∈S, k∈I, C represents the number of channels, D represents the feature dimension extracted within the determination time, and H represents the sample set of all samples under this dataset. In one batch, the fixed emotion sample for subject 1 was determined first Then, in each experiment, the samples of the same emotion of the subjects were taken as positive samples, that is f + The number of n * p + , p + represents the number of all experimental fragments of the data set trigger + emotion; take all samples of the subjects different from the positive sample emotion in each experiment as negative samples, that is f - the number of n * p k (k ∈ I, k ≠ +); To fully capture the characteristics of the sample in a batch, expand the mini batch, a sample 2s, take 6 consecutive sample sequences, define, fix the subject during the process of an experiment, that is, the emotion caused by a segment of stimulation, such as in the SEED dataset, the average duration of the stimulus is 4min, containing 3 kinds of emotional classification, then 4*60 / (2*6)=20 anchor samples will be generated, each anchor will correspond to N positive samples, 2N negative samples, their set e={f,f + ,f -} as a batch, in the next batch, the anchor and positive and negative samples are reselected, until all samples are used as anchors, and an epoch of training is completed.

3. The method of emotion recognition from brain electrical signals across subjects of claim 2, wherein, The feature extraction network is designed and constructed by contrast prediction coding, so that the positive sample pairs are close and the negative sample pairs are far apart. First, the nonlinear encoder g enc maps the input sequence x(t) to a latent representation sequence z(t) = g enc (x t ), an autoregressive model g ar summarizes all z≤t in the latent space and produces a latent representation c(t) = g ar (z(t)); in contrastive predictive coding learning, a residual structure is used as the encoder g enc to avoid overfitting, anchor samples and positive and negative samples are batched into the encoder to obtain z k (t), and the anchor samples are then fed into the LSTM regression prediction network g ar to obtain the prediction result c(t), and LSTM is added as a regression model to improve the temporal resolution of the features; in the prediction process, the network learns the underlying features of the anchor point's sentiment, and the prediction result c t is a feature representation with a fixed point emotion; the prediction result forms a positive sample pair with the feature z + (t) obtained after encoding the positive sample; and the feature z - (t) formed after encoding the negative sample forms a negative sample pair; finally, the supervised contrastive loss function is used to narrow the distance between the positive sample pair and to increase the distance between the negative sample pair, completing the contrastive predictive coding. The model is trained by distinguishing correct samples from a set of noisy samples through an NCE loss function, by maximizing the probability of correct samples and minimizing the probability of noisy samples; in contrastive learning, it trains the model by contrasting positive samples and negative samples, as shown in equation (1): where m, p are the representation vectors of the sample passing through the f(·) network, m·p + is the dot product similarity between the anchor and positive samples, m·p i is the dot product similarity between the anchor and other samples, K represents the number of negative samples, and τ is the temperature parameter; Incorporating the idea of contrastive predictive coding (CPC), the training of both the encoder and the regression model are also included in this loss function, and both the encoder and the autoregressive model are trained to jointly optimize the NCE-based loss as shown in equation (2), c h the prediction vector of the anchor sample h after g ar (g enc (x t )), z q the representation vector of the positive sample of the subject q after g enc (x t ), A(h) = e\h, z a the representation vector of the sample other than the anchor point in a batch after g enc (x t ) ; unlike CPC, which only takes samples from fixed points as positive samples, the label information is combined with the CPC loss, and each fixed point can have multiple positive samples, i.e. the same label is a positive sample, so that the contrast learning is adapted to the full supervision, as shown in the following formula (3), q(h) represents the number of positive samples in the anchor point, i.e. the number of subjects, and the label information generates an embedding space, which is more compact than in the self-supervised case, helping the positive samples to have a more compact distribution in the embedding space.

4. The method of emotion recognition from brain electrical signals across subjects of claim 3, wherein, After contrast learning, the encoder has learned to recognize the underlying logical features. The trained encoder is extracted for the next step of classification. The input is no longer input in the form of positive and negative sample pairs, but is a random and unordered subject sample. The encoder parameters are determined in the previous stage, and the encoder parameters are frozen in this stage. Only the classification head composed of fully connected layers and activation functions is trained through the cross-entropy loss function.

Images