Progressive unsupervised subdomain adaptive electroencephalogram emotion recognition method

Abstract

The invention belongs to the field of signal processing technology research, provides a progressive unsupervised subdomain adaptive electroencephalogram emotion recognition method, and solves the problem of weak generalization ability of a cross-subject emotion recognition model caused by significant individual difference of electroencephalogram signals among subjects. The method comprises the following steps: firstly, extracting EEG differential entropy features for describing EEG signal amplitude distribution complexity; secondly, calculating distribution difference between a source domain and a target domain through a progressive domain self-adaptive algorithm; in the domain self-adaption process, the edge distribution difference between a source domain and a target domain is calculated through a maximum mean difference (MMD) algorithm, and the conditional distribution difference between the source domain and the target domain is calculated through a weighted clustering (WCD) algorithm. And a dynamic factor is designed in the training process to realize conversion from edge distribution difference to conditional distribution difference. And finally, constructing an emotion recognition system, extracting domain invariant features between a source domain and a target domain, and improving cross-subject emotion recognition accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of signal processing, and provides a progressive unsupervised sub-domain adaptive electroencephalogram emotion recognition method for cross-subject emotion recognition. BACKGROUND

[0002] Emotion plays a crucial role in human life, and it affects people's physiological responses and decision-making processes. In the field of human-computer interaction, emotion recognition technology has shown great application potential. Emotion recognition methods are mainly divided into physiological signals and non-physiological signals, among which non-physiological signals include facial expressions, language and gestures, and physiological signals include electroencephalogram (EEG), electromyogram (EMG), electrocardiogram (ECG), etc. Compared with non-physiological signals, physiological signals are more objective and difficult to fake because they directly reflect the physiological processes inside the body, so they have been widely used in the field of emotion recognition. Numerous studies have shown that there is a close and important relationship between the generation of human emotions and the electrical signals released by the cerebral cortex. Electroencephalogram (EEG) is a non-invasive physiological signal that can directly record the brain electrical activity in the emotional state. Compared with other physiological signals, EEG signal has the advantages of high time resolution, high data acquisition and transmission efficiency, and low cost, so it has received widespread attention and research from the academic and industrial communities.

[0003] Currently, the field of electroencephalogram emotion recognition has explored numerous features from different disciplines for EEG (electroencephalogram) emotion recognition, including differential entropy, power spectral density, etc. In order to effectively classify these EEG features, researchers have proposed a variety of machine learning methods. Among them, support vector machine (SVM), K-nearest neighbor algorithm (KNN) and random forest (RF) have been widely used and have achieved significant research results. However, traditional machine learning techniques are highly dependent on manual feature extraction, which is both time-consuming and labor-intensive. Given the significant advantages of deep learning in automatic feature extraction, many EEG emotion recognition methods based on deep neural networks have emerged.

[0004] However, the electroencephalogram signal is weak and is easily contaminated by interference and noise, and there are differences between different subjects and sessions. Therefore, it is a challenging task to establish an emotion recognition model suitable for different subjects, different sessions and different devices in an electroencephalogram-based brain-computer interface system. Patent No. CN202311131272.0 discloses a cross-subject electroencephalogram emotion recognition method and system based on multi-branch sample selection. First, the electroencephalogram signal is converted into an electroencephalogram feature map using three-dimensional mapping. Then, the model is warmed up before training using differential warm-up learning to learn the ability to distinguish physiological differences between subjects in advance. The local refinement and global structure features of the electroencephalogram feature map are processed through high and low frequency windows. Finally, the spatial interaction KAN is used to capture the spatial interaction features of the electroencephalogram signals of different channels between different subjects to enhance the modeling ability of the model for different electroencephalogram patterns. The sample selection module is used to select the most similar samples to the target domain, resulting in a single trend in sample features, which in turn causes fluctuations or inconsistencies in the performance of the model in identifying different emotional states. In addition, the multi-branch architecture is used for cross-subject emotion recognition tasks, which may improve recognition performance but also significantly increases the complexity of the model and leads to a corresponding increase in training costs. SUMMARY

[0005] In view of the above, the present application provides a progressive unsupervised sub-domain adaptive electroencephalogram emotion recognition method, which solves the problem of weak generalization ability of cross-subject emotion recognition models due to significant individual differences between subjects.

[0006] In order to achieve the purpose of the present application, the technical solution provided by the present application is as follows:

[0007] A progressive unsupervised sub-domain adaptive electroencephalogram emotion recognition method, comprising the following steps:

[0008] Step 1: electroencephalogram signal preprocessing, extracting differential entropy features

[0009] The original electroencephalogram signal data is subjected to a downsampling operation to adjust its sampling rate to 200Hz, and a bandpass filter in the frequency range of 1Hz to 75Hz is used to process the signal. Then, the differential entropy features DE are extracted from the preprocessed electroencephalogram signal; Step 2: adaptation of different subject domains in the source domain

[0010] Step 2.1: aligning the EEG feature distribution of different subjects in the source domain;

[0011] Step 2.2: calculating the edge distribution difference between different subjects in the source domain;

[0012] Step 2.3: calculating the conditional distribution difference between different subjects in the source domain;

[0013] Step 2.4: Transforming the marginal distribution difference of different subjects in the source domain to the conditional distribution difference;

[0014] Step 3: Domain adaptation of the source domain and the target domain

[0015] Step 3.1: Performing an alignment operation on the EEG feature distribution of the target subject and multiple subjects in the source domain;

[0016] Step 3.2: Calculating the marginal distribution difference between the source domain and the target domain;

[0017] Step 3.3: Calculating the conditional distribution difference between the source domain and the target domain: classifying the emotional features of the source domain and the target domain using the emotional labels of different subjects in the source domain and the predicted labels in the target domain, respectively, and then realizing the alignment of the features in the emotional sub-domain of the source domain and the target domain;

[0018] Step 3.4: Transforming the marginal distribution difference in the source domain and the target domain to the conditional distribution difference;

[0019] Step 4: Constructing an emotional recognition system for classification and recognition.

[0020] Further, the EEG feature information covering 5 different frequency bands obtained in step 1 is as follows: Delta band (1-4 Hz), Theta band (4-8 Hz), Alpha band (8-14 Hz), Beta band (14-31 Hz), and Gamma band (31-50 Hz), and the differential entropy formula is as follows:

[0021]

[0022] Further, the maximum mean difference MMD algorithm is used to calculate the marginal distribution difference in step 2.2, as shown in equation (2):

[0023]

[0025] Further, the weighted clustering (WCD) algorithm is used to calculate the conditional distribution difference in step 2.3, as shown in equation (3):

[0026]

[0027] wherein, represents the weighted contrastive domain difference, tr(A T S w A) represents the intra-class scatter matrix;

[0028]

[0029] wherein

[0030]

[0031] wherein represents the output of the classifier, y(·) represents the one-hot label of the source domain, and since the target domain is unlabeled, the pseudo label predicted by the network is used, and is denoted, when c=c', the feature distribution difference between the source domain and the target domain under the same category is calculated, and when c≠c', the feature distribution difference between the source domain and the target domain under different categories is calculated;

[0032] wherein the intra-class scatter matrix denotes the distance between the sample and its class mean, and the intra-class scatter matrix is defined as follows:

[0033]

[0034] wherein C represents the number of categories, respectively represent the number of samples of the i-th category in the source domain and the number of samples of the i-th category in the target domain, respectively represent the mean of the i-th category training sample in the source domain and the mean of the i-th category training sample in the target domain.

[0035] Further, a dynamic factor a in step 2.4 is counted to realize the conversion of the edge distribution difference to the conditional distribution difference;

[0036]

[0037] wherein, denotes the edge distribution difference, denotes the conditional distribution difference.

[0038] Further, the calculation in the domain adaptation of the source domain and the target domain in step 3 is the same as step 2.

[0039] Further, the emotion recognition system in step 4 includes a real-time signal acquisition module, a real-time signal processing module and an emotion recognition result display module;

[0040] The real-time signal acquisition module collects the EEG signal through the EEG signal acquisition device, and then transmits the EEG signal to the real-time signal processing module in real time through network communication. The real-time signal processing module performs signal preprocessing on the transmitted EEG signal, extracts differential entropy features, calculates the distribution difference between the source domain and the target domain, including the edge distribution difference and the conditional distribution difference, adjusts the edge distribution difference and the conditional distribution difference using dynamic factors, finally classifies the extracted domain-invariant features and transmits the classification results to the emotion recognition result display module. After receiving the classification results output by the signal processing module, the emotion recognition result display module encodes the display instructions, so that the final classification results can be presented on the display.

[0041] Compared with the prior art, the present application has the following advantages:

[0042] (1) The present application proposes a gradual unsupervised sub-domain adaptive EEG emotion recognition method. First, the data distribution difference between different subjects in the source domain is accurately measured by quantitative analysis means. The core goal is to effectively reduce the domain shift phenomenon between different subjects in the source domain. Then, the alignment operation between the source domain and the target domain is further implemented to avoid the negative transfer effect caused by different subjects in the source domain having similar EEG feature representations but corresponding to different emotional states, and to ensure the performance and stability of the cross-domain emotion recognition model.

[0043] (2) The present application provides an improved weighted clustering difference algorithm (WCD) for calculating the conditional distribution difference. By clustering the samples and assigning weights to different clusters, the distribution difference between different sub-domains is quantified. It can calculate the distribution difference between two sub-domains. By minimizing the distribution difference calculated by WCD, the intra-class compactness and inter-class separability between the source domain and the target domain are improved, thereby obtaining more domain-invariant emotional features. Combined with the subsequent feature alignment and adaptive strategy, the generalization ability of the model to cross-domain emotional features can be enhanced.

[0044] (3) The gradual unsupervised sub-domain adaptive architecture proposed in the present application effectively alleviates the distribution ambiguity problem between the target domain and the source domain. In addition, through the gradual domain adaptation module and the sub-domain adaptation, the data distribution alignment between the source domain and the target domain is further promoted. Therefore, by solving the challenge of EEG signal distribution difference, the present application significantly improves the accuracy of cross-subject EEG emotion recognition. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 It is an implementation flowchart of a gradual unsupervised sub-domain adaptive EEG emotion recognition method.

[0046] Figure 2 It is a framework diagram of an EEG cross-subject emotion recognition system. DETAILED DESCRIPTION

[0047] In order to make the purposes, technical solutions and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below with reference to the drawings.

[0048] A progressive unsupervised sub-domain adaptive electroencephalogram emotion recognition method. As shown in the figure, the method comprises the following steps: Figure 1

[0049] Step 1: EEG signal preprocessing, extracting differential entropy features

[0050] Downsample the original EEG signal data to adjust the sampling rate to 200Hz. In order to effectively filter out noise interference in the signal and eliminate artifact components, a band-pass filter in the frequency range of 1Hz to 75Hz is used to process the signal. Then, the differential entropy features (DE) are extracted from the preprocessed EEG signal, and further subdivided to obtain EEG feature information covering 5 different frequency bands, which are Delta band (1-4Hz), Theta band (4-8Hz), Alpha band (8-14Hz), Beta band (14-31Hz) and Gamma band (31-50Hz). The differential entropy formula is as follows:

[0051]

[0052] Step 2: Domain adaptation of different subjects in source domain

[0053] Step 2.1: Progressive domain adaptation: align the EEG feature distribution of different subjects in the source domain;

[0054] Step 2.2: Calculate the difference of marginal distribution: use the maximum mean difference (MMD) algorithm to calculate the difference of marginal distribution between different subjects in the source domain, as shown in equation (2):

[0055]

[0056] Step 2.3: Calculate the difference of conditional distribution: use the weighted clustering (WCD) algorithm, as shown in equation (3):

[0057]

[0058] Wherein, represents the weighted contrastive domain difference, tr(A T S w A) represents the within-class scatter matrix;

[0059] ​The weighted contrastive discrepancy algorithm is an improvement of the contrastive discrepancy algorithm. Since the target domain is unlabeled, the classifier will make uncertain decisions when predicting the target domain, which may lead to an unclear difference between the class distributions. Therefore, weights are introduced based on the contrastive discrepancy to optimize the distribution discrepancy between the two domains by assigning different weights to the samples. Specifically, if the decision made by the classifier is more ambiguous, the sample will get a larger weight, and the sample will play an important role in calculating the distribution discrepancy, which is beneficial for the sample to move away from the decision boundary. The weighted contrastive discrepancy is represented by formula (4):

[0060]

[0061] wherein

[0062]

[0063] The sample weight formula is calculated by (5) and (6). Wherein represents the output of the classifier, y(·) represents the one-hot label of the source domain, and since the target domain is unlabeled, the pseudo label predicted by the network is used, and is represented. When c = c', the feature distribution discrepancy between the source domain and the target domain under the same category is calculated, and when c≠c', the feature distribution discrepancy between the source domain and the target domain under different categories is calculated.

[0064] wherein the intra-class scatter matrix is used to represent the distance between the sample and its class mean, and by minimizing the distance between each projected sample and its mean, the clustering of the same sample labels between the source domain and the target domain is promoted, and the intra-class scatter matrix is defined as follows:

[0065]

[0066] wherein C represents the number of categories. and represent the number of samples of the i-th category in the source domain and the target domain, respectively. and represent the mean of the i-th category of training samples in the source domain and the target domain, respectively.

[0067] Step 2.4: Dynamic factor adjustment: In the early stage of training, since the target domain data has not been labeled, and there is a significant distribution difference between the source domain and the target domain, which may weaken the accuracy of the pseudo label on the target domain, and further produce a large deviation when calculating the weighted clustering loss. In order to solve the above problems, a dynamic factor a is designed to realize the conversion of the marginal distribution discrepancy to the conditional distribution discrepancy;

[0068]

[0069] Step 3: Domain adaptation of source domain and target domain

[0070] Step 3.1: Progressive domain adaptation: aligning the EEG feature distribution of target subject and multiple subjects in source domain;

[0071] Step 3.2: Computing marginal distribution difference: using the maximum mean difference (MMD) algorithm to calculate the marginal distribution difference between the source domain and the target domain, formula (2) in step 2.2;

[0072] Step 3.3: Computing conditional distribution difference: using the emotional labels of different subjects in the source domain and the predicted labels in the target domain to classify the emotional features of the source domain and the target domain respectively, and then realizing the alignment of the features in the emotional sub-domain of the source domain and the target domain, formula (3) in step 2.3;

[0073] Step 3.4: Dynamic factor adjustment: designing a dynamic factor a to realize the conversion of marginal distribution difference to conditional distribution difference, formula (8) in step 2.4;

[0074] Step 4: Constructing an emotional recognition system for classification and recognition.

[0075] As shown in Figure 2 , the emotional recognition system includes three modules: real-time signal acquisition module, real-time signal processing module and emotional recognition result display module.

[0076] Real-time signal acquisition module: including EEG signal acquisition and network communication real-time transmission, first using EEG signal acquisition device (EEG cap) to collect signals, and then using network communication to transmit EEG signals to the signal processing module in real time.

[0077] Real-time signal processing module: including signal preprocessing, extracting differential entropy features, progressive domain adaptation, computing marginal distribution difference, computing conditional distribution difference, and using dynamic factor to adjust marginal distribution difference and conditional distribution difference. First, the original EEG signal data is down-sampled to adjust the sampling rate to 200Hz, and a band-pass filter in the frequency range of 1Hz to 75Hz is used to process the signal. In order to describe the complexity of the amplitude distribution of EEG signal, the differential entropy feature of EEG is extracted. Then, the progressive domain adaptation algorithm is used to calculate the distribution difference between the source domain and the target domain. In the process of domain adaptation, the maximum mean difference (MMD) algorithm is used to calculate the marginal distribution difference between the source domain and the target domain, and the weighted clustering (WCD) algorithm is used to calculate the conditional distribution difference between the source domain and the target domain. A dynamic factor is designed to realize the conversion of marginal distribution difference to conditional distribution difference in the training process. Finally, the domain-invariant features are classified.

[0078] The emotion recognition result display module comprises real-time wireless transmission and display instruction coding, and after receiving the classification result output by the signal processing module by using the real-time wireless transmission device, the display instruction coding is performed, so that the final classification result can be presented on the display.

[0079] To verify the effectiveness of the method (PUSA method) provided by the present application, comparisons were made with other methods (joint distribution adaptation (JDA), gated recurrent unit-minimum class confusion (GRU-MCC), unsupervised domain adaptation (UDDA), joint feature adaptation and graph propagation of adaptive labels (JAGP), graph-based unsupervised sub-domain adaptation (GUSA), multi-source marginal distribution adaptation (MS-MDA), multi-source feature alignment and label correction method (MFA-LR), multi-source feature representation and alignment network (MS-FRAN), multi-source joint domain adaptation (MSADA), multi-domain geodesic flow kernel dynamic distribution alignment (MGFKD), and multi-source distribution deep adaptive feature normalization network (MSD-DAFNN)), as shown in Table 1. The results show that the PUSA method achieves better performance in the cross-subject emotion recognition task of the Seed and Seed-IV datasets. Since the PUSA method takes into account the distribution difference between different subjects in the source domain, it exhibits better performance compared to the single-source domain adaptation method. Compared to the multi-source domain adaptation method with higher accuracy, the PUSA model shows smaller variance.

[0080] Table 1 Cross-subject classification accuracy of different methods on Seed and Seed-IV

[0081]

[0082] The above describes the specific embodiments of the present application in conjunction with the accompanying drawings, but is not a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A progressive unsupervised sub-domain self-adaptive electroencephalogram emotion recognition method, characterized in that, The method comprises the following steps: Step 1: EEG signal preprocessing, extracting differential entropy features The original EEG signal data is down-sampled to adjust the sampling rate to 200 Hz, and a band-pass filter in the frequency range of 1 Hz to 75 Hz is used to process the signal, and then the differential entropy features DE are extracted from the preprocessed EEG signal; Step 2: Adaptation of different subjects in the source domain Step 2.1: Align the EEG feature distribution of different subjects in the source domain; Step 2.2: Calculate the marginal distribution difference between different subjects in the source domain; Step 2.3: Calculate the conditional distribution difference between different subjects in the source domain; Step 2.4: Convert the marginal distribution difference of different subjects in the source domain to the conditional distribution difference Step 3: Domain adaptation of source domain and target domain Step 3.1: Align the EEG feature distribution of the target subject and multiple subjects in the source domain; Step 3.2: Calculate the marginal distribution difference between the source domain and the target domain; Step 3.3: Calculate the conditional distribution difference between the source domain and the target domain: classify the emotional features of the source domain and the target domain using the emotional labels of different subjects in the source domain and the predicted labels in the target domain, and then align the features of the source domain and the target domain in the emotional sub-domain; Step 3.4: Convert the marginal distribution difference in the source domain and the target domain to the conditional distribution difference; Step 4: Construct an emotion recognition system for classification and recognition.

2. The progressive unsupervised sub-domain adaptive electroencephalogram emotion recognition method according to claim 1, characterized in that: The EEG feature information covering 5 different frequency bands obtained in step 1 is as follows: Delta band (1-4 Hz), Theta band (4-8 Hz), Alpha band (8-14 Hz), Beta band (14-31 Hz), and Gamma band (31-50 Hz). The differential entropy formula is as follows:

3. The progressive unsupervised sub-domain adaptive electroencephalogram emotion recognition method according to claim 2, characterized in that: In step 2.2, the maximum mean difference MMD algorithm is used to calculate the marginal distribution difference, as shown in formula (2): 。 4. The progressive unsupervised sub-domain adaptive electroencephalogram emotion recognition method according to claim 3, characterized in that: In step 2.3, the weighted clustering (WCD) algorithm is used to calculate the conditional distribution difference, as shown in formula (3): wherein, represents the weighted contrast field difference, tr(A T S w A) represents the within-class scatter matrix; Wherein where y(·) represents the one-hot label of the source domain, and since the target domain is unlabeled, the pseudo label predicted by the network is used, and is denoted, and when c = c ′ , the feature distribution difference between the source domain and the target domain under the same category is calculated, and when c≠c ′ , the feature distribution difference between the source domain and the target domain under different categories is calculated; where the within-class scatter matrix Svv is defined as where d^ denotes the distance between a sample and its class mean. The within-class scatter matrix Svv is defined as Wherein, C represents the number of categories, respectively represent the number of samples of the i-th category in the source domain and the number of samples of the i-th category in the target domain, respectively represent the mean of the i-th category of training samples in the source domain and the mean of the i-th category of training samples in the target domain.

5. The progressive unsupervised sub-domain adaptive electroencephalogram emotion recognition method according to claim 4, characterized in that: In step 2.4, a dynamic factor α is used to convert the marginal distribution difference to the conditional distribution difference; wherein, denotes the edge distribution difference, denotes the conditional distribution difference.

6. The progressive unsupervised sub-domain adaptive electroencephalogram emotion recognition method according to claim 5, characterized in that: In step 3, the calculation in the domain adaptation of the source domain and the target domain is the same as step 2.

7. The progressive unsupervised sub-domain adaptive electroencephalogram emotion recognition method according to claim 6, characterized in that: The emotion recognition system in step 4 includes a real-time signal acquisition module, a real-time signal processing module, and an emotion recognition result display module; The real-time signal acquisition module acquires EEG signals through an EEG signal acquisition device, and then transmits the EEG signals to the real-time signal processing module in real time through network communication. The real-time signal processing module performs signal preprocessing, extracts differential entropy features, calculates the distribution difference between the source domain and the target domain, including the marginal distribution difference and the conditional distribution difference, adjusts the marginal distribution difference and the conditional distribution difference using a dynamic factor, finally classifies the extracted domain-invariant features and transmits the classification results to the emotion recognition result display module. After receiving the classification results output by the signal processing module, the emotion recognition result display module encodes the display instructions, so that the final classification results can be presented on the display.

Images