Cross-domain EEG signal emotion recognition method based on multi-level alignment

Abstract

The invention belongs to the field of brain-computer interface emotion recognition, and particularly relates to a cross-domain EEG (electroencephalogram) signal emotion recognition method based on multi-level alignment, which comprises the following steps: acquiring an electroencephalogram signal, and inputting the electroencephalogram signal into a trained emotion recognition model to obtain an emotion classification result; the training process of the emotion recognition model comprises the steps that electroencephalogram signals are obtained and preprocessed, and preprocessed electroencephalogram data are divided into source domain data and target domain data; performing feature extraction on the source domain data and the target domain data to obtain shallow layer features and deep layer features; updating model parameters according to the unconditional MMD loss of the shallow features and the conditional MMD loss of the deep features until a trained emotion recognition model is obtained; according to the method, the universal feature difference between the domains is effectively reduced through the unconditional MMD loss of the shallow-layer features, the consistency of different types of features is further enhanced through the conditional MMD loss of the deep-layer features, and fine feature alignment between the source domain and the target domain is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of brain-computer interface emotion recognition, and specifically relates to a cross-domain EEG signal emotion recognition method based on multi-level alignment. Background Art

[0002] With the rapid development of artificial intelligence and biomedical engineering, emotion recognition technology based on electroencephalogram (EEG) has shown great application potential in the fields of intelligent human-computer interaction, emotion monitoring, mental health assessment, driver fatigue detection, and neurorehabilitation. EEG is a non-invasive technology that records brain electrical activity by placing electrodes on the scalp, which can reflect the emotional state of an individual in real time. With the continuous advancement of technology, EEG emotion recognition systems will be widely used in smart homes, educational assistance, health care, entertainment interaction and other scenarios in the future, promoting the development of personalized and intelligent services.

[0003] In the prior art, emotion recognition is usually based on traditional machine learning methods such as support vector machines (SVM) and artificial neural networks (ANN). These methods can achieve certain results when processing specific data sets, but due to the high nonlinearity and individual differences of EEG signals, cross-individual emotion recognition faces huge challenges. There are significant differences in the spatial and temporal characteristics of EEG data between different subjects. Traditional methods are difficult to generalize between different subjects, and often require separate training models for each individual. This approach is not only time-consuming and labor-intensive, but also inefficient in practical applications and difficult to extend to real-time systems.

[0004] In order to solve the above problems, researchers have introduced deep learning and domain adaptation technologies in recent years to improve the cross-domain generalization ability of EEG emotion recognition. Deep learning models can automatically extract high-dimensional feature representations and capture the complex temporal and spatial relationships in EEG signals, while domain adaptation technology helps the model transfer knowledge between different individuals by narrowing the feature distribution differences between the source domain and the target domain.

[0005] However, existing methods still have shortcomings in the refinement of feature extraction and alignment, especially in how to simultaneously capture local and global features and achieve feature alignment between source and target domains at multiple scales, which urgently requires innovation. Summary of the invention

[0006] In order to solve the problem of cross-domain data differences in EEG emotion recognition, the present invention proposes a cross-domain EEG signal emotion recognition method based on multi-level alignment, which can effectively extract multi-level time-space features and improve the performance of cross-individual emotion recognition through a hierarchical alignment strategy. The specific steps include: obtaining EEG signals, inputting the EEG signals into a trained emotion recognition model, and obtaining emotion classification results; the emotion recognition model includes: a shallow feature extraction module, a deep feature extraction module, a domain discriminator and a classifier;

[0007] The training process of the emotion recognition model includes:

[0008] S1, acquiring EEG signals and preprocessing them to obtain preprocessed EEG data, and dividing the preprocessed EEG data into source domain data and target domain data;

[0009] S2, respectively inputting the source domain data and the target domain data into the shallow feature extraction module for feature extraction to obtain shallow features of the source domain and the target domain;

[0010] S3, respectively inputting the shallow features of the source domain and the target domain into the deep feature extraction module for feature extraction to obtain the deep features of the source domain and the target domain;

[0011] S4, respectively inputting the deep features of the source domain and the target domain into the classifier to obtain the classification prediction results of the source domain and the target domain; respectively inputting the deep features of the source domain and the target domain into the domain discriminator to obtain the domain prediction results of the source domain and the target domain;

[0012] S5. Calculate the overall loss function value according to the shallow features, deep features, classification prediction results, and domain prediction results of the source domain and the target domain, and update the model parameters according to the overall loss function value until the preset training conditions are met to obtain a trained emotion recognition model.

[0013] The preprocessing of the EEG signal includes: downsampling the EEG signal to obtain a downsampled EEG signal E1; using a bandpass filter to filter out high-frequency noise of the downsampled EEG signal E1 to obtain an EEG signal E2; using an independent component analysis method to remove electrooculogram artifacts in the EEG signal E2 to obtain an EEG signal E3 without electrooculogram artifacts; using a moving average and a linear dynamic system to perform feature smoothing on the EEG signal E3 to obtain a smoothed EEG signal E4; using a sliding window to segment the smoothed EEG signal E4, extracting EEG data of five frequency bands of δ, θ, α, β and γ from the segmented signal, respectively, extracting differential entropy features from the EEG data of the five frequency bands, and using the extracted differential entropy features as the preprocessed EEG data.

[0014] The shallow feature extraction module is a parallel convolution layer, including a temporal convolution branch and a spatial convolution branch; the parallel multi-scale convolution layer processes the source domain data and the target domain data respectively, including: inputting the source domain data into the temporal convolution branch and the spatial convolution branch respectively to obtain the multi-scale temporal features and spatial features of the source domain, inputting the target domain data into the temporal convolution branch and the spatial convolution branch respectively to obtain the multi-scale temporal features and spatial features of the target domain, fusing the multi-scale temporal features and spatial features of the source domain data to obtain the shallow features of the source domain, and fusing the multi-scale temporal features and spatial features of the target domain data to obtain the shallow features of the target domain.

[0015] Calculating the overall loss function value includes: calculating the MMD loss of shallow features based on the shallow features of the source domain and the target domain The conditional MMD loss of deep features is calculated based on the deep features and classification prediction results of the source and target domains. Calculate the adversarial loss L of the domain discriminator based on the domain prediction results of the source domain and the target domain domain , calculate the classification loss L based on the classification prediction results of the source domain cls , L cls , L domain The weighted combination gives the overall loss function value.

[0016] The calculation formula of MMD loss of shallow features is:

[0017]

[0018] Among them, φ(·) is the kernel function, represents the i-th shallow feature of the source domain, represents the i-th data in the source domain, represents the jth shallow feature of the target domain, represents the jth data in the target domain, N s is the number of data in the source domain, N t is the number of data in the target domain, i and j are the indexes of the data in the source domain and target domain.

[0019] The classification prediction results of the source domain and the target domain include: the classification prediction label of each data; the calculation formula of the conditional MMD loss of deep features is:

[0020]

[0021] in, represents the data set with classification prediction label c in the source domain, where C is the number of category labels. represents the number of data with classification prediction label c in the source domain, represents the i-th data in the source domain, υ(·) is the kernel function, represents the i-th deep feature of the source domain, represents the number of data with classification prediction label c in the target domain, represents the jth data in the target domain, represents the data set with classification prediction label c in the target domain, Represents the j-th deep feature of the target domain.

[0022] The calculation formula of the adversarial loss of the domain discriminator includes:

[0023]

[0024] in, is the domain prediction result of the i-th data in the source domain, represents the domain prediction result of the jth data in the target domain, N s is the number of data in the source domain, N t is the number of data in the target domain, i and j are the indexes of the data in the source domain and target domain.

[0025] The classification prediction results of the source domain include: the probability distribution of each data in the source domain on C category labels; the classification loss includes:

[0026]

[0027] in, represents the true category label of the i-th data in the source domain, is the probability of the i-th data in the source domain on the category label c, and C is the number of category labels.

[0028] The domain discriminator consists of a fully connected layer, a SELU activation layer, and a Sigmoid layer.

[0029] The classifier consists of a fully connected layer, a SELU activation layer, and a Softmax layer.

[0030] Beneficial effects:

[0031] The present invention extracts shallow and deep features of EEG signals through parallel multi-scale convolution and multi-head attention layers, effectively reduces the common feature differences between domains through unconditional MMD alignment of shallow features, further enhances the feature consistency of different categories through conditional MMD and conditional adversarial alignment of deep features, and realizes fine feature alignment between source domain and target domain, so that the present invention has good cross-domain generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a training flow chart of the EEG signal emotion recognition model of the present invention;

[0033] Figure 2 It is a structural schematic diagram of the EEG signal emotion recognition model of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] like Figure 1 , Figure 2 As shown, the present invention adopts a cross-domain EEG signal emotion recognition method based on multi-level alignment, including: obtaining EEG signal data, inputting the EEG signal data into a trained emotion recognition model, and obtaining an emotion classification result; the emotion recognition model includes: a shallow feature extraction module, a deep feature extraction module, a domain discriminator and a classifier;

[0036] The training process of the emotion recognition model includes:

[0037] S1, acquiring EEG signal data and preprocessing it to obtain preprocessed EEG data, and dividing the preprocessed EEG data into source domain data and target domain data;

[0038] The present invention uses the SEED public data set as experimental data. The SEED data set contains EEG data of 15 people. Each person is collected three times, and each time he watches 15 videos (5 positive, neutral and negative). The data collected for the first time by the 15 subjects is used as experimental data.

[0039] Preprocessing of EEG signals includes:

[0040] The EEG signal is downsampled to 200Hz to obtain the downsampled EEG signal E1; a 0.3-50Hz bandpass filter is used to filter out the high-frequency noise of the downsampled EEG signal E1 to obtain the EEG signal E2; the independent component analysis method (ICA) is used to remove the electrooculogram artifacts in the EEG signal E2 to obtain the EEG signal E3 without the electrooculogram artifacts; the moving average and linear dynamic system (LDS) are used to perform feature smoothing on the EEG signal E3 to obtain the smoothed EEG signal E4; the smoothed EEG signal E4 is segmented using a non-overlapping 1-second sliding window, and the EEG data of five frequency bands of δ (1-3HZ), θ (4-7HZ), α (8-13HZ), β (14-30HZ) and γ (31-50HZ) are extracted from the segmented segments; the differential entropy (DE) feature is extracted from the EEG data of the five frequency bands to obtain the feature representation, which is used as the input of the model. The calculation formula of the DE feature is as follows:

[0041]

[0042] Where x represents the EEG signal, σ represents the standard deviation of x, μ represents the mean of x, and e is the Euler constant.

[0043] All the subjects were divided according to the leave-one-out method, and the preprocessed EEG data were cross-validated to obtain training set, validation set, and test set; after each division, the same number of data fragments were randomly selected from the three emotions in the preprocessed EEG data, and the data fragments were divided into source domain data and target domain data.

[0044] S2, respectively inputting the source domain data and the target domain data into the shallow feature extraction module for feature extraction to obtain shallow features of the source domain and the target domain;

[0045] In one embodiment, the shallow feature extraction module is a parallel multi-scale convolution layer, including a temporal convolution branch and a spatial convolution branch, which respectively extract temporal and spatial features of different scales. Through this parallel architecture, the temporal and spatial branches can synchronously retain the temporal-spatial structural information, thereby facilitating more efficient feature transfer and alignment between the source domain and the target domain. Specifically, the source domain data is input into the temporal convolution branch and the spatial convolution branch respectively to obtain the multi-scale temporal features and spatial features of the source domain, the target domain data is input into the temporal convolution branch and the spatial convolution branch respectively to obtain the multi-scale temporal features and spatial features of the target domain, the multi-scale temporal features and spatial features of the source domain data are fused to obtain the shallow features of the source domain; the multi-scale temporal features and spatial features of the target domain data are fused to obtain the shallow features of the target domain.

[0046] The convolution kernel size of the temporal convolution branch is (k×1×n), where k represents the number of feature maps and n represents the length of the convolution kernel in the time dimension. This branch extracts time-dependent features at different time scales by sliding the convolution kernel in the time dimension.

[0047] The size of the spatial convolution kernel of the spatial convolution branch is (k×m×1), where m is equal to the number of EEG channels; this branch applies spatial convolution simultaneously at each time step to capture spatial dependencies across channels.

[0048] The fusion process includes: the outputs of the temporal and spatial convolution branches are superimposed and merged after passing through an additional convolution layer to adjust the shape of the feature map; the convolution layer is connected to batch normalization and ELU activation function to generate a fused temporal-spatial feature representation F. shallow , that is, shallow features.

[0049] S3, respectively inputting the shallow features of the source domain and the target domain into the deep feature extraction module for feature extraction to obtain the deep features of the source domain and the target domain;

[0050] In one embodiment, the deep feature extraction module is a multi-head attention layer; shallow By performing multi-head attention calculation, the global dependency of EEG signals can be modeled. The specific calculation process of the multi-head attention mechanism is as follows:

[0051] Calculate the query (Q), key (K), and value (V) matrices:

[0052] Q=F shallow W Q ,K=F shallow W K ,V=F shallow W V ,

[0053] Among them, W Q , W K and W V is a learnable linear transformation matrix.

[0054] Compute the scaled dot product of the query and the key to get the attention weights:

[0055]

[0056] Among them, d k is the dimension of the key vector, the Softmax function converts the calculated similarity into weights, and Attention represents the calculation of attention weights. This step ensures that the model can focus on the relationship between important time steps and channels.

[0057] Calculate the attention weight of each head Head i :

[0058] Head i =Attention(Q i ,K i ,V i )

[0059] The output of the multi-head attention layer is:

[0060] MHA(Q,K,V)=Concat(Head1,…,Head H )W O

[0061] Among them, W O is the linear projection matrix, H is the number of heads, i is the index of the head, and Concat is concatenation.

[0062] The output of the multi-head attention layer is the deep feature F deep , for conditional MMD and conditional adversarial alignment.

[0063] S4, respectively inputting the deep features of the source domain and the target domain into the classifier to obtain the classification prediction results of the source domain and the target domain; respectively inputting the deep features of the source domain and the target domain into the domain discriminator to obtain the domain prediction results of the source domain and the target domain;

[0064] The classifier includes two fully connected layers, a SELU activation layer, and a Softmax layer, which are used to map source domain features to specific emotion categories. Specifically, the classifier outputs the probability distribution of source domain and target domain data on C category labels, and takes the label corresponding to the largest probability in the probability distribution as the classification prediction label of the data.

[0065] The domain labels of the source domain and target samples are set to 0 and 1 respectively, so the domain discriminator performs a binary classification task; the discriminator includes two fully connected layers, a SELU activation layer, and a sigmoid layer to generate domain classification probabilities. The label corresponding to the largest domain classification probability is the predicted domain label.

[0066] S5. Calculate the overall loss function value according to the shallow features, deep features, classification prediction results, and domain prediction results of the source domain and the target domain, and update the model parameters based on the overall loss function value until the preset training conditions are met to obtain a trained emotion recognition model.

[0067] The overall loss function value L all for:

[0068]

[0069] in, and λ domain is the weight coefficient, l clsis the classification loss, is the MMD loss of shallow features, is the conditional MMD loss of deep features, L domain is the adversarial loss for the domain discriminator.

[0070] The maximum mean difference (MMD) loss of shallow features is used to perform unconditional MMD alignment on shallow features, making the common features of the source domain and the target domain more consistent and reducing the overall distribution difference between the source domain and the target domain. The MMD loss calculation formula of shallow features is:

[0071]

[0072] Among them, φ(·) is the kernel function, represents the i-th shallow feature of the source domain, represents the i-th data in the source domain, represents the jth shallow feature of the target domain, represents the jth data in the target domain, N s is the number of data in the source domain, N t is the number of data in the target domain, i and j are the indexes of the data in the source domain and target domain.

[0073] In one embodiment, φ(·) is a Gaussian kernel function, which is specifically expressed in the above formula as:

[0074]

[0075] Among them, σ is the bandwidth parameter of the Gaussian kernel, which controls the scale of the high-dimensional feature space.

[0076] The conditional MMD loss of deep features is used to deep Perform category-conditional alignment so that the source domain and the target domain have consistent feature distribution under class conditions. The calculation formula of the conditional MMD loss of deep features is:

[0077]

[0078] in, represents the data set with classification prediction label c in the source domain, where C is the number of category labels. represents the number of data with classification prediction label c in the source domain, represents the i-th deep feature of the source domain, represents the number of data with classification prediction label c in the target domain, represents the data set with classification prediction label c in the target domain, represents the j-th deep feature of the target domain.

[0079] The EEG feature extractor and domain discriminator are further aligned with the deep feature distribution through adversarial training. Adversarial training is a minimax game that aims to train the feature extractor to confuse the domain discriminator so that it cannot identify the source of the learned features. To achieve adversarial training, the domain discriminator is placed after the feature extractor, such as Figure 2 As shown in Figure 2. During the training process, the feature extractor learns domain-invariant feature representations for the two domains, while the domain discriminator identifies the source of the learned features, i.e., the source domain or the target domain. In this way, the distribution difference of deep features is further reduced, and the learned features can be generalized and remain unchanged between the two domains. The adversarial loss is:

[0080]

[0081] Where D(·) is the output of the domain discriminator.

[0082] Classification loss: The classifier is trained through the emotion label supervision of the source domain, and the classification loss is calculated to ensure the accuracy of emotion recognition. The classification loss is defined as:

[0083]

[0084] in, represents the true category label of the i-th data in the source domain, is the probability of the i-th data in the source domain on the category label c.

[0085] The model is back-propagated and parameter updated through the overall loss function, and the feature extractor, classifier, and alignment module are continuously optimized until the preset training conditions are met. The preset training conditions are that the overall loss function value is minimized or the maximum number of training times preset for the model is reached.

[0086] The comparison results between the present invention and other models are shown in Table 1. The results obtained by the present invention are better than the existing methods. Other models include: Deep domain confusion: maximizing for domain invariance (DDC), Deep coral: correlation alignment for deepdomain adaptation (DCORAL), and multi-source marginal distribution adaptation (MS-MDA).

[0087] Table 1 Comparison results of the accuracy of the present invention and other models

[0088]

[0089] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-domain EEG signal emotion recognition method based on multi-level alignment, characterized in that: include: Obtain EEG signals, input the EEG signals into the trained emotion recognition model, and obtain emotion classification results; The emotion recognition model includes: shallow feature extraction module, deep feature extraction module, domain discriminator and classifier; The training process of the emotion recognition model includes: S1, acquiring EEG signals and preprocessing them to obtain preprocessed EEG data, and dividing the preprocessed EEG data into source domain data and target domain data; S2, respectively inputting the source domain data and the target domain data into the shallow feature extraction module for feature extraction to obtain shallow features of the source domain and the target domain; S3, respectively inputting the shallow features of the source domain and the target domain into the deep feature extraction module for feature extraction to obtain the deep features of the source domain and the target domain; S4, respectively inputting the deep features of the source domain and the target domain into the classifier to obtain the classification prediction results of the source domain and the target domain; respectively inputting the deep features of the source domain and the target domain into the domain discriminator to obtain the domain prediction results of the source domain and the target domain; S5. Calculate the overall loss function value according to the shallow features, deep features, classification prediction results, and domain prediction results of the source domain and the target domain, and update the model parameters according to the overall loss function value until the preset training conditions are met to obtain a trained emotion recognition model.

2. A cross-domain EEG signal emotion recognition method based on multi-level alignment according to claim 1, characterized in that: The preprocessing of the EEG signal includes: downsampling the EEG signal to obtain a downsampled EEG signal E1; using a bandpass filter to filter out high-frequency noise of the downsampled EEG signal E1 to obtain an EEG signal E2; using an independent component analysis method to remove electrooculogram artifacts in the EEG signal E2 to obtain an EEG signal E3 without electrooculogram artifacts; using a moving average and a linear dynamic system to perform feature smoothing on the EEG signal E3 to obtain a smoothed EEG signal E4; using a sliding window to segment the smoothed EEG signal E4, extracting EEG data of five frequency bands of δ, θ, α, β and γ from the segmented signal, respectively, extracting differential entropy features from the EEG data of the five frequency bands, and using the extracted differential entropy features as the preprocessed EEG data.

3. The method for cross-domain EEG signal emotion recognition based on multi-level alignment according to claim 1, characterized in that: The shallow feature extraction module is a parallel convolution layer, including a temporal convolution branch and a spatial convolution branch; The parallel multi-scale convolution layer processes the source domain data and the target domain data respectively, including: inputting the source domain data into the time convolution branch and the space convolution branch respectively to obtain the multi-scale time features and space features of the source domain, inputting the target domain data into the time convolution branch and the space convolution branch respectively to obtain the multi-scale time features and space features of the target domain, fusing the multi-scale time features and space features of the source domain data to obtain the shallow features of the source domain, and fusing the multi-scale time features and space features of the target domain data to obtain the shallow features of the target domain.

4. The method for cross-domain EEG signal emotion recognition based on multi-level alignment according to claim 1, characterized in that: Calculating the overall loss function value includes: calculating the MMD loss of shallow features based on the shallow features of the source domain and the target domain The conditional MMD loss of deep features is calculated based on the deep features and classification prediction results of the source and target domains. Calculate the adversarial loss L of the domain discriminator based on the domain prediction results of the source domain and the target domain domain , calculate the classification loss L based on the classification prediction results of the source domain cls , L cls , L domain The weighted combination is used to obtain the overall loss function value; among them, MMD is the maximum mean difference.

5. The method for cross-domain EEG signal emotion recognition based on multi-level alignment according to claim 4, characterized in that: The calculation formula of MMD loss of shallow features is: Among them, φ(·) is the kernel function, represents the i-th shallow feature of the source domain, represents the i-th data in the source domain, represents the jth shallow feature of the target domain, represents the jth data in the target domain, N s is the number of data in the source domain, N t is the number of data in the target domain, i and j are the indexes of the data in the source domain and target domain.

6. A cross-domain EEG signal emotion recognition method based on multi-level alignment according to claim 4, characterized in that: The classification prediction results of the source domain and the target domain include: the classification prediction label of each data; the calculation formula of the conditional MMD loss of deep features is: in, represents the data set with classification prediction label c in the source domain, where C is the number of category labels. represents the number of data with classification prediction label c in the source domain, represents the i-th data in the source domain, φ(·) is the kernel function, represents the i-th deep feature of the source domain, represents the number of data with classification prediction label c in the target domain, represents the jth data in the target domain, represents the data set with classification prediction label c in the target domain, represents the j-th deep feature of the target domain.

7. A cross-domain EEG signal emotion recognition method based on multi-level alignment according to claim 4, characterized in that: The calculation formula of the adversarial loss of the domain discriminator includes: in, is the domain prediction result of the i-th data in the source domain, represents the domain prediction result of the jth data in the target domain, N s is the number of data in the source domain, N t is the number of data in the target domain, i and j are the indexes of the data in the source domain and target domain.

8. The method for cross-domain EEG signal emotion recognition based on multi-level alignment according to claim 4, characterized in that: The classification prediction results of the source domain include: the probability distribution of each data in the source domain on C category labels; the classification loss includes: in, represents the true category label of the i-th data in the source domain, is the probability of the i-th data in the source domain on the category label c, and C is the number of category labels.

9. The method for cross-domain EEG signal emotion recognition based on multi-level alignment according to claim 1, characterized in that: The domain discriminator consists of a fully connected layer, a SELU activation layer, and a Sigmoid layer.

10. The method for cross-domain EEG signal emotion recognition based on multi-level alignment according to claim 1, characterized in that: The classifier consists of a fully connected layer, a SELU activation layer, and a Softmax layer.

Images