Enhanced cross-subject electroencephalogram signal emotion recognition method based on hybrid CNN-transformer architecture

Abstract

The invention relates to the field of brain-computer interface emotion recognition, in particular to an enhanced cross-subject electroencephalogram signal emotion recognition method based on a hybrid CNN-transformer architecture, which comprises the following steps: acquiring electroencephalogram data, and processing the electroencephalogram data by adopting a pre-trained cross-subject electroencephalogram signal emotion recognition model to obtain an emotion recognition result; the cross-subject electroencephalogram signal emotion recognition model comprises a convolution layer, a Transform layer, a cross-subject domain adaptation module, a full connection layer and a SoftMax layer. An adversarial training mechanism is constructed by using a gradient inversion layer and a domain classifier, feature distribution alignment of an EEG signal source domain and a target domain is realized, the model learns domain invariant feature representation between electroencephalogram signals of different testees through a multi-constraint mechanism, and the problem of weak generalization ability of an emotion recognition model is solved.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface emotion recognition, and more specifically to a method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture. Background Technology

[0002] Electroencephalography (EEG) is a non-invasive, low-cost alternative with superior temporal resolution, making it particularly ideal for emotion recognition, although its spatial resolution is not as good as CT and MRI. However, the challenge of EEG emotion recognition lies in how to effectively extract and interpret complex temporal and spatial patterns, which are key manifestations of different emotional states.

[0003] The problem of weak generalization ability of emotion recognition models caused by large differences in EEG signals across subjects in existing technologies urgently needs to be solved. Summary of the Invention

[0004] In view of this, this application discloses an enhanced cross-subject EEG signal emotion recognition method based on a hybrid CNN-transformer architecture to solve the above problems; it includes: acquiring EEG data, processing the EEG data using a pre-trained cross-subject EEG signal emotion recognition model, and obtaining emotion recognition results. The cross-subject EEG signal emotion recognition model includes: convolutional layers, Transformer layers, a cross-subject domain adaptation module, fully connected layers, and SoftMax layers; the convolutional layers and Transformer layers constitute a feature extractor to extract high-dimensional features from the EEG data; the cross-subject domain adaptation module extracts domain-invariant features from the high-dimensional features; and the fully connected layers and SoftMax layers predict the emotion category based on the domain-invariant features, outputting the emotion recognition results.

[0005] The cross-subject domain adaptation module includes: a gradient inversion layer, a domain classifier, a class center alignment module, and a frequency band consistency constraint module;

[0006] The gradient inversion layer, during forward propagation, passes the high-dimensional feature F output by the feature extractor to the domain classifier; during backward propagation, it multiplies the gradient from the domain classifier by a negative constant and then passes it back to the feature extractor.

[0007] The domain classifier is a multilayer perceptron that receives the high-dimensional feature F output by the feature extractor and determines whether the feature comes from the source domain or the target domain.

[0008] The class center alignment module applies emotion category consistency constraints to domain-invariant features by minimizing the feature center distance corresponding to the same emotion category in the source and target domains, thereby reducing inter-domain differences.

[0009] The frequency band consistency constraint module ensures that the representation of the domain-invariant features extracted by the cross-subject domain adaptation module in a specific frequency domain does not produce significant differences among different subjects by measuring the differences between the corresponding frequency band feature representations in the source domain and the target domain.

[0010] This application utilizes a gradient inversion layer and a domain classifier to construct an adversarial training mechanism, thereby aligning the feature distributions of the source and target domains of EEG signals. Simultaneously, it applies emotion category center alignment constraints and five-band feature consistency constraints. Through multiple constraint mechanisms, the model learns domain-invariant feature representations between EEG signals from different subjects, addressing the problem of weak generalization ability of emotion recognition models caused by large cross-subject individual differences in EEG signals. This improves the generalization ability and robustness of emotion recognition models in cross-individual contexts. Attached Figure Description

[0011] Figure 1 This is a schematic diagram illustrating the pre-training steps in an embodiment of this application;

[0012] Figure 2 This is a schematic diagram of the structure of the cross-subject EEG signal emotion recognition model in the embodiments of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, features, and advantages of this application clearer and to enable those skilled in the art to better understand the technical solutions of this application, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments.

[0014] Example 1:

[0015] This embodiment includes a method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture. The method includes: acquiring EEG data; processing the EEG data using a pre-trained cross-subject EEG signal emotion recognition model to obtain emotion recognition results. The cross-subject EEG signal emotion recognition model includes: convolutional layers, Transformer layers, a cross-subject domain adaptation module, fully connected layers, and SoftMax layers. The convolutional layers and Transformer layers constitute a feature extractor, extracting high-dimensional features from the EEG data; the cross-subject domain adaptation module extracts domain-invariant features from the high-dimensional features; and the fully connected layers and SoftMax layers predict the emotion category based on the domain-invariant features, outputting the emotion recognition results.

[0016] The pre-training, such as Figure 1 As shown, it includes:

[0017] S1. Obtain the EEG dataset for training and preprocess the EEG dataset.

[0018] Specifically, this embodiment selects the SEED dataset. Data was collected three times from each subject, and the data collected from the first collection of 15 subjects was used as the experimental data. The EEG signal was downsampled to 200Hz to obtain downsampled data H1. H1 was filtered using a bandpass filter of 0.3-50Hz to remove high-frequency noise and DC drift, resulting in filtered data H2. Independent component analysis was used to remove EEG artifacts from H2, resulting in EEG artifact-free data H3. H3 was then z-score normalized and feature smoothed to obtain feature-smoothed data H4. H4 was segmented using a non-overlapping sliding window to obtain segmented data H5. Data in five frequency bands (δ, θ, α, β, and γ) were extracted from H5 to obtain data H6. Differential entropy features were extracted from H6 to obtain differential entropy feature data H7. The formula for calculating the DE feature is as follows:

[0019]

[0020] in, Indicates brain electrical signals, Let x represent the standard deviation, μ represent the mean of x, and e be Euler's constant.

[0021] S2. Convolutional layers are used to capture local features of the preprocessed EEG data to obtain high-dimensional feature maps.

[0022] Specifically, high-dimensional feature maps It is obtained through two-dimensional convolution, and the formula is:

[0023]

[0024] in, Represents a high-dimensional feature map. This represents the input to the convolutional layer, i.e., the unified time-frequency map (or feature matrix) constructed after preprocessing and feature extraction, at position... Input value at the location, and Indicates the output feature map Spatial coordinates (rows and columns) Indicates size is Convolutional filters, This indicates the bias term.

[0025] Furthermore, Before the output, the image passes through a ReLU activation function and a max-pooling layer. The ReLU activation function introduces non-linear features, ensuring that only positive activation values ​​are passed to the next layer. The max-pooling layer reduces the spatial dimension of the feature map, as shown in the formula:

[0026]

[0027] in, This represents a pooled window, which in this embodiment is 2×2 in size.

[0028] S3. Use the Transformer layer to capture the global dependencies between different regions of the high-dimensional feature map and output the high-dimensional feature F.

[0029] Specifically, the self-attention mechanism in each Transformer layer calculates the attention score between positions i and j in the feature map, enabling the model to focus on relevant and important features throughout the time-frequency map. The formula is as follows:

[0030]

[0031] Where Q, K, and V are the query, key, and value matrices obtained from the input feature map, respectively, and d k This represents the dimension of the key vector.

[0032] S4. Employ the Cross-Subject Domain Adaptation Module (DAAM) to extract high-dimensional features from the outputs of convolutional and Transformer layers to obtain domain-invariant features.

[0033] The cross-subject domain adaptation module includes: a gradient inversion layer, a domain classifier, a class center alignment module, and a frequency band consistency constraint module.

[0034] The Gradient Reversal Layer (GRL), a key component for achieving adversarial domain adaptation, acts as an identity function during forward propagation, passing the high-dimensional features F output by the feature extractor (CNN and Transformer) intact to the domain classifier. During backpropagation, the gradient from the domain classifier is multiplied by a negative constant. The data is then fed back to the feature extractor. The domain classifier itself calculates the domain classification loss. And optimize based on this loss. During the backpropagation phase of the entire model training, the domain classifier Will calculate loss Relative to its input (i.e. features) gradient of ) .gradient This reflects how features need to be adjusted to make the domain classifier's judgment more accurate. The role of GRL is to receive gradients. Multiply it by the negative weight ,Right now The modified gradient is then passed to the feature extractor. Therefore, feature extractor In fact, it is along the way that makes The direction of increase (i.e., confusion) The parameters are updated in the direction of the feature, thereby achieving adversarial alignment.

[0035] This mechanism enables the feature extractor to optimize its own emotion classification loss while attempting to maximize the loss of the domain classifier, thereby forcing the feature extractor to learn features that can deceive the domain classifier, i.e., domain-invariant features.

[0036] Domain Classifier The model is a simple multilayer perceptron (MLP) that receives high-dimensional features F from the feature extractor and attempts to determine whether the features F come from the source domain (D=1, i.e., the training data) or the target domain (D=0, i.e., the test data). The division between the source and target domains is determined before model training begins, based on the specific experimental design of the cross-subject emotion recognition task (usually using the leave-one-subject-out method). The target domain is defined as the set of all subject data used to train the base emotion classifier. The domain is defined as the single set of subject data to be tested or adapted. During the training phase, all samples from the source domain are assigned the domain label D=1, and all samples from the target domain are assigned the domain label D=0. This pre-split is the basis for domain adaptation training, ensuring that the model can specifically learn to eliminate the feature distribution differences between the source and target domains; the training objective of the domain classifier is to minimize the domain classification loss. In this embodiment, the binary cross-entropy loss function is used for calculation, and the formula is as follows:

[0037]

[0038] Where N represents the total number of samples, Represents the true domain label of the sample. express Predicted domain probabilities. During adversarial training, Aiming to minimize feature extractor (Through GRL) the aim is to maximize Ultimately, this achieves adversarial alignment of feature distributions.

[0039] The gradient inversion layer and the domain classifier form an adversarial training mechanism. The goal is to enable the feature extractor to learn EEG signal features that cannot be distinguished by the domain classifier, thereby achieving adversarial alignment of the feature distributions of the source domain and the target domain and obtaining domain-invariant feature representations.

[0040] The class center alignment module imposes emotion category consistency constraints on domain-invariant features by minimizing the feature center distance corresponding to the same emotion category in the source and target domains. This reduces inter-domain differences and ensures that, in the domain-invariant feature space, the same emotion categories from different subjects (domains) remain closely clustered together. Specifically, this is achieved by calculating the source domain... and target domain Feature centers of each emotion category c and And minimize the distance between these corresponding centers to achieve the emotion category consistency constraint. Class center alignment loss It is usually represented by the sum of squares of the center distances of all categories:

[0041]

[0042] Where K represents the total number of emotion categories, by minimizing DAAM ensures that features retain sufficient emotion discrimination ability even after eliminating individual differences.

[0043] The frequency band consistency constraint module measures the frequency band consistency in the source and target domains. The differences between the feature representations of the five frequency bands are assessed to ensure that the domain-invariant features extracted by the cross-subject domain adaptation module do not produce significant differences in their representations in specific frequency domains among different subjects. Specifically, this is measured by calculating the distance between the mean vectors of features in specific frequency bands across two domains, using the square of the L2 norm as the loss function. :

[0044]

[0045] in, Represents the source domain The feature set of mid-frequency band b Represents the target domain The feature set of mid-frequency band b. By minimizing... This forced different subjects to The statistical properties (mean) of the features across the five frequency bands remain consistent in the feature space, thus ensuring the robustness of the frequency domain features. Since adversarial training may unintentionally disrupt the key frequency domain feature structure required for emotion recognition when removing individual information from EEG signals, a frequency band consistency loss is designed. By measuring the differences between the corresponding frequency band feature representations in the source and target domains, it ensures that the domain-invariant features extracted by DAAM do not produce significant differences in their representations in these specific frequency domains among different subjects. This imposes a physiologically soft constraint on the domain-invariant features, further enhancing their robustness and interpretability.

[0046] S5. Flatten the domain-invariant features and input them into the fully connected layer and the SoftMax layer to complete the sentiment category prediction;

[0047] The output calculation formula for a fully connected layer is:

[0048]

[0049] in, Representation domain invariant features, W l b represents the weights of the fully connected layer. l This indicates the bias term.

[0050] The formula for predicting sentiment categories is:

[0051]

[0052] in, This indicates the prediction results for the emotion category. Representing each emotion category The score, Indicates emotion category The score, This represents the total number of categories. The predicted sentiment category is the one with the highest probability.

[0053] S6. Calculate the total loss of the model and update the model parameters. When the preset number of training rounds or early stopping conditions are reached, the model training is completed.

[0054] Specifically, the overall loss function It is emotion classification loss Domain classification loss Class center alignment loss and frequency band consistency loss Weighted sum:

[0055]

[0056] in, , , The weighting coefficients represent the loss. We use cross-entropy loss to classify sentiment in labeled source domain data:

[0057]

[0058] Where n represents the number of categories, It is an indicator variable for the i-th category of the true label (usually 0 or 1). It is the probability of the i-th category predicted by the model.

[0059] Example 2:

[0060] This embodiment includes a method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture. The method includes: acquiring EEG data; processing the EEG data using a pre-trained cross-subject EEG signal emotion recognition model to obtain emotion recognition results. The cross-subject EEG signal emotion recognition model includes: convolutional layers, Transformer layers, a cross-subject domain adaptation module, fully connected layers, and SoftMax layers. The convolutional layers and Transformer layers constitute a feature extractor, extracting high-dimensional features from the EEG data; the cross-subject domain adaptation module extracts domain-invariant features from the high-dimensional features; and the fully connected layers and SoftMax layers predict the emotion category based on the domain-invariant features, outputting the emotion recognition results.

[0061] Specifically, in this embodiment, the structural diagram of the cross-subject EEG signal emotion recognition model is as follows: Figure 2 As shown, the cross-subject EEG signal emotion recognition model mainly consists of three key modules: a convolutional neural network (CNN) module, used to extract local spatial features from EEG signals; a Transformer module, which models the global temporal dependencies of signals through a self-attention mechanism; and a cross-subject domain adaptation module, which solves the problem of cross-subject variability of EEG signals through multiple mechanisms such as adversarial learning, class center alignment, and frequency band consistency constraints, significantly enhancing the model's cross-individual generalization ability and robustness in multi-subject, multi-scenario scenarios.

[0062] Finally, it should be noted that the above description only depicts some embodiments of this application. For those skilled in the art, various changes, modifications, substitutions, and variations can be conceived of these embodiments without departing from the principles and spirit of this application. The scope of protection of this application is defined by the appended claims and their equivalents, and all the above-mentioned behaviors should be covered within the scope of protection of this application.

Claims

1. A method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture, characterized in that, include: EEG data was acquired, and a pre-trained cross-subject EEG signal emotion recognition model was used to process the EEG data to obtain emotion recognition results. The cross-subject EEG signal emotion recognition model includes: a convolutional layer, a Transformer layer, a cross-subject domain adaptation module, a fully connected layer, and a SoftMax layer. The convolutional layer and the Transformer layer constitute a feature extractor to extract high-dimensional features from the EEG data. The cross-subject domain adaptation module extracts domain-invariant features from the high-dimensional features. The fully connected layer and the SoftMax layer predict the emotion category based on the domain-invariant features and output the emotion recognition result.

2. The method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture according to claim 1, characterized in that, The cross-subject domain adaptation module includes: a gradient inversion layer, a domain classifier, a class center alignment module, and a frequency band consistency constraint module; The gradient inversion layer, during forward propagation, passes the high-dimensional feature F output by the feature extractor to the domain classifier; during backward propagation, it multiplies the gradient from the domain classifier by a negative constant and then passes it back to the feature extractor. The domain classifier is a multilayer perceptron that receives the high-dimensional feature F output by the feature extractor and determines whether the feature comes from the source domain or the target domain. The class center alignment module applies emotion category consistency constraints to domain-invariant features by minimizing the feature center distance corresponding to the same emotion category in the source and target domains, thereby reducing inter-domain differences. The frequency band consistency constraint module ensures that the representation of the domain-invariant features extracted by the cross-subject domain adaptation module in a specific frequency domain does not produce significant differences among different subjects by measuring the differences between the corresponding frequency band feature representations in the source domain and the target domain.

3. The method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture according to claim 2, characterized in that, The emotional category consistency constraint is applied to the domain-invariant features by calculating the source domain. and target domain Feature centers of each emotion category c and And minimize the distance between these corresponding centers.

4. The method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture according to claim 2, characterized in that, The difference between the corresponding frequency band feature representations in the source and target domains is measured by calculating the distance between the mean vectors of specific frequency band features in the two domains, using the square of the L2 norm as the loss function. The formula is: in, Represents the source domain The feature set of mid-frequency band b Represents the target domain The feature set of mid-frequency band b.

5. The method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture according to claim 1, characterized in that, The convolutional layer and the Transformer layer together constitute a feature extractor. The operations performed by the convolutional layer include: in, Represents a high-dimensional feature map. Indicates the location of the convolutional layer Input value at the location, and Indicates the output feature map spatial coordinates, Indicates size is Convolutional filters, This indicates the bias term.

6. The method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture according to claim 1, characterized in that, The pre-training includes: S1. Obtain the EEG dataset for training and preprocess the EEG dataset; S2. Convolutional layers are used to capture local features of the preprocessed EEG data to obtain high-dimensional feature maps; S3. Use Transformer layers to capture the global dependencies between different regions of the high-dimensional feature map and output high-dimensional features; S4. Employ a cross-subject domain adaptation module to extract high-dimensional features from the outputs of convolutional and Transformer layers to obtain domain-invariant features. S5. Flatten the domain-invariant features and input them into the fully connected layer and the SoftMax layer to complete the sentiment category prediction; S6. Calculate the total loss of the model and update the model parameters. When the preset number of training rounds or early stopping conditions are reached, the model training is completed.

7. The method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture according to claim 6, characterized in that, The emotion category prediction is completed, and the formula for emotion category prediction is: in, This indicates the prediction results for the emotion category. Representing each emotion category The score, Indicates emotion category The score, This represents the total number of categories, with the predicted sentiment category being the category with the highest probability.

8. The method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture according to claim 1, characterized in that, The total loss of the model is given by the following formula: in, Indicates the total loss. Indicates loss in emotion classification, Representation domain classification loss, Represents the class center alignment loss. Indicates frequency band consistency loss. , , The weighting coefficients represent the loss.

9. The method for enhanced cross-subject EEG signal emotion recognition based on a hybrid CNN-transformer architecture according to claim 8, characterized in that, The cross-entropy loss is used, and the formula is as follows: Where n represents the number of categories, The indicator variable representing the i-th category of the true label. This represents the probability of the i-th category predicted by the model.

Images