Domain-adaptive cross-subject electroencephalogram signal emotion recognition method

Abstract

The invention relates to the technical field of electroencephalogram analysis, in particular to a domain-adaptive cross-subject electroencephalogram signal emotion recognition method, which comprises the following steps: acquiring an electroencephalogram signal, and preprocessing the electroencephalogram signal; inputting the preprocessed electroencephalogram signals into a trained electroencephalogram signal emotion recognition model to obtain an emotion classification result; the electroencephalogram emotion recognition model comprises a graph convolution feature extraction module, an attention module, a domain confrontation module and a classifier module. According to the method, the confrontation module formed by combining the gradient inversion layer and the domain discriminator is designed, and the change rule of the electroencephalogram of a person in positive, neutral and negative states is focused instead of the intensity of the reaction, so that the extracted emotional features have domain invariance, and the extraction accuracy is improved. And identification deviation caused by difference of different individuals in cross-subject emotion identification is eliminated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electroencephalogram analysis, in particular to a domain adaptive cross-subject electroencephalogram signal emotion recognition method. BACKGROUND

[0002] With the continuous development of intelligent human-computer interaction systems, medical health auxiliary systems and intelligent emotion monitoring devices, emotion recognition technology based on electroencephalogram (EEG) has gradually become a research hotspot. Compared with traditional emotion recognition methods such as voice and image, EEG is favored because it can directly reflect brain activity, is difficult to fake, and is highly objective.

[0003] Existing methods mostly use traditional classification models such as support vector machine (SVM) and artificial neural network (ANN) to model EEG. However, due to the large differences in neurophysiological structure and emotional response between individuals, there are innate differences in the cerebral cortex structure, brain region activation pattern and nerve conduction speed of different individuals, making the amplitude, frequency distribution and spatial topology features of EEG signals under the same emotion completely different. The emotional perception threshold and emotional expression intensity of individuals differ, resulting in traditional models often only achieving good performance on a single individual and being difficult to generalize to different individuals, i.e., showing a significant performance decline in cross-subject emotion recognition. At the same time, EEG signals are noisy, have high data dimensionality and complex spatial topology, further exacerbating the difficulty of model design.

[0004] Therefore, there is an urgent need for an EEG signal processing method that can adaptively recognize emotions for different individuals. SUMMARY

[0005] Therefore, the present application discloses a domain adaptive cross-subject electroencephalogram signal emotion recognition method to solve the above problems; comprising:

[0006] Obtaining EEG signals, preprocessing the EEG signals; inputting the preprocessed EEG signals into a trained EEG signal emotion recognition model to obtain an emotion classification result; the EEG signal emotion recognition model comprises a graph convolution feature extraction module, an attention module, a domain adversarial module and a classifier module;

[0007] Further, the training process of the EEG signal emotion recognition model comprises:

[0008] S1, obtaining EEG signals corresponding to the emotions of a user when watching different video clips, and preprocessing the EEG signals;

[0009] S2, dividing the preprocessed EEG data into source domain data and target domain data, and constructing a graph structure;

[0010] S3. Input the source domain map and the target domain map into the convolutional feature extraction module and the attention module to extract features, and obtain the source domain features and the target domain features.

[0011] S4. Input the source domain features and target domain features into the domain adversarial module and classifier to obtain the domain discrimination result and the sentiment classification result;

[0012] S5. Construct the overall loss function by combining the emotion classification loss and the domain discrimination loss, and optimize the model parameters by backpropagation. When the preset number of training iterations is reached, the model training is complete.

[0013] This application designs an adversarial module composed of a gradient inversion layer and a domain discriminator. It focuses on the changes in EEG patterns in positive, neutral, and negative states, rather than the intensity of the reaction. By introducing a domain adversarial training strategy, it models the distribution differences between the source and target domains and distinguishes samples from the source and target domains in a shared feature space. During training, the discriminator is optimized using a cross-entropy loss function, and adversarial learning of the feature extraction module is achieved by inverting the gradient direction. This ensures that the extracted emotional features have domain invariance and eliminates recognition bias caused by individual differences in cross-subject emotion recognition. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the training process of the EEG signal emotion recognition model in the embodiments of this application;

[0015] Figure 2 This is a schematic diagram of the EEG signal emotion recognition model in the embodiments of this application;

[0016] Figure 3 This is a comparison of the EEG signal emotion recognition model with other models in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, features, and advantages of this application clearer and to enable those skilled in the art to better understand the technical solutions of this application, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments.

[0018] Example 1:

[0019] This embodiment includes a domain-adaptive cross-subject EEG signal emotion recognition method, comprising:

[0020] EEG signals are acquired and preprocessed. The preprocessed EEG signals are then input into a trained EEG emotion recognition model to obtain emotion classification results. The EEG emotion recognition model is as follows: Figure 2As shown, it includes: a graph convolutional feature extraction module, an attention module, a domain adversarial module, and a classifier module. The graph convolutional feature extraction module and the attention module extract features from the preprocessed EEG signals to obtain source domain features and target domain features; the domain adversarial module and the classifier module distinguish between source domain and target domain samples in a shared feature space to obtain emotion classification results.

[0021] The preprocessing includes: downsampling the EEG signal to 200Hz to obtain downsampled data H1; filtering H1 using a bandpass filter of 0.3~0Hz to obtain filtered data H2; removing EEG artifacts from H2 using independent component analysis to obtain artifact-free data H3; smoothing H3 to obtain smoothed data H4; segmenting H4 using a non-overlapping sliding window to obtain segmented data H5; extracting data from H5 in five frequency bands: δ (1~3Hz), θ (4~7Hz), α (8~13Hz), β (14~30Hz), and γ (31~50Hz) to obtain data H6; and extracting differential entropy features from H6 to obtain differential entropy (DE) feature data H7. The formula for differential entropy is:

[0022]

[0023] Where x represents the electroencephalogram (EEG) signal, The standard deviation of x is represented by x. Let x represent the mean of x, and e represent Euler's constant.

[0024] The feature smoothing of H3 is performed using moving average and linear dynamic system (LDS).

[0025] The H4 segmentation method uses a non-overlapping sliding window. In this embodiment, the data is segmented using a non-overlapping 1s sliding window. A total of 15 video captures will be divided into 3394 segments, each containing information from 62 channels.

[0026] Furthermore, the training process of the aforementioned EEG signal emotion recognition model is as follows: Figure 1 As shown, it includes:

[0027] S1. Obtain the EEG signals corresponding to the user's emotions when watching different video clips, and preprocess the EEG signals.

[0028] In this embodiment, the EEG signals were obtained from the SEED dataset, which contains EEG data from 15 individuals. Each individual was collected three times, and each time they watched 15 videos (5 positive videos, 5 neutral videos, and 5 negative videos).

[0029] S2. Divide the preprocessed EEG data into source domain data and target domain data, and construct a graph structure.

[0030] The preprocessed EEG data is divided into source domain data and target domain data, using a Leave-One-Subject-Out (LOSO) strategy. In each round, one subject is assigned to the target domain, while the remaining subjects constitute the source domain, enabling cross-subject emotion recognition training. In each round, the source and target domain data are batch-split and loaded in parallel using independent data loaders. The sample size for each emotion category (positive, neutral, negative) is kept consistent to ensure a balanced category distribution.

[0031] S3. The source domain map and the target domain map are input to the convolutional feature extraction module and the attention module to extract features, so as to obtain source domain features and target domain features.

[0032] Specifically, the graph convolution feature extraction module consists of Chebyshev multinomial-based graph convolutional units (Chebynet), batch normalization layers, two-dimensional average pooling layers, and stacked graph convolution output fusion structures. It is used to extract node features from local graphs, global graphs, and fused graph structures. Figure 2 Local brain network graph branches, global brain network graph branches, and fusion common graph branches; batch normalization layers, two-dimensional average pooling layers, and stacked graph convolution output fusion structures are used for modeling and unified representation of multi-scale EEG features.

[0033] The attention module consists of an attention weight learning network composed of a two-layer fully connected structure and Tanh nonlinear activation, which is used to calculate the node-level attention distribution and channel importance score, respectively applied to the graph structure features and multi-band connection matrix, to achieve dynamic weighted fusion of global and local information.

[0034] S4. Input the source domain features and target domain features into the domain adversarial module and classifier to obtain the domain discrimination result and the sentiment classification result.

[0035] Due to individual differences, data measured under the same conditions can exhibit significant deviations. Many factors contribute to these individual differences; for example, the same funny video might elicit strong laughter from some individuals, while others might only offer a slight smile. This is reflected in EEG data, where the former's fluctuations will be much stronger than the latter's. Therefore, this application focuses on the patterns of EEG changes in positive, neutral, and negative states, rather than the intensity of the reaction. It models the distributional differences between the source and target domains by introducing a domain adversarial training strategy. To this end, this application designs an adversarial module consisting of a gradient reversal layer and a domain discriminator, used to distinguish between source and target domain samples in a shared feature space. During training, the discriminator is optimized using a cross-entropy loss function, and adversarial learning of the feature extraction module is achieved by reversing the gradient direction, thereby ensuring that the extracted emotional features possess domain invariance.

[0036] The domain adversarial module includes a gradient inversion layer and a domain discriminator composed of a fully connected layer, a BatchNorm layer, a ReLU activation function, and a Softmax layer. The domain adversarial module receives feature representations extracted by the graph convolutional feature extraction module and learns domain-invariant high-level sentiment features through adversarial training, thereby achieving alignment between the feature spaces of the source and target domains.

[0037] Specifically, domain discriminator It is a parameter The two-layer neural network is responsible for estimating the probability that the input features belong to the source or target domain. It receives the fused feature representation Z and outputs the features using the Softmax function. Belongs to the source domain (label 0) and features Probability estimate of belonging to the target domain (label 1):

[0038] Source domain probability estimation: ;

[0039] Target domain probability estimation: ;

[0040] The classifier includes a fully connected layer and a Softmax layer. The classifier processes source and target domain features: the source and target domain features are input into the fully connected layer for dimensionality reduction, and the dimensionality reduction result is input into the Softmax layer for normalization to obtain the probability value for each emotion. The final formula for obtaining the classification result of the target domain data is:

[0041]

[0042] in, This represents the sentiment prediction result for the target domain data. This represents the emotion-related features of the target domain data after being fused through a graph convolution feature extraction module and an attention mechanism. This represents the weight matrix of the first fully connected layer in the emotion classifier. This indicates a batch normalization operation. This represents the activation function. This represents the weight matrix of the output layer. This means normalizing the classification results into an emotion probability distribution and outputting the probability value corresponding to each emotion category.

[0043] S5. Construct an overall loss function by jointly using the emotion classification loss and the domain discrimination loss, and optimize the model parameters through backpropagation. The model training is complete when the preset number of training iterations is reached. In this embodiment, the cross-entropy loss is calculated based on the classification results and the actual label information. The formula for the overall loss function is:

[0044]

[0045] in, The cross-entropy loss function of the classifier is represented by... This represents the adversarial loss function of the domain adversarial module. Represented as the fusion graph branch loss function, and These represent the weighting coefficients.

[0046] The cross-entropy loss function of the classifier is:

[0047]

[0048] in, Represents cross-entropy loss, Represents classifier Treatment of Classification Signals The classification results Indicates category label.

[0049] The adversarial loss function for the domain adversarial module is:

[0050]

[0051] in, Indicates the source domain input sample. Indicates the target domain input sample. Indicates the parameters of the neighborhood discriminator. This represents the probability that the neighborhood discriminator classifies a sample from the source domain as belonging to the source domain. This represents the probability that the discriminator classifies a sample from the target domain as belonging to the target domain.

[0052] This application constructs a structure-preserving loss to obtain the fusion graph branch loss by L2 normalizing the local and global convolutional embeddings and calculating their mean squared error (MSE). The formula is as follows:

[0053]

[0054] in, and These represent local graph embeddings. and global graph embedding The similarity matrix after L2 normalization is calculated using the following formula:

[0055]

[0056] in, Represents a normalized matrix. The gain coefficient is used to minimize the structural differences between local and global features, promote the fusion of two types of graph convolutional features in the common embedding space, and thus extract more stable emotion recognition features with cross-subject generalization ability.

[0057] The comparison results with other models in this embodiment are as follows: Figure 3 As shown, the results obtained in this application are superior to those of existing methods. Figure 3 The comparative models include: traditional classifier Support Vector Machine (SVM), multi-layer domain adaptive method (DAN) based on maximum mean difference (MMD), domain adaptive method based on graph convolution (DGCNN), bidirectional adversarial network method (BiDANN), graph convolutional network method (PGCN), and residual graph neural network method (RGNN).

[0058] Example 2:

[0059] This embodiment includes a domain-adaptive cross-subject EEG signal emotion recognition method. The difference from Embodiment 1 is that the classifier comprises: a fully connected layer FC1 → batch normalization → Dropout → ReLU activation function → output layer FC2 → Softmax layer. Emotion-related features, after dimensionality reduction and nonlinear transformation, are input into the Softmax layer, which outputs the predicted probabilities of different emotion categories. The category with the highest probability is the final predicted label. During training, the classification loss for source domain samples uses the cross-entropy function, while target domain samples participate in adversarial training and feature alignment, but do not participate in emotion label supervision.

[0060] The data processing of the EEG signal emotion recognition model includes:

[0061] S1. Acquire preprocessed EEG data.

[0062] S2. Divide the preprocessed EEG data into source domain data and target domain data, and construct a graph structure.

[0063] Specifically, the participants were divided according to the leave-one-out cross-validation strategy. In each round, one participant was selected as the target domain and the remaining participants constituted the source domain. In each round of division, the source domain and target domain data were divided and loaded in parallel through independent data loaders. The number of samples for each emotion category was kept consistent.

[0064] S3. The source domain map and the target domain map are input to the convolutional feature extraction module and the attention module to extract features, so as to obtain source domain features and target domain features.

[0065] S4. Input the source domain features and target domain features into the domain adversarial module and classifier to obtain the domain discrimination result and the sentiment classification result.

[0066] Structured EEG features are extracted using a graph convolution feature extraction module and a multi-scale attention mechanism. Finally, it should be noted that the above description only depicts some embodiments of this application. Those skilled in the art will conceive of various changes, modifications, substitutions, and variations to these embodiments without departing from the principles and spirit of this application. The scope of protection of this application is defined by the appended claims and their equivalents, and all such actions should be covered within the scope of protection of this application.

Claims

1. A domain-adaptive cross-subject EEG signal emotion recognition method, characterized in that, include: Acquire EEG signals and preprocess them; The preprocessed EEG signals are input into a trained EEG signal emotion recognition model to obtain emotion classification results. The EEG signal emotion recognition model includes a graph convolution feature extraction module, an attention module, a domain adversarial module, and a classifier module.

2. The domain-adaptive cross-subject EEG signal emotion recognition method according to claim 1, characterized in that, The training process for the EEG signal emotion recognition model includes: S1. Obtain the EEG signals corresponding to the user's emotions when watching different video clips, and preprocess the EEG signals; S2. Divide the preprocessed EEG data into source domain data and target domain data, and construct a graph structure; S3. Input the source domain map and the target domain map into the convolutional feature extraction module and the attention module to extract features, and obtain the source domain features and the target domain features. S4. Input the source domain features and target domain features into the domain adversarial module and classifier to obtain the domain discrimination result and the sentiment classification result; S5. Construct the overall loss function by combining the emotion classification loss and the domain discrimination loss, and optimize the model parameters by backpropagation. When the preset number of training iterations is reached, the model training is complete.

3. The domain-adaptive cross-subject EEG signal emotion recognition method according to claim 2, characterized in that, The preprocessing includes: The EEG signal was downsampled to 200Hz to obtain the downsampled data H1. H1 is filtered using a bandpass filter with a frequency of 0.3-50Hz to obtain the filtered data H2. Independent component analysis was used to remove electrooculography artifacts from H2, resulting in data H3 after removing electrooculography artifacts. Perform feature smoothing on H3 to obtain feature-smoothed data H4; H4 is segmented using a non-overlapping sliding window to obtain segmented data H5; Data from five frequency bands—δ, θ, α, β, and γ—are extracted from H5 to obtain data H6; Differential entropy features are extracted from H6 to obtain differential entropy feature data H7.

4. The domain-adaptive cross-subject EEG signal emotion recognition method according to claim 3, characterized in that, The feature smoothing of H3 is performed using a moving average and a linear dynamic system.

5. The domain-adaptive cross-subject EEG signal emotion recognition method according to claim 2, characterized in that, The preprocessed EEG data is divided into source domain data and target domain data, and the division is carried out according to the leave-one-subject cross-validation strategy. In each round, one subject is selected as the target domain and the remaining subjects constitute the source domain. In each round of division, the source domain data and the target domain data are divided and loaded in parallel through independent data loaders. The number of samples for each emotion category is kept consistent.

6. The domain-adaptive cross-subject EEG signal emotion recognition method according to claim 2, characterized in that, The domain adversarial module includes a gradient inversion layer and a domain discriminator consisting of a fully connected layer, a BatchNorm layer, a ReLU activation function, and a Softmax layer. The domain adversarial module receives feature representations extracted by the graph convolution feature extraction module and learns domain-invariant high-level sentiment features through adversarial training.

7. The domain-adaptive cross-subject EEG signal emotion recognition method according to claim 2, characterized in that, The classifier includes a fully connected layer and a Softmax layer. The classifier processes source and target domain features: the source and target domain features are input into the fully connected layer for dimensionality reduction, and the dimensionality reduction result is input into the Softmax layer for normalization to obtain the probability value for each emotion. The final formula for obtaining the classification result of the target domain data is: in, This represents the sentiment prediction result of the target domain data. This represents the emotion-related features of the target domain data after being fused through a graph convolution feature extraction module and an attention mechanism. This represents the weight matrix of the first fully connected layer in the classifier. This indicates a batch normalization operation. This represents the activation function. This represents the weight matrix of the output layer. This means normalizing the classification results into an emotion probability distribution and outputting the probability value corresponding to each emotion category.

8. The domain-adaptive cross-subject EEG signal emotion recognition method according to claim 2, characterized in that, The overall loss function is constructed by jointly applying the emotion classification loss and the domain discrimination loss. The formula for the overall loss function is as follows: in, The cross-entropy loss function represents the classifier. This represents the adversarial loss function of the domain adversarial module. Represented as the fusion graph branch loss function, and These represent the weighting coefficients, Indicates the source domain input sample. Indicates the target domain input sample. Indicates the parameters of the neighborhood discriminator. This represents the probability that the neighborhood discriminator classifies a sample from the source domain as belonging to the source domain. This represents the probability that the discriminator classifies a sample from the target domain as belonging to the target domain.

9. The domain-adaptive cross-subject EEG signal emotion recognition method according to claim 8, characterized in that, The adversarial loss function of the domain adversarial module is obtained by L2 normalizing the local and global convolutional embeddings, calculating their mean squared errors, and constructing the structure-preserving loss. The formula is as follows: in, and These represent local graph embeddings. and global graph embedding The similarity matrix after L2 normalization Represents a normalized matrix. This represents the gain coefficient.

Images