Cross-subject brain electrical emotion recognition method and device based on domain self-adaption and adversarial fusion

Abstract

The invention discloses a cross-subject electroencephalogram emotion recognition method and device based on domain self-adaption and adversarial fusion, and belongs to the field of electroencephalogram signal processing and emotion calculation. The invention provides a cross-subject electroencephalogram emotion recognition method based on domain self-adaption and adversarial fusion, and aims to solve the problems that in the prior art, electroencephalogram signal preprocessing is insufficient in emotional feature retention capacity and cross-subject model generalization is poor, and the method specifically comprises the steps that an electroencephalogram signal data set is acquired, and electroencephalogram signals are preprocessed; after fractional order Fourier transform is carried out on the preprocessed electroencephalogram signals, differential entropy features are extracted; constructing a pre-training model; constructing a classification model; the classification model adopts an encoder in a pre-trained model after pre-training, and then a classifier is added; and adopting the classification model to realize target domain electroencephalogram signal emotion classification. Experiments show that the average accuracy of cross-subject emotion recognition on an SEED data set reaches 88.29%, and the method is suitable for scenes such as brain-computer interfaces and mental health monitoring.

Description

Technical Field

[0001] The present invention belongs to the field of EEG emotion technology, and in particular relates to a cross-subject EEG emotion recognition method and device based on domain adaptation and adversarial fusion. Background Art

[0002] Human emotions are closely related to health status and behavioral patterns. Real-time monitoring of an individual's emotional state facilitates objective health assessments and early warning of malicious behavior. EEG-based emotion recognition has become an important tool for emotion recognition due to its high temporal resolution and precision. EEG signals originate from specific channels and frequency bands and exhibit distinct responses to different emotional stimuli. However, individuals differ in their emotional perception and experience, leading to varying sensitivities to the same emotion. Consequently, data distributions vary significantly across subjects, making it difficult for models trained on source subject data to effectively generalize to target subjects. This makes it difficult for models trained on source subjects (training data) to directly generalize to unseen subjects (test data), limiting their applicability in practical applications.

[0003] In the existing EEG data preprocessing process, traditional bandpass filtering only statically intercepts frequency bands, which may lose dynamic information and cannot dynamically adapt to the non-stationarity of the signal. After cleaning and segmenting the data, features such as differential entropy are directly extracted. However, the extracted features are often unable to widely represent effective information and are interfered by noise in the original data, making the extracted features ineffective for subsequent module processing.

[0004] Existing domain adaptation methods try to narrow inter-domain differences by aligning the feature distributions of the source and target domains (e.g., adversarial training and distribution alignment loss). However, this relies on target domain data, which is difficult to meet in practical applications. Furthermore, when the target domain data is small, the model risks overfitting. Domain generalization methods assume that the target object is invisible during training and rely solely on source domain data for training. These methods require the model to generalize directly to the unknown target domain, treating different subjects as independent domains and extracting cross-domain invariant features through adversarial training or contrastive learning. However, when source data is insufficient, the model is prone to overfitting, struggles to generate diverse cross-subject features, and lacks robustness to noise.

[0005] In summary, existing technologies suffer from the following major issues: First, traditional EEG preprocessing processes are unable to adapt to the non-stationary characteristics of EEG signals, cannot accurately retain relevant information, are easily affected by noise, and feature extraction is incomplete, unable to capture comprehensive dynamic features, resulting in poor robustness. Second, existing domain adaptation and domain generalization methods rely on target domain data, are prone to model overfitting, have poor ability to extract domain-invariant features, and lack noise immunity. Summary of the Invention

[0006] The purpose of the present invention is to provide a cross-subject EEG emotion recognition method and device based on domain adaptation and adversarial fusion, which can better learn domain invariant features from EEG source data, thereby improving the accuracy of emotion classification and recognition;

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion includes the following steps:

[0009] Obtain an EEG signal data set, and perform preprocessing and linear dynamic smoothing on the EEG signal;

[0010] After performing fractional Fourier transform on the EEG signal after linear dynamic smoothing, differential entropy features are extracted;

[0011] Constructing a pre-training model, the pre-training model includes a domain adaptation model and a domain adversarial model. The domain adaptation model includes an encoder and a decoder. The differential entropy features of the EEG signal are weighted using a self-attention weighting method and input into the encoder. The encoder extracts common features, and the decoder reconstructs the common features to obtain a reconstruction loss. The domain adversarial model distinguishes domain-specific features from the common features and provides the source domain of the individual features, thereby forming an adversarial relationship with the encoder and obtaining an adversarial loss.

[0012] The pre-training loss is obtained by mixing the reconstruction loss and the adversarial loss to supervise the encoder training;

[0013] Constructing a classification model, wherein the classification model uses the encoder in the pre-trained model and adds a classifier after the encoder;

[0014] A classification model is used to realize the target domain EEG signal emotion classification.

[0015] Preferably, preprocessing the EEG signal includes downsampling, filtering, and artifact removal processing of the EEG signal.

[0016] Preferably, the linear dynamic smoothing process comprises the following steps:

[0017] The power spectral density of the preprocessed EEG signal was calculated, and the principal components were extracted as observation vectors through principal component analysis (PCA). The state space model was iteratively optimized using the expectation maximization (EM) algorithm combined with Kalman smoothing. The state mean after Kalman smoothing was used as the denoised alertness-related signal, and the non-correlated components were separated by the observation vector and the denoised alertness-related signal.

[0018] Preferably, the method further includes the following steps: using a fractional-order optimal interval selection algorithm based on bisection to obtain the optimal fractional-order Fourier transform order, wherein the fractional-order optimal interval selection algorithm based on bisection is to select the fractional-order Fourier transform order by bisection, record the training epoch and loss when the pre-training loss stops decreasing, obtain a better interval by comparison, and repeat until the optimal fractional-order Fourier transform order is obtained.

[0019] Among them, the fractional Fourier transform is expressed as:

[0020]

[0021]

[0022] in, is the kernel function, , , p is the order of fractional Fourier transform.

[0023] Preferably, initialize the order search interval from 0.1 to 1, select the midpoint (0.5) and endpoints (0.1 and 1.0) of the interval for experiments, compare the training losses obtained in 50 epochs in the experiment, select the two smaller values ​​to obtain the better interval, select the minimum experimental loss as the loss threshold Δ=0.0052e-8, perform the same selection experiment within the interval, observe the training loss in 50 epochs, stop training if it is greater than the threshold Δ, and try other orders until the optimal fractional Fourier transform order is obtained.

[0024] Preferably, the encoder is composed of a single-layer LSTM network; the number of decoders is the number of divided domains; the decoder is composed of a single-layer LSTM network and a fully connected layer, the decoder randomly selects samples from the initial EEG signal data set to form a batch of supervision data, and repeats the above steps on the supervision data. The supervision data serves as the supervision signal of the decoder. Each decoder is responsible for reconstructing the features related to its corresponding source subject to obtain the reconstructed features, calculating the mean square error between the reconstructed features and the corresponding supervision data, and accumulating the mean square error losses of all decoders to obtain the reconstruction loss.

[0025] Preferably, the domain adversarial model includes a gradient reversal layer and a domain classifier. The gradient reversal layer reverses the gradient direction of the common features, and the data after the gradient reversal is sent to the domain classifier to predict the domain to which the input data belongs. The difference between the prediction result of the domain classifier and the true domain label is calculated to obtain the adversarial loss.

[0026] Preferably, in the step of implementing the emotion classification of the target domain EEG signal using a classification model, the target domain EEG signal is preprocessed and subjected to fractional Fourier transform, and the differential entropy features are extracted. Then, the signals are weighted using the self-attention weight method and input into the pre-trained encoder to extract deep-level features, and the deep-level features are classified using the emotion classifier.

[0027] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion are performed.

[0028] In another aspect, the present invention further provides a cross-subject EEG emotion recognition device based on domain adaptation and adversarial fusion, comprising:

[0029] Memory, for storing software applications,

[0030] A processor is used to execute the software application, and each program of the software application correspondingly executes the steps of the above-mentioned cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion.

[0031] This method effectively retains the key components of emotion recognition in data through fractional Fourier transforms and linear dynamic smoothing. After the fractional Fourier transform, it extracts differential entropy features and uses domain adaptation and adversarial methods for model pre-training. The encoder is trained using only source domain data, and precise parameter adjustment is achieved through hybrid loss. Compared to traditional methods, this method can more efficiently learn invariant features between domains, improving the model's generalization ability. The trained encoder is then used for classification to achieve emotion recognition on unseen data, achieving more stable recognition accuracy and noise resistance in complex environments compared to traditional methods. This method is suitable for scenarios such as sentiment analysis and human-computer interaction, and has high practical value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the process of the present invention;

[0033] Figure 2 It is a schematic diagram of the process of linear dynamic smoothing;

[0034] Figure 3 Flowchart of the fractional-order optimal interval selection algorithm based on bisection. DETAILED DESCRIPTION

[0035] The present invention is further described below with reference to the accompanying drawings and specific embodiments.

[0036] The present invention provides a cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion, comprising the following steps:

[0037] S1. Obtain an EEG signal dataset and perform preprocessing and linear dynamic smoothing on the EEG signal;

[0038] Specifically, an EEG signal dataset is obtained as source domain data. In this embodiment, the SEED database is used. The SEED database contains a list of three basic emotions, "emotions". The data is divided and input according to 15 subjects, of which the data of 14 subjects are used as the source domain for pre-training.

[0039] More specifically, the preprocessing of the EEG signal includes downsampling the EEG signal at a downsampling rate of 1000 Hz; the downsampled EEG signal is subjected to a 0.1 Hz high-pass filter and a 30 Hz low-pass filter using a Butterworth filter to remove artifacts in the EEG signal, including electromyography and electrooculography.

[0040] Specifically, the linear dynamic smoothing process includes the following steps: calculating the power spectral density (PSD) of the preprocessed EEG signal, extracting the principal component as the observation vector by principal component analysis (PCA) method Initialize the linear dynamic system (LDS) parameters, iteratively optimize the state space model through the expectation maximization (EM) algorithm combined with Kalman smoothing, use the state mean after Kalman smoothing as the denoised alertness-related signal, and separate the irrelevant components of the EEG signal by separating the observation vector from the denoised alertness-related signal;

[0041] More specifically, the power spectral density of the preprocessed EEG signal in every 2-second window is calculated to quantify the energy distribution in different frequency bands; principal component analysis is used to reduce the dimension and extract the main features to construct the observation vector ;

[0042] Specifically, the linear dynamic system parameter initialization is expressed as:

[0043]

[0044] Among them, the state transfer matrix Describing potential states The time evolution relationship of the process noise covariance Reflects the covariance characteristics of the noise in the state equation, characterizes the uncertainty of state evolution, and observes the noise covariance represents the covariance of the measurement error in the observation equation, the initial state mean Define the potential state at the initial moment, the expected value initial state covariance Describes the uncertainty of the state at the initial moment.

[0045] More specifically, the state equation for linear dynamic smoothing is

[0046]

[0047] in, represents the potential vigilance-related state at time t (such as the neural activity corresponding to the alertness level), Represents the state transition matrix, describing the temporal dynamics of vigilance, Process noise (covariance RR) represents the random fluctuation of vigilance changes. Vigilance is a key indicator for measuring sustained attention, and its dynamic changes can be objectively quantified through EEG signals (such as the LDS model).

[0048] Specifically, Kalman smoothing calculates the posterior distribution, forward filtering uses Kalman filtering, and backward smoothing uses RTS smoothing.

[0049] Specifically, the EM algorithm parameter update, where the E step: calculates the expectation of the potential state, and the M step: updates the parameter θ to maximize the likelihood. In the E step, the Kalman smoothing result is used to calculate the following expectation:

[0050]

[0051]

[0052] in, represents the potential state mean at time t (smoothed result), reflecting the alertness-related signal after denoising, Represents the potential state covariance matrix at time t (after smoothing), describing the uncertainty of the state estimate.

[0053] In the M step, update 、 and , specifically expressed as:

[0054]

[0055]

[0056]

[0057] in, Indicates potential state, Represents the observation vector at time t.

[0058] Specifically, the state mean after Kalman smoothing is used as the denoised alertness-related signal, and then the non-correlated components are separated, which can be expressed as follows:

[0059]

[0060]

[0061] in, is the estimated alertness-related signal (after denoising), It is the separated non-correlated noise (such as electromyographic artifacts and environmental noise).

[0062] S2, after performing fractional Fourier transform on the EEG signal after linear dynamic smoothing, extract the differential entropy feature;

[0063] Specifically, the EEG signal is processed by linear dynamic smoothing and then subjected to fractional Fourier transform.

[0064] The fractional Fourier transform is expressed as:

[0065]

[0066]

[0067] in, is the kernel function, , , p is the order of fractional Fourier transform, p is 0.6~0.8, and in this embodiment, p =0.6.

[0068] S3. Construct a pre-training model, which includes a domain adaptation model and a domain adversarial model. The domain adaptation model includes an encoder and a decoder. The differential entropy features of the EEG signal are weighted using the self-attention weight method and input into the encoder. The encoder extracts common features, and the decoder reconstructs the common features to obtain reconstruction loss. The domain adversarial model distinguishes domain individual features from the common features and gives the source domain of the individual features, thereby forming an adversarial relationship with the encoder and obtaining adversarial loss.

[0069] Specifically, the self-attention weight method is used to automatically weight the channels and frequency bands of the data to highlight important time steps or features. The self-attention weight method introduces a linear layer to map the original features to a new feature space. The weights are obtained by normalizing the mapped features using the Softmax function:

[0070]

[0071] in, is the weight matrix with dimension D×D, is the bias vector with a dimension of D×1, x is the original data (label segmentation data in a single domain), is the generated one-dimensional weight with dimension D×1.

[0072] Get weighted features :

[0073]

[0074] Specifically, the encoder is composed of a single-layer LSTM network to extract shared features. The encoder learns universal feature representations across data sources to promote knowledge sharing between different data sources. The source domain data is processed in the above steps and input into the encoder. The encoder performs unified encoding on this data. In this embodiment, the encoder input is data from 14 source domains.

[0075] Specifically, the number of the decoders is the number of divided domains, which is the same as the number of encoder inputs. That is, in this embodiment, 14 decoders are established, and samples are randomly selected from the initial 14 source domain data to form a batch of supervision data, and the same processing is performed, that is, the supervision data is processed through steps S1~S2. In the pre-training stage, the supervision data serves as the supervision signal of the decoder. Each decoder is responsible for reconstructing the features related to its corresponding source subject to obtain the reconstructed features. The decoder is supervised by calculating the mean square error (MSE) between the reconstructed features and the corresponding supervision data.

[0076] More specifically, the decoder consists of a single-layer LSTM network and a fully connected layer to achieve reconstruction of the corresponding domain data. The decoder outputs x_out, calculates the mean square error between the decoder output x_out and the corresponding domain supervision data, and accumulates the MSE losses of all decoders to obtain the final reconstruction loss rec_loss.

[0077]

[0078]

[0079] in, represents the output of the decoder, represents the corresponding supervision data, represents the reconstruction loss obtained in domain i.

[0080] Specifically, the domain adversarial model includes a gradient reversal layer (GRL) and a domain classifier. The gradient reversal layer reverses the gradient direction of common features, making it impossible for the domain classifier to directly distinguish domain information from shared features, thereby forcing the model to learn domain-independent features. The data after gradient reversal is fed into the domain classifier to predict the domain to which the input data belongs. The difference between the prediction result of the domain classifier and the true domain label is calculated to obtain the adversarial loss.

[0081] ,

[0082] ,

[0083] in, is the GRL layer, SD is the domain classifier, To calculate the adversarial loss, is the predicted probability of the classifier for the i-th sample, It is the domain label, which indicates the result that the classifier is expected to predict.

[0084] S4. Pre-training loss is obtained by mixing reconstruction loss and adversarial loss to supervise the training of pre-training model.

[0085] Specifically, the pre-training loss in the pre-training stage is obtained by combining the reconstruction loss and the adversarial loss. , adjust the encoder hyperparameters through pre-training loss,

[0086]

[0087] More specifically, the method further includes the following steps: using a fractional-order optimal interval selection algorithm based on bisection to obtain an optimal fractional-order Fourier transform order, wherein the fractional-order optimal interval selection algorithm based on bisection is to select the fractional-order Fourier transform order by bisection, record the training epoch and loss when the pre-training loss stops decreasing, obtain a better interval by comparison, and repeat the process until the optimal fractional-order Fourier transform order is obtained.

[0088] In this embodiment, the order is selected as 0.1, 0.5 and 1.0 for fractional Fourier transform, and the pre-training loss is recorded. The training epoch and loss when the descent stops, the minimum experimental loss is selected as the loss threshold Δ = 0.0052e-8, and the interval of 0.5-1.0 for the next experiment is obtained by comparison. A minimum epoch and training loss group is always maintained, and the midpoint of the interval 0.7 obtained in the previous step is selected for the experiment. Repeat until the optimal order 0.6 is obtained, as shown in Tables 1 and 2. Table 1 shows the results of the fractional order optimal interval selection algorithm based on the dichotomy method, and Table 2 shows the average results of cross-subject recognition when p = 0.6. Then enter the classification stage, greatly reducing the problem scale and improving the experimental efficiency.

[0089] Table 1

[0090] Parameter order Epoch Pre-training loss 0.1 189 0.0026e-8 0.5 176 0.0024e-8 0.6 153 0.0023e-8 0.7 168 0.0023e-8 1.0 182 0.0025e-8

[0091] Table 2

[0092] Order Avg Std 0.6 88.29 5.56

[0093] S5. Build a classification model, wherein the classification model uses the encoder in the pre-trained model and then adds a classifier;

[0094] S6. Use the classification model to realize the emotion classification of EEG signals in the target domain.

[0095] Specifically, we reuse the feature extraction part of the pre-training module, namely the attention mechanism and encoder, and add a multi-layer perceptron (MLP) as the sentiment classifier. The MLP consists of two fully connected layers, with batch normalization layers and ReLU activation functions added between the layers. The negative log-likelihood loss function (NLL Loss) is used to calculate the classification loss, and the final classification probability and classification loss are calculated.

[0096] More specifically, after the data in the target domain is processed through steps S1 to S3, it is weighted using the self-attention weight method to generate weighted features. The weighted features are input into the pre-trained encoder to extract deep features. The deep features are classified using the sentiment classifier, and the classification loss is calculated. The classification loss is used to fine-tune the classification model.

[0097] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion are performed.

[0098] In another aspect, the present invention further provides a cross-subject EEG emotion recognition device based on domain adaptation and adversarial fusion, comprising:

[0099] Memory, for storing software applications,

[0100] A processor is used to execute the software application, and each program of the software application correspondingly executes the steps of the above-mentioned cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion.

Claims

1. A cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion, characterized by: The following steps are involved: Obtain an EEG signal data set, and perform preprocessing and linear dynamic smoothing on the EEG signal; After performing fractional Fourier transform on the EEG signal after linear dynamic smoothing, differential entropy features are extracted; Constructing a pre-training model, the pre-training model includes a domain adaptation model and a domain adversarial model. The domain adaptation model includes an encoder and a decoder. The differential entropy features of the EEG signal are weighted using a self-attention weighting method and input into the encoder. The encoder extracts common features, and the decoder reconstructs the common features to obtain a reconstruction loss. The domain adversarial model distinguishes domain-specific features from the common features and provides the source domain of the individual features, thereby forming an adversarial relationship with the encoder and obtaining an adversarial loss. The pre-training loss is obtained by mixing the reconstruction loss and the adversarial loss to supervise the encoder training; Constructing a classification model, wherein the classification model uses the encoder in the pre-trained model and adds a classifier after the encoder; A classification model is used to realize the target domain EEG signal emotion classification.

2. The cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion according to claim 1 is characterized in that: The preprocessing of EEG signals includes downsampling, filtering and artifact removal.

3. The cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion according to claim 2 is characterized in that: The linear dynamic smoothing process comprises the following steps: The power spectral density of the preprocessed EEG signal was calculated, and the principal components were extracted as observation vectors through principal component analysis. The state space model was iteratively optimized using the expectation maximization algorithm combined with Kalman smoothing. The state mean after Kalman smoothing was used as the denoised alertness-related signal, and the irrelevant components were separated using the observation vector and the denoised alertness-related signal.

4. The cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion according to claim 1 is characterized in that: The following steps are also included: The optimal fractional Fourier transform order is obtained by using a fractional-order optimal interval selection algorithm based on bisection. The fractional-order optimal interval selection algorithm based on bisection selects the fractional Fourier transform order by bisection, records the training epoch and loss when the pre-training loss stops decreasing, obtains a better interval by comparison, and repeats it continuously until the optimal fractional Fourier transform order is obtained.

5. The cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion according to claim 1 is characterized in that: The encoder is composed of a single-layer LSTM network; the number of decoders is the number of divided domains; the decoder is composed of a single-layer LSTM network and a fully connected layer. Samples are randomly selected from the initial EEG signal data set to form a batch of supervision data, and the above steps are repeated for the supervision data. The supervision data serves as the supervision signal of the decoder. Each decoder is responsible for reconstructing the features related to its corresponding source subject to obtain the reconstructed features, and the mean square error between the reconstructed features and the corresponding supervision data is calculated. The mean square error losses of all decoders are accumulated to obtain the reconstruction loss.

6. The cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion according to claim 1 is characterized in that: The domain adversarial model includes a gradient reversal layer and a domain classifier. The gradient reversal layer reverses the gradient direction of common features, and the data after gradient reversal is sent to the domain classifier to predict the domain to which the input data belongs. The difference between the prediction result of the domain classifier and the true domain label is calculated to obtain the adversarial loss.

7. The cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion according to claim 1 is characterized in that: In the step of using a classification model to implement emotion classification of target domain EEG signals, the target domain EEG signals are preprocessed and subjected to fractional Fourier transform to extract differential entropy features. Then, after weighting using the self-attention weight method, the signals are input into the pre-trained encoder to extract deep-level features, and the deep-level features are classified using the emotion classifier.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion described in any one of claims 1 to 7 are performed.

9. A cross-subject EEG emotion recognition device based on domain adaptation and adversarial fusion, characterized by: include: Memory, for storing software applications, A processor is used to execute the software application, each program of the software application correspondingly executes the steps of the cross-subject EEG emotion recognition method based on domain adaptation and adversarial fusion as described in any one of claims 1 to 7.

Images