Cross-subject electroencephalogram emotion recognition method based on dynamic domain invariant representation decoupling and recombination

Abstract

The invention discloses a cross-subject electroencephalogram emotion recognition method and system based on dynamic domain invariant representation decoupling and recombination, and belongs to the technical field of artificial intelligence. The invention provides a non-personalized decoupling and recombination framework for cross-subject electroencephalogram emotion recognition, and aims to separate emotion-related individual invariant features from cross-subject individual invariant features through complex electroencephalogram signal characterization obtained through dynamic decoupling, so that individual differences are eliminated while emotion classification performance is guaranteed. Specifically, the method comprises the following steps: carrying out original EEG data analysis and preprocessing by using MATLAB and Python MNE libraries; frequency spectrum and space features of EEG signals are extracted through a multi-channel frequency spectrum space self-attention mechanism module, and capture of emotional features is enhanced in combination with a self-attention mechanism and a cross-attention mechanism; a joint distribution alignment method based on a category prototype is adopted, and decoupled feature distribution is optimized, so that invariant features in subjects and invariant features among subjects have higher distinction degree in emotion classification; the optimized decoupling features are recombined through a linear network, the two features are coordinated to perform more sufficient emotion representation extraction, and emotion classification is performed through a multi-layer perceptron. According to the method, excellent cross-subject emotion recognition performance is obtained on multiple data sets, and the emotion recognition rate of the electroencephalogram signals in a cross-subject scene can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of domain-adaptive cross-subject emotion recognition based on physiological signals, and in particular to a cross-subject EEG emotion recognition method and system with dynamic domain-invariant representation decoupling and reconstruction. Background Art

[0002] Emotions are a core factor in human cognition and social interaction, playing a vital role in diverse fields, including daily life, healthcare, education, and industrial production. Emotions not only influence human decision-making and behavior but also have broad applications in the diagnosis and treatment of mental illness, mental health monitoring, and human-computer interaction (HCI). Therefore, accurately and objectively identifying human emotional states is crucial for the development of intelligent systems.

[0003] In recent years, with advances in artificial intelligence and biosignal processing technologies, affective computing has become a highly sought-after research area. Among them, electroencephalogram (EEG)-based emotion recognition methods have gradually become a key research area due to their advantages, such as direct reflection of brain activity and high temporal resolution. EEG signals can capture neural activity in the cerebral cortex and reflect emotion-related neural signatures through information in different frequency bands. Compared to traditional overt emotion measurement methods (such as facial expressions, voice, and physiological signals), EEG signals are less susceptible to subjective control by individuals and therefore provide a more objective assessment of emotional state.

[0004] However, due to individual differences in emotional expression, cross-subject EEG emotion recognition still faces severe challenges. Different individuals have significantly different neural activity patterns, and even under the same emotional stimulus, EEG signals from different individuals may still show large inter-individual variability. In addition, EEG data is often affected by physiological state (such as fatigue level and cognitive load) and environmental noise, resulting in poor generalization of the model across different subjects. Therefore, traditional within-subject training methods generally achieve high recognition accuracy, while the performance of cross-subject methods often decreases significantly, restricting the promotion of EEG emotion recognition technology in practical applications.

[0005] To address this issue, researchers have proposed a variety of domain adaptation and transfer learning methods in recent years to improve the cross-subject generalization capabilities of EEG emotion recognition. Traditional within-subject training methods require a large amount of labeled data for each new user, while cross-subject methods attempt to use existing subject data to train the model so that it can generalize to new individuals. However, due to individual differences in EEG signals, directly applying the features of the source subject to the target subject often leads to a decrease in model performance. Therefore, how to establish stable emotion representations across different subjects and improve the adaptability of the model is a key challenge in current EEG emotion recognition research.

[0006] In summary, cross-subject EEG emotion recognition is an important research direction in affective computing, with broad application prospects. However, due to the variability of individual neural activity, traditional methods struggle to achieve good generalization across subjects. To address this issue, this study proposes a non-depersonalized EEG cross-subject emotion recognition method based on dynamic domain-invariant representation decoupling and joint distribution alignment. This approach aims to improve the model's generalization capabilities and promote the development of EEG emotion recognition technology in practical applications. Summary of the Invention

[0007] Purpose of the invention: In response to the shortcomings of cross-subject EEG emotion recognition methods, the first purpose of the present invention is to provide a cross-subject EEG emotion recognition method based on the decoupling and reorganization of dynamic domain invariant representations, which can systematically model the EEG signals of different subjects and optimize the cross-subject representation extraction by decoupling and recoupling the intra-subject invariant representation and the inter-subject invariant representation in the EEG representation. Based on the application of this method, the present invention can provide a cross-subject EEG emotion recognition system based on the decoupling and reorganization of dynamic domain invariant representations.

[0008] Technical Solution: A cross-subject EEG emotion recognition method based on decoupling and reorganization of dynamic domain invariant representations is provided. This method optimizes emotion classification performance by decoupling and reorganizing non-personalized subject-invariant features, promoting inter-subject generalization while optimizing the boundaries of emotion classification. The method includes the following steps:

[0009] S1. Acquire multi-channel EEG emotion data and preprocess the data to obtain a standardized EEG signal sequence;

[0010] S2. Extract spectral features from EEG data using a multi-head self-attention network to capture the multi-dimensional frequency domain dependencies of the signal;

[0011] S3, extract spatial features from EEG data through a multi-head self-attention network, combined with the spectral feature extraction of S2 to capture the frequency-space dependency of the signal;

[0012] S4. Decoupling the EEG features using the subject-invariant feature decoupling module, separating the emotion-related within-subject invariant features from the shared between-subject invariant features through an unshared linear mapping layer;

[0013] S5. Design marginal probability alignment based on maximum mean difference (MMD) loss and conditional probability alignment based on category prototype clustering, and combine the two to form the joint distribution alignment loss of the subjects.

[0014] S6. Align and optimize the intra-subject invariant features of the source and target subject domains through conditional probability alignment based on category prototype clustering.

[0015] S7. Align and optimize the inter-subject invariant features of the source and target domains through the joint distribution alignment loss.

[0016] S8, based on the subject-invariant feature reorganization module, the decoupled and optimized within-subject invariant features and the between-subject invariant features are recoupled and fused to obtain an invariant representation with sufficient emotional characteristics;

[0017] S9. Input the extracted features into the emotion classifier and the subject classifier to achieve emotion classification across subjects.

[0018] Furthermore, the specific process of step S1 includes:

[0019] S1.1. Downsample the EEG data to 200 Hz using MATLAB.

[0020] S1.2. Manually remove myoelectric and oculoscopic artifacts from EEG signals;

[0021] S1.3. Band-pass filtering of EEG data was performed using the Python MNE toolkit. A 0.3 Hz to 50 Hz band-pass filter was used to remove out-of-band noise components.

[0022] S1.4. Divide the EEG data into several non-overlapping segments, each corresponding to an epoch;

[0023] S1.5. Extract EEG spectral features using differential entropy (DE);

[0024] S1.6. Feature smoothing is performed through a linear dynamic system (LDS).

[0025] In the method, the specific process of characterizing the self-representation in step S2 includes:

[0026] The input is the given EEG dataset D = {X, Y cls , Y sub},in is a multi-channel EEG signal, and the emotion label is Y cls , the subject identity label is Y sub , where C represents the number of electrodes and F represents the frequency band. The F spectra are specifically the δ band (1-4 Hz), θ band (4-8 Hz), α band (8-14 Hz), β band (14-30 Hz), and γ band (30-50 Hz). In order to enhance the representation of the original spectral features without changing the spatial dimension C, an attention matrix based on the F frequency band is adopted and constructed to establish connections between different frequency bands. By using the attention mechanism to construct the relationship matrix, X is transposed and several linear transformations are performed to generate the query (Q) and key (K):

[0027]

[0028] Where, X T is the result of transposing X, and Q F and K F Then, for Q F and K F Perform a scaled dot product operation to calculate the correlation between different electrode positions, and process the results through the SoftMax function to obtain the spectrum correlation score. Finally, use the dot product to weight the score X to obtain the feature The whole process can be described as:

[0029]

[0030] Where, d K Represents a normalization parameter equal to K F The dimension of . The residual connection is further used to further constrain the output:

[0031] f F =(W F (g T +X T )+X T ) T

[0032] Where, f F Represents the EEG features after frequency domain enhancement.

[0033] In the method, the specific process of characterizing the self-representation in step S3 includes:

[0034] Through the self-attention mechanism, the spatial information in EEG is further captured on the basis of S2. In order to ensure that the features are extracted in a high dimension, we use a linear transformation layer to transform f F The dimension required for mapping to the subsequent K layers of the self-attention network is: f 0 =W 0 f F , where W 0 represents the parameters of the linear mapping layer, and As output it will be used to generate the query (Q), key (K) and value (V):

[0035]

[0036] Where, and Represents Q i , K i and Vi The parameter matrix of Q i , K i and V i , using the attention mechanism for fusion, the whole process can be expressed as:

[0037]

[0038] Since the encoder adopts a multi-head attention mechanism, the features of each attention head are concatenated and residually connected with the upper layer network to obtain the output features of the i-th layer:

[0039]

[0040] Where, represents the output feature of the j-th attention head, LN() is the normalization layer, and f is the output of the frequency-space feature extractor.

[0041] In the method, the specific process of characterizing the self-representation in step S4 includes:

[0042] High-dimensional features Through the subject-invariant representation decoupling module, it is decoupled into two different parts: emotion-related intra-subject invariant features and inter-subject invariant features. In the subject-invariant representation decoupling module, the high-dimensional feature f K Through two K SIFD Layer MLP E IIF and E MIF Map into a vector. First, f K Flattened to f flat =Flatten(f i ), where and d flat =C×d att . Then the flattened vector f flat Enter E IIF and E MIF , respectively generate emotion-related invariant features within the subject and inter-subject invariant features

[0043] Furthermore, after decoupling is completed, their feature distributions are made as different as possible to achieve better decoupling:

[0044]

[0045] Where Cosine(·) is the cosine similarity function and |·| is the absolute value function. We take the absolute value of the result to ensure that f IIF and f MIF The cosine similarity between them is as close to 0 as possible.

[0046] In the method, the specific process of characterizing the self-representation in step S5 includes:

[0047] Joint subject distribution alignment consists of two parts: marginal probability distribution alignment and conditional probability distribution alignment. In the marginal probability distribution alignment part, MMD loss is used to align the distribution of features of source domain samples and target domain samples:

[0048]

[0049] Where b s and b t are the number of samples in the source subject set and the target subject set, represents the characteristics of the i-th sample in the source domain of the batch, Represents the features of the jth sample in the target domain within the batch.

[0050] In addition, the conditional probability distribution is defined to a certain extent by using the category prototype, which promotes the alignment of the conditional probability distribution by reducing the distance between the features of different category samples and their category prototypes. Therefore, the proposed alignment method based on category prototypes within the training batch enhances the sentiment-related properties of the features. First, according to the category label Y cls The samples in each batch are divided into n groups, where n is the number of emotion categories. For each class, the average feature vector of all samples in the class in the batch is calculated. Set a category alignment constraint That is, the distance between each sample in a class and its class prototype should be within a certain range:

[0051]

[0052] Where n j represents the number of samples of class j, ||·|| F is the Frobenius norm, and μ represents the boundary parameter of the constraint feature.

[0053] In the method, the specific process of characterizing the self-representation in step S6 includes:

[0054] For the within-subject invariant features, this representation represents the relatively representative features of each subject. IIF , uses a subject discriminator and a cross entropy loss function to enable it to learn features related to the subject:

[0055]

[0056] Where G sub is the subject identity discriminator, For E IIFThe output features of the first layer in , is the cross entropy loss. In addition, the capture of emotional features is promoted by combining the conditional probability distribution loss based on the category prototype in the subject distribution alignment loss. Combining the above two constraints, the optimization loss of the invariant features within the subject is defined as where f IIF Contains sample features of both source and target objects.

[0057] In the method, the specific process of characterizing the self-representation in step S7 includes:

[0058] Inter-subject invariant features represent the invariant features shared between subjects and reflect the more common patterns in the group. Therefore, the complete joint subject distribution alignment loss is used for subject representation optimization.

[0059] In the method, the specific process of characterizing the self-representation in step S8 includes:

[0060] To ensure that the decoupling process retains as much useful information as possible, we use cross attention to IIF and f MIF Reconstruct the original signal. First, pass f through a linear layer IIF and f MIF Restore to f K The dimension of in m∈{IIF,MIF}, transpose(a,b) is a function that transposes the dimension of a to the dimension of b. Then, we Perform several linear transformations to generate queries (Q), keys (K), and values ​​(V): in and It's Q rec , K rec and V rec Next, the representations are fused through multi-head attention and the high-dimensional features are mapped back to the original spectral feature dimensions.

[0061] For the previously extracted f IIF and f MIF , connect them together and pass through the encoder E SIFR Further feature extraction is performed to obtain the emotion-related invariant feature f. For feature f, SJDA loss and cross entropy loss are used for constraint and optimization. Therefore, the constraints of the subject-invariant feature reconstruction module are as follows:

[0062]

[0063] These components are reorganized into emotion-related categorical features through the subject-invariant feature recombination (SIFR) module.

[0064] In the method, the specific process of characterizing the self-representation in step S9 includes:

[0065] The overall loss function of this model consists of four parts: cross entropy loss, joint subject distribution alignment loss, subject-invariant feature decoupling loss, and subject-invariant feature reorganization loss. The overall loss function can be expressed as:

[0066]

[0067] Where α and β are hyperparameters, which balance the decoupling process through the weights of IIF and MIF.

[0068] The method achieves a good balance between emotion recognition and cross-subject recognition by jointly optimizing emotion classification loss and cross-subject consistency loss, thereby improving the overall performance of multimodal emotion recognition.

[0069] Furthermore, the source domain data contains EEG signals, emotion labels, and subject identity information from multiple subjects, while the target domain data only contains EEG signals and subject identity information. EEG signals are collected by multiple electrodes and cover multiple frequency bands, including the δ band (1-4 Hz), theta band (4-8 Hz), alpha band (8-14 Hz), beta band (14-30 Hz), and gamma band (30-50 Hz). During the training process, the labeled emotion information in the source domain is combined with the unlabeled emotion data in the target domain for unsupervised domain adaptation, so that the model can make full use of the labeled information in the source domain while adapting to the data distribution of the target domain to achieve emotion recognition across subjects; in the testing phase, only the target domain data is used for emotion recognition.

[0070] By implementing the above method and steps, a cross-subject EEG emotion recognition system with dynamic domain invariant representation decoupling and reorganization can be obtained accordingly. The system includes a feature extraction module based on frequency-space dual attention, a subject-invariant feature decoupling module based on intra-subject and inter-subject features, a subject-invariant feature reorganization module, a cross-subject emotion classification module based on a dual discriminator architecture, and a subject joint distribution alignment module based on marginal probability and conditional probability alignment. The system and the above modules are obtained by implementing the method according to claim 1;

[0071] The feature extraction module based on frequency-space dual attention is implemented through S2 and S3, combining frequency domain and spatial domain for joint modeling to capture the dependency of signals in time domain and spatial domain;

[0072] The subject-invariant feature decoupling module based on intra-subject and inter-subject features is implemented by S4, S6 and S7;

[0073] The subject-invariant feature reconstruction module is implemented by S8, and the adequacy of decoupling is ensured by constructing the reconstruction loss;

[0074] The cross-subject emotion classification module based on the dual discriminator architecture is implemented by S9, which inputs the extracted features into the emotion classifier and the subject classifier to perform emotion recognition and subject recognition respectively, and finally achieves cross-subject emotion classification;

[0075] The subject joint distribution alignment module based on marginal probability and conditional probability alignment is implemented by S5, which optimizes the inter-subject invariant features of the source subject domain and the target subject domain, so that the cross-subject features have higher consistency and recognition.

[0076] Beneficial effects: Compared with the prior art, the significant effects and substantial features of the present invention are mainly:

[0077] (1) A subject-invariant feature decoupling and recombination framework is proposed, which can effectively eliminate feature redundancy, capture both within-subject and between-subject invariant features related to emotion, and fuse them for emotion classification.

[0078] (2) A joint distribution alignment method is introduced, which uses class prototypes and MMD distance to simultaneously constrain the sample probability distribution.

[0079] (3) A spectral-spatial dual-attention emotion feature extractor is proposed, which establishes bidirectional connections across different frequency bands and electrode channels.

[0080] (4) Our method achieves state-of-the-art performance with an accuracy of 93.12% and 77.31% on the SEED and SEED-IV datasets, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a workflow diagram of the method of the present invention;

[0082] Figure 2 It is a data preprocessing flow chart described in the present invention;

[0083] Figure 3 It is the overall framework diagram of the system for applying the method of the present invention; DETAILED DESCRIPTION

[0084] In order to make the purpose, advantages and technical solutions of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be fully and clearly described below with reference to the accompanying drawings;

[0085] Emotions are crucial in fields such as cognition, medicine, and human-computer interaction, and their objective identification is of great significance to the development of intelligent systems. EEG emotion recognition has attracted much attention due to its high temporal resolution and its ability to directly reflect brain activity. However, cross-subject EEG emotion recognition faces severe challenges: there are significant differences in individual neural activity patterns, and the EEG signal distribution of different subjects under the same emotion is different. At the same time, it is affected by physiological state, environmental noise, etc., resulting in poor generalization ability of traditional methods across different subjects. Currently, researchers are trying to use domain adaptation and transfer learning methods to improve generalization ability, but traditional methods still require a large amount of labeled data, and directly migrating source subject features often leads to a decrease in recognition accuracy. Therefore, how to construct a stable emotion representation and improve cross-subject adaptability is the core difficulty of EEG emotion recognition. The present invention proposes a non-depersonalized method based on dynamic domain invariant representation decoupling and joint distribution alignment to optimize cross-subject feature alignment, improve generalization ability, and provide a breakthrough solution for the practical application of EEG emotion recognition.

[0086] The present invention provides a cross-subject EEG emotion recognition method based on dynamic domain invariant representation decoupling and reorganization, combined with Figure 1 The process shown and Figure 3 The method framework diagram shown.

[0087] S1. Obtain multi-channel EEG emotion data and preprocess the data to obtain a standardized EEG signal sequence:

[0088] Combine Figure 2 The data preprocessing flowchart shown in the figure begins by downsampling the EEG data to 200 Hz using MATLAB, and manually removing myoelectric and electrooculographic artifacts. The EEG data is then bandpass filtered from 0.3 Hz to 50 Hz using the Python MNE toolkit to remove noise. Next, the EEG data is divided into several non-overlapping segments, each corresponding to an epoch, and differential entropy (DE) is used to extract EEG spectral features. Finally, the features are smoothed using a linear dynamic system (LDS) to improve signal stability.

[0089] S2. Extract spectral features from EEG data using a multi-head self-attention network to capture the multi-dimensional frequency domain dependencies of the signal:

[0090] The input is the given EEG dataset D = {X, Y cls ,Y sub},in is a multi-channel EEG signal, and the emotion label is Y cls , the subject identity label is Y sub, where C represents the number of electrodes and F represents the frequency band. The F spectra are specifically the δ band (1-4 Hz), the θ band (4-8 Hz), the α band (8-14 Hz), the β band (14-30 Hz), and the γ band (3050 Hz). In order to enhance the representation of the original spectral features without changing the spatial dimension C, an attention matrix based on the F frequency bands is adopted and constructed to establish connections between different frequency bands. By using the attention mechanism to construct the relationship matrix, X is transposed and several linear transformations are performed to generate the query (Q) and key (K):

[0091]

[0092] Where, X T is the result of transposing X, and Q F and K F Then, for Q F and K F Perform a scaled dot product operation to calculate the correlation between different electrode positions, and process the results through the SoftMax function to obtain the spectrum correlation score. Finally, use the dot product to weight the score X to obtain the feature The whole process can be described as:

[0093]

[0094] Where, d K Represents a normalization parameter equal to K F The dimension of . The residual connection is further used to further constrain the output:

[0095] f F =(W F (g T +X T )+X T ) T

[0096] Where, f F Represents the EEG features after frequency domain enhancement.

[0097] S3, extract spatial features from EEG data through a multi-head self-attention network, combined with the spectral feature extraction of S2 to capture the frequency-space dependency of the signal:

[0098] Through the self-attention mechanism, the spatial information in EEG is further captured on the basis of S2. In order to ensure that the features are extracted in a high dimension, we use a linear transformation layer to transform f F The dimension required for mapping to the subsequent K layers of the self-attention network is: f 0 =W 0 fF , where W 0 represents the parameters of the linear mapping layer, and As output it will be used to generate the query (Q), key (K) and value (V):

[0099]

[0100] Where, and Represents Q i , K i and V i The parameter matrix of Q i , K i and V i , using the attention mechanism for fusion, the whole process can be expressed as:

[0101]

[0102] Since the encoder adopts a multi-head attention mechanism, the features of each attention head are concatenated and residually connected with the upper layer network to obtain the output features of the i-th layer:

[0103]

[0104] Where, represents the output feature of the j-th attention head, LN() is the normalization layer, and f is the output of the frequency-space feature extractor.

[0105] s4. Decouple the EEG features using the subject-invariant feature decoupling module, separating the emotion-related within-subject invariant features from the inter-subject shared invariant features through a non-shared linear mapping layer:

[0106] High-dimensional features Through the subject-invariant representation decoupling module, it is decoupled into two different parts: emotion-related intra-subject invariant features and inter-subject invariant features. In the subject-invariant representation decoupling module, the high-dimensional feature f K Through two K SIFD Layer MLP E IIF and E MIF Map into a vector. First, f K Flattened to f flat =Flatten(f i ), where and d flat =C×d att . Then the flattened vector f flat Enter E IIF and E MIF , respectively generate emotion-related invariant features within the subject and inter-subject invariant features

[0107] Furthermore, after decoupling is completed, their feature distributions are made as different as possible to achieve better decoupling:

[0108]

[0109] Where Cosine(·) is the cosine similarity function and |·| is the absolute value function. We take the absolute value of the result to ensure that f IIF and f MIF The cosine similarity between them is as close to 0 as possible.

[0110] S5. Design marginal probability alignment based on maximum mean difference (MMD) loss and conditional probability alignment based on category prototype clustering, and combine the two to form the joint distribution alignment loss of the subjects:

[0111] Joint subject distribution alignment consists of two parts: marginal probability distribution alignment and conditional probability distribution alignment. In the marginal probability distribution alignment part, MMD loss is used to align the distribution of features of source domain samples and target domain samples:

[0112]

[0113] Where b s and b t are the number of samples in the source subject set and the target subject set, represents the characteristics of the i-th sample in the source domain of the batch, Represents the features of the jth sample in the target domain within the batch.

[0114] In addition, the conditional probability distribution is defined to a certain extent by using the category prototype, which promotes the alignment of the conditional probability distribution by reducing the distance between the features of different category samples and their category prototypes. Therefore, the proposed alignment method based on category prototypes within the training batch enhances the sentiment-related properties of the features. First, according to the category label Y cls The samples in each batch are divided into n groups, where n is the number of emotion categories. For each class, the average feature vector of all samples in the class in the batch is calculated. Set a category alignment constraint That is, the distance between each sample in a class and its class prototype should be within a certain range:

[0115]

[0116] Where n j represents the number of samples of class j, ||·|| Fis the Frobenius norm, and μ represents the boundary parameter of the constraint feature.

[0117] S6. Align and optimize the invariant features of the source and target domains through conditional probability alignment based on category prototype clustering:

[0118] Through multimodal feature embedding and self-representation module, multimodal features are decoupled into four features: f A 、g A 、f B and g B , where f A and f B Represents the modality-specific features of EEG and EM, g A and g B represents the complementary features between modalities. For the above features, we use contrastive learning to further disentangle them. In each modality, since its modality-specific features and complementary features originate from the same modality, we use a distance function to reduce the distance between them in the feature space. However, for the two sets of complementary features generated by the cross-attention shared by the two modalities, we need to increase their distance in the feature space. Since there are both positive and negative samples, we use Triplet loss to constrain the features:

[0119] loss T (x,x + ,x - )=max(D(x + ,x)-D(x - ,x)+m,0)

[0120] Where D(·) represents the Jensen-Shannon distance, x + It is the feature that needs to be brought closer to the anchor point x in the feature distance, x - The anchor point x needs to be moved further away, and m is the boundary parameter. Therefore, we set the corresponding two sample pairs (x, x + ,x - ) for comparative learning:

[0121] loss CL =loss T (f A ,g A ,g B )+loss T (f B , g B ,g A ).

[0122] Where, loss CL is the training loss in stage S6.

[0123] S7. Align and optimize the inter-subject invariant features of the source and target domains through the joint distribution alignment loss:

[0124] Inter-subject invariant features represent the invariant features shared between subjects and reflect the more common patterns in the group. Therefore, the complete joint subject distribution alignment loss is used for subject representation optimization.

[0125] S8. Based on the subject-invariant feature reorganization module, the decoupled and optimized within-subject invariant features and the inter-subject invariant features are recoupled to fuse them to obtain an invariant representation with sufficient emotional characteristics:

[0126] To ensure that the decoupling process retains as much useful information as possible, we use cross attention to IIF and f MIF Reconstruct the original signal. First, pass f through a linear layer IIF and f MIF Restore to f K The dimension of in m∈{IIF,MIF}, transpose(a,b) is a function that transposes the dimension of a to the dimension of b. Then, we Perform several linear transformations to generate queries (Q), keys (K), and values ​​(V): in and It's Q rec , K rec and V rec Next, the representations are fused through multi-head attention and the high-dimensional features are mapped back to the original spectral feature dimensions.

[0127] For the previously extracted f IIF andf MIF , connect them together and pass through the encoder E SIFR Further feature extraction is performed to obtain the emotion-related invariant feature f. For feature f, SJDA loss and cross entropy loss are used for constraint and optimization. Therefore, the constraints of the subject-invariant feature reconstruction module are as follows:

[0128]

[0129] These components are reorganized into emotion-related categorical features through the subject-invariant feature recombination (SIFR) module.

[0130] S9. Based on the subject-invariant feature reorganization module, the decoupled and optimized within-subject invariant features and the inter-subject invariant features are recoupled to fuse them to obtain an invariant representation with sufficient emotional characteristics:

[0131] The overall loss function of this model consists of four parts: cross entropy loss, joint subject distribution alignment loss, subject-invariant feature decoupling loss, and subject-invariant feature reorganization loss. The overall loss function can be expressed as:

[0132]

[0133] Where α and β are hyperparameters that balance the decoupling process by weighting IIF and MIF. The proposed method achieves a good balance between emotion recognition and cross-subject recognition by jointly optimizing the emotion classification loss and the cross-subject consistency loss, thereby improving the overall performance of multimodal emotion recognition.

Claims

1. A cross-subject EEG emotion recognition method based on decoupling and reorganization of dynamic domain invariant representations, characterized by: This method optimizes emotion classification performance by decoupling and reorganizing non-personalized subject-invariant features, promoting inter-subject generalization while optimizing the boundaries of emotion classification. It includes the following steps: S1. Acquire multi-channel EEG emotion data and preprocess the data to obtain a standardized EEG signal sequence; S2. Extract spectral features from EEG data using a multi-head self-attention network to capture the multi-dimensional frequency domain dependencies of the signal: S3, extract spatial features from EEG data through a multi-head self-attention network, combined with the spectral feature extraction of S2 to capture the frequency-space dependency of the signal; S4. Decoupling the EEG features using the subject-invariant feature decoupling module, separating the emotion-related within-subject invariant features from the shared between-subject invariant features through an unshared linear mapping layer; S5. Design marginal probability alignment based on maximum mean difference (MMD) loss and conditional probability alignment based on category prototype clustering, and combine the two to form the joint distribution alignment loss of the subjects; S6. Align and optimize the invariant features of the source and target domains through conditional probability alignment based on category prototype clustering; S7, aligning and optimizing the inter-subject invariant features of the source and target domains through the joint distribution alignment loss; S8, based on the subject-invariant feature reorganization module, the decoupled and optimized within-subject invariant features and the between-subject invariant features are recoupled and fused to obtain an invariant representation with sufficient emotional characteristics; S9. Input the extracted features into the emotion classifier and the subject classifier to achieve emotion classification across subjects.

2. The cross-subject EEG emotion recognition method based on dynamic domain invariant representation decoupling and reconstruction according to claim 1 is characterized in that: The specific process of step S1 includes: S1.

1. Downsample the EEG data to 200 Hz using MATLAB: S1.

2. Manually remove myoelectric and oculoscopic artifacts from EEG signals; S1.

3. Band-pass filtering of EEG data was performed using the Python MNE toolkit. A 0.3 Hz to 50 Hz band-pass filter was used to remove out-of-band noise components. S1.

4. Divide the EEG data into several non-overlapping segments, each corresponding to an epoeh; S1.

5. Extract EEG spectral features using differential entropy (DE); S1.

6. Feature smoothing is performed through a linear dynamic system (LDS).

3. The cross-subject EEG emotion recognition method based on dynamic domain invariant representation decoupling and reorganization according to claim 1 is characterized in that: The specific process of step S2 includes: The input is the given EEG dataset D = {X,Y cls , Y sub },in is a multi-channel EEG signal, and the emotion label is Y cls , the subject identity label is Y sub , where C represents the number of electrodes and F represents the frequency band. The F spectra are specifically the δ band (1-4 Hz), θ band (4-8 Hz), α band (8-14 Hz), β band (14-30 Hz), and γ band (30-50 Hz). In order to enhance the representation of the original spectral features without changing the spatial dimension C, an attention matrix based on the F frequency band is adopted and constructed to establish connections between different frequency bands. By using the attention mechanism to construct the relationship matrix, X is transposed and several linear transformations are performed to generate the query (Q) and key (K): Where, X T is the result of transposing X, and Q F and K F Then, for Q F and K F Perform a scaled dot product operation to calculate the correlation between different electrode positions, and process the results through the SoftMax function to obtain the spectrum correlation score. Finally, use the dot product to weight the score X to obtain the feature The whole process can be described as: Where, d K Represents a normalization parameter equal to K F The dimension of . The residual connection is further used to further constrain the output: f F =(W F (g T +X T )+X T ) T Where, f F Represents the EEG features after frequency domain enhancement.

4. The cross-subject EEG emotion recognition method based on dynamic domain invariant representation decoupling and reconstruction according to claim 1 is characterized in that The specific process of step S3 includes: Through the self-attention mechanism, the spatial information in EEG is further captured on the basis of S2. In order to ensure that the features are extracted in a high dimension, we use a linear transformation layer to transform f F The dimension required for mapping to the subsequent K layers of the self-attention network is: f 0 =w 0 f F , where W 0 represents the parameters of the linear mapping layer, and As output it will be used to generate the query (Q), key (K) and value (V): Where, and Represents Q i , K i and V i The parameter matrix of Q i , K i and V i , using the attention mechanism for fusion, the whole process can be expressed as: Since the encoder adopts a multi-head attention mechanism, the features of each attention head are concatenated and residually connected with the upper layer network to obtain the output features of the i-th layer: Where, represents the output feature of the j-th attention head, LN() is the normalization layer, and f is the output of the frequency-space feature extractor.

5. The cross-subject EEG emotion recognition method based on dynamic domain invariant representation decoupling and reorganization according to claim 1 is characterized in that: The specific process of step S4 includes: High-dimensional features Through the subject-invariant representation decoupling module, it is decoupled into two different parts: emotion-related intra-subject invariant features and inter-subject invariant features. In the subject-invariant representation decoupling module, the high-dimensional feature f K Through two K SIFD Layer MLP E IIF and E MIF Map into a vector. First, f K Flattened to f flat =Flatten(f i ), where andd flat =C×d att . Then the flattened vector f flat Enter E IIF and E MIF , respectively generate emotion-related invariant features within the subject and inter-subject invariant features Furthermore, after decoupling is completed, their feature distributions are made as different as possible to achieve better decoupling: Where Cosine(·) is the cosine similarity function and |·| is the absolute value function. We take the absolute value of the result to ensure that f IIF and f MIF The cosine similarity between them is as close to 0 as possible.

6. The cross-subject EEG emotion recognition method based on dynamic domain invariant representation decoupling and reorganization according to claim 1 is characterized in that: The specific process of step S5 includes: Joint subject distribution alignment consists of two parts: marginal probability distribution alignment and conditional probability distribution alignment. In the marginal probability distribution alignment part, MMD loss is used to align the distribution of features of source domain samples and target domain samples: Where b s and b t are the number of samples in the source subject set and the target subject set, represents the characteristics of the i-th sample in the source domain of the batch, Represents the features of the jth sample in the target domain within the batch. In addition, the conditional probability distribution is defined to a certain extent by using the category prototype, which promotes the alignment of the conditional probability distribution by reducing the distance between the features of different category samples and their category prototypes. Therefore, the proposed alignment method based on category prototypes within the training batch enhances the sentiment-related properties of the features. First, according to the category label Y cls The samples in each batch are divided into n groups, where n is the number of emotion categories. For each class, the average feature vector of all samples in the class in the batch is calculated. Set a category alignment constraint That is, the distance between each sample in a class and its class prototype should be within a certain range: Where n j represents the number of samples of class j, ||·|| F is the Frobenius norm, and μ represents the boundary parameter of the constraint feature.

7. The cross-subject EEG emotion recognition method based on dynamic domain invariant representation decoupling and reorganization according to claim 1 is characterized in that: The specific process of step S6 includes: For the within-subject invariant features, this representation represents the relatively representative features of each subject. IIF , uses a subject discriminator and a cross entropy loss function to enable it to learn features related to the subject: Where G sub is the subject identity discriminator, for The output features of the first layer in , is the cross entropy loss. In addition, the capture of emotional features is promoted by combining the conditional probability distribution loss based on the category prototype in the subject distribution alignment loss. Combining the above two constraints, the optimization loss of the invariant features within the subject is defined as where f IIF Contains sample features of both source and target objects.

8. The cross-subject EEG emotion recognition method based on dynamic domain invariant representation decoupling and reconstruction according to claim 1 is characterized in that: The specific process of step S7 includes: Inter-subject invariant features represent the invariant features shared between subjects and reflect the more general patterns in the group. Therefore, the complete joint subject distribution alignment loss is used for subject representation optimization.

9. The cross-subject EEG emotion recognition method based on dynamic domain invariant representation decoupling and reorganization according to claim 1 is characterized in that: The specific process of step S8 includes: To ensure that the decoupling process retains as much useful information as possible, we use cross attention to IIF and f MIF Reconstruct the original signal. First, pass f through a linear layer IIF and f MIF Restore to f K The dimension of in m∈{IIF,MIF}, transpose(a,b) is a function that transposes the dimension of a to the dimension of b. Then, we I performs several linear transformations to generate queries (Q), keys (K), and values ​​(V): in and It's Q rec , K rec and V rec Next, the representations are fused through multi-head attention and the high-dimensional features are mapped back to the original spectral feature dimensions. For the previously extracted f IIF and f MIF , connect them together and pass through the encoder E SIFR Further feature extraction is performed to obtain the emotion-related invariant feature f. For feature f, SJDA loss and cross entropy loss are used for constraint and optimization. Therefore, the constraints of the subject-invariant feature reconstruction module are as follows: These components are reorganized into emotion-related categorical features through the subject-invariant feature recombination (SIFR) module.

10. The cross-subject EEG emotion recognition method based on dynamic domain invariant representation decoupling and reconstruction according to claim 1, characterized in that: The specific process of step S9 includes: The overall loss function of this model consists of four parts: cross entropy loss, joint subject distribution alignment loss, subject-invariant feature decoupling loss, and subject-invariant feature reorganization loss. The overall loss function can be expressed as: Where α and β are hyperparameters, which balance the decoupling process through the weights of IIF and MIF. The method achieves a good balance between emotion recognition and cross-subject recognition by jointly optimizing emotion classification loss and cross-subject consistency loss, thereby improving the overall performance of multimodal emotion recognition.

11. The cross-subject EEG emotion recognition method based on dynamic domain invariant representation decoupling and reorganization according to claim 1 is characterized in that: The source domain data contains EEG signals, emotion labels, and subject identity information from multiple subjects, while the target domain data only contains EEG signals and subject identity information. EEG signals are collected by multiple electrodes and cover multiple frequency bands, including the δ band (1-4 Hz), the θ band (4-8 Hz), the α band (8-14 Hz), the β band (14-30 Hz), and the γ band (30-50 Hz). During the training process, unsupervised domain adaptation is performed using the labeled emotion information in the source domain and the unlabeled emotion data in the target domain. This allows the model to fully utilize the labeled information in the source domain while adapting to the data distribution in the target domain to achieve emotion recognition across subjects. In the testing phase, only the target domain data is used for emotion recognition.

12. A cross-subject EEG emotion recognition system based on dynamic domain invariant representation decoupling and reconstruction, characterized by: The system includes a feature extraction module based on frequency-space dual attention, a subject-invariant feature decoupling module based on intra-subject and inter-subject features, a subject-invariant feature recombination module, a cross-subject emotion classification module based on a dual-discriminator architecture, and a subject joint distribution alignment module based on marginal probability and conditional probability alignment. The system and the above modules are obtained by implementing the method according to claim 1; The feature extraction module based on frequency-space dual attention is implemented through S2 and S3, combining frequency domain and spatial domain for joint modeling to capture the dependency of signals in time domain and spatial domain; The subject-invariant feature decoupling module based on intra-subject and inter-subject features is implemented by S4, S6 and S7; The subject-invariant feature reconstruction module is implemented by S8, and the adequacy of decoupling is ensured by constructing the reconstruction loss; The cross-subject emotion classification module based on the dual discriminator architecture is implemented by S9, which inputs the extracted features into the emotion classifier and the subject classifier to perform emotion recognition and subject recognition respectively, and finally achieves cross-subject emotion classification; The subject joint distribution alignment module based on marginal probability and conditional probability alignment is implemented by S5, which optimizes the inter-subject invariant features of the source subject domain and the target subject domain, so that the cross-subject features have higher consistency and recognition.

Images