Cross-subject and cross-period electroencephalogram emotion recognition model based on sub-domain self-adaption

Abstract

The invention discloses a sub-domain self-adaption-based cross-subject and cross-period electroencephalogram emotion recognition model, and belongs to the technical field of emotion recognition of electroencephalogram signals. The method comprises the following steps: firstly, acquiring domain-invariant shallow feature data through a shared feature extraction module, and inputting the acquired shallow features into a domain specific feature extraction module to extract a plurality of high-level feature representations; then, through an emotion classifier, on one hand, performing emotion prediction on the source domain data, and on the other hand, predicting a pseudo label of a target domain sample for a sub-domain adaptive process; then, a sub-domain adaptation loss PLMMD is adopted to optimize the model, multiple domain invariant representations between a source domain and a target domain are learned, and a hierarchical pseudo-label weight generation module is adopted to relieve pseudo-label noise in the sub-domain adaptation process so as to improve the sub-domain adaptation effect; according to the method, the effect of the electroencephalogram emotion recognition model is obviously improved, and better practicability is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of emotion recognition technology, and specifically relates to cross-subject and cross-period emotion recognition technology of electroencephalogram signals. Background Art

[0002] Emotion recognition (ER) is a growing direction in the field of human-computer interaction, which enables machines to perceive human emotional states, making them more "empathetic" during human-computer interaction. Basically, there are two main ER methods. One is based on non-physiological signals, such as facial expression pictures, body movements, and voice signals. The other is based on physiological signals, such as EEG signals, EMG signals, and ECG signals, which are more objective and less prone to disguise than methods based on non-physiological signals. In recent years, with the rapid development of wearable non-invasive signal acquisition devices and EEG data processing technology, ER based on EEG signals has received increasing attention in academic and commercial fields. However, the non-stationary characteristics of EEG data may lead to differences in data distribution between different subjects and different periods, so it is challenging to use EEG signals to obtain a strong generalization ER model for different subjects and different periods.

[0003] People have tried many different methods to overcome the differences in data distribution of EEG data from different subjects or at different times. A traditional method is to collect and label a large amount of data from new subjects, and then train a subject-specific model. However, this method is time-consuming, has a poor user experience, and the practicality of the model is low. To overcome the differences in feature distribution of EEG data, another method is to use domain adaptation (DA) methods to obtain a general model that is applicable to EEG data from different subjects and at different times. In DA, the test subject is regarded as the target domain and the training subject is regarded as the source domain. DA attempts to narrow the feature distribution between the two domains, thereby improving the performance of the model.

[0004] In order to overcome the feature distribution differences of EEG data, a typical DA technique is the difference-based DA method, which mainly eliminates the difference by minimizing the feature distribution differences between the source domain and the target domain, and uses Multi-Kernel Maximum Mean Discrepancies (MK-MMD) as a measure of domain differences. For example, H. Li et al. applied the Deep Adaptation Network (DAN) to the cross-subject emotion recognition task, showing that DAN can achieve excellent results compared with other baseline methods without domain adaptation. Another commonly used DA technique is the adversarial DA method, which uses domain discriminators and feature generators to alleviate the difference in feature distribution between the source domain and the target domain. For example, S. Liu et al. proposed a multi-branch capsule network based on domain adaptation for cross-subject emotion recognition, using capsule networks to extract discriminative features from EEG and reduce the feature distribution differences of EEG through adversarial domain adaptation.

[0005] Most of the above DA techniques for emotion recognition in EEG data mainly learn global domain shifts, which focus on aligning the global source and target distributions without considering the relationship between subdomains in the two domains (subdomains contain samples in the same category). Therefore, these methods may ignore a lot of fine-grained information of each emotion, which may destroy the discriminative structure in EEG signals.

[0006] Recently, a few researchers tend to focus on subdomain adaptation, that is, adjusting the distribution of EEG data in relevant subdomains in the source and target domains. For example, Z.Li et al. proposed a dynamic domain adaptation algorithm for EEG signals, which alleviated the feature distribution difference of EEG data by minimizing the global difference and local subdomain difference loss, and demonstrated the benefits of reducing the difference from the perspective of fine-grained subdomains. In order to overcome pseudo-label noise, this method allows subdomain alignment to play a major role in the subsequent training process, but if the pseudo-label noise is not processed, the performance of subdomain alignment may be affected. Therefore, it is still challenging to find a subdomain adaptation method that effectively solves domain-invariant emotion representation. Summary of the invention

[0007] The purpose of the present invention is: In response to the above-mentioned problems, the present invention proposes a cross-subject and cross-period EEG emotion recognition model based on unsupervised subdomain adaptation (Unsupervised Subdomain Adaptation Approach Guided by Pseudo Labels, PL-SDA) guided by pseudo labels. Specifically, in order to alleviate the difference in feature distribution of EEG signals, the PL-SDA method performs subdomain alignment (considering the emotion of each sample as a subdomain) through subdomain adaptive loss PLMMD to learn domain-invariant representations. Among them, PL-SDA learns multiple domain-invariant emotion representations to enhance the emotion representation ability of the subdomain adaptation network, because multiple representations can capture more information compared to a single emotion representation. In addition, in order to alleviate the impact of pseudo-label noise on subdomain adaptation, a hierarchical pseudo label weight generation (HPG) method is introduced to improve subdomain adaptation.

[0008] The present invention provides a subdomain-adaptive cross-subject and cross-period EEG signal emotion recognition model, which is obtained by the following steps:

[0009] Step 1: Construct source domain and target domain data, and extract the differential entropy features of the EEG data from the source domain and target domain.

[0010] Step 2: Design a shared feature extractor and input the extracted differential entropy features into the common feature extraction (CFE) module to obtain the shallow domain invariant features of the EEG data.

[0011] Step 3: Design a domain-specific feature extractor and input the shallow features extracted in step 2 into the domain-specific feature extractor (DSFE) to extract multiple high-level representations of EEG data. The multiple high-level representations extracted by DSFE are concatenated into a new vector and input into the following emotion classifier.

[0012] Step 4: Design a sentiment classifier and input the extracted multiple high-level features into the sentiment classifier to obtain the corresponding sentiment prediction value.

[0013] Step 5: Design a pseudo-label weight generation module. In order to overcome the pseudo-label noise in the sub-domain adaptation process, a pseudo-label weight generation module is proposed to adaptively set the pseudo-label weight μ t , reduce the impact of pseudo-label noise of target domain EEG data on sub-domain adaptation process, so as to improve the performance of sub-domain adaptation. And based on the pseudo-label weight μt , we can get the weighted pseudo-label Prepare for the PLMMD loss in the subdomain adaptation process.

[0014] Step 6: Given the high-level representation extracted by DSFE in step 3 and the weighted pseudo-labels, a subdomain adaptive loss (called PLMMD loss) module is proposed to reduce the feature distribution difference of EEG data by fine-grained alignment of a feature distribution of the source domain and the target domain from the emotion subdomain. Since the learned multiple representations may contain more information, while a single representation may miss some important information, multiple emotion representations (i.e., n r Therefore, the present invention proposes a subdomain adaptive loss (defined as L plmmd ), to learn multiple domain-invariant sentiment representations.

[0015] Step 7: Train the subdomain adaptive network, optimize the network model, and solve the network parameters.

[0016] For details of the above steps, please refer to the detailed description of the implementation method.

[0017] Beneficial effects of the present invention:

[0018] The present invention uses subdomain adaptive loss PLMMD to learn multiple domain-invariant representations. At the same time, in order to reduce the pseudo-label noise in the domain adaptation process, a hierarchical pseudo label weight generation (HPG) module is introduced to adaptively set the pseudo-label weights of the target domain EEG data to improve the robustness of subdomain adaptation to pseudo-label noise, obtain robust domain-invariant emotion features, and thus improve the accuracy of cross-domain emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 : is a schematic diagram of the structure of the cross-user and cross-period EEG emotion recognition model based on subdomain adaptation of the present invention; wherein the yellow line represents the training process of the source domain EEG data, and the green line represents the training process of the target domain EEG data;

[0020] Figure 2 Schematic diagram of an example of a hierarchical pseudo-label weight generation (HPG) module for EEG data in the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings.

[0022] The present invention discloses a cross-subject and cross-period EEG signal emotion recognition model based on subdomain adaptation, which mainly includes five parts: a shared feature extractor, a domain-specific feature extractor, a sentiment classifier, a hierarchical pseudo-label weighted weight generation module, and a multi-representation subdomain adaptive loss module. Specifically, 1) First, the source domain and target domain data of the model are input into the shared feature extractor (CFE) module to obtain domain-invariant features. 2) Then, the extracted shared features are input into a domain-specific feature extractor (DSFE) composed of a hybrid structure to extract multiple high-level representations of the EEG signal. 3) Next, the multiple features extracted by the DSFE are input into the sentiment classifier to obtain sentiment prediction. Please note that the sentiment classifier has two purposes: one is to learn the mapping from the source features to their corresponding sentiment label values; the other is to predict the pseudo-labels of the target domain features for use in the subdomain adaptive method. 4) Then, based on the multiple high-level representations extracted from the DSFE, the subdomain adaptive loss (i.e., PLMMD loss) module is used to learn domain-invariant representations from the source domain and the target domain. 5) Finally, the present invention adopts a hierarchical pseudo-label weight generation method to adaptively set the pseudo-label weights to alleviate the impact of pseudo-label noise in sub-domain adaptation and improve the accuracy of emotion recognition.

[0023] The present invention is a subdomain-adaptive cross-subject and cross-period EEG signal emotion recognition model. In this specific implementation, the emotions involved include: negative, neutral, and positive. Figure 1 and Figure 2 , the specific implementation steps are as follows:

[0024] Step 1: Construct source domain and target domain data and extract differential entropy features to prepare for the subdomain adaptive network.

[0025] First, construct the source domain and target domain data. The present invention considers two migration paradigms, namely "cross-subject" and "cross-period". In the cross-subject scenario, it is assumed that there are N user data, one of which is regarded as the target domain, and the remaining user data is regarded as the source domain. In the cross-period scenario, it is assumed that each user has M periods of data, one period of data of a user is regarded as the target domain, and the remaining periods of data of the same user are used as the source domain. The constructed source domain and target domain data are used for subsequent training and testing of the subdomain adaptive network.

[0026] After obtaining the source domain and target domain data, the EEG data can be preprocessed and features can be extracted as shown below.

[0027] The present invention uses EEG data from SEED and SEED-IV public data sets and extracted differential entropy features. In these two data sets, the original EEG data is 64 channels and is downsampled to a sampling rate of 200Hz. In order to filter out noise and artifacts, the EEG data is processed with a bandpass filter from 0.3 to 50Hz. Each channel of the EEG data is divided into 1s cycles of the same length, which do not overlap each other. There are about 3300 clean segments in one experiment.

[0028] Since differential entropy features have the ability to distinguish low-frequency and high-frequency EEG data, the differential entropy (DE) features of EEG data are extracted from the preprocessed EEG data to improve the emotion classification accuracy of the domain adaptation network.

[0029] Specifically, differential entropy features are extracted for each channel in five frequency bands closely related to human physiological activities, where the five frequency bands are (δ: 1-3Hz, θ: 4-7Hz, α: 8-13Hz, β: 14-30Hz, γ: 31-50Hz).

[0030] Assume that the EEG data follows a Gaussian distribution N(μ,σ 2 ),DE features can be extracted by the following formula:

[0031]

[0032] in, μ,σ 2 denote the expectation and variance respectively.

[0033] Step 2: Design a shared feature extractor. The extracted EEG differential entropy features are input into a shared feature extractor composed of a neural network to obtain domain-invariant shallow features.

[0034] The present invention adopts a common feature extractor (CFE) to map the source domain and target domain data to a shared latent space, and then extract the common feature representation of the source domain and the target domain. This module can help extract some shallow domain invariant features. In the present invention, in order to save computing resources, CFE is composed of a three-layer perceptron (MLP). It is worth noting that the features extracted by the shallow neural network are common features, and the migration effect is better. Therefore, in order to extract the shared representation of the source domain and the target domain, the design of CFE is reasonable.

[0035] In this specific implementation, the three-layer perceptron MLP adopted by the shared feature extractor CFE module is a fully connected neural network structure, including three fully connected layers (the input dimension of the first fully connected layer is 64 and the output dimension is 32, the input dimension of the second fully connected layer is 32 and the output dimension is 32, and the input dimension of the third fully connected layer is 32 and the output dimension is 32).

[0036] Step 3: Design a domain-specific feature extractor. In the domain-specific feature extractor, the extracted shared features are sent to a hybrid structure to extract multiple high-level representations of EEG signals.

[0037] After obtaining shallow domain invariant features through CFE, a domain-specific feature extractor (DSFE) composed of a hybrid neural network is used to further extract a variety of different high-level feature representations from the shallow features. Compared with a single representation based on a single structure, the multiple representations obtained by the hybrid structure composed of multiple substructures may contain more EEG signal information to improve the ability of feature representation. Based on multiple representations, the method proposed in the present invention can cover more information by aligning the distribution of multiple representations extracted by the hybrid structure, thereby enhancing the ability to represent emotional features.

[0038] The hybrid neural network structure consists of L different feature extractors, because different neural network structures can extract different representations from low-level features. L feature extractors are used to map source domain and target domain data to L potential feature spaces. Then, multiple high-level features are extracted to obtain more emotion-related information.

[0039] Here, the hybrid neural network structure consists of multiple substructures. In the present invention, since the extracted DE features are used as the input vectors of the network of the present invention, the present invention simplifies this part, and the substructure mainly consists of linear layers. Using multiple representations extracted by DSFE, the source domain is represented as r s,l (l=1,...,L), the target domain is denoted as r t,l (l=1,...,L). The concatenated multi-representation vector is expressed as follows:

[0040] r s =[r s,1 ,...,r s,L ],r t =[r t,1 ,...,r t,L ]

[0041] The concatenated feature vector is input into the subsequent sentiment classifier.

[0042] Assuming that the extracted DE features are used as the input vectors of the network, three substructures based on linear layers (i.e., L=3) are used in the embodiment of the present invention, and in order to construct different substructures, the depth of the linear layer and the number of neurons are different. However, the present invention is not limited to three substructures, and those skilled in the art can set any number of substructures for other applications. (For example, substructure 1: 1 linear layer, 64 input dimensions / neurons and 64 output dimensions; substructure 2: 2 linear layers, 64 input dimensions and 32 output dimensions; substructure 3: 3 linear layers, 64 input dimensions and 32 output dimensions).

[0043] Step 4: Design a sentiment classifier. Input multiple high-level features extracted by DSFE into the sentiment classifier to obtain the corresponding sentiment prediction.

[0044] The sentiment classifier uses the multi-representation features from DSFE to obtain sentiment prediction. In this paper, the classifier consists of a linear layer and a softmax layer. It is worth noting that the classifier has two objectives: one is to learn the mapping from the source embedded features to their corresponding sentiment labels, and the other is to predict the pseudo-labels of the target embedded features, where the pseudo-labels of the target domain can be used in the PLMMD loss function of the subdomain adaptation process.

[0045] In classifier training, cross entropy is chosen to estimate the sentiment classification loss It is expressed as follows

[0046]

[0047] in, represents the i-th source sample, Representation sample The label value of represents the sentiment prediction value obtained through the network, n represents the number of source domain samples, and J(·) represents the cross entropy loss function.

[0048] Step 5: Design a hierarchical pseudo-label selection module. In order to alleviate the impact of pseudo-label noise in the sub-domain adaptation process of EEG data, a hierarchical pseudo-label selection module is proposed to update the pseudo-labels, such as Figure 2 As shown, this improves the performance of domain adaptation.

[0049] In the subdomain adaptation process, the label of the target domain is required, but the present invention considers unsupervised domain adaptation, that is, the target domain has no label. Therefore, a classifier trained in the source domain is used to assign pseudo labels to the target samples. However, when the domain difference is large, the pseudo labels assigned by the source domain classifier to the target samples will inevitably generate noise. Training the network with wrong pseudo labels may lead to error accumulation, thereby affecting the model's ability to classify the sentiment of the target domain samples. In order to reduce the impact of pseudo-label noise, the deep self-training method sets a confidence threshold, selects target samples with high confidence to participate in the training process, and gradually realizes the knowledge transfer process. However, this method is beneficial to samples that are similar to the source domain samples and are easy to transfer, while hard samples with low confidence are difficult to participate in training, which may affect the training performance of the model.

[0050] In order to solve this problem, inspired by the deep self-training method for remote sensing image segmentation proposed by Z.Xi et al., this paper introduces a hierarchical pseudo label weight generation (HPG) method, which does not directly select or ignore candidate pseudo labels based on a confidence threshold, but introduces a weight hierarchical mechanism to adaptively assign weights ρ to pseudo labels. t , while reducing the pseudo-label noise to increase the diversity of samples. Specifically, in the HPG method, simple samples (i.e., reliable pseudo-labels with high confidence) and hard samples (i.e., low-reliability pseudo-labels with medium confidence) are allowed to participate in training, and the hierarchical pseudo-label weights are adaptively assigned to different types of pseudo-labels. In other words, this method enables reliable pseudo-labels to obtain higher weights, low-reliability samples to obtain lower weights, and unreliable samples to be discarded, thereby enhancing pseudo-label diversity while reducing the impact of pseudo-label noise. The HPG method mainly includes three steps: adaptive generation of confidence thresholds, hierarchical weight setting, and pseudo-label weighting.

[0051] Adaptive generation of confidence thresholds. In the HPG method of the present invention, there are firstly two confidence thresholds, namely, a high confidence threshold v h With a low confidence threshold v l , which is used to adaptively select different types of pseudo labels according to the classification confidence. The adaptive generation process of the two confidence thresholds is as follows.

[0052] (1) First, use the trained network model f (including the subdomain adaptive network model composed of CFE, DSFE and classifier) ​​to obtain the classification probability of the target data, and sort the maximum probability of each sample to obtain the confidence list P of the pseudo-label.

[0053] (2) Based on P, obtain the probability value P under a certain proportion αc , to discard pseudo labels with lower probability confidence. The probability value T cur is also the current confidence level, as shown below.

[0054] P c =P[length(P)×α]

[0055] Where length(P) represents the length of the confidence list P, and α is an integer greater than 0 and less than 1. c is a confidence list after selecting length(P)×α from P. It is worth noting that since the model is not stable enough in the early training stage, in order to make the threshold smoother and not affected by model oscillation, the HPG method c Update so that the confidence list depends not only on the input sample of the current iteration, but also on the sample confidence level of the historical iteration. The final probability value list P f It is expressed as follows:

[0056] P f =βP h +(1-β)P c

[0057] Where β is a small positive number used to predict the historical confidence level P. h and the current confidence level P c In this case, P f The highest and lowest values ​​are respectively taken as v h With v l .

[0058] (3) Then, through P f , to obtain the confidence threshold v h With v l , as shown below.

[0059]

[0060] v l =min(P f )

[0061] This formula shows that v l The way to obtain is P f The minimum value min(P f ), v h The way to obtain it is to first obtain the maximum value max(P f ), then max(P f ) applies a function G. Here, the reason for applying G is that in the early training stage, the model is not stable enough to predict each sample with high confidence, resulting in v hThe threshold of is very high, so few samples are selected as easy samples to participate in training. In order to make the model more general, it is necessary to provide more opportunities for target samples and use high confidence threshold v in the training stage. h Therefore, a convex mapping function G(x) is constructed to limit max(P f ). G(x) is a monotonically increasing function with a maximum value of 1, where x represents max(P f ), which is used to ensure that when faced with a large max(P f )(the range is greater than 0.5 and less than or equal to 1) when v h can become sensitive. Finally, after applying function G, v h is an integer ranging from 0.33 to 1.

[0062] Hierarchical pseudo-label weight setting. Then, the present invention is based on the confidence threshold v h With v l To obtain a stratified weight The weight used to determine each pseudo-label is as follows:

[0063]

[0064] in represents the pseudo label weight of the i-th target sample. Represents the soft label of the i-th target sample The maximum probability value in (one-hot encoded vector), Represents the candidate pseudo-label obtained by the sentiment classifier. Parameter v h and v l are two confidence thresholds based on classification confidence, N means greater than v l But less than v h The above formula means that if the class probability Above the threshold v h , indicating that the sample has a high confidence level and is considered an easy sample, and the corresponding pseudo-label weight is set to 1; if the class probability Below the threshold vl, this may be label noise, then the pseudo label weight is set to 0; the category probability Above the threshold v l But below the threshold v h The remaining candidate labels, i.e., hard samples with medium confidence, have their pseudo-label weights determined by their own overall confidence, and the weights are set between 0 and 1.

[0065] Pseudo-label weighting. Finally, based on μ t , we can get the weighted pseudo-label. Represents based on μ tThe weighted pseudo-label of is expressed as follows:

[0066]

[0067] Where * represents the Hadamard product of two vectors (element-wise multiplication), represents the sentiment prediction probability value of the i-th target domain sample, is the pseudo label weight of the i-th target sample. The updated pseudo-label for the i-th sample can be used for PLMMD loss to assign high weights to reliable samples and low weights to unreliable samples, so that target samples with reliable pseudo-labels contribute more to sub-domain adaptation and reduce the contribution of unreliable samples to ensure the performance of sub-domain adaptation.

[0068] Step 6: Design a subdomain adaptive loss module for multiple representations of EEG signals

[0069] Given the high-level representation extracted by DSFE and the weighted pseudo-labels, a subdomain adaptive loss (i.e., PLMMD loss) is proposed to fine-grainedly align the feature distributions of the source and target domains from the emotional subdomain, thereby reducing the feature distribution differences of EEG data. At the same time, since the learned multiple representations may contain more information, while a single representation may miss some important information, multiple emotional representations (i.e., n r Therefore, the present invention proposes a subdomain adaptive loss (called ), to learn multiple domain-invariant sentiment representations.

[0070] To measure the subdomain difference, LMMD is selected to estimate the class-related subdomain difference due to the excellent ability of Local Maximum Mean Discrepancy (LMMD) in estimating the class-related domain difference between probability distributions. To overcome pseudo-label noise, LMMD uses probability prediction values ​​(soft labels) as pseudo-labels and weights the target domain samples according to the probability prediction values. However, soft labels may still contain noise because the model may be unstable in the early training stage and some unreliable samples may still be assigned to high probability predictions and high weights, which may lead to error accumulation. Therefore, in order to overcome the influence of pseudo-label noise when performing subdomain adaptation, a pseudo-label-guided multi-representation subdomain adaptation loss, namely PLMMD, is proposed to measure the subdomain difference in the source and target domain EEG data. Here, PLMMD is guided by the weighted pseudo-labels to determine the contribution of each pseudo-label to the model optimization, that is, to determine that the samples with reliable pseudo-labels in the target samples contribute more, while the target samples with noisy pseudo-labels contribute less. In order to overcome the influence of pseudo-label noise on sub-domain adaptation, the present invention obtains μ by applying the hierarchical pseudo-label weight generation (HPG) method (see step 5). t To improve LMMD, a PLMMD loss is proposed.

[0071] Specifically, the pseudo-label weight μ obtained based on the HPG method t , we can get the weighted pseudo-label based on The subdomain adaptive loss PLMMD for the lth representation in L multiple representations can be expressed as follows:

[0072]

[0073] Where l indicates that the formula is the PLMMD loss of the lth feature representation among the L high-level representations obtained by DSFE. l (X s ,X t ) represents the sub-domain adaptation loss for the l-th representation. represents the reproducing kernel Hillbert space (RKHS) with kernel k, C represents the number of sentiment categories, n represents the number of source domain samples, m represents the number of target domain samples, ‖·‖ 2 represents the square of the element ·. s represents the source domain, D t represents the target domain, which has different users or periods from the source domain. Φ(·) represents the feature map, which maps the original sample to the feature map of RKHS. The kernel k is denoted as k(r s,l ,rt,l )=<Φ(r s,l ),Φ(r t,l )>, <·,·> represents the inner product of vectors, for example Indicates that the source domain samples With the target domain sample After feature mapping by Φ(·), the inner product operation is performed. In addition, and represent the weights of source samples and target samples respectively.

[0074] Among them, the weight of the source sample The calculation formula is:

[0075]

[0076] in, represents the weight of the cth class of the ith source sample. For the ith source sample, the true label value The one-hot encoding of is used to calculate the weight for the i-th source sample

[0077] The weight of the target sample The calculation formula is:

[0078]

[0079] in, Represents the weighted pseudo label for the i-th target sample The cth item of (one-hot encoded vector), Based on The expanded one-hot encoding vector (obtained by the HPG method) has the same length as the sentiment category, and each element value is Represents the weighted pseudo label corresponding to the i-th sample, namely Item c of . Based on The expanded one-hot encoding vector (obtained by the HPG method) has the same length as the sentiment category, and each element value is is the sentiment prediction value obtained by the sentiment classifier, which is also a one-hot encoding vector. * represents the Hadamard product of two vectors. Here, Means: Use stratified weights Original soft tag Weighted to obtain updated pseudo labels Reliable pseudo labels are given higher weights, while unreliable pseudo labels are given lower weights, aiming to make more reliable samples contribute more to model optimization, while the contribution of unreliable samples in model optimization is weakened, thereby reducing the impact of pseudo label noise on the model.

[0080] Based on PLMMD, this paper proposes a subdomain adaptive loss (called ) to learn multiple domain-invariant sentiment representations. Based on the L representations obtained by DSFE, a subdomain adaptive loss function L is used to align the multiple representations of different subdomains in the source and target domains. plmmd as follows:

[0081]

[0082] Wherein, L represents the number of multiple representations obtained by the DSFE module.

[0083] Step 7: Optimization of emotion recognition network model.

[0084] The goal of model optimization is to find the value of θ that minimizes the total loss function, which can be expressed as follows:

[0085]

[0086] where γ is a hyperparameter, is the sentiment classification loss based on cross entropy, is a subdomain adaptive loss using a pseudo-label weighting method, which is a distance metric for different subdomains.

[0087] For the above formula, since the cross entropy loss function is easy to optimize, this paper mainly discusses how to estimate and optimize the loss function in the domain adaptation network When training a model based on this loss function, it is necessary to jointly optimize the pseudo labels and the parameters θ of the neural network, where m represents the number of samples in the target domain and θ represents the parameters to be learned in the neural network. The present invention optimizes these two objectives by performing iterative updates. In each iteration, the present invention first fixes the model parameters θ and uses the fixed network to generate and update the target pseudo labels In obtaining Finally, in order to overcome the pseudo-label noise in the target domain, the present invention weights the pseudo-labels, adaptively weights the pseudo-labels, and obtains weighted pseudo-labels to participate in the subsequent training process. Based on the weighted pseudo-labels, the loss can be estimated. And update the network parameters θ by minimizing the total loss.

[0088] This strategy follows the idea of ​​the Expectation Maximization (EM) algorithm. At the beginning, due to domain shift, the performance of the model on the target domain may be poor. Therefore, based on the pseudo-labeling method of the present invention, some pseudo-labels can participate in the training in the early training stage, because the pseudo-label threshold μ t is relatively small at the beginning, which results in fewer target domain samples to choose from. As training progresses, Optimization, the model can generate relatively reliable pseudo labels on the target samples. More pseudo labels can be selected to estimate And optimize the parameters to get a better model; a better model will generate better pseudo-labels. According to the theory of the EM algorithm, both the pseudo-labels and the model parameters will eventually converge.

[0089] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-subject and cross-period EEG emotion recognition model based on subdomain adaptation, characterized in that: The model is obtained by the following steps s1-s6: S1: Extract differential entropy features of EEG data from source and target domains; S2: Design a shared feature extractor and input the extracted differential entropy features into the shared feature extractor (Common feature extraction, CFE) to obtain domain-invariant shallow features; S3: Design a domain-specific feature extractor and input the extracted shallow features into the domain-specific feature extractor (DSFE) to extract multiple high-level representations of the EEG signal; S4: Design a sentiment classifier, concatenate multiple high-level representations extracted by DSFE into a new vector, and input it into the sentiment classifier to obtain the corresponding sentiment prediction value; S5: Design a hierarchical pseudo label weight generation (HPG) module to generate hierarchical pseudo label weights μ t , reduce the impact of pseudo-label noise of target domain EEG data on subdomain adaptation to improve the performance of subdomain adaptation, based on the weight μ t , get the weighted pseudo-label And apply it to the subdomain adaptation loss in step S6; S6: Combine the high-level representation extracted by DSFE in step S3 with the weighted pseudo-label A subdomain adaptive loss PLMMD is proposed. On this basis, a subdomain adaptive loss function for multiple high-level representations of EEG signals is designed, which is defined as Fine-grained alignment of the feature distributions of the source and target domains is performed through the emotion subdomain to reduce the feature distribution differences of EEG data; S7: Train and optimize the model to solve the adaptive network parameters.

2. According to claim 1, a subdomain-adaptive cross-subject and cross-period EEG signal emotion recognition model, characterized in that: The implementation of extracting differential entropy features of EEG data in S1 includes: In the five frequency bands closely related to human physiological activities, the differential entropy features are extracted for each channel, where the five frequency bands are (δ: 1-3Hz, θ: 4-7Hz, α: 8-13Hz, β: 14-30Hz, γ: 31-50Hz); Assume that the EEG data follows a Gaussian distribution N(μ,σ 2 ), the differential entropy feature DE can be extracted by the following formula:

3. According to the subdomain adaptive cross-subject and cross-period EEG emotion recognition model of claim 1, it is characterized in that: The shared feature extractor in S2 includes three fully connected layers, which maps the source domain and target domain data into a shared latent space and then extracts common feature representations of the source domain and the target domain.

4. The cross-subject and cross-period EEG emotion recognition model based on subdomain adaptation according to claim 1, characterized in that: The domain-specific feature extractor in S3 adopts a hybrid neural network structure, which is composed of L different feature extractors, and the L feature extractors are used to map the source domain and target domain data to L potential feature spaces; Then, multiple high-level features are extracted to obtain more emotion-related information, and multiple high-level features are concatenated into a new vector as the input of the subsequent classifier.

5. The subdomain-adaptive cross-subject and cross-period EEG emotion recognition model according to claim 1, characterized in that: The sentiment classifier in S4 consists of a linear layer and a softmax layer. The sentiment classifier has two goals: one is to learn the mapping from the source domain embedded features to their corresponding sentiment labels, and the other is to predict the original pseudo labels of the target domain embedded features, where the pseudo labels can be used in the hierarchical pseudo label weight generation module; In the training of this sentiment classifier, cross entropy is chosen to estimate the classification loss, which is expressed as follows in, represents the i-th source sample, Representation sample The label value of represents the sentiment prediction value obtained through the network, n represents the number of source domain samples, and J(·) represents the cross entropy loss function.

6. The subdomain-adaptive cross-subject and cross-period EEG emotion recognition model according to claim 1, characterized in that: In S5, a hierarchical pseudo-label weight μ t The setting method is as follows: in represents the pseudo label weight of the i-th target sample, Represents the soft label of the i-th target sample The maximum probability value in (one-hot encoded vector), represents the candidate pseudo-label obtained by the sentiment classifier, and the parameter v h and v l are two confidence thresholds based on classification confidence, N means greater than v l But less than v h The above formula shows that if the class probability Above the threshold v h , indicating that the sample has a high confidence level and is considered an easy sample. The corresponding pseudo-label weight is set to 1. The probability is lower than the threshold v l , which may be label noise, then the pseudo label weight is set to 0; the category probability Above the threshold v l But below the threshold v h The remaining candidate labels are hard samples with medium confidence. The pseudo-label weights are determined by their own overall confidence, and the weights are set between 0 and 1. Finally, based on The weighted pseudo-label can be obtained; let Indicates based on The weighted pseudo-label of is expressed as follows: Where * represents the Hadamard product of two vectors (element-wise multiplication), represents the sentiment prediction probability value of the i-th target domain sample, is the pseudo label weight of the i-th target sample, The updated pseudo-label for the i-th sample can be used for PLMMD loss to assign high weights to reliable samples and low weights to unreliable samples, so that target samples with reliable pseudo-labels contribute more to sub-domain adaptation and reduce the contribution of unreliable samples to ensure the performance of sub-domain adaptation.

7. The subdomain-adaptive cross-subject and cross-period EEG emotion recognition model according to claim 6, characterized in that: The threshold parameter v h With v l The adaptive generation process of is as follows: (1) First, obtain the classification probability of the target data and sort the maximum probability of each sample to obtain the confidence list P of the pseudo-label; (2) Based on P, obtain the probability value P under a certain proportion α c , to discard pseudo labels with lower probability confidence, as shown below. P c =P[length(P)×α] Among them, length(P) represents the length of the confidence list P, α is an integer greater than 0 and less than 1, P c is the confidence list after selecting length(P)×α ratio from P, which is the current confidence level; since the model is not stable enough in the early training stage, in order to make the threshold smoother and not affected by model oscillation, P c Update so that the confidence list depends not only on the input sample of the current iteration, but also on the sample confidence level of the historical iteration. The final probability value list P f It is expressed as follows: P f =βP h +(1-β)P c Where β is a positive number used to predict the historical confidence level P. h and the current confidence level P c trade-offs between; (3) Then, through P f Get the confidence threshold v h With v l , as shown below: v l =main(P f ) This formula shows that v l The way to obtain is P f The minimum value of main(P f ), v h The way to obtain it is to first obtain the maximum value max(P f ), then max(P f ) applies a function G(x). After applying the function G, v h The range is approximately from 0.33 to 1.

8. The cross-subject and cross-period EEG emotion recognition model based on subdomain adaptation according to claim 1 or 7, characterized in that: The implementation of S6 includes: Based on weighted pseudo-labels The subdomain adaptive loss function PLMMD for the lth representation in L multiple representations is designed as follows: PLMMD l (X s , X t ) represents the sub-domain adaptation loss for the lth representation, represents the reproducing kernel Hilbert space with kernel k, C represents the number of sentiment categories, n represents the number of source domain samples, m represents the number of target domain samples, ||·|| 2 represents the square of the element ·, D s represents the source domain, D t represents the target domain, Φ(·) represents the feature map, that is, the original sample is mapped to the feature map of RKHS, and the kernel k is represented by k(r s，l , r t，l )=<Φ(r s，l ), Φ(r t，l )>, <·, ·> represents the inner product of vectors, and Represent the weights of the cth class of the i-th sample in the source domain and the i-th sample in the target domain respectively; Among them, the weight of the i-th sample in the source domain is The calculation formula is: in, Represents the weight of the cth class of the ith source sample, using the true label value for the ith source sample The one-hot encoding is used to calculate the weight for the i-th source sample The subscript c indicates the cth class; The weight of the i-th target sample The calculation formula is: in, Represents the weighted pseudo label for the i-th target sample The cth item of (one-hot encoded vector), Based on The expanded one-hot encoding vector has the same length as the sentiment category, and each element value is Represents the weighted pseudo label corresponding to the i-th sample, namely Item c of Based on The expanded one-hot encoding vector has the same length as the sentiment category, and each element value is is the sentiment prediction value obtained by the sentiment classifier, which is also a one-hot encoding vector. * represents the Hadamard product of two vectors. Representation: Using stratified weights Original soft tag Weighted to obtain updated pseudo labels Reliable pseudo-labels are assigned higher weights, while unreliable pseudo-labels are assigned lower weights, so that more reliable samples contribute more to model optimization, while the contribution of unreliable samples in model optimization is weakened, thereby reducing the impact of pseudo-label noise on the model.

9. The cross-subject and cross-period EEG emotion recognition model based on subdomain adaptation according to claim 8, characterized in that: The implementation of S6 also includes: designing a subdomain adaptive loss for multiple high-level representations of EEG signals, defined as To learn multiple domain-invariant sentiment representations; based on the L representations obtained by DSFE, a subdomain adaptive loss function is used to align the multiple representations of different subdomains in the source domain and the target domain as follows: Wherein, L represents the number of multiple representations obtained by the DSFE module.

10. The subdomain-adaptive cross-subject and cross-period EEG emotion recognition model according to claim 9, characterized in that: The optimization of the model is to find the network parameters θ that minimize the total loss function, which is expressed as follows: where γ is a hyperparameter, is the sentiment classification loss based on cross entropy, It is a subdomain adaptation loss using multiple high-level representations guided by pseudo labels, and is a distance measure for different subdomains of EEG data.

Images