An electroencephalogram emotion recognition method based on passive domain adaptation
By using a passive domain adaptation method, pseudo-labels are generated using source domain model parameters and unlabeled target domain data to train the target model, which solves the problems of decreased accuracy and privacy leakage in cross-subject EEG emotion recognition, and achieves efficient emotion recognition and privacy protection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI UNIV OF TECH
- Filing Date
- 2023-03-29
- Publication Date
- 2026-07-07
AI Technical Summary
Existing EEG emotion recognition technology suffers from decreased model accuracy in cross-subject emotion recognition tasks. At the same time, the privacy of EEG data poses a risk of privacy information leakage, making it difficult to improve recognition accuracy while protecting user privacy.
A passive domain adaptation method is adopted, which trains the target model by generating pseudo-labels. Using the source domain model parameters and unlabeled target domain data, differential entropy features are extracted and trained using a multilayer perceptron to generate the optimal target model and achieve emotion recognition.
Without accessing source domain data or features, the model's recognition accuracy is improved, user privacy is protected, and the safe use of EEG signals is achieved, promoting a balanced development between brain science research and personal privacy protection.
Smart Images
Figure CN116327196B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of affective computing, specifically to an EEG emotion recognition method based on passive domain adaptation. Background Technology
[0002] Emotions are a comprehensive expression of human consciousness and behavior. In the transportation sector, emotional stability is crucial for drivers of aircraft, high-speed trains, and long-distance buses. Real-time assessment of a driver's emotional state helps determine if rest is needed, thus ensuring driving safety. Therefore, accurate emotion recognition has significant research and application value in the transportation field. The signals relied upon for emotion recognition can be categorized into non-physiological and physiological signals. Non-physiological signals include facial expressions, voice tone, and body posture, while physiological signals primarily include electroencephalogram (EEG), electrocardiogram (ECG), and electromyogram (EMG) signals. Because non-physiological signals are highly subjective and difficult to accurately reflect emotional states, physiological signals are more commonly used in emotion recognition. Furthermore, research indicates that the areas closely related to emotions are mainly in the cerebral cortex; therefore, EEG-based emotion recognition methods have become mainstream.
[0003] Traditional methods for EEG-based emotion recognition primarily characterize the original physiological signals by manually extracting their time-domain, frequency-domain, and time-frequency-domain features. These features are then classified using traditional machine learning classifiers, such as random forests, support vector machines, and linear discriminant analysis. However, with the rapid development of deep learning and numerous studies demonstrating that deep learning models far outperform traditional machine learning methods in emotion recognition tasks, researchers have increasingly applied deep learning to EEG emotion recognition, particularly convolutional neural networks, which have exhibited superior recognition and classification capabilities.
[0004] However, due to individual differences and the non-stationarity of EEG signals, the data distribution varies among different subjects, leading to a significant decrease in model accuracy in cross-subject emotion recognition tasks. Furthermore, the high cost and inefficient nature of data collection make it impossible to address the issue of data distribution differences among subjects by collecting large amounts of data. Therefore, to improve the generalization ability of models in practical applications, researchers have proposed domain adaptation methods. Following the knowledge transfer paradigm, most of these methods can be categorized into instance-based methods and feature-based methods. The basic idea of instance-based methods is to select source domain data with high similarity to the target domain data for target domain adaptation. Feature-based methods typically construct a feature space or subspace to seek a feature mapping that reduces the difference between the source and target domains.
[0005] However, most existing domain adaptation methods require access to the source domain's EEG data or features. Since EEG data is highly confidential and can reflect a user's personality traits, cognitive abilities, and physical and mental health, if the subject's EEG data is directly shared, attackers can infer the subject's personal privacy information without the subject's consent or knowledge, which could lead to the leakage of the subject's privacy information. Summary of the Invention
[0006] To overcome the shortcomings of existing technologies, this invention proposes an EEG emotion recognition method based on passive domain adaptation, aiming to protect user privacy in emotion recognition tasks across subjects, while also solving the problem of decreased model recognition accuracy due to differences in data distribution among different subjects.
[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:
[0008] The present invention provides a method for EEG emotion recognition based on passive domain adaptation, characterized by the following steps:
[0009] Step 1: Acquire raw EEG signal data from N subjects and perform preprocessing including downsampling, filtering, and segmentation to obtain preprocessed EEG signal data; use the preprocessed EEG signal data from one subject as the target domain data. The preprocessed EEG signal data from the remaining N-1 subjects were used as source domain data. Let source domain data X s The tag is in, Represents source domain data X s The i-th data in Represents target domain data X t The j-th data point in the dataset, where Q represents the number of channels in the EEG signal and P represents the number of sampling points. Represents source domain data X s The i-th data The corresponding labels, where C represents the number of label categories, and n s n t Representing source domain data X respectively s and target domain data X t The number of samples;
[0010] Step 2: Generate the optimal source domain model for the subject.
[0011] Step 2.1: Extract the i-th data using Short Time Fourier Transform. The differential entropy features of several frequency bands in each channel are obtained, and the frequency band features of all channels are concatenated to obtain the i-th data. Differential entropy characteristics E represents the number of features after concatenation;
[0012] Step 2.2, the i-th data Differential entropy characteristics The data is input into a multilayer perceptron (MLP) for processing and outputs the i-th data. Predicted labels
[0013] Based on source domain data X s and its label Y s Based on the predicted labels, the multilayer perceptron (MLP) is trained using the SGD optimizer, and the cross-entropy loss function is minimized until it converges, thus obtaining the optimal source domain model. and the parameters of the trained source domain model
[0014] Step 3: Utilize the trained source domain model parameters And unlabeled target domain data X t To generate the optimal target model for the subjects
[0015] Step 3.1: Use source domain model parameters After initializing another multilayer perceptron (MLP) with the same structure, the target model g is obtained. t The target domain data X is obtained by following step 2.1. t The differential entropy features are input into the target model g. t The data is processed in the hidden layer, and the target domain data X is output by the last layer of the hidden layer. t Hidden features After processing by the softmax function after the output layer, the target domain data X is obtained. t Category probability Where B represents the number of neurons in the last layer of the hidden layer;
[0016] Step 3.2: Transfer the target domain data X t The target model g is trained by dividing it into multiple mini-batches and using batch training. t :
[0017] Step 3.2.1: Transfer the current small batch of target domain data Input into the current mini-batch target model In the middle, the last layer of the hidden layer outputs the current small batch of target domain data. Hidden features The data is then processed by the softmax function after the output layer to obtain the current mini-batch of target domain data. Category probability in, This represents the current small batch of target domain data. The m-th data in the dataset; M represents the target domain data in the current mini-batch. The number of samples;
[0018] Step 3.2.2, using F, And P generate the current small batch of target domain data pseudo-tags in, express pseudo-tags;
[0019] Step 3.2.3, from Obtaining samples Category probability Then according to pseudo-tags The categories, from Obtain the probability value of the same category and use it as the m-th data. weight
[0020] Step 3.2.4: In the current small batch of target domain data and its pseudo-labels Above, the weighted cross-entropy shown in equation (1) is adopted. The target model g is used as a loss function for iterative training. t To obtain the optimal target model for the current small batch.
[0021]
[0022] In equation (1), Indicates pseudo-tags The c-th element of the one-hot encoding, δ c It is the c-th softmax function;
[0023] Step 3.3: Input the target domain data of the next mini-batch into the optimal target model of the current mini-batch. In the process of steps 3.2.1-3.2.3, pseudo-labels and weights are obtained, and then the weighted cross-entropy is calculated. To update the current optimal target model for the mini-batch
[0024] After all mini-batch data has been fed into the target model for training, the optimal target model trained with the last mini-batch of target domain data is taken as the final optimal target model. It is used to achieve the recognition of emotions by the subject's brainwaves.
[0025] The characteristic of the EEG emotion recognition method based on passive domain adaptation described in this invention is that step 3.2.2 is performed as follows:
[0026] Step a, for The distance matrix is obtained by performing matrix multiplication with the transpose of F. Where, the values in the m-th row of D represent The m-th sample With X t n t Distance values between samples;
[0027] Step b: In the m-th row of the distance matrix D, find the k distance values in ascending order and obtain the k distance values in X. t The indices of the corresponding k samples;
[0028] Step c: Obtain the class probability values of the corresponding k samples from P according to the k sample indices, and take the mean of the class probability values of the k samples as the sample. The probability values of each category are then used, and the category corresponding to the highest category probability value is taken as... pseudo-tags
[0029] The present invention provides an electronic device comprising a memory and a processor, wherein the memory is used to store a program supporting the processor in executing the method of claim 1, and the processor is configured to execute the program stored in the memory.
[0030] The present invention provides a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, performs the steps of the method described in claim 1.
[0031] Compared with existing technologies, the beneficial effects of this invention are reflected in:
[0032] 1. This invention adapts to target domain data using only source domain model parameters without accessing source domain data or features. Compared to existing domain-adaptive emotion recognition methods, this invention can protect user privacy, improve the model's emotion recognition accuracy, ensure the security and confidentiality of data during the use of EEG signals, and promote a balanced development of brain science research and personal privacy protection.
[0033] 2. This invention proposes a method for generating pseudo-labels for data. During the training process of the target model, this method can generate pseudo-labels for the target domain data using the model's output. Thus, the model can be trained using pseudo-labels without using any artificial labels for the target domain data. Compared with domain-adaptive emotion recognition methods that require artificial labels for the target domain data, the method of this invention is more practical. Attached Figure Description
[0034] Figure 1 This is a flowchart of the method of the present invention;
[0035] Figure 2 This is a schematic diagram of the method of the present invention;
[0036] Figure 3 This is a flowchart of the EEG signal data processing of the present invention;
[0037] Figure 4 An image showing the accuracy of emotion recognition for 15 subjects in three experiments on the SEED dataset.
[0038] Figure 5 The image shows the accuracy of emotion recognition for 15 subjects in three experiments on the SEED-IV dataset. Detailed Implementation
[0039] In this embodiment, a method for EEG emotion recognition based on passive domain adaptation first preprocesses the raw EEG signal data, dividing the data into a source domain and a target domain. Then, it extracts the differential entropy features of the source domain data, constructs a multilayer perceptron, and trains it on the source domain data to generate a source domain model for the subject. Next, the parameters of the source domain model are transferred to the target model. Finally, the target model is trained using the target domain data and the generated target domain data pseudo-labels, ultimately obtaining the optimal target model. This target model is then used to perform emotion classification on the target domain data, achieving the goal of emotion recognition for the subject. Figure 1 and Figure 2 As shown, specifically, the method is carried out in the following steps:
[0040] Step 1: Obtain raw EEG signal data from N subjects and perform downsampling, filtering, and segmentation preprocessing to obtain preprocessed EEG signal data. In this embodiment, two publicly available datasets were used: SEED and SEED-IV, both established by the laboratory of Shanghai Jiao Tong University. Both the SEED and SEED-IV datasets contain 15 subjects, and the EEG signals related to emotion were recorded by a 62-channel neural scanning system and the eye movement signals were recorded by eye-tracking glasses, respectively, under the induction of 15 movie clips (approximately 4 minutes in length) and 24 movie clips (approximately 2 minutes in length). Each subject underwent three experiments, with each experiment spaced several tens of days apart. The emotion labels in the SEED dataset were divided into three categories: positive, negative, and neutral. The emotion labels in the SEED-IV dataset were... The labels were divided into four categories: happiness, sadness, fear, and neutral. Preprocessing of the raw EEG signal data included: ① downsampling the EEG signals using a sampling rate of 200Hz to improve the signal-to-noise ratio; ② filtering the EEG signals using a bandpass filter of 0-75Hz; ③ segmenting the EEG signal data in the SEED and SEED-IV datasets using non-overlapping 1s and 4s time windows, respectively. After segmentation, each subject's experimental data in the SEED dataset contained 3394 samples (signal duration × number of movie clips) in one experiment, while each subject in the SEED-IV dataset contained 851 samples in the first experiment, 832 samples in the second experiment, and 822 samples in the third experiment. The preprocessed EEG signal data from one of the subjects was used as the target domain data. The preprocessed EEG signal data from the remaining N-1 subjects were used as source domain data. Let source domain data X s The tag is in, Represents source domain data X s The i-th data in Represents target domain data X t The j-th data point in the dataset, where Q represents the number of channels in the EEG signal and P represents the number of sampling points. Represents source domain data X s The i-th data The corresponding labels, where C represents the number of label categories, and n s n t Representing source domain data X respectively s and target domain data X t The number of samples;
[0041] Step 2: Generate the optimal source domain model for the subject.
[0042] Step 2.1: Extract the i-th data using Short Time Fourier Transform. The differential entropy features of several frequency bands in each channel are obtained, and the frequency band features of all channels are concatenated to obtain the i-th data. Differential entropy characteristics E represents the number of features after splicing. In this embodiment, differential entropy features of five frequency bands—δ, θ, α, β, and γ—were extracted. Since the EEG signal data has 62 channels, the number of features E = 62 × 5 = 310. The formula for calculating differential entropy is as follows:
[0043]
[0044] Where p(x) represents the probability density function of continuous information, which approximately follows a Gaussian distribution N(μ,σ) for a given length. 2 The differential entropy of the EEG signal data is:
[0045]
[0046] It is equal to the logarithm of its energy spectrum in a specific frequency band;
[0047] Step 2.2, the i-th data Differential entropy characteristics The data is input into a multilayer perceptron (MLP) for processing and outputs the i-th data. Predicted labels Specifically, such as Figure 3 As shown, a Multilayer Perceptron (MLP) consists of an input layer, hidden layers, and an output layer, containing four hidden layers that process the input data. The feature number E is output sequentially from 310 to 256, 128, 64, and 32. Then, the output layer outputs C values from the 32 feature numbers. Finally, the softmax function is used to transform these C values into probability values corresponding to C types of sentiment. The category corresponding to the highest probability value is the predicted label. The layers of a multilayer perceptron (MLP) are fully connected. A non-linear activation function is added to the output of each hidden layer. Considering the sensitivity of EEG signal data, using the ReLU activation function would result in the loss of a lot of information because values less than 0 are discarded. Therefore, this embodiment uses the LeakyReLU activation function, whose expression is:
[0048]
[0049] Where x is the input value, and α is a constant, which is 0.01 in this example;
[0050] Based on source domain data X s and its label Y sBased on the predicted labels, the multilayer perceptron (MLP) is trained using the SGD optimizer, and the cross-entropy loss function is minimized until it converges, thus obtaining the optimal source domain model. and the parameters of the trained source domain model
[0051] Step 3: Utilize the trained source domain model parameters And unlabeled target domain data X t To generate the optimal target model for the subjects
[0052] Step 3.1: Use source domain model parameters After initializing another multilayer perceptron (MLP) with the same structure, the target model g is obtained. t The target domain data X is obtained by following step 2.1. t The differential entropy features are input into the target model g. t The data is processed in the hidden layer, and the target domain data X is output by the last layer of the hidden layer. t Hidden features The target domain data X is obtained by processing the data through the softmax function after passing through the output layer. t Category probability Where B represents the number of neurons in the last layer of the hidden layer. In this embodiment, B = 32.
[0053] Step 3.2: Transfer the target domain data X t The target model g is trained by dividing it into multiple mini-batches and using batch training. t :
[0054] Step 3.2.1: Transfer the current small batch of target domain data Input into the current mini-batch target model In the middle, the last layer of the hidden layer outputs the current small batch of target domain data. Hidden features After passing through the output layer, the target domain data is obtained by processing it through the softmax function. Category probability in, This represents the current small batch of target domain data. The m-th data in the dataset; M represents the target domain data in the current mini-batch. The number of samples; in this implementation, each small batch has 128 data points, i.e., M = 128;
[0055] Step 3.2.2, using F, And P generate the current small batch of target domain data pseudo-tags in, express pseudo-tags;
[0056] Step 3.2.3, from Get Category probability Then according to pseudo-tags The categories, from Obtain the probability value of the same category and use it as the m-th data. weight
[0057] Step 3.2.4: In the current small batch of target domain data and its pseudo-labels Above, the weighted cross-entropy shown in equation (1) is adopted. The target model g is used as a loss function for iterative training. t To obtain the optimal target model for the current small batch.
[0058]
[0059] in, Indicates pseudo-tags The c-th element of the one-hot encoding, δ c It is the c-th softmax function;
[0060] Step 3.3: Input the target domain data of the next mini-batch into the optimal target model of the current mini-batch. In the process of steps 3.2.1-3.2.3, pseudo-labels and weights are obtained, and then the weighted cross-entropy is calculated. To update the current optimal target model for the mini-batch
[0061] After all mini-batch data has been fed into the target model for training, the optimal target model trained with the last mini-batch of target domain data is taken as the final optimal target model. It is used to achieve the recognition of emotions by the subject's brainwaves.
[0062] In practice, step 3.2.2 is carried out as follows:
[0063] Step a, for The distance matrix is obtained by performing matrix multiplication with the transpose of F. The values in the m-th row of D represent The m-th sample With X t n t Distance values between samples;
[0064] Step b: In the m-th row of the distance matrix D, find the k distance values in ascending order, and obtain the distances of these k distance values in X. t The index of the corresponding k samples, in this embodiment, k = 9;
[0065] Step c: Obtain the class probability values of the k samples from P according to the index of these k samples, and take the mean of the class probability values of the k samples as the sample value. The probability values of each category, and then the samples The category corresponding to the largest category probability value is used as pseudo-tags
[0066] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.
[0067] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.
[0068] The EEG emotion recognition method based on passive domain adaptation of the present invention was tested on the public datasets SEED and SEED-IV. The emotion recognition accuracy on the target domain data was used as the evaluation index. The higher the recognition accuracy, the better the recognition ability of the model. Table 1 shows the average recognition accuracy of 15 subjects in each experiment on the SEED and SEED-IV datasets using the source model and the target model for emotion recognition on their target domain data. The source model refers to the subject's source domain model obtained in step 2, and the target model refers to the subject's target model obtained in step 3.
[0069] Table 1. Average recognition accuracy of 15 subjects in each experiment on the target domain data of the SEED and SEED-IV datasets.
[0070]
[0071] Table 1 shows the experimental results, indicating that the target model significantly improves the average recognition accuracy of the target domain data compared to the source model in both datasets. Furthermore, from... Figure 4 and Figure 5 As can be seen, the accuracy of each subject in both datasets was improved when using the target model compared to using the source model on their target domain data, thus verifying the effectiveness of the method of this invention. Furthermore, this experiment also tested the loss function of equation (1). Whether weighted ablation experiments were performed, the results are shown in Table 2:
[0072] Table 2. Loss functions on the SEED and SEED-IV datasets Weighted ablation test results
[0073]
[0074] The values in Table 2 represent the weighted or unweighted loss function used for 15 subjects in each experiment of the SEED and SEED-IV datasets. The target model was trained, and then the average recognition accuracy of emotion recognition was measured on the target domain data using the target model. The ablation experiment results showed that the recognition effect was better when the loss function was weighted.
[0075] In summary, the EEG emotion recognition method based on passive domain adaptation proposed in this invention uses only source domain model parameters and unlabeled target domain data to build a target model to improve the accuracy of emotion recognition in the target domain. This not only protects user privacy and prevents privacy leaks, but also improves the accuracy of emotion recognition on the SEED and SEED-IV datasets.
Claims
1. A brainwave emotion recognition method based on passive domain adaptation, characterized in that, The procedure is as follows: Step 1: Obtain raw EEG signal data from N subjects and perform preprocessing such as downsampling, filtering, and segmentation to obtain preprocessed EEG signal data; Preprocessed EEG data from one of the subjects was used as the target domain data. The preprocessed EEG signal data from the remaining N-1 subjects were used as source domain data. Let source domain data X s The tag is in, Represents source domain data X s The i-th data in Represents target domain data X t The j-th data point in the dataset, where Q represents the number of channels in the EEG signal and P represents the number of sampling points. Represents source domain data X s The i-th data The corresponding labels, where C represents the number of label categories, and n s n t Representing source domain data X respectively s and target domain data X t The number of samples; Step 2: Generate the optimal source domain model for the subject. Step 2.1: Extract the i-th data using Short Time Fourier Transform. The differential entropy features of several frequency bands in each channel are obtained, and the frequency band features of all channels are concatenated to obtain the i-th data. Differential entropy characteristics E represents the number of features after concatenation; Step 2.2, the i-th data Differential entropy characteristics The data is input into a multilayer perceptron (MLP) for processing and outputs the i-th data. Predicted labels Based on source domain data X s and its label Y s Based on the predicted labels, the multilayer perceptron (MLP) is trained using the SGD optimizer, and the cross-entropy loss function is minimized until it converges, thus obtaining the optimal source domain model. and the parameters of the trained source domain model Step 3: Utilize the trained source domain model parameters And unlabeled target domain data X t To generate the optimal target model for the subjects Step 3.1: Use source domain model parameters After initializing another multilayer perceptron (MLP) with the same structure, the target model g is obtained. t The target domain data X is obtained by following step 2.
1. t The differential entropy features are input into the target model g. t The data is processed in the hidden layer, and the target domain data X is output by the last layer of the hidden layer. t Hidden features After processing by the softmax function after the output layer, the target domain data X is obtained. t Category probability Where B represents the number of neurons in the last layer of the hidden layer; Step 3.2: Transfer the target domain data X t The target model g is trained by dividing it into multiple mini-batches and using batch training. t : Step 3.2.1: Transfer the current small batch of target domain data Input into the current mini-batch target model In the middle, the last layer of the hidden layer outputs the current small batch of target domain data. Hidden features The data is then processed by the softmax function after the output layer to obtain the current mini-batch of target domain data. Category probability in, This represents the current small batch of target domain data. The m-th data in the dataset; M represents the target domain data in the current mini-batch. The number of samples; Step 3.2.2, using F, And P generate the current small batch of target domain data pseudo-tags in, express pseudo-tags; Step 3.2.3, from Obtaining samples Category probability Then according to pseudo-tags The categories, from Obtain the probability value of the same category and use it as the m-th data. weight Step 3.2.4: In the current small batch of target domain data and its pseudo-labels Above, the weighted cross-entropy shown in equation (1) is adopted. The target model g is used as a loss function for iterative training. t To obtain the optimal target model for the current small batch. In equation (1), Indicates pseudo-tags The c-th element of the one-hot encoding, δ c It is the c-th softmax function; Step 3.3: Input the target domain data of the next mini-batch into the optimal target model of the current mini-batch. In the process of steps 3.2.1-3.2.3, pseudo-labels and weights are obtained, and then the weighted cross-entropy is calculated. To update the current optimal target model for the mini-batch After all mini-batch data has been fed into the target model for training, the optimal target model trained with the last mini-batch of target domain data is taken as the final optimal target model. It is used to achieve the recognition of emotions by the subject's brainwaves.
2. The EEG emotion recognition method based on passive domain adaptation according to claim 1, characterized in that, Step 3.2.2 is performed as follows: Step a, for The distance matrix is obtained by performing matrix multiplication with the transpose of F. Where, the values in the m-th row of D represent The m-th sample With X t n t Distance values between samples; Step b: In the m-th row of the distance matrix D, find the k distance values in ascending order and obtain the k distance values in X. t The indices of the corresponding k samples; Step c: Obtain the class probability values of the corresponding k samples from P according to the k sample indices, and take the mean of the class probability values of the k samples as the sample. The probability values of each category are then used, and the category corresponding to the highest category probability value is taken as... pseudo-tags 3. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the EEG emotion recognition method of claim 1 or 2, and the processor is configured to execute the program stored in the memory.
4. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the EEG emotion recognition method according to claim 1 or 2.