Methods, systems, devices, and media for EEG emotion recognition combining multi-source domain adaptation and long short-term memory networks.

Abstract

This invention provides a method, system, device, and medium for EEG emotion recognition combining multi-source domain adaptation and long short-term memory networks. The method includes first constructing a common feature extractor using LSTM to extract common features from EEG data, then constructing a domain-specific feature extractor using two LSTM layers connected to a linear layer and an activation function to extract domain-specific features from the EEG data. Simultaneously, the dynamic features of EEG over time are captured during feature extraction. Finally, the extracted common and domain-specific features are fed into a domain classifier for error calculation and classification prediction to obtain the emotion prediction result. The system, device, and medium are used to implement the method for EEG emotion recognition combining multi-source domain adaptation and long short-term memory networks. It features the ability to utilize the temporal characteristics of EEG signals and the ability to predict the target domain using knowledge from multiple source domains, thereby improving the accuracy of classification prediction.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition technology, specifically relating to methods, systems, devices, and media for EEG emotion recognition that combine multi-source domain adaptation and long short-term memory networks. Background Technology

[0002] In recent years, with the development of artificial intelligence, emotion recognition technology has been increasingly applied in real-life scenarios. Currently, there are many methods for emotion recognition, including text recognition, speech recognition, facial expression recognition, and physiological signal recognition. Among these methods, electroencephalogram (EEG) signals are more objective, realistic, and difficult to fake, thus demonstrating significant advantages in accuracy and reliability. EEG signals are electrical activities recorded on the scalp through an electrode array, reflecting the electrical activity of neurons in the brain. However, EEG signals exhibit strong individual variability, leading to significant differences in the distribution of training data and actual application data, thereby limiting the generalization of emotion classifiers across subjects. Due to physiological and cognitive differences between individuals, as well as noise and interference in EEG signals, cross-subject EEG emotion recognition has always been a challenging problem. To address this issue, researchers have proposed cross-subject EEG emotion recognition based on domain adaptation networks. Domain adaptation networks can reduce the differences between the source and target domains in the common space, thereby using labeled source domain data to train a classifier to predict the label of the target domain data. Therefore, applying domain adaptation networks to EEG emotion recognition can greatly enhance the generalization ability of emotion classifiers across different subjects. In recent years, with the rapid rise of artificial intelligence and brain-computer interfaces, more and more research on this topic has emerged. For example, combining Temporal Convolutional Networks (TCNs) with Adversarial Discriminative Domain Adaptation (ADDA) algorithms utilizes TCNs to learn more transferable EEG intrinsic features, eliminates domain transfer between different subjects, and learns dynamic temporal information; introducing dynamic training strategies allows the model to focus on optimizing global domain differences in early training steps, and then gradually switch to local subdomain differences; introducing EEG-Mixup-based data augmentation methods to generate learning samples, and using semi-supervised two-step pairwise learning and semi-supervised multi-domain adaptation methods to align data representations across multiple domains to solve the problem of distribution mismatch, etc. As can be seen from the above, many research papers have proposed different domain adaptation methods, which have achieved certain successes in cross-subject emotion recognition tasks. However, research in this field still faces some challenges, such as: (1) using multilayer perceptrons to construct feature extractors leads to the neglect of temporal relationships and dynamic information in EEG signals; (2) using single-source domain adaptation methods results in models failing to fully utilize knowledge from multiple source domains, leading to insufficient prediction accuracy for the target domain. On the other hand, Long Short-Term Memory (LSTM) networks are recurrent neural network models whose internal control units can learn and memorize long-term dependencies in sequential data, achieving significant success in natural language processing, speech recognition, and time series prediction. However, there is currently a lack of methods that combine multi-source domain adaptation with LTM networks to address cross-conversation and cross-subject EEG emotion recognition problems.Therefore, a new approach is needed to combine multi-source domain adaptation and long short-term memory networks to improve the performance and generalization of cross-session and cross-subject EEG emotion recognition.

[0003] Jin Yiming et al. published EEG-based Emotion Recognition Using Domain Adaptation Network (2017 International Conference on Orange Technologies (ICOT), 222-225). This method uses a single-source domain adaptation model and a traditional multilayer perceptron structural feature extractor. However, due to the difficulty in capturing temporal relationships and the inability to fully utilize knowledge from multiple source domains, the model's classification prediction accuracy is not high enough. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, the present invention aims to provide a method, system, device, and medium for EEG emotion recognition that combines multi-source domain adaptation and long short-term memory networks. First, a single-layer LSTM network is used to construct a common feature extractor to extract common features from multiple source domains, solving the problem that single-source domain models cannot fully utilize knowledge from multiple source domains. Then, a two-layer LSTM network, a fully connected layer, and activation functions are used as N specific domain feature extractors to extract specific domain features. Using an LSTM network to construct the feature extractors can capture the dynamic features of EEG in the time dimension, solving the problem that other non-temporal models ignore the temporal relationship of EEG signals. The extracted common and specific features are then fed into a domain classifier for error calculation and classification prediction. By combining multi-source domain adaptation and long short-term memory networks, the problem of cross-conversation and cross-subject EEG emotion recognition is solved, thus considering the temporal connection of EEG signals. Simultaneously, by utilizing the common and specific features of multiple domains, new technical means and directions are provided for research and application in the field of emotion recognition.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for EEG emotion recognition that combines multi-source domain adaptation and long short-term memory networks includes the following steps:

[0007] Step 1: Set the cycle number;

[0008] Step 2: Use a 1-layer LSTM network as a common feature extractor to extract common features from the EEG data;

[0009] Step 3: Using a 2-layer LSTM network, a fully connected layer, and an activation function as N specific domain feature extractors, specific domain features of the EEG data are extracted from the common EEG feature data extracted in Step 2. The difference between the specific domain features of the source domain and the target domain in the feature space is calculated using the maximum mean difference (mmd loss), where N is the number of source domains.

[0010] Step 4: Construct N softmax classifiers, use the specific domain features of the EEG data extracted in Step 3 to perform emotion classification prediction for each branch, calculate the classification loss (clsloss) for each classification, and reduce the difference in emotion prediction results among the N classifiers using discloss to obtain N convergent emotion prediction results.

[0011] Step 5: Take the average of the N converged sentiment prediction results obtained in Step 4 as the final sentiment prediction result. Finally, obtain the final classification prediction loss L by weighted summation of the maximum mean difference (mmdloss), classification loss (cls loss), and difference loss (disc loss). Update the network weights with the final classification prediction loss L, and repeat Steps 2 to 5 until the round set in Step 1 is reached.

[0012] The LSTM network parameters in step 2 include: the original data input dimension (input_size), the common feature data output dimension (output_size), and the number of LSTM network layers (num_layers = 2). The common feature extractor maps all independent source domain S and target domain T data to a common Hilbert space. Obtain the common characteristics of all data domains.

[0013] The LSTM network parameters mentioned in step 3 include: input_size of common feature data, output_size of specific domain feature data, and num_layers = 1 of LSTM network layers;

[0014] Then, the output dimension output_size of the specific domain feature data is used as input to the fully connected layer. The parameters of the fully connected layer are: Linear(in_features, out_features);

[0015] Finally, the fully connected layer is followed by an activation function, and the activation function parameter used is: LeakyRelu(negative_slope);

[0016] Each source domain and target domain is mapped as a branch to the common Hilbert space. Extract specific features for each pair of source and target domains, and use the maximum mean difference (MMD loss) to calculate the difference between the source and target domains in the feature space:

[0017]

[0018] In the formula, N represents the maximum mean difference between specific domain features of the source and target domains. S N represents the number of source domains. T Let Φ be the number of target domains, and Φ be the mapping function. For the data of the i-th source domain, For the data of the j-th target domain, For Hilbert space.

[0019] Step 4 specifically includes:

[0020] The classification loss (clsloss) is used to calculate the classification loss for each training iteration of the classifier:

[0021]

[0022] In the formula, For classification loss, N is the number of source domains, x is a specific feature of the current source domain, and X is the classification loss. S For other source domain-specific features, The true label for the i-th source domain feature. Let be the predicted label for the i-th source domain feature;

[0023] Use disc loss to reduce the discrepancies in the prediction results of N classifiers:

[0024]

[0025] In the formula, For difference loss, N is the number of source domains, x is a specific feature of the current target domain, and X is the difference loss. t For other target domain-specific features, For the j-th target domain feature, Let be the predicted label for the j-th target domain feature.

[0026] The loss L for the final classification prediction in step 5 is defined as:

[0027]

[0028] In the formula, For the total loss, For classifying losses, For the maximum mean difference, For difference loss, β = 0.01, where epoch is the number of cycles set in step 1.

[0029] Systems for EEG emotion recognition that combine multi-source domain adaptation and long short-term memory networks include:

[0030] The extractor module uses an LSTM network, fully connected layers, and activation functions as extractors to extract features.

[0031] The classifier module constructs a classifier to perform classification prediction for each branch.

[0032] The results output module sets the number of iterations and averages the converged prediction results based on the number of iterations to output the final prediction result.

[0033] EEG emotion recognition devices that combine multi-source domain adaptation and long short-term memory networks include:

[0034] Memory: Used to store the computer program for implementing the EEG emotion recognition method that combines multi-source domain adaptation and long short-term memory networks;

[0035] Processor: Used to implement the EEG emotion recognition method combining multi-source domain adaptation and long short-term memory network when executing the computer program.

[0036] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for EEG emotion recognition combining multi-source domain adaptation and long short-term memory networks.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] 1. The present invention adopts a multi-source domain adaptation network as a whole, constructs a common feature extractor and N specific domain feature extractors to obtain the features of each source domain, thereby enabling the use of knowledge from multiple source domains to predict the target domain, enhancing the generalization of the model and improving the classification prediction accuracy of the model in the target domain;

[0039] 2. This invention uses a 1-layer LSTM as a common feature extractor and a 2-layer LSTM + linear layer + activation function as a domain-specific feature extractor, thereby capturing and utilizing information of EEG signals in the time dimension for classification, further improving the classification accuracy of the model.

[0040] In summary, this invention employs a multi-source domain adaptation network, using a single-layer LSTM as a common feature extractor and two layers of LSTM plus a linear layer plus an activation function to construct N specific domain feature extractors. This approach enables the model to predict the target domain using temporal information from EEG signals and knowledge from multiple source domains, further enhancing the model's generalization ability and improving its classification and prediction accuracy in cross-conversation and cross-subject EEG emotion recognition problems. Attached Figure Description

[0041] Figure 1 This is a model structure diagram of the EEG emotion recognition method combining multi-source domain adaptation and long short-term memory network provided in an embodiment of the present invention.

[0042] Figure 2 This is a flowchart illustrating the steps of an EEG emotion recognition method combining multi-source domain adaptation and long short-term memory networks, as provided in an embodiment of the present invention.

[0043] Figure 3 The flowchart illustrates the algorithm implementation of the EEG emotion recognition method combining multi-source domain adaptation and long short-term memory networks provided in this embodiment of the invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings.

[0045] This embodiment uses leave-one-out cross-validation on the real EEG datasets SEED and SEED-IV to evaluate the model's performance. The overall model structure diagram is as follows. Figure 1 As shown;

[0046] A method for EEG emotion recognition combining multi-source domain adaptation and long short-term memory networks is illustrated in the flowchart below. Figure 2 As shown, it includes the following steps:

[0047] Step 1: Set the cycle number;

[0048] Step 2: Use a 1-layer LSTM network as a common feature extractor to extract common features from the EEG data;

[0049] Step 3: Using a 2-layer LSTM network, a fully connected layer, and an activation function as N specific domain feature extractors, specific domain features of the EEG data are extracted from the common EEG feature data extracted in Step 2. The difference between the specific domain features of the source domain and the target domain in the feature space is calculated using the maximum mean difference (mmd loss), where N is the number of source domains.

[0050] Step 4: Construct N softmax classifiers, use the specific domain features of the EEG data extracted in Step 3 to perform emotion classification prediction for each branch, calculate the classification loss (cls loss) for each classification, and use discloss to reduce the difference in emotion prediction results among the N classifiers, so as to obtain N convergent emotion prediction results.

[0051] Step 5: Take the average of the N converged sentiment prediction results obtained in Step 4 as the final sentiment prediction result. Finally, obtain the final classification prediction loss L by weighted summation of the maximum mean difference (mmd loss), classification loss (cls loss), and difference loss (disc loss). Update the network weights with the final classification prediction loss L, and repeat Steps 2 to 5 until the round set in Step 1 is reached.

[0052] The LSTM network parameters in step 2 include: input size = 310 for the original data input, output size = 64 for the common feature data output, and num_layers = 2 for the LSTM network. The common feature extractor maps all data from the independent source domain S and the target domain T to a common Hilbert space. Obtain the common characteristics of all data domains.

[0053] The LSTM network parameters mentioned in step 3 include: input_size = 64 for common feature data input, output_size = 32 for specific domain feature data output, and num_layers = 1 for the number of LSTM network layers.

[0054] Then, the output dimension output_size=32 of the specific domain feature data is used as input to the fully connected layer. The parameters of the fully connected layer are: Linear(in_features=32, out_features=32);

[0055] Finally, the fully connected layer is followed by an activation function with the following parameters: LeakyRelu(negative_slope = 0.01);

[0056] Each source domain and target domain is mapped as a branch to the common Hilbert space. Extract specific features for each pair of source and target domains, and use the maximum mean difference (MMD loss) to calculate the difference between the source and target domains in the feature space:

[0057]

[0058] In the formula, N represents the maximum mean difference between specific domain features of the source and target domains. S N represents the number of source domains. T Let the number of target domains be 1, and Φ be the mapping function. For the data of the i-th source domain, For the data of the j-th target domain, For Hilbert space;

[0059] Step 4 specifically includes:

[0060] The classification loss (cls loss) is used to calculate the classification loss for each classifier training iteration:

[0061]

[0062] In the formula, For classification loss, N=14 is the number of source domains, x is a specific feature of the current source domain, and X S For other source domain-specific features, The true label for the i-th source domain feature. Let be the predicted label for the i-th source domain feature;

[0063] Use disc loss to reduce the discrepancies in the predictions of N=14 classifiers:

[0064]

[0065] In the formula, For difference loss, N=14 is the number of source domains, x is a specific feature of the current target domain, and X t For other target domain-specific features, For the j-th target domain feature, The predicted label for the j-th target domain feature;

[0066] The loss L for the final classification prediction in step 5 is defined as:

[0067]

[0068] In the formula, For the total loss, For classifying losses, For the maximum mean difference, For difference loss, β = 0.01, where epoch is the number of cycles set in step 1.

[0069] Systems for EEG emotion recognition that combine multi-source domain adaptation and long short-term memory networks include:

[0070] The extractor module uses an LSTM network, a fully connected layer, and an activation function as an extractor to extract features, and is used to implement steps 2 and 3 of the EEG emotion recognition method that combines multi-source domain adaptation and long short-term memory network.

[0071] The classifier module constructs a classifier to perform classification prediction for each branch, which is used to implement step 4 of the EEG emotion recognition method that combines multi-source domain adaptation and long short-term memory network.

[0072] The results output module sets the number of loop rounds and takes the average of the converged prediction results based on the number of rounds as the final prediction result output. This is used to implement steps 1 and 5 of the EEG emotion recognition method that combines multi-source domain adaptation and long short-term memory network.

[0073] EEG emotion recognition devices that combine multi-source domain adaptation and long short-term memory networks include:

[0074] Memory: Used to store the computer program for implementing the EEG emotion recognition method that combines multi-source domain adaptation and long short-term memory networks;

[0075] Processor: Used to implement the EEG emotion recognition method combining multi-source domain adaptation and long short-term memory network when executing the computer program.

[0076] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for EEG emotion recognition combining multi-source domain adaptation and long short-term memory networks.

[0077] Evaluation of the effectiveness of EEG emotion recognition:

[0078] like Figure 3 As shown below, the application effects of this invention will be described in detail with reference to the experimental algorithm flow.

[0079] The real-world examples used the publicly available EEG datasets SEED and SEED-Ⅳ from Shanghai Jiao Tong University. For the SEED dataset, 15 healthy Chinese subjects participated in three identical experiments, one week apart. Each experiment involved watching 15 movie clips, categorized by positive, neutral, and negative emotions. Emotional labels were categorized into three types: negative (-1), neutral (0), and positive (1). Electrodes were placed according to the international 10-20 system, and a 62-channel ESI neuroscanning system was used to capture EEG signals during the experiments. For the SEED-Ⅳ dataset, 15 subjects participated in three experiments, each watching 24 movie clips. However, the selected movies differed across experiments, categorizing them by positive, neutral, fear, and negative emotions. Emotional labels were categorized into four types: sad (0), neutral (1), fear (2), and happy (3). Electrodes were also placed according to the international 10-20 system, and the ESI system was used for capture.

[0080] Table 1 compares the accuracy of the proposed method for cross-subject emotion recognition on the SEED dataset with that of other methods. Table 2 compares the accuracy of the proposed method for cross-conversation and cross-subject emotion recognition on the SEED-Ⅳ dataset with that of other methods. As shown in Table 1, the proposed method achieves an accuracy of 89.78% for cross-conversation recognition and 80.55% for cross-subject recognition on the SEED dataset. As shown in Table 2, the proposed method achieves an accuracy of 63.00% for cross-conversation recognition and 61.33% for cross-subject recognition on the SEED-Ⅳ dataset. Compared to other methods, the proposed method demonstrates improved accuracy in classification prediction.

[0081] Table 1 Comparison of emotion recognition accuracy on the SEED dataset.

[0082] method Cross-session Cross-subjects DDC 81.53 68.99 DAN 79.93 65.84 DCORAL 76.86 66.29 DANN - 79.19 Method of the present invention 89.78 80.55

[0083] Table 2 Comparison of emotion recognition accuracy on the SEED-Ⅳ dataset.

[0084] method Cross-session Cross-subjects DDC 57.63 37.41 DAN 55.14 32.44 DCORAL 44.63 37.43 Method of the present invention 63.00 61.33

[0085] Based on the classification accuracy results on the two real datasets above, the effectiveness of the method of the present invention in cross-subject EEG emotion recognition can be verified.

[0086] The above description is only a preferred embodiment of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for electroencephalogram emotion recognition by combining multi-source domain adaptation and long short-term memory network, characterized in that, The method comprises the following steps: Step 1: setting a cycle round; Step 2: using a 1-layer LSTM network as a public feature extractor to extract public features of the electroencephalogram data; Step 3: using a 2-layer LSTM network, a full connection layer and an activation function as N specific domain feature extractors to extract specific domain features of the electroencephalogram data from the public feature data of the electroencephalogram extracted in step 2, wherein N is the number of source domains, and using a maximum mean difference (mmdloss) to calculate the difference between the specific domain features of the source domain and the target domain in the feature space; Step 4: constructing N softmax classifiers, using the specific domain features of the electroencephalogram data extracted in step 3 to perform emotion classification prediction for each branch, using a classification loss (cls loss) to calculate the classification loss of each time, using a difference loss (disc loss) to reduce the difference between the emotion prediction results of the N classifiers, and obtaining N converged emotion prediction results; Step 5: taking the average of the N converged emotion prediction results obtained in step 4 as the final emotion prediction result, and finally obtaining a final classification prediction loss L by weighted summation of the maximum mean difference (mmdloss), the classification loss (cls loss) and the difference loss (disc loss), updating the network weight by using the final classification prediction loss L, and re-executing steps 2-5 until the round set in step 1 is reached.

2. The method of electroencephalogram emotion recognition using combined multi-source domain adaptation and long short-term memory network according to claim 1, characterized in that, The LSTM network parameters in step 2 include: original data input dimension input_size, public feature data output dimension output_size, LSTM network layer number num_layers = 2, and data of all independent source domains S and data of the target domain T are mapped to a common Hilbert space through the public feature extractor Common features of all data domains are obtained.

3. The method of electroencephalogram emotion recognition using combined multi-source domain adaptation and long short-term memory network according to claim 1, characterized in that, The LSTM network parameters in step 3 include: a public feature data input dimension input_size, a specific domain feature data output dimension output_size, and an LSTM network layer number num_layers=1; The specific domain feature data output dimension output_size is input into a full connection layer again, and the full connection layer parameters used are: Linear(in_features, out_features); Finally, a full connection layer is connected with an activation function, and the activation function parameters used are: LeakyRelu(negative_slope); mapping each source and target domain as a branch to a common hilbert space extracting specific features for each pair of source and target domains, and using maximum mean discrepancy (mmd loss) to compute the difference between source and target domains in the feature space: wherein is the maximum mean difference between the specific domain features of the source and target domains, N S is the number of source domains, N T is the number of target domains, Φ is a mapping function, is the data of the i-th source domain, is the data of the j-th target domain, is a Hilbert space.

4. The method of electroencephalogram emotion recognition using combined multi-source domain adaptation and long short-term memory network according to claim 1, wherein, The step 4 specifically comprises: A classification loss (cls loss) is used to calculate the classification loss of each time of classifier training: wherein, is a classification loss, N is the number of source domains, x is the current source domain-specific feature, X S is other source domain-specific features, is the true label of the i-th source domain-specific feature, is the predicted label of the i-th source domain-specific feature; A difference loss (disc loss) is used to reduce the difference between the prediction results of the N classifiers: In the formula, is a difference loss, N is the number of source domains, x is the current target domain-specific feature, X t is other target domain-specific features, is the true label of the jth target domain feature, is the predicted label of the jth target domain feature.

5. The method of electroencephalogram emotion recognition using combined multi-source domain adaptation and long short-term memory network according to claim 1, wherein, The loss L of the final classification prediction in step 5 is defined as: wherein, is the total loss, is the classification loss, is the maximum mean discrepancy, is the discrepancy loss, β = 0.01, wherein epoch is the number of cycles set in step 1.

6. The electroencephalogram emotion recognition system based on the electroencephalogram emotion recognition method combining multi-source domain adaptation and long short-term memory network according to any one of claims 1-5, characterized in that, It comprises: An extractor module uses an LSTM network, a full connection layer and an activation function as an extractor to perform feature extraction; A classifier module constructs a classifier to perform classification prediction for each branch; A result output module sets a cycle round, and takes the average of the converged prediction result values as the final prediction result output according to the round.

7. A device for electroencephalogram emotion recognition combining multi-source domain adaptation and long short-term memory network, characterized in that, It comprises: A memory: used for storing a computer program for implementing the electroencephalogram emotion recognition method combining multi-source domain adaptation and long short-term memory network as claimed in any one of claims 1-5; A processor: used for implementing the electroencephalogram emotion recognition method combining multi-source domain adaptation and long short-term memory network as claimed in any one of claims 1-5 when the computer program is executed.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the electroencephalogram emotion recognition method combining multi-source domain adaptation and long short-term memory network according to any one of claims 1-5.

Images