Electroencephalogram emotion recognition method based on double-branch damaged channel feature reconstruction model

Abstract

The invention discloses an electroencephalogram emotion recognition method based on a double-branch damaged channel feature reconstruction model. The method comprises the steps of preprocessing a data set, dividing a training set and a test set, extracting dynamic random mask channel features, constructing a double-branch pre-training network, training the double-branch pre-training network, constructing a double-branch fine-tuning network, training the double-branch fine-tuning network and testing the double-branch fine-tuning network. According to the method, dynamic random mask features are extracted, damaged electroencephalogram channel features in a real environment are simulated, time sequence features, spatial features and emotion features are extracted from the electroencephalogram channel features after mask input, and the simulated damaged electroencephalogram channel features are input into a trained fine tuning network to obtain an emotion classification prediction result. The technical problem that in the prior art, random mask channels are not considered, and electroencephalogram channel characteristics of masks are reconstructed from the two dimensions of time and space is solved. The method has the advantages of high robustness, high recognition accuracy and the like, and can be applied to the technical field of emotion recognition from the damaged electroencephalogram channel.

Description

Technical Field

[0001] The present invention belongs to the field of computer application technology, and in particular relates to an electroencephalogram emotion recognition method based on a double-branch damaged channel feature reconstruction model. Background Art

[0002] EEG emotion recognition has attracted attention in human-computer interaction due to its high accuracy and unpretendability, and EEG emotion recognition is currently an important research in this field. The goal of EEG emotion recognition is to identify the emotion type of the EEG signal based on the input EEG signal, which has a wide range of application value. Existing methods mainly focus on experiments within subjects. Since EEG signals vary significantly between individuals, there are certain limitations and flexibility in experiments. The discussion on the channel loss caused by interference when collecting EEG signals in reality is not comprehensive enough. How to design a model that can simulate the channel damage and extract generalized features between subjects has become a key technical issue in improving EEG emotion recognition.

[0003] Existing technologies mainly include two categories: cross-subject domain generalization and mask channel. Cross-subject domain generalization is to use only source subject data in the model training stage to improve the model's adaptability to the unseen domain (target domain). Existing methods usually use conventional neural network models to conduct simple cross-subject experiments. However, due to differences between individuals, some conventional neural network models do not consider how to reduce the significant differences between the source domain and the target domain. In order to reduce the differences, the domain adaptation problem in transfer learning has been widely used. Gong et al. proposed the CiABL model in "CiABL: Completeness-induced Adaptive Broad Learning for Cross-Subject Emotion Recognition with EEG and Eye Movement Signals". The model minimizes inter-subject differences by aligning marginal distributions and conditional distributions to improve cross-subject emotion recognition performance.

[0004] Liu et al. constructed the DA-CapsNet method in "DA-CapsNet: A multi-branch capsule network based on adversarial domain adaption for cross-subject EEG emotion recognition", mapped the differential entropy features into a two-dimensional matrix to form a homomorphic difference cubic sequence, and constructed a multi-branch parallel main capsule network. An adversarial domain adaptation mechanism was also added to reduce the distribution difference between domains. Although these methods can reduce the difference between the source domain and the target domain, they will over-rely on the target domain data and still have limitations when processing new target domains.

[0005] In recent years, mask channel methods have been introduced for EEG data, masking EEG channels with fixed ratios to simulate channel damage in real environments. Most methods are based on prior knowledge to judge EEG signals. These methods basically mask some fixed EEG channels, which are difficult to adapt to the situation in real environments where damaged channels are inconsistent with fixed masked channels, and lack flexibility. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned technology and provide an EEG emotion recognition method based on a double-branch damaged channel feature reconstruction model with good generalization characteristics, strong robustness and high recognition accuracy.

[0007] The technical solution adopted to solve the above technical problems is:

[0008] (1) Dataset preprocessing

[0009] Fifteen subjects were selected from the SEED dataset, which contains three emotion categories: positive emotion p, neutral emotion u, and negative emotion q. The differential entropy features of EEG signals in five frequency bands, δ of 1-3Hz, θ of 4-7Hz, α of 8-13Hz, β of 14-30Hz, and γ of 31-50Hz, were obtained as input. The EEG signals in the five frequency bands were divided into time series segments according to a 1-second sliding window with a fixed time step of 30 seconds.

[0010] (2) Divide the training set and test set

[0011] The EEG features and corresponding labels after the masked channel were cross-validated according to the leave-one-out cross-validation of the subjects. Each time, the data of one subject was selected as the test set, and the data of the remaining 14 subjects were used as the training set.

[0012] (3) Extract dynamic random mask channel features

[0013] According to the international 10-20 standard EEG channels, for the 62 EEG channels of the SEED dataset, the mask ratios were selected as 10%, 20%, and 30%, and 6, 12, and 18 EEG channels were randomly masked in each batch. The differential entropy features after masking were extracted to simulate the channel damage in the real environment. The masked matrix X was determined as follows: m :

[0014] X m =X⊙(1-M)

[0015] Among them, M represents a mask matrix with the same dimension as the input data X, whose value is 0 or 1, indicating the masked matrix, ⊙ represents the dot product, and 1-M represents the unmasked matrix.

[0016] (4) Constructing a dual-branch pre-training network

[0017] The dual-branch pre-trained network consists of a temporal branch network and a spatial branch network in parallel.

[0018] The timing branch network is composed of an LSTM encoder 1, an LSTM encoder 2, 28 LSTM decoders, a gradient flip layer 1, a main discriminator 1, and a mixed layer 1. One output end of the LSTM encoder 1 is connected to the main discriminator 1 through the gradient flip layer 1, and the other output end of the LSTM encoder 1 is connected to the input end of the parallel LSTM decoder 1-LSTM decoder 14. The output end of the parallel LSTM decoder 1-LSTM decoder 14 is connected to the input end of the mixed layer 1, and the output end of the mixed layer 1 is connected to the input end of the parallel LSTM decoder 15-LSTM decoder 28 through the LSTM encoder 2.

[0019] The spatial branch network is composed of a GCN encoder 1, a GCN encoder 2, 28 GCN decoders, a gradient flip layer 2, a main discriminator 2, and a mixing layer 2. One output end of the GCN encoder 1 is connected to the main discriminator 2 through the gradient flip layer 2, and the other output end of the GCN encoder 1 is connected to the input end of the parallel GCN decoder 1-GCN decoder 14. The output end of the parallel GCN decoder 1-GCN decoder 14 is connected to the input end of the mixing layer 2, and the output end of the mixing layer 2 is connected to the input end of the parallel GCN decoder 15-GCN decoder 28 through the GCN encoder 2.

[0020] (5) Training the dual-branch pre-trained network

[0021] 1) Construct loss function

[0022] The loss function L includes the temporal branch feature reconstruction loss function l trec , spatial branch feature reconstruction loss function l srec, temporal branch adversarial loss function l t , spatial branch adversarial loss function l s .

[0023] The temporal branch feature reconstruction loss function l is constructed as follows: trec :

[0024]

[0025] Where i∈{1,2,…,n}, n is the number of source subjects, which is 14. ti is the output of the ith decoder in the sequential branch, x i is the differential entropy feature of the i-th source subject, and MSE represents the mean square error.

[0026] The spatial branch feature reconstruction loss function l is constructed as follows: srec :

[0027]

[0028] Among them, O si is the output of the i-th decoder in the spatial branch.

[0029] The temporal branch adversarial loss function l is constructed as follows: t :

[0030]

[0031] in, The probability that the feature belongs to a source subject is predicted by subject discriminator 1, is the true domain label to which the i-th sample belongs.

[0032] Construct the spatial branch adversarial loss function l as follows s :

[0033]

[0034] in, The probability that the feature belongs to a source subject is predicted by subject discriminator 2, is the true domain label to which the i-th sample belongs.

[0035] 2) Training a dual-branch pre-trained network

[0036] The training set was input into the two-branch pre-trained network for training. The training parameters were: learning rate was 0.003, weight decay rate was 0.0005, and Adam optimizer was used. It was run on NVIDIAA40 GPUs for 150 iterations until the loss function converged.

[0037] (6) Constructing a dual-branch fine-tuning network

[0038] The dual-branch fine-tuning network is composed of an LSTM encoder 3, a GCN encoder 3, a feature fusion layer, and a sentiment classifier. The output ends of the LSTM encoder 3 and the GCN encoder 3 are connected to the input end of the feature fusion layer, and the output end of the feature fusion layer is connected to the sentiment classifier. The structure of the GCN encoder 3 is the same as that of the GCN encoder 1.

[0039] (7) Training the dual-branch fine-tuning network

[0040] 1) Constructing the cross entropy loss function

[0041] The cross entropy loss function l is constructed as follows cls :

[0042]

[0043] Among them, y j is the real emotion category, is the predicted emotion category, j represents the three emotion categories of positive emotion p, neutral emotion u and negative emotion q, j∈{1,2,3}.

[0044] 2) Training a dual-branch fine-tuning network

[0045] The training set is input into the two-branch fine-tuning network for training. The training parameters are: learning rate is 0.003, weight decay rate is 0.0005, Adam optimizer is used, and it is iterated 300 times until the cross entropy loss function converges.

[0046] (8) Test the dual-branch fine-tuning network

[0047] The test set is input into the trained two-branch fine-tuning network for testing, and the predicted emotion category is output.

[0048] In the step (4) of constructing the pre-training model of the present invention, the GCN encoder 1 is composed of a graph convolution layer 1, a batch normalization layer 1, a graph convolution layer 2, and a batch normalization layer 2 connected in series in sequence; the structure of the GCN encoder 2 is the same as that of the GCN encoder 1.

[0049] In the step (4) of constructing the pre-training model of the present invention, the GCN decoder 1 is composed of a graph convolution layer 1, a batch normalization layer 3, a linear layer 1, a graph convolution layer 4, a batch normalization layer 4, and a linear layer 2, the graph convolution layer 1 is connected in series with the batch normalization layer 3, the linear layer 1, the graph convolution layer 4, the batch normalization layer 4, and the linear layer 2, and the other output end of the linear layer 1 is connected to the other input end of the linear layer 2; the structure of the GCN decoder 2 to the GCN decoder 28 is the same as that of the GCN decoder 1.

[0050] In step (4) of the present invention, in constructing a dual-branch pre-training network, the graph convolution layer is constructed as follows:

[0051] Construct the graph convolution layer O according to formula (1) g :

[0052] O g =S×(X×W)+b (1)

[0053]

[0054]

[0055] Where k,l∈{1,2,...,N}, N is 62, P k and P l Represent two EEG channels, ‖P k -P l ‖2 is the Euclidean norm between two EEG channels; A k,l is the obtained adjacency matrix, is the square of the three-dimensional distance between nodes k and l, λ is the sparsity factor, λ takes values ​​of 1 to 10, and the sparse graph matrix S is obtained; W is the weight parameter; b is the bias, b∈[-0.125,0.125].

[0056] In the method for constructing a graph convolutional layer of the present invention, λ is a sparse factor, and the optimal value of λ is 8; b is a bias, and the optimal value of b is 0.1.

[0057] In step (4) of the present invention, in constructing a dual-branch pre-trained network, the construction method of the GCN decoder 1 is as follows:

[0058] Construct GCN decoder 1 as follows:

[0059] h s =Linear(h s0 +D(X,S)

[0060] Among them, h s0 is an initial all-zero hidden state tensor, D is the graph convolution operation, and h s is the output of the current decoder 1, and Linear is the linear layer.

[0061] Compared with the prior art, the present invention has the following advantages:

[0062] Since the present invention constructs a method for extracting dynamic random mask features, the EEG channel features are randomly masked according to a certain ratio to simulate the damaged EEG channel features in a real environment. In the pre-training stage, the temporal features are extracted from the temporal branches by inputting the masked EEG channel features, and the GCN encoder and the GCN decoder are used to further obtain the spatial features on the basis of extracting the temporal features, which solves the problem of learning and reconstructing generalized spatiotemporal features from the masked EEG channel features. In the step of testing the dual-branch fine-tuning network, the test set is input into the trained dual-branch fine-tuning network to extract effective generalized features and output the predicted emotion category. The present invention has the advantages of simulating real damaged EEG channel features, extracting good generalized features, strong robustness, high recognition accuracy, etc., and can be applied to the technical field of extracting generalized spatiotemporal features from damaged EEG channel features and recognizing emotions. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of embodiment 1 of the present invention.

[0064] Figure 2 It is a structural diagram of the dual-branch pre-training network.

[0065] Figure 3 yes Figure 2 Schematic diagram of the structure of the sequential branch network.

[0066] Figure 4 yes Figure 2 Schematic diagram of the structure of the medium-space branch network.

[0067] Figure 5 yes Figure 4 Schematic diagram of the structure of GCN encoder 1

[0068] Figure 6 yes Figure 4 Schematic diagram of the structure of GCN decoder 1.

[0069] Figure 7 It is a structural diagram of the fine-tuning network. DETAILED DESCRIPTION

[0070] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples, but the present invention is not limited to the following embodiments.

[0071] Example 1

[0072] The EEG emotion recognition method based on the dual-branch damaged channel feature reconstruction model of this embodiment consists of the following steps (see Figure 1 ):

[0073] (1) Dataset preprocessing

[0074] Fifteen subjects were selected from the SEED dataset, which contains three emotion categories: positive emotion p, neutral emotion u, and negative emotion q. The differential entropy features of EEG signals in five frequency bands, δ of 1-3Hz, θ of 4-7Hz, α of 8-13Hz, β of 14-30Hz, and γ of 31-50Hz, were obtained as input. The time series were divided into segments according to a 1-second sliding window with a fixed time step of 30 seconds.

[0075] (2) Divide the training set and test set

[0076] The EEG features and corresponding labels after the masked channel were cross-validated according to the leave-one-out cross-validation of the subjects. Each time, the data of one subject was selected as the test set, and the data of the remaining 14 subjects were used as the training set.

[0077] (3) Extract dynamic random mask channel features

[0078] According to the international 10-20 standard EEG channels, for the 62 EEG channels of the SEED dataset, the mask ratios were selected as 10%, 20%, and 30%, and 6, 12, and 18 EEG channels were randomly masked in each batch. The differential entropy features after masking were extracted to simulate the channel damage in the real environment. The masked matrix X was determined as follows: m :

[0079] X m =X⊙(1-M)

[0080] Among them, M represents a mask matrix with the same dimension as the input data X, whose value is 0 or 1, indicating the masked matrix, ⊙ represents the dot product, and 1-M represents the unmasked matrix.

[0081] (4) Constructing a dual-branch pre-training network

[0082] Figure 2 A schematic diagram of the structure of the dual-branch pre-training network of this embodiment is given. Figure 2 In the embodiment, the dual-branch pre-training network is composed of a temporal branch network and a spatial branch network in parallel.

[0083] Figure 3 Given Figure 2 Schematic diagram of the structure of the sequential branch network. Figure 3In the embodiment, the temporal branch network is composed of an LSTM encoder 1, an LSTM encoder 2, 28 LSTM decoders, a gradient flipping layer 1, a main discriminator 1, and a mixing layer 1. One output end of the LSTM encoder 1 is connected to the main discriminator 1 through the gradient flipping layer 1, and the other output end of the LSTM encoder 1 is connected to the input end of the parallel LSTM decoder 1-LSTM decoder 14. The output end of the parallel LSTM decoder 1-LSTM decoder 14 is connected to the input end of the mixing layer 1, and the output end of the mixing layer 1 is connected to the input end of the parallel LSTM decoder 15-LSTM decoder 28 through the LSTM encoder 2.

[0084] Figure 4 Given Figure 2 Schematic diagram of the structure of the spatial branch network. Figure 4 In the figure, the spatial branch network of this embodiment is composed of a GCN encoder 1, a GCN encoder 2, 28 GCN decoders, a gradient flipping layer 2, a main discriminator 2, and a mixing layer 2. One output end of the GCN encoder 1 is connected to the main discriminator 2 through the gradient flipping layer 2, and the other output end of the GCN encoder 1 is connected to the input end of the parallel GCN decoder 1-GCN decoder 14. The output end of the parallel GCN decoder 1-GCN decoder 14 is connected to the input end of the mixing layer 2, and the output end of the mixing layer 2 is connected to the input end of the parallel GCN decoder 15-GCN decoder 28 through the GCN encoder 2.

[0085] Figure 5 Given Figure 4 Schematic diagram of the structure of GCN encoder 1. Figure 5 In the embodiment, the GCN encoder 1 is composed of a graph convolution layer 1, a batch normalization layer 1, a graph convolution layer 2, and a batch normalization layer 2 connected in series in sequence. The structure of the GCN encoder 2 is the same as that of the GCN encoder 1.

[0086] The construction method of the graph convolution layer 1 of this embodiment is as follows:

[0087] Construct the graph convolution layer O according to formula (1) g :

[0088] O g =S×(X×W)+b (1)

[0089]

[0090]

[0091] Where k,l∈{1,2,...,N}, N is 62, P k and P l Represent two EEG channels, ‖Pk -P l ‖2 is the Euclidean norm between two EEG channels; A k,l is the obtained adjacency matrix, is the square of the three-dimensional distance between nodes k and l, λ is the sparse factor, λ is 1 to 10, and λ in this embodiment is 8, and the sparse graph matrix S is obtained, W is the weight parameter, b is the bias, b∈[-0.125,0.125], and b in this embodiment is 0.1. The construction method of graph convolution layer 2 is the same as that of graph convolution layer 1.

[0092] Figure 6 Given Figure 4 Schematic diagram of the structure of GCN decoder 1. Figure 6 In the embodiment, the GCN decoder 1 is composed of a graph convolution layer 3, a batch normalization layer 3, a linear layer 1, a graph convolution layer 4, a batch normalization layer 4, and a linear layer 2. The graph convolution layer 3 is connected in series with the batch normalization layer 3, the linear layer 1, the graph convolution layer 4, the batch normalization layer 4, and the linear layer 2. The other output end of the linear layer 1 is connected to the other input end of the linear layer 2. The structures of GCN decoders 2 to 28 are the same as those of the GCN decoder 1.

[0093] The construction method of the GCN decoder 1 of this embodiment is as follows:

[0094] Construct GCN decoder 1 as follows:

[0095] h s =Linear(h s0 +D(X,S)

[0096] Among them, h s0 is an initial all-zero hidden state tensor, D is the graph convolution operation, and h s is the output of the current decoder 1, and Linear is linear. The construction method of GCN decoder 2 to GCN decoder 28 is the same as the construction method of GCN decoder 1.

[0097] (5) Training the dual-branch pre-trained network

[0098] 1) Construct loss function

[0099] The loss function L includes the temporal branch feature reconstruction loss function l trec , spatial branch feature reconstruction loss function l srec , temporal branch adversarial loss function l t , spatial branch adversarial loss function l s .

[0100] The temporal branch feature reconstruction loss function l is constructed as follows: trec :

[0101]

[0102] Where i∈{1,2,…,n}, n is the number of source subjects, which is 14. ti is the output of the ith decoder in the sequential branch, x i is the differential entropy feature of the i-th source subject, and MSE represents the mean square error.

[0103] The spatial branch feature reconstruction loss function l is constructed as follows: srec :

[0104]

[0105] Among them, O si is the output of the i-th decoder in the spatial branch.

[0106] The temporal branch adversarial loss function l is constructed as follows: t :

[0107]

[0108] in, The probability that the feature belongs to a source subject is predicted by subject discriminator 1, is the true domain label to which the i-th sample belongs.

[0109] Construct the spatial branch adversarial loss function l as follows s :

[0110]

[0111] in, The probability that the feature belongs to a source subject is predicted by subject discriminator 2, is the true domain label to which the i-th sample belongs.

[0112] 2) Training a dual-branch pre-trained network

[0113] The training set was input into the two-branch pre-trained network for training. The training parameters were: learning rate was 0.003, weight decay rate was 0.0005, and Adam optimizer was used. It was run on NVIDIAA40 GPUs for 150 iterations until the loss function converged.

[0114] Since the present invention constructs a method for extracting dynamic random mask features, the EEG channel features are randomly masked according to a certain ratio to simulate the damaged EEG channel features in a real environment. In the pre-training stage, the temporal features are extracted from the temporal branch by inputting the masked EEG channel features, and the GCN encoder and GCN decoder are used to further obtain the spatial features based on the extracted temporal features, thereby solving the problem of learning and reconstructing generalized spatiotemporal features from the masked EEG channel features.

[0115] (6) Constructing a dual-branch fine-tuning network

[0116] Figure 7 A schematic diagram of the structure of the dual-branch fine-tuning network of this embodiment is given. Figure 7 In the embodiment, the dual-branch fine-tuning network is composed of an LSTM encoder 3, a GCN encoder 3, a feature fusion layer, and a sentiment classifier. The output ends of the LSTM encoder 3 and the GCN encoder 3 are connected to the input end of the feature fusion layer, and the output end of the feature fusion layer is connected to the sentiment classifier.

[0117] (7) Training the dual-branch fine-tuning network

[0118] 1) Constructing the cross entropy loss function

[0119] The cross entropy loss function l is constructed as follows cls :

[0120]

[0121] Among them, y j is the real emotion category, is the predicted emotion category, j represents the three emotion categories of positive emotion p, neutral emotion u and negative emotion q, j∈{1,2,3}.

[0122] 2) Training a dual-branch fine-tuning network

[0123] The training set is input into the two-branch fine-tuning network for training. The training parameters are: learning rate is 0.003, weight decay rate is 0.0005, Adam optimizer is used, and it is iterated 300 times until the cross entropy loss function converges.

[0124] (8) Test the dual-branch fine-tuning network

[0125] The test set is input into the trained two-branch fine-tuning network for testing, and the predicted emotion category is output.

[0126] Completed the EEG emotion recognition method based on the dual-branch damaged channel feature reconstruction model.

[0127] Example 2

[0128] The EEG emotion recognition method based on the dual-branch damaged channel feature reconstruction model of this embodiment consists of the following steps:

[0129] (1) Dataset preprocessing

[0130] This step is the same as in Example 1.

[0131] (2) Divide the training set and test set

[0132] This step is the same as in Example 1.

[0133] (3) Extract dynamic random mask channel features

[0134] This step is the same as in Example 1.

[0135] (4) Constructing a dual-branch pre-training network

[0136] The structure of the dual-branch pre-training network is the same as that of Example 1.

[0137] The construction method of the graph convolution layer 1 of this embodiment is as follows:

[0138] Construct the graph convolution layer O according to formula (1) g :

[0139] O g =S×(X×W)+b (1)

[0140]

[0141]

[0142] In formula (1), λ is a sparse factor, and the value of λ is 1 to 10. In this embodiment, the value of λ is 1. b is a bias, and b∈[-0.125, 0.125]. The value of b in this embodiment is -0.125. The meanings and values ​​of other parameters and coefficients are the same as those in embodiment 1.

[0143] The other steps of this step are the same as those in Example 1.

[0144] The other steps are the same as those in Example 1. The EEG emotion recognition method based on the double-branch damaged channel feature reconstruction model is completed.

[0145] Example 3

[0146] The EEG emotion recognition method based on the dual-branch damaged channel feature reconstruction model of this embodiment consists of the following steps:

[0147] (1) Dataset preprocessing

[0148] This step is the same as in Example 1.

[0149] (2) Divide the training set and test set

[0150] This step is the same as in Example 1.

[0151] (3) Extract dynamic random mask channel features

[0152] This step is the same as in Example 1.

[0153] (4) Constructing a dual-branch pre-training network

[0154] The structure of the dual-branch pre-training network is the same as that of Example 1.

[0155] The construction method of the graph convolution layer 1 of this embodiment is as follows:

[0156] Construct the graph convolution layer O according to formula (1) g :

[0157] O g =S×(X×W)+b (1)

[0158]

[0159]

[0160] In formula (1), λ is a sparse factor, and the value of λ is 1 to 10. In this embodiment, the value of λ is 10. b is a bias, and b∈[-0.125, 0.125]. The value of b in this embodiment is 0.125. The meanings and values ​​of other parameters and coefficients are the same as those in embodiment 1.

[0161] The other steps of this step are the same as those in Example 1.

[0162] The other steps are the same as those in Example 1. The EEG emotion recognition method based on the double-branch damaged channel feature reconstruction model is completed.

[0163] In order to verify the beneficial effects of the present invention, a comparative simulation experiment was carried out using the method of Example 1 of the present invention (hereinafter referred to as the present invention) and “Anovel bi-hemispheric discrepancy model for EEG emotion recognition” (hereinafter referred to as Comparative Experiment 1), “Gmss: Graph-Based Multi-Task Self-Supervised Learning for EEG emotion Recognition.” (hereinafter referred to as Comparative Experiment 2), and “DMMR: Cross-Subject Domain Generalization for EEG-Based Emotion Recognition via Denoising Mixed Mutual Reconstruction.” (hereinafter referred to as Comparative Experiment 3).

[0164] Determine the evaluation indicators: average accuracy (AVG) and standard deviation (STD). The higher the average accuracy, the better the classification effect. The larger the standard deviation, the worse the stability; the smaller the standard deviation, the better the stability. Calculate the average accuracy AVG according to the following formula:

[0165]

[0166]

[0167] Among them, ACC is the accuracy of a single subject, t∈{1,2,...,15} is the number of subjects, C t is the number of samples correctly predicted by the tth subject, N t is the total number of samples of the tth subject, K is the total number of subjects, and its value is 15.

[0168] The standard deviation STD is calculated as follows:

[0169]

[0170] Among them, AVG is the average accuracy of all subjects.

[0171] The experimental results are shown in Table 1.

[0172] Table 1 Experimental results of the present invention and comparative experimental methods

[0173] Comparative experiment type AVG(%) STD(%) Comparative experiment 1 85.40 7.53 Comparative experiment 2 86.52 6.22 Comparative experiment 3 86.54 6.02 The present invention 88.69 6.07

[0174] As can be seen from Table 1, the accuracy of the present invention is 3.19% higher than that of comparative experiment 1, 2.17% higher than that of comparative experiment 2, and 2.15% higher than that of comparative experiment 3. The standard deviation of the present invention is 1.47% lower than that of comparative experiment 1, 0.15% lower than that of comparative experiment 2, and slightly higher than that of comparative experiment 3 by 0.05%.

Claims

1. The EEG emotion recognition method based on the dual-branch damaged channel feature reconstruction model consists of the following steps: (1) Dataset preprocessing 15 subjects were selected from the SEED dataset containing three emotion categories: positive emotion p, neutral emotion u, and negative emotion q. The differential entropy features of EEG signals in five frequency bands, δ 1-3Hz, θ 4-7Hz, α 8-13Hz, β 14-30Hz, and γ 31-50Hz, were obtained as input. The EEG signals in the five frequency bands were divided into time series segments according to a 1-second sliding window with a fixed time step of 30 seconds. (2) Divide the training set and test set The EEG features and corresponding labels after the masked channel were cross-validated according to the leave-one-out cross-validation of the subjects. Each time, the data of one subject was selected as the test set, and the data of the remaining 14 subjects were used as the training set. (3) Extract dynamic random mask channel features According to the international 10-20 standard EEG channels, for the 62 EEG channels of the SEED dataset, the mask ratios were selected as 10%, 20%, and 30%, and 6, 12, and 18 EEG channels were randomly masked in each batch. The differential entropy features after masking were extracted to simulate the channel damage in the real environment. The masked matrix X was determined as follows: m : X m =X⊙(1-M) Where M represents a mask matrix with the same dimension as the input data X, whose value is 0 or 1, indicating the masked matrix, ⊙ represents the dot product, and 1-M represents the unmasked matrix; (4) Constructing a dual-branch pre-training network The dual-branch pre-trained network is composed of a temporal branch network and a spatial branch network in parallel; The described temporal branch network is composed of an LSTM encoder 1, an LSTM encoder 2, 28 LSTM decoders, a gradient flip layer 1, a main discriminator 1, and a mixed layer 1. An output end of the LSTM encoder 1 is connected to the main discriminator 1 through the gradient flip layer 1, and the other output end of the LSTM encoder 1 is connected to the input end of the parallel LSTM decoder 1-LSTM decoder 14. The output end of the parallel LSTM decoder 1-LSTM decoder 14 is connected to the input end of the mixed layer 1, and the output end of the mixed layer 1 is connected to the input end of the parallel LSTM decoder 15-LSTM decoder 28 through the LSTM encoder 2; The spatial branch network is composed of a GCN encoder 1, a GCN encoder 2, 28 GCN decoders, a gradient flip layer 2, a main discriminator 2, and a mixing layer 2. An output end of the GCN encoder 1 is connected to the main discriminator 2 through the gradient flip layer 2, and the other output end of the GCN encoder 1 is connected to the input end of the parallel GCN decoder 1-GCN decoder 14. The output end of the parallel GCN decoder 1-GCN decoder 14 is connected to the input end of the mixing layer 2, and the output end of the mixing layer 2 is connected to the input end of the parallel GCN decoder 15-GCN decoder 28 through the GCN encoder 2; (5) Training the dual-branch pre-trained network 1) Construct loss function The loss function L includes the temporal branch feature reconstruction loss function l trec , spatial branch feature reconstruction loss function l srec , temporal branch adversarial loss function l t , spatial branch adversarial loss function l s ; The temporal branch feature reconstruction loss function l is constructed as follows: trec : Where i∈{1,2,…,n}, n is the number of source subjects, which is 14. ti is the output of the ith decoder in the sequential branch, x i is the differential entropy feature of the i-th source subject, MSE represents the mean square error; The spatial branch feature reconstruction loss function l is constructed as follows: srec : Among them, O si is the output of the i-th decoder in the spatial branch; The temporal branch adversarial loss function l is constructed as follows: t : in, The probability that the feature belongs to a source subject is predicted by subject discriminator 1, is the real domain label to which the i-th sample belongs; Construct the spatial branch adversarial loss function l as follows s : in, The probability that the feature belongs to a source subject is predicted by subject discriminator 2, is the real domain label to which the i-th sample belongs; 2) Training a dual-branch pre-trained network The training set was input into the dual-branch pre-trained network for training. The training parameters were: learning rate was 0.003, weight decay rate was 0.0005, Adam optimizer was used, and it was run on NVIDIAA40 GPUs for 150 iterations until the loss function converged. (6) Constructing a dual-branch fine-tuning network The dual-branch fine-tuning network is composed of an LSTM encoder 3, a GCN encoder 3, a feature fusion layer, and a sentiment classifier. The output ends of the LSTM encoder 3 and the GCN encoder 3 are connected to the input end of the feature fusion layer, and the output end of the feature fusion layer is connected to the sentiment classifier. The structure of the GCN encoder 3 is the same as that of the GCN encoder 1. (7) Training the dual-branch fine-tuning network 1) Constructing the cross entropy loss function The cross entropy loss function l is constructed as follows cls : Among them, y j is the real emotion category, is the predicted emotion category, j represents the three emotion categories of positive emotion p, neutral emotion u and negative emotion q, j∈{1,2,3}; 2) Training a dual-branch fine-tuning network The training set is input into the two-branch fine-tuning network for training. The training parameters are: learning rate is 0.003, weight decay rate is 0.0005, Adam optimizer is used, 300 iterations are performed, and training is performed until the cross entropy loss function converges. (8) Test the dual-branch fine-tuning network The test set is input into the trained two-branch fine-tuning network for testing, and the predicted emotion category is output.

2. The EEG emotion recognition method based on the dual-branch damaged channel feature reconstruction model according to claim 1 is characterized in that: In step (4) of constructing the pre-trained model, the GCN encoder 1 is composed of a graph convolution layer 1, a batch normalization layer 1, a graph convolution layer 2, and a batch normalization layer 2 connected in series in sequence; the structure of the GCN encoder 2 is the same as that of the GCN encoder 1.

3. The EEG emotion recognition method based on the dual-branch damaged channel feature reconstruction model according to claim 1 is characterized in that: In step (4) of constructing the pre-trained model, the GCN decoder 1 is composed of a graph convolution layer 1, a batch normalization layer 3, a linear layer 1, a graph convolution layer 4, a batch normalization layer 4, and a linear layer 2, the graph convolution layer 1 is connected in series with the batch normalization layer 3, the linear layer 1, the graph convolution layer 4, the batch normalization layer 4, and the linear layer 2, and the other output end of the linear layer 1 is connected to the other input end of the linear layer 2; the structure of the GCN decoder 2 to the GCN decoder 28 is the same as that of the GCN decoder 1.

4. The EEG emotion recognition method based on the dual-branch damaged channel feature reconstruction model according to claim 1 is characterized in that In step (4) of constructing a dual-branch pre-trained network, the graph convolution layer is constructed as follows: Construct the graph convolution layer O according to formula (1) g : O g =S×(X×W)+b (1) Where k,l∈{1,2,...,N}, N is 62, P k and P l Represent two EEG channels, ‖P k -P l ‖2 is the Euclidean norm between two EEG channels; A k,l is the obtained adjacency matrix, is the square of the three-dimensional distance between nodes k and l, λ is the sparsity factor, λ takes values ​​of 1 to 10, and the sparse graph matrix S is obtained; W is the weight parameter; b is the bias, b∈[-0.125,0.125].

5. The method for EEG emotion recognition based on the dual-branch damaged channel feature reconstruction model according to claim 1, characterized in that: In the method for constructing the graph convolutional layer, the λ is a sparse factor, the value of λ is 8, and b is a bias, the value of b is 0.

1.

6. The EEG emotion recognition method based on the dual-branch damaged channel feature reconstruction model according to claim 1 is characterized in that In step (4) of constructing a dual-branch pre-trained network, the construction method of the GCN decoder 1 is as follows: Construct GCN decoder 1 as follows: h s =Linear(h s0 +D(X,S)) Among them, h s0 is an initial all-zero hidden state tensor, D is the graph convolution operation, and h s is the output of the current decoder 1, and Linear is the linear layer.

Images