EEG emotion recognition method and system based on combination of multi-head attention network and time convolution

Abstract

The invention discloses a multi-head attention network and time convolution combined EEG emotion recognition method and system, and the system comprises three convolution modules (CV) which are used for extracting the frequency and space information in an EEG signal; a multi-head self-attention module (AT) which highlights important information in frequency and space; performing time convolution (TCN), and extracting advanced time features in the electroencephalogram signals; meanwhile, a convolutional sliding window is combined, so that the classification efficiency is improved; according to the method, the residual network and the attention mechanism are combined, depth information related to emotion recognition in the electroencephalogram signals in frequency, space and dimensions in the electroencephalogram signals is effectively extracted, then time dimension information in the extracted depth information is extracted through the LSTM and the network, final classification is carried out, and the accuracy of emotion recognition in the electroencephalogram signals is improved. And the binary classification accuracy rates of the wake-up dimension and the titer dimension in the DEAP data set respectively reach 98.15% and 97.13%.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and specifically to an EEG emotion recognition method and system combining a multi-head attention network with temporal convolution. Background Art

[0002] The frequency, time and space characteristics of EEG signals are crucial for emotion recognition. However, most literature only considers one or two of these three features. Therefore, it is very necessary to adaptively capture important frequency, space and time features in a unified network by integrating information from different fields, which is of vital significance to improve the accuracy of emotion recognition from EEG signals.

[0003] Emotion recognition methods are divided into two categories. One type of emotion recognition method based on non-physiological signals mainly infers emotional state by analyzing non-physiological signals such as language, behavior and facial expressions. However, these methods are often greatly affected by subjective factors and cannot ensure the authenticity of emotions. From the perspective of neuroscience, another type of emotion recognition method based on physiological signals believes that the physiological signals of the human body are related to the individual's autonomic nervous system and are not controlled by the individual's subjective consciousness. Therefore, physiological signals can more accurately reflect the individual's emotional state. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides an EEG emotion recognition method and system combining a multi-head attention network with time convolution to solve the above problems and perform feature extraction and recognition on EEG signals. First, the differential entropy features of different frequencies are extracted after time domain segmentation, and the differential entropy features of different channels are mapped into a four-dimensional structure. CV is used to extract the spatial and frequency information of EEG signals, and the performance of the model is enhanced by MSA learning weights. Convolution sliding window parallel operation is adopted, and TCN is used to extract time-related information from the input data for classification and recognition, which solves the problems raised in the above background technology.

[0005] The present invention discloses an EEG emotion recognition method combining a multi-head attention network with TCN, comprising a data collection and reading module, a network model reading and operation module, and a data test module;

[0006] The data collection and reading module reads the original EEG signals in the database, divides the data into N segments according to the time domain, calculates the differential entropy of the eigenvalue h(X) through the formula, maps the corresponding coordinates of the read signal features to the two-dimensional space matrix, and sets the height h and width w of the two-dimensional feature map, then stacks the two-dimensional feature maps of different frequency bands d to generate a three-dimensional feature matrix, and then generates a four-dimensional feature matrix R according to the N segments of original EEG signals, and stores each input sample S in the four-dimensional feature matrix R N*d*w*hIn the data collection and reading module, after generating the four-dimensional feature matrix, the data is passed to the network model reading and operation module;

[0007] The network model reading operation module establishes a network model. The network model includes three modules, namely, a CV module, an MHA module and a TCN module. The CV module decouples the channel dimension and the spatial position feature in the feature structure by using a deep separable convolution, and extracts a four-dimensional feature matrix R N*d*w*h The spatial and frequency information Q n , the multi-head attention mechanism collects and reads the input Q n , capturing the dependencies of various ranges within the sequence, outputting the matrix Rn, TCN collects and reads Rn to calculate the spatiotemporal information in the EEG signal of the corresponding time period, and sends the data to the data experiment module;

[0008] The data test module reads the calculation data in the network model reading operation module to analyze the test results through the indicators of accuracy, precision, recall rate and established model performance, and attaches a prediction label to the analysis results.

[0009] As a preferred technical solution of the present invention, the data collection and reading module extracts the original EEG signals in the DEA public data set and divides the extracted DE features into N segments.

[0010] As a preferred technical solution of the present invention, the differential entropy calculation formula for each DE feature is as follows:

[0011] H(X)=-∫f(x)log[f(x)]dx

[0012] Among them, f(x) is the probability density function of the random variable X, x is the value of the random variable, and the differential entropy of a random variable that approximately obeys the Gaussian distribution N(θ, σ2) is equal to the logarithm of its energy spectrum in a specific frequency band. The specific expression is as follows:

[0013]

[0014] Where e and σ represent the Euler constant and standard deviation of the time series, respectively.

[0015] As a preferred technical solution of the present invention, the data collection and reading module converts the differential entropy feature vector of the electrical signal of the external device into a compact 2D graph, and then maps it one by one in the two-dimensional space to form a two-dimensional feature graph with a height of h and a width of w, and combines the frequency band feature d to form a three-dimensional feature matrix, sets the reading time of each frequency band to T, and combines R N*d*h*w , N = 1, 2, 3...2T, it is concluded that each EEG segment can be expressed as Sn∈R 2T*d*h*w .

[0016] As a preferred technical solution of the present invention, the network model reading operation module includes the following steps when performing reading operation:

[0017] Step 1: Network input: extract the four-dimensional feature map of differential entropy of each EEG signal;

[0018] Step 2: Extract frequency and space information, using a residual network with an attention mechanism to extract the frequency and space information from each time slice;

[0019] Step 3: Extract time information: Use the LSTM network to extract the hidden dynamic time information from the output of the residual network with an attention mechanism;

[0020] Step 4: Use the fully connected layer to classify the final emotional state.

[0021] As a preferred technical solution of the present invention, the calculation formula for the minimum number of layers n required in the TCN module is as follows:

[0022]

[0023] In the formula, n is the number of layers, b is the dilation rate, k is the convolution kernel size, and L is the input data dimension;

[0024] Where L=512, b=8, k=8, n=3.

[0025] As a preferred technical solution of the present invention, the CV module adds depthwise convolution and depthwise separable convolution after two layers of convolution, including a maximum pooling layer and a fully connected layer. n ∈R 512*2T , where Qn=(q1,q2,q3...q 2T ), q t ∈R 512 .

[0026] As a preferred technical solution of the present invention, the calculation formula of the data test module is as follows:

[0027]

[0028] TP means the true label is positive and the model predicts it is also positive, TN means the true label is negative and the model predicts it is also negative, FN means the true label is positive but the model predicts it is negative, FP means the true label is negative but the model predicts it is positive, and F1 Score means the final value of the prediction calculation

[0029] The present invention combines the residual network with the attention mechanism to effectively extract the deep information related to the emotion recognition in the EEG signal in the frequency, space, and dimensions. Then, the extracted depth information is extracted through the LSTM, network to extract the time dimension information, and finally classify it. In the DEAP dataset, the accuracy of the two-classification of the arousal dimension and the valence dimension reached 98.15% and 97.13% respectively, the accuracy of the four-classification of the arousal-valence dimension reached 96.96%, and the accuracy of the three-classification of the positive-neutral-negative dimension of the SEED dataset reached 97.64%. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the CV-MSA-TCN model;

[0031] Figure 2 It is the CV-MHT-TCN flow chart;

[0032] Figure 3 It is a four-dimensional feature construction graph;

[0033] Figure 4 It is a schematic diagram of a two-dimensional structure;

[0034] Figure 5 It is the CV structure diagram;

[0035] Figure 6 It is a multi-head self-attention mechanism;

[0036] Figure 7 It is a temporal convolutional network composed of two residual blocks;

[0037] Figure 8 It is a TCN design drawing;

[0038] Fig. 9 It is the ACC index of each sample of "Arousal" in DEAP;

[0039] Fig.10 It is the ACC index of each sample of “Valence” in DEAP;

[0040] Fig.11 is the score of the model evaluation index on DEAP;

[0041] Fig.12 is the score of the model evaluation index on SEED;

[0042] Fig.13 It is an ablation experiment;

[0043] Fig.14 It is a comparison chart with the prior art.

[0044] Fig.15Compare the calculation parameters and time with the existing technology DETAILED DESCRIPTION

[0045] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0046] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or a direct connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] An EEG emotion recognition method combining a multi-head attention mechanism with TCN includes a data collection and reading module, a network model reading and operation module, and a data test module;

[0048] Please refer to Figure 1 and Figure 2 The network model reading operation module establishes a network model. The network model includes three modules, namely, a CV module, an MHA module and a TCN module. The CV module decouples the channel dimension and the spatial position feature in the feature structure by using a deep separable convolution, and extracts a four-dimensional feature matrix R N*d*w*h The spatial and frequency information Q n , the multi-head attention mechanism collects and reads the input Q n , capturing the dependencies of various ranges within the sequence, outputting the matrix Rn, TCN collects and reads Rn to calculate the spatiotemporal information in the EEG signal of the corresponding time period, and sends the data to the data experiment module,

[0049] The data test module reads the calculation data in the network model reading operation module to analyze the test results through the indicators of accuracy, precision, recall rate and established model performance, and attaches a prediction label to the analysis results.

[0050] The DEAP dataset contains 32 subjects, each with 32 channels of EEG signals and 8 channels of peripheral physiological signals. Each subject watched 40 one-minute emotional music videos. After watching each short video, the subject rated the level of arousal, valence, liking, and control affect. Binarization was used, with lower scores indicating weaker emotions and higher scores indicating stronger emotions, with five points as the dividing line. Emotions can be discretized into different labels by combining different dimensions. In this experiment, only two dimensions of arousal and valence were selected for testing.

[0051] Please refer to Figure 3 In the data processing stage, the original signal is divided into T signals with a length of 0.5 seconds, and the differential entropy feature H(X) is calculated for each signal in each frequency band. The specific expression is as follows: H(X) = -∫f(x)log[f(x)]dx, where f(x) is the probability density function of the random variable X, and x is the value of the random variable, which approximately obeys the Gaussian distribution N(θ, σ 2 ), where e and σ represent the Euler constant and standard deviation of the time series, respectively

[0052] Please refer to Figure 2 According to the relative electrode position during acquisition, the differential entropy feature vector was transformed into a two-dimensional graph. In order to explore the advantages and disadvantages of the two-dimensional graph, a compact Figure 8 ×9 and a sparse map 19×19, where the positions that do not correspond to electrode positions are filled with zeros, such as Figure 3 As shown. By combining the differential entropy feature maps of different frequencies, 5×8×9 and 5×19×19 three-dimensional feature matrices are formed. The three-dimensional feature matrix greatly retains the electrode position information, space, and frequency information of the EEG signal. T signals are stacked to form four-dimensional feature matrices of size T×5×8×9 and T×5×9×19, and N samples are linearly combined to form NT×5×8×9 and NT×5×9×19 four-dimensional feature matrices. Each sample can be represented by Z n ∈R NT ×f×h×w , where f represents the number of frequency bands, h and w represent the length and width of the two-dimensional feature map, respectively, N is the number of samples in the dataset, and T is the number of signals into which the samples are decomposed.

[0053] The network model reading operation module includes the following steps when performing the reading operation:

[0054] Step 1: Network input: extract the four-dimensional feature map of differential entropy of each EEG signal;

[0055] Step 2: Extract frequency and space information, using a CV network with a multi-head attention mechanism to extract the frequency and space information from each time slice;

[0056] Step 3: Extract temporal information: Use the TCN network to extract the hidden dynamic temporal information from the output of the residual network with an attention mechanism;

[0057] Step 4: Use the fully connected layer to classify the final emotional state.

[0058] Please refer to Figure 5, after two layers of standard convolution, a deep convolution and a deep separable convolution are added to extract the spatial and frequency features in the time series. The first convolution layer uses the convolution kernel (h,1), where h is the maximum value of the two-dimensional feature map in the vertical direction of the four-dimensional feature structure to ensure the mutual spatial information of the signal in the vertical direction, the step size is 1, and 64 feature maps are output. The second layer uses the convolution kernel (1,w), where w is the maximum value of the two-dimensional EEG feature map in the horizontal direction of the four-dimensional feature structure to ensure the mutual spatial information of the signal in the horizontal direction, the step size is 1, and 64 feature maps are output. In order to reduce the amount of calculation and improve the operation speed, we choose deep convolution as the third layer. Deep convolution can effectively reduce the parameters required by the convolution layer by reducing the number of input channels, thereby reducing the amount of calculation. At the same time, because it can directly learn the spatial features of each time segment, it effectively extracts the spatial features of a specific frequency. The convolution kernel size is (4,4), and 128 feature maps are output. The fourth layer uses deep separable convolution, which is a combination of deep convolution and point-by-point convolution. Deep convolution is used to extract spatial features, and point-by-point convolution is used to extract channel features. Deep separable convolution performs grouped convolutions on the feature dimension, extracts independent spatial features for each frequency segment, and uses a 1×1 convolution to aggregate all segments to extract features before output, thereby expressing the relationship between the internal and cross-graphs of the feature map, effectively extracting frequency and spatial features, with a convolution kernel of (1, 1) and outputting 256 feature maps. In order to reduce overfitting and improve the robustness of the program, a maximum pooling convolution with a convolution kernel of (2, 2) and a step size of 2 is used, and softmax is selected for the final fully connected layer.

[0059] For each time slice data Sn∈R 2T*d*h*w , N = 1, 2, 3...2T, the output obtained by the CV model is Q n ∈R 512*2T , please refer to Figure 6 Apply the multi-head attention mechanism (MHSA) to calculate the attention weights;

[0060] The process can be defined as:

[0061] F Q / K / V =Flatten(Conv(Reshape(LN(x)))

[0062] F S =MHSA(F Q ,F K ,F V )+LN(x)

[0063] F S =MHSA(F Q ,F K ,F V )+LN(x)

[0064] Finally, after the reshape operation (Reshape), it is mapped into a suitable dimension to obtain the final multi-head attention output tensor FS;

[0065] Among them, LN represents the layer normalization operation, and T represents the transposition operation;

[0066] Furthermore, the multi-scale feedforward network is used to process features at different scales, and the multi-scale feedforward network includes multiple parallel convolutional layers, each of which has a different receptive field or output scale.

[0067] Please refer to Figure 7 and Figure 8 The time domain convolutional network TCN consists of one-dimensional full convolution, dilated convolution and causal convolution;

[0068] The one-dimensional full convolution in TCN makes each hidden layer of the network the same length as the input network layer, and fills in zeros to keep the input and output of the previous and next networks the same length. Causal convolution ensures that the output at time t in the sequence model is only related to the elements before (t-1) in the previous layer of the network, and the data after time t will not interfere with the learning of features. Compared with traditional convolution, the expansion factor is introduced in the dilated convolution, and the receptive field of the convolution kernel is expanded exponentially by increasing the expansion factor;

[0069]

[0070] However, for conventional causal convolution, only the values ​​of the first few fixed moments can be captured. If you want to capture information at a longer distance, you need to increase the number of network layers. Dilated Convlution is adopted in TCN, which exponentially expands the receptive field of the convolution kernel by increasing the dilation factor. The following is a schematic diagram of dilated causal convolutions, where the dilation factor (d) = [1,9,18], and the convolution kernel size (k) = 9. The d = 1 of the bottom layer means continuous sampling of the input sequence, and the d = 9 of the middle layer means that one sample is taken for every two input points as the input of the next layer, and so on. Generally speaking, the higher the level, the larger the size of d used. At this time, the dilated convolution with a dilation factor of d at xt is:

[0071]

[0072] By adding residual links to the middle layer of the convolutional neural network, the problem of the network being too deep and difficult to train can be effectively solved. The residual module of the TCN network is encapsulated by methods such as one-dimensional full convolution, causal convolution and dilated convolution, and also includes activation layer ReLU units and weight normalization. A residual block contains two layers of convolution and nonlinear mapping (ReLU units). Weight normalization WeightNorm and regularization Dropout are also added in each layer to regularize the network.

[0073] Through the time domain convolutional network TCN, the N features selected from the training set are input into the TCN model as the input sequence of TCN. By increasing the number of training rounds and adjusting related parameters, the accuracy of the model is continuously improved, thereby obtaining a better classification effect.

[0074] The formula for calculating the minimum number of layers n required in a TCN module is as follows:

[0075]

[0076] Please refer to the experimental results of CV-MSA-TCN on the DEAP dataset. Fig. 9 The average accuracy (ACC) of arousal classification is 97.49% with a standard deviation of 0.93%. The average accuracy of valence classification is 97.36% with a standard deviation of 1.05%. The accuracy of all subjects (#2 and #22) is above 90, except for subject (#22) whose accuracy is below 90%. On the SEED dataset, the recognition rate reached 96.60. Please refer to Fig.10 .

[0077] The accuracy, precision, recall and f1-Score are used to evaluate the performance of CV-MSA-TCN. The evaluation indicators are calculated based on all the prediction results of DEAP and SEED datasets. Please refer to Fig.11

[0078] To justify the study setting, we performed ablation experiments, such as Fig.12 To verify the computational performance of our method, we conducted five-fold cross-validation experiments on these methods on SEED and DEAP data. Fig.13 The average accuracy of each method on the SEED dataset and the DEAP dataset is listed in Figure 1, which proves that our experimental results have achieved good results. At the same time, we used LSTM in the time convolution to construct the CV-MHT-LSTM model and compared it with the time convolution CV-MHT-TCN we designed, as shown in Figure 1. Fig.14 and 15As shown in the figure, we list the number of parameters and computing power of each method on the SEED dataset and the DEAP dataset. It can be found that in terms of time cost, the method in this paper is 171s faster than the 4D-CRNN method. On the DEAP dataset, our compact mapping outperforms the sparse mapping in terms of accuracy and training time cost, and outperforms the 4D-CRNN method with the same compact mapping. For Valence classification, the average accuracy of our method is 97.49%, which is higher than the 4D-FSTception method under the same feature map. In addition, the time cost of this method is even 30s faster than the 4D-FSTception method. In the DEAP dataset, the number of model parameters and floating-point operations we obtained are 10.7M and 21.5G respectively. Compared with other methods, the algorithm in this paper has better performance. The results of the two datasets make us believe that the method in this paper shows better computing performance when classifying emotions.

Claims

1. A multi-head attention network combined with temporal convolution EEG emotion recognition method, characterized in that: The method comprises the following steps: a data collection and reading module reads the original EEG signal in the database, divides the data into N segments according to the time domain, calculates the differential entropy of the eigenvalue h(X) by the formula, maps the corresponding coordinates of the read signal feature to a two-dimensional space matrix, and sets the height h and width w of the two-dimensional feature map, then stacks the two-dimensional feature maps of different frequency bands d to generate a three-dimensional feature matrix, then generates a four-dimensional feature matrix R according to the N segments of original EEG signals, and stores each input sample S in the four-dimensional feature matrix R. N*d*w*h In the data collection and reading module, after generating the four-dimensional feature matrix, the data is passed to the network model reading and operation module; The network model reading operation module establishes the network model. The network model includes three modules, namely, the CV module, the MHA module and the TCN module. The CV module decouples the channel dimension and the spatial position feature in the feature structure by using a deep separable convolution, and extracts the four-dimensional feature matrix R N*d*w*h The spatial and frequency information Q n , the multi-head attention mechanism collects and reads the input Q n , capturing the dependencies of various ranges within the sequence, outputting the matrix Rn, TCN collects and reads Rn to calculate the spatiotemporal information in the EEG signal of the corresponding time period, and sends the data to the data experiment module; The data experiment module reads the calculation data in the network model reading operation module to analyze the test results through the accuracy, precision, recall rate and the established model performance indicators, and attaches prediction labels to the analysis results.

2. The EEG emotion recognition method based on multi-head attention network combined with temporal convolution according to claim 1, characterized in that: The data collection and reading module extracts the original EEG signals in the DEAP public data set and divides the extracted DE features into N segments.

3. The EEG emotion recognition method combining the multi-head attention mechanism with TCN according to claim 2, characterized in that: The differential entropy calculation formula of the DE feature is as follows: In the formula, f(x) represents the probability density function of X, and the distribution of f(x) in a fixed frequency band is Gaussian distribution (μ, σ 2 ) is satisfied, the calculation formula is as follows: The e in the formula represents the Euler constant of the time series, represents the standard deviation of the time series, and μ represents the expected value.

4. The EEG emotion recognition method based on multi-head attention network combined with temporal convolution according to claim 3, characterized in that: The data collection and reading module converts the differential entropy feature vector of the electrical signal of the external device into a compact 2D graph, and then maps it one by one into the two-dimensional space to form a two-dimensional feature graph with a height of h and a width of w, and combines the frequency band feature d to form a three-dimensional feature matrix, sets the reading time of each frequency band to T, and combines R N*d*h*w , N = 1, 2, 3…2T, and each EEG segment can be expressed as S n ∈R 2T*d*h*w .

5. The EEG emotion recognition method based on multi-head attention network combined with temporal convolution according to claim 1, characterized in that: The network model reading operation module includes the following steps when performing the reading operation: Step 1: Network input: extract the four-dimensional feature map of differential entropy of each EEG signal; Step 2: Extract frequency and space information, using a CV network with a multi-head attention mechanism to extract the frequency and space information from each time slice; Step 3: Extract temporal information: Use the TCN network to extract the hidden dynamic temporal information from the output of the residual network with an attention mechanism; Step 4: Use the fully connected layer to classify the final emotional state.

6. The EEG emotion recognition method using a multi-head attention network combined with temporal convolution according to claim 5, characterized in that: The calculation formula for the minimum number of layers n required in the TCN module is as follows: In the formula, n is the number of layers, b is the dilation rate, k is the convolution kernel size, and L is the input data dimension.

7. The EEG emotion recognition method using a multi-head attention network combined with temporal convolution according to claim 6, characterized in that: The depthwise separable convolution in the CV module reduces the parameter storage and computational cost. Comparing the number of parameters and computational complexity of the two, the calculation formula is as follows: The number of parameters occupied by the standard convolution is H×W×M×N, and the amount of calculation is The total number of parameters of this method is H×W×1×M+1×1×M×N, and the amount of calculation is 8. The EEG emotion recognition method combining multi-head attention network and temporal convolution according to claim 1, characterized in that: The calculation formula of the data test module is as follows: Among them, TP means that the true label is positive and the model predicts it is also positive, TN means that the true label is negative and the model predicts it is also negative, FN means that the true label is positive but the model predicts it is negative, FP means that the true label is negative but the model predicts it is positive, and F1 Score represents the final value of the prediction calculation.

9. An EEG emotion recognition system combining multi-head attention network with temporal convolution, characterized in that: The method described in any one of claims 1 to 7 can be executed, and the system comprises a data collection and reading module, a network model reading and operation module, and a data test module.

Images