Electroencephalogram emotion recognition method based on dual-flow space-time asymmetric physiological region division

Abstract

The application provides an electroencephalogram emotion recognition method based on double-flow time-space asymmetry physiological region division. The recognition method comprises the following steps: step one, after frequency division extraction of the electroencephalogram signal, the signal is sent to a dynamic time layer; step two, the electroencephalogram signal with extracted time-frequency features is divided into four regions of frontal lobe, temporal lobe, parietal lobe and occipital lobe according to physiology, and is sent to an asymmetry space layer as a first flow; step three, the original electroencephalogram signal is first divided into four regions of frontal lobe, temporal lobe, parietal lobe and occipital lobe, and is sent to an asymmetry space layer; step four, the signal with extracted space features is sent to a dynamic time layer as a second flow; step five, five regions of the four regions in the first flow plus global electroencephalogram region and five regions of the second flow are sent to a high-level fusion layer for fusion. Compared with the prior art, the application realizes weight distribution of different physiological regions on specific emotions, so that each region of the electroencephalogram is more sensitive to emotion recognition.

Description

TECHNICAL FIELD

[0001] The application relates to a brain electrical emotion recognition method based on double-flow space-time asymmetric physiological region division, and belongs to the technical field of artificial intelligence. BACKGROUND

[0002] Emotion is a basic factor in human daily life, affecting decision-making, perception, interpersonal communication and human intelligence. In order to obtain a more sensitive method of emotional state, researchers begin to explore the correlation between brain activity and emotional changes. Electroencephalogram is a widely used brain imaging technology that can directly measure brain activity. By placing several electrodes on the surface of the head, brain electrical signals can be collected. Brain electrical signals have high temporal resolution and can capture sub-second different signals. With the help of machine learning and signal processing technology, a brain-computer interface system can recognize human emotions through electroencephalogram. Brain electrical signals contain rich brain activity information in different frequency bands. Due to the non-stationarity and dynamics of electroencephalogram, a fixed single kernel may not effectively capture potential emotions occurring at different time scales and durations. Some research has proposed a dynamic time layer based on different scale convolution kernels to learn more rich time-frequency features of brain electrical signals. The time-frequency features and spatial features in brain electrical signals have different effects on brain electrical emotion recognition at different depths. As for the asymmetry of brain emotion, some research has proposed to divide the brain into left and right hemispheres, but the frontal lobe, parietal lobe and temporal lobe are usually functional brain regions related to emotional processes in the brain.

[0003] Therefore, it is necessary to provide a brain electrical emotion recognition method based on double-flow space-time asymmetric physiological region division to extract the effects of time-frequency features and spatial features in brain electrical signals on brain electrical emotion recognition at different depths and capture the activity of emotions in different regions of the brain. SUMMARY

[0004] The purpose of the present application is to provide a brain electrical emotion recognition method based on double-flow space-time asymmetric physiological region division to extract the effects of time-frequency features and spatial features in brain electrical signals on brain electrical emotion recognition at different depths and capture the activity of emotions in different regions of the brain.

[0005] To achieve the above purpose, the present application provides a brain electrical emotion recognition method based on double-flow space-time asymmetric physiological region division, comprising the following steps:

[0006] Step one, after the brain electrical signal is frequency-division extracted, it is sent to a dynamic time layer for time-frequency feature extraction;

[0007] Step two, the brain electrical signal with extracted time-frequency features is divided into four regions of frontal lobe, temporal lobe, parietal lobe and occipital lobe according to physiology, and is sent to an asymmetric space layer as a first flow;

[0008] Step 3: The original EEG signal is partitioned into four regions: frontal lobe, temporal lobe, parietal lobe, and occipital lobe, and sent to the asymmetric spatial layer for spatial feature extraction;

[0009] Step 4: Send the signal after extracting spatial features into the dynamic time layer as the second stream;

[0010] Step 5: Send the four regions in the first stream plus the global EEG region (a total of five regions) and the five regions in the second stream into the advanced fusion layer for fusion.

[0011] As a further improvement of the present invention, a multi-scale one-dimensional convolution kernel is used in the dynamic time layer in step 1. The calculation formula of the multi-scale one-dimensional convolution kernel is as follows:

[0012]

[0013] where f S is the EEG sampling rate, α i is the length of the one-dimensional convolution kernel and is set to the ratio of the EEG sampling rate, where i is the level of the dynamic time layer. There are three layers in the dynamic time layer, so i will change from 1 to 3, and the ratio coefficient will become [0.5, 0.25, 0.125]. Therefore, the size of the one-dimensional convolution kernel in the i-th layer can be defined as Multi-scale T-kernels are able to capture both long-term and short-term temporal patterns, and higher-level T-kernels have smaller ratio coefficients, i.e., shorter convolution kernel lengths.

[0014] As a further improvement of the present invention, the calculation formula of the dynamic time layer in step 1 is:

[0015]

[0016] Where X represents the EEG input sample, Conv1D(·) represents the one-dimensional convolution operation, Φ L-ReLU (·) represents the LeakyReLU activation function, AvgPool(·) represents the average pooling layer, Represents the output of the i-th level time kernel.

[0017] As a further improvement of the present invention, the output calculation formula of the dynamic time layer in step 1 is:

[0018]

[0019] where f bn (·) is a batch normalization operation, which concatenates the results extracted at different frequencies along the f direction.

[0020] As a further improvement of the application, in step two, the electroencephalogram signal is divided into four regions of frontal lobe, occipital lobe, temporal lobe and parietal lobe, and the adopted method is to rearrange the signal channels in the original database, that is, ['Fp1', 'AF3', 'F3', 'F7', 'FC5', 'FC1', 'Fp2', 'AF4', 'Fz', 'F4', 'F8', 'FC6', 'FC2', 'C3', 'CP5', 'CP1', 'P3', 'P7', 'PO3', 'CP2', 'P4', 'P8', 'PO4', 'Pz', 'Cz', 'C4', 'CP6', 'T7', 'T8', 'O1', 'Oz', 'O2'], the array subscript is from 0 to 31, wherein the channels with subscript from 0 to 12 represent the frontal lobe, the channels with subscript from 13 to 26 represent the parietal lobe, the channels with subscript from 27 to 28 represent the temporal lobe, and the channels with subscript from 29 to 31 represent the occipital lobe.

[0021] As a further improvement of the application, in step two, the size of the convolution kernel of the asymmetric spatial layer is:

[0022]

[0023] Wherein, δ is a coefficient for controlling the ratio of the space and the length to the total number of channels, and δ j =[13,14,2,3], wherein j is the number of regions of the electroencephalogram signal, the electroencephalogram signal is divided into four regions, so j will change from 0 to 3, and thus the size of the convolution kernel of the jth asymmetric spatial layer is defined as

[0024] As a further improvement of the application, in step two, the calculation formula of the asymmetric spatial layer is:

[0025]

[0026] Wherein Z T represents the output of the dynamic temporal layer, Conv1D(·) represents the one-dimensional convolution operation, Φ L-ReLU (·) represents the LeakyReLU activation function, and AvgPool(·) represents the average pooling layer.

[0027] As a further improvement of the application, in step two, the output calculation formula of the asymmetric spatial layer is:

[0028]

[0029] Wherein f bn (·) is a batch normalization operation, and represents a concatenation operation in the channel c dimension.

[0030] As a further improvement of the present application, the fusion method of the high-level fusion layer in step five is:

[0031] Z fusion =GAP(f bn (Φ L-ReLU (Conv1D(Z S ,(10,1)))

[0032] Output=Φ softmax (W′Φ dp (Φ ReLU (W(Γ(Z S ))+b))+b′)

[0033] Wherein, the convolution kernel size is (10, 1), and information fusion is carried out along the spatial dimension; Φ dp represents a dropout layer, Φ softmax is an activation function, Γ(·) is a compression operation, W' and W are optional connection weight matrices, and b and b' are bias terms.

[0034] The present application has the beneficial effects that: the present application realizes the weight distribution of different physiological regions on specific emotions to the greatest extent by physiological partitioning of the electroencephalogram signal and setting the first flow dynamic time layer-asymmetric space layer and the second flow asymmetric space layer-dynamic time layer, and finally sending them into the high-level fusion layer, and the characteristics of different levels of time and space make the electroencephalogram regions more sensitive to emotion recognition. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 It is a method flow block diagram of the present application.

[0036] Figure 2 It is a unit network structure framework diagram of the present application.

[0037] Figure 3 It is an electroencephalogram partitioning diagram of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in detail below in combination with the drawings and specific embodiments.

[0039] Here, it should be noted that, in order to avoid obscuring the present application due to unnecessary details, only structures and / or processing steps closely related to the scheme of the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0040] It is also to be understood that the terminology "include", "includes" or any other variation thereof is intended to cover a non-exclusive inclusion, such that processes, methods, articles, or apparatuses that comprise a list of elements are not required to comprise only those elements recited.

[0041] As shown in Figure 1 is a flow chart of a brain electrical emotion recognition method based on double-flow space-time asymmetric physiological region division of the application, the application proposes a physiological partition kernel on the basis of existing research, which divides the brain into frontal lobe, parietal lobe, occipital lobe and temporal lobe, as shown in Figure 3 The kernel size of each region is its EEG signal channel number, and a first flow dynamic time layer-asymmetric space layer and a second flow asymmetric space layer-dynamic time layer are proposed to extract the influence of time-frequency features and spatial features on EEG emotion recognition at different depths, wherein the network structure of the dynamic time layer and the asymmetric space layer is as shown in Figure 2 The network structure of the dynamic time layer and the asymmetric space layer is as shown in

[0042] The application discloses a brain electrical emotion recognition method based on double-flow space-time asymmetric physiological region division, comprising the following steps:

[0043] Step one, after the frequency extraction of the EEG signal, the signal is sent to the dynamic time layer for time-frequency feature extraction;

[0044] Step two, the EEG signal with extracted time-frequency features is divided into four regions of frontal lobe, temporal lobe, parietal lobe and occipital lobe according to physiology, and is sent to the asymmetric space layer as the first flow;

[0045] Step three, the original EEG signal is first divided into four regions of frontal lobe, temporal lobe, parietal lobe and occipital lobe, and is sent to the asymmetric space layer for spatial feature extraction;

[0046] Step four, the signal with extracted spatial features is sent to the dynamic time layer as the second flow;

[0047] Step five, the five regions of the four regions in the first flow plus the global EEG region and the five regions of the second flow are sent to the high-level fusion layer for fusion.

[0048] The steps one to five will be described in detail below.

[0049] In order to extract more discriminative time-frequency features, a multi-scale one-dimensional convolution kernel is used in the dynamic time layer in step one, because the EEG signal contains different rich information in different frequency bands. The multi-scale one-dimensional convolution kernel calculation formula is as follows:

[0050]

[0051] where f S is the electroencephalogram sampling rate, a i is the length of one-dimensional convolution kernel and is set as the ratio of the electroencephalogram sampling rate, where i is the level of the dynamic temporal layer, and the dynamic temporal layer has three levels in total, so i will change from 1 to 3, and the ratio coefficient will become [0.5, 0.25, 0.125], and thus the size of the one-dimensional convolution kernel of the i-th layer can be defined as From the perspective of time, the multi-scale T kernel can capture long-term and short-term temporal patterns and learn more diverse features. The T kernel of a higher level has a smaller ratio coefficient, which makes the convolution kernel length shorter, while the T kernel of a lower level has a larger ratio coefficient, which makes the convolution kernel length longer. The long convolution kernel can learn different features of long time and low frequency. The short convolution kernel extracts short time and high frequency features.

[0052] In this embodiment, the calculation formula of the dynamic temporal layer in step one is:

[0053]

[0054] where X represents the electroencephalogram input sample, Conv1D(·) represents the one-dimensional convolution operation, Φ L-ReLU (·) represents the LeakyReLU activation function, AvgPool(·) represents the average pooling layer, represents the output of the i-th level of the temporal kernel.

[0055] Further, the output calculation formula of the dynamic temporal layer in step one is:

[0056]

[0057] where f bn (·) is the batch normalization operation, and the results extracted at different frequencies are spliced in the f direction to extract time-frequency features.

[0058] In step two, the electroencephalogram signal with extracted time-frequency features is divided into four regions, i.e., frontal lobe, occipital lobe, temporal lobe, and parietal lobe, as shown in Figure 2As shown, this is because the asymmetric spatial layer utilizes the neuroscience finding that the brain activity in frontal lobe, temporal lobe, parietal lobe and occipital lobe is asymmetrically related to emotion. The implemented method is to rearrange the signal channels in the original database, i.e., ['Fp1', 'AF3', 'F3', 'F7', 'FC5', 'FC1', 'Fp2', 'AF4', 'Fz', 'F4', 'F8', 'FC6', 'FC2', 'C3', 'CP5', 'CP1', 'P3', 'P7', 'PO3', 'CP2', 'P4', 'P8', 'PO4', 'Pz', 'Cz', 'C4', 'CP6', 'T7', 'T8', 'O1', 'Oz', 'O2'], the array subscript is from 0 to 31, wherein the channels with subscript from 0 to 12 represent the frontal lobe, the channels with subscript from 13 to 26 represent the parietal lobe, the channels with subscript from 27 to 28 represent the temporal lobe, and the channels with subscript from 29 to 31 represent the occipital lobe.

[0059] In step two, a regional kernel is proposed to learn the asymmetric features between brain regions, and the size of the convolution kernel of the asymmetric spatial layer is:

[0060]

[0061] wherein δ is a coefficient for controlling the ratio of space and length to the total number of channels, and δ j =[13,14,2,3], wherein j is the number of regions of the electroencephalogram signal, the electroencephalogram signal is divided into four regions, so j will change from 0 to 3, and thus the size of the convolution kernel of the jth asymmetric spatial layer is defined as After obtaining the size of the convolution kernel, the calculation formula of the asymmetric spatial layer can be derived, and the calculation formula of the asymmetric spatial layer is:

[0062]

[0063] wherein Z T represents the output of the dynamic temporal layer, Conv1D(·) represents a one-dimensional convolution operation, Φ L-ReLU (·) represents a LeakyReLU activation function, and AvgPool(·) represents an average pooling layer.

[0064] Further, the output calculation formula of the asymmetric spatial layer in step two is:

[0065]

[0066] wherein f bn (·) is a batch normalization operation, and represents a concatenation operation in channel c dimension. The first stream is to extract time-frequency features first, and then extract spatial features.

[0067] In step three, the original brain electrical signals are divided into frontal lobe, occipital lobe, temporal lobe and parietal lobe according to the method implemented in step two, and are sent to the asymmetric spatial layer, the convolution kernel size of the asymmetric spatial layer is calculated according to the calculation formula of step two, and the calculation formula of the asymmetric spatial layer and the output calculation formula of the asymmetric spatial layer are derived, and the spatial features are extracted. This is because the spatial features are more effective for brain electrical signals, and a double-flow feature extraction is further designed, and the second flow first extracts spatial features and then extracts time-frequency features.

[0068] In step four, the brain electrical signals with extracted spatial features are sent to the multi-scale dynamic time layer, and the time-frequency features are extracted according to the operation of step one.

[0069] In step five, the fusion method of the high-level fusion layer is:

[0070] Z fusion =GAP(f bn (Φ L-ReLU (Conv1D(Z S ,(10,1)))))

[0071] Output=Φ softmax (W′Φ dp (Φ ReLU (W(Γ(Z S ))+b))+b′)

[0072] Wherein, the convolution kernel size is (10, 1), including four regions in the first flow plus five regions in the second flow, a total of 10. Information fusion is performed along the spatial dimension. Φ dp represents a dropout layer, Φ softmax is an activation function, Γ(·) is a compression operation, W' and W are optional connection weight matrices, and b and b' are bias terms. The feature signals are sent to the high-level fusion layer to learn the fusion features of the region and global kernel, so as to achieve more compact real-time use.

[0073] In summary, the present application proposes a first flow dynamic time layer-asymmetric spatial layer and a second flow asymmetric spatial layer-dynamic time layer to extract time-frequency features and spatial features, and to study the influence of different depths on brain electrical emotion recognition, and to perform physiological partitioning on brain electrical signals, thereby realizing the weight distribution of different physiological regions in a specific emotion.

[0074] The above examples are only used to illustrate the technical solutions of the present application and not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A method for EEG emotion recognition based on dual-stream spatiotemporal asymmetric physiological region division, characterized by: The following steps are involved: Step 1: Perform frequency division extraction on the EEG signal and send it to the dynamic time layer to extract the time-frequency features; Step 2: The EEG signals after time-frequency features are extracted are divided into four regions according to physiological conditions: frontal lobe, temporal lobe, parietal lobe, and occipital lobe, and sent to the asymmetric spatial layer as the first stream; Step 3: The original EEG signal is partitioned into four regions: frontal lobe, temporal lobe, parietal lobe, and occipital lobe, and sent to the asymmetric spatial layer to extract spatial features; Step 4: Send the signal after extracting spatial features into the dynamic time layer as the second stream; Step 5: Send the four regions in the first stream plus the global EEG region (a total of five regions) and the five regions in the second stream into the advanced fusion layer for fusion.

2. The EEG emotion recognition method based on dual-stream spatiotemporal asymmetric physiological region division according to claim 1 is characterized by: In the dynamic time layer in step 1, a multi-scale one-dimensional convolution kernel is used. The calculation formula of the multi-scale one-dimensional convolution kernel is as follows: where f S is the EEG sampling rate, α i is the length of the one-dimensional convolution kernel and is set to the ratio of the EEG sampling rate, where i is the level of the dynamic time layer. There are three layers in the dynamic time layer, so i will change from 1 to 3, and the ratio coefficient will become [0.5, 0.25, 0.125]. Therefore, the size of the one-dimensional convolution kernel in the i-th layer can be defined as Multi-scale T-kernels are able to capture both long-term and short-term temporal patterns, and higher-level T-kernels have smaller ratio coefficients, i.e., shorter convolution kernel lengths.

3. The method for EEG emotion recognition based on dual-stream spatiotemporal asymmetric physiological region division according to claim 2 is characterized by: The calculation formula of the dynamic time layer in step 1 is: Where X represents the EEG input sample, Conv1D(·) represents the one-dimensional convolution operation, Φ L-ReLU (·) represents the LeakyReLU activation function, AvgPool(·) represents the average pooling layer, Represents the output of the i-th level time kernel.

4. The method for EEG emotion recognition based on dual-stream spatiotemporal asymmetric physiological region division according to claim 3 is characterized by: The output calculation formula of the dynamic time layer in step 1 is: where f bn (·) is a batch normalization operation, which concatenates the results extracted at different frequencies along the f direction.

5. The EEG emotion recognition method based on dual-stream spatiotemporal asymmetric physiological region division according to claim 1 is characterized by: In step 2, the EEG signals are divided into four regions: frontal lobe, occipital lobe, temporal lobe, and parietal lobe. The method used is to rearrange the signal channels in the original database. The array after arrangement is ['Fp1','AF3','F3','F7','FC5','FC1','Fp2','AF4','Fz','F4','F8','FC6','FC2','C3','CP5','CP1','P3','P 7','PO3','CP2','P4','P8','PO4','Pz','Cz','C4','CP6','T7','T8','O1','Oz','O2'], where the array subscripts range from 0 to 31, where the channels with subscripts from 0 to 12 represent the frontal lobe, the channels with subscripts from 13 to 26 represent the parietal lobe, the channels with subscripts from 27 to 28 represent the temporal lobe, and the channels with subscripts from 29 to 31 represent the occipital lobe.

6. The method for EEG emotion recognition based on dual-stream spatiotemporal asymmetric physiological region division according to claim 1, characterized in that: The size of the convolution kernel of the asymmetric spatial layer in step 2 is: where δ is a coefficient that controls the ratio of space and length to the total number of channels, and δ j =[13,14,2,3], where j is the number of regions where the EEG signal is partitioned. The EEG signal is divided into four regions, so j will vary from 0 to 3. The size of the convolution kernel of the jth asymmetric spatial layer is defined as 7. The method for EEG emotion recognition based on dual-stream spatiotemporal asymmetric physiological region division according to claim 6, characterized in that: The calculation formula for the asymmetric spatial layer in step 2 is: where Z T represents the output of the dynamic time layer, Conv1D(·) represents the one-dimensional convolution operation, Φ L-ReLU (·) represents the LeakyReLU activation function, and AvgPool(·) represents the average pooling layer.

8. The method for EEG emotion recognition based on dual-stream spatiotemporal asymmetric physiological region division according to claim 7, characterized in that: The output calculation formula of the asymmetric spatial layer in step 2 is: where f bn (·) is a batch normalization operation, which represents a concatenation operation on the channel c dimension.

9. The method for EEG emotion recognition based on dual-stream spatiotemporal asymmetric physiological region division according to claim 1, characterized in that: The fusion method of the advanced fusion layer in step 5 is: WITH fusion =GAP(f bn (Φ L-ReLU (Conv1D(Z S ,(10,1))))) Output=Φ softmax (W ′ Φ dp (Φ ReLU (W(Γ(Z S ))+b))+b ′ ) Among them, the convolution kernel size is (10, 1), and information fusion is performed along the spatial dimension; Φ dp represents the dropout layer, Φ softmax is the activation function, Γ(·) is the compression operation, W ′ and W are optional weight matrices, b and b ′ It is a partial item.

Images