Electroencephalogram signal emotion detection method based on multi-stage feature fusion

Abstract

The invention relates to the field of medical signal processing and the field of artificial intelligence, in particular to an electroencephalogram signal emotion detection method based on multi-stage feature fusion, which comprises the following steps of: constructing a first electroencephalogram signal feature extraction network by taking a time sequence electroencephalogram signal as input to extract frequency domain features; a second electroencephalogram signal feature extraction network is constructed with differential entropy features as input, the network takes EEGNet as a trunk network, and space attention mechanism extraction space and time domain feature extraction space, channel and time domain features are introduced. And constructing a third electroencephalogram signal feature extraction network by taking the first statistical feature as input to extract a second statistical feature, fusing the five features to obtain a fused feature, and performing recognition by using an electroencephalogram signal recognition network to obtain an emotion detection result and output the emotion detection result. According to the method, the problem that information such as space, time domain, frequency domain and channel in the electroencephalogram signals cannot be fully utilized in existing electroencephalogram signal emotion detection is solved.

Description

Technical Field

[0001] The present invention relates to the fields of medical signal processing and artificial intelligence, and specifically to an EEG signal emotion detection method based on multi-stage feature fusion. Background Art

[0002] Emotions are a comprehensive expression that integrates human will and behavior. They largely reflect people's attitudes and cognitions towards things, and also play a very important role in affecting people's behavior and reactions. In advanced human-computer interaction systems, accurate recognition of emotions has become a key research field for achieving harmonious human-computer interaction and personalized interaction, and has set off a research boom in the fields of neuroscience, psychology, and clinical application. In the field of intelligent driving, since emotions have an important impact on driver behavior and driving safety, and there are particularities in collecting signals during driving, driver emotion detection has become a relatively special part of the field of emotion recognition.

[0003] As far as the special field of driver emotion recognition is concerned, it can be mainly divided into two categories: one is recognition based on non-physiological signals, and the other is recognition based on physiological signals. Recognition based on non-physiological signals mainly focuses on the recognition of facial expressions and voice tones. However, this type of signal has the problem of poor reliability, because it can be blurred by deliberately hiding, and some people have expression barriers in this aspect, all of which will lead to inaccurate recognition. Recognition based on physiological signals is to detect human physiological signals such as EEG signals, electrooculographic signals, electromyographic signals, and electrocardiographic signals, and then conduct in-depth research on their relationship with human emotions. Among them, EEG signals are particularly special. It is known as the "gold standard" for emotion recognition. This is because EEG signals cover a lot of information such as human mental activities and physiological states, which can provide a strong basis for emotion detection. However, most of the current research is based on traditional machine learning, such as support vector machine (SVM), K nearest neighbor algorithm (KNN), etc., or universal deep learning networks such as convolutional neural network (CNN), long short-term memory network (LSTM), etc. for emotion detection. This type of research ignores the spatial particularity of the special structure of EEG signals, such as the spatial discontinuity between channels and the strong coupling relationship, which wastes a large amount of data information contained in the EEG signals and reduces the effectiveness of recognition.

[0004] In the prior art, the Chinese patent application with publication number CN116509392A discloses a method for emotion detection and recognition based on brain waves and convolutional neural networks. The method uses the GCN-NGCU structure to extract the differential entropy features and power spectral density features of manually extracted EEG signals, and finally outputs the emotion classification through the fully connected layer. The Chinese patent application with publication number CN117204856A discloses a method for detecting emotions using popular features. After calculating the differential entropy features of the EEG signal, the features are extracted through channel feature extraction, spatial popular mapping modules, etc., and finally the classification of emotions is output. However, in these two inventions, there is still a lack of research on the channel features of EEG signals. Either the extraction of channel features is ignored, or only the information in a single channel is studied, but the mutual coupling and mutual influence between different channels in the EEG signal are ignored, and the time domain, frequency domain, channel and other information contained in the EEG signal are not fully utilized. In addition, only the classification function is applied to the driver's emotion detection, ignoring the objective fact that only extreme emotions affect driving behavior, resulting in certain problems in its practical application. In view of the above problems, the present invention designs a special network structure to design a special feature extraction network structure for different field characteristics, and realizes the final continuous evaluation of the driver's emotion through the feature scale matching and fusion module. Summary of the invention

[0005] In view of this, the present invention provides an EEG signal emotion detection method based on multi-stage feature fusion to overcome the above problems.

[0006] The specific technical solutions of the present invention are as follows:

[0007] A method for detecting emotion of an electroencephalogram signal based on multi-stage feature fusion, comprising the following steps:

[0008] Step 1: Acquire and process EEG signal data to obtain differential entropy features, time series EEG signals and first statistical features used as input vectors;

[0009] Step 2: Use the EEG signal feature extraction network to extract and fuse EEG signal features:

[0010] The EEG signal feature extraction network includes a first EEG signal feature extraction network, a second EEG signal feature extraction network, a third EEG signal feature extraction network, and a feature scale matching and fusion module, wherein:

[0011] The input of the first EEG signal feature extraction network is the time series EEG signal, and the output is the frequency domain feature;

[0012] The input of the second EEG signal feature extraction network is the differential entropy feature used as an input vector, and the output is the spatial and channel features, as well as the time domain features;

[0013] The input of the third EEG signal feature extraction network is the first statistical feature, and the output is the second statistical feature;

[0014] The feature scale matching and fusion module receives frequency domain features, spatial features, time domain features and second statistical features, and fuses them to obtain multi-dimensional fusion features;

[0015] Step 3: The EEG signal recognition network processes the fusion features to obtain an emotion detection result, which is a continuous value from -1 to 1 for the subject's emotional state from negative emotion, neutral emotion to positive emotion; wherein: -1 indicates that the subject is in a negative emotion, 1 indicates that the subject is in a positive emotion, and 0 indicates that the subject is in a neutral emotion. Negative emotions are fear or sadness, and positive emotions are happiness or excitement.

[0016] Furthermore, the EEG signal emotion detection method based on multi-stage feature fusion also includes steps 4 and 5;

[0017] Step 4: Train and optimize the EEG signal feature extraction network and the EEG signal recognition network to obtain an EEG signal emotion detection model;

[0018] Step 5: Input the EEG signal to be detected into the EEG signal emotion detection model for detection, and obtain the emotion detection result output of the EEG signal.

[0019] Furthermore, in step 2, when the EEG signal feature extraction network performs a fusion operation on the frequency domain signal features, the space and channel features, the time domain features and the second statistical features:

[0020] If the magnitude difference of the feature vector lengths of the five types of features, namely, time domain and frequency domain signal features, second statistical features, space and channel features, is within 2 orders of magnitude, feature fusion is performed; if the magnitude difference of the feature vector lengths of time domain and frequency domain signal features, space and channel features, and second statistical features exceeds 2 orders of magnitude, it is necessary to perform dimensionality reduction processing on the corresponding feature vectors.

[0021] Furthermore, the implementation method of step 1 of acquiring EEG signal data and processing to obtain differential entropy features is:

[0022] Unify the sampling rate of EEG signal data, use non-overlapping sliding windows to slice and intercept the EEG signal data after the unified sampling rate, and perform normalization operation on each slice;

[0023] In a single slice, five frequency bands of δ (1 Hz to 4 Hz), θ (4 Hz to 8 Hz), α (8 Hz to 14 Hz), β (14 Hz to 31 Hz), and γ (31 Hz to 50 Hz) are filtered using a fifth-order Butterworth filter to obtain signals of five frequency bands;

[0024] The differential entropy features of each frequency band signal are calculated and extracted respectively, and all differential entropy features extracted from a single channel are concatenated and combined to generate differential entropy features used as input vectors.

[0025] Furthermore, the first statistical features extracted in step 1 include: average amplitude (mean), kurtosis (K), standard deviation (std), waveform factor (wf), pulse factor (f) and frequency domain power spectral density (PSD) of the time series signal.

[0026] Furthermore, the second EEG signal feature extraction network in step 2 is a second EEG signal feature extraction network structure composed of EEGNet as the backbone network and by introducing a spatial attention mechanism and a time domain feature extraction module.

[0027] Furthermore, in the EEG signal feature extraction network:

[0028] The first EEG signal feature extraction network is composed of a frequency domain feature calculation module and a frequency domain feature extraction module; the frequency domain feature calculation module receives the time series EEG signal and transforms it into the frequency domain to obtain the frequency domain EEG signal; the frequency domain feature extraction module is used to extract the frequency domain features of the received frequency domain EEG signal and send the frequency domain features to the feature fusion module;

[0029] The second EEG signal feature extraction network is composed of an EEGNet module, a spatial attention module, and a time domain feature extraction module, wherein: the EEGNet module is used to receive the differential entropy used as an input vector and perform data dimensionality reduction on it, and send the differential entropy after dimensionality reduction to the spatial attention module and the time domain feature extraction module respectively; the spatial attention feature extraction module is used to extract spatial and channel features from the differential entropy after dimensionality reduction using the spatial attention mechanism, and send the spatial and channel features to the feature fusion module; the time domain feature extraction module is used to extract time domain features from the differential entropy after dimensionality reduction, and send the time domain features to the feature fusion module;

[0030] The third EEG signal feature extraction network is composed of a statistical feature calculation module and a statistical feature extraction module, wherein: the statistical feature calculation module is used to perform statistics on the received first statistical feature and send the statistical feature result to the statistical feature extraction module; the statistical feature extraction module is used to extract the second statistical feature from the received statistical feature result and send the second statistical feature to the feature fusion module.

[0031] Furthermore, the EEG signal recognition network includes a vector flattening layer, a fully connected layer, an activation function layer, and a fully connected layer.

[0032] Furthermore, in step 4, when training the EEG signal feature extraction network and the EEG signal recognition network, the input training data is based on the EEG data collected in step 1. The noise signal actually collected in the public data set is added using an overlapping sliding window with a length of 3s and a sliding distance of 1s, and the data obtained after expanding the data is used as the training data.

[0033] The present invention is based on a method for detecting emotions of EEG signals based on multi-stage feature fusion. By processing the acquired EEG signals, differential entropy features, time-series EEG signals and first statistical features used as input vectors are obtained. The differential entropy features used as input vectors are extracted by performing sliding window slicing on the acquired EEG signal data, filtering and normalizing the data in the slices. Then, a first EEG signal feature extraction network is constructed with the time-series EEG signal as input to extract frequency domain features; a second EEG signal feature extraction network is constructed with the differential entropy features as input. The network is based on EEGNet as the backbone network, and spatial, channel and time domain features are extracted by introducing a spatial attention mechanism to extract spatial and time domain features. After obtaining the fused features by fusing these five features, the fused features are identified using an EEG signal recognition network to obtain the output of the emotion detection results. The present invention solves the problem that the existing EEG signal emotion detection fails to make full use of the spatial, time domain, frequency domain, channel and other information in the EEG signal.

[0034] Due to the adoption of the above-mentioned scheme, the present invention has the following beneficial effects:

[0035] 1. The present invention uses differential entropy data as input vector, uses a sliding window method to slice the collected EEG signal data, and removes outliers and normalizes the data within the slice, thereby retaining data distribution characteristics while reducing noise impact.

[0036] 2. In the EEG signal feature extraction network, multiple feature extraction branches are used and statistical features are added to improve the overall performance of the model. In the second EEG signal feature extraction network, the spatial attention mechanism is introduced in EEGNe to extract channel and spatial features, and the time domain feature extraction module is introduced to extract time domain features; in conjunction with the idea of ​​multi-stage feature fusion, the extracted channel features, spatial features, time domain features, frequency domain features and statistical features are fused, which can better retain the effective information of the data and thus improve the recognition accuracy.

[0037] 3. In the process of feature fusion, the feature scale matching and fusion module used judges the similarity of features in scale and dimension, and applies normalization, one-dimensional convolution and other methods to promote feature matching, avoiding the "long tail" problem in multi-feature fusion recognition and improving algorithm performance.

[0038] 4. During the training process, the present invention uses a partially overlapping sliding window method to expand the training data and improve the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of an embodiment of an EEG signal emotion detection method based on multi-stage feature fusion;

[0040] Figure 2 is a flow chart of differential entropy feature extraction used as an input vector in an embodiment;

[0041] Figure 3 This is a diagram of the EEG signal recognition network structure of an embodiment. DETAILED DESCRIPTION

[0042] The modules of the present invention are described in further detail below in conjunction with the illustrations, but the embodiments of the present invention are not limited thereto.

[0043] like Figure 1 As shown, the present embodiment provides an EEG signal emotion detection method based on multi-stage feature fusion, including step 1, collecting and processing EEG signals, step 2, building an EEG signal feature extraction network to extract features from EEG signals, and step 3, using an EEG signal recognition network to perform emotion recognition on the features of the EEG signal input, thereby obtaining an emotion detection result. The specific description of each step is as follows:

[0044] Step 1: Collect and process EEG signal data: EEG signal data collection, this embodiment uses the self-collected driver EEG signals as the EEG signals collected by this embodiment. When collecting signals, the subjects are placed in a simulated driving environment created by a 6D driving simulator, and the subjects are induced to have corresponding emotions by playing a video. At the same time, an EEG acquisition device with a sampling frequency of 1000 Hz and 32 channels is used to collect EEG signals. However, this embodiment is also applicable to emotion recognition using driver EEG signals obtained by other acquisition methods.

[0045] The differential entropy feature, the time series EEG signal and the first statistical feature used as an input vector are obtained from the collected EEG signal; wherein:

[0046] The differential entropy feature used as the input vector is obtained as Figure 2 As shown, the following steps are included:

[0047] The acquired EEG signal data is uniformly adjusted to 200 Hz, and samples with a sampling rate higher than 200 Hz are downsampled, as shown in the following formula:

[0048] x dn (n) = x(nM) (1)

[0049] In formula (1), x dn (n) is the nth sampling point data of the downsampled signal, x(nM) is the nMth sampling point data of the original signal, and M is the downsampling multiple.

[0050] At the same sampling rate, a sliding window with a window size of 3s is used to perform non-overlapping slice interception, record the maximum and minimum values ​​in the slice, and perform normalization operation on a single slice. The normalization operation formula is shown in (2):

[0051]

[0052] In formula (2), after is the data after normalization calculation, before is the original data, max and min represent the maximum and minimum values ​​of the data in a single data slice respectively.

[0053] In a single slice, a fifth-order Butterworth filter is used to filter the five frequency bands of δ (1Hz to 4Hz), θ (4Hz to 8Hz), α (8Hz to 14Hz), β (14Hz to 31Hz), and γ (31Hz to 50Hz). The Butterworth filter can be expressed by the following formula:

[0054]

[0055] In formula (3), |G(ω)| 2 is the square of the amplitude of the signal after filtering, ω is the frequency, n is the filter order, ω C is the cut-off frequency.

[0056] After obtaining the signals of the five frequency bands, the differential entropy of the signal is calculated every 0.5s for the signal in each frequency band. A total of 6 differential entropy features are calculated within the 3s time window of a single recognition signal, and a total of 30 differential entropy features are obtained by calculating the 5 frequency bands. The 30 differential entropy features extracted from a single channel are spliced ​​and combined to generate a final input vector of size N*32*30, where N is the number of samples in a single epoch, and the sampling adopts the 32-channel EEG sampling device of the international 10-20 EEG sampling electrode map.

[0057] To obtain the time series EEG signal features, this paper uses the time domain analysis method to extract the time series EEG signal features from the EEG signal for the first time.

[0058] The first statistical features are obtained, including: the average amplitude (mean), kurtosis (K), standard deviation (std), waveform factor (wf), pulse factor (f) and frequency domain power spectral density (PSD) of the time series signal. The main calculation method is as follows:

[0059]

[0060] Where N is the number of sample points and x(i) is the sample point value.

[0061]

[0062]

[0063] Where f is the sampling frequency.

[0064] Step 2: Use the EEG signal feature extraction network to extract and fuse EEG signal features:

[0065] The EEG signal feature extraction network includes a first EEG signal feature extraction network, a second EEG signal feature extraction network, a third EEG signal feature extraction network, and a feature scale matching and fusion module, wherein:

[0066] The input of the first EEG signal feature extraction network is the time series EEG signal, and the output is the frequency domain feature. The first EEG signal feature extraction network is composed of a frequency domain feature calculation module and a frequency domain feature extraction module; the frequency domain feature calculation module receives the time series EEG signal and transforms it into the frequency domain to obtain the frequency domain EEG signal; the frequency domain feature extraction module is used to extract the frequency domain features of the received frequency domain EEG signal and send the frequency domain features to the feature fusion module.

[0067] The input of the second EEG signal feature extraction network is the differential entropy feature used as an input vector, and the output is the spatial and channel features, as well as the time domain features. The second EEG signal feature extraction network is composed of an EEGNet module, a spatial attention module, and a time domain feature extraction module, wherein: the EEGNet module is used to receive the differential entropy used as an input vector and perform data dimensionality reduction on it, and send the reduced differential entropy to the spatial attention module and the time domain feature extraction module respectively; the spatial attention feature extraction module is used to extract the spatial and channel features of the reduced differential entropy using the spatial attention mechanism, and send the spatial and channel features to the feature fusion module; the time domain feature extraction module is used to extract the time domain features of the reduced differential entropy, and send the time domain features to the feature fusion module.

[0068] The input of the third EEG signal feature extraction network is the first statistical feature, and the output is the second statistical feature. The third EEG signal feature extraction network is composed of a statistical feature calculation module and a statistical feature extraction module, wherein: the statistical feature calculation module is used to perform statistics on the received first statistical feature and send the statistical feature result to the statistical feature extraction module; the statistical feature extraction module is used to extract the second statistical feature from the received statistical feature result and send the second statistical feature to the feature fusion module.

[0069] The feature fusion module is used to fuse time domain features, frequency features, spatial features, statistical information features and spatial channel features to obtain fused features and output them to the EEG signal recognition network for emotion detection.

[0070] Table 1 Actual structure and parameters of each part of the EEG signal feature extraction network

[0071]

[0072]

[0073] It should be noted that in the EEG signal feature extraction network, its feature scale matching and fusion module is composed of feature dimension determination and fusion layer. By judging the size of the vector feature input to the layer and implementing the splicing operation, the splicing fusion of the EEG signal features extracted at different stages is realized, thereby improving the recognition accuracy. That is, the feature fusion module of this embodiment can realize adaptive feature fusion. When the magnitude difference of the feature vector length input to the feature fusion module is within 2 orders of magnitude, feature fusion is performed. When the magnitude difference of the feature vector length input to the feature fusion module is greater than 2 orders of magnitude, a certain type of feature has too low an effect or is offset. At this time, the input feature vector will be subjected to dimensionality reduction processing. It should be noted that the magnitude is measured based on the dimensions of all features before the final fusion. In this embodiment, the lowest feature is used as the standard magnitude measurement standard.

[0074] Step 3: The EEG signal recognition network processes the fusion features to obtain the emotion detection results. The implementation method is as follows:

[0075] Construct an EEG signal recognition network. The EEG signal recognition network consists of a vector flatten layer, a first fully connected (FC1) layer, an activation function layer (ReLu), and a second fully connected (FC2) layer; wherein:

[0076] The vector flatten layer is used to expand the fused features output by the EEG feature extraction network into one-dimensional features to facilitate the calculation of the subsequent two fully connected layers.

[0077] The parameters of the first fully connected layer are 806*60, and the parameters of the second fully connected layer are 60*1. The activation function layer selects Relu as the activation function to achieve the nonlinear performance of the network.

[0078] This embodiment uses the EEG signal recognition network to accurately identify a total of five features, namely, spatial, channel, time domain, frequency domain, and statistical features, based on the results of the EEG signal feature extraction network, thereby achieving continuous evaluation of the driver's emotions.

[0079] In terms of loss function selection, this embodiment uses the RMSE loss function and the Adam optimizer to train the EEG signal feature extraction network and the EEG signal recognition network to obtain the EEG signal emotion detection model. The RMSE loss function is calculated as follows:

[0080]

[0081] The network is trained using the Adam optimizer, with the maximum number of training rounds set to 128 and the initial learning rate set to 1e-3. The loss function Loss is calculated and back-propagated to the network for optimization, and the training is stopped when the loss function converges or the maximum number of iterations is reached.

[0082] The EEG signal emotion detection model includes a trained EEG signal feature extraction network and an EEG signal recognition network. The EEG signal emotion detection model can be used to detect the driver's emotions.

[0083] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A method for detecting emotion from EEG signals based on multi-stage feature fusion, characterized in that: The method comprises: Step 1: Acquire and process EEG signal data to obtain differential entropy features, time series EEG signals and first statistical features used as input vectors; Step 2: Use the EEG signal feature extraction network to extract and fuse EEG signal features: The EEG signal feature extraction network includes a first EEG signal feature extraction network, a second EEG signal feature extraction network, a third EEG signal feature extraction network, and a feature scale matching and fusion module, wherein: The input of the first EEG signal feature extraction network is the time series EEG signal, and the output is the frequency domain feature; The input of the second EEG signal feature extraction network is the differential entropy feature used as an input vector, and the output is the spatial and channel features, as well as the time domain features; The input of the third EEG signal feature extraction network is the first statistical feature, and the output is the second statistical feature; The feature scale matching and fusion module receives frequency domain features, spatial features, time domain features and second statistical features, and fuses them to obtain multi-dimensional fusion features; Step 3: The EEG signal recognition network processes the fusion features to obtain an emotion detection result, which is a continuous value from -1 to 1 for the subject's emotional state from negative emotion, neutral emotion to positive emotion; wherein: -1 indicates that the subject is in a negative emotion, 1 indicates that the subject is in a positive emotion, and 0 indicates that the subject is in a neutral emotion. Negative emotions are fear or sadness, and positive emotions are happiness or excitement.

2. The method for detecting emotion based on EEG signals based on multi-stage feature fusion according to claim 1, characterized in that: The EEG signal emotion detection method based on multi-stage feature fusion also includes steps 4 and 5; Step 4: Train and optimize the EEG signal feature extraction network and the EEG signal recognition network to obtain an EEG signal emotion detection model; Step 5: Input the EEG signal to be detected into the EEG signal emotion detection model for detection, and obtain the emotion detection result output of the EEG signal.

3. The method for detecting emotion based on EEG signals based on multi-stage feature fusion according to claim 1, characterized in that: In step 2, when the EEG signal feature extraction network performs a fusion operation on the frequency domain signal features, the space and channel features, the time domain features and the second statistical features: If the magnitude difference of the feature vector lengths of the five types of features, namely, time domain and frequency domain signal features, second statistical features, space and channel features, is within 2 orders of magnitude, feature fusion is performed; if the magnitude difference of the feature vector lengths of time domain and frequency domain signal features, space and channel features, and second statistical features exceeds 2 orders of magnitude, it is necessary to perform dimensionality reduction processing on the corresponding feature vectors.

4. The method for detecting emotion of an EEG signal based on multi-stage feature fusion according to claim 1, characterized in that: The implementation method of step 1 of acquiring EEG signal data and processing to obtain differential entropy features is: Unify the sampling rate of EEG signal data, use non-overlapping sliding windows to slice and intercept the EEG signal data after the unified sampling rate, and perform normalization operation on each slice; In a single slice, five frequency bands of δ (1 Hz to 4 Hz), θ (4 Hz to 8 Hz), α (8 Hz to 14 Hz), β (14 Hz to 31 Hz), and γ (31 Hz to 50 Hz) are filtered using a fifth-order Butterworth filter to obtain signals of five frequency bands; Calculate and extract the differential entropy features of each frequency band signal respectively, extract a single channel to all differential entropy features for splicing and combination, so as to generate differential entropy features used as input vectors; The first statistical features extracted in step 1 include: the average amplitude (mean), kurtosis (K), standard deviation (std), waveform factor (wf), pulse factor (f) and frequency domain power spectral density (PSD) of the time series signal.

5. The method for detecting emotion of EEG signals based on multi-stage feature fusion according to claim 1, characterized in that: The second EEG signal feature extraction network in step 2 is a second EEG signal feature extraction network structure composed of EEGNet as the main network and by introducing a spatial attention mechanism and a time domain feature extraction module.

6. The method for detecting emotions using EEG signals based on multi-stage feature fusion according to claim 5, characterized in that: The first EEG signal extraction network of the EEG signal feature extraction network is composed of a frequency domain feature calculation module and a frequency domain feature extraction module; The frequency domain feature calculation module receives the time series EEG signal and transforms it into the frequency domain to obtain the frequency domain EEG signal; The frequency domain feature extraction module is used to extract the frequency domain features of the received frequency domain EEG signal and send the frequency domain features to the feature fusion module; The second EEG signal extraction network of the EEG signal feature extraction network is composed of an EEGNet module, a spatial attention module, and a time domain feature extraction module, wherein: the EEGNet module is used to receive the differential entropy used as an input vector and perform data dimensionality reduction on it, and send the differential entropy after dimensionality reduction to the spatial attention module and the time domain feature extraction module respectively; the spatial attention feature extraction module is used to extract spatial and channel features from the differential entropy after dimensionality reduction using the spatial attention mechanism, and send the spatial and channel features to the feature fusion module; the time domain feature extraction module is used to extract time domain features from the differential entropy after dimensionality reduction, and send the time domain features to the feature fusion module; The third EEG signal extraction network of the EEG signal feature extraction network is composed of a statistical feature calculation module and a statistical feature extraction module, wherein: the statistical feature calculation module is used to perform statistics on the received first statistical features and send the statistical feature results to the statistical feature extraction module; the statistical feature extraction module is used to extract the second statistical features from the received statistical feature results and send the second statistical features to the feature fusion module.

7. The method for detecting emotion of EEG signals based on multi-stage feature fusion according to claim 1, characterized in that: The EEG signal recognition network includes a vector flattening layer, a fully connected layer, an activation function layer, and a fully connected layer.

8. The method for detecting emotion of an EEG signal based on multi-stage feature fusion according to claim 2, characterized in that: When training the EEG signal feature extraction network and the EEG signal recognition network, the input training data of step 4 is based on the EEG data collected in step 1. The noise signal actually collected in the public data set is added using an overlapping sliding window with a length of 3s and a sliding distance of 1s, and the data obtained after expanding the data is used as the training data.

Images