A method and device for recognizing electroencephalogram signals based on spatiotemporal deep learning

Abstract

The present invention discloses a method and device for EEG signal recognition based on spatiotemporal deep learning, comprising preprocessing the collected original EEG signal to obtain a preprocessed EEG signal; calculating manual features on the preprocessed EEG signal and establishing a three-dimensional manual feature matrix in combination with the spatial distribution of the brain electrodes of the original EEG signal; constructing a spatiotemporal deep learning model, using the preprocessed EEG signal and the three-dimensional manual feature matrix as inputs of the spatiotemporal deep learning model, and training the spatiotemporal deep learning model; finally, inputting the EEG signal into the trained spatiotemporal deep learning model to obtain an EEG signal recognition result. The present invention combines the spatial distribution of the brain electrodes of the EEG signal with the manual features, and more comprehensively mines the valuable information contained in the EEG signal than the existing technology, and more fully combines prior knowledge, thereby significantly improving the accuracy of EEG signal recognition.

Description

Technical Field

[0001] The present invention belongs to the field of electroencephalogram (EEG) signal processing, and specifically relates to an EEG signal recognition method and device based on spatiotemporal deep learning. Background Art

[0002] EEG signals are low-cost to acquire and contain rich information, making them uniquely valuable in scientific research and disease diagnosis. However, as highly random and non-stationary signals, EEG signals have extremely weak amplitudes and are easily interfered with by other physiological signals, such as electrocardiogram (ECG) and electrooculogram (EOG), resulting in artifacts. Furthermore, because EEG signals are acquired in real time and have a long duration, it is currently difficult to deploy a large number of experts in hospital settings to interpret these highly rhythmic, real-time EEG signals.

[0003] In recent years, some studies have used artificial intelligence methods such as deep learning to identify EEG signals, such as CN2023104777954, an EEG monitoring system and method for neurology, CN2021116144703, an EEG recognition method based on a time-channel cascaded Transformer network, and CN2021100488180, an EEG recognition method and system based on graph convolution and gated recurrent units. However, existing studies, including the above-mentioned patent applications, have ignored the spatial information of the distribution of brain electrodes involved in EEG signals, and have not combined the temporal characteristics of EEG signals with traditional manual features. They have not fully considered high-dimensional features and prior knowledge, and therefore have not achieved ideal performance. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems raised in the background technology and propose an EEG signal recognition method and device based on spatiotemporal deep learning.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes an EEG signal recognition method based on spatiotemporal deep learning, comprising:

[0007] Preprocessing the collected original EEG signal to obtain a preprocessed EEG signal;

[0008] Calculate the manual features of the preprocessed EEG signal and build a three-dimensional manual feature matrix based on the spatial distribution of the brain electrodes of the original EEG signal;

[0009] Build a spatiotemporal deep learning model, use the preprocessed EEG signals and the three-dimensional hand-crafted feature matrix as input to train the spatiotemporal deep learning model;

[0010] Finally, the EEG signal is input into the trained spatiotemporal deep learning model to obtain the EEG signal recognition result.

[0011] Preferably, preprocessing the collected original EEG signal to obtain a preprocessed EEG signal includes:

[0012] The original EEG signal is processed in sequence through filtering, downsampling, independent component analysis algorithm, normalization and sliding window segmentation to obtain the preprocessed EEG signal.

[0013] Preferably, calculating manual features of the preprocessed EEG signal and establishing a three-dimensional manual feature matrix based on the spatial distribution of the brain electrodes of the original EEG signal include:

[0014] The preprocessed EEG signals are respectively calculated with manual features in time domain, manual features in frequency domain and manual features of nonlinear entropy;

[0015] Calculate the manual features in the time domain, including calculating the mean, variance, standard deviation, normalized first-order difference and normalized second-order difference of the preprocessed EEG signals to obtain the corresponding manual features;

[0016] Calculate the manual features in the frequency domain, including calculating the band power of the delta frequency band, theta frequency band, alpha frequency band, beta frequency band and gamma frequency band of the preprocessed EEG signal to obtain the corresponding manual features;

[0017] Calculate the manual features of nonlinear entropy, including calculating the approximate entropy, sample entropy, Raine entropy, Tsallis entropy and differential entropy of the preprocessed EEG signals to obtain the corresponding manual features;

[0018] According to the spatial distribution of brain electrodes involved in the original EEG signal, a two-dimensional matrix is ​​constructed, and combined with the calculated manual feature stacking, a three-dimensional manual feature matrix is ​​obtained.

[0019] Preferably, a spatiotemporal deep learning model is constructed, and the preprocessed EEG signal and the three-dimensional manual feature matrix are used as inputs of the spatiotemporal deep learning model. The spatiotemporal deep learning model is trained, including:

[0020] The spatiotemporal deep learning module includes a preprocessed EEG signal branch, a three-dimensional handcrafted feature matrix branch, a fully connected layer, and a softmax layer;

[0021] The pre-processed EEG signal branch uses multiple convolutional layers to extract high-dimensional features from the input pre-processed EEG signal. After flattening the extracted high-dimensional features into one-dimensional vectors, input embedding and position encoding are performed. A preset number of encoding blocks are used to extract features from the position-encoded data. Each encoding block contains a multi-head attention layer, a first normalization layer, a forward feedback layer, and a second normalization layer.

[0022] The 3D hand-crafted feature matrix branch uses a preset number of convolution blocks and maximum pooling layers to extract high-dimensional features from the input 3D hand-crafted feature matrix. Each convolution block includes a convolution layer, a batch normalization layer, and a random dropout layer.

[0023] Flatten the output of each branch into a one-dimensional vector, and merge the two one-dimensional vectors into a one-dimensional vector;

[0024] Finally, the merged and connected one-dimensional vector is input into the fully connected layer and the Softmax layer in sequence to obtain the prediction result;

[0025] Calculate the loss function, perform backpropagation, and complete the training of the spatiotemporal deep learning model.

[0026] Preferably, a binary cross entropy function is used as the loss function of the spatiotemporal deep learning model, and the loss function Loss calculation formula is as follows:

[0027]

[0028] Among them, N' represents the number of samples for training the spatiotemporal deep learning module, y u represents the binary label of the u-th sample, p(y u ) indicates that the output of the spatiotemporal deep learning module belongs to the binary label y u probability.

[0029] This application also proposes an EEG signal recognition device based on spatiotemporal deep learning, which includes a processor and a memory storing a number of computer instructions. When the computer instructions are executed by the processor, the steps of the EEG signal recognition method based on spatiotemporal deep learning are implemented.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention combines the spatial distribution of brain electrodes of EEG signals with manual features, and more comprehensively mines the valuable information contained in EEG signals than the existing technology, and more fully combines prior knowledge, thereby significantly improving the accuracy of EEG signal recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a module block diagram of the EEG signal recognition method and device based on spatiotemporal deep learning of the present invention;

[0033] Figure 2 Schematic diagram of the three-dimensional manual feature matrix of the present invention;

[0034] Figure 3 Schematic diagram of the structure of the spatiotemporal deep learning model of the present invention;

[0035] Figure 4 Schematic diagram of different electrode combinations of the present invention;

[0036] Figure 5 It is the class activation map of different electrode reduction schemes of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0038] It should be noted that when a component is referred to as being "connected" to another component, it may be directly connected to the other component or there may be an intermediate component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0039] In one embodiment, Figure 1-3 As shown, a method for EEG signal recognition based on spatiotemporal deep learning includes:

[0040] Step 1: preprocess the collected original EEG signal to obtain a preprocessed EEG signal.

[0041] Specifically, the collected original EEG signals are preprocessed to obtain preprocessed EEG signals, including:

[0042] The original EEG signal is processed in sequence through filtering, downsampling, independent component analysis algorithm, normalization and sliding window segmentation to obtain the preprocessed EEG signal.

[0043] In this embodiment, there is no restriction on the object of the collected raw EEG signal. Filtering, downsampling, independent component analysis algorithm, normalization and sliding window segmentation are performed as follows:

[0044] A 1-60 Hz bandpass filter was used to remove high-frequency noise and baseline drift from the original EEG signal;

[0045] Use 50Hz notch filter to filter out the influence of power frequency interference;

[0046] Downsample the signal with an original sampling rate of 1kHz to 250Hz;

[0047] The independent component analysis (ICA) algorithm is used to remove the electrooculogram and electromyography components contained in the original EEG signal;

[0048] The Min-Max method is used to normalize the signal. The specific calculation formula is:

[0049]

[0050] Among them, b i is the i-th element of the sequence B of EEG signal amplitudes, min{B} means finding the minimum value of sequence B, and max{B} means finding the maximum value of sequence B;

[0051] The normalized signal is segmented by a sampling sliding window. The length of each sliding window is 5 seconds. Combined with the sampling rate of 250 Hz, the length of each signal segment is 1250 sampling points.

[0052] Step 2: Calculate the manual features of the preprocessed EEG signal and establish a three-dimensional manual feature matrix based on the spatial distribution of the brain electrodes of the original EEG signal.

[0053] Specifically, the manual features of the preprocessed EEG signals are calculated and combined with the spatial distribution of the electrodes of the original EEG signals to establish a three-dimensional manual feature matrix, including:

[0054] The preprocessed EEG signals are respectively calculated with manual features in time domain, manual features in frequency domain and manual features of nonlinear entropy;

[0055] Calculate the manual features in the time domain, including calculating the mean, variance, standard deviation, normalized first-order difference and normalized second-order difference of the preprocessed EEG signals to obtain the corresponding manual features;

[0056] Calculate the manual features in the frequency domain, including calculating the band power of the delta frequency band, theta frequency band, alpha frequency band, beta frequency band and gamma frequency band of the preprocessed EEG signal to obtain the corresponding manual features;

[0057] Calculate the manual features of nonlinear entropy, including calculating the approximate entropy, sample entropy, Raine entropy, Tsallis entropy and differential entropy of the preprocessed EEG signals to obtain the corresponding manual features;

[0058] According to the spatial distribution of brain electrodes involved in the original EEG signal, a two-dimensional matrix is ​​constructed, and combined with the calculated manual feature stacking, a three-dimensional manual feature matrix is ​​obtained.

[0059] It should be noted that the normalized first-order difference δ in the manual features of the time domain is calculated ξ and normalized second-order differences The calculation formula is as follows:

[0060]

[0061]

[0062] Among them, ξ represents a period with a standard deviation of σ ξ And the length of the time series is T', t represents the tth moment of the time series.

[0063] In this embodiment, when calculating the manual features in the frequency domain, the band powers of the delta band (1-4 Hz), theta band (4-8 Hz), alpha band (8-12 Hz), beta band (12-30 Hz), and gamma band (30-45 Hz) are calculated.

[0064] In this embodiment, when calculating the manual features of nonlinear entropy, the approximate entropy ApEn is used to measure the randomness of the time series. The specific calculation formula is as follows:

[0065]

[0066] Where n=N-m+1, N represents the number of data points in the time series, and m is the embedding dimension. represents the probability of approximation under the embedding dimension m and similarity tolerance r, and 1≤i≤n.

[0067] Sample entropy is an extension of approximate entropy and can also measure randomness and regularity.

[0068] Renyi Entropy RenyiEn is an extension of Shannon Entropy. The specific calculation formula is:

[0069]

[0070] Among them, α represents the order and is a non-negative real number not equal to 1, p i It is the probability calculated from the energy of the original EEG signal, and 1≤i≤n.

[0071] The Tsallis entropy TsEn is calculated for the complexity of non-superposition systems. The specific calculation formula is as follows:

[0072]

[0073] Differential entropy DiffEn is also extended from Shannon entropy, and the specific calculation formula is as follows:

[0074]

[0075] In this embodiment, since different brain electrodes collect activities of the cortex in different areas of the brain during the acquisition of EEG signals, it is necessary to fully consider the spatial position distribution of the brain electrodes. The spatial distribution of the brain electrodes is a three-dimensional distribution. First, the three-dimensional distribution of the brain electrodes is mapped to a two-dimensional plane to construct a two-dimensional matrix, such as a 7*7 two-dimensional matrix. Combined with 15 manual features (mean, variance, standard deviation, normalized first-order difference, normalized second-order difference, delta band, theta band, alpha band, beta band, gamma band, approximate entropy, sample entropy, Reine entropy, Tsallis entropy and differential entropy manual features), a 7*7*15 three-dimensional manual feature matrix is ​​constructed, such as Figure 2 shown.

[0076] Step 3: Build a spatiotemporal deep learning model, use the preprocessed EEG signals and three-dimensional manual feature matrix as input of the spatiotemporal deep learning model, and train the spatiotemporal deep learning model.

[0077] Specifically, the spatiotemporal deep learning module includes a preprocessed EEG signal branch, a three-dimensional handcrafted feature matrix branch, a fully connected layer, and a softmax layer;

[0078] The pre-processed EEG signal branch uses multiple convolutional layers to extract high-dimensional features from the input pre-processed EEG signal. After flattening the extracted high-dimensional features into one-dimensional vectors, input embedding and position encoding are performed. A preset number of encoding blocks are used to extract features from the position-encoded data. Each encoding block contains a multi-head attention layer, a first normalization layer, a forward feedback layer, and a second normalization layer.

[0079] The 3D hand-crafted feature matrix branch uses a preset number of convolution blocks and maximum pooling layers to extract high-dimensional features from the input 3D hand-crafted feature matrix. Each convolution block includes a convolution layer, a batch normalization layer, and a random dropout layer.

[0080] Flatten the output of each branch into a one-dimensional vector, and merge the two one-dimensional vectors into a one-dimensional vector;

[0081] Finally, the merged and connected one-dimensional vector is input into the fully connected layer and the Softmax layer in sequence to obtain the prediction result;

[0082] Calculate the loss function, perform backpropagation, and complete the training of the spatiotemporal deep learning model.

[0083] The parameters of the spatiotemporal deep learning model are shown in Table 1:

[0084] Table 1

[0085]

[0086]

[0087] It should be noted that the preprocessed EEG signal branch: In this embodiment, the size of the preprocessed EEG signal input to the preprocessed EEG signal branch is 21*1250*1 (such as input_2 in Table 1), and six convolution layers with consistent parameters are used (such as 2D convolution_5, 2D convolution_6, 2D convolution_7, 2D convolution_8, 2D convolution_9 and 2D convolution_10 in Table 1, and the dimensions of the outputs of the six convolution blocks are 21*618*32, 21*3 02*32, 21*144*16, 21*65*16, 21*25*8 and 21*5*4) are used to extract high-dimensional features of the input signal, and the convolution layer is composed of multiple convolution kernels, each of which has a size of 1*16 and a step size of 1*2; the extracted high-dimensional features are flattened into a one-dimensional vector (such as flatten_2 in Table 1, and the dimension after flattening is 420), and then input embedding and position encoding are performed. The extracted high-dimensional features are defined as X, and the vector after input embedding is represented as X embedding (As shown in the embedding in Table 1, the output dimension is 420*32). Sin and cosine functions are used for position encoding (position encoding originated in the field of natural language processing. Different arrangements of words in a sentence will lead to different meanings. EEG signals are similar to text information and also emphasize temporal sequence). The specific calculation formula is as follows:

[0088]

[0089]

[0090] in, PE (pos,2q) PE (pos,2q+1) Respectively represent the position pos of the data point in the current time series, the corresponding codes in the even case (2q) and the odd case (2q+1), d model represents the input embedding dimension of the encoding vector, and the output after input embedding and position encoding is represented as X embedded =X embedding +X pos , and X pos (PE (pos,2q) , PE (pos,2q+1) ).

[0091] In this embodiment, four encoding blocks with the same structure (such as encoding block_1, encoding block_2, encoding block_3 and encoding block_4 in Table 1, and the dimensions of the output of these four encoding blocks are all 420*32) are used to extract features from the data after position encoding. Each encoding block includes a multi-head attention layer, a first normalization layer, a forward feedback layer and a second normalization layer:

[0092] The multi-head attention layer needs to complete the position encoding of the data X embedded The input mapping is the query vector Q, the key-value vector K, and the value vector V. The specific calculation formula is as follows:

[0093] Q=X embedded ×W Q

[0094] K=X embedded ×w K

[0095] V=X embedded ×W V

[0096] Among them, w Q 、w K 、w V Represent the weight matrices of the corresponding vectors respectively.

[0097] The dot product attention mechanism with scaling is used to calculate the attention information of the query vector, key-value vector, and value vector after the mapping is completed. The specific calculation formula is as follows:

[0098] Multihead(Q,K,V)=Concat(H1,H2,...,H h )

[0099] H s =Attention(Q s , K s , V s )

[0100]

[0101] Among them, Multihead represents multiple heads, Concat represents the merge connection operation, h represents the number of heads in the multi-head attention, and H s Indicates the sth head, Q s , K s , V s They represent the query vector matrix, key vector matrix, and value vector matrix of the s-th head, respectively. T represents the matrix bias operation. k Represents a balancing parameter used to prevent the dimensionality from being too high.

[0102] After the multi-head attention layer, the first normalization layer, the forward feedback layer and the second normalization layer are used to obtain the output X of an encoding block. output , the specific calculation formula is as follows:

[0103] X add&norm1 =Layernorm(X embedded +Multihead(Q, K, V)

[0104] X ff =max(0, W1X add&norm1 +b1)w2+b2

[0105] X output =Layernorm(X add&norm1 +X rr )

[0106] Among them, X add&norm1 represents the output of the first normalization layer, Layernorm represents the layer normalization operation (the operation of the first normalization layer and the second normalization layer), X ff represents the output of the forward feedback layer, w1 and W2 represent the weight matrix, and b1 and b2 represent the bias matrix.

[0107] The output of the encoding block X output and the input X of the encoding block embedded The dimensions are consistent, so that a preset number of coding blocks can be directly connected end to end.

[0108] Three-dimensional manual feature matrix branch:

[0109] In this embodiment, the three-dimensional hand-made feature matrix branch uses four convolution blocks to extract high-dimensional features from the input three-dimensional hand-made feature matrix (such as input_1 in Table 1, with a dimension of 7*7*15) (passing through four convolution blocks in sequence, such as two-dimensional convolution_1, two-dimensional convolution_2, two-dimensional convolution_3 and two-dimensional convolution_4 in Table 1, and the dimensions of the outputs after the four convolution blocks are 7*7*32, 7*7*64, 7*7*128 and 7*7*32 respectively), and each convolution block contains One convolutional layer, one batch normalization layer and a random dropout layer; the convolution kernel size of the convolution layer in the first convolution block is 1*1, and the dropout rate of the random dropout layer is 0.2; the convolution kernel size of the convolution layer in the second, third, and fourth convolution blocks is 3*3, and the dropout rate of the random dropout layer is 0.2; the maximum pooling layer (such as the two-dimensional maximum pooling in Table 1, the dimension of the output after this layer is 7*7*32) is used to reduce the dimension of the output of the fourth convolution block, and the pooling size is 2*2.

[0110] The output of each branch (the output of the preprocessed EEG signal branch and the output of the three-dimensional manual feature matrix branch) is flattened into a one-dimensional vector (the output of the three-dimensional manual feature matrix branch is flattened into a one-dimensional vector as shown in Flatten_1 in Table 1, and the flattened dimension is 1568; the output of the preprocessed EEG signal branch is flattened into a one-dimensional vector as shown in Flatten_3 in Table 1, and the flattened dimension is 13440), and the two one-dimensional vectors are merged and connected into one one-dimensional vector (such as the merge in Table 1, and the output dimension is 15008); finally, the merged and connected one-dimensional vector is input into the fully connected layer (such as the fully connected_1 in Table 1, and the output dimension is 512) and the Softmax layer (such as the fully connected_2 in Table 1, and the output dimension is 2) in sequence to obtain the prediction result:

[0111] In this embodiment, the fully connected layer has 512 neurons, and the softmax layer has 512 neurons.

[0112] During the training of the spatiotemporal deep learning model, the accuracy of the model can be improved by properly setting the model's training hyperparameters. The specific steps are as follows:

[0113] The Adam algorithm is used as the optimizer, with a learning rate of 0.01 and a weight decay of 1e-6;

[0114] The maximum number of iterations is 100;

[0115] The batch size is 64;

[0116] The binary cross entropy function is used as the loss function of the spatiotemporal deep learning model. The loss function Loss calculation formula is as follows:

[0117]

[0118] Among them, N' represents the number of samples for training the spatiotemporal deep learning module, y u represents the binary label of the u-th sample, p(y u ) indicates that the output of the spatiotemporal deep learning module belongs to the binary label y u probability.

[0119] Step 4: Finally, the EEG signal is input into the trained spatiotemporal deep learning model to obtain the EEG signal recognition result.

[0120] It should be noted that this method is applicable to a variety of different scenarios. For example, taking pain recognition as an example, the recognition result of the EEG signal is whether there is pain (i.e. pain or no pain); if in scenarios such as depression, epilepsy, Parkinson's disease, etc., the recognition result is whether symptoms such as depression, epilepsy, Parkinson's disease, etc. occur.

[0121] The present invention combines the spatial distribution of brain electrodes of EEG signals with manual features, and more comprehensively mines the valuable information contained in EEG signals than the existing technology, and more fully combines prior knowledge, thereby significantly improving the accuracy of EEG signal recognition.

[0122] In one embodiment, a device for EEG signal recognition based on spatiotemporal deep learning is also proposed, comprising a processor and a memory storing a plurality of computer instructions. When the computer instructions are executed by the processor, the steps of the EEG signal recognition method based on spatiotemporal deep learning are implemented.

[0123] It should be noted that the specific limitations of the EEG signal recognition device based on spatiotemporal deep learning can be found in the above limitations of the EEG signal recognition method based on spatiotemporal deep learning, which will not be repeated here.

[0124] Evaluating spatiotemporal deep learning models:

[0125] The constructed spatiotemporal deep learning model was evaluated using cross-subject 10-fold cross-validation. The subjects were randomly shuffled and evenly divided into ten groups. Eight of these groups served as the training set for the spatiotemporal deep learning model, one served as the validation set, and one served as the test set. During training, the best performing model on the validation set was saved and tested on the test set, with the EEG signal recognition accuracy reported.

[0126] To prove the effectiveness and innovation of the present invention, as shown in Table 2:

[0127] Table 2

[0128]

[0129] For the preprocessed EEG signal branch and the three-dimensional manual feature matrix branch, the accuracy and F1 scores of the present invention and other existing commonly used machine learning algorithms were compared. The preprocessed EEG signal branch (using multi-head attention) was compared with the traditional one-dimensional convolutional neural network, and the three-dimensional manual feature matrix branch (using two-dimensional convolution) was compared with the existing support vector machine, random forest, and K-nearest neighbor algorithm. The combination of the preprocessed EEG signal branch and the three-dimensional manual feature matrix branch (using multi-head attention + two-dimensional convolution) was compared with the existing one-dimensional convolution + two-dimensional convolution. The accuracy and F1 scores of the present invention are both high, which reflects the rationality of the present invention.

[0130] In one embodiment, two electrode reduction schemes for the EEG signal-based pain recognition method are as follows:

[0131] In this embodiment, a gradient-weighted class activation heat map (Grad-CAM) is used to measure the contribution of each EEG channel to pain recognition. The calculation of the gradient-weighted class activation heat map mainly uses the feature map output by the last convolution block in the three-dimensional manual feature matrix branch (i.e., the fourth convolution block, 2D convolution_4 in Table 1), because it contains the most representative high-dimensional features. The specific calculation formula is as follows:

[0132]

[0133]

[0134] Among them, A represents the feature map output by the last convolution block in the three-dimensional hand-crafted feature matrix branch, k represents the k-th channel of the feature map A, and c represents the category. Represents weight, ReLU represents nonlinear activation function, represents the global average pooling operation, Indicates that the gradient is calculated by backpropagation.

[0135] The 21 brain electrodes involved in collecting the original EEG signals are divided into regions. Figure 4 As shown in Figure 2; Based on this division, the gradient-weighted class activation heat map is used to evaluate the contribution of different regions in a targeted manner, focusing on the 15-electrode combination, 9-electrode combination, 9-electrode combination, 6-electrode combination, and 6-electrode combination. The class activation heat maps obtained under different electrode combinations are shown in Figure 2. Figure 5 As shown, Figure 5 (a) reflects the contribution of 21-channel EEG signals; Figure 5 (a) The contribution of 15 electrodes in the frontal lobe, temporal lobe, middle lobe, and parietal lobe is relatively higher. These 15 channels can be used as the first electrode reduction scheme. The class activation heat map is calculated for this. The results are as follows Figure 5 (b) shown; comprehensive Figure 5 (a) and Figure 5 (b) The left hemisphere of the frontal lobe, temporal lobe, middle lobe, and parietal lobe have a relatively higher contribution, so these 9 electrodes are used as the second electrode reduction scheme, and the class activation heat map is calculated for this. The results are as follows Figure 5 (c) Figure 5 (d) Figure 5 (e) and Figure 5 (f) Shows the corresponding electrode contributions in the 9-electrode combination, 6-electrode combination, and 6-electrode combination.

[0136] In order to quantitatively compare the accuracy of children's automatic pain recognition under different electrode reduction schemes, as shown in Table 3:

[0137] Table 3

[0138]

[0139] The recognition accuracy and F1 score of the constructed spatiotemporal deep learning model (21 electrodes) were compared for five different electrode reduction combinations. The results showed that the spatiotemporal deep learning model (21 electrodes) had the highest accuracy and F1 score, reflecting the advanced nature and effectiveness of the present invention.

[0140] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The above-described embodiments merely represent specific and detailed examples of the present application and should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for EEG signal recognition based on spatiotemporal deep learning, characterized by: The EEG signal recognition method based on spatiotemporal deep learning includes: Preprocessing the collected original EEG signal to obtain a preprocessed EEG signal; Calculate the manual features of the preprocessed EEG signal and build a three-dimensional manual feature matrix based on the spatial distribution of the brain electrodes of the original EEG signal; Build a spatiotemporal deep learning model, use the preprocessed EEG signals and the three-dimensional hand-crafted feature matrix as input to train the spatiotemporal deep learning model; Finally, the EEG signal is input into the trained spatiotemporal deep learning model to obtain the EEG signal recognition result; The construction of the spatiotemporal deep learning model uses the preprocessed EEG signal and the three-dimensional manual feature matrix as inputs of the spatiotemporal deep learning model, and trains the spatiotemporal deep learning model, including: The spatiotemporal deep learning module includes a preprocessed EEG signal branch, a three-dimensional manual feature matrix branch, a fully connected layer, and a softmax layer; The preprocessed EEG signal branch uses multiple convolutional layers to extract high-dimensional features from the input preprocessed EEG signal, flattens the extracted high-dimensional features into one-dimensional vectors, performs input embedding and position encoding, and uses a preset number of encoding blocks to extract features from the data after position encoding. Each encoding block includes a multi-head attention layer, a first normalization layer, a forward feedback layer, and a second normalization layer; The three-dimensional hand-crafted feature matrix branch uses a preset number of convolution blocks and maximum pooling layers to extract high-dimensional features from the input three-dimensional hand-crafted feature matrix, and each convolution block includes a convolution layer, a batch normalization layer, and a random dropout layer; Flatten the output of each branch into a one-dimensional vector, and merge the two one-dimensional vectors into a one-dimensional vector; Finally, the merged and connected one-dimensional vector is sequentially input into the fully connected layer and the Softmax layer to obtain the prediction result; Calculate the loss function, perform backpropagation, and complete the training of the spatiotemporal deep learning model; The binary cross entropy function is used as the loss function of the spatiotemporal deep learning model, and the loss function The calculation formula is as follows: ； in, represents the number of samples for training the spatiotemporal deep learning module, Indicates the The binary labels of samples, Indicates that the output of the spatiotemporal deep learning module belongs to a binary label probability.

2. The method for EEG signal recognition based on spatiotemporal deep learning according to claim 1, wherein: The preprocessing of the collected original EEG signal to obtain the preprocessed EEG signal includes: The original EEG signal is processed in sequence through filtering, downsampling, independent component analysis algorithm, normalization and sliding window segmentation to obtain the preprocessed EEG signal.

3. The method for EEG signal recognition based on spatiotemporal deep learning according to claim 1, wherein: The method of calculating manual features of the preprocessed EEG signal and establishing a three-dimensional manual feature matrix based on the spatial distribution of the brain electrodes of the original EEG signal includes: The preprocessed EEG signals are respectively calculated with manual features in time domain, manual features in frequency domain and manual features of nonlinear entropy; Calculate the manual features in the time domain, including calculating the mean, variance, standard deviation, normalized first-order difference and normalized second-order difference of the preprocessed EEG signals to obtain the corresponding manual features; Calculate the manual features in the frequency domain, including calculating the band power of the delta frequency band, theta frequency band, alpha frequency band, beta frequency band and gamma frequency band of the preprocessed EEG signal to obtain the corresponding manual features; Calculate the manual features of nonlinear entropy, including calculating the approximate entropy, sample entropy, Raine entropy, Tsallis entropy and differential entropy of the preprocessed EEG signals to obtain the corresponding manual features; According to the spatial distribution of brain electrodes involved in the original EEG signal, a two-dimensional matrix is ​​constructed, and combined with the calculated manual feature stacking, a three-dimensional manual feature matrix is ​​obtained.

4. A device for EEG signal recognition based on spatiotemporal deep learning, comprising a processor and a memory storing a plurality of computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images