Electroencephalogram decoding method based on brain map filter and multi-scale adaptive convolutional network

Abstract

The invention discloses an electroencephalogram decoding method based on an electroencephalogram filter and a multi-scale adaptive convolutional network, which comprises the following steps of: firstly, carrying out preprocessing such as data standardization on an original electroencephalogram signal, establishing a function connection relation graph between different electroencephalogram electrodes according to a phase-locked value (PLV), and constructing the electroencephalogram filter based on the function connection relation graph; according to the method, mixed information in electroencephalogram signals can be effectively filtered out, the resolution of the electroencephalogram signals is enhanced, then a multi-scale adaptive convolutional network based on an attention mechanism is designed, one-dimensional convolution kernels of different lengths are adopted to extract relative global and local time features in the electroencephalogram signals, and meanwhile, due to physiological factors, the time features of the electroencephalogram signals are extracted. According to the significant difference of electroencephalogram signals among different subjects, an attention mechanism is adopted to weight features extracted from different convolution kernel lengths so as to adaptively select a feature extraction scheme adaptive to the electroencephalogram signals of different subjects, the generalization ability of a model in a cross-subject scene is improved, then the electroencephalogram signals are subjected to convolution in a channel dimension, and the convolution performance of the electroencephalogram signals is improved. And finally, inputting the extracted features into a full connection layer to obtain a classification result of the electroencephalogram signals.

Description

Technical Field

[0001] The present invention belongs to the field of electroencephalogram (EEG) signal intention decoding, and particularly relates to an EEG decoding method based on an EEG graph filter and a multi-scale adaptive convolutional network. Background Art

[0002] A brain-computer interface (BCI) is a technology that can convert electrical signals captured from the human cerebral cortex into computer instructions, thereby realizing information interaction with external devices. There are tens of billions of neurons in the human brain, and a large number of neuron cells communicate with each other by generating electric currents. By decoding the EEG signals generated by nerve cells, the intentions of the subjects can be obtained. The BCI system has now been widely applied in many fields such as movement assistance, neurorehabilitation, and disease diagnosis.

[0003] Currently, the intention decoding methods for EEG signals are mainly divided into traditional machine learning methods and deep learning methods. The decoding process of traditional methods is divided into two parts: feature extraction and feature classification. Common traditional EEG signal feature extraction methods include autoregressive components (AR), principal component analysis (PCA), and independent component analysis (ICA), etc. AR extracts characteristic parameters reflecting dynamic characteristics by establishing a linear relationship between EEG signals and their historical data; PCA and ICA respectively achieve data dimensionality reduction and signal separation by maximizing variance and independence. Traditional EEG signal classification methods include linear discriminant analysis (LDA), support vector machine (SVM), and Bayesian statistical classifier. LDA finds a hyperplane for classification by estimating the means and covariances of each class and using a linear discriminant function; SVM selects a hyperplane by maximizing the margin and uses a kernel function to handle non-linear problems; the Bayesian classifier calculates the posterior probability based on Bayes' theorem and assumes that the data conforms to a Gaussian distribution. However, since traditional methods rely on prior knowledge of EEG signals and require manual design and extraction of specific features, it is difficult to capture all the key information in EEG signals. Deep learning methods can automatically extract complex and high-level features from raw data, reducing the dependence on artificial settings.

[0004] Common deep learning-based electroencephalogram (EEG) signal decoding methods include backpropagation neural network (BPNN), recurrent neural network (RNN), and convolutional neural network (CNN), etc. BPNN uses gradient descent and error backpropagation algorithms to continuously optimize parameters for training and learning; RNN realizes the memory and transmission of historical information and establishes time-dependent relationships through the joint calculation of hidden states and current inputs; CNN extracts multi-level features through the local perception and weight sharing mechanisms of sliding convolutional kernels. However, due to the characteristics of EEG signals such as non-linearity, non-stationarity, and low signal-to-noise ratio, there are still many difficulties in the decoding research of EEG signals, and there is still much room for improvement in the performance of the model.

[0005] To address the above problems, the present invention proposes an EEG decoding method based on a brain map filter and a multi-scale adaptive convolutional network. It can effectively extract the features of EEG signals, enhance the resolution of EEG signals, and improve the decoding performance of the model. Summary of the Invention

[0006] The present invention aims to solve the problems of the above prior art. It proposes an EEG decoding method based on a brain map filter and a multi-scale adaptive convolutional network. The technical solution of the present invention is as follows:

[0007] An EEG decoding method based on a brain map filter and a multi-scale adaptive convolutional network, which includes the following steps:

[0008] Step 1: Collect motor imagery EEG data and label different motor imagery signal paradigms.

[0009] Step 2: Divide the EEG data obtained in Step 1 into a training set and a test set. The training set is used to complete the training of the classification model, and the test set is used to verify the classification performance of the model. At the same time, preprocessing operations such as band-pass filtering and normalization are performed on the EEG signals to enhance the signal quality.

[0010] Step 3: Establish a functional connection map between different EEG electrodes according to PLV.

[0011] Step 4: Construct a brain map filter based on the functional connection map to filter out the confounding information in the EEG signals and improve the overall resolution of the signals.

[0012] Step 5: Extract the relatively global and local temporal features in the EEG signals using one-dimensional convolutional kernels of different scales, and adaptively select the signal features extracted by convolutional kernels of different scales based on the attention mechanism, and weight them to enhance the feature representation of the EEG signals.

[0013] Step 6: Extract spatial features between different channels using a spatial convolution kernel, and input the extracted features into a fully connected layer to complete the feature classification of the signal.

[0014] 2. A brain decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1, characterized in that, in the said step 1: an electroencephalogram (EEG) acquisition device is used to acquire four different types of EEG signals, with a frequency of 250 Hz. Each category paradigm includes 4 s of motor imagery signals, and there is a 2 s rest interval after each acquisition. Then the next trial is acquired, and there is a 2 s cue time for the corresponding category before each acquisition. After acquisition, the EEG signals of different categories are labeled.

[0015] 3. A brain decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1, characterized in that, in the said step 2, the EEG signal data is divided, and preprocessing operations of band-pass filtering and normalization are performed on the signals, specifically including: dividing the acquired EEG data into a training set and a test set according to 3:1 for training and performance detection of the model. A sixth-order Butterworth filter is used to filter the EEG signals at 0.5 - 40 Hz to remove high-frequency noise and low-frequency interference in the signals. A Z-score normalization operation is performed on the band-pass filtered EEG signals to eliminate the influence of different dimensions in the data and optimize the distribution difference of the data.

[0016] 4. A brain decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1, characterized in that, in the said step 3, the phase locking value calculates the strength of phase synchronization or locking between different electrode channels. To obtain the phase locking value, the Hilbert transform is performed on the EEG signals to convert them from the time domain to the frequency domain signal to obtain phase information. The defined formula is as follows:

[0017]

[0018] where is the phase difference between channels x and y at time point t, and N is the data length. Further, the connection strength between the functional connection edges between different electrodes obtained is screened, and the connection edges with weights greater than the average value of all connection edge weights are retained, and a brain functional connection graph G=(V, E) is established based on this. V is the nodes in the graph corresponding to the EEG electrodes, and E is the set of edges in the graph corresponding to the retained electrode connection edges. The connection edge weight between any two nodes is:

[0019]

[0020] where is the PLV connection edge weight calculated between nodes i and j.

[0021] 5. A brain signal decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1. It is characterized in that the steps for constructing a brain map filter in step 4 include: based on the Laplacian matrix of the brain functional connection graph G constructed in step 3, which is defined as:

[0022] L = D - A (3)

[0023] where D = {d(v1), 0,..., d(v i ), 0,..., d(v c )} is the degree matrix of graph G, d(v i ) is the degree of node i, and A is the adjacency matrix of graph G, which is composed of the weights of the connecting edges between the obtained nodes. The Laplacian operator represents the local rate of change of each point in a scalar field affected by the sources in the field. The Laplacian matrix is the representation of the Laplacian operator in the graph structure. Using the Laplacian matrix can remove the confounding information contained between electrode nodes. To ensure that the sum of the weights of all adjacent edges around a single electrode is 1, the Laplacian matrix is normalized by random walk: L rw = D -1 L = I - D -1 A, and the elements in the obtained L rw are:

[0024]

[0025] L rw is the brain map filter, and the signal filtered by the brain map filter can be expressed as X filter = L rw X.

[0026] 6. A brain signal decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1. It is characterized in that the steps of the multi-scale attention adaptive convolutional network in step 5 include:

[0027] First, one-dimensional convolutional kernels of different scale sizes are used to extract the relatively global and local temporal features in the EEG signal. The sizes of the one-dimensional convolutional kernels are 1×20, 1×60, and 1×80 respectively. Then, the dimension of the obtained feature map is compressed through max pooling, and Dropout is used after max pooling to suppress overfitting during model training.

[0028] Next, the three groups of obtained features are respectively expanded in a single dimension and concatenated together in the channel dimension. The concatenated features are input into the SE channel attention mechanism module, and each channel, that is, the importance weight of the scale, is learned through global average pooling and a fully connected layer, and the weight coefficient is output. Finally, the obtained weights are used to perform weighted fusion on the original three groups of features to adaptively weight the features from different-scale convolutional kernels and find the optimal scale combination. The output data of each channel is:

[0029] x c =F se (u c ,w c )=u c w c (5)

[0030] Among them, x c is the output data after weighting a single channel, u c is the input data, and w c is the weight coefficient.

[0031] 7. A brain decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1. It is characterized in that in step 6, the feature maps weighted by the channel attention mechanism are concatenated according to the channel dimension, and then spatial convolution is used to extract the spatial features between different electrode channels of the electroencephalogram. The extracted electroencephalogram features are input into a fully connected layer to complete the final signal category classification, and the loss function used is the cross-entropy loss function:

[0032]

[0033] Among them, C represents the number of classification categories, y ij represents the true label of the data sample, and x ij represents the predicted probability of the i-th data sample in category j.

[0034] The advantages and beneficial effects of the present invention are as follows:

[0035] The present invention uses the phase-locked value to calculate the connection edge strength between different electrodes, establishes a functional connection relationship diagram between electroencephalogram electrodes, and can accurately reflect the mutual relationship between electrodes. On this basis, a brain map filter is constructed by combining the Laplace operator to suppress the confounding information in the electroencephalogram signal. Compared with the traditional method of directly using the original electroencephalogram data, the electroencephalogram signal processed by the brain map filter has enhanced signal resolution and improved the overall decoding accuracy of the model.

[0036] Other advantages of the present invention include the adoption of multi-scale one-dimensional temporal convolution, which can fully extract relatively global and local temporal features in EEG signals. In view of the individual differences existing in EEG signals, an attention mechanism is further adopted to weight the features extracted by convolution kernels of different scales, adaptively select a suitable combination of convolution scales, and can more effectively extract the temporal features of EEG signals, enhancing the generalization ability of the model in the cross-subject scenario. Description of the Drawings

[0037] Figure 1 It is a flowchart of an EEG decoding method based on an EEG map filter and a multi-scale adaptive convolution network provided by the present invention.

[0038] Figure 2 It is a model based on an EEG map filter and a multi-scale adaptive convolution network. Detailed Embodiments

[0039] Next, the technical solutions in the embodiments of the present invention will be clearly and detailedly described in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only a part of the embodiments of the present invention.

[0040] The technical solution for the present invention to solve the above technical problems is as follows:

[0041] As Figure 1 shown, the present invention provides an EEG decoding method based on an EEG map filter and a multi-scale adaptive convolution network, including the following steps:

[0042] Step 1: EEG data of different categories can be collected using an EEG device of Brain Products company, with a frequency of 250 Hz. Each category paradigm includes 4 s of motor imagery signals. After collection, there will be a 2 s rest interval, and then the next trial will be collected. There will be a 2 s prompt time for the corresponding category before each collection. After collection, the EEG signals of different categories will be labeled.

[0043] Subsequently, the collected EEG data is divided into a training set and a test set according to a ratio of 3:1 for model training and performance detection. A sixth-order Butterworth filter is used to filter the EEG signals at 0.5 - 40 Hz to remove high-frequency noise and low-frequency interference in the signals. A Z-score normalization operation is performed on the band-pass filtered EEG signals to eliminate the influence of different dimensions in the data and optimize the distribution difference of the data. The formula is as follows:

[0044]

[0045] where x t is the data point at time point t, u is the mean of the sample data, δ is the standard deviation of the sample data, and z tis the data at time point t after normalization.

[0046] Step 2: Calculate the connection edge strength between different electrode channels using the phase locking value. To obtain the phase locking value, perform a Hilbert transform on the EEG signal to convert it from the time domain to the frequency domain signal, obtain the phase information, and define the formula as follows:

[0047]

[0048] where is the phase difference between channels x and y at time point t, and N is the data length. Further screen the connection strength between the functional connection edges of different electrodes obtained, retain the connection edges whose weights are greater than the average value of all connection edge weights, and establish a brain functional connection graph G=(V, E) based on this. V is the node in the graph corresponding to the EEG electrode, and E is the set of edges in the graph corresponding to the retained electrode connection edges. The weight of the connection edge between any two nodes is:

[0049]

[0050] where is the PLV connection edge weight calculated between node i and node j.

[0051] Step 3: Construct the Laplacian matrix of the brain functional connection graph G, which is defined as:

[0052] L = D - A (4)

[0053] where D = {d(v1), 0,..., d(v i ), 0,..., d(v c )} is the degree matrix of graph G, d(v i ) is the degree of node i, and A is the adjacency matrix of graph G, which is composed of the connection edge weights between the obtained nodes. The Laplacian operator represents the local change rate of each point in a scalar field affected by the sources in the field. The Laplacian matrix is the representation of the Laplacian operator in the graph structure. Using the Laplacian matrix can remove the confounding information contained between electrode nodes. To ensure that the sum of the weights of all adjacent edges around a single electrode is 1, perform random walk normalization on the Laplacian matrix: L rw = D -1 L = I - D -1 A, and the elements in the obtained L rw are:

[0054]

[0055] L rw is the brain graph filter, and the signal filtered by the brain graph filter can be expressed as X filter = L rw X.

[0056] Step 4: Construct a multi-scale attention adaptive convolutional network. First, extract the relatively global and local temporal features in the EEG signals through one-dimensional convolutional kernels of different scale sizes. The sizes of the one-dimensional convolutional kernels are 1×20, 1×60, and 1×80 respectively. The calculation formula for one-dimensional convolution is:

[0057]

[0058] where, is the j-th feature map corresponding to the c-th convolutional layer, is the connection weight established between the j-th feature of the c-th layer and the i-th feature of the c-1-th layer, * represents the convolution operation, b is the bias, M j is the set of input features, and f is the exponential linear unit (ELU) activation function. Then, compress the dimension of the obtained feature map through max pooling, and use Dropout after max pooling to suppress overfitting during model training.

[0059] Next, expand the three groups of obtained features separately in a single dimension and concatenate them together in the channel dimension. The concatenated features are input into the SE channel attention mechanism module, and each channel, that is, the importance weight of the scale, is learned through global average pooling and a fully connected layer, and the weight coefficient is output. Finally, use the obtained weight to perform weighted fusion on the original three groups of features to adaptively weight the features from different scale convolutional kernels and find the optimal scale combination. The output data for each channel is:

[0060] x c =F se (u c ,w c ) = u c w c (7)

[0061] where, x c is the output data of a single channel after weighting, u c is the input data, and w c is the weight coefficient.

[0062] Step 5: Concatenate the feature maps weighted by the channel attention mechanism along the channel dimension, and then use spatial convolution to extract the spatial features between different electrode channels of the EEG. The extracted EEG features are input into a fully connected layer, and the loss function used is the cross-entropy loss function:

[0063]

[0064] where, C represents the number of classification categories, y ij represents the true label of the data sample, and x ijIt represents the predicted probability of the i-th data sample for class j. Finally, the classification result is obtained in the fully connected layer.

[0065] The systems, devices, modules or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0066] Computer-readable media includes both permanent and non-permanent, removable and non-removable media and can be implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0067] It should also be noted that the term "comprises", "comprising", or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, commodity, or device that comprises a series of elements includes not only those elements but also other elements not expressly listed, or elements that are inherent to such process, method, commodity, or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, commodity, or device that comprises the element.

[0068] The above embodiments should be understood as being only for illustrative purposes of the present invention and not for limiting the protection scope of the present invention. After reading the content recorded in the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A brain decoding method based on a mind map filter and a multi-scale adaptive convolutional network, characterized in that, It includes the following steps: Step 1: Collect electroencephalogram (EEG) data of motor imagery, and label different motor imagery signal paradigms; Step 2: Divide the EEG data obtained in Step 1 into a training set and a test set. The training set is used to complete the training of the classification model, and the test set is used to verify the classification performance of the model. At the same time, preprocessing operations such as band-pass filtering and normalization are performed on the EEG signals to enhance the signal quality; Step 3: Establish a functional connectivity map between different EEG electrodes according to the phase locking value (PLV); Step 4: Construct a brain map filter based on the functional connectivity map to filter out the mixed information in the EEG signals and improve the overall resolution of the signals; Step 5: Extract relatively global and local temporal features in the EEG signals using one-dimensional convolutional kernels of different scales, and adaptively select the signal features extracted by the convolutional kernels of different scales based on the attention mechanism, and weight them to enhance the feature representation of the EEG signals; Step 6: Extract spatial features between different channels using a spatial convolutional kernel, and input the extracted features into a fully connected layer to complete the feature classification of the signals.

2. The brain decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1, wherein In Step 1: An EEG acquisition device is used to collect EEG signals of different categories. The sampling frequency is 250 Hz, and each category paradigm includes 4 s of motor imagery signals. There is a 2 s rest interval after each acquisition, and then the next trial is collected. There is a 2 s prompt time corresponding to the category before each acquisition. After the acquisition, the EEG signals of different categories are labeled.

3. A brain decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1, characterized in that, In Step 2, the EEG signal data is divided, and preprocessing operations such as band-pass filtering and normalization are performed on the signals, which specifically include: dividing the collected EEG data into a training set and a test set according to a ratio of 3:1 for model training and performance detection. A sixth-order Butterworth filter is used to filter the EEG signals at 0.5 - 40 Hz to remove high-frequency noise and low-frequency interference in the signals. A Z-score normalization operation is performed on the band-pass filtered EEG signals to eliminate the influence of different dimensions in the data and optimize the distribution difference of the data.

4. A brain decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1, characterized in that In Step 3, the phase locking value calculates the strength of phase synchronization or locking between different electrode channels. To obtain the phase locking value, the Hilbert transform is performed on the EEG signals to convert them from the time domain to the frequency domain signals to obtain the phase information. The definition formula is as follows: where is the phase difference between channels x and y at time point t, N is the data length. Further, the connection strengths between the functional connection edges between different electrodes obtained are screened, and the edges with weights greater than the mean of all connection edge weights are retained, and a brain functional connection graph G=(V, E) is established with this. V is the node in the graph corresponding to the EEG electrodes, E is the set of edges in the graph corresponding to the retained electrode connection edges, and the weight of the connection edge between any two nodes is: wherein is the calculated PLV connection edge weight between node i and node j.

5. A brain decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1, characterized in that, The steps of constructing a brain map filter in Step 4 include: Based on Step 3, construct the Laplacian matrix of the brain functional connectivity graph G, which is defined as: L = D - A (3) where \(D = \{d(v_1), 0, \ldots, d(v i ), 0, \ldots, d(v c )\}\) is the degree matrix of graph \(G\), \(d(v i )\) is the degree of node \(i\), \(A\) is the adjacency matrix of graph \(G\), which is composed of the weights of the connecting edges between the obtained nodes. The Laplace operator represents the local change rate of each point in a scalar field affected by the sources in the field. The Laplace matrix is the representation of the Laplace operator in the graph structure. Using the Laplace matrix can remove the confounding information contained between electrode nodes. To ensure that the sum of the weights of all adjacent edges around a single electrode is 1, the Laplace matrix is normalized by random walk: \(L rw = D -1 L = I - D -1 A\), and the elements in the obtained \(L rw \) are: L rw Namely, it is the brain map filter, and the signal filtered by the brain map filter can be expressed as X filter = L rw X 6. A brain decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1, characterized in that The steps of the multi-scale attention adaptive convolutional network in Step 5 include: First, extract relatively global and local temporal features in the EEG signals using one-dimensional convolutional kernels of different scales. The sizes of the one-dimensional convolutional kernels are 1×20, 1×60, and 1×80 respectively. Then, compress the dimension of the obtained feature map through max pooling, and use Dropout after max pooling to suppress overfitting during model training; Next, the three sets of obtained features are respectively expanded in a single dimension and concatenated together in the channel dimension. The concatenated features are input into the SE channel attention mechanism module, and each channel, that is, the importance weight of the scale, is learned through global average pooling and a fully connected layer, and the weight coefficient is output. Finally, the obtained weights are used to perform weighted fusion on the original three sets of features to adaptively weight the features from different-scale convolutional kernels and find the optimal scale combination. The output data of each channel is as follows: x c = F se (u c , w c ) = u c w c (5) where x c is the output data after weighting a single channel, u c is the input data, and w c is the weight coefficient.

7. A brain decoding method based on a brain map filter and a multi-scale adaptive convolutional network according to claim 1, characterized in that In step 6, the feature maps weighted by the channel attention mechanism are concatenated according to the channel dimension. Then, spatial convolution is used to extract the spatial features between different electrode channels of the EEG. The extracted EEG features are input into a fully connected layer to complete the final signal category classification. The loss function used is the cross-entropy loss function: Among them, C represents the number of classification categories, and y ij represents the true label of the data sample, and x ij represents the predicted probability of the i-th data sample on the j-th category.

Images