3D interpolation and 3DCNN-based motor imagery task classification method

Abstract

This invention discloses a classification method for motor imagery tasks based on 3D interpolation and 3DCNN. First, bandpass filtering is applied to the motor imagery EEG signals. Then, Fast Fourier Transform (FFT) is used to perform frequency domain transformation on the EEG signals of each electrode, and the power value is calculated. Next, the 3D coordinates of the scalp electrodes are projected into 3D space, and a 3D interpolation algorithm is used to interpolate the power values, generating a 3D interpolated feature image containing the true 3D spatial location information of the electrodes. Finally, a 3D convolutional neural network (3DCNN) is designed to match the characteristics of the 3D interpolated feature image for feature extraction and classification. This invention reflects the depth information of motor imagery activation, encodes the precise three-dimensional spatial information of the electrodes into the 3D interpolated image, and effectively matches the spatial convolution capability of 3DCNN.

Description

Technical Field

[0001] This invention relates to three-dimensional (3D) interpolation and a three-dimensional convolutional neural network (3DCNN) for the field of motor imagery electroencephalography (MI-EEG) signal recognition technology. Specifically, it involves: performing frequency domain transformation on the EEG signal of each lead based on Fast Fourier Transform (FFT) and calculating the power value; combining this with the precise three-dimensional position information of the electrodes; and generating a three-dimensional interpolated image using a three-dimensional interpolation algorithm. Finally, a 3D convolutional neural network (3DCNN) is designed to match the characteristics of the 3D interpolated feature image for feature extraction and recognition, thereby achieving the classification of motor imagery EEG signals. Background Technology

[0002] Brain-computer interfaces (BCIs) enable direct communication between the brain and external devices, completely independent of muscle and peripheral nerve involvement. Non-invasive BCI systems have become a popular research topic due to their simplicity, low cost, and non-invasive nature. Motor imagery electroencephalography (MI-EEG) signals collected using electrode caps are commonly used as input signals for non-invasive BCI systems. By decoding MI-EEG signals, the subject's motor intentions are directly converted into control signals for external devices, finding wide applications in assistive devices, rehabilitation medicine, entertainment, and gaming.

[0003] MI-EEG signals are non-stationary signals with distinct rhythmic characteristics, and the activated three-dimensional spatial information differs depending on the motion imagery task. With the development of deep learning technology, some researchers have proposed various MI-EEG two-dimensional (2D) imaging methods based on frequency domain or time-frequency analysis, and designed two-dimensional convolutional neural networks (2DCNNs) to achieve automatic feature extraction and recognition. The utilization of the two-dimensional spatial information of the electrodes has significantly improved the decoding effect of motion imagery. However, the depth information activated by motion imagery has not yet been reflected, and the precise three-dimensional spatial information of the electrodes cannot be encoded into the MI-EEG image, resulting in incomplete spatial information of the electrodes and affecting the decoding accuracy of MI-EEG. Summary of the Invention

[0004] To address the aforementioned shortcomings, this invention provides a method for classifying motion imagination tasks based on 3D interpolation and 3DCNN.

[0005] (1) The original MI-EEG of all channels was filtered by using a bandpass filter to obtain the frequency band most relevant to the motion visualization task. The signal of each lead was transformed in the frequency domain by Fast Fourier Transform (FFT) and the power value was obtained.

[0006] (2) A three-dimensional mesh is established using the actual 3D spatial coordinates of the electrodes, and the power value is interpolated based on the 3D spatial position of the electrodes and the established mesh. The frequency domain characteristics are extended to three-dimensional space to generate a three-dimensional interpolated image.

[0007] This feature map contains information about the actual spatial location of the electrodes.

[0008] (3) Based on the characteristics of the feature matrix of the three-dimensional interpolated imaging map, a 3DCNN with four cascaded modules was designed for feature matrix extraction and motion imagination task decoding, so that the spatial convolution calculation of 3DCNN is more consistent with the spatial distribution characteristics of MI-EEG signal in a physical sense.

[0009] The specific steps of this invention are as follows:

[0010] Step 1: Preprocessing of EEG signals.

[0011] By performing 8-32Hz bandpass filtering on the EEG signals, the EEG signals most relevant to the motor imagery task were obtained.

[0012] Step 2: Calculate the power value based on Fast Fourier Transform (FFT).

[0013] Step 2.1 Use Fast Fourier Transform (FFT) to extract frequency domain features. This refers to the MI-EEG signal of the m-th lead in a single acquisition experiment, where m represents the lead number, m∈{1,2,3,...,N}. c}, N c N represents the total number of leads. s This represents the number of sampling points included in a single acquisition experiment. The MI-EEG data of the m-th lead can be represented as:

[0014]

[0015] Then, x m Divided into N D N windows D ∈N + N + It is a positive natural number. The data for each window can be represented as:

[0016]

[0017] Where j is the window number, j∈{1,2,3,...,N} D}.so The number of sampling points included is:

[0018]

[0019] The FFT is used to transform the time-domain signal of each window into the frequency domain. To improve the resolution of frequency calculations, the signal sequence of each window is padded with zeros to a length of N. FFT The transformed sequence can be represented as:

[0020]

[0021] Furthermore, the 8-30Hz frequency band sequence corresponding to FOI is represented as follows:

[0022]

[0023] The length of the frequency band sequence is N F Solve using the following formula:

[0024]

[0025] In the formula, F H and F L f represents the upper and lower frequency limits for each sub-band. s It is the sampling frequency.

[0026] Step 2.2 Calculate the power value for each electrode. For each window of the frequency band sequence, the average power is calculated independently as follows:

[0027]

[0028] By N D The average power values ​​in each window are then averaged to obtain the following solution:

[0029]

[0030] Step 3: Generation of 3D interpolated image.

[0031] Step 3.1 Establish a three-dimensional mesh based on the 3D spatial coordinates of the electrodes. The three-dimensional coordinates of the electrodes are obtained from the electrode distribution map of the BCI system. A three-dimensional mesh of size 32×32×32 is established based on the three-dimensional spatial coordinates of the electrodes.

[0032] Step 3.2 Interpolates the power value based on the 3D spatial location of the electrodes and the established grid. The frequency domain characteristics are then extended to three-dimensional space to generate a 3D interpolated image.

[0033] Step 4: Four-module cascaded 3DCNN for feature matrix recognition.

[0034] Step 4.1 Based on the characteristics of 3D interpolated imaging maps, a 3DCNN with four cascaded modules was designed for feature matrix extraction and classification. This four-module cascaded structure effectively decodes the encoded information of motion visualization tasks. Modules 1 and 2 have the same structure: two 3D convolutional layers and one max-pooling layer, with the same kernel size (3×3×3) and stride (3×3×3). Both convolutional layers use ReLU activation. Module 3 contains one 3D convolutional layer and one max-pooling layer, with a 3×3×3 kernel and stride, and ReLU activation. Module 4 contains two fully connected layers that flatten the extracted spatial features and output the category. To avoid overfitting and accelerate training, batch normalization (BN) and dropout techniques are used in all four modules. Detailed network parameters for the designed 3DCNN are shown in Table 1.

[0035] Table 14 M3DCNN Network Structure

[0036]

[0037] Step 4.2 Use 3DCNN to identify the feature matrix of the 3D interpolated imaging map. The feature matrix of the 3D interpolated imaging map obtained from each EEG experiment is concatenated to form a total dataset, which is used for training and testing of 3DCNN to achieve motor imagery task classification.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] (1) The method of the present invention combines the precise three-dimensional spatial position and feature information of the electrodes to preserve the depth information of motor imagery activation and avoid the loss of the spatial position information of the electrodes of motor imagery EEG signals.

[0040] (2) The method of the present invention uses a three-dimensional interpolation algorithm to generate a feature matrix, which combines the frequency domain features with the precise position information of the electrode, thus fully preserving the frequency-space features of the electrode.

[0041] (3) The present invention uses a 3DCNN that matches the characteristics of the feature matrix of the three-dimensional interpolation imaging map, which can fully extract and recognize the features of the feature matrix, and effectively improve the classification effect of EEG signals. Attached Figure Description

[0042] Figure 1 This is a timing diagram of an EEG acquisition experiment.

[0043] Figure 2 This is a distribution diagram of the electrode locations.

[0044] Figure 3This is a slice image after 3D interpolation imaging.

[0045] Figure 4 This is a diagram of the 3DCNN architecture.

[0046] Figure 5 This is a flowchart of the algorithm of the present invention. Detailed Implementation

[0047] Specifically, this invention was carried out in the Matlab R2019a simulation environment and Tensorflow environment under the Windows 10 (64-bit) operating system.

[0048] The dataset used in this invention is Dataset 2a from the "BCI Competition 2008". This dataset records the EEG signals of 9 subjects, collected using 22-conductance electrode caps conforming to the international 10-20 standard, at a sampling frequency of 250Hz. The data was divided into training and testing sets, and each subject performed 576 experiments (144 experiments for each type of task, totaling four types of motor imagery tasks). Figure 1 As shown, the experiment lasted 7.5 seconds. At t=0 seconds, a "+" sign appeared on the screen, accompanied by a short alarm sound, indicating the start of the experiment. At t=2 seconds, indicator arrows appeared on the screen and were held for 1.25 seconds each for left, right, up, or down, corresponding to the imagery tasks of the left hand, right hand, tongue, and foot. From t=3 seconds to t=6 seconds, the subjects completed a 3-second motor imagery task based on prompts. From t=6 to 7.5 seconds, the subjects rested and the next experiment was conducted.

[0049] Based on the above-mentioned motor imagery EEG dataset, the specific implementation steps of this invention are as follows:

[0050] Step 1: Preprocessing of EEG signals.

[0051] Step 1.1 Based on the labels of the four types of motor imagery tasks—left hand (a=1), right hand (a=2), foot (a=3), and tongue (a=4)—extract the single-experiment M for each type of motor imagery task from 9 subjects. a ∈R 22×1875 .

[0052] Step 1.2 Use a Butterworth filter to perform an 8-32Hz bandpass filter on the EEG signal to obtain the EEG signal most relevant to the motor imagery task, denoted as .

[0053] Step 2: Calculate the power value based on Fast Fourier Transform (FFT).

[0054] Step 2.1 Signal Frequency domain features are extracted using Fast Fourier Transform (FFT). The transformed sequence is...

[0055] Step 2.2: Squaring each sampling point of each sequence and then averaging the results yields the average power value.

[0056] Step 3: Generation of 3D interpolated image.

[0057] Step 3.1 Obtain the 3D coordinate information of the electrodes C∈R based on the BCI Competition 2008 system lead distribution diagram. 3x22 Electrode positions as follows Figure 2 As shown. A three-dimensional mesh G∈R of size 32×32×32 is constructed based on the 3D spatial coordinates of the electrodes. 32x32x32 The three-dimensional coordinates of the electrodes are obtained from the electrode distribution map of the BCI system, and a three-dimensional mesh of size is established based on the three-dimensional spatial coordinates of the electrodes.

[0058] Step 3.2 According to C∈R 3x22 Interpolation mapping to G∈R 32x32x32 Within the mesh, frequency domain features are extended to three-dimensional space, resulting in a three-dimensional interpolated image G. M ∈R 32x32x32 Visualization of slices in 3D interpolated images, such as Figure 3 As shown, the concatenated matrix yields a 32x32x32x1 eigenma matrix G. F ∈R 32x32x32x1 As input to the network.

[0059] Step 4 is based on 3DCNN classification.

[0060] Based on the frequency-space characteristics of the 3D interpolated imaging map, a matching 3DCNN was designed for feature extraction and classification. The network structure is as follows: Figure 4 As shown, the average classification accuracy was obtained using 10-fold cross-validation.

[0061] The MI-EEG signals of subjects S1, S2, S3, ..., S9 were processed using the above procedure, and the average recognition accuracy of 10-fold cross-validation is shown in Table 2.

[0062] Table 2. Accuracy of 10-fold cross-validation among nine subjects.

[0063]

Claims

1. A method for classifying motion imagination tasks based on 3D interpolation and 3DCNN, characterized by: First, the raw motor imagery EEG signals (MI-EEG) are bandpass filtered. Then, the EEG signals of each electrode are frequency domain transformed and the power value is calculated. Combined with the precise three-dimensional position information of the electrodes, a three-dimensional interpolation algorithm is used to generate a three-dimensional interpolated image. The three-dimensional interpolated images are then stitched together to form a feature matrix. Finally, a 3D convolutional neural network (3DCNN) is designed to match the characteristics of the feature matrix for feature extraction and recognition, thereby achieving the classification of motor imagery EEG signals. Its features are: Step 1: Preprocessing of the EEG signal; By performing 8-32Hz bandpass filtering on the EEG signals, the EEG signals most relevant to the motor imagery task were obtained. Step 2: Calculate the power value based on Fast Fourier Transform (FFT); Step 2.1 Use Fast Fourier Transform (FFT) to extract time-frequency features; This refers to the MI-EEG signal of lead m in a single acquisition experiment, where m represents the lead number, m∈{1,2,3,...,N}. c } represents the total number of leads; N s This represents the number of sampling points included in a single acquisition experiment; the MI-EEG data of the m-th lead can be represented as: Then x m Divided into N D N windows D ∈N + The data for each window can be represented as: Where j is the window index, j∈{1, 2, 3, ..., N} D };so The number of sampling points included is: FFT is used to transform the time-domain signal of each window into the frequency domain; to improve the resolution of frequency calculation, the signal sequence of each window is padded with zeros to a length of N. FFT The transformed sequence is represented as: Furthermore, the 8-30Hz frequency band sequence corresponding to FOI is represented as follows: The length of the frequency band sequence is N F Solve using the following formula: In the formula, F H and F L f represents the upper and lower frequency limits for each sub-band. s It is the sampling frequency; Step 2.2 Calculate the power value for each electrode; for each window of the frequency band sequence, the average power is calculated independently as follows: By N D The average power values ​​in each window are then averaged to obtain the following solution: Step 3: Generation of the 3D interpolated image; Step 3.1 Establish a three-dimensional mesh based on the 3D spatial coordinates of the electrodes; the three-dimensional coordinates of the electrodes are obtained from the electrode distribution map of the BCI system, and a three-dimensional mesh of size 32×32×32 is established based on the three-dimensional spatial coordinates of the electrodes. Step 3.2 Interpolate the power value based on the 3D spatial position of the electrode and the established grid; extend the time-frequency characteristics to three-dimensional space to generate a three-dimensional interpolated image; Step 4: Four-module cascaded 3DCNN for feature matrix recognition; Step 4.1 Based on the characteristics of 3D interpolated imaging maps, a 3DCNN with four cascaded modules was designed for feature matrix extraction and classification. This four-module cascaded structure effectively decodes the encoded information of motion visualization tasks. Modules 1 and 2 have the same structure: two 3D convolutional layers and one max-pooling layer, with identical 3×3×3 kernel sizes and strides. Both convolutional layers use ReLU activation functions. Module 3 contains one 3D convolutional layer and one max-pooling layer, with a 3×3×3 kernel size and a 3×3×3 stride. The activation function of the convolutional layer is ReLU. Module 4 contains two fully connected layers that flatten the extracted spatial features and output the category. To avoid overfitting and accelerate network training, batch normalization and Dropout techniques are used in all four modules. Step 4.2 Use 3DCNN to identify the feature matrix of the three-dimensional interpolated imaging map; the feature matrix of the three-dimensional interpolated imaging map obtained from each EEG experiment is spliced ​​together to form a total dataset, which is used for training and testing of 3DCNN to achieve motor imagery task classification.

Images