A multi-class motor imagery brain-computer interface decoding method combining EEGNet and FBCSP

Abstract

A multi-class motor imagery brain-computer interface decoding method combining EEGNet and FBCSP includes: 1) segmenting the continuous EEG signals of the subject into single-test signals and using filter banks for sub-band filtering; 2) using the training data of all subjects to perform transfer learning training on EEGNet, and classifying the test data to obtain classification scores for all tasks; 3) based on the OVO strategy, training the test subjects' training data on FBCSP to obtain spatial filters for each sub-band and each pair of tasks, concatenating the training and test feature signals of all sub-bands for each pair of tasks after spatial filtering, using the former to train an SVM to classify the latter to obtain classification scores for each pair of tasks, and summing all scores for each task; 4) standardizing the score vectors obtained from EEGNet and FBCSP and summing them, with the maximum value corresponding to the task as the test signal label. This invention can improve the classification accuracy of BCI and promote its practical application.

Description

Technical Field

[0001] This invention belongs to the field of neuroscience technology, specifically relating to brain-computer interface technology and electroencephalogram (EEG) signal recognition technology. Background Technology

[0002] Brain-computer interfaces (BCIs) monitor a user's brain activity, interpret their intentions, and translate those intentions into external commands. The initial purpose of BCIs was to help people with disabilities and neurological disorders regain their ability to live and communicate independently, thereby improving their quality of life. With continuous technological development and advancements, BCIs can now also serve healthy individuals with normal motor function, such as through BCI-based video games. BCIs convert collected brain signals into commands that drive external devices, replacing the human limbs or speech organs to achieve communication with the outside world and control of the external environment.

[0003] Motor imagery (MI) is one of the most widely used cognitive tasks in BCI system design. MI refers to motor behavior imagined in the brain without any visibly performed action. During MI, μ-rhythm (8-12Hz) and β-rhythm (18-26Hz) electroencephalography (EEG) signals can be recorded in the sensorimotor cortex of the brain. When a user imagines a limb movement, the amplitude of the contralateral μ / β rhythm decreases; this electrophysiological phenomenon is called event-related desynchronization (ERD). When the user completes the imagined movement, the amplitude of the μ / β rhythm increases; this is called event-related synchronization (ERS). The spatial distribution of ERD / ERS in the brain varies significantly across different MI tasks and can be measured using various algorithms to recognize EEG signals and user intentions.

[0004] For decades, BCI research has primarily employed traditional machine learning algorithms to classify and identify EEG signals. Common Spatial Pattern (CSP) is a commonly used MI-BCI classification algorithm. It decomposes the raw EEG signal into spatial patterns, maximizing the power difference between two types of EEG signals (e.g., left-hand and right-hand motor imagery) in the μ and β rhythm bands. To improve the classification performance of MI-BCI, researchers have made various improvements to the CSP algorithm. Among these, Filter Bank CSP (FBCSP) is one of the most effective algorithms. It uses a filter bank to decompose the EEG signal into multiple sub-bands, extracts features using CSP in each sub-band, and then fuses the feature signals from each sub-band to classify the EEG signal.

[0005] In recent years, with the rapid development of deep learning technology, researchers have begun to use convolutional neural networks (CNNs) for feature extraction and classification of EEG signals. The compact convolutional neural network (EEG-based convolutional neural network, EEGNet) is a general-purpose deep learning model for BCI (Browser Interaction Center) systems, applicable to EEG signal decoding in various BCI systems, including MI-BCI. EEGNet combines feature extraction and pattern classification into a single network for simultaneous learning, avoiding the complex step of manual feature extraction. It achieves high classification performance even with a limited number of training samples.

[0006] While significant progress has been made in MI-BCI research, its classification accuracy still needs further improvement for many practical applications. FBCSP and EEGNet each have their own advantages and limitations. The former can extract discriminative feature signals but has relatively weak learning capabilities; the latter can model complex EEG signals but requires a large amount of labeled data. In particular, FBCSP struggles to effectively utilize information from other subjects, which is precisely EEGNet's strength. Therefore, effectively integrating these two methods and leveraging their advantages holds promise for improving the classification performance of MI-BCI. This invention combines the FBCSP algorithm and the EEGNet model to classify multiple classes of MI signals. Even with a small training set, it significantly improves the classification performance of BCI, promoting the practical application of MI-BCI. Summary of the Invention

[0007] The purpose of this invention is to provide a multi-class motor imagery brain-computer interface decoding method that combines EEGNet and FBCSP. Its feature is that it can significantly improve the classification accuracy of motor imagery brain-computer interfaces under the condition of a small number of training set samples, thereby promoting the practical application of motor imagery-based BCI systems.

[0008] This invention is achieved through the following technical solutions.

[0009] The present invention discloses a multi-class motor imagery brain-computer interface decoding method combining EEGNet and FBCSP, comprising the following steps:

[0010] Step 1: Data Preprocessing. The continuous EEG signal of each subject is segmented into single-experiment EEG signals, and sub-band filtering of the single-experiment EEG signals is performed using a filter bank;

[0011] Step 2: EEG signal classification strategy. The experimental data for each subject are divided into a training set and a test set, and each subject is used as a test subject only once using the leave-one-out method.

[0012] Step 3: EEGNet Model Training and Classification. The EEGNet model is trained using transfer learning. The trained model is then used to classify the test data, obtaining the feature score vectors (i.e., the classifier's values ​​before the output labels) for a single test experiment across all motion visualization tasks (i.e., all classes).

[0013] Step 4: FBCSP Model Training and Classification. The FBCSP algorithm is trained using a one-versus-one (OVO) strategy to obtain the spatial filter for each sub-band and each task pair. Spatial filtering is applied to the training and test data to obtain the training and test feature signals for each sub-band and each task pair, respectively. The training and test feature signals from all sub-bands of each task pair are concatenated together. The former is used to train a two-class support vector machine (SVM) to classify the latter, obtaining two classification scores for each task pair. All classification scores for each task are summed to obtain the classification score for each task (i.e., the class) in a single test experiment.

[0014] Step 5: EEG Signal Classification. The classification score vectors obtained from the EEGNet and FBCSP models are standardized respectively. Then, these two normalized classification score vectors are added together. Finally, the motion imagery task corresponding to the maximum score is determined as the task label for a single test signal.

[0015] Further, step (1) of truncating the continuous EEG signal refers to dividing the continuous EEG data into data from a single experiment, that is, truncating a segment of data after each experimental task prompt for subsequent data processing; the step of using a filter bank to perform sub-band filtering on the EEG signal of a single experiment refers to performing bandpass filtering on the EEG data of a single experiment across the filter bank, including each sub-band. The bandpass filtering uses a fourth-order Butterworth infinite impulse response (IIR) digital filter.

[0016] Furthermore, step (2) of dividing the experimental data of each subject into training set and test set means that if the original dataset is not divided into training and test parts, then cross-validation and other methods are used to divide the experimental data of each subject into training set and test set.

[0017] The use of leave-one-out method, where each subject is used as a test subject once, means that each subject is used as a test subject sequentially. Their training data is used to train the classification model, while their test data is used to evaluate the classification performance of BCI. The average classification accuracy of all subjects is used as a metric for BCI performance.

[0018] Furthermore, in step (3):

[0019] EEGNet, or EEG-based convolutional neural network, is a deep learning platform capable of processing EEG signals induced by motion and visual perception. EEGNet consists of three modules: Module 1 includes a temporal convolution of size (1, 64), a depthwise convolution of size (C, 1) (where C is the number of channels), two batch normalization layers, one exponential linear unit activation layer, one average pooling layer, and one random deactivation layer; Module 2 includes a separable convolution, one batch normalization layer, one exponential linear unit activation layer, one average pooling layer, one random deactivation layer, and one flattening layer; Module 3 is a classification module where features are directly input into a softmax classification layer with N (N is the number of classes) units. EEGNet uses the Adam optimizer with default parameters for model fitting, minimizing the class cross-entropy loss function.

[0020] The training of the EEGNet model using transfer learning involves two phases: The first phase uses training data from all subjects to globally train the EEGNet, obtaining weights and biases for convolutional and fully connected layers common to all subjects; the second phase uses training data from the test subjects to locally train the EEGNet, transferring the weights and biases obtained in the first phase as initial values ​​for the EEGNet weights and biases. After these two phases of training, a specific EEGNet model (net) is obtained for each subject.

[0021] The process of classifying test data using a trained model refers to classifying the test data of each subject using the neural network classification function `classify()` based on a specific EEGNet model for that subject, thus obtaining the results of the i-th test experiment. Feature scores for all motion imagery tasks

[0022]

[0023] Furthermore, in step (4):

[0024] The FBCSP algorithm refers to the CSP algorithm based on filter banks, that is, applying the CSP algorithm for two types of signals to the EEG signal filtered by each subband of the filter bank.

[0025] The use of a one-versus-one (OVO) strategy to train the FBCSP model refers to training the FBCSP model if the EEG data includes N P If there are 1 task, it can be decomposed into N. J =N P (N P -1) / 2 task pairs (i.e. class pairs), and use the FBCSP algorithm for each pair (i.e. each two classes) of EEG data;

[0026] The process of training the FBCSP algorithm model using the training data of the test subjects is as follows: Assume that the EEG signal of a single experiment, after subband k-filtering, is represented as... Where, N c N s N t With N k These represent the number of channels, the number of sampling points, the total number of experiments per class, and the number of subbands in the filter bank, respectively. The EEG signal X in a single experiment... t k The spatial covariance matrix can be calculated using the following formula:

[0027]

[0028] In the formula, the superscript T denotes the transpose operation, and trace(A) denotes the trace operation of matrix A, which is the sum of the diagonal elements of matrix A. t The mean spatial covariance matrix of this experiment is

[0029]

[0030] The FBCSP algorithm aims to estimate a spatial filtering matrix by minimizing the following objective function. To maximize the variance difference between the two types of signals.

[0031]

[0032] In the formula, and Let be the mean spatial covariance matrices of the first and second classes of training experiments in subband k, respectively. The solution to the above equation can be applied to the matrix... and Obtained by generalized eigenvalue decomposition. Take the matrix. The eigenvectors corresponding to the m largest and m smallest eigenvalues ​​can be used to obtain the spatial domain filter for subband k.

[0033] The process of obtaining the training and testing feature signals for each sub-band and each pair of tasks is as follows: Assume the training and testing signals for a single experiment in sub-band k are respectively... and Training features With test features The calculation is as follows:

[0034]

[0035] The process of obtaining the two test feature scores for each pair of tasks is as follows: The training and test feature signals of all sub-bands are concatenated to obtain the training feature vector f for each pair of tasks. tr With the test feature vector f te :

[0036]

[0037] In the formula, A Support Vector Machine (SVM) classification model is trained using the feature vectors from all training experiments for each task pair, and the test feature vector f is used for the i-th experiment. te,i After classification, the classification scores for the j-th task in this experiment are as follows:

[0038]

[0039] In the formula, predict() is the SVM prediction function.

[0040] The process of summing all test scores for each task is as follows: For a task containing N... P The EEG data for each task needs to be processed by N. J =N P (N P -1) / 2 OVO classifications, generating a total of 2N J Each category score generates N classification scores, where each category of signal produces N. P -1 category score.

[0041]

[0042] Furthermore, step (5) includes the following specific steps:

[0043] (5.1) Standardize the classification scores of the EEGNet model and the FBCSP model respectively, that is, divide them by the average score of the training set of the EEGNet model and the FBCSP model respectively, and then add them together to obtain the total classification score of the i-th test experiment as follows:

[0044]

[0045] In the formula, normalize represents the normalization function.

[0046] (5.2) Test Signal Corresponding class tag l i The class label corresponding to the highest score:

[0047]

[0048] In the formula, N te This represents the total number of test experiments.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] (1) This invention provides a multi-class motor imagery brain-computer interface decoding method that combines EEGNet and FBCSP, which can effectively utilize the complementarity between the feature signals of the two models, thereby significantly improving the separability of the feature signals and their robustness to noise.

[0051] (2) This invention provides a feature fusion method based on the sum of classification scores, which can achieve high classification accuracy and classification robustness under the condition of small training dataset, and promote the practical application of multi-class motion imagination BCI system. Attached Figure Description

[0052] Figure 1 EEGNet and FBCSP's multi-class motion imagery BCI decoding method.

[0053] Figure 2 Example dataset: Motion Imagination BCI Experiment Timing Scheme.

[0054] Figure 3 The network structure and parameters of the convolutional neural network EEGNet.

[0055] Figure 4 Training and classification methods for convolutional neural networks (EEGNet).

[0056] Figure 5 Training and classification method for filter bank common spatial mode FBCSP.

[0057] Figure 6 Classification performance of EEGNet, FBCSP, and combinations thereof. Detailed Implementation

[0058] To clearly illustrate the objectives, technical solutions, and advantages of the present invention, a detailed description is provided below in conjunction with the accompanying drawings and embodiments. The specific embodiments described herein are merely illustrative of the technical solutions of the present invention and are not intended to limit the scope of the invention.

[0059] This invention provides a multi-class motor imagery brain-computer interface decoding method combining EEGNet and FBCSP. The method mainly includes training and classifying the EEGNet model, training and classifying the FBCSP model, summing the classification scores of the two models, and a classification method for the test signal, such as... Figure 1 As shown.

[0060] Using dataset 2a, a four-class motion imagery dataset provided by the 4th International BCI Competition, this paper illustrates the task recognition method for multi-class motion imagery BCI provided by this invention. This embodiment uses Matlab to complete all program code. A brief description of this dataset is as follows:

[0061] This dataset was recorded in an uncontrolled environment, resulting in numerous artifacts, making decoding the motor imagery task a challenging task. The dataset contains EEG data from nine subjects performing four tasks: left-hand, right-hand, bilateral foot, and tongue motor imagery. Each trial's task was given via a cue. The dataset was acquired at a sampling rate of 250 Hz, using 22 electrodes, and at two time points (T and E). Each subject underwent 288 trials per time point, and 72 trials per task (i.e., per category). The timing scheme for the motor imagery BCI trial is as follows: Figure 2 As shown. The motor imagery begins with the task prompt and ends after 4 seconds of performing the imagery task.

[0062] The multi-class motor imagery brain-computer interface decoding method combining EEGNet and FBCSP described in this invention has the following implementation steps:

[0063] Step 1: Data Preprocessing. Continuous EEG signals from each subject are truncated, and subband filtering of the EEG signals from a single experiment is performed using a filter bank.

[0064] (1.1) The truncation of continuous EEG signals refers to dividing continuous EEG data into data for a single experiment, that is, truncating a segment of data after each experimental task prompt for subsequent data processing. For the EEGNet model, a data segment of 2s-6s is truncated, so each experiment is a matrix with a dimension of 22×1000; for the FBCSP algorithm, a data segment of 2.5s-5.5s is truncated, so each experiment is a matrix with a dimension of 22×750.

[0065] (1.2) Subband filtering of the EEG signal from a single experiment using a filter bank refers to bandpass filtering of the EEG data from a single experiment across the various subbands included in the filter bank. The bandpass filtering employs a fourth-order Butterworth Infinite Impulse Response (IIR) digital filter. For the EEGNet model, one subband (0-30Hz) is used to filter the data from a single experiment; for the FBCSP algorithm, seven subbands (4-9Hz, 9-14Hz, 14-19Hz, 19-24Hz, 24-29Hz, 29-34Hz, and 34-39Hz) are used to filter the data from a single experiment.

[0066] Step 2: EEG signal classification strategy. The experimental data for each subject is divided into a training set and a test set, and each subject is used as a test subject once using the leave-one-out method.

[0067] (2.1) The division of each subject's experimental data into training and testing sets refers to the use of methods such as cross-validation to divide each subject's experimental data into training and testing sets if the original dataset is not divided into training and testing parts. Since the dataset includes datasets from two time periods (E and E), the data collected in time period T is used as the training set, while the dataset collected in time period E is used as the testing set;

[0068] (2.2) The use of leave-one-out method, where each subject is used as a test subject once, means that each subject is used as a test subject sequentially. Their training data is used to train the classification model, while their test data is used to evaluate the classification performance of BCI. Since this dataset contains 9 subjects, each subject is used as a test subject once. The average classification accuracy of all 9 subjects is taken as the classification accuracy of EEGNe, FBCSP, and the combination of the two.

[0069] Step 3: EEGNet Model Training and Classification. The EEGNet model is trained using transfer learning. The trained model is then used to classify the test data, obtaining the classification score vector (i.e., the classifier's value before the output label) for a single test experiment across all motion visualization tasks (i.e., all classes). The training and classification method for the convolutional neural network EEGNet is as follows: Figure 3 As shown.

[0070] (3.1) EEGNet refers to an EEG-based convolutional neural network (EEGNet) that can perform deep learning on EEG signals induced by motion visualization, etc. The structure and parameters of EEGNet are as follows: Figure 4As shown, the network consists of 17 layers, including one input layer, four convolutional layers, three batch normalization layers, two rectified linear units (RCLs), two pooling layers, two dropout layers, one fully connected layer, one softmax layer, and one output layer. The four convolutions serve the following functions: the first convolution is a temporal convolution to learn frequency filters; the second convolution is a depth-oriented convolution to learn frequency-specific spatial filters; the third convolution is a separable convolution, a combination of depth-oriented convolutions, learning a temporal summary for each feature map; and the fourth convolution is a point-oriented convolution, which optimally integrates the feature maps. The last three layers are used for classification, directly inputting features into a softmax layer with N units (N = 4, representing the number of classes), and determining the class label based on its output probability. EEGNet uses the Adam optimizer with default parameters for model fitting, minimizing the class cross-entropy loss function.

[0071] (3.2) The training of the EEGNet model using transfer learning refers to a two-stage training process: The first stage uses training data from all subjects to perform global training on the EEGNet, obtaining weights and biases for convolutional and fully connected layers common to all subjects; the second stage uses training data from the test subjects to perform local training on the EEGNet, transferring the weights and biases obtained from global training to the local training as initial values ​​for the EEGNet weights and biases. After these two stages of training, a specific EEGNet model (net) is obtained for each subject.

[0072] (3.3) The classification of test data using the trained model refers to classifying the test data of each subject using the neural network classification function classify() based on the specific EEGNet model net for each subject, and obtaining the i-th test experiment. Feature scores for all motion imagery tasks

[0073]

[0074] Step 4: FBCSP Model Training and Classification. The FBCSP algorithm is trained using a one-versus-one (OVO) strategy to obtain the spatial filter for each sub-band and each task pair. Spatial filtering is applied to the training and test data to obtain the training and test feature signals for each sub-band and each task pair, respectively. The training and test feature signals from all sub-bands of each task pair are concatenated together. The former is used to train a two-class support vector machine (SVM) to classify the latter, obtaining two classification scores for each task pair. All test scores for each task are summed to obtain the classification score for each task (i.e., class) in a single test experiment. The FBCSP training method based on the OVO strategy and the classification method based on SVM are as follows: Figure 5 As shown.

[0075] (4.1) The FBCSP algorithm mentioned refers to the CSP algorithm based on the filter bank, that is, applying the CSP algorithm for two types of signals to the EEG signal filtered by each sub-band of the filter bank. In this embodiment, the filter bank includes 7 sub-bands, so the CSP algorithm is applied to the EEG signal filtered by these 7 sub-bands;

[0076] (4.2) The use of a one-versus-one (OVO) strategy to train the FBCSP model refers to training the FBCSP model if the EEG data includes N P If there are 1 task, it can be decomposed into N. J =N P (N P -1) / 2 task pairs (i.e., class pairs), and the FBCSP algorithm is used for each pair (i.e., each two classes) of EEG data. This dataset contains N P = 4 motion visualization tasks, which can form N J = There are 6 task pairs: left hand-right hand, left hand-foot, left hand-tongue, right hand-foot, right hand-tongue, and foot-tongue. The FBCSP model is trained using the OVO strategy, that is, the CSP algorithm is used for each task pair of signals in each subband filter;

[0077] (4.3) The process of training the FBCSP algorithm model using the training data of the test subjects is as follows: Assume that the EEG signal of a single experiment is represented by subband k filtering as follows: Where, N c N s N t With N k These represent the number of channels, the number of sampling points, the total number of experiments per class, and the number of subbands in the filter bank, respectively. In this embodiment, N c N s N t With N kThe values ​​are 22, 750, 72, and 7 respectively. Single-sample EEG signal. The spatial covariance matrix can be calculated using the following formula.

[0078]

[0079] In the formula, the superscript T denotes the transpose operation, and trace(A) denotes the trace operation of matrix A, which is the sum of the diagonal elements of matrix A. t The average spatial covariance matrix of this experiment is:

[0080]

[0081] The FBCSP algorithm aims to estimate a spatial filtering matrix by minimizing the following objective function. Maximize the variance difference between the two types of signals:

[0082]

[0083] In the formula, and Let be the mean spatial covariance matrices of the first and second classes of training experiments in subband k, respectively. The solution to the above equation can be expressed as a matrix. The eigenvectors of . Take The eigenvectors corresponding to the m largest and m smallest eigenvalues ​​can be used to obtain the spatial domain filter for subband k. In this embodiment, m = 1, therefore

[0084]

[0085] (4.4) The process of obtaining the training and testing feature signals for each pair of tasks in each subband is as follows: Assume that the training and testing signals for a single experiment in subband k are respectively and Training features With test features The calculation is as follows:

[0086]

[0087] (4.5) The process of obtaining the two test feature scores for each pair of tasks is as follows: concatenate the training and test feature signals of all sub-bands respectively to obtain the training feature vector f for each pair of tasks. tr With the test feature vector f te :

[0088]

[0089] In the formula, A Support Vector Machine (SVM) classification model is trained using the feature vectors from all training experiments for each task pair, and the test feature vector f is used for the i-th experiment. te,i After classification, the classification scores for the j-th task in this experiment are as follows:

[0090]

[0091] In the formula, predict() is the SVM prediction function.

[0092] (4.6) The process of summing all test scores for each task is as follows: For a task containing N P The EEG data for each task needs to be processed by N. J =N P (N P -1) / 2 OVO classifications, generating a total of 2N J Each category score generates N classification scores, where each category of signal produces N. P -1 category score:

[0093]

[0094] Step 5: EEG Signal Classification. The feature score vectors obtained from the EEGNet and FBCSP models are standardized respectively. Then, these two feature score vectors are added together. Finally, the motion visualization task corresponding to the maximum score is determined as the task label for a single test signal.

[0095] (5.1) Standardize the classification scores of the EEGNet model and the FBCSP model respectively, that is, divide them by the average score of the training set of the EEGNet model and the FBCSP model respectively, and then add them together to obtain the total classification score of the i-th test experiment as follows:

[0096]

[0097] In the formula, nornalize represents the normalization function.

[0098] (5.2) Test Signal Corresponding class tag l i The class label corresponding to the highest score:

[0099]

[0100] In the formula, N te This represents the total number of test experiments.

[0101] Step 6: Performance Evaluation. In MI-based BCI research, EEGNet and FBCSP are two of the most typical methods in deep learning and traditional machine learning, respectively. To evaluate the actual effect of the signal decoding method provided in this invention, we compared the classification performance of these three methods.

[0102] Figure 6 The graph shows the classification accuracy for each subject using these three algorithms, compared to the average accuracy. As can be seen from the graph, for each subject, the classification accuracy of the combination of EEGNet and FBCSP (EEGNet+FBCSP) is greater than or equal to the classification accuracy of EEGNet and FBCSP alone. On average for each subject, the classification accuracies of FBCSP, EEGNet, and EEGNet+FBCSP are 68.47%, 70.19%, and 76.43%, respectively, with the latter being 7.69% and 6.74% higher than FBCSP and EEGNet, respectively.

[0103] Using paired t-tests (confidence levels set at 95%), we performed a statistical analysis of the average classification accuracy of these three methods, such as... Figure 6 As shown in the figure. ** indicates p < 0.01. The t-test results show that the classification accuracy of EEGNet+FBCSP is significantly higher than that of FBCSP and EEGNet.

[0104] These results fully demonstrate that the method provided by this invention for decoding multiple types of motion imagery BCI signals by combining EEGNet and FBCSP is extremely effective and helps to promote the practical application of motion imagery BCI systems.

Claims

1. A multi-class motor imagery brain-computer interface decoding method combining EEGNet and FBCSP, characterized by: Includes the following steps: Step 1: Data preprocessing: The continuous EEG signal of each subject is segmented into EEG signals of a single experiment, and the EEG signals of the single experiment are subband filtered using a filter bank. Step 2: EEG signal classification strategy: Divide the experimental data of each subject into a training set and a test set, and use the leave-one-out method to treat each subject as a test subject once; Step 3: EEGNet Model Training and Classification: Train the EEGNet model using transfer learning; use the trained model to classify the test data and obtain the feature score vectors for all classes of all motion visualization tasks in a single test experiment; Step 4: FBCSP Model Training and Classification: Train the FBCSP algorithm using a one-to-one strategy to obtain the spatial filter for each sub-band and each task pair; perform spatial filtering on the training and test data to obtain the training and test feature signals for each sub-band and each task pair respectively; concatenate the training and test feature signals of all sub-bands for each task pair together, and use the training feature signals to train a two-class support vector machine (SVM) to classify the test feature signals, obtaining two classification scores for each task pair; sum all classification scores for each task to obtain the classification score for each task category in a single test experiment. Step 5: EEG signal classification: Standardize the classification score vectors obtained from the EEGNet and FBCSP models respectively, then add the two normalized classification score vectors together, and finally determine the motion imagery task corresponding to the maximum score as the task label corresponding to the single test signal.

2. The multi-class motor imagery brain-computer interface decoding method combining EEGNet and FBCSP according to claim 1, characterized in that in step 1: The term "segmenting continuous EEG signals into single-experiment EEG signals" refers to dividing continuous EEG data into data for a single experiment, that is, extracting a segment of data after each experimental task prompt for subsequent data processing. The use of a filter bank to perform subband filtering on the EEG signal of a single experiment refers to bandpass filtering of the EEG data of a single experiment in the filter bank, including each subband. The bandpass filter described uses a fourth-order Butterworth infinite impulse response (IIR) digital filter.

3. The multi-class motor imagery brain-computer interface decoding method combining EEGNet and FBCSP according to claim 1, characterized in that in step 2: The phrase "dividing the experimental data of each subject into training and testing sets" refers to using cross-validation to divide the experimental data of each subject into training and testing sets when the original dataset is not divided into training and testing parts. The use of leave-one-out method to treat each subject as a test subject once means that each subject is treated as a test subject in turn, and their training data is used to train the classification model, while their test data is used to evaluate the classification performance of BCI; the average classification accuracy of all subjects is used as the metric for BCI performance.

4. The multi-class motor imagery brain-computer interface decoding method combining EEGNet and FBCSP according to claim 1, characterized in that: In step 3: EEGNet refers to a convolutional neural network based on EEG data, which performs deep learning on EEG signals induced by motion visualization. EEGNet consists of three modules: Module 1 includes a temporal convolution of size (1, 64), a depthwise convolution of size (C, 1), where C is the number of channels, two batch normalization layers, one exponential linear unit activation layer, one average pooling layer, and one random deactivation layer; Module 2 includes a separable convolution, one batch normalization layer, one exponential linear unit activation layer, one average pooling layer, one random deactivation layer, and one flattening layer; Module 3 is a classification module, where features are directly input into a softmax classification layer with N units, where N is the number of classes; EEGNet uses the Adam optimizer with default parameters for model fitting, minimizing the class cross-entropy loss function. The training of the EEGNet model using the transfer learning method refers to the training being carried out in two stages: the first stage uses the training data of all subjects to train the EEGNet globally, obtaining the weights and biases of the convolutional and fully connected layers that are common to all subjects; The second stage uses the training data of the test subjects to perform local training on EEGNet, and transfers the weights and biases obtained in the first stage of training to the second stage of training as the initial values ​​of EEGNet weights and biases; after two stages of training, a specific EEGNet model net is obtained for each subject. The use of a trained model to classify test data refers to using a neural network classification function based on a specific EEGNet model for each subject. The test data of the subject is classified to obtain the i-th test experiment. Feature scores for all motion imagery tasks: 。 5. The multi-class motor imagery brain-computer interface decoding method combining EEGNet and FBCSP according to claim 1, characterized in that in step 4: The FBCSP algorithm refers to the CSP algorithm based on filter banks, that is, applying the CSP algorithm for two types of signals to the EEG signal filtered by each subband of the filter bank. The use of a one-to-one strategy to train the FBCSP model refers to training the FBCSP model when the EEG data includes... If there are multiple tasks, then break them down into... Each task pair uses the FBCSP algorithm for each pair of EEG data. The process of training the FBCSP algorithm model using the training data of the test subjects is as follows: The EEG signal of a single experiment, after subband k-filtering, is represented as... ,in, , , and These represent the number of channels, the number of sampling points, the total number of experiments per class, and the number of subbands in the filter bank; the EEG signal in a single experiment. The spatial covariance matrix is ​​calculated using the following formula: In the formula, the superscript T denotes the transpose operation, while This represents the trace operation of matrix A; The average spatial covariance matrix of this experiment is: The FBCSP algorithm aims to estimate a spatial filtering matrix by minimizing the following objective function. To maximize the variance difference between the two types of signals: In the formula, and Let be the mean spatial covariance matrices of the first and second class training experiments in subband k, respectively; and let be the solution pair matrix of the above equation. and Obtained by generalized eigenvalue decomposition; matrix is ​​taken. The eigenvectors corresponding to the m largest and m smallest eigenvalues ​​are used to obtain the spatial domain filter for sub-band k. ; The process of obtaining the training and testing feature signals for each subband and each pair of tasks is as follows: The training and testing signals for a single experiment in subband k are respectively... and Training features With test features The calculation is as follows: The process of obtaining the two test feature scores for each pair of tasks is as follows: Connect the training and test feature signals of all sub-bands respectively to obtain the training feature vector for each pair of tasks. With test feature vector : In the formula, Train a Support Vector Machine (SVM) classification model using the feature vectors from all training experiments for each task pair, and test the feature vectors from the i-th experiment. The classification scores for the j-th task in this experiment are as follows: In the formula, For SVM prediction functions; The process of summing all test scores for each task is as follows: For tasks containing... The EEG data for each task needs to be processed. This one-to-one classification generates a total of Each category score is generated, where each signal category produces... Individual category scores: 。 6. The multi-class motor imagery brain-computer interface decoding method combining EEGNet and FBCSP according to claim 1, characterized in that: Step 5 includes the following steps: (5.1) Standardize the classification scores of the EEGNet model and the FBCSP model respectively, divide them by the average training set scores of the EEGNet model and the FBCSP model respectively, and then add them together to obtain the total classification score of the i-th test experiment as follows: In the formula, normalize represents the normalization function; (5.2) Test signal Corresponding class tags The class label that was judged as the highest score In the formula, This represents the total number of test experiments.

Images