A visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis

Abstract

A visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis, comprising: 1) channel extraction, frequency reduction, data cutting and sub-band filtering are performed on the EEG data of each subject; 2) the sbCNN model is trained by using the training data of all subjects, the test data is classified, the classification scores of all stimulus frequencies are obtained and normalized; 3) the TRCA model is trained by using the training data of the test subject, the spatial filter and the template of each stimulus frequency are obtained, the spatial filter of all stimulus frequencies is integrated to filter the template and the test experiment, and the classification score of the test experiment is obtained by performing correlation operation on the filtered template and the test experiment and normalized; 4) the classification scores of the two models are added, the mode classification is performed according to the classification score, and the stimulus frequency of the test experiment is identified. The application can improve the classification accuracy of the brain-computer interface and promote its practical application.

Description

TECHNICAL FIELD

[0001] The present application relates to brain science technology and computer technology. More particularly, the present application relates to brain-computer interface technology and electroencephalogram signal processing technology. BACKGROUND

[0002] Brain-Computer Interface (BCI) refers to creating a connection path for information exchange between the brain and a device with information processing capability, so as to realize the exchange of information or the control of the device. BCI technology can realize the interaction between the brain and the external device, cross the conventional brain information output path, and has broad application prospects in the fields of medical health and the like. Meanwhile, with the continuous exploration of the structure and function of the brain by modern medicine, humans have a relatively in-depth understanding of the brain function areas such as movement, vision, hearing and language. Through BCI, the information of the brain areas can be acquired and analyzed, and the BCI will have a wide application in the fields of diagnosis, screening, treatment and rehabilitation of nervous system diseases.

[0003] Due to the convenience of collection and low cost, electroencephalogram (EEG) signal is the most widely used BCI input signal. Among various EEG-based BCI systems, visual BCI has obvious advantages in information transmission rate, number of recognizable targets, applicable population, and user training time, and thus has attracted widespread attention. BCI based on steady-state visual evoked potential (SSVEP) is a typical visual BCI. SSVEP is the response of the brain to an external visual stimulus of a specific frequency. When the human eye is subjected to a flickering or changing pattern of visual stimulus of a fixed frequency, the electrical activity of the brain cortex will be modulated, thereby generating a continuous response related to the stimulus frequency. SSVEP causes the power spectrum of the EEG signal to have spectral peaks at the stimulus frequency and its harmonic frequencies. By analyzing the power spectrum of the EEG signal and detecting the frequency corresponding to the maximum spectral peak, the user's gaze target can be determined, so as to identify the user's intention and output the corresponding command.

[0004] Although great progress has been made in BCI research, it does not support wide practical applications at present. One of the main challenges faced by SSVEP-based BCI is to correctly decode SSVEP signals, that is, to accurately detect the stimulation frequency. Frequency detection methods can be divided into two categories: traditional machine learning methods and newer deep learning methods. The former can develop discriminative feature extraction algorithms according to the researchers' professional knowledge, but these algorithms have weak learning ability; the latter has strong ability to automatically represent high-level abstraction and can accurately model complex EEG signals, but it requires a large amount of labeled data, which is difficult to obtain for BCI research. Therefore, both methods have their own advantages and limitations, and their classification performance cannot meet the needs of practical applications. Combining the two methods to decode SSVEP signals is expected to take advantage of their strengths to improve frequency detection accuracy and the performance of BCI systems. SUMMARY

[0005] The purpose of the present application is to provide a visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis, which combines task-related component analysis (TRCA) and sub-band convolutional neural network (sbCNN) to detect the stimulation frequency of SSVEP signals. Under the condition of small training set samples, the classification performance of visual BCI and the robustness of frequency detection are significantly improved, thereby promoting the practical application of visual BCI system.

[0006] The present application is realized by the following technical solutions.

[0007] The visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis according to the present application comprises the following steps:

[0008] (1) Data preprocessing: For a given data set, the continuous EEG data of the subject is subjected to channel extraction, frequency reduction, data truncation and sub-band filtering;

[0009] (2) EEG signal classification strategy: using k-fold cross-validation method, the experimental data of each subject is divided into training set and test set, and using leave-one-out method, each subject is used as a test subject once;

[0010] (3) sbCNN model training and classification: using transfer learning method to train sbCNN model. Using the trained model to classify the test data, obtaining the classification score vector (i.e. the value before the output label of the classifier) of all stimulation targets (i.e. stimulation frequency) in a single experiment;

[0011] (4) TRCA model training and classification: training the TRCA algorithm model using the training data of the test subject, obtaining the spatial filter and multi-channel template of each stimulation target (i.e. stimulation frequency); integrating the spatial filter of all stimulation targets to spatially filter the multi-channel template and the test signal; using the template matching method to identify the frequency of the test signal, obtaining the classification score vector of all frequencies for this test experiment;

[0012] (5) Stimulation frequency identification: for a single test experiment, the classification scores obtained by the sbCCN and TRCA models are normalized to the interval [0, 1] respectively, then the two classification scores are added, and finally the stimulation frequency corresponding to the maximum score is determined as the stimulation frequency of the SSVEP signal.

[0013] The step (1) comprises the following specific steps:

[0014] (1.1) Channel extraction of continuous EEG signal: extracting the EEG data of nine electrode channels (Pz, PO5, PO3, POz, PO4, PO6, O1, Oz, O2) located in the occipital region from the original recording channels;

[0015] (1.2) Truncation of continuous EEG signal: dividing the continuous EEG data into single experiment data, i.e. truncating a segment of data from the task prompt in the time window [0.64s, (0.64+d)s], where 0.5s is the eye shift time and 0.14s is the latency delay time of the visual system, and d is the data length for frequency identification;

[0016] (1.3) Down-sampling of continuous EEG signal: if the sampling frequency of the original EEG data is too high, it needs to be down-sampled to about 250Hz to reduce the complexity of data processing;

[0017] (1.4) Sub-band filtering of continuous EEG signal: using a Chebyshev Infinite Impulse Response (IIR) filter to band-pass filter the single experiment EEG data in the frequency band [8*n, 90]Hz (n=1, 2, …, N b , N b is the number of sub-bands). In order to avoid phase distortion, the signal is filtered from both positive and negative directions;

[0018] The step (2) comprises the following specific steps:

[0019] (2.1) Using k-fold cross-validation method to divide the experimental data of each subject into training set and test set: dividing the experimental data of each subject into k equal parts, using each part of data as test data once, and using the remaining k-1 parts of data as training data;

[0020] (2.2) Using leave-one-out method, each subject is used as a test subject once: its training data is used to train the classification model, while its test data is used to evaluate the classification performance of BCI. The average classification accuracy and information transfer rate of all subjects and k-fold cross-validation are used as the performance measure of BCI.

[0021] The step (3) comprises the following steps:

[0022] (3.1) sbCNN refers to a convolutional neural network based on sub-band filtered data (sub-band convolutional neural network, sbCNN), which can perform deep learning on multiple sub-band SSVEP signals. The sbCNN is composed of 12 network layers, including 4 convolutional layers, 1 fully connected layer and an output layer. The first convolutional layer is used for sub-band combination, the second convolutional layer is used for electrode channel combination, the third and fourth convolutional layers are used for feature signal extraction, and the fully connected layer is used for prediction of stimulation frequency, and the highest probability value returned by the softmax function is used to achieve it;

[0023] (3.2) The sbCNN model is trained using a transfer learning method: the training is carried out in two stages: in the first stage, the sbCNN is globally trained using the training data of all subjects to obtain the weights and biases of the convolutional layer and the fully connected layer common to all subjects; in the second stage, the sbCNN is locally trained using the training data of the test subject, and the weights and biases obtained in the first stage are transferred to the second stage training as the initial values of the sbCNN network weights and biases. After two-stage training, the sbCNN model net specific to each subject is obtained;

[0024] (3.3) The trained model is used to classify the test data: according to the sbCNN model specific to each subject, the neural network classification function is used to classify the test data of the subject, and the classification score of the ith test experiment of the subject is obtained Classification score for all stimulation frequencies:

[0025]

[0026] The step (4) comprises the following steps:

[0027] (4.1) The process of training the TRCA algorithm model using the training data of the test subject is as follows: assuming that the training data of the nth stimulation target of the test subject in the kth sub-band is represented as k = 1, 2, …, N K n = 1, 2, …, N F , where N C , NS T K F are the number of electrode channels, the number of sampling points in a single experiment, the number of experiments for each stimulation frequency, the number of subbands in a filter bank, and the number of stimulation frequencies, respectively. The ith experiment of stimulation frequency n in subband k can be denoted as The optimization goal of the TRCA algorithm is to obtain the spatial filter for each stimulation frequency in each subband, project the multi-channel EEG signal into a single-channel signal, maximize the sum of the covariance between experiments, and find the spatial filter for each stimulation target and each subband maximize the following equation:

[0028]

[0029] where the superscript T is the transpose operator, denotes the sum of the covariance matrix between all experiments of stimulation frequency n in subband k. To obtain a finite solution, the following constraint condition needs to be imposed on the above equation:

[0030]

[0031] Then, the weight vector can be calculated as follows:

[0032]

[0033] The solution of the above equation can be represented as the eigenvector of the matrix .

[0034] For stimulation target n and subband k, the multi-channel template signal of the test subject is the average of all training signals:

[0035]

[0036] (4.2) The spatial filter of all stimulation targets is integrated to perform spatial filtering on the multi-channel template and test signals: the spatial filter of all stimulation targets in each subband is connected into an integrated spatial filter

[0037]

[0038] The TRCA algorithm for estimating the integrated spatial filter is called the ensemble TRCA (eTRCA) algorithm.

[0039] (4.3) The process of frequency identification of the test signal using the template matching method is as follows: assuming that in subband k, a single test experiment can be denoted as The W k ​​​The template signals and single test experiments are filtered in space domain, and the filtered test signals are correlated with the i-th template signal to obtain the correlation coefficient as follows:

[0040]

[0041] In the formula, p represents the Pearson correlation coefficient. For each stimulation frequency, the correlation coefficients of N K sub-bands are integrated according to the following formula to obtain the classification score obtained by the eTRCA algorithm:

[0042]

[0043] The step (5) comprises the following specific steps:

[0044] (5.1) The classification scores of the sbCNN model and the eTRCA model are normalized to the interval [0, 1] respectively, and then the two are added to obtain the total classification score of each stimulation frequency as follows:

[0045]

[0046] In the formula, nor represents the normalization function.

[0047] (5.2) The test signal corresponding to the stimulation frequency f i is identified using the following equation:

[0048]

[0049] Task-related component analysis (TRCA) and sub-band convolutional neural network (sbCNN) are the most advanced traditional machine learning algorithm and deep learning algorithm respectively, and have achieved good performance in the SSVEP-based BCI system. However, the TRCA algorithm is limited in effectively utilizing the information of other subjects, which is exactly the advantage of the deep learning method. The method of effectively integrating TRCA and sbCNN of the present application effectively improves the classification performance of the visual BCI.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] (1) The present application provides a visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis, which can effectively utilize the complementarity between the feature signals of the two models and significantly improve the classification performance of the BCI.

[0052] (2) The algorithm provided by the present application can achieve high classification accuracy under the condition of small data set, significantly improve the information transmission rate (ITR) of the visual BCI system, and promote the practical application of the BCI system. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The present application provides a visual BCI frequency detection method combining sub-band convolutional neural network and task-related analysis.

[0054] Figure 2 The stimulation target frequency and phase coding schematic diagram of the BCI speller based on SSVEP.

[0055] Figure 3 The present application provides a visual BCI frequency detection method combining sub-band convolutional neural network and task-related analysis.

[0056] Figure 4 The present application provides a visual BCI frequency detection method combining sub-band convolutional neural network and task-related analysis.

[0057] Figure 5 The present application provides a visual BCI frequency detection method combining sub-band convolutional neural network and task-related analysis.

[0058] Figure 6 The present application provides a visual BCI frequency detection method combining sub-band convolutional neural network and task-related analysis. DETAILED DESCRIPTION

[0059] In order to clearly illustrate the purpose, technical scheme and advantages of the present application, the following will be described in detail in combination with the drawings and embodiments. The specific embodiments described herein are only used to explain the technical scheme of the present application, and are not limited to the present application.

[0060] The present application provides a visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis, which mainly includes training and classification of sbCNN model, training and classification of TRCA model, adding classification scores of two models, and classification method of test signal, as shown in Figure 1 A benchmark data set provided by the neural engineering laboratory of Tsinghua University is used to illustrate the implementation of the SSVEP signal frequency recognition method provided by the present application. In this embodiment, all program codes are programmed by using Matlab language.

[0061] This data set is briefly described as follows:

[0062] The data set comes from a BCI spelling experiment based on SSVEP. This visual speller includes 40 stimulation targets, and uses frequency and phase joint coding method for modulation, as shown in Figure 2The range of stimulation frequency was 8 Hz and 15.8 Hz with an interval of 0.2 Hz, while the range of stimulation phase was 0 rad and 1.5π rad with an interval of 0.5π rad. 35 healthy subjects (18 males, average age 22 years) participated in the experiment. Among them, 8 had experience of using SSVEP-based BCI, and the others had no experience. The experiment consisted of 6 blocks, each of which included 40 trials corresponding to 40 targets prompted in random order, i.e. each target contained 6 experiments. Each experiment started with a red visual cue with a length of 0.5 s. After the start of each experiment, the subjects were required to turn their gaze to the stimulation target as soon as possible. Then, all stimulation targets were flashed for 5 s. After the visual stimulation, the screen remained blank for 0.5 s, and then the next experiment started. 64 electrodes were placed according to the international 10 / 20 standard for recording experimental data, and the sampling frequency of the EEG data was 1000 Hz.

[0063] The implementation steps of the frequency detection method of the application are as follows:

[0064] Step 1: EEG data preprocessing. The continuous EEG signals of all subjects were subjected to channel extraction, down-sampling, data cutting and filter bank filtering.

[0065] (1.1) Channel extraction of continuous EEG signals: the data of nine electrodes (Pz, PO5, PO3, POz, PO4, PO6, O1, Oz, O2) located in the occipital region were extracted from the 64 recording channels of the original EEG data;

[0066] (1.2) Cutting of continuous EEG signals: the continuous EEG data were cut into single experiment data, i.e. cutting a segment of data in the time window [0.64 s, (0.64+d) s] after the start of the task prompt, where 0.5 s is the time of gaze shift, 0.14 s is the latency delay time of the visual system, and d is the data length for frequency detection;

[0067] (1.3) Down-sampling of continuous EEG signals: the original sampling frequency of EEG data was 1000 Hz, which was reduced to 250 Hz by down-sampling to reduce the complexity of data processing;

[0068] (1.4) Filter filtering of continuous EEG signals: a Chebyshev type I infinite impulse response (IIR) filter was used to filter the single trial EEG data in the frequency range of 8×n b -88 Hz (n b = 1, 2, …, N b , N bBandpass filtering is performed on the frequency band (number of sub-bands). To avoid phase distortion, the Matlab function `filtfilt` is used to filter the EEG data.

[0069] Step 2: EEG signal classification strategy. In the dataset used in this embodiment, each subject's experiment includes 6 blocks. Therefore, a 6-fold cross-validation method is used to divide each subject's experimental data into a training set and a test set, and a leave-one-out method is used to select each subject as a test subject once.

[0070] (2.1) The experimental data of each subject was divided into training and test sets using the 6-fold cross-validation method: the experimental data of each subject was divided into 6 equal parts, each part (each block) of data was used as test data once, and the remaining 5 parts (5 blocks) of data were used as training data.

[0071] (2.2) Each subject was selected as a test subject once using the leave-one-out method: their training data was used to train the classification model, while the test data was used to evaluate the classification performance of the BCI. The average classification accuracy and information transfer rate of all 35 subjects and 6x cross-validation were used as metrics for BCI performance.

[0072] Step 3: sbCNN Model Training and Classification: Train the sbCNN model using transfer learning. Use the trained sbCNN model to classify the test data, obtaining the classification score vector (i.e., the classifier's value before the output label) for each experiment for all stimulus targets (i.e., frequencies).

[0073] (3.1) The sbCNN mentioned refers to a sub-band convolutional neural network (sbCNN) based on sub-band filtered data. This embodiment uses 3 sub-bands [8*n, 90] Hz (n = 1, 2, ..., N). b N b Deep learning is performed on SSVEP signals with a sub-band count of 3. For example... Figure 3 As shown, sbCNN consists of 12 network layers, including 4 convolutional layers, 3 dropout layers, 1 ReLU layer, 1 fully connected layer, 1 softmax layer, and 1 output layer. The first convolutional layer is used for subband binding, the second convolutional layer is used for electrode channel binding, the third and fourth convolutional layers are used for extracting feature signals, and the fully connected layer is used to predict stimulus frequencies, which is achieved by the highest probability value returned by the softmax function.

[0074] (3.2) Training sbCNN using transfer learning method: training is conducted in two stages, the first stage globally trains sbCNN using training data of all subjects to obtain weights and biases common to all subjects; the second stage locally trains sbCNN using training data of test subjects, and transfers the weights and biases of the convolutional layer and fully connected layer obtained in the first stage to the second stage training as the initial values of the sbCNN network weights and biases, as shown in Figure 4 After two-stage training, the subject-specific sbCNN model net;

[0075] (3.3) Classifying test data using the trained sbCNN model refers to using the sbCNN model specific to each subject to classify the test data of the subject according to the neural network classification function, obtaining the classification score vector of the i-th test experiment Classification scores for all stimulus frequencies:

[0076]

[0077] Step 4: TRCA model training and classification. Train the TRCA algorithm model using the training data of the test subject to obtain the spatial filter and multi-channel template for each stimulus target (i.e. stimulus frequency); integrate the spatial filters of all stimulus targets to spatially filter the multi-channel template and test signal; use template matching method to identify the frequency of the test signal to obtain the classification score vector of the test experiment for all frequencies;

[0078] (4.1) The process of training the TRCA algorithm model using the training data of the test subject is as follows: suppose the training data of the n-th stimulus target of the test subject in the k-th sub-band is represented as k = 1, 2, …, N K n = 1, 2, …, N F , where N C , N S , N T , N K , N F are the number of electrode channels, the number of sampling points in a single experiment, the number of experiments for each stimulus target, the number of sub-bands in the filter bank, and the number of stimulus targets, respectively. This embodiment uses 5 sub-bands [8*n, 90] Hz (n = 1, 2, …, N b , N b = 5 sub-band number) to train the TRCA model. The i-th experiment of the n-th stimulus target in the k-th sub-band can be represented as The optimization objective of the TRCA algorithm is to obtain the spatial filter for each stimulus frequency in each subband, projecting the multi-channel signal into a single-channel signal, and maximizing the sum of covariances between experiments. In other words, it seeks the spatial filter for each stimulus target and each subband. Maximize the following equation:

[0079]

[0080] In the formula Let represent the sum of the covariance matrices of stimulus target n across all experimental pairs in subband k. To obtain a finite solution, the following constraints need to be imposed on the above equation:

[0081]

[0082] Then, the weight vector can be calculated using the following formula:

[0083]

[0084] The solution to the above equation can be expressed as a matrix. eigenvectors.

[0085] For a stimulus target n and a subband k, the multichannel template signal of the test subject is the average of all training signals.

[0086]

[0087] (4.2) Spatial filtering of multi-channel template and test signal by integrating spatial filters of all stimulus targets: connect the spatial filters of all stimulus targets in each subband into an integrated spatial filter.

[0088]

[0089] The TRCA algorithm for estimating integrated spatial filters is called the ensemble TRCA (eTRCA) algorithm.

[0090] (4.3) The process of frequency identification of the test signal using the template matching method is as follows: Assuming that in subband k, a single test experiment can be expressed as Use W k Spatial filtering is performed on the template signal and the single test experiment. The filtered test signal is then correlated with the i-th template signal to obtain the correlation coefficient as follows:

[0091]

[0092] In the formula, ρ represents the Pearson correlation coefficient. For each stimulus frequency, N is integrated according to the following formula. Kcorrelation coefficient of the sub-band, the classification score obtained by the eTRCA algorithm

[0093]

[0094] Step 5: Stimulus frequency identification. For single test experiment, the classification scores obtained by the deep learning model and the TRCA model are normalized to the interval [0, 1] respectively, then the two classification scores are added, and finally the stimulus frequency corresponding to the maximum score is determined as the frequency of the SSVEP signal.

[0095] (5.1) The classification scores of the sbCNN model and the eTRCA model are normalized to the interval [0, 1] respectively, then the two scores are added to obtain the total classification score of each stimulus frequency as follows

[0096]

[0097] where nor represents the normalization function.

[0098] (5.2) Test signal Corresponding stimulus frequency f i is identified using the following equation

[0099]

[0100] Step 6: Performance evaluation. In the SSVEP-based BCI research, sbCCN and eTRCA are the most advanced methods in deep learning and traditional machine learning respectively. In order to evaluate the actual effect of the signal decoding method provided by the present application, we compared the classification performance of the three methods under different data lengths.

[0101] Figure 6 shows the classification accuracy and information transfer rate (ITR) of the three algorithms under five different data lengths (0.2s, 0.4s, 0.6s, 0.8s and 1.0s). From Figure 6 (a), we can see that the classification accuracy of the three models monotonically increases with the increase of data length. At each data length, the classification accuracy of eTRCA+sbCNN is higher than that of eTRCA and sbCNN; from Figure 6 (b), we can see that the information transfer rate of the three models first increases with the increase of data length, reaches a maximum of 277 bits / minute at 0.4s, and then monotonically decreases with the increase of data length.

[0102] Using paired student's t-test (confidence probability set at 95%), we made statistical analysis of the classification accuracy and information transfer rate of the three methods, as shown in Figure 6(a) and (b) are shown. In the figure, * indicates p < 0.05, ** indicates p < 0.01, and *** indicates p < 0.001. The results of t-test show that the classification accuracy and information transfer rate of eTRCA+sbCNN are significantly higher than those of eTRCA and sbCNN at each data length. The information transfer rate of 277 bits / min achieved in this embodiment is the highest information transfer rate achieved so far for this data set.

[0103] These results fully demonstrate that the method for decoding visual BCI signals provided by the present application in combination with the sub-band convolutional neural network and task-related analysis is extremely effective, and helps to promote the practical application of the visual BCI system.

Claims

1. A visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis, characterized in that, The method comprises the following steps: (1) data preprocessing: for a given data set, the continuous EEG data of the subject is extracted, down-sampled, data truncated and sub-band filtered; (2) EEG signal classification strategy: using k-fold cross-validation method, the experimental data of each subject is divided into training set and test set, and each subject is used as a test subject once using leave-one-out method; (3) sbCNN model training and classification: using transfer learning method to train sbCNN model; using the trained model to classify the test data, and obtaining the classification score vector of all stimulation targets of a single experiment; (4) TRCA model training and classification: using the training data of the test subject to train the TRCA algorithm model, and obtaining the spatial filter and multi-channel template of each stimulation target; integrating the spatial filter of all stimulation targets to perform spatial filtering on the multi-channel template and the test signal; using template matching method to identify the frequency of the test signal, and obtaining the classification score vector of all frequencies of the test experiment; (5) stimulation frequency identification: for a single test experiment, the classification scores obtained by sbCCN and TRCA model are normalized to the interval [0, 1], then the two classification scores are added, and finally the stimulation frequency corresponding to the maximum score is determined as the stimulation frequency of the SSVEP signal.

2. The visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis according to claim 1, characterized in that, The step (1) comprises the following steps: (1.1) extracting the EEG data of nine electrode channels Pz, PO5, PO3, POz, PO4, PO6, O1, Oz and O2 located in the occipital region from the original recording channel; (1.2) dividing the continuous EEG data into single experiment data, i.e. truncating a segment of data from the task prompt in the time window [0.64s, (0.64+d)s], wherein 0.5s is the eye shift time, 0.14s is the latency delay time of the visual system, and d is the data length for frequency identification; (1.3) if the sampling frequency of the original EEG data is too high, it needs to be down-sampled to about 250Hz to reduce the complexity of data processing; (1.4) The EEG data of a single trial is filtered in the frequency band [8*n, 90] Hz, n = 1, 2,..., N using Chebyshev infinite impulse response filters b , N b is the number of subbands, and the signal is filtered from both forward and backward directions. 3.The visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis according to claim 1, wherein, The step (2) comprises the following steps: (2.1) dividing the experimental data of each subject into k equal parts, and using each part of data as test data once, and the remaining k-1 parts of data as training data; (2.2) using the training data to train the classification model, and using the test data to evaluate the classification performance of BCI; the average classification accuracy and information transmission rate of all subjects and k-fold cross-validation are used as the performance measurement standard of BCI.

4. The visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis according to claim 1, characterized in that, The step (3) comprises the following steps: (3.1) training the sbCNN model using a transfer learning method, which is performed in two stages: in the first stage, the sbCNN is globally trained using the training data of all subjects to obtain the weights and biases of the convolutional layers and the fully connected layer common to all subjects; in the second stage, the sbCNN is locally trained using the training data of the test subject, and the weights and biases obtained in the first stage are transferred to the second stage training as the initial values of the sbCNN network weights and biases; after two-stage training, the sbCNN model net specific to each subject is obtained; The sbCNN is composed of 12 network layers, including 4 convolutional layers, 1 fully connected layer and an output layer; the first convolutional layer is used for sub-band combination, the second convolutional layer is used for electrode channel combination, the third and fourth convolutional layers are used for extracting feature signals, and the fully connected layer is used for predicting the stimulation frequency, and the highest probability value returned by the softmax function is used to achieve this; (3.2) using the neural network classification function on the test data of the subject to obtain the ith test experiment Classification scores for all stimulation frequencies 5. The visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis according to claim 1, characterized in that, The step (4) comprises the following steps: (4.1) The process of training the TRCA algorithm model using the training data of the test subject is as follows: assuming that the training data of the test subject of the nth stimulus target in the kth sub-band is represented as k = 1, 2, …, N K n = 1, 2, …, N F where N C N S N T N K N F are the number of electrode channels, the number of sampling points in a single experiment, the number of experiments for each stimulation frequency, the number of sub-bands of the filter bank, and the number of stimulation frequencies, respectively; the ith experiment of the stimulation frequency n in the kth sub-band can be represented as The optimization goal of the TRCA algorithm is to obtain the spatial filter of all stimulation frequencies in each sub-band, project the multi-channel EEG signal into a single-channel signal, maximize the sum of the covariance between experiments, that is, to find the spatial filter of each stimulation target and each sub-band maximize the following equation: where the superscript T is the transpose operator, represents the sum of the covariance matrices between all experimental pairs for the stimulation frequency n in the sub-band k; in order to obtain a finite solution, the following constraint must be imposed on the above equation: Then, the weight vector can be calculated as follows: The solution to the above equation can be expressed as an eigenvector of the matrix ​ For the stimulation target n and the sub-band k, the multi-channel template signal of the test subject is the average value of all training signals (4.2) A spatial filter integrating all the spatial filters of the stimuli targets spatially filters the multi-channel template and test signals: the spatial filters of all the stimuli targets of each sub-band are concatenated into an integrated spatial filter The TRCA algorithm for estimating the integrated spatial filter is referred to as the integrated TRCA algorithm; (4.3) The process of frequency identification of the test signal using the template matching method is as follows: assuming in the sub-band k, a single test experiment can be expressed as Using W k The spatial domain filtering is performed on the template signal and the single test experiment, and the correlation coefficient of the filtered test signal and the i-th template signal is obtained as follows: where p represents the Pearson correlation coefficient; for each stimulation frequency, the correlation coefficients of N K sub-bands are integrated according to the following equation to obtain the classification score obtained by the eTRCA algorithm:

6. The visual brain-computer interface frequency detection method combining convolutional neural network and task-related analysis according to claim 1, characterized in that, The step (5) comprises the following steps: (5.1) the classification scores of the sbCNN model and the eTRCA model are normalized to the interval [0, 1] respectively, and then the sum of the two can obtain the total classification score of each stimulation frequency as follows: In the formula, nor represents a normalization function; (5.2) test signal corresponding stimulation frequency f i , using the following equation:

Images