Spontaneous electroencephalogram decoding method of function predefined convolution

Abstract

The invention discloses a spontaneous electroencephalogram decoding method based on function predefined convolution, relates to the technical field of information, and solves the problems of overlarge neural network mapping space and poor learning ability and interpretability in spontaneous electroencephalogram signal decoding in the prior art. The method comprises the following steps: constructing a function predefined convolutional layer and determining parameters of the function predefined convolutional layer; constructing a decoding network according to the function predefined convolutional layer, and training the decoding network to obtain a trained decoding network; wherein the decoding network comprises a first function predefined convolution layer, a first batch normalization layer, a maximum pooling layer, a second function predefined convolution layer, a second batch normalization layer, a correction linear unit activation layer, an AvgPool layer, a Dropout layer, a flattening layer and a full connection layer which are connected in sequence; decoding and classifying the electroencephalogram data by using a decoding network to obtain a classification result; according to the method, the mapping space of the features is limited to signal frequency points and channel weights, and the calculation efficiency is improved.

Description

Technical Field

[0001] The invention relates to the field of information technology, and in particular to a spontaneous electroencephalogram decoding method of function predefined convolution. Background Art

[0002] Spontaneous EEG-based brain-computer interfaces are an ideal way for brain-computer interaction. Traditional spontaneous EEG decoding methods use neuroscience prior knowledge to extract handcrafted spatiotemporal features, which are computationally efficient but have insufficient performance. Neural network-based methods use end-to-end training, which greatly improves performance; however, they are less interpretable and computationally efficient than traditional methods.

[0003] In recent years, with the development of deep learning technology, more and more spontaneous EEG decoding methods based on neural networks have been proposed. Agarwal et al. used Morse wavelets to convert EEG signals in the motor cortex area into scale maps, and used deep convolutional neural networks to classify these scale map images. Alnaanah et al. studied the performance of 1D-CNN, 2D-CNN and 3D-CNN in motor imagery EEG decoding tasks. Patel et al. studied the performance of CNN and long short-term memory methods in speech imagery EEG decoding tasks. Kamble et al. designed a smoothed pseudo-Wigner-Ville distribution method based on CNN to identify word imagery EEG signals. Li et al. proposed a mixed-scale space-time dilated convolutional network to classify EEG signals of word imagery and achieved the current best performance.

[0004] Traditional EEG signal decoding methods design interpretable and computationally efficient hand-crafted features based on neuroscience methods. However, the performance of these methods cannot meet the practical application requirements of BCI. Deep learning-based decoding methods have greatly improved the decoding performance of spontaneous EEG signals. However, the "black box" nature of NN and the large number of trainable parameters lead to poor interpretability of EEG signal decoding tasks and easy overfitting. Summary of the invention

[0005] The present invention solves the problems in the prior art of excessively large neural network mapping space, poor learning ability and interpretability in spontaneous EEG decoding by providing a spontaneous EEG decoding method with a predefined function convolution, limits the mapping space of features to signal frequencies and channel weights, and improves the computational efficiency of spontaneous EEG decoding.

[0006] The present invention provides a spontaneous EEG decoding method of function predefined convolution, the method comprising:

[0007] Constructing a function-predefined convolutional layer, and determining parameters of the function-predefined convolutional layer;

[0008] Constructing a decoding network according to the function predefined convolution layer, and training the decoding network using the EEG data set to obtain a trained decoding network; wherein the decoding network includes a first function predefined convolution layer, a first batch normalization layer, a maximum pooling layer, a second function predefined convolution layer, a second batch normalization layer, a rectified linear unit activation layer, an AvgPool layer, a Dropout layer, a flattening layer, and a fully connected layer connected in sequence;

[0009] The decoding network is used to decode and classify the EEG data to obtain an EEG decoding result.

[0010] In a possible implementation manner, determining the parameters of the function predefined convolutional layer includes:

[0011] Determine a prototype function corresponding to each channel of the convolution kernel of the function predefined convolution layer, and determine the parameters of the prototype function;

[0012] Determine a parameter matrix of a predefined convolutional layer of the function according to the parameters of the prototype function, and determine a parameter vector corresponding to each channel according to the parameter matrix;

[0013] Determine the length of the convolution kernel of the function predefined convolution layer, calculate multiple convolution kernel parameter values ​​corresponding to each channel according to the length, and use the multiple convolution kernel parameter values ​​to form the function predefined convolution kernel corresponding to each channel;

[0014] Predefine the convolution kernel and the parameter vector corresponding to each channel according to the function, and calculate the weight corresponding to each channel;

[0015] According to the function, the input signal of the convolution layer and the weight corresponding to each channel are predefined, and the convolution result corresponding to each channel is calculated using the forward propagation method;

[0016] According to the input signal of the function predefined convolution layer and the convolution result corresponding to each channel, the parameters of the prototype function corresponding to each channel are updated by using the back propagation method to determine the parameters of the function predefined convolution layer.

[0017] In a possible implementation, the reverse calculation of multiple convolution kernel parameter values ​​corresponding to each channel includes:

[0018] Determining a plurality of integer positions of each channel according to the length;

[0019] The convolution kernel parameter values ​​at multiple integer positions are calculated using the prototype function corresponding to each channel and the parameters corresponding to the prototype function.

[0020] In a possible implementation, the input signal of the convolution layer and the weight corresponding to each channel are predefined according to the function, and the convolution result corresponding to each channel is calculated by the forward propagation method, including:

[0021] The weight corresponding to each channel is calculated according to the parameter vector corresponding to each channel;

[0022] The convolution calculation is performed on the input signal of the convolution layer predefined by the function and the weight corresponding to each channel to obtain the convolution result corresponding to each channel.

[0023] In a possible implementation, the updating of the parameters of the prototype function corresponding to each channel by using a back propagation method according to the input signal of the function predefined convolution layer and the convolution result corresponding to each channel to determine the parameters of the function predefined convolution layer includes:

[0024] By using the gradient descent method, the parameters of the prototype function corresponding to each channel are updated according to the input signal of the function predefined convolution layer and the convolution result corresponding to the input signal of the function predefined convolution layer.

[0025] In a possible implementation, during the training of the decoding network, a cross entropy loss function is used to calculate the error of the decoding network, and the parameters of the decoding network are adjusted according to the error using an Adam estimation method.

[0026] In a possible implementation, the first function predefines the number of convolution kernels of the convolution layer, the number of convolution kernels of the first batch normalization layer, the number of convolution kernels of the first batch normalization layer, and the number of convolution kernels of the second batch normalization layer to be positively correlated with the number of categories of the EEG data.

[0027] In a possible implementation, the convolution kernel lengths of the first function predefined convolution layer, the second function predefined convolution layer, and the AvgPool layer are all related to the length of the input signal of the function predefined convolution layer.

[0028] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0029] The present invention proposes a function predefined convolution layer to capture the key frequency points and channel weights of spontaneous EEG signals. Subsequently, the present invention designs a decoding network based on FPC to improve the efficiency of spontaneous EEG decoding. It can effectively utilize the computational efficiency of traditional methods and the advantages of learning ability based on neural network methods, which has important application value for deploying high-performance spontaneous EEG signal decoding algorithms on devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flowchart of the steps of the spontaneous EEG decoding method of the predefined convolution function provided in an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of a function predefined convolutional layer structure provided in an embodiment of the present invention;

[0032] Figure 3A A schematic diagram of frequency distribution of a decoding network provided in an embodiment of the present invention after training on the GigaScience dataset;

[0033] Figure 3B A schematic diagram of the frequency distribution of the decoding network provided in an embodiment of the present invention after training on the Linguistic BCI dataset;

[0034] Figure 3C A schematic diagram of frequency distribution of a decoding network provided in an embodiment of the present invention after training on an Imagined Speech dataset;

[0035] Figure 4A A region division result diagram in the channel weight visualization experiment result of the FPC layer provided by an embodiment of the present invention;

[0036] Figure 4B A schematic diagram of channel weight distribution of FPC layer parameters trained with the GigaScience dataset provided in an embodiment of the present invention;

[0037] Figure 4C A schematic diagram of the channel weight distribution of the FPC layer parameters trained with the Linguistic BCI dataset provided in an embodiment of the present invention;

[0038] Figure 4D A schematic diagram of the channel weight distribution of FPC layer parameters trained with the Imagined Speech dataset provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The present invention provides a spontaneous EEG decoding method with a predefined convolution function, such as Figure 1 As shown, the method includes the following steps S101 to S103.

[0041] S101, constructing a function predefined convolution layer, and determining parameters of the function predefined convolution layer;

[0042] Specifically, in step S101, a function predefined convolution layer is constructed, and parameters of the function predefined convolution layer are determined, including the following steps S1011 to S1016.

[0043] S1011, determining a prototype function corresponding to each channel of a convolution kernel of a function predefined convolution layer, and determining parameters of the prototype function;

[0044] S1012, determining a parameter matrix of a predefined convolutional layer of the function according to the parameters of the prototype function, and determining a parameter vector corresponding to each channel according to the parameter matrix;

[0045] S1013, determine the length of the convolution kernel of the function predefined convolution layer, calculate multiple convolution kernel parameter values ​​corresponding to each channel according to the length, and use the multiple convolution kernel parameter values ​​to construct the function predefined convolution kernel corresponding to each channel.

[0046] Here, multiple convolution kernel parameter values ​​corresponding to each channel are calculated according to the length, including:

[0047] (1) Determine multiple integer positions of each channel according to the length;

[0048] (2) Calculate the convolution kernel parameter values ​​at multiple integer positions through the prototype function corresponding to each channel and the parameters corresponding to the prototype function.

[0049] S1014, calculating the weight corresponding to each channel according to the function predefined convolution kernel and the parameter vector corresponding to each channel;

[0050] S1015, according to the function pre-defines the input signal of the convolution layer and the weight corresponding to each channel, and uses the forward propagation method to calculate the convolution result corresponding to each channel.

[0051] Here, the forward propagation method is used to calculate the convolution result corresponding to each channel according to the input signal of the predefined convolution layer and the weight corresponding to each channel, including:

[0052] (1) Calculate the weight corresponding to each channel according to the parameter vector corresponding to each channel;

[0053] (2) Perform convolution calculation based on the input signal of the predefined convolution layer and the weight corresponding to each channel to obtain the convolution result corresponding to each channel.

[0054] S1016, based on the input signal of the function predefined convolution layer and the convolution result corresponding to each channel, the parameters of the prototype function corresponding to each channel are updated using the back propagation method to determine the parameters of the function predefined convolution layer.

[0055] Here, the back propagation method is used to update the parameters of the prototype function corresponding to each channel according to the input signal of the function predefined convolution layer and the convolution result corresponding to the input signal of the function predefined convolution layer, including: using the gradient descent method to update the parameters of the prototype function corresponding to each channel according to the input signal of the function predefined convolution layer and the convolution result corresponding to the input signal of the function predefined convolution layer.

[0056] Exemplarily, in the structure of the function predefined convolutional layer (FPC), such as Figure 2 As shown in the figure, the differentiable function f(·) is pre-set as the prototype function, so as to limit the feature mapping space to the transformation space of the prototype function, where the parameters of the prototype function f(·) {a1, a2, …, a n} is the object to be optimized. The parameter matrix in FPC stores the parameters of the m channels of f(·), as follows:

[0057]

[0058] For channel m, the parameter vector is expressed as:

[0059] p m =[a m1 ,a m2 ,…,a mn ].

[0060] In addition, an integer vector is set to store the length information of the convolution kernel i = {1, 2, 3, ..., k}, where k corresponds to the length of the convolution kernel.

[0061] During the forward propagation process, each set of parameters is a constant term of the prototype function f(·), and the output value of each integer position is calculated as the convolution kernel parameter value at that position.

[0062] The set of outputs at all positions constitutes a function predefined convolution kernel (Function PredefinedKernal), and m groups of parameters are calculated in parallel to obtain m function predefined convolution kernels.

[0063] For the mth function, the convolution kernel is predefined, and the convolution kernel weight vector w m It can be expressed as:

[0064] w m =[f(1,p m ),f(2,p m ),…,f(k,pm )].

[0065] The input signal X of the function predefined convolution layer is convolved with the function predefined convolution kernel to obtain the output Y of the FPC layer. For the mth channel of the output Y, the convolution result can be expressed as:

[0066] y m =X*w m ,

[0067] Among them, * represents the convolution operation.

[0068] During the back-propagation process, the gradient descent method is used to optimize the parameters of the prototype function f(·).

[0069] Error value of FPC layer Use the chain derivation method to calculate, where L represents the loss function of the neural network, y l Represents y m The lth element of Represents the partial derivative operation.

[0070] Parameter a mn The partial derivative of The calculation formula is:

[0071]

[0072] Parameter a mn The update formula is:

[0073]

[0074] in, Indicates a mn The original value of Indicates a mn Updated value. η and batchsize are the hyperparameters of the decoding network. The hyperparameter η determines the update speed of training, and batchsize represents the number of samples used for each update during training.

[0075] In particular, when the prototype function f(·) is set to a sine function, according to the convolution theorem:

[0076]

[0077] Among them, F represents Fourier transform, and the function predefines that only signals with the same frequency components in the input signal X and f(·) of the convolution layer will be retained by the FPC layer. This shows that the FPC layer can extract specific frequency points in the signal in a trainable way.

[0078] S102, constructing a decoding network according to the function predefined convolutional layer, and training the decoding network using the EEG data set to obtain a trained decoding network; wherein the decoding network includes a first function predefined convolutional layer, a first batch normalization layer, a maximum pooling layer, a second function predefined convolutional layer, a second batch normalization layer, a rectified linear unit activation layer, an AvgPool layer, a Dropout layer, a flattening layer and a fully connected layer connected in sequence.

[0079] Specifically, in step S102, during the training of the decoding network, the error of the decoding network is calculated using the cross entropy loss function, and the parameters of the decoding network are adjusted according to the error using the Adam estimation method.

[0080] Specifically, the first function predefines the number of convolution kernels of the convolution layer, the number of convolution kernels of the first batch normalization layer, the number of convolution kernels of the first batch normalization layer, and the number of convolution kernels of the second batch normalization layer, which are all positively correlated with the number of categories of the EEG data.

[0081] Specifically, the convolution kernel lengths of the first function predefined convolution layer, the second function predefined convolution layer, and the AvgPool layer are all related to the length of the input signal of the function predefined convolution layer.

[0082] Exemplarily, in order to improve the decoding efficiency of spontaneous EEG signals, the present invention proposes a decoding network (FPCNN). FPCNN uses two FPC layers to extract channel weights and signal frequency features. The two FPC layers are followed by batch normalization (BatchNorm) layers to provide regularization. FPCNN uses a rectified linear unit (ReLU) activation layer for nonlinear transformation, a Dropout layer to improve the generalization ability of the network, and two pooling layers for auxiliary dimensionality reduction.

[0083] Specifically, a maximum pooling layer (MaxPool) with a kernel length of 2 is placed between the two FPC layers to maintain the shape of the feature waveform, and an AvgPool layer is placed after the second FPC layer to further smooth the features. Finally, a fully connected layer (FC) layer is used for classification. During the training process, the cross entropy loss function is used to calculate the error between the FPCNN output result and the label, and the self-Adam estimation method is used to optimize the parameters of the FPCNN.

[0084] The proposed FPCNN has two hyperparameters R and S. The hyperparameter R represents the number of convolution kernels in the FPC layer. The first FPC layer has 2R convolution kernels, and the second FPC layer has R convolution kernels. In addition, the parameters of the first and second BatchNorm layers are 2R and R, respectively. The empirical value of R is 8 times the number of data categories to be decoded. The hyperparameter R represents the kernel length of the FPC and AvgPool layers. When the sampling rate of the data is q Hz, S is set to q / 4, and the FPC layer can capture the signal frequency of 4 to q / 2 Hz. The above hyperparameter settings are applicable to spontaneous EEG signals.

[0085] S103, using a decoding network to decode and classify the EEG data to obtain an EEG decoding result.

[0086] In a specific simulation embodiment provided by the present invention, in order to verify the performance of the proposed decoding model on the spontaneous EEG decoding task, the present invention conducted experiments on three data sets: GigaScience (Giga), Linguistic BCI (LBCI) and Imagined Speech (IS).

[0087] For the GigaScience (Giga) dataset: This dataset implements the motor imagery experimental paradigm, including the motor imagery tasks of "left hand" and "right hand". In one trial of the experiment, the paradigm program first presents a 2-second attention-guiding cross; then randomly gives the left and right hand motor imagery prompts, and the subjects perform the left and right hand motor imagery tasks for 3 seconds after seeing the prompts; finally, the 2-second attention-guiding cross is presented again. This paradigm includes 100 or 120 trials of imagination tasks for the left / right hand. During the execution of the paradigm, the EEG acquisition device synchronously records the subject's EEG signal and the time tag signal issued by the paradigm program, thereby completing the acquisition of a trial of motor imagery EEG signals. The 52 subjects in this dataset are 24.8±3.86 years old. The EEG acquisition device uses 64-lead Ag / AgCl dynamic electrodes and records data at a sampling frequency of 512Hz.

[0088] For the Linguistic BCI (LBCI) dataset: This dataset implements the pronunciation imagination experiment paradigm, which includes four imagined pronunciations: "BA", "FO", "LE" and "RY".

[0089] In one trial of the experiment, the prompt for the pronunciation imagination task was first randomly given; then there was a random interval of 3 to 7 seconds to eliminate the possible practice effect; then the experimental paradigm program issued a buzzer prompt, and the subject began to perform the pronunciation imagination task given by the task prompt after the prompt. The task was completed within 1 second or 312 milliseconds according to the requirements; the EEG acquisition equipment synchronously recorded the subject's EEG signal and the time label signal issued by the paradigm program, thereby completing the acquisition of pronunciation imagination EEG signals for one trial.

[0090] The 10 subjects in this dataset are aged between 21 and 33 years old. Each subject underwent 4 rounds of experiments, each of which contained 30 trials. Each trial collected an EEG signal of pronunciation imagination. The EEG acquisition device was Biosemi's ActiveTwo TM The release team performed a 256Hz downsampling process and used a 49.2-50.8Hz FIR filter to perform a 50Hz power frequency filter. The AAR tool of the EEGLAB toolbox was used for preprocessing, and the SOBI and CCA methods were used to remove the effects of EOG and EMG artifacts in the EEG signal.

[0091] For the Imagined Speech (IS) dataset: This dataset implements the Chinese word imagination experimental paradigm, which includes the Chinese word pronunciation imagination tasks of four kinds of fruits, such as "apple", "banana", "grape", and "watermelon", and four kinds of animals, such as "baboon", "lion", "rhino", and "zebra". In one trial of the experiment, the paradigm program first appears an attention-guiding cross for 2 seconds; then the screen displays a picture of one of the eight words, and the subject performs the word imagination task according to the picture prompt. This paradigm contains a total of 10 experimental modules, each module contains 20 experimental units, each experimental unit contains 8 trials, and each experimental unit contains 3 to 5 seconds of random preparation time and 5 seconds of rest time before and after. During the execution of the paradigm, the EEG acquisition equipment synchronously records the EEG signals of the subjects and the time tag signals issued by the paradigm program, thereby completing the acquisition of EEG signals of word imagination in one trial. The age range of the 9 subjects in this dataset is 22 to 26 (22.89±1.20) years old. Each subject collected EEG data for 200 trials of each word (a total of 1600 trial data samples). The EEG acquisition device was Biosemi's ActiveTwo TM The release team downsampled the video to 256 Hz and performed a 50 Hz notch filter.

[0092] In the following experiments, the time interval after the start prompt is [0s, 3s] in each sample of the GigaScience dataset, and the time interval is [0s, 1s] for the LBCI and IMAGINED SPEECH datasets. Specifically, the original EEG signal of the trial is first bandpass filtered using a zero-phase 1-128Hz FIR filter to remove DC bias and high-frequency EMG noise; then a 50Hz FIR notch filter is used to remove the mains line noise; finally, the filtered EEG trial data samples are resampled to 256Hz.

[0093] Simulation experiment settings:

[0094] On the GigaScience dataset, the FPCNN involved in the present invention is compared with five existing algorithms related to the field, namely FBCSP, PSCSP, ConvNet, CWT-CNN and CNN1D_MF. On the Linguistic BCI dataset, the FPCNN involved in the present invention is compared with five existing algorithms related to the field, namely CSP+SVM, DWT+MLP, EEGNET, CNN-LSTM and SPWVD-CNN. On the Imagined Speech dataset, the FPCNN involved in the present invention is compared with five existing algorithms related to the field, namely CSP+SVM, DWT+MLP, EEGNET, SPWVD-CNN and HS-STDCN.

[0095] The classification accuracy is used to evaluate the performance of all the above methods, and the hyperparameters [R, S] of FPCNN are set as follows: GigaScience dataset [3, 64], LINGUISTIC BCI dataset [32, 64], IMAGINED SPEECH dataset [64, 64]. During the network training process, the Adam optimizer is used and the batch size (batchsize) is set to 32.

[0096] Experimental results:

[0097] (1) The experimental results of GigaScience dataset are shown in Table 1:

[0098] Table 1 Classification accuracy and standard deviation experimental results on the GigaScience dataset (1 = 100%)

[0099] Comparison Method Accuracy Standard Deviation FBCSP 0.7067 0.1361 PSCSP 0.7350 0.1401 ConvNet 0.7874 0.1463 CWT-CNN 0.8061 0.1410 CNN1D_MF 0.7823 0.1376 FPCNN 0.8270 0.1415

[0100] Table 1 lists the classification results on the GigaScience dataset. The FPCNN method proposed in this paper achieves the best classification performance. The classification accuracy is 9.20% higher than the best traditional signal processing method PSCSP, 3.96% higher than the classic deep learning technology ConvNet, and 2.09% higher than the best existing deep learning technology CWT-CNN. It can be seen that FPCNN outperforms all competitors in motor imagery spontaneous EEG decoding.

[0101] (2) Experimental results on the Linguistic BCI dataset

[0102] Table 2 shows the classification results of all the comparison methods on the Linguistic BCI dataset. FPCNN outperforms other competitors in the syllable imagination decoding task. This method is 18.6% higher than the best traditional signal processing technology DWT+MLP, 5.56% higher than the classic deep learning technology EEGNET, and 3.08% higher than the current best deep learning technology SPWVD-CNN.

[0103] Table 2 Classification accuracy and standard deviation experimental results on the Linguistic BCI dataset (1 = 100%)

[0104]

[0105]

[0106] (3) Experimental results of Imagined Speech dataset

[0107] Table 3 Classification accuracy and standard deviation experimental results on the Imagined Speech dataset (1 = 100%)

[0108] Comparison Method Accuracy Standard Deviation CSP+SVM 0.1643 0.0138 DWT+MLP 0.2420 0.0412 EEGNET 0.4993 0.0495 SPWVD-CNN 0.5104 0.0503 HS-STDCN 0.5431 0.0522 FPCNN 0.5772 0.0535

[0109] The classification results on the Imagined Speech dataset are shown in Table 3. Compared with the best traditional signal processing method DWT+MLP, the classification accuracy of this method is improved by 33.52%, compared with the classic deep learning method EEGNET, it is improved by 7.79%, and compared with the current best deep learning method HS-STDCN, it is improved by 3.41%. It can be seen that FPCNN is superior to other comparison methods in the word imagination decoding task.

[0110] Experimental results analysis

[0111] (1) Visualization of prototype function frequency distribution

[0112] In order to further study the information learned by the network during training, this work conducted a visualization experiment to show the frequency distribution of the prototype function parameters in the trained FPC layer. This experiment calculated all frequency parameters between 0 and 128 Hz of the FPC layer after training for all subjects on the three datasets. The visualization results are compared with the correlation of brain tasks as follows:

[0113] (1) For the motor imagery task (GigaScience dataset), Figure 3A The visualization results in show that the 8-40Hz frequency band has the highest weight after training. This learned frequency band matches the activation rhythm (μ and β rhythm) of motor imagery tasks in existing studies. In addition, the high-frequency band (60-80Hz) also shows a high weight. In previous studies, the role of this frequency band in motor imagery decoding tasks has often been overlooked. This visualization experiment proves that the high-frequency band also plays a role in the classification of spontaneous EEG of motor imagery.

[0114] (2) For the pronunciation imagination task (LINGUISTIC BCI dataset), the high-frequency band (>70Hz) has a higher weight, such as Figure 3B This means that the most useful information learned by the network for classifying imaginary sounds comes from the high-frequency band. This finding is consistent with previous research results on language BCI.

[0115] (3) For the word imagination task (IMAGINED SPEECH dataset), Figure 3C It shows that the FPC layer has learned high weights in the high-frequency band, which is a common feature of language EEG. Unlike the pronunciation imagination task, in the word imagination task, the 28-44Hz frequency band (belonging to the gamma rhythm) also learned a higher weight. This phenomenon shows that although both pronunciation imagination and word imagination tasks are language tasks, there are certain differences in brain activity. This finding has practical value for designing language BCI paradigms and decoding methods. Figure 3A , Figure 3B and Figure 3C The frequency distribution after training on the datasets Giga, LBCI and IS. The visualization results calculate all frequency parameters between 0-128Hz. The horizontal axis represents the frequency (Hz) and the vertical axis represents the normalized weight value. The frequency visualization experiment of the FPC layer proves the effectiveness of this work for spontaneous EEG feature extraction.

[0116] The trained FPC layer can clearly extract the frequency information of brain activity in spontaneous EEG tasks, and can be cross-corroborated with existing brain activity research results in the field of neuroscience. The design of the FPC layer in this work enhances the interpretability of the network and improves the decoding efficiency of spontaneous EEG signals.

[0117] Prototype function spatial weight visualization: In order to further study the spatial features learned by the proposed FPCNN, this experiment visualizes the channel weight distribution of the FPC layer.

[0118] The average channel weight parameters of all subjects in each dataset were calculated and plotted as brain topography, as shown in Figure 4A , Figure 4B , Figure 4C and Figure 4D shown. Figure 4A A schematic diagram of the functional area division of the cerebral cortex is given. Figure 4B , Figure 4C and Figure 4D The channel weight distribution visualization results of the Giga, LBCI and IMAGINED SPEECH datasets are shown respectively.

[0119] (1) For the motor imagery task (GigaScience dataset), Figure 4B It shows that motor function and sensory areas are activated during motor imagery tasks. These two brain areas are the main areas used to perform motor imagery tasks, and correspondingly, the trained FPC layer also learns the highest channel weights in the corresponding areas. In addition, information from the higher mental function area, somatosensory association area, Broca's area, and Wernicke's area that assist in motor imagery tasks also contributes to the classification of motor imagery EEG. This shows that the cerebral cortical areas that receive task instructions and control movements during the task also play a role in motor imagery decoding.

[0120] (2) For the articulation imagery task (LINGUISTIC BCI dataset), Broca's area, Wernicke's area, motor function area, and association area have higher channel weights in the trained FPC layer, such as Figure 4C As shown. Existing studies have considered these areas as the main areas for language tasks. In addition, visual and auditory areas also showed higher weights on the parameter brain topography. This phenomenon may be related to the experimental paradigm that uses visual and auditory stimuli to issue imagination task instructions.

[0121] (3) For the word imagination task (IMAGINED SPEECH dataset), Figure 4D The visualization results show results that are common to language tasks. At the same time, it can be observed that, unlike the pronunciation imagination task, the higher mental function area showed a higher level of activity in the word imagination task. This phenomenon may be related to the experimental paradigm that uses pictures to issue task instructions. Specifically, the experimental paradigm of the IMAGINED SPEECH dataset uses 4 kinds of fruit and 4 kinds of animal pictures to stimulate subjects to imagine words. The information generated by the judgment process of the higher mental function area also helps the network to classify EEG signals.

[0122] The visualization results of the channel weights of the FPC layer clearly show the spatial feature extraction process of spontaneous EEG. This experiment proves the interpretability of the proposed FPCNN. In addition, the method proposed in this work also provides a practical tool for the problem of brain region functional localization in the study of spontaneous EEG signals.

[0123] Figure 4A The functional area division of the cerebral cortex is shown, which is specifically: (1) higher psychological function area; (2) motor function area (eye movement and orientation); (3) motor function area (voluntary muscles); (4) sensory area; (5) somatosensory association area; (6) visual area; (7) association area; (8) Broca's area; (9) auditory area; (10) Wernicke's area.

[0124] Figure 4B , Figure 4C and Figure 4D These are the channel weight distributions of the FPC layer parameters trained on the GigaScience dataset, the Linguistic BCI dataset, and the IMAGINED SPEECH dataset, where the red area represents high weights and the blue area represents low weights.

[0125] Comparison of network parameters: In order to compare the complexity of different neural network methods, this experiment calculates the number of trainable parameters of all recent comparison methods. Assume that the EEG dataset has 32 channels, a sample duration of 1 second, a sampling frequency of 512Hz, and 2 categories. The number of trainable parameters of all recent comparison methods is shown in Table 4. It can be clearly observed from Table 4 that there are significant differences in the number of trainable parameters of these models. For example, the model with the most parameters is SPWVD-CNN, which has 39,132,384 trainable parameters. The number of parameters is 22,858 times that of the proposed FPCNN, which has only 1,712 trainable parameters. Compared with the three existing best methods (CWT-CNN, SPWVD-CNN, and HS-STDCN) mentioned above, the proposed FPCNN reduces the number of trainable parameters by 99.96%, 99.99%, and 92.17%, respectively.

[0126] The main reason for the large difference in the number of trainable parameters is the difference in network structure. SPWVD-CNN and CWT-CNN extract features from EEG data in the preprocessing stage and save them in the form of images. This makes these networks image processing networks, and accordingly, they also have image processing network-level parameters. CNN1D_MF and HS-STDCN have fewer parameters because their input is the original EEG signal and uses a one-dimensional convolution operator for feature extraction.

[0127] Table 4 Parameter statistics of the comparison methods

[0128] method Parameter quantity SPWVD-CNN 39，132，384 CNN-LSTM 9，456，752 CWT-CNN 4，603，380 CNN1D_MF 226，575 HS-STDCN 21，872 FPCNN 1，712

[0129] Unlike image processing tasks, models with large parameter counts can easily overfit the training data of the EEG dataset, resulting in the inability to extract common features between trials. In addition, neural network models with large parameter counts consume a lot of storage and computing resources, limiting the practical application of these methods. The proposed FPCNN fully considers the prior knowledge of existing research and is more conducive to extracting common frequency features between trials; at the same time, the number of parameters of this method is greatly reduced and the computational efficiency is much higher than that of existing methods.

[0130] Experimental summary: In order to improve the spontaneous electroencephalogram (EEG) decoding performance, a novel FPCNN method is proposed to decode spontaneous EEG signals. This method combines the advantages of the computational efficiency of traditional methods and the learning ability of NN-based methods. In order to evaluate the performance of FPCNN, experiments were conducted on three datasets: GigaScience (Giga), Linguistic BCI (LBCI), and ImaginedSpeech (IS). The experimental results show that compared with the comparative methods, FPCNN has a significant improvement in classification accuracy results, and has the advantages of being efficient, stable, and interpretable.

[0131] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of the present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0132] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the present invention.

Claims

1. A spontaneous EEG decoding method with predefined convolution function, characterized in that: include: Constructing a function-predefined convolutional layer, and determining parameters of the function-predefined convolutional layer; Constructing a decoding network according to the function predefined convolution layer, and training the decoding network using the EEG data set to obtain a trained decoding network; wherein the decoding network includes a first function predefined convolution layer, a first batch normalization layer, a maximum pooling layer, a second function predefined convolution layer, a second batch normalization layer, a rectified linear unit activation layer, an AvgPool layer, a Dropout layer, a flattening layer, and a fully connected layer connected in sequence; The decoding network is used to decode and classify the EEG data to obtain an EEG decoding result.

2. The spontaneous EEG decoding method of the function predefined convolution according to claim 1, characterized in that: The determining of the parameters of the function predefined convolutional layer comprises: Determine a prototype function corresponding to each channel of the convolution kernel of the function predefined convolution layer, and determine the parameters of the prototype function; Determine a parameter matrix of a predefined convolutional layer of the function according to the parameters of the prototype function, and determine a parameter vector corresponding to each channel according to the parameter matrix; Determine the length of the convolution kernel of the function predefined convolution layer, calculate multiple convolution kernel parameter values ​​corresponding to each channel according to the length, and use the multiple convolution kernel parameter values ​​to form the function predefined convolution kernel corresponding to each channel; Predefine the convolution kernel and the parameter vector corresponding to each channel according to the function, and calculate the weight corresponding to each channel; According to the function, the input signal of the convolution layer and the weight corresponding to each channel are predefined, and the convolution result corresponding to each channel is calculated using the forward propagation method; According to the input signal of the function predefined convolution layer and the convolution result corresponding to each channel, the parameters of the prototype function corresponding to each channel are updated by using the back propagation method to determine the parameters of the function predefined convolution layer.

3. The spontaneous EEG decoding method of the function predefined convolution according to claim 2, characterized in that: The calculating a plurality of convolution kernel parameter values ​​corresponding to each channel according to the length includes: Determining a plurality of integer positions of each channel according to the length; The convolution kernel parameter values ​​at multiple integer positions are calculated using the prototype function corresponding to each channel and the parameters corresponding to the prototype function.

4. The spontaneous EEG decoding method of the function predefined convolution according to claim 2, characterized in that: The method of predefining the input signal of the convolution layer and the weight corresponding to each channel according to the function and calculating the convolution result corresponding to each channel by the forward propagation method includes: The weight corresponding to each channel is calculated according to the parameter vector corresponding to each channel; The convolution calculation is performed on the input signal of the convolution layer predefined by the function and the weight corresponding to each channel to obtain the convolution result corresponding to each channel.

5. The spontaneous EEG decoding method of the function predefined convolution according to claim 2, characterized in that: The method of updating the parameters of the prototype function corresponding to each channel by using a back propagation method according to the input signal of the function predefined convolution layer and the convolution result corresponding to each channel to determine the parameters of the function predefined convolution layer includes: By using the gradient descent method, the parameters of the prototype function corresponding to each channel are updated according to the input signal of the function predefined convolution layer and the convolution result corresponding to the input signal of the function predefined convolution layer.

6. The spontaneous EEG decoding method of the function predefined convolution according to claim 1, characterized in that: In the process of training the decoding network, the error of the decoding network is calculated using a cross entropy loss function, and the parameters of the decoding network are adjusted according to the error using an Adam estimation method.

7. The spontaneous EEG decoding method of the function predefined convolution according to claim 1, characterized in that: The first function predefines the number of convolution kernels of the convolution layer, the number of convolution kernels of the first batch normalization layer, the number of convolution kernels of the first batch normalization layer, and the number of convolution kernels of the second batch normalization layer to be positively correlated with the number of categories of the EEG data.

8. The spontaneous EEG decoding method of the function predefined convolution according to claim 1, characterized in that: The convolution kernel lengths of the first function predefined convolution layer, the second function predefined convolution layer, and the AvgPool layer are all related to the length of the input signal of the function predefined convolution layer.

Images