Decoding method based on frequency band combination enhancement and multi-view convolutional neural network

Abstract

The present invention discloses a decoding method based on frequency band combination enhancement and multi-view convolutional neural network, comprising the following steps: S1, collecting human brain electrical signals through motor imagery; S2, preprocessing the brain electrical signals collected in step S1; S3, constructing a frequency band combination enhancement and multi-view convolutional neural network; S3-1, constructing a frequency band combination enhancement module FBC(·), constructing a multi-view convolutional module F MV (·), 1D-CNN construction; S3-2, the constructed frequency band combination enhancement module FBC (·), multi-view convolution module F MV (·) and 1D-CNN are sequentially combined into a convolutional neural network F based on frequency band combination enhancement and multi-view FBC_MV_CNN(·) ; S4. Using the preprocessed EEG signal obtained in step S2 to train the frequency band combination enhancement and multi-view convolutional neural network, this method can effectively decode the motor imagery EEG signal (MI-EEG) in brain-computer interface (BCI) research, thereby effectively improving the prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition, and in particular to a decoding method based on frequency band combination enhancement and multi-view convolutional neural network. Background Art

[0002] Electroencephalogram (EEG) signals have become a common focus for exploring the underlying physiological processes associated with various human behaviors, such as emotion, memory, and motor control. EEG-based motor intention recognition has been widely used in brain-computer interface (BCI) research. Compared with traditional evoked potential signals (such as P300 and steady-state visual evoked potential), motor imagery EEG (MI-EEG) is more spontaneous and more suitable for online BCI research, particularly in the field of rehabilitation medicine. For example, previous studies have utilized MI-BCI to design autonomously operated bionic prostheses and develop feedback rehabilitation devices for stroke patients. Furthermore, accurate prediction of motor execution (ME) for different tasks has great potential for optimizing applications such as robotic control.

[0003] In order to precisely control BCI systems, it is crucial to accurately identify and decode control signals. In recent years, common spatial patterns (CSP) and its related improved algorithms (e.g., FBCSP) have become the mainstream for extracting MI-EEG features. In addition, machine learning methods (e.g., support vector machines, SVMs) have been used for EEG feature extraction and MI classification. Sahar et al. used CSP and SVM based on open source datasets to obtain good motor imagery classification accuracy. However, machine learning algorithms rely too much on feature selection, so extracting appropriate features for different tasks remains a challenge.

[0004] Recently, automatic feature extraction methods based on convolutional neural networks (CNNs) have been applied to various engineering fields, including computer vision, time series prediction, and motor imagery recognition. The convolutional layers of CNNs use learnable convolution kernels and artificial neural networks (ANNs) to implement a process that integrates feature extraction and decoding, thereby overcoming the reliance of machine learning on feature extraction algorithms. Uktveris T. et al. proposed a method to generate two-dimensional feature maps from one-dimensional EEG feature vectors and used CNNs to classify EEG on a four-category task, achieving a classification accuracy of up to 68%. Lee et al. combined a CNN with a time-frequency representation of EEG and achieved a classification accuracy of 78.93% on two classification tasks.

[0005] To address the problem of feature extraction for each channel, Sun et al. designed a squeeze and excitation (SE) block on CNN to adaptively recalibrate channel feature responses, further improving the overall performance of CNN in MI classification. Tabar et al. used stacked autoencoders (SAE) to extract EEG time, frequency, and position information and achieved good results based on CNN on the BCI IV-2b dataset. However, these CNN algorithms only consider the time domain features of the EEG sequence and the spatial domain features of the electrode position, ignoring the correlation information between different channels and between frequency bands, which has been shown to be important in the functional brain network analysis of stroke patients. In addition, most studies require pre-extraction of EEG features and use them as input to CNN (such as spectrograms), without fully utilizing the ability of CNN to automatically extract features.

[0006] To address these problems, this paper proposes a frequency band combination (FBC) module and a multi-view feature extraction structure (MV structure) for EEG-CNN. Summary of the Invention

[0007] In response to the shortcomings of the existing technology, the present invention proposes a decoding method based on frequency band combination enhancement and multi-view convolutional neural network, which can effectively decode motor imagery electroencephalogram (MI-EEG) signals in brain-computer interface (BCI) research, thereby effectively improving the prediction accuracy.

[0008] In order to solve the above technical problems, the technical solution of the present invention is:

[0009] A decoding method based on frequency band combination enhancement and multi-view convolutional neural network includes the following steps:

[0010] S1. Collect human brain electrical signals through motor imagery;

[0011] S2, preprocessing the EEG signal collected in step S1;

[0012] S3, construct frequency band combination enhancement and multi-view convolutional neural network;

[0013] S3-1, construct frequency band combination enhancement module FBC(·), construct multi-view convolution module F MV (·),1D-CNN construction;

[0014] S3-2, the constructed frequency band combination enhancement module FBC(·), multi-view convolution module F MV (·) and 1D-CNN are sequentially combined into a convolutional neural network F based on frequency band combination enhancement and multi-view FBC_MV_CNN (·);

[0015] S4. Using the preprocessed EEG signal obtained in step S2, training the frequency band combination enhancement and multi-view convolutional neural network.

[0016] Preferably, in step S1, the method for collecting EEG signals is:

[0017] With the mastoid process behind the left ear as the reference and the forehead as the ground, EEG signals of 22 channels, including CZ, C2, C3, C4, C5, C6, FCZ, FC1, FC2, FC3, FC4, FC5, FC6, C-1, CP2, CP3, CP4, CP5, CP6, CP7, and CP8, were collected at a frequency of 1000 Hz;

[0018] The subjects were asked to imagine the movements of their left hand, right hand, feet and tongue according to visual cues.

[0019] Preferably, in step S2, the EEG signal preprocessing method is: performing ICA independent component analysis, segmenting the processed EEG signal, and extracting EEG signals during four types of actions.

[0020] The ICA independent component analysis method is analyzed using the SOBI algorithm.

[0021] Preferably, the method for constructing the frequency band combination enhancement module FCB(·) is:

[0022] Assuming that the size of the original data X is N*P, where N is the number of channels and P is the number of sampling points, four bandpass filters of different frequency bands are performed on X respectively:

[0023]

[0024]

[0025]

[0026]

[0027] Then the original data and the four filtered data are superimposed and combined along the channel direction to form the enhanced data X FBC :

[0028]

[0029] Preferably, the multi-view convolution module F MV (·) construction method:

[0030] Six feature extraction groups of different sizes and step sizes are defined, wherein the feature extraction groups include two small-field-of-view S modules and four large-field-of-view Sx modules;

[0031] The small field of view S module includes 1-2 convolution kernels, a batch normalization layer and an activation function layer;

[0032] The large field of view Sx module includes a convolution layer, a pooling layer, a batch normalization layer and an activation function layer;

[0033] The output feature maps of the six modules are connected along the depth direction as the input signal of 1D-CNN.

[0034] Preferably, the construction method of the 1D-CNN is:

[0035] The incoming signal Input is input in a two-dimensional matrix format, where the vertical direction is the channel alignment direction and the horizontal direction is the time progression direction.

[0036] Input∈R (C×T) (6)

[0037] Where C represents the number of features and T represents the length of the time series;

[0038] Establish a spatial feature extraction part S(·), which includes 1-2 convolutional layers S i , the vertical size of the convolution kernel is the number of input channels. Each convolution kernel extracts information between EEG channels when moving and forms a feature map with a vertical dimension of 1, S i and S(·) are defined as follows:

[0039] S(·)=Conv Ti [BN Ti (·)]∈R (C′×T) (7)

[0040] S(·)=S i [S i-1 (...)]∈R (C′×T) (8)

[0041] Where C′ represents the number of convolution kernels, and the size of the convolution kernel is (C, K Si ), K Si is the convolution kernel length;

[0042] Establish a time feature extraction part G(·), which contains three convolutional layers g i(·), the convolution kernel in each convolution layer is 1 in the longitudinal direction, that is, the horizontal convolution kernel. The horizontal convolution kernel abstracts the time domain information of the feature map and forms a high-dimensional feature. Each convolution layer contains a BN operation and a nonlinear mapping based on Leaky ReLU. The time feature extraction part G(·) takes the feature map output by the previous part as input and outputs a high-dimensional feature map with spatiotemporal information. The feature map is downsampled in the average pooling layer and then expanded into a feature vector using a 1*1 convolution operation. i The definitions of (·) and G(·) are as follows:

[0043] g i，1 (·)=Conv2{BN2[LeakyReLU 12 (·)]}∈R (C″×T) (9)

[0044] g i，2 (·)=Conv3{BN3[LeakyReLU 23 (·)]}∈R (C″′×T) (10)

[0045] g i (·)=g i，2 (g i，1 (·)) (11)

[0046] G(·)=g i (g i-1 (...)) (12)

[0047] Where C″ represents the number of convolution kernels in the second layer, the convolution kernel size is (1, K2), and K2 is the convolution kernel length; C″· represents the number of convolution kernels in the third layer, the convolution kernel size is (1, K3), and K3 is the convolution kernel length;

[0048] Build 1D-CNN model F 1D_CNN (·), using the vertical convolution kernel and the horizontal convolution kernel to extract the spatial and temporal features of the input signal Input in sequence, the 1D-CNN model F 1D_CNN The fully connected part of (·) includes an input layer, a hidden layer, and an output layer. The number of nodes in the hidden layer is set to 32.

[0049] F 1D_CNN (·)=FC 1，2，3 {AveragePooling[G(S(·))]} (13)

[0050] Among them FC i Indicates the full connection of the i-th layer.

[0051] Preferably, the convolutional layers all use a valid padding strategy, the BN operations all perform batch normalization on the convolutional layer outputs in the channel dimension, and the LeakyReLU activation function is defined as: f(z)=max(αz,z), where α is a LeakyReLU hyperparameter.

[0052] Preferably, the frequency band combination enhancement and multi-view convolutional neural network F FBC_MV_CNN (·) The softmax loss function is used for training, and the F FBC_MV_CNN (·)The expression is as follows:

[0053] F FBC_MV_CNN (·)=F 1D_CNN (F MV (FBC(·))) (14).

[0054] Preferably, in the network training described in step 4, the learning rate during network training is adjusted to a cosine annealing function with restart, and network training is performed, wherein the update formula of the learning rate under the cosine annealing function with restart is:

[0055]

[0056] Among them, new lr The symbol represents the new learning rate, Init lr represents the initial learning rate, eta min represents the minimum learning rate, T max Indicates the set iteration half cycle, and epoch indicates the number of cycles in the current iteration.

[0057] The present invention has the following characteristics and beneficial effects:

[0058] The frequency band combination enhancement and multi-view convolutional neural network constructed by the present invention directly uses time domain EEG signals as input and performs well.

[0059] The frequency band combination enhancement module FBC(·) (FNC module) constructed by the present invention enables the signal to retain more frequency domain information while maintaining the time domain form.

[0060] The multi-view convolution module F constructed by the present invention MV (·)(MV module) can obtain EEG frequency band and inter-frequency band information from multiple perspectives, and simultaneously obtain EEG spatiotemporal information, thereby reducing the impact of individual variations between samples on the network to a certain extent.

[0061] In summary, the present invention focuses on the important role that motor imagery EEG (MI-EEG) plays in brain-computer interface (BCI) research. However, effectively decoding these signals remains a problem to be solved. When EEG data is input in the original time series, the traditional 1D-CNN cannot obtain frequency domain and channel correlation information. The frequency band combination enhancement module FBC (·) superimposes the original EEG and its four rhythm signals (δ, θ, α and β) to form an FBC signal, which is used as the input of the next module. The multi-view convolution module F MV (·) Two types of convolutional modules with different field-of-view sizes are used to extract both intra-band and inter-band information from FBC signals. We validated the MV and FBC modules on two datasets and compared the performance of the FBC-MV-CNN with other state-of-the-art algorithms. The comparison shows that the proposed solution achieves higher prediction accuracy. Furthermore, to prevent overfitting, a cosine annealing algorithm with a restart strategy is used to update the learning rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 It is an implementation flow chart of the present invention.

[0064] Figure 2 This is a diagram of the experimental paradigm.

[0065] Figure 3 is the multi-view convolution module F MV Schematic diagram of (·).

[0066] Figure 4 Schematic diagram of the model framework of frequency band combination enhancement and multi-view convolutional neural network.

[0067] Figure 5 The frequency band combination enhancement module FBC(·) and the multi-view convolution module F MV (·) Performance diagram of the model under different combinations. DETAILED DESCRIPTION

[0068] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0069] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0070] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0071] This embodiment provides a decoding method based on frequency band combination enhancement and multi-view convolutional neural network, such as Figure 1 As shown, the following steps are included:

[0072] Step 1: Use Neuracle's Neusen W series wireless EEG instrument with 22 channels (position remains the same as the experimental paradigm in 2.1.1) and a sampling frequency of 1000 Hz to collect data. Refer to the experimental paradigm of BCI IV-2a, and replace the MI process with the key press ME stage. Figure 2 As shown, each trial starts after the voice prompt. The first is a gaze stage of 0 to 2 seconds. Then, in the second second, a movement prompt of "L" or "R" lasting 1.25 seconds appears, which respectively represent that the next stage is left hand movement or right hand movement. The subject presses the left or right button according to the movement prompt between the 3rd and 6th seconds. At this time, the instrument will record the moment when the button is pressed and stipulate that it is the 0th second of the analysis stage. For each trial, -0.8 to -0.3 seconds of the analysis stage is taken as the data sample to be analyzed. Each round of experiments includes 50 movements of the left and right hands, with a total of 3 subjects (numbered B01 to B05), and each subject collects 2 rounds of experiments. The experimental flow chart is shown as follows Figure 2 shown.

[0073] Step 2: Preprocess the data. First, perform 1-32Hz bandpass filtering through an FIR filter. Independent Component Analysis (ICA) based on the SOBI algorithm is used in the EEGLAB toolbox to remove eye movement and blink artifact components from the original signal.

[0074] Step 3: Construct frequency band combination enhancement and multi-view convolutional neural network F FBC_MV_CNN (·),like Figure 4 As shown, frequency band combination enhancement and multi-view convolutional neural network F FBC_MV_CNN (·) All parameters are randomly generated.

[0075] Specifically, the frequency band combination enhancement module FBC(·):

[0076] The size of the original data X is N*P, where N is the number of channels and P is the number of sampling points. Then, four bandpass filters of different frequency bands are performed on X respectively:

[0077]

[0078]

[0079]

[0080]

[0081] Then the original data and the four filtered data are superimposed and combined along the channel direction to form the enhanced data X FBC :

[0082]

[0083] Multi-view convolution module F MV (·),like Figure 3 As shown, it includes 2 small-field-of-view S modules and 4 large-field-of-view Sx modules;

[0084] The small field of view S module includes 1-2 convolution kernels, a batch normalization layer and an activation function layer;

[0085] The large field of view Sx module includes a convolution layer, a pooling layer, a batch normalization layer and an activation function layer;

[0086] The output feature maps of the six modules are connected along the depth direction and used as the input signal of 1D-CNN. Specifically, 1D-CNN:

[0087] The incoming signal Input is input in a two-dimensional matrix format, where the vertical direction is the channel alignment direction and the horizontal direction is the time progression direction.

[0088] Input∈R (C×T)

[0089] Where c represents the number of features and T represents the length of the time series;

[0090] Establish a spatial feature extraction part S(·), which includes 1-2 convolutional layers S i , the vertical size of the convolution kernel is the number of input channels. Each convolution kernel extracts information between EEG channels when moving and forms a feature map with a vertical dimension of 1, s i and S(·) are defined as follows:

[0091] S i (·)=Conv Ti [BN Ti (·)]∈R (C′×T)

[0092] S(·)=S i [S i-1 (...)]∈R (C′×T)

[0093] Where C′ represents the number of convolution kernels, and the size of the convolution kernel is (C, K Si ), K Si is the convolution kernel length;

[0094] Establish a time feature extraction part G(·), which contains three convolutional layers g i (·), the convolution kernel in each convolution layer is 1 in the longitudinal direction, that is, the horizontal convolution kernel. The horizontal convolution kernel abstracts the time domain information of the feature map and forms a high-dimensional feature. Each convolution layer contains a BN operation and a nonlinear mapping based on Leaky ReLU. The time feature extraction part G(·) takes the feature map output by the previous part as input and outputs a high-dimensional feature map with spatiotemporal information. The feature map is downsampled in the average pooling layer and then expanded into a feature vector using a 1*1 convolution operation. i The definitions of (·) and G(·) are as follows:

[0095] g i，1 (·)=Conv2{BN2[LeakyReLU 12 (·)]}∈R (C″×T)

[0096] g i，2 (·)=Conv3{BN3[LeakyReLU 23 (·)]}∈R (C″′×T)

[0097] gi (·)=g i，2 (g i，1 (·))

[0098] G(·)=g i (g i-1 (...))

[0099] Where C″ represents the number of convolution kernels in the second layer, the convolution kernel size is (1, K2), and K2 is the convolution kernel length; C″· represents the number of convolution kernels in the third layer, the convolution kernel size is (1, K3), and K3 is the convolution kernel length;

[0100] Build 1D-CNN model F 1D_CNN (·), using the vertical convolution kernel and the horizontal convolution kernel to extract the spatial and temporal features of the input signal Input in sequence, the 1D-CNN model F 1D_CNN The fully connected part of (·) includes an input layer, a hidden layer, and an output layer. The number of nodes in the hidden layer is set to 32.

[0101] F 1D_CNN (·)=FC 1，2，3 {AveragePooling[G(S(·))]}

[0102] Among them FC i Indicates the full connection of the i-th layer.

[0103] Step 4: Divide the data U into a training set U1 and an evaluation set U2. Set the learning rate update method to simulated annealing with restarts. Use U1 to train the constructed FBC-MV-CNN, and evaluate the model's accuracy acc and Kappa coefficient on U2. When the accuracy reaches the specified threshold, stop training and obtain the final model. The calculation formula for the Kappa coefficient is as follows:

[0104]

[0105] Where C is the number of classification categories.

[0106] In order to verify that FBC and MV can improve the decoding ability of 1D-CNN for motor imagery, two datasets (4-classification dataset A and 2-classification dataset B) were used to evaluate the network models with different module combinations (MV-CNN, FBC-CNN and FBC-MV-CNN). The evaluation results are as follows: Figure 5 shown.

[0107] Overall analysis shows that compared to 1D-CNN, MV-CNN's average recognition accuracy improved by 3.87% and 6.99%, respectively, indicating that the use of a multi-view spatial feature extraction structure can significantly improve the average recognition rate of the decoding network under the same input. For a single subject, S07, the use of the MV structure improved the average accuracy by 13.03%. In addition, after using frequency band combination enhancement, the overall correct classification rate of 1D-CNN and MV-CNN increased by 1.77% and 4.89%, respectively, and the overall standard deviation was reduced by 0.71% and 1.84%, respectively, relative to their respective predecessors.

[0108] like Figure 5 As shown in (a), in a sample of nine subjects, the decoding method using FBC and MV structures achieved higher recognition accuracy, which was better than the decoding network using only these two methods or only one method, indicating that FBC and MV help to extract the features of raw EEG data more comprehensively. Figure 5 (b) shows the standard deviation of the network. Except for the sample S02, the Std value of FBC-MV-CNN is lower than that of other models, indicating that the network using the combined FBC and MV methods is more robust and its accuracy is less likely to fluctuate.

[0109] In order to verify the impact of the attention module CBAM on the network, two datasets (4-classification dataset A and 2-classification dataset B) are used to evaluate the models without and with CBAM. The results are shown in Table 1.

[0110] Table 1 Evaluation comparison of FBC-MV-CNN before and after adding CBAM attention

[0111]

[0112] As can be seen, adding the CBAM module improves the network's performance on most samples, but remains the same or deteriorates on others. Overall average accuracy indicates improved performance. Therefore, it is recommended to retain the CBAM module in the network.

[0113] In order to verify the classification effect of FBC-MV-CNN on motor imagery, the trained model was evaluated using two datasets (4-classification dataset A and 2-classification dataset B). The evaluation results are shown in Tables 2 and 3, respectively.

[0114] Table 2 Performance of traditional models and FBC-MV-CNN on four-category datasets

[0115]

[0116] Table 3 Performance of traditional models and FBC-MV-CNN on binary classification datasets

[0117]

[0118]

[0119] Table 2 shows that the mean Kappa coefficient obtained by the FBC-MV-CNN method is 0.6, indicating that the proposed method has high consistency. However, the mean Kappa coefficients calculated by the 1D-CNN, CSP+SVM, and FBCSP+SVM methods are all less than 0.6, indicating that these methods have only average or moderate consistency. Table III also lists the Kappa values ​​of the proposed method and the top three methods in the competition. The Kappa values ​​of the top three algorithms are all less than 0.6, indicating only average or moderate consistency. The proposed FBC-MV-CNN decoding algorithm improves on the top three algorithms on the majority of the tested samples, demonstrating its effectiveness.

[0120] Table 3 shows that the FBC-MV-CNN decoding algorithm achieved higher prediction accuracy across all subjects compared to the traditional CSP+SVM and 1-D CNN algorithms. For example, in subject B01, the proposed method improved prediction accuracy by 20% compared to CSP+SVM. Furthermore, the standard deviation of this method across all tested samples was lower than that of SVM and CSP, demonstrating that the FBC-MV-CNN decoding algorithm is more robust. The Wilcoxon signed-rank test showed that the FBC+MV-CNN decoding algorithm had a significant effect on the kappa coefficient compared to the CSP+SVM and 1D-CNN methods (p < 0.02, p < 0.04). Overall, the proposed algorithm achieved higher average prediction accuracy than the other methods, demonstrating the feasibility of the proposed model.

[0121] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A decoding method based on frequency band combination enhancement and multi-view convolutional neural network, characterized in that: The steps include: S1. Collect human brain electrical signals through motor imagery; S2, preprocessing the EEG signal collected in step S1; S3, construct frequency band combination enhancement and multi-view convolutional neural network; S3-1, construct frequency band combination enhancement module FBC(·), construct multi-view convolution module F MV (·),1D-CNN construction; The method for constructing the frequency band combination enhancement module FBC(·) is as follows: Assuming that the size of the original data X is N*P, where N is the number of channels and p is the number of sampling points, four bandpass filters of different frequency bands are performed on X respectively: Then the original data and the four filtered data are superimposed and combined along the channel direction to form the enhanced data X FBC : The multi-view convolution module F MV (·) construction method: Six feature extraction groups of different sizes and step sizes are defined, wherein the feature extraction groups include two small-field-of-view S modules and four large-field-of-view Sx modules; The small field of view S module includes 1-2 convolution kernels, a batch normalization layer and an activation function layer; The large field of view Sx module includes a convolution layer, a pooling layer, a batch normalization layer and an activation function layer; The output feature maps of the six modules are connected along the depth direction as the input signal of 1D-CNN; S3-2, the constructed frequency band combination enhancement module FBC(·), multi-view convolution module F MV (·) and 1D-CNN are sequentially combined into a convolutional neural network F based on frequency band combination enhancement and multi-view FBC_MV_CNN (·); S4. Using the preprocessed EEG signal obtained in step S2, training the frequency band combination enhancement and multi-view convolutional neural network.

2. The decoding method based on frequency band combination enhancement and multi-view convolutional neural network according to claim 1, characterized in that: In step S1, the method for collecting EEG signals is: With the mastoid process behind the left ear as the reference and the forehead as the ground, EEG signals of 22 channels, including CZ, C2, C3, C4, C5, C6, FCZ, FC1, FC2, FC3, FC4, FC5, FC6, C-1, CP2, CP3, CP4, CP5, CP6, CP7, and CP8, were collected at a frequency of 1000 Hz; The subjects were asked to imagine the movements of their left hand, right hand, feet and tongue according to visual cues.

3. The decoding method based on frequency band combination enhancement and multi-view convolutional neural network according to claim 1, characterized in that: In step S2, the EEG signal preprocessing method is as follows: performing ICA independent component analysis, segmenting the processed EEG signal, and extracting EEG signals during four types of actions.

4. The decoding method based on frequency band combination enhancement and multi-view convolutional neural network according to claim 1, characterized in that: The construction method of the 1D-CNN is as follows: The incoming signal Input is input in a two-dimensional matrix format, where the vertical direction is the channel alignment direction and the horizontal direction is the time progression direction. Input∈R (C×T) (6) Where C represents the number of features and T represents the length of the time series; Establish a spatial feature extraction part S(·), which includes 1-2 convolutional layers S i , the vertical size of the convolution kernel is the number of input channels. Each convolution kernel extracts information between EEG channels when moving and forms a feature map with a vertical dimension of 1, S i and S(·) are defined as follows: S i (·)=Conv Ti [BN Ti (·)]∈R (C′×T) (7) S(·)=S i [S i-1 (...)]∈R (C′×T) (8) Where C′ represents the number of convolution kernels, and the size of the convolution kernel is (C, K Si ), K S i is the length of the convolution kernel; Establish a time feature extraction part G(·), which contains three convolutional layers g i (·), the convolution kernel in each convolution layer is 1 in the longitudinal direction, that is, the horizontal convolution kernel. The horizontal convolution kernel abstracts the time domain information of the feature map and forms a high-dimensional feature. Each convolution layer contains a BN operation and a nonlinear mapping based on Leaky ReLU. The time feature extraction part G(·) takes the feature map output by the previous part as input and outputs a high-dimensional feature map with spatiotemporal information. The feature map is downsampled in the average pooling layer and then expanded into a feature vector using a 1*1 convolution operation. i The definitions of (·) and G(·) are as follows: g i，1 (·)=Conv2{BN2[LeakyReLU 12 (·)]}∈R (C′’×T) (9) g i，2 (·)=Conv3{BN3[LeakyReLU 23 (·)]}∈R (C′’’×T) (10) g i (·)=g i，2 (g i，1 (·)) (11) G(·)=g i (g i-1 (...)) (12) Where C'' represents the number of convolution kernels in the second layer, the convolution kernel size is (1, K2), and K2 is the convolution kernel length; C''' represents the number of convolution kernels in the third layer, the convolution kernel size is (1, K3), and K3 is the convolution kernel length; Build 1D-CNN model F 1D_CNN (·), using the vertical convolution kernel and the horizontal convolution kernel to extract the spatial and temporal features of the input signal Input in sequence, the 1D-CNN model F 1D_CNN The fully connected part of (·) includes an input layer, a hidden layer, and an output layer. The number of nodes in the hidden layer is set to 32. F 1D_CNN (·)=FC 1，2，3 {AveragePooling[G(S(·))]} (13) Among them FC i Indicates the full connection of the i-th layer.

5. The decoding method based on frequency band combination enhancement and multi-view convolutional neural network according to claim 4, characterized in that: The convolutional layers all use a valid padding strategy, the BN operations all perform batch normalization on the convolutional layer outputs in the channel dimension, and the LeakyReLU activation function is defined as: f(z)=max(αz,z), where α is a LeakyReLU hyperparameter.

6. The decoding method based on frequency band combination enhancement and multi-view convolutional neural network according to claim 4, characterized in that: The frequency band combination enhancement and multi-view convolutional neural network F FBC_MV_CNN (·) The softmax loss function is used for training, and the F FBC_MV_CNN (·)The expression is as follows: F FBC_MV_CNN (·)=F 1D_CNN (F MW (FBC(·))) (14)。 7. The decoding method based on frequency band combination enhancement and multi-view convolutional neural network according to claim 6, characterized in that: In the network training described in step S4, the learning rate during network training is adjusted to a cosine annealing function with restart, and network training is performed, wherein the update formula of the learning rate under the cosine annealing function with restart is: Among them, new lr The symbol represents the new learning rate, Init lr represents the initial learning rate, eta min represents the minimum learning rate, T max Indicates the set iteration half cycle, and epoch indicates the number of cycles in the current iteration.

Images