End-to-end deep learning method and device based on EEG motor imagery classification and medium

Abstract

The invention discloses an end-to-end deep learning method and device based on EEG motor imagery classification and a medium. The method comprises the following steps: acquiring an electroencephalogram signal, and inputting the electroencephalogram signal into an MBCNet model; the MBCNet model comprises a first branch and a second branch, and the first branch and the second branch have the same structure and different convolution kernel sizes; respectively extracting space-time information of the electroencephalogram signal through the first branch and the second branch to obtain a first space-time feature and a second space-time feature; performing feature fusion on the first spatial-temporal feature and the second spatial-temporal feature to obtain fused feature representation; and performing feature classification on the fused feature representation to obtain a motor imagery result represented by the electroencephalogram signal. The end-to-end deep learning method and device based on EEG motor imagery classification and the medium disclosed by the invention have good performance in electroencephalogram signal motor imagery classification decoding; the provided MBCNet model has good robustness and generalization ability, and compared with other baseline models, the provided MBCNet model has better decoding ability.

Description

Technical Field

[0001] The present invention relates to the field of motor imagery technology, and in particular to an end-to-end deep learning method, device, and medium based on EEG motor imagery classification. Background Art

[0002] Brain-computer interface (BCI) technology is an innovative communication method that allows the human brain to directly interact with and control external devices through neural signals. Motor imagery (MI)-based BCI is a key application of this technology. It allows users to control external devices by imagining specific physical movements (such as hand or foot movements) without actually moving them. This is a revolutionary way of interaction for those who have lost mobility due to illness or disability.

[0003] In traditional machine learning for MI classification, commonly used methods such as Common Spatial Pattern (CSP) and its variants are used to capture and decode different signal features from EEG signals. The main purpose of CSP is to optimize spatial filters to maximize the differences between EEG signals. Various methods using wavelet transforms have also been explored for extracting time-frequency features from EEG signals. These time-frequency features are then classified using algorithms such as support vector machines (SVM) and linear discriminant analysis (LDA) to determine the specific body movements expressed by the time-frequency features.

[0004] However, traditional machine learning methods often require manual extraction of signal features. Due to the complex characteristics of EEG signals, it is difficult to effectively extract effective motor imagery classification information from these signal features. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an end-to-end deep learning method, device, and medium based on EEG motor imagery classification.

[0006] A first aspect of the present invention provides an end-to-end deep learning method based on EEG motor imagery classification, comprising the following steps:

[0007] Acquire an EEG signal and input the EEG signal into an MBCNet model; the MBCNet model includes a first branch and a second branch, the first branch and the second branch having the same structure and different convolution kernel sizes;

[0008] Extracting the spatiotemporal information of the EEG signal through the first branch and the second branch respectively to obtain a first spatiotemporal feature and a second spatiotemporal feature;

[0009] Performing feature fusion on the first spatiotemporal feature and the second spatiotemporal feature to obtain a fused feature representation;

[0010] Feature classification is performed on the fused feature representation to obtain a motor imagery result represented by the EEG signal.

[0011] Furthermore, the first branch and the second branch each include a spatiotemporal convolution module, an attention extraction module and a Li-TCN module; wherein the spatiotemporal convolution module is used to extract the primary spatiotemporal features of the EEG signal; the attention extraction module is used to determine the channel weights of the primary spatiotemporal features; the Li-TCN module is used to extract high-level spatiotemporal features from the primary spatiotemporal features in combination with the channel weights, wherein the high-level spatiotemporal features of the first branch are output as the first spatiotemporal features, and the high-level spatiotemporal features of the second branch are output as the second spatiotemporal features.

[0012] Furthermore, the spatiotemporal convolution module specifically includes a temporal convolution layer, a temporal attention layer and a dual-branch network layer;

[0013] The temporal convolution layer is used to extract temporal features of EEG signals; the temporal convolution layer of the first branch has a smaller convolution kernel, and the temporal convolution layer of the second branch has a larger convolution kernel;

[0014] The temporal attention layer is used to capture the inter-channel interaction of the EEG signal;

[0015] The dual-branch network layer includes a first branch and a second branch in parallel; the first branch and the second branch have the same structure and different convolution depths; the first branch and the second branch each include a depth convolution layer and a spatiotemporal convolution layer; the depth convolution layer is used to extract spatial features of the EEG signal based on the inter-channel interaction of the EEG signal; the spatiotemporal convolution layer is used to integrate the temporal features and spatial features of the EEG signal to obtain a primary spatiotemporal sequence; the depth convolution layer of the first branch has a smaller convolution depth, and the depth convolution layer of the second branch has a larger convolution depth;

[0016] The primary spatiotemporal features output by the spatiotemporal convolution module are obtained by adding the primary spatiotemporal sequence output by the first branch and the primary spatiotemporal sequence output by the second branch.

[0017] Furthermore, the attention extraction module includes a global average pooling layer, a first fully connected layer, and a second fully connected layer; the attention execution module specifically performs the following steps to determine the channel weights of the primary spatiotemporal features:

[0018] The primary spatiotemporal features are compressed into scalars through a global average pooling layer to generate a channel vector that describes the global information of the channel;

[0019] Performing dimensionality reduction on the channel vector through a first fully connected layer to capture the nonlinear dependency of the channel vector;

[0020] Restoring the dimension of the channel vector through a second fully connected layer to generate a weight vector;

[0021] The weight vector is multiplied element-wise by the primary spatiotemporal feature to determine the channel weight of the primary spatiotemporal feature.

[0022] Furthermore, the Li-TCN module includes a LiteMLA module and a TCN module;

[0023] The LiteMLA module is used to extract multi-scale features of the primary spatiotemporal features;

[0024] The TCN module is used to perform dilated causal convolution and residual connection based on the extracted multi-scale features to obtain high-level spatiotemporal feature output.

[0025] Furthermore, the LiteMLA module includes a first projection layer, a first convolution kernel, a second convolution kernel, a ReLU linear attention layer, and a second projection layer; the first convolution kernel and the second convolution kernel have different sizes; the LiteMLA module extracts multi-scale features of the primary spatiotemporal features through the following steps;

[0026] Using a first projection layer, the primary spatiotemporal features are divided into a first projection, a second projection, and a third projection;

[0027] For the first projection, the ReLU linear attention layer is directly used to extract the first scale features;

[0028] For the second projection, after convolution aggregation is performed on the second projection using the first convolution kernel, the second scale feature is extracted using the ReLU linear attention layer;

[0029] For the third projection, after convolution aggregation is performed on the third projection using the second convolution kernel, the third scale feature is extracted using the ReLU linear attention layer;

[0030] The first scale feature, the second scale feature, and the third scale feature are aggregated using the second projection layer to obtain multi-scale features of the primary spatiotemporal feature.

[0031] Furthermore, the TCN module includes multiple residual blocks, each of which consists of two dilated causal convolutional layers; each of the causal convolutional layers is followed by a batch normalization and ELU activation layer; the TCN module obtains high-level spatiotemporal feature outputs through the following steps:

[0032] In each residual block, dilated causal convolution is performed on the multi-scale features using the causal convolution layer, and the feature dimensions of the multi-scale features are maintained using the batch normalization and ELU activation layers;

[0033] At the end of each residual block, the input and output of the residual block are added element by element using a residual connection; the output of the last residual block passes through the ELU activation layer to obtain high-level spatiotemporal features.

[0034] Furthermore, the MBCNet model also includes a fully connected layer; the feature classification of the fused feature representation is specifically performed through the fully connected layer.

[0035] A second aspect of the present invention discloses an electronic device, comprising a processor and a memory;

[0036] The memory is used to store programs;

[0037] The processor executes the program to implement the end-to-end deep learning method based on EEG motor imagery classification.

[0038] The third aspect of the present invention discloses a computer-readable storage medium, which stores a program. The program is executed by a processor to implement the end-to-end deep learning method based on EEG motor imagery classification.

[0039] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0040] The embodiments of the present invention have the following beneficial effects: the end-to-end deep learning method, device and medium based on EEG motor imagery classification disclosed in the present invention have good performance in EEG signal motor imagery classification decoding. The MBCNet model designed by the present invention adopts a two-branch parallel structure to extract primary spatiotemporal features at different scales, uses the channel attention module to selectively focus on valuable features, extracts high-level spatiotemporal features through the Li-TCN module, and then fuses the multi-branch features, and finally sends them to the classification module to obtain the classification results. The MBCNet model provided by the embodiment of the present invention has good robustness and generalization ability, and has better decoding ability compared with other baseline models.

[0041] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 This is a basic implementation flow chart of an end-to-end deep learning method based on EEG motor imagery classification in the present invention.

[0044] Figure 2 This is a schematic diagram of the MBCNet model structure used in the present invention.

[0045] Figure 3 This is a structural diagram of the attention extraction module in the MBCNet model used in the present invention.

[0046] Figure 4 It is a schematic diagram of the LiteMLA module structure in the MBCNet model used in the present invention.

[0047] Figure 5 This is a schematic diagram of the TCN module structure in the MBCNet model used in the present invention.

[0048] Figure 6 It is a structural schematic diagram of an electronic device provided by the present invention.

[0049] Figure 7 It is a schematic diagram of the structure of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] Attention-based methods are becoming increasingly popular in image processing due to their inherent ability to capture long-term dependencies. Many studies have also attempted to integrate attention modules into deep learning frameworks to improve classification accuracy.

[0052] Based on the above inspiration, the first embodiment of the present invention provides an end-to-end deep learning method based on EEG motor imagery classification, such as Figure 1 As shown, the following steps are included:

[0053] S1. Obtain EEG signals and input them into the MBCNet model;

[0054] S2. extracting the spatiotemporal information of the EEG signal through the first branch and the second branch, respectively, to obtain a first spatiotemporal feature and a second spatiotemporal feature;

[0055] S3. Fusing the first spatiotemporal feature and the second spatiotemporal feature to obtain a fused feature representation;

[0056] S4. Perform feature classification on the fused feature representation to obtain the motor imagery result represented by the EEG signal.

[0057] The end-to-end deep learning method based on EEG motor imagery classification provided by the embodiment of the present invention is implemented by the designed end-to-end deep learning model MBCNet. Figure 2 As shown, the network structure of the MBCNet model adopts a two-branch structure, including a first branch and a second branch, wherein the first branch and the second branch have the same structure and different convolution kernel sizes. In each branch, filters with convolution kernels of different sizes are used to extract primary spatiotemporal features, which are then added in the time dimension, and then the channel attention mechanism is used to identify which of the different channels (different electrode positions) are most important for the current task. The extracted primary spatiotemporal features are passed to the Li-TCN module to extract high-level spatiotemporal features. Through the feature fusion module, the high-level spatiotemporal features from the two branches are combined to obtain a feature representation with more information and characterization capabilities, and finally passed to the classifier for classification. The MBCNet model designed in this embodiment of the present invention has strong generalization ability and robustness, and has better decoding capabilities compared to other baseline models.

[0058] As a preferred embodiment, the spatiotemporal convolution module specifically includes a temporal convolution layer, a temporal attention layer and a dual-branch network layer.

[0059] Among them, the temporal convolution layer is used to extract the temporal features of the EEG signal; the temporal convolution layer of the first branch has a smaller convolution kernel, and the temporal convolution layer of the second branch has a larger convolution kernel.

[0060] The temporal attention layer is used to capture the inter-channel interactions of EEG signals.

[0061] The dual-branch network layer includes a parallel first branch and a second branch; the first branch and the second branch have the same structure and different convolution depths; the first branch and the second branch each include a depth convolution layer and a spatiotemporal convolution layer; the depth convolution layer is used to extract spatial features of EEG signals based on the inter-channel interaction of EEG signals; the spatiotemporal convolution layer is used to integrate the temporal features and spatial features of EEG signals to obtain a primary spatiotemporal sequence; the depth convolution layer of the first branch has a smaller convolution depth, and the depth convolution layer of the second branch has a larger convolution depth.

[0062] The primary spatiotemporal features output by the spatiotemporal convolution module are obtained by adding the primary spatiotemporal sequence output by the first branch and the primary spatiotemporal sequence output by the second branch.

[0063] For example, in the temporal convolution layer, an embodiment of the present invention uses a convolution kernel size of 31 with a filter number of 9 in the first branch to extract temporal features; in the second branch, a convolution kernel size of 63 with a filter number of 9 is used to extract temporal features. After the temporal convolution layer, a temporal attention layer is used to capture the inter-channel interactions of the EEG signal for use in the dual-branch network layer. In the dual-branch network layer, the two convolution branches have the same structure, and the initial layer is a depth convolution layer, where the depth value of the first branch depth convolution layer is 2 and the depth value of the second branch depth convolution layer is 4; spatial features are extracted from the feature map of the EEG signal through the depth convolution layer. After the depth convolution layer, both branches use an average pooling layer of size (1, 8) to reduce the sampling rate and reduce the feature dimension. This reduces the temporal data and sampling frequency to 1 / 8 of the input, that is, the sampling rate of the signal is about 32Hz. The second convolutional layer of both branches consists of a spatiotemporal convolutional layer with a filter size of F2 = F1 * D = 9 * 2 = 18 and a kernel size of (1, 16). The spatiotemporal convolutional layer then learns to optimally integrate spatiotemporal features, resulting in a high-level spatiotemporal feature representation. Following the spatiotemporal convolutional layer, an average pooling layer of size (1, 7) is used to reduce the sampling rate to 5 Hz. Finally, the spatiotemporal sequences from the two branches are fused by addition.

[0064] As a preferred embodiment, the attention extraction module includes a global average pooling layer, a first fully connected layer, and a second fully connected layer; the attention execution module specifically performs the following steps to determine the channel weights of the primary spatiotemporal features:

[0065] The primary spatiotemporal features are compressed into scalars through a global average pooling layer to generate a channel vector that describes the global information of the channel;

[0066] The channel vector is reduced in dimension by the first fully connected layer to capture the nonlinear dependency of the channel vector;

[0067] The dimension of the channel vector is restored through the second fully connected layer to generate a weight vector;

[0068] The weight vector is multiplied element-wise with the primary spatiotemporal features to determine the channel weights of the primary spatiotemporal features.

[0069] like Figure 3 As shown, the attention extraction module in this embodiment of the present invention mainly performs two steps:

[0070] Squeezing operation Fsq: First, the feature map of the primary spatiotemporal features is reduced to a scalar through global average pooling.

[0071] Excitation operation Fex: It includes two fully connected layers, where the first fully connected layer reduces the input vector dimension, and the second fully connected layer expands the reduced vector dimension, and finally generates a weight vector.

[0072] Finally, the weight vector is element-wise multiplied with the initial feature map, so that the network can weight features from different feature maps. This module suppresses unimportant channels while emphasizing the importance of useful channels, thereby enhancing the performance and generalization ability of the model.

[0073] As a preferred embodiment, the Li-TCN module includes a LiteMLA module and a TCN module;

[0074] LiteMLA module is used to extract multi-scale features of primary spatiotemporal features;

[0075] The TCN module is used to perform dilated causal convolution and residual connection based on the extracted multi-scale features to obtain high-level spatiotemporal feature output.

[0076] like Figure 4 As shown in Figure 1, the LiteMLA module includes a first projection layer, a first convolution kernel, a second convolution kernel, a ReLU linear attention layer, and a second projection layer; the first convolution kernel and the second convolution kernel have different sizes; the LiteMLA module extracts multi-scale features of primary spatiotemporal features through the following steps;

[0077] Use the first projection layer to separate the primary spatiotemporal features into the first projection, the second projection, and the third projection;

[0078] For the first projection, the ReLU linear attention layer is directly used to extract the first-scale features;

[0079] For the second projection, after convolution aggregation of the second projection using the first convolution kernel, the second scale features are extracted using the ReLU linear attention layer;

[0080] For the third projection, the second convolution kernel is used to perform convolution aggregation on the third projection, and then the ReLU linear attention layer is used to extract the third scale features;

[0081] The second projection layer is used to aggregate the first-scale features, the second-scale features, and the third-scale features to obtain multi-scale features of the primary spatiotemporal features.

[0082] The LiteMLA module used in the embodiments of the present invention can achieve global receptive field and multi-scale learning, thereby extracting multi-scale features for classification. After obtaining the Q / K / V projections through the first projection layer, multi-scale features are generated by aggregating nearby features through lightweight small-kernel convolution. The projection of the Q channel is not convolved, the projection of the K channel is convolved with a convolution kernel of size (3, 3), and the projection of the V channel is convolved with a convolution kernel of size (5, 5).

[0083] A ReLU linear attention layer is applied to multi-scale features, and the outputs are concatenated and fed into a second projection layer for feature fusion. This embodiment of the present invention uses a ReLU linear attention layer to achieve a global receptive field, unlike the computationally complex softmax attention. ReLU linear attention is implemented by computing a linear combination of ReLU activation functions, which has linear computational complexity and is therefore more efficient on hardware.

[0084] like Figure 5 As shown in the figure, the TCN module consists of multiple residual blocks, each of which consists of two dilated causal convolutional layers; each causal convolutional layer is followed by a batch normalization and ELU activation layer; the TCN module obtains high-level spatiotemporal feature outputs through the following steps:

[0085] In each residual block, a causal convolution layer is used to perform dilated causal convolution on multi-scale features. Batch normalization and ELU activation layers are used to maintain the feature dimensionality of multi-scale features. In this embodiment of the present invention, causal convolution is used to prevent any information from propagating from the future to the past. That is, the output at time t depends only on the input at time t and before. Dilated convolution increases network depth while allowing the receptive field to expand exponentially. Therefore, dilated causal convolution can learn relationships in long sequences.

[0086] At the end of each residual block, a residual connection is used to perform element-by-element addition of the input and output of the residual block. The output of the last residual block is activated by the ELU layer to obtain high-level spatiotemporal features. Residual connections perform element-by-element addition of the input and output feature maps F(x)+x. Because they can learn identity functions, they are effective in deep networks. In residual blocks, we use the identity mapping because the input and output dimensions are the same. Otherwise, a linear transformation, namely a 1×1 convolution, is used.

[0087] The receptive field size (RFS) of the TCN in the embodiment of the present invention increases exponentially with the number of stacked residual blocks L. This is because the expansion D corresponding to each additional residual block increases exponentially. RFS is controlled by two parameters: the number of residual blocks L and the kernel size K T , defined as follows:

[0088] RFS=1+2(K T -1)(2L -1);

[0089] After obtaining the high-level spatiotemporal feature output, the embodiment of the present invention fuses the first and second spatiotemporal features through a feature fusion module. Finally, the fused features are sent to the classifier block for prediction. This embodiment of the present invention uses the SoftMax activation function as the classification module, selecting the label with the highest probability as the final decoding result. To prevent overfitting, this embodiment of the present invention imposes a maximum norm constraint of 0.25 on the weights.

[0090] When evaluating the performance of the model proposed in the embodiment of the present invention, accuracy and kappa value (k) are selected as the main indicators. The calculation formulas of accuracy and kappa value are shown in (3) and (4).

[0091]

[0092] Among them, n left 、n right 、n feet and n tongue Represents the number of correct predictions for each MI task sample, N left 、N right 、N feet and N tongue Represents the number of samples for each MI task.

[0093]

[0094] Among them, p0 is the consistency of the observations, that is, the accuracy of the model, and p e Indicates the consistency of the expected value, which is the probability or accuracy of random guessing.

[0095] Compared with other models, the results are shown in the following table. The model proposed in the present invention achieved an overall accuracy of 79.21% and a score of 0.71, which is much higher than the average classification results of several other models. Compared with EEGNet, ATCNet, DeepConvNet, and ShallowConvNet, the accuracy rates were improved by 8.96%, 4.95%, 8.25%, and 9.12%, respectively. The standard deviation of the accuracy between subjects was 9.75, which to some extent shows that the model proposed in the present invention has good performance in the four-category decoding of EEG signal motor imagery. For the Kappa score, people pay more attention to its value rather than its standard deviation. Because the Kappa coefficient already provides useful information about consistency, the standard deviation of the Kappa score is not given in the present invention.

[0096]

[0097] Figure 6 : is a structural diagram of the electronic device proposed in the second embodiment of the present invention. The memory of this embodiment stores program instructions for implementing the end-to-end deep learning method based on EEG motor imagery classification of any of the above embodiments. The processor is used to execute the program instructions stored in the memory to perform end-to-end deep learning based on EEG motor imagery classification. Among them, the processor can also be called a CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0098] The contents of the method in the first embodiment of the present invention are all applicable to this electronic device embodiment. The functions specifically implemented by this electronic device embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method.

[0099] Figure 7 Schematic diagram of the structure of the computer-readable storage medium of the third embodiment of the present invention. The computer-readable storage medium of the fourth embodiment of the present invention stores program instructions that can implement the above-mentioned end-to-end deep learning method based on EEG motor imagery classification, wherein the program instructions can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0100] The contents of the method in the first embodiment of the present invention are all applicable to this computer-readable storage medium embodiment. The functions specifically implemented by this computer-readable storage medium embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method.

[0101] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement the end-to-end deep learning method based on EEG motor imagery classification provided by the above embodiment.

[0102] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0103] Those skilled in the art will appreciate that the modules in the devices in the embodiments of the present invention can be adaptively changed and set in one or more devices different from the embodiments. The modules or units or components in the embodiments of the present invention can be combined into one module or unit or component, and in addition they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including corresponding claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including corresponding claims, abstracts and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0104] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0105] It should be noted that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0106] In addition, each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. In particular, for embodiments such as devices and equipment, since they are basically similar to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The embodiments of the devices and equipment described above are merely schematic, wherein the modules, units, etc. described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed to multiple places, such as nodes in a system network. Specifically, some or all of the modules and units may be selected according to actual needs to achieve the purpose of the above-mentioned embodiment scheme. Those skilled in the art can understand and implement it without paying any creative work.

[0107] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0108] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0109] In addition, the terms "first" and "second" used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in this embodiment. Therefore, the features defined by the terms "first" and "second" in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of such features. In the description of the present invention, the word "plurality" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0110] In the embodiments of the present invention, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or apparatus comprising the element. In addition, components, features, and elements with the same name in different embodiments of the present invention may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0111] Although embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention, and those of ordinary skill in the art may change, modify, replace, and modify the above embodiments within the scope of the present invention. Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the present invention. This application is intended to cover any variations, uses, or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the present invention. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the claims below.

Claims

1. An end-to-end deep learning method based on EEG motor imagery classification, characterized by: The following steps are involved: Acquire an EEG signal and input the EEG signal into an MBCNet model; the MBCNet model includes a first branch and a second branch, the first branch and the second branch having the same structure and different convolution kernel sizes; Extracting the spatiotemporal information of the EEG signal through the first branch and the second branch respectively to obtain a first spatiotemporal feature and a second spatiotemporal feature; Performing feature fusion on the first spatiotemporal feature and the second spatiotemporal feature to obtain a fused feature representation; Feature classification is performed on the fused feature representation to obtain a motor imagery result represented by the EEG signal.

2. The end-to-end deep learning method based on EEG motor imagery classification according to claim 1, characterized in that The first branch and the second branch each include a spatiotemporal convolution module, an attention extraction module and a Li-TCN module; wherein the spatiotemporal convolution module is used to extract the primary spatiotemporal features of the EEG signal; the attention extraction module is used to determine the channel weights of the primary spatiotemporal features; the Li-TCN module is used to extract high-level spatiotemporal features from the primary spatiotemporal features in combination with the channel weights, wherein the high-level spatiotemporal features of the first branch are output as the first spatiotemporal features, and the high-level spatiotemporal features of the second branch are output as the second spatiotemporal features.

3. The end-to-end deep learning method based on EEG motor imagery classification according to claim 2, characterized in that The spatiotemporal convolution module specifically includes a temporal convolution layer, a temporal attention layer, and a dual-branch network layer; The temporal convolution layer is used to extract temporal features of EEG signals; the temporal convolution layer of the first branch has a smaller convolution kernel, and the temporal convolution layer of the second branch has a larger convolution kernel; The temporal attention layer is used to capture the inter-channel interaction of the EEG signal; The dual-branch network layer includes a first branch and a second branch in parallel; the first branch and the second branch have the same structure and different convolution depths; the first branch and the second branch each include a depth convolution layer and a spatiotemporal convolution layer; the depth convolution layer is used to extract spatial features of the EEG signal based on the inter-channel interaction of the EEG signal; the spatiotemporal convolution layer is used to integrate the temporal features and spatial features of the EEG signal to obtain a primary spatiotemporal sequence; the depth convolution layer of the first branch has a smaller convolution depth, and the depth convolution layer of the second branch has a larger convolution depth; The primary spatiotemporal features output by the spatiotemporal convolution module are obtained by adding the primary spatiotemporal sequence output by the first branch and the primary spatiotemporal sequence output by the second branch.

4. The end-to-end deep learning method based on EEG motor imagery classification according to claim 2, characterized in that The attention extraction module includes a global average pooling layer, a first fully connected layer, and a second fully connected layer; the attention execution module specifically performs the following steps to determine the channel weights of the primary spatiotemporal features: The primary spatiotemporal features are compressed into scalars through a global average pooling layer to generate a channel vector that describes the global information of the channel; Performing dimensionality reduction on the channel vector through a first fully connected layer to capture the nonlinear dependency of the channel vector; Restoring the dimension of the channel vector through a second fully connected layer to generate a weight vector; The weight vector is multiplied element-wise by the primary spatiotemporal feature to determine the channel weight of the primary spatiotemporal feature.

5. The end-to-end deep learning method based on EEG motor imagery classification according to claim 2, characterized in that The Li-TCN module includes a LiteMLA module and a TCN module; The LiteMLA module is used to extract multi-scale features of the primary spatiotemporal features; The TCN module is used to perform dilated causal convolution and residual connection based on the extracted multi-scale features to obtain high-level spatiotemporal feature output.

6. The end-to-end deep learning method based on EEG motor imagery classification according to claim 5, characterized in that: The LiteMLA module includes a first projection layer, a first convolution kernel, a second convolution kernel, a ReLU linear attention layer, and a second projection layer; the first convolution kernel and the second convolution kernel have different sizes; the LiteMLA module extracts multi-scale features of the primary spatiotemporal features through the following steps; Using a first projection layer, the primary spatiotemporal features are divided into a first projection, a second projection, and a third projection; For the first projection, the ReLU linear attention layer is directly used to extract the first scale features; For the second projection, after convolution aggregation is performed on the second projection using the first convolution kernel, the second scale feature is extracted using the ReLU linear attention layer; For the third projection, after convolution aggregation is performed on the third projection using the second convolution kernel, the third scale feature is extracted using the ReLU linear attention layer; The first scale feature, the second scale feature, and the third scale feature are aggregated using the second projection layer to obtain multi-scale features of the primary spatiotemporal feature.

7. The end-to-end deep learning method based on EEG motor imagery classification according to claim 5, characterized in that The TCN module includes multiple residual blocks, each of which consists of two dilated causal convolutional layers; each causal convolutional layer is followed by a batch normalization and ELU activation layer; the TCN module obtains high-level spatiotemporal feature outputs through the following steps: exist In each residual block, the multi-scale features are dilated by using the causal convolution layer, and the feature dimensions of the multi-scale features are maintained by using the batch normalization and ELU activation layer; At the end of each residual block, the input and output of the residual block are added element by element using a residual connection; the output of the last residual block passes through the ELU activation layer to obtain high-level spatiotemporal features.

8. The end-to-end deep learning method based on EEG motor imagery classification according to claim 1, characterized in that: The MBCNet model also includes a fully connected layer; the feature classification of the fused feature representation is specifically performed through the fully connected layer.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement an end-to-end deep learning method based on EEG motor imagery classification as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement an end-to-end deep learning method based on EEG motor imagery classification according to any one of claims 1 to 8.

Images