Motor imagery electroencephalogram signal decoding method, device, equipment, medium and product
By introducing coordinate attention mechanism and band adaptive networks in the decoding of motor imagination EEG signals, the problem of difficulty in capturing the spatiotemporal characteristics of EEG signals is solved, and the decoding accuracy and classification accuracy are significantly improved.
Patent Information
- Application Number
- CN202510312807.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-17
- Publication Date
- 2025-06-10
AI Technical Summary
The prior art is difficult to fully capture the spatiotemporal characteristics of EEG signals, resulting in limited decoding accuracy of EEG signals for motor imagination.
The coordinate attention mechanism module is used to fuse the spatiotemporal information of EEG signals in the channel dimension and time dimension, and combine the band adaptive network to select the convolution kernel size to extract spatiotemporal features for classification.
By accurately extracting the critical space-time information of EEG signals, the decoding performance is improved, the ability to capture features of different frequency bands is enhanced, and the accuracy of EEG signals classification is significantly improved.
Smart Images
Figure CN120123745A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of electroencephalogram signal processing, and particularly relates to a method, device, equipment, medium and product for decoding motor imagery electroencephalogram signals. Background Art
[0002] MI-EEG (Motor Imagery Electroencephalography) technology is a technology for monitoring brain activities through electroencephalogram, and is used to record and analyze the changes in brain waves when an individual performs motor imagery. There are mainly two methods for MI-EEG decoding: traditional machine learning and deep learning. Traditional machine learning algorithms include feature learning and classification. In the feature learning stage, taking the common spatial pattern (CSP for short) which is common as an example, it has been proved to be one of the most popular and efficient electroencephalogram feature extraction algorithms, especially in the BCI competition dataset (a standardized dataset provided by a competition organized by developers in the field of Brain-Computer Interface (BCI)). F. Lotte introduced regularization into CSP to extract more spatial filters related to neurophysiology information. The filter bank spatial pattern has achieved high decoding accuracy in the scenario of limited and highly noisy electroencephalogram signal training data.
[0003] However, CSP is limited to binary classification (i.e., two types of imagination tasks). In the classification stage, many researchers use traditional machine learning algorithms such as linear discriminant analysis and support vector machine to classify EEG signals. However, these classification algorithms still rely on features manually designed according to human experience, which may limit the improvement of classification accuracy and consume a lot of time. On the contrary, deep learning algorithms can automatically learn features from data and reduce the dependence on artificial feature design. This method can greatly improve the classification accuracy and shorten the decoding time, and has also achieved great success in the field of motor imagery brain-computer interface.
[0004] At present, the convolutional neural network (CNN) has become one of the most widely used deep learning models for electroencephalogram (EEG) signal classification, and significant research progress has been made in multi-classification tasks related to motor imagery. In the current context, single-branch CNN provides a relatively simple method that focuses on features at a single scale and has achieved good results in multiple BCI paradigms. However, all these methods use a single branch, which has limited ability to effectively capture the rich temporal and spectral information in MI-EEG. Therefore, many methods use multi-branch CNN to decode MI-EEG, but there are still some problems. First, the convolutional scales of multi-branch CNN cannot adapt to different frequency bands. Second, multi-branch CNN is difficult to fully capture spatial information, and the introduction of the attention mechanism can alleviate this problem. However, most existing attention mechanisms generate spatial attention maps through two-dimensional global pooling, which makes it difficult to extract the correlation information between channels and the time dimension. The above problems will further lead to the phenomenon of MI-EEG decoding errors. Summary of the Invention
[0005] In view of this, the present invention provides a method, device, equipment, medium and product for decoding motor imagery EEG signals to solve the problem in the prior art that it is difficult to comprehensively capture the spatio-temporal characteristics of EEG signals, resulting in limited decoding.
[0006] In a first aspect, the present invention provides a method for decoding motor imagery EEG signals, the method comprising:
[0007] Obtain the original EEG signal;
[0008] Preprocess the original EEG signal to obtain the initial EEG signal after preprocessing;
[0009] Decompose the initial EEG signal into a plurality of to-be-processed signals in different frequency bands;
[0010] Input the to-be-processed signals into a pre-established coordinate attention mechanism module to fuse the spatio-temporal information of each to-be-processed signal in the channel dimension and the time dimension;
[0011] Extract the spatio-temporal features of the to-be-processed signals based on the spatio-temporal information corresponding to each to-be-processed signal and a pre-established frequency band adaptability network, where the spatio-temporal features are used for EEG signal classification; wherein, the frequency band adaptability network adaptively selects the convolutional kernel size according to the frequency band characteristics of the to-be-processed signals.
[0012] In the present invention, by fusing spatio-temporal information through the coordinate attention mechanism, key information in EEG signals can be extracted more accurately, improving the decoding performance. Additionally, the convolutional kernel size is adaptively selected through the frequency band adaptation network, effectively enhancing the ability to capture features of different frequency bands and improving the classification accuracy. The present invention fully considers the frequency band differences and spatio-temporal characteristics of EEG signals, optimizes the signal processing of EEG signals, and effectively improves the accuracy of EEG signal classification.
[0013] In an alternative embodiment, the coordinate attention mechanism module includes:
[0014] An input layer;
[0015] Two juxtaposed average pooling layers, the input ends of the average pooling layers are all connected to the input layer;
[0016] A first convolutional module, the input end of the first convolutional module is connected to the output end of the average pooling layer;
[0017] Input the signal to be processed into the coordinate attention mechanism module, and fuse the spatio-temporal information of each signal to be processed in the channel dimension and time dimension, including:
[0018] Input the signal to be processed into the average pooling layer through the input layer, and output the time information feature in the time dimension and the spatial information feature in the channel dimension respectively; one of the average pooling layers is used to extract features of the signal to be processed along the time direction, and the other average pooling layer is used to extract features of the signal to be processed along the channel direction;
[0019] Based on the first convolutional module, fuse the time information feature and the spatial information feature to obtain the fused spatio-temporal information.
[0020] In this embodiment, the features in the time and space dimensions are respectively extracted through the juxtaposed average pooling layers, realizing the efficient fusion of spatio-temporal information. At the same time, it also improves the model's ability to capture the spatio-temporal features of EEG signals and enhances the decoding accuracy.
[0021] In an alternative embodiment, the coordinate attention mechanism module further includes:
[0022] A first normalization layer, the input end of the first normalization layer is connected to the output end of the first convolutional module;
[0023] A first activation function layer, the input end of the first activation function layer is connected to the output end of the first normalization layer;
[0024] Two juxtaposed second convolutional modules, the input ends of the second convolutional modules are connected to the output end of the first activation function layer;
[0025] The output end of each second convolutional module is also provided with a second activation function layer, and the output end of the second activation function layer is used to output corresponding attention weights.
[0026] In an alternative embodiment, after obtaining the fused spatio-temporal information, it further includes:
[0027] Based on the first normalization layer and the first activation function layer, the spatio-temporal information is split into two independent tensors;
[0028] Based on the second convolutional module and the second activation function layer, the two independent tensors are transformed into tensors of the same size as when inputting into the coordinate attention mechanism module, and attention weights of the channel dimension and the time dimension are obtained.
[0029] In this embodiment, through the processing of the normalization layer and the activation function layer, the spatio-temporal information can be fully split and fused during feature extraction, improving the model's ability to understand features of different dimensions, and being able to generate accurate attention weights at the output end, enabling the model to adaptively adjust the attention to each channel and time dimension, and enhancing the classification accuracy.
[0030] In an alternative embodiment, the frequency band adaptation network includes:
[0031] Multiple spatio-temporal convolutional modules with different convolutional scales arranged in sequence, where each spatio-temporal convolutional module includes at least two convolutional layers, and the size of the convolutional kernel of each convolutional layer decreases as the frequency band bandwidth increases;
[0032] A depth convolutional module is arranged at the output end of the last spatio-temporal convolutional module;
[0033] The output end of each spatio-temporal convolutional module and the output end of the depth convolutional module are successively provided with a third normalization layer and a third activation function layer.
[0034] In this embodiment, the frequency band adaptation network can capture the details and global features of different frequency bands through multi-scale convolution, and at the same time use the depth convolutional layer in EEGNet to extract the spatial information between channels, enhancing the feature expression ability, and can effectively improve the accuracy of the classification result of the electroencephalogram signal.
[0035] In an alternative embodiment, after extracting the spatio-temporal features of each signal to be processed, it further includes:
[0036] Using two fully connected layers arranged in sequence to map the spatio-temporal features of each signal to be processed into feature vectors;
[0037] Based on the feature vectors of all signals to be processed, calculate the classification probability of the electroencephalogram signal.
[0038] In this embodiment, by mapping the outputs of each branch to feature vectors and calculating the average value, the features from different frequency bands and scales can be effectively fused, avoiding the key information that may be lost in a single branch, thereby effectively improving the accuracy of the final electroencephalogram (EEG) signal classification result.
[0039] In a second aspect, the present invention provides a device for decoding motor imagery EEG signals, the device comprising:
[0040] An acquisition module, configured to acquire raw EEG signals;
[0041] A preprocessing module, configured to preprocess the raw EEG signals to obtain initial EEG signals after preprocessing;
[0042] A decoding module, configured to decompose the initial EEG signals into a plurality of to-be-processed signals in different frequency bands; input the to-be-processed signals into a pre-established coordinate attention mechanism module to fuse the spatio-temporal information of each to-be-processed signal in the channel dimension and the time dimension; based on the spatio-temporal information corresponding to each to-be-processed signal and a pre-established frequency band adaptation network, extract the spatio-temporal features of the to-be-processed signals, where the spatio-temporal features are used for EEG signal classification; wherein, the frequency band adaptation network adaptively selects the convolution kernel size according to the frequency band characteristics of the to-be-processed signals.
[0043] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, which are communicatively connected to each other, wherein the memory stores computer instructions, and the processor executes the computer instructions to execute the motor imagery EEG signal decoding method according to the first aspect or any corresponding embodiment thereof.
[0044] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the motor imagery EEG signal decoding method according to the first aspect or any corresponding embodiment thereof.
[0045] In a fifth aspect, the present invention provides a computer program product comprising computer instructions for causing a computer to execute the motor imagery EEG signal decoding method according to the first aspect or any corresponding embodiment thereof.
[0046] It should be noted that since the device for decoding motor imagery EEG signals, the computer device, the computer-readable storage medium, and the computer program product provided by the present invention correspond to the above-mentioned motor imagery EEG signal decoding method. Therefore, for the beneficial effects of the device for decoding motor imagery EEG signals, the computer device, the computer-readable storage medium, and the computer program product, please refer to the description of the corresponding beneficial effects of the above-mentioned motor imagery EEG signal decoding method, and details are not described herein again. Description of the Drawings
[0047] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings required for the description of the specific embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.
[0048] Figure 1 is a schematic flowchart of a method for decoding motor imagery electroencephalogram signals according to an embodiment of the present invention;
[0049] Figure 2 is a schematic overall structure diagram of the decoding of motor imagery electroencephalogram signals according to an embodiment of the present invention;
[0050] Figure 3 is a schematic structure diagram of a coordinate attention mechanism module according to an embodiment of the present invention;
[0051] Figure 4 is a schematic structure diagram of a frequency band adaptability network according to an embodiment of the present invention;
[0052] Figure 5 is a structural block diagram of a device for decoding motor imagery electroencephalogram signals according to an embodiment of the present invention;
[0053] Figure 6 is a schematic hardware structure diagram of a computer device according to an embodiment of the present invention. Specific Embodiments
[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.
[0055] Since there are still some problems in the currently used multi-branch CNN for decoding MI-EEG, at present, there is an urgent need for a decoding method that can not only adapt to different frequency bands but also achieve the fusion of channel and time dimension information.
[0056] In view of this, according to an embodiment of the present invention, an embodiment of a method for decoding motor imagery electroencephalogram signals is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.
[0057] In this embodiment, a method for decoding motor imagery electroencephalogram (EEG) signals is provided, which can be executed by devices such as servers, terminals, and mobile terminals. Figure 1 It is a flowchart of the method for decoding motor imagery EEG signals according to an embodiment of the present invention, as Figure 1 shown. The process includes the following steps:
[0058] Step S101, obtain the original EEG signal. The original EEG signal can be collected by an electroencephalograph device, which can detect the weak electrical signals generated during the activities of brain neurons, that is, the EEG signal.
[0059] Step S102, preprocess the original EEG signal to obtain the initial EEG signal after preprocessing.
[0060] For the original EEG signal, the Z-score normalization method can be used to eliminate the scale difference of the signal, and convert the signals in different frequency bands to the same scale for subsequent signal comparison and analysis. The normalization formula is: where μ and σ represent the mean and variance of the original EEG signal respectively. The preprocessed EEG signal can be a two-dimensional matrix with the sampling time of the EEG signal as the width and the electrode channels arranged in parallel as the length.
[0061] Step S103, decompose the initial EEG signal into multiple processed signals in different frequency bands. The original EEG signal can be filtered into three overlapping different frequency band signals through a multi-band decomposition module, for example: 0.1Hz - 16Hz, 8Hz - 30Hz, 22Hz - 40Hz.
[0062] Step S104, input the processed signal into a pre-established coordinate attention mechanism module to fuse the spatio-temporal information of each processed signal in the channel dimension and the time dimension.
[0063] In this embodiment, a coordinate attention mechanism is introduced to enhance the decoding ability of the model by fusing the channel and time direction information of each frequency band of MI-EEG. In EEG signals, different channels and time periods may carry different important information. Especially in the MI-EEG task, signals in different frequency bands reflect different brain activity patterns. The coordinate attention mechanism provided in this embodiment can calculate the attention distribution in space (channel direction) and time (time direction), enabling the model to adaptively focus on more critical spatial regions and time periods, thereby improving the decoding performance.
[0064] The coordinate attention mechanism provided in this embodiment adaptively assigns different attention weights to each channel and time step according to the characteristics of the input signal. In this way, the model can dynamically adjust its focus according to the importance of the signals in different channels and time periods, effectively avoiding the interference of irrelevant or noisy information.
[0065] Step S105: Based on the spatio-temporal information corresponding to each signal to be processed and the pre-established band adaptability network, extract the spatio-temporal features of the signal to be processed, where the spatio-temporal features are used for electroencephalogram (EEG) signal classification; among them, the band adaptability network adaptively selects the convolution kernel size according to the frequency band characteristics of the signal to be processed.
[0066] The coordinate attention mechanism module and the band adaptability network can be set in a pre-constructed processing model. In actual use, the signal to be processed can be directly input into the processing model.
[0067] Refer to Figure 2 As shown, after the coordinate attention mechanism module, a band adaptability network CNN is further introduced in this embodiment to extract the time-frequency related features of the signal and further integrate the time-frequency and spatial information. Different frequency bands of EEG signals usually reflect different brain activity patterns, and the signal characteristics of each frequency band are different. The band adaptability network can automatically select an appropriate convolution kernel size to process the signal of that frequency band according to the frequency band characteristics of the input signal, so that the convolutional layer can more accurately capture the time-frequency features of that frequency band. By combining the information in the time domain and the frequency domain, the band adaptability network can extract the time-frequency related features of the EEG signal, and these features can reflect the changes of the signal at different time points and frequency bands. In this embodiment, the use of adaptive convolution kernel selection avoids the problems of insufficient or excessive feature extraction that may be caused by a fixed convolution kernel size, which helps to improve the classification accuracy.
[0068] In the EEG signal processing task, in addition to time-frequency features, spatial information also plays a crucial role. The band adaptability network can help the model better integrate features in different dimensions by extracting time-frequency features and combining them with spatial information, thereby improving the accuracy of classification or decoding tasks. By combining with the coordinate attention mechanism module, the band adaptability network can further enhance the spatio-temporal feature integration ability of the model, enabling the model to have a more refined understanding of the spatial and time features of different signals.
[0069] In this embodiment, by fusing spatio-temporal information through the coordinate attention mechanism, the key information in the EEG signal can be extracted more accurately, improving the decoding performance; in addition, by adaptively selecting the convolution kernel size through the band adaptability network, the ability to capture features of different frequency bands is effectively improved, enhancing the classification accuracy. The present invention fully considers the frequency band differences and spatio-temporal characteristics of EEG signals, optimizes the signal processing of EEG signals, and effectively improves the accuracy of EEG signal classification.
[0070] As an objective neurophysiological biomarker, electroencephalogram (EEG) signals can provide direct information about the patient's brain activity, helping to more accurately identify and evaluate the presence and severity of brain diseases for assisting in treatment.
[0071] In some alternative embodiments, referring to Figure 3 as shown, the coordinate attention mechanism module includes:
[0072] Input layer (Input);
[0073] Two juxtaposed average pooling layers (Avg.Pool), and the input ends of the average pooling layers are all connected to the input layer;
[0074] The first convolutional module (Conv2D), and the input end of the first convolutional module is connected to the output end of the average pooling layer.
[0075] Input the signal to be processed into the coordinate attention mechanism module, and fuse the spatio-temporal information of each signal to be processed in the channel dimension and the time dimension, including:
[0076] Input the signal to be processed into the average pooling layer through the input layer, and respectively output the time information feature in the time dimension and the spatial information feature in the channel dimension; one of the average pooling layers is used to extract features of the signal to be processed along the time direction, and the other average pooling layer is used to extract features of the signal to be processed along the channel direction;
[0077] Based on the first convolutional module, fuse the time information feature and the spatial information feature to obtain the fused spatio-temporal information.
[0078] Specifically, the EEG signal of each frequency band can be described as: X ∈ R N×1×C×T . N represents the number of samples, 1 represents the number of convolutional channels, C represents the number of electrodes, and T represents the number of sampling points.
[0079] The two average pooling layers are respectively used to encode the EEG signal along the time and channel directions, and the features along the time direction will be transposed to align with the features along the channel direction.
[0080] The output of the coordinate information embedding can be expressed as:
[0081] where n represents the sample, c represents the number of channels, t represents the sampling point, and i, j represent variables.
[0082] Two groups of feature maps sensitive to the spatial direction and Make a connection and fuse it using the convolution transformation function F with a convolution kernel size of 1×1 to obtain: f = δ(F([z c ,z t ));
[0083] where [·,·] represents the concatenation operation in the channel and time dimensions, δ represents the non-linear activation function, and f ∈ R 8 ×1×(C+T) is the intermediate feature map encoding spatial information in both the channel and time directions, i.e., spatio-temporal information.
[0084] In this embodiment, the features in the time and space dimensions are respectively extracted through the parallel average pooling layers, realizing the efficient fusion of spatio-temporal information. At the same time, it also improves the model's ability to capture the spatio-temporal features of EEG signals and enhances the decoding accuracy.
[0085] In some alternative embodiments, as shown in Figure 3 the coordinate attention mechanism module further includes:
[0086] The first normalization layer (BatchNorm), the input end of the first normalization layer is connected to the output end of the first convolution module;
[0087] The first activation function layer (Sigmoid), the input end of the first activation function layer is connected to the output end of the first normalization layer;
[0088] Two parallel second convolution modules (Conv2D), the input end of the second convolution module is connected to the output end of the first activation function layer;
[0089] Each output end of the second convolution module is also provided with a second activation function layer (Sigmoid), and the output end of the second activation function layer is used to output the corresponding attention weight.
[0090] In some alternative embodiments, after obtaining the fused spatio-temporal information, it further includes:
[0091] Based on the first normalization layer and the first activation function layer, split the spatio-temporal information into two independent tensors;
[0092] Based on the second convolution module and the second activation function layer, transform the two independent tensors into tensors of the same size as when inputting into the coordinate attention mechanism module to obtain the attention weights in the channel dimension and time dimension.
[0093] Specifically, along the (C + T) dimension, split f ∈ R 8×1×(C+T) into two independent tensors f c ∈ R 8×1×C and f t ∈ R 8 ×1×T, and then use two 1×1 convolutional transformations F t and F c , respectively transform f t and f c into tensors of the same size as the input X∈R N×1×C×T , obtaining: g t =σ(F t (f t )); g c =σ(F c (f c ));
[0094] where σ represents the sigmoid function. g t and g c serve as attention weights respectively. The channel attention output of a single branch is:
[0095] In this embodiment, through the processing of the normalization layer and the activation function layer, the spatio-temporal information can be fully split and fused during feature extraction, improving the model's ability to understand features of different dimensions, and being able to generate accurate attention weights at the output end, enabling the model to adaptively adjust its attention to each channel and time dimension, enhancing the classification accuracy.
[0096] In some alternative embodiments, as shown in Figure 4 , the frequency band adaptation network includes:
[0097] Multiple spatio-temporal convolution modules (Conv2D) with different convolution scales arranged in sequence, where each spatio-temporal convolution module includes at least two convolutional layers, and the convolution kernel size of each convolutional layer decreases as the frequency band bandwidth increases;
[0098] The depth convolution module (DepthwiseConv2D) is arranged at the output end of the last spatio-temporal convolution module;
[0099] The output end of each spatio-temporal convolution module and the output end of the depth convolution module are successively provided with a third normalization layer (Batchnorm2D) and a third activation function layer (ELU).
[0100] In the frequency band adaptation network CNN, three spatio-temporal convolution modules with different convolution scales are built in parallel, respectively focusing on the detail and contour information of the signal. Specifically, each spatio-temporal convolution module contains two convolutional layers, and its convolution kernel size decreases as the bandwidth increases. Then, the depth convolutional layer in EEGNet is used to extract the spatial information between channels. A normalization layer and an activation function are added between every two convolutional layers to avoid network overfitting and gradient explosion. As the number of convolutional layers increases, the number of convolution kernel channels will also increase accordingly. The output size of each module is: (N, 32×(T - 2×Si + 2)), where S i represents the step size.
[0101] In this embodiment, the frequency-band adaptive network can capture the details and global features of different frequency bands through multi-scale convolution. At the same time, the depth convolution layer in EEGNet is used to extract the spatial information between channels, enhancing the feature expression ability and effectively improving the accuracy of the classification result of EEG signals.
[0102] In some alternative embodiments, after extracting the spatio-temporal features of each signal to be processed, it further includes:
[0103] Mapping the spatio-temporal features of each signal to be processed into feature vectors by using two sequentially arranged fully connected layers;
[0104] Calculating the classification probability of the EEG signal based on the feature vectors of all signals to be processed.
[0105] In this embodiment, after the frequency-band adaptive network CNN, two fully connected layers are used to map the output of each branch into feature vectors Then calculate the average value of the outputs of all branches, that is, the classification probability, and the final output can be described as: where b is the number of branches.
[0106] In this embodiment, by mapping the outputs of each branch into feature vectors and calculating the average value, the features from different frequency bands and scales can be effectively fused, avoiding the key information that may be lost in a single branch, thereby effectively improving the accuracy of the final classification result of EEG signals.
[0107] In this embodiment, a device for decoding motor imagery EEG signals is also provided. This device is used to implement the above embodiments and preferred embodiments, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.
[0108] This embodiment provides a device for decoding motor imagery EEG signals, as Figure 5 shown, the device includes:
[0109] An acquisition module 201, configured to acquire raw EEG signals;
[0110] A preprocessing module 202, configured to preprocess the raw EEG signals to obtain the initial EEG signals after preprocessing;
[0111] The decoding module 203 is used to decompose the initial EEG signal into multiple to-be-processed signals of different frequency bands; input the to-be-processed signals into a pre-established coordinate attention mechanism module to fuse the spatio-temporal information of each to-be-processed signal in the channel dimension and the time dimension; extract the spatio-temporal features of the to-be-processed signals based on the spatio-temporal information corresponding to each to-be-processed signal and the pre-established band adaptability network, and the spatio-temporal features are used for EEG signal classification; wherein, the band adaptability network adaptively selects the convolution kernel size according to the frequency band characteristics of the to-be-processed signals. It includes: inputting the to-be-processed signals through the input layer into the average pooling layer, and respectively outputting the time information features in the time dimension and the spatial information features in the channel dimension; one average pooling layer is used to extract features of the to-be-processed signals along the time direction, and the other average pooling layer is used to extract features of the to-be-processed signals along the channel direction; based on the first convolution module, fuse the time information features and the spatial information features to obtain the fused spatio-temporal information. The decoding module is also used to split the spatio-temporal information into two independent tensors based on the first normalization layer and the first activation function layer; based on the second convolution module and the second activation function layer, transform the two independent tensors into tensors of the same size as when input into the coordinate attention mechanism module to obtain the attention weights in the channel dimension and the time dimension. The decoding module is also used to map the spatio-temporal features of each to-be-processed signal into feature vectors by using two sequentially arranged fully connected layers; calculate the classification probability of the EEG signal based on the feature vectors of all to-be-processed signals.
[0112] The motor imagery EEG signal decoding device in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.
[0113] The further function descriptions of the above-mentioned various modules and units are the same as those in the corresponding embodiments above, and will not be repeated here.
[0114] The embodiment of the present invention also provides a computer device having the above-mentioned Figure 5 shown motor imagery EEG signal decoding device.
[0115] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of a computer device provided by an optional embodiment of the present invention. As Figure 6As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting the components, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, if needed, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (such as an array of servers, a set of blade servers, or a multi-processor system). Figure 6 In the figure, a processor 10 is taken as an example.
[0116] The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above-mentioned hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above-mentioned programmable logic device can be a complex programmable logic device, a field-programmable gate array, a generic array logic, or any combination thereof.
[0117] Among them, the memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiments.
[0118] The memory 20 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 can optionally include a memory remotely set relative to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.
[0119] The memory 20 can include a volatile memory, such as a random access memory; the memory can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state drive; the memory 20 can also include a combination of the above types of memories.
[0120] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.
[0121] Embodiments of the present invention also provide a computer-readable storage medium. The method according to the embodiments of the present invention can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium. Thus, the method described herein can be stored as such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.
[0122] A part of the present invention can be applied as a computer program product, such as computer program instructions. When executed by a computer, through the operation of the computer, the method and / or technical solution according to the present invention can be called or provided. Those skilled in the art should be able to understand that the forms of existence of computer program instructions in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Herein, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible by the computer.
[0123] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.
Claims
1. A method for decoding motor imagery EEG signals, characterized in that: The method comprises: Obtaining raw EEG signals; Preprocessing the original EEG signal to obtain a preprocessed initial EEG signal; Decomposing the initial EEG signal into a plurality of signals to be processed in different frequency bands; Input the signal to be processed into a pre-established coordinate attention mechanism module to fuse the spatiotemporal information of each signal to be processed in the channel dimension and the time dimension; Based on the spatiotemporal information corresponding to each of the signals to be processed and a pre-established frequency band adaptive network, the spatiotemporal features of the signals to be processed are extracted, and the spatiotemporal features are used for EEG signal classification; wherein the frequency band adaptive network adaptively selects the convolution kernel size according to the frequency band features of the signals to be processed.
2. The method according to claim 1, characterized in that The coordinate attention mechanism module includes: Input layer; Two average pooling layers arranged in parallel, wherein input ends of the average pooling layers are both connected to the input layer; A first convolution module, wherein an input end of the first convolution module is connected to an output end of the average pooling layer; The step of inputting the signal to be processed into a coordinate attention mechanism module and fusing the spatiotemporal information of each signal to be processed in the channel dimension and the time dimension includes: The signal to be processed is input into the average pooling layer through the input layer, and the time information features in the time dimension and the spatial information features in the channel dimension are output respectively; one of the average pooling layers is used to extract features of the signal to be processed along the time direction, and the other average pooling layer is used to extract features of the signal to be processed along the channel direction; Based on the first convolution module, the temporal information feature and the spatial information feature are fused to obtain the fused temporal and spatial information.
3. The method according to claim 2, characterized in that The coordinate attention mechanism module also includes: A first normalization layer, wherein an input end of the first normalization layer is connected to an output end of the first convolution module; A first activation function layer, wherein an input end of the first activation function layer is connected to an output end of the first normalization layer; Two second convolution modules arranged in parallel, wherein the input end of the second convolution module is connected to the output end of the first activation function layer; The output end of each of the second convolution modules is also provided with a second activation function layer, and the output end of the second activation function layer is used to output the corresponding attention weight.
4. The method according to claim 3, characterized in that After obtaining the fused spatiotemporal information, the method further includes: Based on the first normalization layer and the first activation function layer, split the spatiotemporal information into two independent tensors; Based on the second convolution module and the second activation function layer, the two independent tensors are transformed into tensors of the same size as those input into the coordinate attention mechanism module to obtain the attention weights of the channel dimension and the time dimension.
5. The method according to claim 1, characterized in that The frequency band adaptive network comprises: A plurality of sequentially arranged spatiotemporal convolution modules of different convolution scales, wherein each of the spatiotemporal convolution modules comprises at least two convolution layers, and the convolution kernel size of each convolution layer decreases as the frequency band bandwidth increases; A depth convolution module, arranged at the output end of the last spatiotemporal convolution module; The output end of each of the spatiotemporal convolution modules and the output end of the depth convolution module are sequentially provided with a third normalization layer and a third activation function layer.
6. The method according to claim 1, characterized in that After extracting the spatiotemporal features of each of the signals to be processed, the method further includes: Mapping the spatiotemporal features of each signal to be processed into a feature vector using two fully connected layers arranged in sequence; Based on the feature vectors of all the signals to be processed, the classification probability of the EEG signal is calculated.
7. A motor imagery EEG signal decoding device, characterized in that: The device comprises: An acquisition module is used to acquire raw EEG signals; A preprocessing module, used to preprocess the original EEG signal to obtain a preprocessed initial EEG signal; A decoding module is used to decompose the initial EEG signal into multiple signals to be processed in different frequency bands; input the signals to be processed into a pre-established coordinate attention mechanism module to fuse the spatiotemporal information of each signal to be processed in the channel dimension and the time dimension; based on the spatiotemporal information corresponding to each signal to be processed and a pre-established frequency band adaptive network, the spatiotemporal features of the signals to be processed are extracted, and the spatiotemporal features are used for EEG signal classification; wherein the frequency band adaptive network adaptively selects the convolution kernel size according to the frequency band features of the signals to be processed.
8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the motor imagery EEG signal decoding method according to any one of claims 1 to 6 by executing the computer instructions.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the motor imagery EEG signal decoding method described in any one of claims 1-6.
10. A computer program product, characterized in that It comprises computer instructions, and the computer instructions are used to make a computer execute the motor imagery EEG signal decoding method according to any one of claims 1-6.