A cortical electroencephalographic decoding method
By introducing temporal and spatial convolutional units of spiking neural networks and fusing them with handcrafted features, the problem of high energy consumption in cortical EEG decoding was solved, achieving efficient and energy-saving motor intention recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2025-04-08
- Publication Date
- 2026-07-03
AI Technical Summary
Existing cortical EEG decoding methods have high energy consumption while improving model performance, and traditional methods have neglected the improvement of model performance and the loss of information due to the singleness of features.
A cortical EEG decoding method is adopted, which introduces a spiking neural network, including temporal convolutional units and spatial convolutional units. The spatiotemporal features of multi-channel EEG signals are extracted through temporal and spatial convolution calculations, and then fused with handcrafted features to predict motion intention, thereby reducing energy consumption and improving model performance.
It significantly reduced the energy consumption of floating-point operations, improved the accuracy of cortical EEG decoding, stabilized the training process, made full use of EEG signal information, and solved the problems of energy consumption and performance.
Smart Images
Figure CN120408140B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of cortical electroencephalography (EEG) decoding, and more specifically, relates to a cortical EEG decoding method. Background Technology
[0002] Brain-computer interfaces (BCIs), as a communication pathway between the biological brain and external devices (controllers, wheelchairs, robotic arms, etc.), are mainly divided into non-invasive and invasive types based on the placement of sensors during signal acquisition. Invasive EEG signals, compared to non-invasive signals, possess more complete spatiotemporal characteristics. Among invasive BCIs, intracortical brain-computer interfaces (iBCIs) are widely used, covering areas such as motor decoding based on non-human mammals (NHPs) and human rehabilitation therapy.
[0003] In cortical EEG studies of rhesus monkeys, decoding motor intentions has enabled the control of sights and robotic arms. During signal acquisition, sensors are typically placed in three brain regions associated with movement (M1, S1, and PPC). In each region, a multi-electrode Utah array is used to acquire multi-channel EEG signals, with each electrode recording the neural activity of one or more neurons. Since neuronal activity and communication primarily rely on firing impulses, the EEG signals of rhesus monkeys exhibit different impulse states under different limb movements or spatial perception states, providing a basis for decoding their motor intentions.
[0004] The experimental paradigms mainly include the limb movement paradigm and the grasping paradigm. In the limb movement paradigm, three identical cylindrical objects are fixed on the experimental board, and the monkeys grasp the target objects using limb movements according to signal lights. In the grasping paradigm, objects of different shapes (such as cylinders, spheres, and triangular prisms) are transported from the same position on the experimental board at different times by motors, and the monkeys grasp them according to indicator lights. Both paradigms belong to multi-classification tasks.
[0005] Traditional rhesus monkey cortical EEG decoding generally involves three main steps: signal preprocessing, feature extraction, and feature classification. Signal preprocessing often employs the root mean square thresholding method to convert continuous analog signals into discrete pulse signals, which helps reflect the original neuronal activity and retains useful information. Feature extraction typically calculates neural activity vectors (NAVs) based on pulse firing frequencies as handcrafted features, while also appropriately downsampling the signal. In the feature classification stage, the decoding algorithms used are mostly linear machine learning models. For example, some literature proposes sparse Bayesian regressors for 3D motion decoding; others have used steady-state Kalman filters and their Gaussian process regression-based methods in experiments on three quadriplegic patients, achieving rapid closed-loop neural cursor control. However, traditional methods mainly focus on system control or data levels, employing relatively simple models and neglecting the improvement of the model's own performance. Furthermore, the features used are relatively limited, leading to the loss of some spatiotemporal information from the original signal.
[0006] Deep learning has been widely used in image and speech recognition, natural language processing and other fields. Artificial Neural Networks (ANNs) have high energy consumption due to the large number of floating-point operations involved. Spiking Neural Networks (SNNs) mimic the working mechanism of human brain neurons and reduce power consumption on neuromorphic chips through pulse drive. Reference [4] introduces a spiking neural network-long short-term memory network structure, which decodes movement through brain signals, highlighting the low energy consumption of spiking neural networks and utilizing the ability of long short-term memory networks to capture long-term time dependencies. Some literature uses a neural engineering framework to map control algorithms into spiking neural network models; some literature implements spiking neural networks on neuromorphic processors, using synaptic time-dependent plasticity as a learning method, and experiments have shown that the chip can correctly learn decoding tasks. In addition, Reference [7] innovatively uses spiking neural networks as data generators, showing that they can generate a small amount of training data that conforms to neural population dynamics, thereby improving the performance of cortical brain-computer interface decoders. Although these methods have shown that spiking neural networks have the potential to be highly energy-efficient in cortical EEG decoding, the relatively simple network structure leads to weak model performance. Therefore, controlling energy consumption while improving model performance has become an urgent problem to be solved. Summary of the Invention
[0007] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a cortical EEG decoding method, the purpose of which is to improve the model performance while controlling the energy consumption of the method operation.
[0008] To achieve the above objectives, according to one aspect of the present invention, a cortical electroencephalogram (EEG) decoding method is provided, comprising:
[0009] A spiking neural network is obtained, comprising a temporal convolutional unit, a spatial convolutional unit, and a pooling unit. The temporal convolutional unit is used to calculate the temporal pulses of the cortical electroencephalogram (EEG) signals by channel to obtain multi-channel temporal pulse features. The spatial convolutional unit is used to calculate the spatial pulses of the multi-channel temporal pulse features to obtain spatiotemporal pulse features. The pooling unit is used to average the spatiotemporal pulse features in the temporal dimension to obtain a depth feature vector.
[0010] The manual features of the cortical EEG signal are calculated and denoted as NAV feature vectors; the depth feature vector and the NAV feature vector are linearly projected to obtain feature vectors of corresponding preset dimensions; the projected feature vectors of preset dimensions are concatenated to obtain joint features; and motor intention is predicted based on the joint features to achieve cortical EEG decoding.
[0011] Furthermore, the temporal convolutional unit includes a temporal convolutional layer, a first normalization layer, and a first PLIF neuron;
[0012] The temporal convolutional layer is used to perform one-dimensional convolution operations on cortical EEG signals by channel;
[0013] The first normalization layer is used to calculate the mean and variance of each channel in the output of the temporal convolutional layer, and to perform element-wise normalization on the feature vector of each channel based on the mean and variance; and to perform element-wise linear transformation on the normalized feature vector to obtain the normalized temporal feature of the channel.
[0014] The first PLIF neuron is used to perform pulse calculation on the standardized time features of each channel to obtain the pulse feature vector of that channel, which is composed of the number of pulses at each time step. The pulse feature vectors of each channel constitute the time pulse features.
[0015] Furthermore, the spatial convolution branch includes a spatial convolutional layer, a second normalized layer, and a second PLIF neuron;
[0016] The spatial convolutional layer is used to perform two-dimensional convolution operations on the multi-channel temporal features;
[0017] The second normalization layer is used to calculate the mean and variance of each channel in the output of the spatial convolutional layer, and to perform element-wise normalization on the feature vector of each channel based on the mean and variance; and to perform element-wise linear transformation on the normalized feature vector to obtain the normalized spatial features of the channel.
[0018] The second PLIF neuron is used to perform pulse calculation on the standardized spatial features of each channel to obtain the pulse feature vector of that channel, which is composed of the number of pulses at each time step. The spatiotemporal pulse features are composed of the pulse feature vectors of each channel.
[0019] Furthermore, the implementation method for motion intent prediction based on joint features is as follows:
[0020] The joint features are multiplied by the weight matrix of the classifier, and the prediction result is obtained through classification.
[0021] Furthermore, the NAV feature vector is determined as follows:
[0022] The data of each channel of the cortical EEG signal is divided into multiple time periods by time dimension. The number of pulses in multiple time periods of the channel is calculated to obtain the pulse number vector corresponding to the channel. A matrix composed of the pulse number vectors of all channels is constructed. The matrix is flattened to obtain the NAV feature vector. Each time period contains multiple time steps.
[0023] Furthermore, the multiple time periods are multiple non-overlapping time periods of equal length.
[0024] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.
[0025] According to another aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the storage medium is located to perform the steps of the method described above.
[0026] According to another aspect of the invention, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the steps of the method described above.
[0027] In summary, compared with the prior art, the technical solutions conceived by this invention have the following main advantages:
[0028] 1. This invention proposes a cortical EEG decoding method. First, a spiking neural network (SNN) is introduced, comprising temporal convolutional units and spatial convolutional units. For each channel of the EEG signal, temporal convolution and pulse signal calculation are performed separately to obtain the temporal pulse features for each channel. These multi-channel temporal pulse features are simultaneously input into the spatial convolutional unit, which performs spatial convolution between the multi-channel temporal pulse features to obtain spatiotemporal pulse features, which serve as depth features. The method further introduces handcrafted features to calculate NAV features. The depth features and handcrafted features obtained from the spiking neural network are projected separately and then concatenated for motion intention prediction. The spiking neural network used in this method significantly reduces floating-point operations, making it more efficient and energy-saving on corresponding hardware. Furthermore, this method integrates depth features and handcrafted features, fully utilizing the information in the EEG signal, thus solving the problems of model performance and energy consumption in current cortical EEG-based motion intention recognition models.
[0029] 2. The present invention further proposes that both the temporal convolutional unit and the spatial convolutional unit should first normalize after the convolution operation and then perform the spiking operation. The batch normalization layer makes the training more stable, and the spiking neuron can convert the data into a spiking format, thereby reducing energy consumption. Attached Figure Description
[0030] Figure 1 A framework diagram of a cortical EEG decoding method provided in an embodiment of the present invention;
[0031] Figure 2 This is a schematic flowchart of a cortical electroencephalogram (EEG) decoding method provided in an embodiment of the present invention. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0033] Example 1
[0034] A cortical EEG decoding method, such as Figure 1 As shown, it includes:
[0035] A spiking neural network is obtained, comprising temporal convolutional units, spatial convolutional units, and pooling units. The temporal convolutional units are used to calculate temporal pulses of the cortical EEG signal by channel to obtain multi-channel temporal pulse features. The spatial convolutional units are used to calculate spatial pulses of the multi-channel temporal pulse features to obtain spatiotemporal pulse features. The pooling units are used to average the spatiotemporal pulse features over the temporal dimension to obtain a depth feature vector. The handcrafted features of the cortical EEG signal are calculated and denoted as the NAV feature vector. The depth feature vector and the NAV feature vector are linearly projected to obtain feature vectors of corresponding preset dimensions. The projected feature vectors of preset dimensions are concatenated to obtain joint features. Based on the joint features, motion intention prediction is performed to achieve cortical EEG decoding.
[0036] Traditional cortical EEG classification relies on handcrafted features related to pulse firing rate and uses relatively simple mathematical models. While deep learning methods are widely used in other fields, their application in cortical brain-computer interface classification is relatively limited. This embodiment proposes a novel spiking neural network architecture aimed at efficiently decoding EEG signals. Furthermore, it combines traditional features with deep features through feature fusion (FF) to further improve decoding accuracy. The spiking neural network model extracts spatiotemporal features of the original signal through spatiotemporal convolution. The feature fusion method combines features extracted by the deep model with traditional NAV features, incorporating pulse firing rate-related information while preserving the spatiotemporal characteristics of the data, and can be applied to various deep models.
[0037] Temporal convolutional units and spatial convolutional units constitute the feature extractor of a spiking neural network. In a preferred embodiment, the temporal convolutional unit includes a temporal convolutional layer, a first normalization layer, and a first PLIF neuron. The temporal convolutional layer performs one-dimensional convolution on the cortical EEG signal by channel. The first normalization layer calculates the mean and variance of each channel in the output of the temporal convolutional layer, and performs element-wise normalization on the feature vector of each channel based on the mean and variance. An element-wise linear transformation is then performed on the normalized feature vector to obtain the normalized temporal feature of that channel. The first PLIF neuron performs pulse calculation on the normalized temporal feature of each channel to obtain the pulse feature vector of that channel, which is composed of the number of pulses at each time step. The pulse feature vectors of each channel constitute the temporal pulse feature, which can be represented as z = [z1; z2; ...].
[0038] In a preferred embodiment, the spatial convolution branch includes a spatial convolutional layer, a second normalization layer, and a second PLIF neuron. The spatial convolutional layer performs a two-dimensional convolution operation on the multi-channel temporal features. The second normalization layer calculates the mean and variance of each channel in the output of the spatial convolutional layer, and performs element-wise normalization on the feature vector of each channel based on the mean and variance. An element-wise linear transformation is then performed on the normalized feature vector to obtain the normalized spatial features of that channel. The second PLIF neuron performs pulse calculation on the normalized spatial features of each channel to obtain the pulse feature vector of that channel, which is composed of the number of pulses at each time step. The pulse feature vectors of each channel constitute the spatiotemporal pulse features.
[0039] The convolution formulas for temporal and spatial convolution units are as follows:
[0040]
[0041] F = K S *z
[0042] Where K is the convolution kernel, * is the convolution operation, and F is the impulse feature.
[0043] Normalization and Neuron Processing: After each convolutional layer, a one-dimensional batch normalization layer and a PLIF neuron (Parametric Leaky-and-Intergrate Fire) are set up. The batch normalization layer calculates normalization per channel. For the output from the convolutional layer, the mean and variance of each channel are first calculated, and then a standardization operation is performed (the output of the convolutional layer is subtracted from the mean and then divided by the variance) to obtain standardized features. Then, a linear transformation is performed on the standardized features of each channel to obtain the final output. For the PLIF neuron, the time dimension of each channel's data is treated as a time step, and impulse calculation is performed to obtain impulse features.
[0044] As a preferred implementation method, motion intent prediction based on joint features can be achieved by multiplying the joint features with the weight matrix of the classifier and obtaining the prediction result through classification.
[0045] The classifier employs a fully connected layer and uses a classification strategy based on pulse firing rate. The spatiotemporal pulse features obtained from the feature extractor are averaged along the time step dimension to obtain a deep feature vector f, denoted as: In the formula F t Let F be the column vector representing the spatiotemporal impulse feature F at the t-th time step.
[0046] As a preferred implementation method, the NAV feature vector is determined as follows:
[0047] The data of each channel of the cortical EEG signal is divided into multiple time periods by time. The number of pulses in multiple time periods of the channel is calculated to obtain the pulse number vector corresponding to the channel. A matrix composed of the pulse number vectors of all channels is constructed. The matrix is flattened to obtain the NAV feature vector. Each time period contains multiple time steps.
[0048] As an implementation example, such as Figure 2 As shown, a decoding method is illustrated, which includes the implementation of multi-channel time pulse features, spatiotemporal pulse features, manual feature calculation, mapping, etc.
[0049] As a preferred embodiment, the multiple time periods are multiple non-overlapping time periods of equal length.
[0050] Deep feature vector f and handcrafted feature vector f NAV The splicing and fusion is represented as:
[0051] FF = Proj(f) ⊕ Proj(f) NAV )
[0052] In the formula, FF represents the joint features. Then, it is combined with the weight matrix W of the classifier. c Performing a dot product operation yields the final prediction result: y = W c ·FF, where y represents the prediction result.
[0053] Experiments verified that, on the rhesus monkey cortical EEG dataset, the model proposed in this invention achieves higher classification accuracy and lower energy consumption compared to ANN-based deep models (ShallowConvNet, DeepConvNet, EEGConformer, EEGDeforer, EEGNet). Furthermore, the feature fusion method can further improve the classification accuracy of the deep model, as shown in Table 1.
[0054] Table 1: Classification accuracy of different methods on the rhesus monkey cortical EEG dataset.
[0055]
[0056] In the table above, the highest result in each experimental data segment is marked in bold, and the second highest result is marked with an underline.
[0057] The method in this embodiment, through experimental testing, verifies that the proposed spiking neural network model and feature fusion method can achieve stable performance improvement and energy consumption reduction in rhesus monkey cortical EEG data processing, and can be used as a practical method for decoding motor intentions based on cortical EEG.
[0058] Example 2
[0059] A model construction method for decoding cortical electroencephalogram (EEG) signals includes: constructing a spiking neural network unit, a handcrafted feature calculation unit, and a prediction unit. The spiking neural network unit includes a temporal convolution unit, a spatial convolution unit, and a pooling unit. The temporal convolution unit calculates temporal pulses on the cortical EEG signals by channel to obtain multi-channel temporal pulse features. The spatial convolution unit calculates spatial pulses on the multi-channel temporal pulse features to obtain spatiotemporal pulse features. The pooling unit averages the spatiotemporal pulse features over the temporal dimension to obtain a depth feature vector. The handcrafted feature calculation unit calculates handcrafted features of the cortical EEG signals, denoted as the NAV feature vector. The prediction unit linearly projects the depth feature vector and the NAV feature vector to obtain feature vectors of corresponding preset dimensions. The projected feature vectors of the preset dimensions are concatenated to obtain joint features. Based on the joint features, motion intention is predicted to achieve cortical EEG decoding. The related technical solutions are the same as in Embodiment 1 and will not be repeated here.
[0060] In summary, this invention proposes a novel spiking neural network architecture that extracts multi-level temporal information through channel-wise temporal convolution, followed by a spatial convolutional layer to extract information between different channels. Simultaneously, batch normalization layers stabilize training, and spiking neurons convert data into a pulse format, thereby reducing energy consumption. Furthermore, feature fusion methods can further improve the classification accuracy of the proposed network and other artificial neural networks.
[0061] Example 3
[0062] This application also relates to an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.
[0063] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.
[0064] The relevant technical solutions are the same as above, and will not be repeated here.
[0065] Example 4
[0066] This application also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.
[0067] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.
[0068] The relevant technical solutions are the same as above, and will not be repeated here.
[0069] Example 5
[0070] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method described in the above embodiments of this application.
[0071] The relevant technical solutions are the same as above, and will not be repeated here.
[0072] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A cortical electroencephalographic decoding method, characterized by, include: A spiking neural network is obtained, comprising a temporal convolutional unit, a spatial convolutional unit, and a pooling unit. The temporal convolutional unit is used to calculate the temporal pulses of the cortical electroencephalogram (EEG) signals by channel to obtain multi-channel temporal pulse features. The spatial convolutional unit is used to calculate the spatial pulses of the multi-channel temporal pulse features to obtain spatiotemporal pulse features. The pooling unit is used to average the spatiotemporal pulse features over the time dimension to obtain a depth feature vector; The manual features of the cortical EEG signal are calculated and denoted as NAV feature vectors; the depth feature vector and the NAV feature vector are linearly projected to obtain feature vectors of corresponding preset dimensions; the projected feature vectors of preset dimensions are concatenated to obtain joint features; motor intention is predicted based on the joint features to achieve cortical EEG decoding. The temporal convolutional unit includes a temporal convolutional layer, a first normalization layer, and a first PLIF neuron. The temporal convolutional layer is used to perform one-dimensional convolution operations on cortical EEG signals by channel; The first normalization layer is used to calculate the mean and variance of each channel in the output of the temporal convolutional layer, and to perform element-wise normalization on the feature vector of each channel based on the mean and variance; and to perform element-wise linear transformation on the normalized feature vector to obtain the normalized temporal feature of the channel. The first PLIF neuron is used to perform pulse calculation on the standardized time features of each channel to obtain the pulse feature vector of that channel, which is composed of the number of pulses at each time step. The pulse feature vectors of each channel constitute the time pulse features. The spatial convolutional unit includes a spatial convolutional layer, a second normalization layer, and a second PLIF neuron. The spatial convolutional layer is used to perform two-dimensional convolution operations on the multi-channel temporal pulse features; The second normalization layer is used to calculate the mean and variance of each channel in the output of the spatial convolutional layer, and to perform element-wise normalization on the feature vector of each channel based on the mean and variance; and to perform element-wise linear transformation on the normalized feature vector to obtain the normalized spatial features of the channel. The second PLIF neuron is used to perform pulse calculation on the standardized spatial features of each channel to obtain the pulse feature vector of that channel, which is composed of the number of pulses at each time step. The spatiotemporal pulse features are composed of the pulse feature vectors of each channel.
2. A cortical electroencephalographic decoding method as claimed in claim 1, characterized in that, The implementation method for motion intent prediction based on joint features is as follows: The joint features are multiplied by the weight matrix of the classifier, and the prediction result is obtained through classification.
3. The cortical EEG decoding method as described in claim 1, characterized in that, The NAV feature vector is determined as follows: The data of each channel of the cortical EEG signal is divided into multiple time periods by time dimension. The number of pulses in multiple time periods of the channel is calculated to obtain the pulse number vector corresponding to the channel. A matrix composed of the pulse number vectors of all channels is constructed. The matrix is flattened to obtain the NAV feature vector. Each time period contains multiple time steps.
4. The cortical EEG decoding method as described in claim 3, characterized in that, Multiple time periods are multiple non-overlapping time periods of equal length.
5. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 4.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein the computer program, when executed by a processor, controls the device on which the storage medium is located to perform the steps of the method as described in any one of claims 1 to 4.
7. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 4.