Cortical electroencephalogram decoding method

Abstract

The invention belongs to the cortical electroencephalogram decoding related technical field, and particularly relates to a cortical electroencephalogram decoding method comprising the steps that a pulse neural network is acquired, the pulse neural network comprises a time convolution unit, a space convolution unit and a pooling unit, the time convolution unit is used for carrying out time pulse calculation on cortical electroencephalogram signals according to channels, and multi-channel time pulse features are obtained; the spatial convolution unit is used for carrying out spatial pulse calculation on the multi-channel time pulse characteristics to obtain space-time pulse characteristics; the pooling unit is used for averaging the space-time pulse features in the time dimension to obtain a depth feature vector; manual features of the cortical electroencephalogram signals are calculated and recorded as NAV feature vectors; respectively carrying out linear projection on the depth feature vector and the NAV feature vector to obtain a feature vector corresponding to a preset dimension; splicing the projected feature vectors of the preset dimension to obtain a joint feature; and motion intention prediction is carried out based on the joint features, and cortical electroencephalogram decoding is realized. The method is low in energy consumption and high in precision.

Description

Technical Field

[0001] The present invention belongs to the technical field related to cortical brain electrodecoding, and more specifically, relates to a cortical brain electrodecoding method. Background Art

[0002] As a communication path between a biological brain and external devices (controllers, wheelchairs, robotic arms, etc.), a brain-computer interface is mainly divided into non-invasive and invasive types according to the sensor placement positions when collecting signals. Compared with non-invasive signals, invasive electroencephalogram signals have more perfect spatio-temporal characteristics. In invasive brain-computer interfaces, intracortical brain-computer interfaces (iBCIs) are widely used, covering aspects such as motion decoding based on non-human mammals (NHPs) and rehabilitation therapy for humans.

[0003] In the study of rhesus monkey cortical electroencephalogram, the control of a reticle and a robotic arm can be achieved by decoding the motion intention. When collecting signals, sensors are usually placed in three brain regions related to motion (M1, S1, and PPC regions). In each brain region, a multi-electrode Utah array is used to collect multi-channel electroencephalogram signals, and each electrode can record the neural activities of one or several neurons. The activities of brain neurons and the communication between neurons are mainly achieved by firing pulses. Therefore, when the rhesus monkey is in different limb movement or different spatial perception states, its electroencephalogram signals show different pulse states, which provides the possibility for decoding its motion intention.

[0004] The experimental paradigms mainly include the limb movement paradigm and the grasping paradigm. In the limb movement paradigm, three identical cylindrical objects are fixed on an experimental board, and the experimental monkey grasps the target object through limb movement according to the indication of a signal lamp; in the grasping paradigm, objects of different shapes (such as cylindrical, spherical, and triangular prism-shaped) are transported by a motor at different times at the same position on the experimental board, and the experimental monkey grasps according to the indicator light. Both of these paradigms belong to multi-classification tasks.

[0005] Traditional decoding of the rhesus monkey cortical brain electrical signals generally involves three main steps: signal preprocessing, feature extraction, and feature classification. The root mean square threshold method is often used for signal preprocessing to convert continuous analog signals into discrete pulse signals, which helps to reflect the original neuronal activities and retain useful information. Feature extraction usually calculates the neural activity vector (NAV) based on the spike firing frequency as a handcrafted feature and can also appropriately downsample the signals. In the feature classification step, the decoding algorithms used are mostly linear machine learning models. For example, some literature has proposed a sparse Bayesian regressor for three-dimensional motion decoding; some literature has used a steady-state Kalman filter and its method based on Gaussian process regression to conduct experiments on three tetraplegic patients and achieved fast closed-loop neural cursor control. However, traditional methods mainly focus on system control or the data level, use relatively simple models, and neglect the improvement of the model's own performance. Moreover, the features used are relatively single, resulting in the loss of some spatio-temporal information of the original signals.

[0006] Deep learning has been widely applied in fields such as image and speech recognition, and natural language processing. Since the artificial neural network involves a large number of floating-point operations, its energy consumption is relatively high. The "spiking" neural network (SNN) mimics the working mechanism of human brain neurons and reduces the power consumption on neuromorphic chips through spike driving. Reference [4] introduced the spiking neural network-long short-term memory network structure to decode motion through brain signals, which not only highlights the low energy consumption characteristics of the spiking neural network but also utilizes the ability of the long short-term memory network to capture long-term time dependencies. Some literature has mapped control algorithms to the spiking neural network model using a neuroengineering framework; some literature has implemented the spiking neural network on a neuromorphic processor and used spike-timing-dependent plasticity as a learning method, and experiments have proven that the chip can correctly learn the decoding task. In addition, reference [7] innovatively used the spiking neural network as a data generator, indicating that it can generate a small amount of training data that conforms to neural population dynamics, thereby improving the performance of the cortical brain-computer interface decoder. Although these methods show the potential of the spiking neural network for efficient energy conservation in cortical brain electrical signal decoding, the relatively simple network structure leads to weak model performance. Therefore, it has become an urgent problem to control energy consumption while improving the model performance. Summary of the Invention

[0007] In view of the above defects or improvement requirements of the prior art, the present invention provides a method for decoding cortical brain electrical signals, aiming to control the energy consumption during the operation of the method while improving the model performance.

[0008] To achieve the above objective, according to one aspect of the present invention, a method for decoding cortical brain electrical signals is provided, including:

[0009] Obtain a spiking neural network, which includes a temporal convolutional unit, a spatial convolutional unit, and a pooling unit. The temporal convolutional unit is used to calculate temporal spikes for electrocorticogram signals by channel to obtain multi-channel temporal spike features; the spatial convolutional unit is used to calculate spatial spikes for the multi-channel temporal spike features to obtain spatio-temporal spike features; the pooling unit is used to average the spatio-temporal spike features in the temporal dimension to obtain a depth feature vector;

[0010] Calculate the handcrafted features of the electrocorticogram signals, denoted as NAV feature vectors; linearly project the depth feature vector and the NAV feature vectors respectively to obtain feature vectors corresponding to a preset dimension; splice the projected feature vectors of the preset dimension to obtain joint features; perform motion intention prediction based on the joint features to achieve electrocorticogram decoding.

[0011] Furthermore, the temporal convolutional unit includes a temporal convolutional layer, a first normalization layer, and a first PLIF neuron;

[0012] The temporal convolutional layer is used to perform one-dimensional convolutional operations on electrocorticogram signals by channel;

[0013] The first normalization layer is used to calculate the mean and variance of each channel in the output of the temporal convolutional layer, and perform element-wise normalization processing on the feature vectors of each channel based on the mean and variance; perform element-wise linear transformation on the normalized feature vectors to obtain the normalized temporal features of the channel;

[0014] The first PLIF neuron is used to perform spike calculations on the normalized temporal features of each channel to obtain a spike feature vector composed of the number of spikes at each time step of the channel, and the time pulse features are composed of the spike feature vectors of each channel.

[0015] Furthermore, the spatial convolution branch includes a spatial convolutional layer, a second normalization layer, and a second PLIF neuron;

[0016] The spatial convolutional layer is used to perform two-dimensional convolutional operations on the multi-channel temporal features;

[0017] The second normalization layer is used to calculate the mean and variance of each channel in the output of the spatial convolutional layer, and perform element-wise normalization processing on the feature vectors of each channel based on the mean and variance; perform element-wise linear transformation on the normalized feature vectors to obtain the normalized spatial features of the channel;

[0018] The second PLIF neuron is used to perform impulse calculation on the normalized spatial features of each channel to obtain an impulse feature vector of each channel composed of the number of impulses at each time step, and the spatio-temporal impulse features are composed of the impulse feature vectors of each channel.

[0019] Further, the implementation manner of predicting the motion intention based on the joint features is as follows:

[0020] Multiply the joint features with the weight matrix of the classifier, and obtain the prediction result through classification.

[0021] Further, the determination method of the NAV feature vector is as follows:

[0022] Divide the data of each channel of the cortical electroencephalogram signal in the time dimension to obtain multiple time periods, calculate the number of impulses in multiple time periods of this channel to obtain an impulse number vector corresponding to this channel, construct a matrix composed of the impulse number vectors of all channels, and flatten the matrix to obtain the NAV feature vector, where each time period contains multiple time steps.

[0023] Further, the multiple time periods are multiple non-overlapping and equal-length time periods.

[0024] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps of the method described above are implemented.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium includes a stored computer program, wherein when the computer program is run by a processor, the device where the storage medium is located is controlled to execute the steps of the method described above.

[0026] According to another aspect of the present invention, a computer program product is provided, including a computer program or instruction, and when the computer program or instruction is executed by a processor, the steps of the method described above are implemented.

[0027] Generally speaking, compared with the prior art by the above technical solution conceived by the present invention, the technical solution provided by the present invention mainly has the following beneficial effects:

[0028] 1. The present invention proposes a method for decoding cortical electroencephalogram. First, a spiking neural network is introduced, which includes a temporal convolutional unit and a spatial convolutional unit. For the electroencephalogram signals of each channel, temporal convolution and calculation of spiking signals are respectively performed to obtain the temporal spiking features of each channel. The multi-channel temporal spiking features are simultaneously input into the spatial convolutional unit, and the spatial convolutional unit performs convolution between channels in the spatial dimension on the multi-channel temporal spiking features to obtain spatio-temporal spiking features as deep features. The method further introduces handcrafted features and calculates NAV features. The deep features obtained by the spiking neural network and the handcrafted features are respectively projected and then concatenated and fused for motion intention prediction. The spiking neural network adopted by this method significantly reduces floating-point operations, is more efficient and energy-saving when running on corresponding hardware, and this method fuses deep features and handcrafted features, making full use of the information of electroencephalogram signals. Therefore, it can solve the problems of model performance and running energy consumption in current motion intention recognition based on cortical electroencephalogram.

[0029] 2. The present invention further proposes that in both the temporal convolutional unit and the spatial convolutional unit, normalization is first performed after the convolution operation and then the spiking calculation operation. The batch normalization layer makes the training more stable, and the spiking neurons can convert the data into a spiking format, thereby reducing energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a framework diagram of a method for decoding cortical electroencephalogram provided by an embodiment of the present invention;

[0031] Figure 2 It is a schematic flowchart of a method for decoding cortical electroencephalogram provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0033] Embodiment 1

[0034] A method for decoding cortical electroencephalogram, as Figure 1 shown, includes:

[0035] Obtain a spiking neural network, which includes a temporal convolutional unit, a spatial convolutional unit, and a pooling unit. The temporal convolutional unit is used to calculate temporal spikes for electrocorticogram (ECoG) signals channel by channel to obtain multi-channel temporal spike features; the spatial convolutional unit is used to calculate spatial spikes for the multi-channel temporal spike features to obtain spatio-temporal spike features; the pooling unit is used to average the spatio-temporal spike features in the temporal dimension to obtain a deep feature vector; calculate the handcrafted features of the ECoG signals, denoted as the NAV feature vector; linearly project the deep feature vector and the NAV feature vector respectively to obtain feature vectors corresponding to a preset dimension; splice the projected feature vectors of the preset dimension to obtain a joint feature; perform motion intention prediction based on the joint feature to achieve ECoG decoding.

[0036] Traditional ECoG classification relies on handcrafted features related to spike firing rates and uses relatively simple mathematical models. Although deep learning methods are widely used in other fields, their application in ECoG-based brain-computer interface classification is relatively insufficient. This embodiment proposes a brand-new spiking neural network architecture, aiming to decode ECoG signals efficiently and energy-savingly. At the same time, through the Feature Fusion (FF) method, traditional features are combined with deep features to further improve the decoding accuracy. The spiking neural network model extracts spatio-temporal features of the original signal through spatio-temporal convolution. The feature fusion method can fuse the features extracted by the deep model with the traditional NAV features, incorporate information related to spike firing rates while retaining the spatio-temporal features of the data, and can be applied to various deep models.

[0037] The temporal convolutional unit and the spatial convolutional unit constitute the feature extractor of the spiking neural network. As a preferred implementation, the temporal convolutional unit includes a temporal convolutional layer, a first normalization layer, and a first PLIF neuron; wherein, the temporal convolutional layer is used to perform one-dimensional convolutional operations on the ECoG signals channel by channel; the first normalization layer is used to calculate the mean and variance of each channel in the output of the temporal convolutional layer, and perform element-wise normalization processing on the feature vectors of each channel based on the mean and variance; perform element-wise linear transformation on the normalized feature vectors to obtain the normalized temporal features of each channel; the first PLIF neuron is used to perform spike calculations on the normalized temporal features of each channel to obtain a spike feature vector composed of the number of spikes at each time step for each channel, and the spike feature vectors of each channel constitute the temporal spike features, which can be expressed as z = [z1; z2;...].

[0038] As a preferred embodiment, the spatial convolution branch includes a spatial convolution layer, a second normalization layer, and a second PLIF neuron; wherein, the spatial convolution layer is used to perform a two-dimensional convolution operation on the multi-channel temporal features; the second normalization layer is used to calculate the mean and variance of each channel in the output of the spatial convolution layer, and perform element-wise normalization processing on the feature vectors of each channel based on the mean and variance; perform element-wise linear transformation on the normalized feature vectors to obtain the normalized spatial features of the channel; the second PLIF neuron is used to perform pulse calculation on the normalized spatial features of each channel to obtain a pulse feature vector composed of the number of pulses at each time step of the channel, and the spatio-temporal pulse features are composed of the pulse feature vectors of each channel.

[0039] The convolution calculation formulas of the temporal convolution unit and the spatial convolution unit are as follows:

[0040]

[0041] F = K S * z

[0042] where, K is the convolution kernel, * is the convolution operation, and F is the pulse feature.

[0043] Normalization and neuron processing: After each convolution layer, a one-dimensional batch normalization (batchnorm) layer and a PLIF neuron (Parametric leaky-and-intergrate fire) are immediately set. The batchnorm layer calculates normalization by channel. For the output from the convolution layer, first calculate the mean and variance of each channel, perform normalization operations (subtract the mean from the output of the convolution layer and then divide by the variance) to obtain the normalized features, and then perform linear transformation on the normalized features of each channel to obtain the final output. For the PLIF neuron, regard the time dimension of the data of each channel as time steps, perform pulse calculation, and obtain the pulse features.

[0044] As a preferred embodiment, the implementation manner of predicting the motion intention based on the joint features is: perform a dot product of the joint features and the weight matrix of the classifier, and obtain the prediction result through classification.

[0045] The classifier adopts a fully connected layer and uses a classification strategy based on the firing rate. Average the spatio-temporal pulse features obtained from the feature extractor in the time step dimension to obtain a deep feature vector f, which is expressed as: In the formula, F t represents the column vector of the spatio-temporal pulse feature F at the t-th time step.

[0046] As a preferred embodiment, the determination method of the NAV feature vector is:

[0047] Divide the data of each channel of the cortical electroencephalogram signal in the time dimension to obtain multiple time periods, calculate the number of pulses in multiple time periods of this channel, obtain the pulse number vector corresponding to this channel, construct a matrix composed of the pulse number vectors of all channels, and flatten the matrix to obtain the NAV feature vector, where each time period contains multiple time steps.

[0048] As an implementation example, as Figure 2 shown, a decoding method is shown, which includes implementation methods such as multi-channel time pulse features, spatio-temporal pulse features, manual feature calculation, mapping, etc.

[0049] As a preferred implementation method, the multiple time periods are multiple non-overlapping and equal-length time periods.

[0050] Deep feature vector f and the manual feature vector f NAV are spliced and fused to be expressed as:

[0051] FF = Proj(f) ⊕ Proj(f NAV )

[0052] In the formula, FF represents the combined feature. Then, through the dot product operation with the weight matrix W c of the classifier, the final prediction result is obtained: y = W c ·FF, where y represents the prediction result.

[0053] Experimental verification shows that on the rhesus monkey cortical electroencephalogram dataset, the model proposed by the present invention has a higher classification accuracy rate and lower energy consumption compared with the ANN-based deep models (ShallowConvNet, DeepConvNet, EEGConformer, EEGDeforer, EEGNet). At the same time, the feature fusion method can further improve the classification accuracy of the deep model, as shown in Table 1.

[0054] Table 1: Classification accuracy rates of different methods on the rhesus monkey cortical electroencephalogram dataset.

[0055]

[0056] In the above table, the highest result on each experimental data segment is marked in bold, and the second highest result is marked with an underline.

[0057] The method of this embodiment is tested through experiments, and it is verified that the proposed pulsed neural network model and feature fusion method can achieve stable performance improvement and energy consumption reduction in the processing of rhesus monkey cortical electroencephalogram data, and can be used as an actual cortical electroencephalogram-based movement intention decoding method.

[0058] Embodiment 2

[0059] A method for constructing a model for decoding cortical electroencephalogram signals, comprising: constructing a spiking neural network unit, a handcrafted feature calculation unit, and a prediction unit, wherein the spiking neural network unit includes a temporal convolutional unit, a spatial convolutional unit, and a pooling unit. The temporal convolutional unit is used to calculate temporal spikes of the cortical electroencephalogram signals channel by channel to obtain multi-channel temporal spike features; the spatial convolutional unit is used to calculate spatial spikes of the multi-channel temporal spike features to obtain spatio-temporal spike features; the pooling unit is used to average the spatio-temporal spike features in the temporal dimension to obtain a deep feature vector; the handcrafted feature calculation unit is used to calculate the handcrafted features of the cortical electroencephalogram signals, denoted as NAV feature vectors; the prediction unit is used to linearly project the deep feature vector and the NAV feature vector respectively to obtain feature vectors corresponding to a preset dimension; the projected feature vectors of the preset dimension are concatenated to obtain a joint feature; and motion intention prediction is performed based on the joint feature to achieve cortical electroencephalogram decoding. The related technical solutions are the same as those in Embodiment 1 and will not be elaborated here.

[0060] In summary, the present invention proposes a new spiking neural network architecture, which extracts multi-level temporal information through temporal convolution by channel, and then extracts information between different channels through a spatial convolutional layer. At the same time, the batch normalization layer makes the training more stable, and the spiking neurons can convert the data into a spike format, thereby reducing energy consumption. At the same time, the feature fusion method can further improve the classification accuracy of the network proposed by the present invention and other artificial neural networks.

[0061] Embodiment 3

[0062] This application also relates to an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0063] The electronic device may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The so-called processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor runs or executes the computer programs and / or modules stored in the memory, and calls the data stored in the memory to implement various functions of the electronic device.

[0064] The related technical solutions are the same as above and will not be elaborated here.

[0065] Embodiment 4

[0066] This application also relates to a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0067] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0068] The related technical solutions are the same as above and will not be elaborated here.

[0069] Embodiment 5

[0070] The embodiment of this application provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps of the method in the above embodiments of this application.

[0071] The related technical solutions are the same as above and will not be elaborated here.

[0072] It is easy for those skilled in the art to understand that the above are only the preferred embodiments of the present invention, and are not used to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cortical brain electrical decoding method, characterized in that Including: Obtain a spiking neural network, which includes a temporal convolutional unit, a spatial convolutional unit, and a pooling unit. The temporal convolutional unit is used to calculate temporal spikes for electrocorticogram signals by channel, obtaining multi-channel temporal spike features; the spatial convolutional unit is used to calculate spatial spikes for the multi-channel temporal spike features, obtaining spatio-temporal spike features; The pooling unit is used to average the spatio-temporal spike features in the time dimension, obtaining a deep feature vector; Calculate the handcrafted features of the electrocorticogram signals, denoted as NAV feature vectors; linearly project the deep feature vector and the NAV feature vectors respectively to obtain feature vectors corresponding to a preset dimension; splice the projected feature vectors of the preset dimension to obtain a joint feature; perform movement intention prediction based on the joint feature to achieve electrocorticogram decoding.

2. The cortical brain electrodecoding method according to claim 1, characterized in that, The temporal convolutional unit includes a temporal convolutional layer, a first normalization layer, and a first PLIF neuron; The temporal convolutional layer is used to perform one-dimensional convolutional operations on electrocorticogram signals by channel; The first normalization layer is used to calculate the mean and variance of each channel in the output of the temporal convolutional layer, and perform element-wise normalization processing on the feature vectors of each channel based on the mean and variance; perform element-wise linear transformation on the normalized feature vectors to obtain the normalized temporal features of the channel; The first PLIF neuron is used to perform spike calculations on the normalized temporal features of each channel, obtaining a spike feature vector composed of the number of spikes at each time step of the channel, and the temporal spike features are composed of the spike feature vectors of each channel.

3. A cortical brain decoding method according to claim 1, characterized in that, The spatial convolution branch includes a spatial convolutional layer, a second normalization layer, and a second PLIF neuron; The spatial convolutional layer is used to perform two-dimensional convolutional operations on the multi-channel temporal features; The second normalization layer is used to calculate the mean and variance of each channel in the output of the spatial convolutional layer, and perform element-wise normalization processing on the feature vectors of each channel based on the mean and variance; perform element-wise linear transformation on the normalized feature vectors to obtain the normalized spatial features of the channel; The second PLIF neuron is used to perform spike calculations on the normalized spatial features of each channel, obtaining a spike feature vector composed of the number of spikes at each time step of the channel, and the spatio-temporal spike features are composed of the spike feature vectors of each channel.

4. The cortical brain electrocode decoding method according to claim 1, characterized in that The implementation method of performing movement intention prediction based on the joint feature is: Perform a dot product of the joint feature and the weight matrix of the classifier, and obtain a prediction result through classification.

5. The corticocerebral decoding method according to claim 1, wherein, The determination method of the NAV feature vector is: Divide the data of each channel of the electrocorticogram signal in the time dimension to obtain multiple time periods, calculate the number of spikes in multiple time periods of the channel to obtain a spike number vector corresponding to the channel, construct a matrix composed of the spike number vectors of all channels, and flatten the matrix to obtain the NAV feature vector, where each time period contains multiple time steps.

6. The cortical brain electro decoding method according to claim 5, wherein, The multiple time periods are multiple non-overlapping and equal-length time periods.

7. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program is run by a processor, it controls the device where the storage medium is located to execute the steps of the method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instruction is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images