A method for extracting features from brain neuronal spike potential sequences based on a dual self-attention mechanism
By combining dual self-attention and segmented attention mechanisms, a time series prediction model is constructed, which solves the problems of sparsity and variability of spike potential sequences, improves the feature extraction and time-series prediction capabilities of neuronal spike potential sequences, and achieves high prediction accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2024-07-23
- Publication Date
- 2026-07-31
AI Technical Summary
Existing self-attention architecture-based features extraction models for spike potential sequences fail to effectively adapt to the sparsity and high variability of spike potential sequences, resulting in insufficient features extraction and time series prediction capabilities.
A dual self-attention mechanism is adopted, combining a stabilization-destabilization framework and a segmented attention mechanism to construct a time series prediction model. By using time series self-attention and channel self-attention, the instability caused by data sparsity is mitigated, and the robustness of the model is improved.
It improves the robustness and prediction accuracy of the spike potential sequence time series prediction model, and can achieve good results in multi-neuron spike potential time series analysis tasks, with fast computation speed and high consistency prediction performance.
Smart Images
Figure CN118861628B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of time series modeling of brain electrical signal spike potential sequences, and in particular relates to a method for extracting features of brain neuronal spike potential sequences based on a dual self-attention mechanism. Background Technology
[0002] In the fields of neuroscience and biomedical engineering, understanding the complex dynamics and information encoding mechanisms of the nervous system is one of the core challenges. Action potentials, as markers of neuronal excitability, carry fundamental information about brain activity through their firing sequences, making them crucial for cognitive function, sensory processing, and even disease diagnosis. Especially with the increasing sophistication of multichannel recording technology, simultaneously monitoring the action potential activity of multiple neurons has become a key method for studying the encoding and decoding of neural information flow in the brain. Therefore, accurately predicting the subsequent firing times of multichannel action potential sequences is essential for elucidating neural encoding mechanisms, designing neural prostheses, and developing advanced brain-computer interface systems.
[0003] Currently, due to the high dimensionality, sparsity, and non-stationary nature of neural signals, modeling the firing time series of neuron clusters and predicting the firing dynamics of subsequent neurons is a challenging task. The mainstream neural network-based modeling methods currently include:
[0004] (1) Variational Auto-Encoder (VAE) series: This series of models uses VAE structures to analyze low-dimensional dynamic trajectories of neuron population firing from the perspective of latent variable analysis. For example, the LFADS (latent factor analysis via dynamical systems) algorithm uses two RNN network structures as encoder and generator, respectively. The encoder compresses and encodes the spike firing matrix into latent code, which is then used by the generator to generate subsequent spike firing rate curves.
[0005] (2) Sequence-to-sequence models based on Transformers are based on the introduction of a self-attention mechanism. Self-attention assigns different weights to each element in the sequence by calculating the similarity between the query, key, and value, and then sums them up to obtain the context representation of the current position. For example, the NDT (Neural Data Transformer) model adopts the original encoder-decoder architecture and performs self-attention calculation in time to complete the dynamic modeling of the spike potential firing of neural groups.
[0006] In general, due to the high variability of neuronal firing and the correlation of population activity, spike potential sequence data are highly non-stationary, sparse, high-dimensional discrete-time data, making dynamic modeling of neuronal population activity from spike potential firing sequences more challenging. Meanwhile, compared to LFADS series algorithms, network models based on self-attention mechanisms offer advantages in improved execution efficiency and parallelism, strong scalability in network construction, and, due to their inherent sequence-to-sequence structure, easier model building and training.
[0007] However, existing feature extraction models for spike potential sequences based on self-attention architecture only use the original encoding and decoding architecture and have not made adaptive adjustments to the sparse and highly variable data characteristics of spike potential sequences. Therefore, it is necessary to adjust and improve the original attention architecture to achieve better feature extraction and time series prediction capabilities. Summary of the Invention
[0008] This invention provides a method for extracting features of brain neuron spike potential sequences based on a dual self-attention mechanism, which can improve the robustness of spike potential sequence time series prediction models and achieve good prediction accuracy in multi-neuron spike potential time series analysis and prediction tasks.
[0009] A method for extracting features from brain neuronal spike potential sequences based on a dual self-attention mechanism includes the following steps:
[0010] (1) Brain-computer interface was used to obtain brain neuron spike potential data, including multi-channel spike firing timestamp sequence and stimulus signal timestamp sequence in the experiment. After preprocessing, the data was divided into training set, validation set and test set.
[0011] (2) Construct a time series prediction model with dual self-attention mechanism. The model includes a positional encoding layer and a temporal embedding layer, followed by an encoder composed of multiple dual attention encoding modules. The outer layer of the entire encoder has a stabilization-de-stabilization framework and a segmented attention framework. Finally, the model includes a linear time series prediction layer.
[0012] (3) Construct a cross-entropy loss function, use the training set and validation set to train and evaluate the constructed time series prediction model, optimize the network parameters by reducing the network loss, and finally make the network converge to obtain a model for front potential time series prediction.
[0013] (4) Test the trained model on the test set to predict the peak value distribution sequence of each channel in the next period of time, and evaluate the model using the number of bits per peak.
[0014] (5) Use the tested and evaluated model to predict the timing of the spike potential. The input is the historical multi-channel spike potential count matrix, and the output is the predicted multi-channel spike potential emission rate matrix.
[0015] This invention considers both temporal self-attention and channel self-attention. In addition, it proposes a segmented attention mechanism based on neural signals to alleviate the instability caused by data sparsity. It also proposes a stabilization-de-stabilization framework for extracting non-stationary information from neural signals to improve the robustness of the temporal prediction model for spike potential sequences.
[0016] In step (1), brain neuron spike potential data are obtained based on the maze task;
[0017] The maze task involves a center-to-outward delayed arrival task with obstacles. Subjects need to quickly control the cursor to move from the current position to the target position after the target appears and an instruction is given, without touching the obstacles during the movement. The device records neural activity from the anterior dorsal motor cortex and primary motor cortex of the monkeys. The recorded data is sorted by spike potential to obtain a multi-channel spike emission timestamp sequence. In addition, each experiment has several external stimuli, and the timestamp sequence of the stimulus signals in the experiment is recorded.
[0018] In step (1), the preprocessing involves converting the multi-channel spike potential time series into a spike counting matrix around the stimulus based on the experimental stimulus time; specifically as follows:
[0019] Action-oriented stimulus 0 As a baseline, where 0 represents an action instruction, its length is E, and it indicates that a total of E experiments were conducted, then... This indicates the moment when the action instruction stimulus occurs in the e-th experiment;
[0020] Using the multi-channel spike potential time series Spikes as the preprocessing object, its size is C, indicating that there are a total of C channels. c This represents the sequence of neuron spike firing timestamps recorded in the c-th channel; the number of spike firings in each channel is not necessarily equal. An offset interval [-bias_start, bias_stop] is specified for the stimulus. 0 Each stimulus timestamp And each channel of Spikes c :
[0021] In the time interval The Spikes are calculated using bin_size as the sampling interval. c The number of spike potentials in each bin within this time interval is calculated. Note that T = (bias_start + bias_stop) / bin_size is an integer representing the number of sampling time slots in a single experiment. The spike count matrix (s) around the stimulus is obtained by counting the records of each stimulus timestamp and acquisition channel. The shape of this matrix is (E, T, C).
[0022] In step (1), dividing the data into training, validation, and test sets is specifically as follows:
[0023] The preprocessed spike count matrix around the stimulus was used as the original dataset. In the first dimension (experimental dimension), it was randomly divided into training set, validation set and test set in a ratio of 6:2:2. At the same time, in the second dimension (time dimension), the training set, validation set and test set were divided into input part and prediction part in a ratio of 10:4.
[0024] In step (2), the stabilization-destabilization framework is used to stabilize the input. Normalizing along the time dimension T yields the following result. Simultaneously, the mean μ and variance σ of the input are retained for prediction; Se represents the peak count matrix of the e-th experiment. Let T be a set of real number matrices with T rows and C columns, where T represents the number of sampling time slots in a single experiment and C represents the number of channels.
[0025] The segmented attention framework is used for input... The original time series is segmented along the time dimension T according to a set window size, dividing it into multiple non-overlapping or slightly overlapping time segments. Each time segment represents a local time period in the time series, resulting in a matrix. Where N is the number of time segments and P is the length of a time segment.
[0026] In step (2), the temporal embedding layer contains a fully connected layer F1:P→d, and the positional encoding layer contains a learnable positional encoding parameter. matrix After passing through the temporal embedding layer and the positional encoding layer, we obtain:
[0027]
[0028] in, d is the hidden layer dimension.
[0029] In step (2), the encoder contains multiple dual-attention encoding modules. As input to the dual attention encoding module, sequence-based attention is first calculated, followed by channel-based attention; finally, the outputs of the two attentions are summed and normalized, and residual connections are added after randomly discarding a certain percentage of parameters.
[0030] After encoding by multiple dual-attention encoding modules, the final encoded output is obtained.
[0031]
[0032] The linear time-series prediction layer receives the encoder output. Then, the last two dimensions are merged first, followed by a linear layer.
[0033] F out :N×d→T pred
[0034] Finally, by adding the input mean μ and variance σ, we obtain the final prediction result:
[0035]
[0036] In step (3), channel-independent binary cross-entropy loss is used, and its calculation method is as follows:
[0037]
[0038] The loss function is calculated time-by-time. All are C-dimensional vectors; yt is the true value, indicating whether each channel contains a sharp value at time t; The predicted value represents the probability that each channel will contain a sharp point at time t, and its value is between 0 and 1. In actual calculation, each channel is calculated independently and then summed to obtain the loss value. After backpropagation of the model, the model parameters are updated.
[0039] In step (4), evaluating the model using bits per spike refers to comparing the predicted neuron firing rate matrix with the actual spike count matrix. The comparison strategy is as follows:
[0040] Calculate the negative log-likelihood of the predicted firing rate based on the Poisson distribution model with the actual peak count matrix:
[0041] NLL = -log(exp(-r) / n!*r) n )
[0042] Where n is the actual number of distributions, n! represents the factorial of the actual number of distributions, and r is the predicted distribution rate;
[0043] Calculate the negative log-likelihood of the predicted firing rate based on the average value of the Poisson distribution and the true peak count matrix:
[0044]
[0045] Where n is the actual number of distributions, r0 is the average predicted value, and the difference is divided by the total number of spikes and converted to bits to get the number of bits per spike. The higher this value, the more information the model carries per spike, meaning the model's prediction is more efficient and information-rich.
[0046]
[0047] Where, n sp This represents the total number of issues contained in the true spike count matrix.
[0048] Compared with the prior art, the present invention has the following beneficial effects:
[0049] (1) The attention model trained by this invention has a smaller training parameter scale, but the effect can still meet expectations;
[0050] (2) This invention proposes a stationarization-nonstationarization training framework that can learn the nonstationarity of neural firing;
[0051] (3) This invention proposes a segmented attention mechanism, which effectively integrates temporal information, reduces the impact of the sparsity of the original data, and enhances the robustness of the model;
[0052] (4) This invention proposes a dual attention mechanism based on segmented data, which takes into account both temporal attention and channel attention. Attached Figure Description
[0053] Figure 1 This is a flowchart of a method for extracting brain neuronal spike potential sequence features based on a dual self-attention mechanism according to the present invention.
[0054] Figure 2 This is a schematic diagram of the dual attention encoding module. Detailed Implementation
[0055] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.
[0056] like Figure 1 As shown in the figure, A represents the acquisition and preprocessing of the dataset. The monkey maze dataset was collected during a monkey maze experiment. The monkeys were pre-implanted with electrode arrays, enabling high-density neural signal recording. The experiment trained the monkeys to navigate a maze, recording neuronal action potential signals and various experimental stimulus times, such as target appearance and action instructions, while simultaneously recording the task. Subsequently, at the end of the experiment, the multi-channel neuronal action potential signals were processed by spike sorting to obtain a multi-channel spike potential time series. Subsequent dataset construction and partitioning will be based on this sequence data.
[0057] Next, windowing sampling and discretization matrix representation are performed on the multi-channel spike potential time series based on the experimental signal stimulation time. The sequence contains C channels, with the spike potential sequence recorded in channel c. And the event sequence e = {e1, e2, ..., e} E For example: n c This represents the number of spike potentials recorded in this channel, and E represents the number of events contained in the event sequence e. E can also be interpreted as the number of experiments. Traverse the event sequence, starting with e... i Reference front potential sequence Perform windowed sampling, i.e., specify the offset interval [-bias_start, bias_stop] and the sampling interval bin_size, denoted as T = (bias_start + bias_stop) / bin_size, ensuring that T is an integer, and calculate the values falling within each sampling interval [e i -bias_start,e i -bias_start+bin_size),..,[e i -bias_start+(T-1)*bin_size,e i The number of spike potentials in +bias_sop) is obtained based on event e. i and the front potential sequence s C Peak potential counting vector Therefore, the final experimental stimulation-surround spike potential firing counting matrix is S(E,T,C).
[0058] Finally, the dataset is divided according to S(E,T,C). The first division is the division of the training set, validation set, and test set. Here, the division is carried out according to the first dimension, which is the experiment dimension. At this time, each experiment represents a sample, and the ratio of the training set, validation set, and test set is 6:2:2. In the second dimension, which is the time dimension, the dataset is divided into the input part and the prediction part according to the ratio of 10:4.
[0059] Figure 1 B in the diagram represents the data flow for model training and optimization. The model consists of a positional encoding layer and an embedding layer, followed by an encoder composed of multiple dual attention encoding modules. The entire encoder is then wrapped by a stabilization-destabilization framework and a segmented attention framework. Finally, the model includes a linear temporal prediction layer.
[0060] Taking the e-th experimental sample as an example, for the input Stabilization over the time dimension yields... Simultaneously retain the mean μ and variance σ of the input; next, for The time dimension T is segmented according to the window patch_size to obtain the matrix. Where N is the number of segments and P is the segment length; then a fully connected layer F1:P→d is introduced as a temporal embedding layer, along with a learnable position encoding parameter. get:
[0061]
[0062] in d is the hidden layer dimension.
[0063] like Figure 2 The diagram shows the structure of the dual attention encoding module, where... As input to the dual attention encoding module, sequence-based attention is first calculated, followed by channel-based attention; specifically, the model obtains three matrices, Q, K, and V, through linear transformation.
[0064] Q = F q (X), K = F k (X),VB=F v (X)
[0065] Among them, matrix Fully connected layer F q ,F k ,F v :d→d, based on the multi-head attention calculation mechanism, Q, K, V are further segmented into:
[0066]
[0067] Next is the sequence self-attention calculation module.
[0068]
[0069] here, And the channel attention calculation module:
[0070]
[0071] here, Finally, the multiple heads are merged, the outputs of the two attention points are summed and normalized, and residual connections are added after randomly discarding a certain percentage of parameters.
[0072]
[0073] The above describes the process of attention calculation at one layer. After multiple layers of encoding, the final encoded output is obtained. The linear time-series prediction layer receives the encoded output. Then, the last two dimensions are merged first, followed by a linear layer.
[0074] F out :N×d→T pred
[0075] Finally, note that adding the input mean μ and variance σ will give you the final prediction result:
[0076]
[0077] A loss function is constructed, and the model is trained and evaluated using the training and validation sets. The network parameters are optimized by reducing the network loss, and the network eventually converges, resulting in a model that can be used for spike potential time series prediction. The converged model is then tested on the test set to predict the subsequent firing rate of each channel.
[0078] Based on the characteristics of neural firing and discrete sampling, more than 99% of the values in the firing count matrix of the spike potential sequence are 0 or 1, and the proportion of 0 is about 95% to 98%. This indicates the sparsity of neuron firing. On the other hand, it also shows that for a certain neuron (channel), it either fires (value 1) or does not fire (value 0) within a sampling window. Therefore, a binary cross-entropy loss function is adopted here. On the other hand, in order to keep the model output in the range of 0 and 1, a sigmoid activation function is used. Finally, when calculating the loss, the loss of each channel is calculated independently and then summed.
[0079] This invention transforms multi-channel spike potential time-series data into spike potential counting matrices through time-series windowing sampling and discretization, constructing a self-attention-based temporal prediction model to predict the subsequent firing rates of each neuron. Simultaneously, it adapts to the non-stationarity of neural impulse signals using a stationarization-destationarization mechanism, alleviates the problem of excessively sparse neural impulse signals through a segmented attention mechanism, and improves the model's robustness through a dual attention mechanism, achieving excellent prediction accuracy in multi-neuron spike potential time-series analysis and prediction tasks. The method of this invention has a fast computation speed, which can improve the speed of spike potential sequence feature extraction, maintains high consistency in predictions across different datasets, and helps to deepen the understanding of neuronal activity patterns and their encoding strategies in specific behavioral or cognitive tasks.
[0080] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for extracting features of brain neuron spike sequence based on a double self-attention mechanism, characterized in that, Includes the following steps: (1) Brain-computer interface was used to obtain brain neuron spike potential data, including multi-channel spike firing time stamp sequence and stimulus signal time stamp sequence in the experiment. After preprocessing, the data was divided into training set, validation set and test set. (2) Construct a time series prediction model with dual self-attention mechanism. The model includes a positional encoding layer and a temporal embedding layer, followed by an encoder composed of multiple dual attention encoding modules. The outer layer of the entire encoder has a stabilization-de-stabilization framework and a segmented attention framework. Finally, the model includes a linear time series prediction layer. The stabilization-destabilization framework is used for stabilizing the input. Normalizing along the time dimension T yields the following result. At the same time, the mean of the input is preserved. and variance Used for prediction; Indicates the first The peak counting matrix of this experiment. express OK The set of real matrices of columns, Indicates the number of channels; The segmented attention framework is used for processing the input The original time series is cut into multiple non-overlapping or slightly overlapping time segments in the time dimension T according to a set window size, each time segment representing a local period in the time series, obtaining a matrix where N is the number of time segments, and P is the length of the time segment. (3) Construct the cross-entropy loss function, use the training set and validation set to train and evaluate the constructed time series prediction model, optimize the network parameters by reducing the network loss, and finally make the network converge to obtain a model for front potential time series prediction. (4) Test the trained model on the test set to predict the peak value distribution sequence of each channel in the next period of time, and evaluate the model using the number of bits per peak. (5) Use the tested and evaluated model to predict the timing of the spike potential. The input is the historical multi-channel spike potential count matrix, and the output is the predicted multi-channel spike potential emission rate matrix.
2. The brain neuron spike sequence feature extraction method based on a dual self-attention mechanism according to claim 1, characterized in that, In step (1), brain neuron spike potential data are obtained based on the maze task; The maze task involves a center-to-outward delayed arrival task with obstacles. Subjects need to quickly control the cursor to move from the current position to the target position after the target appears and an instruction is given, without touching the obstacles during the movement. The device records neural activity from the anterior dorsal motor cortex and primary motor cortex of the monkeys. The recorded data is sorted by spike potential to obtain a multi-channel spike emission timestamp sequence. In addition, each experiment has several external stimuli, and the timestamp sequence of the stimulus signals in the experiment is recorded.
3. The brain neuron spike sequence feature extraction method based on a dual self-attention mechanism according to claim 1, characterized in that, In step (1), the preprocessing involves converting the multi-channel spike potential time series into a spike counting matrix around the stimulus based on the experimental stimulus time; specifically as follows: action cue is the baseline, where represents the action cue, which has a length of represents that a total of experiments were performed, then represents the time at which the action cue occurred in the experiment. Multi-channel spike potential time series This is a preprocessed object, its size is , indicating a total of One channel, Indicates the first Each channel records a sequence of neuron spike firing timestamps, and the number of spike firings in each channel is not necessarily equal; a specified offset interval is provided. ,right Each stimulus timestamp as well as Each channel : In time interval The number of spikes falling into each bin_size interval is calculated The number of spikes falling into each bin_size interval is calculated The number of spikes falling into each 4. The brain neuron spike sequence feature extraction method based on a dual self-attention mechanism according to claim 1, characterized in that, In step (1), dividing the data into training, validation, and test sets is specifically as follows: The pre-processed stimulation surrounding spike count matrix is taken as the original data set, which is randomly divided into a training set, a validation set and a test set in the first dimension according to a ratio of , and at the same time, the training set, the validation set and the test set are divided into an input part and a prediction part in the second dimension according to a ratio of .
5. The method for extracting brain neuronal spike potential sequence features based on a dual self-attention mechanism according to claim 1, characterized in that, In step (2), the temporal embedding layer includes a fully connected layer. The positional coding layer contains a learnable positional coding parameter. ,matrix After passing through the temporal embedding layer and the positional encoding layer, we obtain: in, d is the dimension of the hidden layer.
6. The method for extracting brain neuronal spike potential sequence features based on a dual self-attention mechanism according to claim 5, characterized in that, In step (2), the encoder contains multiple dual-attention encoding modules. As input to the dual attention encoding module, sequence-based attention is first calculated, followed by channel-based attention; finally, the outputs of the two attentions are summed and normalized, and residual connections are added after randomly discarding a certain percentage of parameters. After encoding by multiple dual-attention encoding modules, the final encoded output is obtained. ; The linear time-series prediction layer receives the encoder output. Then, the last two dimensions are merged first, followed by a linear layer. Finally, add the input mean. and variance The final prediction result is obtained: 。 7. The method for extracting brain neuronal spike potential sequence features based on a dual self-attention mechanism according to claim 1, characterized in that, In step (3), channel-independent binary cross-entropy loss is used, and its calculation method is as follows: The loss function is calculated time-by-time. All are C-dimensional vectors; It is the actual value, indicating whether each channel contains a peak value at time t; The predicted value represents the probability that each channel will contain a sharp point at time t, and its value is between 0 and 1. In actual calculation, each channel is calculated independently and then summed to obtain the loss value. After backpropagation of the model, the model parameters are updated.
8. The method for extracting brain neuronal spike potential sequence features based on a dual self-attention mechanism according to claim 1, characterized in that, In step (4), evaluating the model using bits per spike refers to comparing the predicted neuron firing rate matrix with the actual spike count matrix. The comparison strategy is as follows: Calculate the negative log-likelihood of the predicted firing rate based on the Poisson distribution model with the actual peak count matrix: Where n is the actual number of distributions, The factorial represents the actual number of distributions, and r is the predicted distribution rate; Calculate the negative log-likelihood of the predicted firing rate based on the average value of the Poisson distribution and the true peak count matrix: Where n is the actual number of distributions, The average prediction is calculated, and the difference is divided by the total number of spikes and converted to bits to obtain the number of bits per spike. The higher this value, the more information the model carries per spike, meaning the model's prediction is more efficient and information-rich. in, This represents the total number of issues contained in the true spike count matrix.