A neuron firing rate time series prediction method based on spatiotemporal feature fusion

Abstract

The application discloses a neuron firing rate timing prediction method based on space-time feature fusion, and comprises the following steps: (1) obtaining original neural pulse data from a neural activity public data set, an experimental acquisition system or a neural signal simulation platform; (2) constructing a space-time Transformer model combined with transfer entropy, wherein the model structure comprises an input representation layer, a time attention layer, a space attention layer, a transfer entropy calculation layer and an output layer; (3) constructing a joint loss function, and training the space-time Transformer model constructed in the step (2) by using the original neural pulse data obtained in the step (1); and (4) in an inference test stage, inputting neural pulse data to be predicted into the trained space-time Transformer model to obtain a neuron firing rate prediction result. The space-time Transformer model combined with transfer entropy is constructed, and accurate prediction of neuron group firing rate is realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and neural engineering, and in particular to a method for time-series prediction of neuron firing rate based on spatiotemporal feature fusion. Background Technology

[0002] With the development of neural recording technology, researchers are able to simultaneously acquire pulse firing data from hundreds or thousands of neurons using multi-electrode arrays (MEAs) or optogenetic imaging techniques. These neural signals reflect the complex dynamic characteristics of neural populations in both time and space.

[0003] For example, Chinese patent document CN118886458A discloses a signal processing method based on a spiking neural network, which includes: for any time step of any neuron, determining the membrane potential of the neuron at the time step based on the input pulse signal of the neuron within the preset time step; determining the output pulse signal of the neuron at each time step in parallel according to the membrane potential of each time step and the preset firing threshold, and executing the signal firing of the neuron at each time step in parallel according to the output pulse signal of each time step.

[0004] In neuroscience and brain-computer interface research, predicting the firing rate of neurons is of great significance: in brain-computer interfaces, firing rate prediction can be used to infer motor intentions or perceptual states in advance, enabling control feedback with lower latency; in neurodynamic modeling, predicting the activity patterns of neurons helps to reveal the causal coupling relationships between neural populations; in disease analysis and neuromodulation, predicting abnormal firing patterns can help identify early signals of neural circuit dysregulation.

[0005] Traditional neural signal analysis methods, such as linear regression, PSTH (impulse-triggered histogram), and state-space models, typically assume linear relationships or Gaussian distributions between signals, making it difficult to effectively capture nonlinear dependencies and long-range interactions between neurons. In recent years, deep learning methods, especially the Transformer architecture, have demonstrated powerful global modeling capabilities in time series forecasting. However, directly applying them to predict neural firing rates still faces the following challenges: (1) Lack of neurophysiological interpretability: Traditional self-attention mechanisms rely on dot product-based similarity measures, which only reflect the linear correlation between signals and ignore the nonlinear information transmission mechanisms that are widely present in the nervous system, making it difficult to reflect the causal relationship between neurons.

[0006] (2) Lack of selectivity and constraint in attention: Standard self-attention mechanism calculates attention weights globally, which may establish false connections between noisy channels and unrelated neurons in neural signals, resulting in a decrease in the robustness of feature extraction.

[0007] (3) Static masking mechanisms do not take into account the characteristics of neural signals: Existing masking methods statically mask samples based on fixed probabilities. However, the activity of neuronal populations has significant sparsity and time-varying characteristics, and the degree of firing activity varies significantly in different time segments. Static masking may weaken the model's ability to model high-information-density regions.

[0008] Therefore, there is an urgent need for a model that combines time dependence and spatial dependence and has physiological interpretability to accurately predict neuronal firing rates. Summary of the Invention

[0009] To address the problems existing in the spatiotemporal Transformer for predicting the firing rate of neurons in time series, this invention provides a method for predicting the firing rate of neurons in time series based on spatiotemporal feature fusion. By constructing a spatiotemporal Transformer model that incorporates transfer entropy, it achieves accurate prediction of the firing rate of the neuron population.

[0010] A method for time-series prediction of neuron firing rate based on spatiotemporal feature fusion includes the following steps: (1) Obtain raw neural impulse data from publicly available neural activity datasets, experimental acquisition systems, or neural signal simulation platforms; (2) Construct a spatiotemporal Transformer model that incorporates transfer entropy. The model structure includes an input representation layer, a temporal attention layer, a spatial attention layer, a transfer entropy calculation layer, and an output layer. In the input representation layer, the raw neural impulse counts are mapped to high-dimensional temporal embedding features. and spatial embedding features ; In the temporal attention layer, based on temporal embedding features Calculate the temporal attention weight matrix between different time steps to obtain the temporal latent features. ; In the spatial attention layer, based on spatial embedding features Calculate the spatial attention weight matrix between different neurons. ; In the transfer entropy calculation layer, the transfer entropy matrix between neurons is calculated. and the transfer entropy matrix Spatial attention weight matrix The enhanced spatial attention matrix is ​​obtained by fusion. ; Time potential features With the enhanced spatial attention matrix After performing spatiotemporal attention fusion, spatiotemporal latent features are obtained. Spatiotemporal latent features are expressed through the output layer. This is mapped to the predicted neuron firing rate. (3) Construct a joint loss function and use the original neural impulse data obtained in step (1) to train the spatiotemporal Transformer model constructed in step (2); (4) During the inference test phase, the neural impulse data to be predicted is input into the trained spatiotemporal Transformer model to obtain the predicted firing rate of neurons.

[0011] Temporal embedding features in the input representation layer and spatial embedding features They are respectively: ; ; in, This is a neural impulse counting matrix, where each element... Neuron In the Pulse count within a time window; This indicates an embedding operation. express Transpose of; and They are time and spatial location codes, respectively.

[0012] Preferably, a sine function is used to generate the time position code. and spatial location coding .

[0013] In the transfer entropy calculation layer, the transfer entropy matrix between neurons is calculated. The formula is as follows: ; ; In the formula, Represents neurons For neurons Information transmission intensity, neurons and The time series are respectively and ; Describes the joint probability distribution. Represents a known neuron and neurons Conditional probability in past states. This indicates that only the neurons are known. Conditional probability in past states. Indicates the number of neurons. It represents the set of real numbers.

[0014] The transfer entropy matrix Spatial attention weight matrix Integration, specifically: ; in, This is the adjustment coefficient.

[0015] Time potential features With the enhanced spatial attention matrix Spatiotemporal attention fusion is performed, specifically as follows: ; in, For time potential features The transpose of .

[0016] In step (3), the constructed joint loss function is: ; in, For joint losses, Loss due to mask reconstruction To compare learning loss, and These are the weighting coefficients.

[0017] Mask modeling loss An adaptive masking mechanism is employed to dynamically adjust the masking probability based on the local activation intensity of neurons. The calculation formula is as follows: ; in, The normalized local activation intensity This is the intensity scaling factor. Based on the basic mask probability, This is the maximum mask probability limit.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention explicitly incorporates the information flow relationship between neurons into spatial attention computation through transfer entropy, enhancing the model's ability to capture information transmission among neuronal groups; and through an adaptive masking mechanism, it enables the model to concentrate more learning resources on high-information regions, thereby improving the accuracy of neural signal prediction. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the spatiotemporal Transformer model structure incorporating transfer entropy in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the training process of the model in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0024] This invention presents a method for time-series prediction of neuron firing rate based on spatiotemporal feature fusion. The core of this method is the design of a spatiotemporal Transformer model incorporating transfer entropy, such as... Figure 1 As shown, the model structure includes an input representation layer, a temporal attention layer, a spatial attention layer, a transfer entropy calculation layer, and an output layer. The specific implementation steps are as follows: Step 1, data preprocessing.

[0025] Raw data is obtained from publicly available neural activity datasets and divided into training, validation, and test sets according to a preset ratio.

[0026] Specifically, the following two datasets were primarily used: The MC_Maze dataset was collected from the primary motor cortex and dorsal premotor cortex of monkeys. During the data collection process, the monkey first observed the target point on the screen, then waited for a period of time before manipulating a handle to move the cursor outward from the center point, avoiding the boundaries of the virtual maze to reach the target. The MC_Maze dataset contains 2869 samples, each covering 182 neurons, with a length of several hundred milliseconds. Analysis was performed using a 5-millisecond time step. The MC_RTT dataset was collected from the motor cortex region of monkeys. During the data collection process, the monkey moved the handle to the target point; once reached, a next random target immediately appeared. The MC_RTT dataset contains 1351 samples, each covering 130 neurons, with a length of 600 milliseconds. Analysis was performed using a 5-millisecond time step.

[0027] Both datasets are hosted in NWB (Neurodata without Borders) format on the DANDI (Distributed Archives for Neurophysiology Data Integration) platform.

[0028] Step 2: Construct a spatiotemporal Transformer model that incorporates transfer entropy.

[0029] 2-1, The original pulse count is mapped to a high-dimensional feature vector through the embedding layer.

[0030] We set up a temporal embedding layer and a spatial embedding layer to embed the input neural impulse activity from the two dimensions of time and space, respectively.

[0031] Specifically, let the neural impulse activity of a single trial be represented as a matrix. ,in For time steps, For the number of neurons, each element Neuron In the Pulse counts within a time window. The original pulse counts are mapped to high-dimensional feature vectors through an embedding layer. and Add time and location codes respectively. and spatial location coding To obtain temporal embedding features and spatial embedding features This preserves temporal and spatial order information, providing structured input for subsequent attention mechanisms.

[0032] Preferably, a sine function is used to generate the time position code. and spatial location coding .

[0033] 2-2, Calculate the attention weight matrix between different time steps in the temporal attention layer.

[0034] Temporal embedding features after encoding time location Perform attention calculations.

[0035] Specifically, the query, key, and value matrix is ​​first generated using learnable weights: ; ; ; in, .

[0036] Attention calculation is performed using a scaled dot product: ; in, This indicates the concatenated processing of multi-head attention and feedforward network (ReLU activation). Represents the dimension of the key vector and outputs the temporal latent features. .

[0037] 2-3. Calculate the attention weight matrix between different neurons in the spatial attention layer.

[0038] Spatial embedding features after spatial location encoding Perform attention calculations.

[0039] Specifically, the query, key, and value matrix is ​​first generated using learnable weights: ; ; ; in, .

[0040] Attention calculation is performed using a scaled dot product: .

[0041] It is worth noting that, in order to inject the statistical information of the samples as prior information into the model, this invention calculates the transfer entropy between each pair of neurons and adds the transfer entropy matrix to the spatial attention weight matrix. Specifically, let the neurons... and The time series are respectively and The method for calculating the transfer entropy is as follows: ; in: Describes the joint probability distribution. Represents conditional probability. This indicates that only the known neurons Conditional probability in past states.

[0042] By calculating the transfer entropy matrix for all neuron pairs, we obtain: ; Wherein, the first of the matrix Element-representing neurons For neurons The intensity of information transmission.

[0043] Finally, the transfer entropy matrix and the attention matrix are fused to obtain the enhanced attention matrix: ; in, This is an adjustment coefficient used to balance the contributions of transfer entropy information and original attention information.

[0044] The enhanced attention matrix obtained through the above steps It also includes attention weights based on similarity relationships and causal relationship information based on transfer entropy, which are used to improve the dynamic modeling effect of neuron populations.

[0045] 2-4, Spatiotemporal attention fusion.

[0046] The enhanced spatial attention matrix With time potential features Multiplying by the transpose: This is to achieve the fusion of spatiotemporal features and output spatiotemporal latent features. .

[0047] 2-5, The model output layer and training process are set as follows: The model output is the predicted neuron firing rate, derived from the fusion features obtained above. The linear transformation maps the distribution rate prediction matrix. Training employs a joint loss function, including mask modeling loss and contrastive learning loss. The training process is as follows: Figure 2 As shown.

[0048] Specifically, the goal of masking modeling loss is to reconstruct the true logarithmic discharge rate of the masked portion from the input data. The method involves, during training, working along the input matrix... A subset of the spatiotemporal axis is randomly selected for masking (its pulse count is set to zero or changed), and then the model is allowed to predict the discharge rate at the mask location. Mask reconstruction is achieved by minimizing the negative log-likelihood of Poisson. .

[0049] It is worth noting that, to further enhance the model's adaptive learning capability, this invention introduces an adaptive masking mechanism based on the original masking modeling. This mechanism aims to dynamically adjust the masking probability according to the local activity intensity of neurons. This mechanism ensures that the model concentrates more learning resources on reconstructing regions with higher information content and more active neuronal activity, thereby improving the model's learning efficiency and reconstruction accuracy. For the input spatiotemporal matrix... any element in (i.e., neurons) In the time window (pulse count), its local activation intensity Through the After normalization (such as Sigmoid or Softmax normalization), we get: This normalization maps the original pulse count to intensity values ​​in the range [0,1]. The probability of being selected as a mask Its local activation intensity They are positively correlated. Specifically, this probability can be calculated using the following formula: ; in, This is the intensity scaling factor, used to control the degree to which the activation intensity affects the probability; Based on the basic masking probability (i.e., the minimum masking probability), it ensures that even low-activity areas have a chance to be masked; The maximum mask probability is limited to avoid excessive information loss.

[0050] Therefore, in each iteration, the model is based on the calculated probability distribution matrix. For the input matrix Non-uniform sampling is employed. By allocating more mask weights to highly active, information-rich regions, the model is forced to learn the precise dependencies and underlying driving mechanisms of these key regions. This not only accelerates convergence but also improves the final model's ability to capture complex neurodynamics.

[0051] The purpose of contrastive learning loss is to further constrain the model's discharge rate output, making it robust to neural discharge variability. The method involves first generating data augmentation operations for each test sample, where the augmentation transformation is defined as an adjustment of the original input matrix... Random elements in the model are randomly discarded and their pulse counts are changed to obtain two corresponding random perturbation versions. Then, a contrastive loss is used to bring the model predictions of different augmented samples of the same experimental sample closer to each other, while maintaining distance from the model predictions of other experimental samples. The contrastive loss is defined as: ; in, Cosine similarity is used to measure the distance between the model predictions of two samples. This refers to the temperature parameter.

[0052] Final composite loss function This is constructed by weighted linear summation of two independent loss terms: ; in, and These are the weight coefficients of the two loss terms. This combination allows for precise control over the contribution of different training objectives to the overall model optimization through hyperparameters.

[0053] Step 3, Reasoning Test Phase.

[0054] The model uses forward prediction as the primary evaluation metric.

[0055] Specifically, each sample input to the model contains There are 10 time steps, of which the first part is 10 time steps. For the observation time step, the neural activity is known; the subsequent time steps... To predict time steps, the neural activity is unknown and is masked in the input samples. During inference, the model... The neural group activity within a time step is used to extract a spatiotemporal latent representation through temporal and spatial attention mechanisms. The output layer then maps this latent feature to a complete representation through a linear transformation. The prediction matrix for each time step. The output corresponding to each time step is the predicted firing rate of the neuron.

[0056] Preferably, the number of bits per pulse can be used to quantify how close the prediction result is to the actual pulse delivery: ; in, The firing rate of the predicted neurons, as indicated by the model. Relative to actual pulse count The resulting log-likelihood, The log-likelihood represents the average firing rate of neurons. This represents the total pulse count of the predicted neuron. The calculated result can be used as a score to evaluate the model's effectiveness in predicting neuron firing rate; a higher score indicates better performance.

[0057] To illustrate the effectiveness of the proposed spatiotemporal feature fusion-based neuron firing rate time-series prediction method, ablation experiments were conducted on the MC_Maze dataset. Three model settings were used: setting one was a complete spatiotemporal Transformer model combining transfer entropy and adaptive masking; setting two was a spatiotemporal Transformer model with only the transfer entropy module removed; setting three was a spatiotemporal Transformer model with only the adaptive masking mechanism removed; and setting four was a spatiotemporal Transformer model with both the transfer entropy module and adaptive masking removed. The experimental results are as follows: setting one achieved a forward prediction metric of 0.231 bps, which is an improvement of 0.03, 0.04, and 0.09 compared to settings two, three, and four, respectively. These results demonstrate that the method proposed in this invention can enhance the spatiotemporal Transformer, improving its ability to predict neuronal population activity.

[0058] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for temporal prediction of neuron firing rate based on spatiotemporal feature fusion, characterized in that, Includes the following steps: (1) Obtain raw neural impulse data from publicly available neural activity datasets, experimental acquisition systems, or neural signal simulation platforms; (2) Construct a spatiotemporal Transformer model that incorporates transfer entropy. The model structure includes an input representation layer, a temporal attention layer, a spatial attention layer, a transfer entropy calculation layer, and an output layer. In the input representation layer, the raw neural impulse counts are mapped to high-dimensional temporal embedding features. and spatial embedding features ; In the temporal attention layer, based on temporal embedding features Calculate the temporal attention weight matrix between different time steps to obtain the temporal latent features. ; In the spatial attention layer, based on spatial embedding features Calculate the spatial attention weight matrix between different neurons. ; In the transfer entropy calculation layer, the transfer entropy matrix between neurons is calculated. and the transfer entropy matrix Spatial attention weight matrix The enhanced spatial attention matrix is ​​obtained by fusion. ; Time potential features With the enhanced spatial attention matrix After performing spatiotemporal attention fusion, spatiotemporal latent features are obtained. Spatiotemporal latent features are expressed through the output layer. This is mapped to the predicted neuron firing rate. (3) Construct a joint loss function and use the original neural impulse data obtained in step (1) to train the spatiotemporal Transformer model constructed in step (2); (4) In brain-computer interfaces, during the inference test phase, the neural pulse data to be predicted is input into the trained spatiotemporal Transformer model to obtain the firing rate prediction results of neurons; based on the firing rate prediction, the motor intention or perception state is inferred in advance to achieve control feedback with lower latency.

2. The method for time-series prediction of neuron firing rate based on spatiotemporal feature fusion according to claim 1, characterized in that, Temporal embedding features in the input representation layer and spatial embedding features They are respectively: ； ； in, This is a neural impulse counting matrix, where each element... Neuron In the Pulse count within a time window; This indicates an embedding operation. express transpose, and They are time and spatial location codes, respectively.

3. The method for temporal prediction of neuron firing rate based on spatiotemporal feature fusion according to claim 2, characterized in that, Time position coding is generated using a sine function. and spatial location coding .

4. The method for time-series prediction of neuron firing rate based on spatiotemporal feature fusion according to claim 1, characterized in that, In the transfer entropy calculation layer, the transfer entropy matrix between neurons is calculated. The formula is as follows: ； ； In the formula, Represents neurons For neurons Information transmission intensity, neurons and The time series are respectively and ; Describes the joint probability distribution. Represents a known neuron and neurons Conditional probability in past states. This indicates that only the neurons are known. Conditional probability in past states. Indicates the number of neurons. It represents the set of real numbers.

5. The method for temporal prediction of neuron firing rate based on spatiotemporal feature fusion according to claim 1, characterized in that, The transfer entropy matrix Spatial attention weight matrix Integration, specifically: ； in, This is the adjustment coefficient.

6. The method for time-series prediction of neuron firing rate based on spatiotemporal feature fusion according to claim 1, characterized in that, Time potential features With the enhanced spatial attention matrix Spatiotemporal attention fusion is performed, specifically as follows: ； in, For time potential features The transpose of .

7. The method for temporal prediction of neuron firing rate based on spatiotemporal feature fusion according to claim 1, characterized in that, In step (3), the constructed joint loss function is: ； in, For joint losses, Loss due to mask reconstruction To compare learning loss, and These are the weighting coefficients.

8. The method for time-series prediction of neuron firing rate based on spatiotemporal feature fusion according to claim 7, characterized in that, Mask modeling loss An adaptive masking mechanism is employed to dynamically adjust the masking probability based on the local activation intensity of neurons. The calculation formula is as follows: ； in, The normalized local activation intensity This is the intensity scaling factor. Based on the basic mask probability, This is the maximum mask probability limit.

Images