Neuron signal classification method based on data enhancement

Abstract

The invention discloses a data enhancement-based neuron signal classification method, which comprises the following steps of: (1) collecting neuron reaction signals, calculating the discharge rate of neurons in a specific time box by adopting a sliding window method, and averagely dividing the sequences into a plurality of fragments along the time dimension; (2) constructing a generator based on transformer and SNN, receiving a random vector, and outputting generated data after the random vector passes through an embedded layer, a multi-head self-attention layer and forward transmission; (3) constructing a discriminator, inputting the generated data and the real data into the discriminator, and judging whether the input data is the real data or not; (4) in the training process, using a binary cross entropy error as a loss function to update parameters of a generator and a discriminator; and (5) performing data enhancement by using data generated by the trained generator to improve the classification performance on the SNN classifier. According to the method, the problem of long sparse sequence generation is solved to amplify the neuron reaction signal, so that the classification effect after model training is improved.

Description

Technical Field

[0001] The invention belongs to the field of electroencephalogram signal analysis, and in particular relates to a neuron signal classification method based on data enhancement. Background Art

[0002] Neuronal responses in the ventral intraparietal area (VIP) suggest distinct spatiotemporal dynamics. Decoding heading direction using VIP neuron population responses is crucial for understanding primate self-motion. Neural networks (SNNs) are biologically plausible models known for their ability to efficiently process sequence data with rich spatiotemporal dynamics. SNNs facilitate spatiotemporal dynamics and energy efficiency through sequential communication, making them suitable for neuromorphic chips and energy conservation. SNN-based heading direction decoding models are expected to improve spike-based neural population response decoding with low power consumption by leveraging the rich spatiotemporal features and event-driven nature of SNNs.

[0003] Training SNNs requires diverse and stable data. However, acquiring VIP neuron responses in monkey brains is expensive and time-consuming, hindering the collection of sufficient data for effective SNN model training. To alleviate the need for data collection in deep neural network model training, generative models have emerged. For example, generative adversarial networks (GANs) consist of a generator and a discriminator. The generator produces data that matches the dimensions of real data, while the discriminator evaluates the authenticity of the generated data. Through adversarial training, the generator and discriminator strive to achieve equilibrium so that the generated data is more diverse than the real data, thereby improving model training.

[0004] Kotariya et al. introduced generative adversarial networks to tackle static image generation using temporal encoding. The model extended the refractory period of the integrating and firing neurons to ensure a single firing of each neuron.

[0005] Rosenfeld et al. proposed SpikeGAN, which contains a hybrid SNN-ANN architecture. The model aims to train SNN to match the distribution of signals rather than a single signal, supporting Bayesian learning of generator weights.

[0006] Yoon et al. introduced Time GAN, proposing a framework that combines traditional unsupervised GAN training methods with more controllable supervised learning methods.

[0007] Li et al. proposed TTS-GAN, in which both the generator and the discriminator are transformer encoders. They viewed time series sequences as images of height 1, introducing a new perspective for the representation of time series data in GANs.

[0008] While generative models for continuous time series data based on traditional artificial neural networks have been widely explored, using SNNs to generate synthetic CNN-based data to enhance neural decoding of population neuronal responses remains an unexplored area. Therefore, the goal of this invention is to enhance CNN-based data decoding by introducing a CNN-based GAN model, with a special focus on decoding neural signals from VIP neuron populations. Summary of the invention

[0009] The present invention provides a neuron signal classification method based on data enhancement, which amplifies the collected neuron response signals by solving the problem of long sparse sequence generation, thereby improving the classification effect after model training.

[0010] A neuron signal classification method based on data enhancement comprises the following steps:

[0011] (1) Collect neuronal response signals, calculate the firing rate of neurons in specific time bins using a sliding window method, and divide these sequences into multiple segments along the time dimension;

[0012] (2) Build a generator based on transformer and SNN; the generator receives a randomly generated vector I∈R D , the vector follows a Gaussian distribution, I i ~N(0,1), D represents the dimension of the input vector; after the received vector passes through the embedding layer, the multi-head self-attention layer and the forward pass, the generator outputs the generated data with the same dimension as the input data;

[0013] (3) constructing a discriminator, inputting the generated data obtained in step (2) and the real data obtained in step (1) into the discriminator, and the discriminator outputs binary data 0 and 1 to determine whether the input data is real data;

[0014] (4) During the training process, the binary cross entropy error is used as the loss function to update the parameters of the generator and the discriminator. The true label is set to a list filled with the number 1, indicating that the input data of the discriminator is true, and the false label of the generated data is 0;

[0015] (5) The data generated by the trained generator is used to perform data enhancement on the neuron signal to improve the neuron signal classification performance on the downstream SNN classifier.

[0016] In step (1), neuronal response signals are collected from the neurons in the ventral parietal area of ​​the brain.

[0017] In step (2), the generator used is a conditional generator, which aims to use the label to influence the generation result and generate data corresponding to the label; the label first passes through the embedding layer, and the embedded label is connected with the input vector to generate a new vector, which is used to combine the label and data to generate data with different labels.

[0018] In step (2), a time dimension segmentation is designed before each multi-head self-attention layer. After two segmentations, each time it is divided into a different number of segments, so that each part can independently obtain information through self-attention.

[0019] In step (2), in the multi-head self-attention layer, after performing temporal self-attention, a spatial self-attention module is added to extract spatial correlation; temporal self-attention calculates the correlation between time points, while spatial self-attention calculates the correlation between different cells in a segment.

[0020] In step (2), the neurons used in the generator's SNN network are LIF neurons, and the formula is as follows:

[0021]

[0022] S[t]=Θ(H[t]-V th )

[0023] V[t]=H[t](1-S[t])+V reset S[t]

[0024] where X[t] is the input current at time step t, Θ(v) is a discharge function that outputs 1 when v ≥ 0 and 0 otherwise; τ is the membrane time constant, V reset is the resting membrane potential, V th is the threshold potential; if the current potential is greater than V th , the neuron fires a spike.

[0025] In step (3), the discriminator is also built based on transformer and SNN. Compared with the generator, the discriminator reduces the extraction operation of spatial attention. The last layer uses a classification head composed of a linear layer and a sigmoid activation function to make a decoding decision to determine whether the input data is real data or generated data.

[0026] In step (4), the input of the generator is represented as d, the output of the generator is represented as G(d), and the judgment of the discriminator is represented as D(G(d)); the loss functions of the discriminator and the generator are as follows:

[0027] real_loss=BCELoss(D(real_data),real_label),

[0028] fake_loss=BCELoss(D(G(d)),fake_label),

[0029] dLoss=real_loss+fake_loss,

[0030] gLoss=BCELoss(D(G(d)),real_label).

[0031] Among them, real_loss represents the loss between the discriminator output and the real label when the input is real data, and fake_loss represents the loss between the discriminator output and the real label when the input is generated data; dLoss is selected to update the parameters of the discriminator, and it is expected that the discriminator will make correct judgments on real data and generated data. GLoss is selected to update the parameters of the generator, which will deceive the discriminator and judge that the data generated by the generator is real data.

[0032] In step (5), the SNN classifier used integrates the temporal attention mechanism and the spatial attention mechanism to select the key time points and cells respectively; during the selection process, the input sequence is divided into several segments, and the key time is searched in units of a segment rather than a single time point.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. We propose a peak-based unified decoding framework, including a temporal spike generative adversarial network (T-SGAN) for data augmentation and a recurrent SNN with spatial and temporal attention for robust decoding.

[0035] 2. The present invention introduces a time dimension segmentation in T-SGAN to shorten the time steps to be processed, and combines spatial self-attention to extract spatially relevant information. This design enhances the generation of long and sparse spike data.

[0036] 3. By incorporating the spatiotemporal attention mechanism into the recurrent SNN decoder, the ReSNN decoder can adaptively select the key time period and sensitive neurons for each head orientation data, thereby enhancing the decoding performance of the head orientation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the overall framework of a neuron signal classification method based on data enhancement of the present invention;

[0038] Figure 2 Schematic diagram of the spatiotemporal attention components and time dimension segmentation method used in the generator of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0040] The data set used in this embodiment is collected from the ventral parietal intra-region (VIP) of the monkey brain, and the neuronal response in this region has obvious spatiotemporal dynamics. Two data sets are used, data set 1 comes from 90 cells in the brain of a monkey, and data set 2 comes from 210 cells in the brains of three monkeys. During the sampling process, the monkey was stimulated from eight horizontal directions and the response of the VIP neurons was recorded. The collected data set contains a total of 160 trials, with 20 trials in each direction. In each trial, it contains 1200 time points to reflect the change of the signal. The shape of the data set of data set 1 is (160, 1200, 90), and the shape of data set 2 is (160, 1200, 210). The shape of the label is (160, 1), and 8 directions are represented by integers from 1 to 8.

[0041] like Figure 1 As shown, a neuron signal classification method based on data enhancement includes the following steps:

[0042] Step (1), obtaining the data to be processed, which is collected from the neurons in the ventral parietal region of the monkey's brain. The sliding window method is used to calculate the firing rate of neurons in a specific time bin. These sequences are evenly divided into multiple segments along the time dimension;

[0043] Specifically, before inputting the data into the generator, we set a sliding window of size 30ms. The unit of these windows is "ms" because the sample frequency of the data is 1000HZ. For each sequence, slide the window from the starting position to the end position with a step size of 30ms, add up all the points in the window, and calculate the frequency in the window. Therefore, for a sequence of length 1200ms, after processing, the length of the input sequence is 40.

[0044] Step (2), for the experiment, set the batch size to 20 and initialize a random vector with a dimension of 50. The generator receives a random vector I∈R D . This vector follows a Gaussian distribution, I i ~N(0,1). R and D represent the batch size and input potential dimension, which are 20 and 50 respectively. After the embedding layer, multi-head self-attention layer and forward pass, the generator outputs data with the same dimension as the input data. Assuming the dimension of the input data is (T, C), the dimension of the output data should also be (T, C);

[0045] Specifically, for the generator, its specific implementation is as follows:

[0046] 2-1. Structure: The structure of the generator is as follows Figure 1 As shown in (a). In the Embedding layer, the label first passes through the embedding layer, and the embedded label is connected to the input vector to generate a new vector to combine the label and data to generate data with different labels. After the data is segmented in the time series dimension, it is sent to the transformer to extract time and space information. The output data is then segmented again to extract time and space information again. Finally, the data is output through a forward transfer layer and a (1, 1) convolution layer. The output generated data has the same dimension as the input data, that is, the time series length of the input data after passing through the sliding window is 40, and the generated data is consistent with it.

[0047] 2-2. The neurons used in the network are LIF neurons. The basic unit in the LIF neuron model is the postsynaptic spike neuron of the postsynaptic neuron, which accumulates its membrane potential by receiving the synthetic current of the presynaptic neuron. When the discharge threshold is exceeded, the postsynaptic spike neuron sends a pulse. The calculation formula of the LIF neuron is described as follows:

[0048]

[0049] S[t]=Θ(H[t]-V th )

[0050] V[t]=H[t](1-S[t])+V reset S[t]

[0051] Where X[t] is the input current at time step t, and Θ(v) is a discharge function. When v ≥ 0, the discharge function outputs 1, otherwise it outputs 0. τ is the membrane time constant, V reset is the resting membrane potential, V th is the threshold potential. If the current potential is greater than V th , the neuron fires a spike.

[0052] In this embodiment, the resting membrane potential of the neuron is set to 0, the time constant is set to 2, and the threshold potential is set to 1. The membrane voltage of the neuron at the current moment decays by half and is transmitted to the next moment. After receiving the input at the next moment, if the membrane voltage is greater than 1, it is emitted and the potential is restored to 0, otherwise it is not emitted.

[0053] 2-3. A time dimension split is designed before each multi-head attention layer, such as Figure 2 As shown. For example, consider a matrix I∈R T×E, dividing it into several segments with several points (seg_num, points_num, E). Here, seg_num represents the number of segments, and point_num represents the number of time points within each segment. The product of these two values ​​is equal to T. Each part undergoes multi-head attention independently. After segmentation, the number of time points is reduced to the number of points, which significantly reduces the computational load compared to the original T. In order to solve the problem that points between different segments cannot be connected, secondary segmentation is introduced. The data is re-divided into different numbers of segments. The data dimension after segmentation is (seg_num', points_num', E). seg_num is not equal to seg_num', points_num is not equal to points_num', and the two cannot be multiples.

[0054] In this example, the sequence is initially divided into 5 segments, and then re-divided into 4 segments, that is, the data dimension after the first segmentation is (5, 8, E), and the data dimension after the second segmentation is (4, 10, E). After passing through the temporal self-attention layer, the dimension is restored to (40, E);

[0055] 2-4. In the multi-head self-attention layer, after performing temporal self-attention, a spatial self-attention module is added to extract spatial correlation. Temporal self-attention calculates the correlation between time points, while spatial attention calculates the correlation between different cells in a segment. In a multi-head attention block, we describe its operation as:

[0056] X=X+dropout(att(layernorm(X))),

[0057] X=transpose(X),

[0058] X=transpose(X+dropout(att(layernorm(X)))),

[0059] Among them, transpose refers to converting the time dimension and the space dimension. The att function is the same as the normal self-attention, which contains Query, Key and Value. This embodiment selects the dropout layer to avoid overfitting;

[0060] Step (3), input the generated data and real data obtained in step (2) into the discriminator. The structure of the discriminator is similar to that of the generator, both of which are based on transformer and SNN. The discriminator outputs binary data 0 and 1 to determine whether the input data is real data;

[0061] 3-1. The structure of the discriminator is as follows Figure 1As shown in (a). Compared with the generator, the discriminator reduces the extraction operation of spatial attention to reduce the strength of the discriminator, so that the generation ability of the generator and the judgment ability of the discriminator can match each other. The last layer uses a classification head composed of a linear layer and a sigmoid activation function to make a decoding decision to determine whether the input data is real data or generated data.

[0062] Step (4), using the binary cross entropy error as the loss function to update the parameters. In this embodiment, the true label is set to a list filled with the number 1, indicating that the input data of the discriminator is true, and the false label of the synthetic data is 0.

[0063] 4-1. Label the input of the generator as d. Then G(d) represents the output of the generator, and D(G(d)) represents the judgment of the discriminator. The loss functions of the discriminator and the generator are as follows:

[0064] real_loss=BCELoss(D(real_data),real_label),

[0065] fake_loss=BCELoss(D(G(d)),fake_label),

[0066] dLoss=real_loss+fake_loss,

[0067] gLoss=BCELoss(D(G(d)),real_label).

[0068] Among them, real_loss represents the loss between the discriminator output and the real label when the input is real data, and fake_loss represents the loss when the input is generated data. dLoss is selected to update the parameters of the discriminator, and it is expected that the discriminator will make correct judgments on real data and generated data. The generator uses gLoss, which will deceive the discriminator and judge that the data generated by the generator is real data.

[0069] In this example, a random vector is put into the generator to generate generated data with the same shape as the real data. Then the real data and the generated data are input into the discriminator, and the parameters of the discriminator are first updated using dLoss. Then the updated discriminator is asked to judge the generated data again to obtain gLoss. Finally, gLoss is used to update the parameters of the generator. Thus, a complete update is formed within one round.

[0070] Step (5) uses the data generated by the trained generator to perform data augmentation to improve the classification performance on the downstream classifier.

[0071] 5-1. The structure of the recurrent SNN is as follows Figure 1As shown in (b), the temporal attention mechanism and the spatial attention mechanism are integrated into the recurrent SNN classifier to select the key time points and cells that are important to the results. During the selection process, the input sequence is divided into several segments, and the key time is found in units of a segment, rather than a single time point.

[0072] In this example, the decoder is built by combining recurrent SNN and full connection. We define the output of the recurrent SNN as the hidden layer. We implement the hidden layer dimension size of 32, 64, 128 or 256, and the window size of 30, 60, 120 or 300.

[0073] 5-2. During the decoding process, in order to obtain a stable decoding accuracy, the entire dataset is divided into a training set and a test set using a ratio of 3:1. The training set is specifically used for training and generation within the T-SGAN framework, while the test set is specifically used for decoding testing. Subsequently, the generated data is combined with the training set to obtain a merged dataset, which is further divided into a training set and a validation set. This division follows a 2:1 ratio and uses a three-fold cross-validation method. After 50 cycles of training, validation, and testing, the average accuracy of these cycles is calculated and used as the final result.

[0074] Step (6), using the decoder and training step in step (5) to obtain the decoding accuracy.

[0075] In order to verify the effect of the present invention, this embodiment compares the decoding performance with and without generated data on two data sets to evaluate the performance of the generative model. As shown in Table 1, the performance improvement on the data set after data enhancement using GAN is very obvious, with a maximum improvement of 1.65% on data set 1 and a maximum improvement of 7% on data set 2.

[0076] Table 1 Performance comparison between without data augmentation and with data augmentation

[0077]

[0078] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for classifying neural signals based on data enhancement, characterized in that: The following steps are involved: (1) Collect neuronal response signals, calculate the firing rate of neurons in specific time bins using a sliding window method, and divide these sequences into multiple segments along the time dimension; (2) Build a generator based on transformer and SNN; the generator receives a randomly generated vector I∈R D , the vector follows a Gaussian distribution, I i ~N(0,1), D represents the dimension of the input vector; after the received vector passes through the embedding layer, the multi-head self-attention layer and the forward pass, the generator outputs the generated data with the same dimension as the input data; (3) constructing a discriminator, inputting the generated data obtained in step (2) and the real data obtained in step (1) into the discriminator, and the discriminator outputs binary data 0 and 1 to determine whether the input data is real data; (4) During the training process, the binary cross entropy error is used as the loss function to update the parameters of the generator and the discriminator. The true label is set to a list filled with the number 1, indicating that the input data of the discriminator is true, and the false label of the generated data is 0; (5) The data generated by the trained generator is used to perform data enhancement on the neuron signal to improve the neuron signal classification performance on the downstream SNN classifier.

2. The method for neuron signal classification based on data enhancement according to claim 1, characterized in that: In step (1), neuronal response signals are collected from the neurons in the ventral parietal area of ​​the brain.

3. The method for neuron signal classification based on data enhancement according to claim 1, characterized in that: In step (2), the generator used is a conditional generator, which aims to use the label to influence the generation result and generate data corresponding to the label; the label first passes through the embedding layer, and the embedded label is connected with the input vector to generate a new vector, which is used to combine the label and data to generate data with different labels.

4. The method for neuron signal classification based on data enhancement according to claim 1, characterized in that: In step (2), a time dimension segmentation is designed before each multi-head self-attention layer. After two segmentations, each time it is divided into a different number of segments, so that each part can independently obtain information through self-attention.

5. The method for neuron signal classification based on data enhancement according to claim 1, characterized in that: In step (2), in the multi-head self-attention layer, after performing temporal self-attention, a spatial self-attention module is added to extract spatial correlation; temporal self-attention calculates the correlation between time points, while spatial self-attention calculates the correlation between different cells in a segment.

6. The method for neuron signal classification based on data enhancement according to claim 1, characterized in that: In step (2), the neurons used in the generator's SNN network are LIF neurons, and the formula is as follows: S[t]=Θ(H[t]-V th ) V[t]=H[t](1-S[t])+V reset S[t] where X[t] is the input current at time step t, Θ(v) is a discharge function that outputs 1 when v ≥ 0 and 0 otherwise; τ is the membrane time constant, V reset is the resting membrane potential, V th is the threshold potential; if the current potential is greater than V th , the neuron fires a spike.

7. The method for neuron signal classification based on data enhancement according to claim 1, characterized in that: In step (3), the discriminator is also built based on transformer and SNN. Compared with the generator, the discriminator reduces the extraction operation of spatial attention. The last layer uses a classification head composed of a linear layer and a sigmoid activation function to make a decoding decision to determine whether the input data is real data or generated data.

8. The method for neuron signal classification based on data enhancement according to claim 1, characterized in that: In step (4), the input of the generator is represented as d, the output of the generator is represented as G(d), and the judgment of the discriminator is represented as D(G(d)); the loss functions of the discriminator and the generator are as follows: real_loss=BCELoss(D(real_data),real_label), fake_loss=BCELoss(D(G(d)),fake_label), dLoss=real_loss+fake_loss, gLoss=BCELoss(D(G(d)),real_label). Among them, real_loss represents the loss between the discriminator output and the real label when the input is real data, and fake_loss represents the loss between the discriminator output and the real label when the input is generated data; dLoss is selected to update the parameters of the discriminator, and it is expected that the discriminator will make correct judgments on real data and generated data. GLoss is selected to update the parameters of the generator, which will deceive the discriminator and judge that the data generated by the generator is real data.

9. The method for neuron signal classification based on data enhancement according to claim 1, characterized in that: In step (5), the temporal attention mechanism and the spatial attention mechanism are integrated into the SNN classifier used to select the key time points and cells respectively; During the selection process, the input sequence is divided into several segments, and the key time is found in units of a segment rather than a single time point.

Images