A neuromorphic data classification method based on multi-scale residual spiking neural network

Abstract

The present invention discloses a neuromorphic data classification method based on a multi-scale residual spike neural network, which solves the problems of difficulty in accurate characterization of neuromorphic data and insufficient feature mining, including: dividing a data set into a training set and a test set, performing time domain adaptive sampling to obtain discrete pulse blocks; performing tensor mapping to obtain tensor representations of the training set and the test set; constructing a G‑PLIF; sequentially splicing the G‑PLIF, Conv and tdBN to obtain a GCB module; GCB is used as a residual path F r (·), add a skip connection F s (·), forming a pulse residual block; embedding multiple layers of F on the basis of the pulse residual block r (·) and F s (·), construct a multi-scale pulse residual block MSR; replace the convolution part in the convolutional neural network with MSR to obtain a multi-scale residual pulse neural network model MSR-SNN; input the tensor of the training set into the MSR-SNN for training to obtain a neuromorphic data classification model; input the tensor of the test set into the neuromorphic data classification model to obtain the classification accuracy. The present invention expands the firing mode of neurons and achieves high-accuracy classification.

Description

Technical Field

[0001] The invention relates to a neuromorphic data classification method based on a multi-scale residual pulse neural network, and belongs to the technical field of target classification. Background Art

[0002] Neuromorphic sensors are new bio-inspired sensors that can quickly and asynchronously record changes in light intensity of pixels and output event data, so they are also called event cameras. Compared with traditional cameras, event cameras have unique advantages: high event resolution, high dynamic range, low power consumption, and greatly reduced motion blur. Even in challenging low-light, high-speed scenes and high dynamic range environments, they can provide high-quality visual information. The emergence of event cameras has provided a new development area for visual applications. Recently, event-based cameras have been proposed for target classification tasks.

[0003] Neuromorphic data object classification methods can be divided into three categories:

[0004] Based on the image paradigm, the asynchronous pulse stream is projected in the time domain or the frequency is accumulated into an image-like plane, which can quickly adapt to the existing target detection algorithm. However, these representation methods do not exploit the spatiotemporal characteristics of asynchronous pulse signals and are difficult to meet the needs of high-precision detection of some targets.

[0005] The 3D point cloud structure considers the similarity between event data and point cloud data in the 3D space dimension and recognizes the target from the perspective of the 3D graph. Compared with the standard image paradigm, the computational complexity is significantly reduced. Although graph-based methods have achieved good performance in target recognition tasks, they rely on constructing sparse event graphs on a certain number of events, collecting information contained in the event group at the expense of discarding the low latency characteristics of the event data.

[0006] The Spiking Neural Network (SNN), combined with the information processing method of biological neurons in the brain, is regarded as a third-generation neural network model with great potential and research space in processing asynchronous timing signals. Its working mode is highly consistent with neuromorphic data, and it can extract spatiotemporal mixed features in event data, realize low-power target recognition, and has achieved considerable results in public data sets. However, the difficulty of accurately representing neuromorphic data, the limited information capacity of SNN, and the simplified neuron model limit its expressive power and dynamic diversity, which makes it difficult to achieve high classification accuracy in neuromorphic data classification tasks. Summary of the invention

[0007] In view of the difficulty of accurate characterization of neuromorphic data, insufficient feature extraction and low classification accuracy, the present invention provides a neuromorphic data classification method based on a multi-scale residual spike neural network. This method designs a generalized parameter leakage integral and annealing neuron model G-PLIF model to increase the heterogeneity of neurons and improve the firing rate in the network; designs a new multi-scale pulse residual block MSR structure to efficiently mine the features in the pulse stream, and based on this, constructs a new framework multi-scale pulse residual network MSR-SNN for neuromorphic data, achieving better classification results.

[0008] In order to achieve the above object, the technical solution disclosed in the present invention is as follows:

[0009] The present invention discloses a neuromorphic data classification method based on a multi-scale residual spiking neural network, comprising:

[0010] Step 1: Divide the data set into a training set and a test set, perform time-domain adaptive sampling on the training set and the test set respectively, and obtain corresponding discrete pulse blocks;

[0011] Step 2: Perform tensor mapping on the discrete pulse blocks corresponding to the training set and the test set to obtain the tensor representation of the training set and the test set;

[0012] Step 3: Based on the variability of the membrane potential threshold of biological neurons, a generalized parameter leakage integral and annealing neuron model G-PLIF is constructed;

[0013] Step 4: Sequentially concatenate the generalization parameter leakage integral and annealing neuron model G-PLIF, the convolution model Conv, and the threshold-dependent batch normalization model tdBN to obtain the activated convolution batch processing module GCB; GCB is used as the residual path F r (·) and add a skip connection F s (·), forming the impulse residual block;

[0014] Step 5: Embed multiple layers of F based on the pulse residual block r (·) and F s (·) structure, constructing a multi-scale pulse residual block MSR;

[0015] Step 6: Use the multi-scale pulse residual block MSR to replace the convolution part in the convolutional neural network CNN to obtain the constructed multi-scale residual pulse neural network model MSR-SNN;

[0016] Step 7: Input the tensor of the training set into the multi-scale residual spike neural network model MSR-SNN, and perform network training through the time back propagation method and the substitute gradient method. After the training is completed, the neuromorphic data classification model is obtained;

[0017] Step 8: Input the tensor of the test set into the neuromorphic data classification model to obtain the classification accuracy of the test set.

[0018] In step 1, the method of dividing the training set and the test set includes:

[0019] In the public neuromorphic dataset, 80% of the data is randomly selected as the training set and the remaining 20% ​​is used as the test set.

[0020] In step 1, the method of performing time-domain adaptive sampling on the training set and the test set to obtain corresponding discrete pulse blocks includes:

[0021]

[0022] Among them, Δt i is the time window length of the i-th discrete pulse block; Δt i-1 is the time window length of the i-1th discrete pulse block; Δt α is the set time window value, β and Υ are the feedback control parameters of the time domain and the number of pulses, respectively, which can dynamically adjust the time domain window length of the pulse block; N i is the number of pulses in the i-th time-space window; N i-1 is the number of pulses in the i-1th space-time window; i is a digital index.

[0023] In the initial process, β=0, Υ=0 is set, and the number of pulses in the first pulse block is N0; the selection of β and Υ depends on the resolution of the data set, the complexity of the scene, and the speed of motion, and there are differences between different data sets;

[0024] Iterate the calculations in sequence to obtain the time window length Δt of all pulse blocks i ; All events in each time window form a pulse block S, and N i A discrete pulse block S.

[0025] In step 2, the method of performing tensor mapping on the discrete pulse blocks corresponding to the training set and the test set to obtain the tensor representation of the training set and the test set includes:

[0026]

[0027] Among them, E(x, y, t) is a tensor representation, which contains the spatiotemporal statistical characteristics of sparse pulse signals, so as to maximize the performance of downstream tasks; n is the event index number; N is the pulse block S(x n ,y n ,t n ) is the total number of events in the pulse block; S is the pulse block; S(x n ,y n ,t n) is a pulse block containing n events; x n is the horizontal coordinate position of the nth event, y n is the ordinate position of the nth event; t n is the time when the nth event occurs; k(xx n ,yy n ,tt n ) is the kernel function and a pulse neural network structure is adopted.

[0028] In step 3, based on the variability of the biological neuron membrane potential threshold, the method of constructing the generalized parameter leakage integral and annealing neuron model G-PLIF includes:

[0029] When the membrane potential voltage u(t) is less than the membrane potential threshold υ, the G-PLIF neuron is in a charging state, and the G-PLIF charging process is expressed as:

[0030]

[0031] Where τ = 1 + e -α is the membrane potential constant, α is a constant; u(t) represents the neuron membrane potential voltage at time t; u rest represents the resting potential; X(t) represents the input of the neuron at time t.

[0032] At this time, the functional relationship between the membrane potential threshold υ and the membrane potential voltage u(t) can be expressed as:

[0033]

[0034] Among them, ∞ It represents the stable value of υ when there is no external input and time tends to infinity. a and b are coefficients. The first term a(u(t)-u rest ) indicates that the membrane potential voltage u(t) has an effect on υ. If there is no effect, a=0 is set. The second term b(υ-υ ∞ ) represents the membrane potential threshold υ with υ ∞ As the benchmark, it decays at rate b;

[0035] When the membrane potential voltage u(t) is greater than or equal to the membrane potential threshold υ, the G-PLIF neuron fires a pulse. The G-PLIF pulse firing process is expressed as:

[0036]

[0037] Where s[t] is the pulse emission function. When u(t) is greater than or equal to υ, s[t] is 1 and a pulse is emitted. When u(t) is less than υ, s[t] is 0 and no pulse is emitted.

[0038] When the G-PLIF neuron pulse is released, the membrane potential threshold υ and the membrane potential voltage u(t) are reset. The G-PLIF voltage reset process is expressed as:

[0039] u(t)←u reset ,υ←max(υ,υ ∞ )when u(t)>υ.

[0040] In step 4, the generalization parameter leakage integral and annealing neuron model G-PLIF, the convolution model Conv and the threshold-dependent batch normalization model tdBN are sequentially spliced ​​to obtain the activation convolution batch processing module GCB as follows:

[0041] The output of the generalized parameter leakage integral and annealing neuron model G-PLIF is used as the input of the convolution model Conv, and the output of the convolution model Conv is used as the input of the threshold-dependent batch normalization model tdBN to obtain the activated convolution batch processing module GCB.

[0042] In step 4, GCB is used as the residual path F r (·) and add a skip connection F s (·), the method of forming the pulse residual block is:

[0043] X L =ADD(F r (X L-1 ),F s (X L-1 ));

[0044] Among them, X L is the output of the Lth layer residual block of the pulse residual block; F r (X L-1 ) is the input X L-1 The residual path of s (X L-1 ) is the input X L-1 The skip connection of X L-1 is the output of the L-1th layer residual block of the impulse residual block.

[0045] In step 5, multiple layers of F are embedded on the basis of the impulse residual block. r (·) and F s (·) structure, the method of constructing the multi-scale pulse residual block MSR includes:

[0046]

[0047] Among them, X * L is the output of the L-th layer residual block of the multi-scale pulse residual block MSR; the input X of the previous layer L-1Divided into three equal parts and Its residual path F r (·) is implemented through GCB, while the jump connection F s (·) uses a Conv and tdBN to maintain the identity mapping, and ADD(·) represents the addition operation of tensors.

[0048] In step 6, the constructed multi-scale residual spiking neural network model MSR-SNN includes: a residual module and a classifier module;

[0049] The residual module is used for feature extraction and consists of a series of sampling layers, each of which includes an MSR and a maximum pooling layer; it allows object features to be extracted across various dimensions and numbers of channels, thereby improving the robustness of the network;

[0050] The classifier module is used to perform linear transformation on the output of the residual module through the fully connected layer, which is then activated by neurons and finally classified through the softmax function.

[0051] In step 7, the tensor of the training set is input into the multi-scale residual spiking neural network model MSR-SNN, and the network is trained by time back propagation method and substitute gradient method. After the training is completed, the method of obtaining the neuromorphic data classification model includes:

[0052] The tensor of the training set is input into the multi-scale residual spiking neural network model. The time step, learning rate and number of iterations are determined based on the requirements of different data sets to perform network training. After the network training is completed, a neuromorphic data classification model is obtained.

[0053] The beneficial effects of the present invention are as follows:

[0054] The present invention constructs a generalized parameter leakage integral and annealing model G-PLIF, simulates the variable threshold of biological neurons, expands the firing mode of neurons, and enhances the expression ability of the spiking neural network SNN.

[0055] A multi-scale spike residual network model MSR-SNN was constructed. Through multi-scale feature fusion and residual learning, the features of event data were fully mined from different dimensions and numbers of channels, thereby improving the classification accuracy of neuromorphic data.

[0056] The embodiment of the present invention uses the neuromorphic datasets DVS-Gesture dataset, DVS-CIFAR10 dataset, ASL-DVS dataset and N-MNIST dataset as benchmarks to evaluate the method disclosed in the present invention. The method proposed in the present invention achieves the highest accuracy on all datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of time-domain adaptive sampling of neuromorphic data;

[0058] Figure 2 This is a schematic diagram of the MSR-SNN construction process. DETAILED DESCRIPTION

[0059] The embodiment of the present invention discloses a neuromorphic data classification method based on a multi-scale residual spiking neural network, which is based on the asynchronous, discrete and sparse characteristics of the existing bionic neural network - spiking neural network, and is perfectly compatible with neuromorphic data. Recent research is devoted to applying spiking neural networks to neuromorphic data classification tasks, but its development is greatly limited by the characterization capabilities of neuromorphic data and the efficient mining technology of time series information. To solve this problem, the present invention starts from two aspects:

[0060] First, a time-domain adaptive sampling strategy is proposed to dynamically adjust the time window size and improve the event data representation capability.

[0061] Second, a multi-scale pulse residual network MSR-SNN based on the generalized parameter leakage integral and annealing neuron model G-PLIF is designed to improve the pulse activity of neurons and enhance the feature mining ability of the network. G-PLIF expands its expression ability and dynamic diversity of neurons through variable membrane potential thresholds. The multi-scale pulse residual block structure introduces the concept of multi-scale pyramid, which can fully mine the feature information in event data. And the highest accuracy is achieved on the DVS-Gesture dataset, DVS-CIFAR10 dataset, ASL-DVS dataset and N-MNIST dataset. The method disclosed in the present invention includes:

[0062] Step 1: Divide the data set into a training set and a test set, perform time-domain adaptive sampling on the training set and the test set respectively, and obtain corresponding discrete pulse blocks to meet the event representation of different scenes and motion states.

[0063] Step 2: Perform tensor mapping on the discrete pulse blocks corresponding to the training set and the test set to obtain the tensor representation of the training set and the test set; these can be directly used as the input of the network.

[0064] Step 3: Based on the variability of the membrane potential threshold of biological neurons, a generalized parameter leakage integral and annealing neuron model G-PLIF is constructed.

[0065] Step 4: Sequentially concatenate the generalized parameter leakage integral and annealing neuron model G-PLIF, the convolution model Conv, and the batch processing model tdBN to obtain the activated convolution batch processing module GCB; GCB is used as the residual path F r (·) and add a skip connection F s (·), forming an impulse residual block.

[0066] Step 5: Embed multiple layers of F based on the pulse residual block r (·) and F s (·) structure, constructing a multi-scale pulse residual block MSR;

[0067] Step 6. Use the multi-scale pulse residual block MSR to replace the convolution part in the convolutional neural network CNN to obtain the constructed multi-scale residual pulse neural network model MSR-SNN.

[0068] Step 7: Input the tensor of the training set into the multi-scale residual spike neural network model MSR-SNN, and perform network training through time backpropagation method and substitute gradient method. After the training is completed, the neuromorphic data classification model is obtained.

[0069] Step 8: Input the tensor of the test set into the neuromorphic data classification model to obtain the classification accuracy of the test set. At the same time, compare it with the current most advanced method to evaluate the classification effect.

[0070] In step 1, the method of dividing the training set and the test set includes:

[0071] Among the publicly available neuromorphic datasets, DVS-Gesture dataset, DVS-CIFAR10 dataset, ASL-DVS dataset, and N-MNIST dataset, 80% of the data is randomly selected from each dataset as a training set, and the remaining 20% ​​is used as a test set.

[0072] In step 1, the training set and the test set are respectively subjected to time-domain adaptive sampling, and the method of obtaining the corresponding discrete pulse blocks can be expressed as a problem of solving a minimization optimization function, including:

[0073]

[0074] Among them, Δt i is the time window length of the i-th discrete pulse block; Δt i-1 is the time window length of the i-1th discrete pulse block; Δt α is the set time window value, β and Υ are the feedback control parameters of the time domain and the number of pulses, respectively, which can dynamically adjust the time domain window length of the pulse block; N i is the number of pulses in the i-th time-space window; N i-1 is the number of pulses in the i-1th space-time window; i is a digital index.

[0075] In the initial process, β=0, Υ=0 is set, and the number of pulses in the first pulse block is N0; the selection of β and Υ depends on the resolution of the data set, the complexity of the scene, and the speed of motion, and there are differences between different data sets;

[0076] Iterate the calculations in sequence to obtain the time window length Δt of all pulse blocks i ; All events in each time window form a pulse block S, and N i The schematic diagram of the adaptive sampling in the time domain of neuromorphic data is shown in Figure 1 As shown. Figure 1 It can be seen that the time domain windows of the pulse blocks are of unequal lengths, and the windows are allowed to overlap with each other to ensure the robustness of the generated pulse windows to the scene motion speed.

[0077] In step 2, the method of performing tensor mapping on the discrete pulse blocks corresponding to the training set and the test set to obtain the tensor representation of the training set and the test set includes:

[0078] Through a mapping, the pulse block S containing sparse pulse signals is mapped to a pulse tensor, so that the pulse data can be quickly and efficiently applied to neural network learning tasks. The mapping process is as follows:

[0079]

[0080] Among them, E(x, y, t) is a tensor representation, which contains the spatiotemporal statistical characteristics of sparse pulse signals, so as to maximize the performance of downstream tasks; n is the event index number; N is the pulse block S(x n ,y n ,t n ) is the total number of events in the pulse block; S is the pulse block; S(x n ,y n ,t n ) is a pulse block containing n events; x n is the horizontal coordinate position of the nth event, y n is the ordinate position of the nth event; t n is the time when the nth event occurs; k(xx n ,yy n ,tt n ) is the kernel function and a pulse neural network structure is adopted.

[0081] In step 3, based on the variability of the biological neuron membrane potential threshold, the method of constructing the generalized parameter leakage integral and annealing neuron model G-PLIF includes:

[0082] The basic computing unit of a spiking neural network is a spiking neuron. When simulating a large-scale spiking neural network, a simplified spiking neuron model needs to be introduced. The most common one is the LIF neuron. However, the neuron model generally uses a fixed threshold, and sufficient voltage needs to be accumulated to reach the threshold and trigger a pulse, thereby limiting the discharge rate of the spiking neuron. As the network layer deepens, neurons in the deep layer will be difficult to drive, resulting in a decrease in the convergence speed. In response to the above problems, in step three, based on the variable membrane potential threshold of biological neurons, a generalization parameter leaky integral and annealing neuron model (Generalization Parametric Leaky Integrate-and-Fire, G-PLIF) is constructed. The construction method includes:

[0083] When the membrane potential voltage u(t) is less than the membrane potential threshold υ, the G-PLIF neuron is in a charging state, and the G-PLIF charging process is expressed as:

[0084]

[0085] Where τ = 1 + e -α is the membrane potential constant, α is a constant; u(t) represents the neuron membrane potential voltage at time t; u rest represents the resting potential; X(t) represents the input of the neuron at time t;

[0086] For G-PLIF neurons, the membrane potential threshold υ is no longer a constant, but a variable related to the membrane potential voltage and the membrane potential threshold. The variable relationship between the membrane potential threshold υ and the membrane potential voltage u(t) is:

[0087]

[0088] Among them, ∞ It represents the stable value of υ when there is no external input and time tends to infinity. a and b are coefficients. The first term a(u(t)-u rest ) indicates that the membrane potential voltage u(t) has an effect on it. If it has no effect, a=0 is set; the second term b(υ-υ ∞ ) represents the membrane potential threshold υ with υ ∞ As the benchmark, it decays at rate b;

[0089] When the membrane potential voltage u(t) is greater than or equal to the membrane potential threshold υ, the G-PLIF neuron fires a pulse. The G-PLIF pulse firing process is expressed as:

[0090]

[0091] Where s[t] is the pulse firing function. When u(t) is greater than or equal to υ, s[t] is 1 and a pulse is fired. When u(t) is less than υ, s[t] is 0 and no pulse is fired. After the G-PLIF neuron pulse is fired, both the membrane potential threshold υ and the membrane potential voltage u(t) are reset. The G-PLIF voltage reset process is expressed as:

[0092] u(t)←u reset ,υ←max(υ,υ ∞ )when u(t)>υ.

[0093] G-PLIF neuron threshold is a variable, so that in different states, even if the membrane potential value of the neuron is the same, its behavior may not be the same. Its update rule will affect the variable, and then the neuron can also produce a variety of firing patterns.

[0094] In step 4, the generalization parameter leakage integral and annealing neuron model G-PLIF, the convolution model Conv and the threshold-dependent batch normalization model tdBN are sequentially spliced ​​to obtain the activation convolution batch processing module GCB as follows:

[0095] The output of the generalized parameter leakage integral and annealing neuron model G-PLIF is used as the input of the convolution model Conv, and the output of the convolution model Conv is used as the input of the threshold-dependent batch normalization model tdBN to obtain the activated convolution batch processing module GCB.

[0096] In step 4, GCB is used as the residual path F r (·) and add a skip connection F s (·), the method of forming the pulse residual block is:

[0097] X L =ADD(F r (X L-1 ),F s (X L-1 ));

[0098] Among them, X L is the output of the Lth layer residual block of the pulse residual block; F r (X L-1 ) is the input X L-1 The residual path of s (X L-1 ) is the input X L-1 The network structure uses only a single F r (·) and F s(·), only limited target feature information can be extracted. In order to fully mine the timing information in the pulse signal, the present invention introduces the concept of feature pyramid into the design of multi-scale pulse residual block MSR. In step 5, two layers of F are embedded on the basis of the pulse residual block. r (·) and F s (·) structure, the method of constructing the multi-scale pulse residual block MSR includes:

[0099]

[0100] Among them, X * L is the output of the L-th layer residual block of the multi-scale pulse residual block MSR; the input X of the previous layer L-1 Divided into three equal parts and Its residual path F r (·) is implemented through GCB, while the jump connection F s (·) uses a Conv and tdBN to maintain the identity mapping, and ADD(·) represents the addition operation of the tensor. Figure 2 As shown in , the MSR residual block is based on the pulse residual block and has a two-layer residual structure.

[0101] In step 6, the constructed multi-scale residual spiking neural network model MSR-SNN includes: a residual module and a classifier module;

[0102] The residual module is used for feature extraction and consists of a series of sampling layers, each of which includes an MSR and a maximum pooling layer; it allows object features to be extracted across various dimensions and numbers of channels, thereby improving the robustness of the network;

[0103] The classifier module is used to perform linear transformation on the output of the residual module through the fully connected layer, which is then activated by neurons and finally classified through the softmax function.

[0104] like Figure 2 As shown, the MSR-SNN network constructed by the present invention consists of a residual module and a classifier module. The neuromorphic data is converted into a pulse tensor through time domain adaptive sampling and is directly sent to the input layer of the network. The residual module is a feature extraction layer, which consists of several MSR and maximum pooling layers. The present invention uses the MSR residual operation to extract object features from different dimensions and channel numbers, which can enhance the robustness of the network. The classifier module realizes neuromorphic data classification through a fully connected layer, after neuron activation and softmax operation.

[0105] In step 7, the tensor of the training set is input into the multi-scale residual spiking neural network model MSR-SNN, and the network is trained by time back propagation method and substitute gradient method. After the training is completed, the method of obtaining the neuromorphic data classification model includes:

[0106] The tensor of the training set is input into the multi-scale residual spiking neural network model. The time step, learning rate and number of iterations are determined based on the requirements of different data sets to perform network training. After the network training is completed, a neuromorphic data classification model is obtained.

[0107] Experiment and result analysis:

[0108] Experimental setup:

[0109] The experimental software environment is based on the SpikingJelly deep learning framework of PyTorch1.1. Neuromorphic datasets include two categories, one is a dataset that records the motion transformation of objects on the computer screen, such as the N-MNIS dataset and the DVS-CIFAR10 dataset, and the other is a dataset obtained by recording real-world scenes, such as the DVS-Gesture dataset and the ASL-DVS dataset. All experimental training processes use the AdamW optimizer with a learning rate of 0.02. The initial values ​​of the membrane potential threshold and membrane time constant of the neuron are υ0=1, τ=2, respectively. The back-propagation training method using the alternative gradient method and the time back-propagation method is compared with the current advanced algorithms.

[0110] Compared with current advanced algorithms:

[0111] The experimental results are shown in Table 1, which compares the performance of the method proposed in the present invention with the current advanced methods, verifying the effectiveness of the model in processing neuromorphic data. The highest accuracy was achieved on the DVS-Gesture dataset, DVS-CIFAR10 dataset, ASL-DVS dataset, and N-MNIST dataset. The method proposed in the present invention achieved significant improvements in the recognition accuracy of the DVS-CIFAR10 dataset and the ASL-DVS dataset, which were increased by 6.69% and 2.47% respectively; and slightly improved on the DVS-Gesture dataset and the N-MNIST dataset, which were increased by 0.3% and 0.12% respectively.

[0112] Table 1

[0113]

[0114] These results demonstrate the effectiveness of the proposed MSR-SNN in processing various types of neuromorphic data.

[0115] Experimental conclusion:

[0116] In an embodiment of the present invention, it is considered to combine neuromorphic data with pulse neural networks to explore their potential in neuromorphic data target classification tasks. Taking into account the binary, discrete, and time-domain correlation of neuromorphic data, the present invention designs a multi-scale pulse residual block MSR, which fully mines the characteristics of event data through multi-scale feature fusion and residual learning. Experimental results show that the model disclosed in the present invention achieves the most advanced accuracy in event data and has a faster convergence speed. The MSR-SNN designed by the present invention will promote further exploration of SNN in neuromorphic data classification tasks, and the pulse structure makes it possible to deploy it on neuromorphic hardware in the future.

[0117] The experimental results achieved results in two aspects: improved classification accuracy and increased neuronal pulse firing frequency (i.e. enhanced representation ability).

[0118] The beneficial effects of the present invention are as follows:

[0119] The present invention constructs a generalized parameter leakage integral and annealing model G-PLIF, simulates the variable threshold of biological neurons, expands the firing mode of neurons, and enhances the expression ability of the spiking neural network SNN.

[0120] A multi-scale spike residual network model MSR-SNN was constructed. Through multi-scale feature fusion and residual learning, the features of event data were fully mined from different dimensions and numbers of channels, thereby improving the classification accuracy of neuromorphic data.

[0121] The embodiment of the present invention uses the neuromorphic datasets DVS-Gesture dataset, DVS-CIFAR10 dataset, ASL-DVS dataset and N-MNIST dataset as benchmarks to evaluate the method disclosed in the present invention. The method proposed in the present invention achieves the highest accuracy on all datasets.

Claims

1. A neuromorphic data classification method based on a multi-scale residual spiking neural network, characterized in that: include: Step 1: Divide the data set into a training set and a test set, perform time-domain adaptive sampling on the training set and the test set respectively, and obtain corresponding discrete pulse blocks; Step 2: Perform tensor mapping on the discrete pulse blocks corresponding to the training set and the test set to obtain the tensor representation of the training set and the test set; Step 3: Based on the variability of the membrane potential threshold of biological neurons, a generalized parameter leakage integral and annealing neuron model G-PLIF is constructed; The method of constructing the generalized parameter leakage integral and annealing neuron model G-PLIF includes: When the membrane potential voltage u(t) is less than the membrane potential threshold υ, the G-PLIF neuron is in a charging state, and the G-PLIF charging process is expressed as: Where τ = 1 + e -α is the membrane potential constant, α is a constant; u(t) represents the neuron membrane potential voltage at time t; u rest represents the resting potential; X(t) represents the input of the neuron at time t; At this time, the functional relationship between the membrane potential threshold υ and the membrane potential voltage u(t) can be expressed as: Among them, ∞ It represents the stable value of υ when there is no external input and time tends to infinity. a and b are coefficients. The first term a(u(t)-u rest ) indicates that the membrane potential voltage u(t) has an effect on υ. If there is no effect, a=0 is set. The second term b(υ-υ ∞ ) represents the membrane potential threshold υ with υ ∞ As the benchmark, it decays at rate b; When the membrane potential voltage u(t) is greater than or equal to the membrane potential threshold υ, the G-PLIF neuron fires a pulse. The G-PLIF pulse firing process is expressed as: Where s[t] is the pulse emission function. When u(t) is greater than or equal to υ, s[t] is 1 and a pulse is emitted. When u(t) is less than υ, s[t] is 0 and no pulse is emitted. When the G-PLIF neuron pulse is released, the membrane potential threshold υ and the membrane potential voltage u(t) are reset. The G-PLIF voltage reset process is expressed as: u(t)←u reset ,υ←max(υ,υ ∞ )when u(t)>υ; Step 4: Sequentially concatenate the generalization parameter leakage integral and annealing neuron model G-PLIF, the convolution model Conv, and the threshold-dependent batch normalization model tdBN to obtain the activated convolution batch processing module GCB; GCB is used as the residual path F r (·) and add a skip connection F s (·), forming the impulse residual block; Step 5: Embed multiple layers of F based on the pulse residual block r (·) and F s (·) structure, constructing a multi-scale pulse residual block MSR; Step 6: Use the multi-scale pulse residual block MSR to replace the convolution part in the convolutional neural network CNN to obtain the constructed multi-scale residual pulse neural network model MSR-SNN; Step 7: Input the tensor of the training set into the multi-scale residual spike neural network model MSR-SNN, and perform network training through the time back propagation method and the substitute gradient method. After the training is completed, the neuromorphic data classification model is obtained; Step 8: Input the tensor of the test set into the neuromorphic data classification model to obtain the classification accuracy of the test set.

2. The method according to claim 1, characterized in that In step 1, the method of dividing the training set and the test set includes: In the public neuromorphic dataset, 80% of the data is randomly selected as the training set and the remaining 20% ​​is used as the test set.

3. The method according to claim 1, characterized in that In step 1, the method of performing time-domain adaptive sampling on the training set and the test set to obtain corresponding discrete pulse blocks includes: Among them, Δt i is the time window length of the i-th discrete pulse block; Δt i-1 is the time window length of the i-1th discrete pulse block; Δt α is the set time window value, β and Υ are the feedback control parameters of the time domain and the number of pulses, respectively, which can dynamically adjust the time domain window length of the pulse block; N i is the number of pulses in the i-th spatiotemporal window; N i-1 is the number of pulses in the i-1th spatiotemporal window; i is a digital index; In the initial process, β=0, Υ=0 is set, and the number of pulses in the first pulse block is N0; the selection of β and Υ depends on the resolution of the data set, the complexity of the scene, and the speed of motion, and there are differences between different data sets; Iterate the calculations in sequence to obtain the time window length Δt of all pulse blocks i ; All events in each time window form a pulse block S, and N i A discrete pulse block S.

4. The method according to claim 1 or 3, characterized in that In step 2, the method of performing tensor mapping on the discrete pulse blocks corresponding to the training set and the test set to obtain the tensor representation of the training set and the test set includes: Among them, E(x, y, t) is a tensor representation, which contains the spatiotemporal statistical characteristics of sparse pulse signals, so as to maximize the performance of downstream tasks; n is the event index number; N is the pulse block S(x n ,y n ,t n ) is the total number of events in the pulse block; S is the pulse block; S(x n ,y n ,t n ) is a pulse block containing n events; x n is the horizontal coordinate position of the nth event, y n is the ordinate position of the nth event; t n is the time when the nth event occurs; k(xx n ,yy n ,tt n ) is the kernel function and a pulse neural network structure is adopted.

5. The method according to claim 1, characterized in that In step 4, the generalization parameter leakage integral and annealing neuron model G-PLIF, the convolution model Conv and the threshold-dependent batch normalization model tdBN are sequentially spliced ​​to obtain the activation convolution batch processing module GCB as follows: The output of the generalized parameter leakage integral and annealing neuron model G-PLIF is used as the input of the convolution model Conv, and the output of the convolution model Conv is used as the input of the threshold-dependent batch normalization model tdBN to obtain the activated convolution batch processing module GCB.

6. The method according to claim 1, characterized in that In step 4, GCB is used as the residual path F r (·) and add a skip connection F s (·), the method of forming the pulse residual block is: X L =ADD(F r (X L-1 ),F s (X L-1 )); Among them, X L is the output of the Lth layer residual block of the pulse residual block; F r (X L-1 ) is the input X L-1 The residual path of s (X L -1 ) is the input X L-1 The skip connection of X L-1 is the output of the L-1th layer residual block of the impulse residual block.

7. The method according to claim 1 or 6, characterized in that In step 5, multiple layers of F are embedded on the basis of the impulse residual block. r (·) and F s (·) structure, the method of constructing the multi-scale pulse residual block MSR includes: Among them, X * L is the output of the L-th layer residual block of the multi-scale pulse residual block MSR; the input X of the previous layer L-1 Divided into three equal parts and Its residual path F r (·) is implemented through GCB, while the jump connection F s (·) uses a Conv and tdBN to maintain the identity mapping, and ADD(·) represents the addition operation of tensors.

8. The method according to claim 1, characterized in that In step 6, the constructed multi-scale residual spiking neural network model MSR-SNN includes: a residual module and a classifier module; The residual module is used for feature extraction and consists of a series of sampling layers, each of which includes an MSR and a maximum pooling layer; it allows object features to be extracted across various dimensions and numbers of channels, thereby improving the robustness of the network; The classifier module is used to perform linear transformation on the output of the residual module through the fully connected layer, which is then activated by neurons and finally classified through the softmax function.

9. The method according to claim 1, characterized in that In step 7, the tensor of the training set is input into the multi-scale residual spiking neural network model MSR-SNN, and the network is trained by time back propagation method and substitute gradient method. After the training is completed, the method of obtaining the neuromorphic data classification model includes: The tensor of the training set is input into the multi-scale residual spiking neural network model. The time step, learning rate and number of iterations are determined based on the requirements of different data sets to perform network training. After the network training is completed, a neuromorphic data classification model is obtained.

Images