A Training Method for Spiking Neural Networks Based on Adaptive Threshold Integer-Activated Neurons
Through the LIF neuron module activated by adaptive threshold integers, the traditional pulsed neural network training problem is solved, efficient feature extraction and operational reduction is achieved, and the application scope of pulsed neural network is expanded.
Patent Information
- Application Number
- CN202510470143.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-15
- Publication Date
- 2025-08-01
- Estimated Expiration
- 2045-04-15
AI Technical Summary
Traditional pulsed neural networks are difficult to directly apply backpropagation optimization algorithm training, resulting in high energy consumption, large delay and inability to capture spatiotemporal information. The existing alternative gradient direct training methods have deteriorated performance in simple tasks, making it difficult to expand their application range.
LIF neurons activated based on adaptive threshold integers are used to generate pulse sequences through time encoding, and downsample and feature extraction are used to use adaptive threshold integer-activated LIF neuron modules to perform downsampling and feature extraction, and combined with processing modules for different visual tasks, a multi-level feature representation is formed.
It improves the training performance of pulsed neural networks, reduces computing volume and power consumption, expands its application range, makes it easier to learn general features and retain detailed information.
Smart Images
Figure CN119990198B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of spiking neural network training, and specifically relates to a method for training a spiking neural network based on an adaptive threshold integer activation neuron. Background Art
[0002] In recent years, the field of artificial intelligence has achieved great achievements comparable to or even exceeding human performance in many fields such as computer vision, speech recognition, and natural language processing (NLP). As the third generation of neural networks, spiking neural networks (SNNs) imitate biological neurons and use sparse, event-driven spike activations for inter-neuron communication, which can greatly reduce the energy consumption burden of autonomous driving calculations and have broad prospects. Traditional spiking neural networks are limited by complex neuron dynamics and non-differentiable characteristics, making it difficult to directly apply the backpropagation optimization algorithm for direct training.
[0003] The mainstream method of converting artificial neural networks (ANNs) into spiking neural networks often has extremely high time steps to achieve performance similar to that of ANNs, resulting in increased energy consumption and latency, and being unable to capture spatio-temporal information in data, so it has gradually been abandoned. The emerging alternative gradient direct training methods have a serious performance degradation at shorter time steps and are currently only mostly applied to simple image classification tasks. Using larger time steps leads to excessive memory overhead and makes the model difficult to train. To overcome the shortcomings of directly trained SNNs using alternative gradients and expand the application scope of SNNs, many studies have been explored. For example, EMS-YOLO has become the first model to use directly trained SNNs for object detection, and Meta-SpikeFormer can first handle object detection in a pre-training and fine-tuning manner. However, there is still a gap between these studies and the performance of advanced artificial neural networks. Summary of the Invention
[0004] The purpose of the present invention is to provide a method for training a spiking neural network based on an adaptive threshold integer activation neuron to solve the problems raised in the above background art.
[0005] To solve the above technical problems, the present invention provides the following technical solution: A method for training a spiking neural network based on an adaptive threshold integer activation neuron, comprising the following steps:
[0006] S1: Input a neuromorphic data stream or a static image, generate a spike sequence through time encoding, and use it as the input of the model;
[0007] Preferably, the network inputs neuromorphic data streams (such as event camera outputs) or images in time steps, encodes them over time, and outputs a four-dimensional tensor of [T, C, H, W] to obtain a pulse sequence. Here, T represents the number of discrete time steps, C represents the number of channels (such as polarity, grayscale), H and W represent the image height and width respectively, and then it is normalized to obtain the input of the backbone network.
[0008] S2: Use an SNN module with LIF neurons based on adaptive threshold integer activation in the backbone network to perform downsampling and extract multi-scale features;
[0009] Preferably, the backbone network using an SNN module with LIF neurons based on adaptive threshold integer activation successively performs downsampling and feature extraction, capturing edge, texture, and semantic features layer by layer to form a multi-level feature representation. The SNN module is expressed as:
[0010] SepConv(U) = Conv pw (Conv dw (SN(Conv pw (SN(U))))))
[0011] ChannelConv(U′) = Conv(SN(Conv(SN(U′))))
[0012] U′ = U + SepConv(U)
[0013] U″ = U′ + ChannelConv(U′)
[0014] Among them, Conv pw is a pointwise convolution, which is used to perform convolution on the input feature map in the channel dimension using a convolution kernel of size 1×1; Conv dw is a depthwise convolution, which is used to perform spatial convolution on each input channel separately; Conv is a standard convolution, which is used in other branches or modules. SN is a spiking neuron layer, which is used for LIF neurons with adaptive threshold integer activation to map the continuous convolution output to discrete pulse activations; SepConv is a separable convolution, which is used to combine Conv pw and Conv dw . ChannelConv is an optional channel mixer, which is used to further enhance the information fusion between channels.
[0015] Preferably, the SN layer uses LIF neurons with adaptive threshold integer activation as the basic unit. The LIF neurons are LIF neurons with adaptive threshold integer activation, and the membrane potential update formula of the LIF neurons is:
[0016] V t+1 = τVt (1 - S t ) + WX t
[0017] S t = Θ(V t - V th )
[0018] where V t is the membrane potential at time step t, S t is the pulse signal at time step t, W is the weight matrix connecting the input X t and the membrane potential, τ is the leakage factor of the neuron, V th is the threshold, Θ is the Heaviside step function, and when the cumulative membrane potential is greater than the threshold, a step signal is generated.
[0019] Preferably, let the pulse threshold of the neuron be inversely proportional to the rate of change of the membrane potential, so that the more important and active neurons have lower thresholds. Its continuous - time form is:
[0020] V th = a+(1 - a)e -ψ / λ
[0021]
[0022] where a is a trainable parameter, ψ is the rate of change of the membrane potential, λ is a hyper - parameter set to 0.1 during training, Δt is the time difference between two consecutive changes in the membrane potential, and ΔV m is the difference in the membrane potential between two consecutive changes. Based on the characteristics of exponential decay, an increase in the membrane potential will lead to a decrease in the threshold, achieving a sensitive response to the input intensity; further modify the pulse signal generation function and the membrane potential update function as follows:
[0023]
[0024] V t+1 = τ(V t - S t V th ) + WX t
[0025] Based on the above - modified pulse, integer - valued activation can be achieved, and more detailed information can be retained through integer - valued activation during the training phase.
[0026] S3: Divide the visual task into image recognition, image detection, and image segmentation, and connect them to different processing modules respectively.
[0027] Preferably, for the image classification task, a classification head is connected to map the spatial dimension of the end feature map to the classification category for image classification. For the image detection task, the neck module is used to perform upsampling to align the feature maps in terms of spatial dimensions. Then, the high-level features and low-level features are concatenated to combine details and semantics. Finally, a detection head is connected to output the final detection boxes and categories. For the image segmentation task, the feature pyramid module is used to capture context information, combine high and low features, and perform mask embedding. Finally, a segmentation head is connected to generate the final segmentation mask.
[0028] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: The backbone network of the present invention adopts an SNN module based on LIF neurons with adaptive threshold integer activation, which performs downsampling and feature extraction in sequence, captures edge, texture, and semantic features layer by layer, forms a multi-level feature representation, and by introducing an adaptive threshold, the threshold of more important and active neurons is lower, making it not only easier to trigger pulses, but also reducing the computational amount while making the network easier to learn general features and avoiding overfitting. And through integer activation, the membrane potential accumulation amount is converted into an integer number of pulses, which is easier to train while retaining rich details, and also further reduces power consumption by extending the time step without increasing multiplication operations during inference, improving the performance of the directly trained network as a whole and expanding the application scope of the spiking neural network. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, but do not constitute a limitation to the present invention.
[0030] In the drawings:
[0031] Figure 1 is a schematic diagram of the steps of a training method for a spiking neural network based on LIF neurons with adaptive threshold integer activation provided by an embodiment of the present invention;
[0032] Figure 2 is a schematic diagram of the structure of an SNN module based on LIF neurons with adaptive threshold integer activation provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0033] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
[0034] The embodiments of the present invention are combined with Figure 1 and Figure 2, specifically providing the following technical solution: A training method for a spiking neural network based on an adaptive threshold integer activation neuron, combined with Figure 1 as shown, the method includes the following steps:
[0035] S1: Input the neuromorphic data stream or static image, generate a pulse sequence through time encoding, and use it as the input of the model;
[0036] Exemplarily, the network inputs the neuromorphic data stream (such as the output of an event camera) or image at each time step, outputs a four-dimensional tensor of [T, C, H, W] through time encoding to obtain a pulse sequence, where T represents the discrete number of time steps, C represents the number of channels (such as polarity, grayscale), and H and W respectively represent the height and width of the image, and then obtains the input of the backbone network after normalization.
[0037] S2: In the backbone network, use the SNN module based on the LIF neuron with adaptive threshold integer activation for downsampling and extracting multi-scale features;
[0038] Exemplarily, for the backbone network using the SNN module based on the LIF neuron with adaptive threshold integer activation, perform downsampling and feature extraction in sequence, capture edge, texture, and semantic features layer by layer, and form a multi-level feature representation. The SNN module is expressed as:
[0039] SepConv(U) = Conv pw (Conv dw (SN(Conv pw (SN(U))))))
[0040] ChannelConv(U′) = Conv(SN(Conv(SN(U′))))
[0041] U′ = U + SepConv(U)
[0042] U″ = U′ + ChannelConv(U′)
[0043] Specifically combined with Figure 2 as shown, where the explanations of each module are as follows:
[0044] Conv pw : Pointwise convolution, using a 1×1 convolution kernel to perform convolution on the input feature map in the channel dimension;
[0045] Conv dw : Depthwise convolution, performing spatial convolution on each channel of the input respectively;
[0046] Conv: VanillaConv, which can be used in other branches or modules;
[0047] SN: Spiking Neuron Layer. In the present invention, the LIF neuron with adaptive threshold integer activation is adopted to map the continuous convolution output into discrete pulse activation;
[0048] BN: Batch Normalization Layer. In the present invention, the Threshold-dependent Batch Normalization is adopted. It is a batch normalization method specially designed for spiking neural networks. On the basis of considering the spatial dimension, the feature statistics and threshold dependence in the time dimension are introduced to ensure the stability and efficiency of training;
[0049] SepConv: used to combine Conv pw and Conv dw to achieve efficient separable convolution, which not only maintains the feature extraction performance but also significantly reduces the number of parameters and the amount of computation;
[0050] ChannelConv: an optional channel mixer for further enhancing the information fusion between channels.
[0051] Exemplarily, the SN layer takes the LIF neuron with adaptive threshold integer activation as the basic unit. In the network, the LIF neuron plays the role of the activation function of the traditional neural network. Here, the LIF neuron with adaptive threshold integer activation is used, and the specific formula concept derivation is as follows:
[0052] The membrane potential update formula of the LIF neuron is:
[0053] V t+1 = τV t (1 - S t ) + WX t
[0054] S t = Θ(V t - V th )
[0055] Among them, V t is the membrane potential at time step t, S t is the pulse signal at time step t, W is the weight matrix connecting the input X t and the membrane potential, τ is the leakage factor of the neuron, V th is the threshold, and Θ is the Heaviside step function. When the cumulative membrane potential is greater than the threshold, a step signal is generated;
[0056] Furthermore, in the present invention, it is intended that the more important and active neurons have lower thresholds, so that pulses are more easily triggered. By making the pulse threshold of neurons inversely proportional to the membrane potential change rate, its continuous-time form can be described as:
[0057] V th = a + (1 - a)e -ψ / λ
[0058]
[0059] where a is a trainable parameter, ψ is the membrane potential change rate, λ is a hyperparameter, set to 0.1 during training, Δt is the time difference between two consecutive membrane potential changes, and ΔV m is the difference between two consecutive membrane potential changes. Due to the exponential decay characteristic, an increase in the membrane potential will bring about a rapid and smooth threshold reduction, achieving a sensitive response to the input intensity. At the same time, the upper and lower limits of the threshold are also restricted to prevent the threshold from getting out of control. That is, the pulse signal generation function and the membrane potential update function are modified as follows:
[0060] S t = [V t / V th
[0061] V t+1 = τ(V t - S t V th ) + WX t
[0062] Thus, the pulse can be activated by integer values. In the training stage of the present invention, more detailed information is retained through integer activation, improving the network performance. At the same time, the integer values replace the multi-time-step pulses, avoiding the problem of excessive memory overhead caused by too long time steps, which makes it difficult to train;
[0063] In the inference stage, by extending the virtual time step, that is, splitting the integer activation value n into pulse values of 1 × n time steps to maintain pulse drive, avoiding the cumulative multiplication calculation introduced by integer activation, reducing power consumption, and giving play to the advantages of the spiking neural network. Moreover, an adaptive threshold is introduced, only making the thresholds of important neurons lower and more easily trigger pulses, reducing the amount of computation while making the network easier to learn general features and avoid overfitting; and integer activation is performed to convert the membrane potential accumulation amount into integer pulse numbers, which is easier to train while retaining rich details. During inference, by extending the time step without increasing multiplication operations, power consumption is reduced.
[0064] Specifically, 4 downsampling modules and 4 SNN modules are adopted in the backbone network. The downsampling modules and SNN modules are stacked alternately in 4 groups in sequence, so that the size of the feature map is gradually reduced, and multi-scale feature information from the first shallow feature to the fourth deep feature is extracted.
[0065] S3: Divide the vision tasks into image recognition, image detection and image segmentation, and connect them to different processing modules for processing and output respectively;
[0066] Exemplarily, for the image classification task, connect to the classification head, map the spatial dimension of the end feature map to the classification category for image classification. For the image detection task, use the neck module to perform upsampling to align the feature map in the spatial dimension. Then, splice the high-level feature and the low-level feature, combine the details and semantics, and finally connect to the detection head to output the final detection box and category. For the image segmentation task, use the feature pyramid module to capture the context information, combine the high and low features, and perform mask embedding. Finally, connect to the segmentation head to generate the final segmentation mask.
[0067] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.
[0068] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A training method for a spiking neural network based on an adaptive threshold integer activation neuron, characterized in that: It includes the following steps: S1: Input the neuromorphic data stream or static image, generate a pulse sequence through time encoding, and use it as the input of the model; S2: Use the SNN module with LIF neurons based on adaptive threshold integer activation in the backbone network to perform downsampling and extract multi-scale features; The membrane potential update formula of the LIF neuron is: V t+1 = τV t (1 - S t ) + WX t S t = Θ(V t - V th ) Among them, V t is the membrane potential at time step t, S t is the pulse signal at time step t, W is the weight matrix, connecting the input X t and the membrane potential, τ is the leakage factor of the neuron, V th is the threshold, Θ is the Heaviside step function, and when the cumulative membrane potential is greater than the threshold, a step signal is generated; The LIF neuron includes: making the pulse threshold of the neuron inversely proportional to the membrane potential change rate, so that the more important and active neurons have lower thresholds. Its continuous-time form is: V th = a + (1 - a)e -ψ / λ where a is a trainable parameter, ψ is the rate of change of membrane potential, λ is a hyperparameter set to 0.1 during training, Δt is the time difference between two consecutive changes in membrane potential, and ΔV m is the difference in membrane potential between two consecutive changes. Based on the exponentially decaying characteristic, an increase in membrane potential will lead to a decrease in the threshold, achieving a sensitive response to the input intensity; Further modify the pulse signal generation function and the membrane potential update function as follows: V t+1 = τ(V t - S t V th ) + WX t Based on the fact that the modified pulse can be integer-activated, more detailed information can be retained through integer activation during the training phase; S3: Divide the visual tasks into image recognition, image detection, and image segmentation, and connect them to different processing modules respectively.
2. The training method of a spiking neural network based on an adaptive threshold integer activation neuron according to claim 1, wherein: The specific input of the model includes: The pulsed neural network inputs the neuromorphic data stream or image according to time steps; After time encoding, a four-dimensional tensor in the [T, C, H, W] format is output to obtain a pulse sequence; Then it is normalized to obtain the input of the backbone network.
3. A training method for a spiking neural network based on an adaptive threshold integer activation neuron according to claim 1, characterized in that: The specific content of S2 includes: Adopt the backbone network of the SNN module with LIF neurons based on adaptive threshold integer activation; Perform downsampling and feature extraction in sequence; Capture edge, texture, and semantic features layer by layer; Form a multi-level feature representation.
4. A method for training a spiking neural network based on an adaptive threshold integer activation neuron according to claim 3, characterized in that: The SNN module is expressed as: SepConv(U) = Conv pw (Conv dw (SN(Conv pw (SN(U)))))) ChannelConv(U′) = Conv(SN(Conv(SN(U′)))) U′ = U + SepConv(U) U″ = U′ + ChannelConv(U′) Among them, Conv pw is pointwise convolution, which is used to perform convolution on the input feature map in the channel dimension using a convolution kernel of size 1×1; Conv dw is depthwise convolution, which is used to perform spatial convolution on each input channel separately; Conv is standard convolution, which is used in other branches or modules, SN is the spiking neuron layer, which is used for LIF neurons with adaptive threshold integer activation to map the continuous convolution output to discrete pulse activation; SepConv is separable convolution, which is used to combine Conv pw and Conv dw , and ChannelConv is an optional channel mixer, which is used to further enhance the information fusion between channels.
5. A method for training a spiking neural network based on an adaptive threshold integer activation neuron according to claim 4, characterized in that: The SN layer uses the LIF neuron with adaptive threshold integer activation as the basic unit, and the LIF neuron is the LIF neuron with adaptive threshold integer activation.
6. A method for training a spiking neural network based on an adaptive threshold integer activation neuron according to claim 3, wherein: The specific content of performing downsampling and feature extraction in sequence includes: Adopt 4 downsampling modules and 4 SNN modules in the backbone network; Stack 4 groups of downsampling modules and SNN modules alternately in sequence, so that the size of the feature map gradually decreases; Extract multi-scale feature information from the first shallow feature to the fourth deep feature.
7. A training method for a spiking neural network based on an adaptive threshold integer activation neuron according to claim 1, characterized in that: The specific content of step S3 includes: For the image classification task, connect to the classification head; Map the spatial dimension of the end feature map to the classification category for image classification; For the image detection task, use the neck module to perform upsampling to align the feature map in terms of spatial size; Stitch the high-level features and low-level features together to combine details and semantics; Finally, connect to the detection head to output the final detection box and category; For the image segmentation task, use the feature pyramid module to capture context information; Combine high and low features for mask embedding; Finally, connect to the segmentation head to generate the final segmentation mask.