Pulse neural network training method based on adaptive threshold integer activation neurons

Abstract

The invention discloses a pulse neural network training method based on adaptive threshold integer activation neurons, and the method specifically comprises the following steps: inputting a neuromorphic data stream or a static image, generating a pulse sequence through time coding, and taking the pulse sequence as the input of a model; in the backbone network, performing down-sampling and extracting multi-scale features by using an SNN module based on LIF neurons activated by an adaptive threshold integer; the visual tasks are divided into image recognition, image detection and image segmentation, and are respectively accessed to different processing modules for processing and outputting; according to the invention, on the basis of a Meta-SpikeFormer element module, the spiking neural network based on LIF neurons activated by an adaptive threshold integer value is used, so that the memory overhead during training is reduced, the network performance of direct training is improved, the application range of the spiking neural network is expanded, and the network is better applied to image classification, recognition and segmentation tasks.

Description

Technical Field

[0001] The invention relates to the technical field of pulse neural network training, and in particular to a pulse neural network training method based on adaptive threshold integer activation neurons. Background Art

[0002] In recent years, the field of artificial intelligence has made great achievements in computer vision, speech recognition, and natural language processing (NLP), which are comparable to or even exceed human performance. As the third generation of neural networks, spiking neural networks (SNNs) imitate biological neurons and use sparse, event-driven pulse activation for inter-neuron communication. They can greatly reduce the energy consumption burden of autonomous driving calculations and have broad prospects. Traditional spiking neural networks are limited by complex neuron dynamics and non-differentiable characteristics, making it difficult to directly apply back-propagation optimization algorithms for direct training.

[0003] The mainstream artificial neural network (ANN) conversion spike neural network method often has extremely high time steps in order to achieve performance similar to that of ANN, resulting in increased energy consumption and latency, and the inability to capture spatiotemporal information in the data, and is gradually being abandoned. The emerging alternative gradient direct training method has serious performance degradation at shorter time steps, and is currently only used for simple image classification tasks. The use of larger time steps leads to excessive memory overhead and difficulty in training the model. In order to overcome the shortcomings of alternative gradient direct training SNN and expand the application scope of SNN, many studies have been explored, such as EMS-YOLO, which became the first model to use directly trained SNN to handle object detection, and Meta-SpikeFormer, which was able to handle target detection in a pre-trained and fine-tuned manner for the first time. However, these studies still have a gap with the performance of advanced artificial neural networks. Summary of the invention

[0004] The purpose of the present invention is to provide a pulse neural network training method based on adaptive threshold integer activation neurons to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a pulse neural network training method based on adaptive threshold integer activation neurons, comprising the following steps:

[0006] S1: Input neuromorphic data stream or static image, generate pulse train through time encoding, as input of the model;

[0007] Preferably, the network inputs a neuromorphic data stream (such as event camera output) or an image in time steps, and outputs a four-dimensional tensor of [T, C, H, W] after time encoding to obtain a pulse sequence, where T represents the number of discrete time steps, C represents the number of channels (such as polarity, grayscale), H and W represent the image height and width, respectively, and then normalized to obtain the backbone network input.

[0008] S2: Using adaptive threshold integer activation in the backbone network The SNN module of neurons performs downsampling and extracts multi-scale features;

[0009] Preferably, a backbone network of an SNN module based on LIF neurons activated by adaptive threshold integer value is used, and downsampling and feature extraction are performed in sequence to capture edge, texture and semantic features layer by layer to form a multi-level feature representation. The SNN module is represented as:

[0010]

[0011]

[0012]

[0013] in, It is a point-by-point convolution, which is used to convolve the input feature map in the channel dimension using a convolution kernel of size 1×1; It is a depth convolution, which is used to perform spatial convolution on each channel of the input; It is a standard convolution, which is used in other branches or modules. is a spiking neuron layer for activation with adaptive threshold integer values Neurons, which map continuous convolution outputs to discrete spike activations; is a separable convolution, which is used to and To combine, It is an optional channel mixer used to further enhance the information fusion between channels.

[0014] Preferably, The LIF neuron activated by adaptive threshold integer value is used as the basic unit. Neurons are activated by adaptive threshold integer values Neurons, the The membrane potential update formula of a neuron is:

[0015]

[0016]

[0017] in, is the time step The membrane potential, is the time step The pulse signal, is the weight matrix, connecting the input and membrane potential, is the leak factor of neurons, is the threshold value, It is a Heaviside step function. When the accumulated membrane potential is greater than the threshold, a step signal is generated.

[0018] Preferably, the pulse threshold of the neuron is inversely proportional to the rate of change of the membrane potential, so that the more important and active the neuron, the lower the threshold. Its continuous time form is:

[0019]

[0020]

[0021] in, are trainable parameters, is the membrane potential change rate, is a hyperparameter, set to 0.1 during training. is the time difference between two membrane potential changes, is the difference between the two membrane potential changes. Based on the characteristics of exponential decay, an increase in membrane potential will lead to a decrease in the threshold, thus achieving a sensitive response to the input intensity. The pulse signal generation function and the membrane potential update function are further modified as follows:

[0022]

[0023]

[0024] Based on the above modified pulses capable of integer-valued activation, more detailed information can be retained through integer-valued activation during the training stage.

[0025] S3: Divide the visual tasks into image recognition, image detection and image segmentation, and connect them to different processing modules respectively.

[0026] Preferably, for image classification tasks, a classification head is connected to map the spatial dimensions of the terminal feature map to classification categories for image classification. For image detection tasks, a neck module is used to perform upsampling to align the feature maps in spatial dimensions, and then high-level features are concatenated with low-level features, combining details with semantics. Finally, a detection head is connected to output the final detection box and category. For image segmentation tasks, a feature pyramid module is used to capture contextual information, combining high and low features, and performing mask embedding. Finally, a segmentation head is connected to generate the final segmentation mask.

[0027] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention adopts a backbone network of an SNN module of LIF neurons based on adaptive threshold integer activation, performs downsampling and feature extraction in sequence, captures edge, texture and semantic features layer by layer, forms a multi-level feature representation, and introduces an adaptive threshold so that the threshold of more important and more active neurons is lower, which not only makes it easier to trigger pulses, but also reduces the amount of calculation while making it easier for the network to learn common features and avoid overfitting; and by performing integer activation, the accumulated amount of membrane potential is converted into an integer number of pulses, which makes it easier to train while retaining rich details, and also further reduces power consumption by extending the time step without increasing multiplication operations during inference, thereby improving the overall performance of the network for direct training and expanding the application scope of pulse neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0029] Figure 1 It is a schematic diagram of the steps of a pulse neural network training method based on adaptive threshold integer activation neurons provided by an embodiment of the present invention;

[0030] Figure 2 It is a schematic diagram of the SNN module structure of the LIF neuron based on adaptive threshold integer activation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Embodiments of the present invention are combined Figure 1 and Figure 2 , specifically providing the following technical solutions: a pulse neural network training method based on adaptive threshold integer activation neurons, combined with Figure 1 As shown, the method comprises the following steps:

[0033] S1: Input neuromorphic data stream or static image, generate pulse train through time encoding, as input of the model;

[0034] Exemplarily, the network inputs a neuromorphic data stream (such as event camera output) or an image in time steps, and outputs a four-dimensional tensor of [T, C, H, W] after time encoding to obtain a pulse sequence, where T represents the number of discrete time steps, C represents the number of channels (such as polarity, grayscale), H and W represent the image height and width, respectively, which are then normalized to obtain the backbone network input.

[0035] S2: In the backbone network, an SNN module based on LIF neurons activated with adaptive threshold integer value is used to downsample and extract multi-scale features;

[0036] Exemplarily, a backbone network of an SNN module based on LIF neurons activated by adaptive threshold integer value is used, and downsampling and feature extraction are performed in sequence to capture edge, texture and semantic features layer by layer to form a multi-level feature representation. The SNN module is represented as:

[0037]

[0038]

[0039]

[0040]

[0041] Specific combination Figure 2 As shown, each module is explained as follows:

[0042] : Pointwise Conv, convolution the input feature map in the channel dimension using a 1×1 convolution kernel;

[0043] : Depthwise Conv, which performs spatial convolution on each channel of the input;

[0044] : Standard convolution (Vanilla Conv), which can be used in other branches or modules;

[0045] : Spiking Neuron Layer, in the present invention, LIF neurons activated by adaptive threshold integer values ​​are used to map continuous convolution outputs into discrete spike activations;

[0046] : Batch Normalization Layer. In the present invention, a threshold-dependent batch normalization layer is used. It is a batch normalization method specially designed for spiking neural networks. On the basis of considering the spatial dimension, it introduces feature statistics and threshold dependence of the time dimension, thereby ensuring the stability and efficiency of training.

[0047] :Used to and Combined with the traditional convolution operation, the convolution operation can achieve efficient separable convolution, which not only maintains the feature extraction performance but also significantly reduces the number of parameters and calculations.

[0048] : An optional channel mixer to further enhance the information fusion between channels.

[0049] For example, Layers activated with adaptive thresholds Neurons are the basic units, and the activation function of traditional neural networks is Neurons, here activated using adaptive threshold integer values Neuron, the specific formula concept is derived as follows:

[0050] The membrane potential update formula of a neuron is:

[0051]

[0052]

[0053] in, is the time step The membrane potential, is the time step The pulse signal, is the weight matrix, connecting the input and membrane potential, is the leak factor of neurons, is the threshold value, It is the Heaviside step function. When the accumulated membrane potential is greater than the threshold, a step signal is generated.

[0054] Furthermore, the present invention intends to make the threshold of more important and active neurons lower, so that it is easier to trigger pulses. By making the pulse threshold of neurons inversely proportional to the rate of change of membrane potential, its continuous time form can be described as:

[0055]

[0056]

[0057] in, are trainable parameters, is the membrane potential change rate, is a hyperparameter, set to 0.1 during training. is the time difference between two membrane potential changes, is the difference between the two membrane potential changes. Due to the exponential decay characteristics, the increase in membrane potential will lead to a rapid and smooth decrease in the threshold, realizing a sensitive response to the input intensity. At the same time, the upper and lower limits of the threshold are also limited to prevent the threshold from getting out of control. The pulse signal generation function and the membrane potential update function are modified as follows:

[0058]

[0059]

[0060] Thus, the pulse can be activated by integer value. In the training stage, the present invention retains more detailed information through integer value activation, improves network performance, and at the same time, the integer value replaces the multi-time step pulse, avoiding the problem of excessive memory overhead caused by too long time step, which makes training difficult.

[0061] In the inference stage, the virtual time step is extended, that is, the integer activation value n is split into pulse values ​​1×n time steps to maintain pulse drive, avoiding the cumulative multiplication calculation introduced by integer value activation, reducing power consumption, and giving full play to the advantages of pulse neural networks. In addition, an adaptive threshold is introduced to only make the threshold of important neurons lower and easier to trigger pulses, reducing the amount of calculation while making it easier for the network to learn common features and avoid overfitting; and integer value activation is performed to convert the accumulated membrane potential into an integer number of pulses, which is easier to train while retaining rich details. During inference, the time step is extended without increasing multiplication operations to reduce power consumption.

[0062] Specifically, 4 downsampling modules and 4 SNN modules are used in the backbone network. The downsampling modules and SNN modules are alternately stacked in 4 groups in sequence to gradually reduce the size of the feature map and extract multi-scale feature information from the first shallow feature to the fourth deep feature.

[0063] S3: Divide the visual tasks into image recognition, image detection and image segmentation, and connect them to different processing modules for processing and output;

[0064] Exemplarily, for image classification tasks, the classification head is connected to map the spatial dimensions of the terminal feature map to classification categories for image classification. For image detection tasks, the neck module is used to perform upsampling to align the feature maps in spatial dimensions. Then, the high-level features are concatenated with the low-level features, combining details and semantics. Finally, the detection head is connected to output the final detection box and category. For image segmentation tasks, the feature pyramid module is used to capture contextual information, combine high and low features, and perform mask embedding. Finally, the segmentation head is connected to generate the final segmentation mask.

[0065] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0066] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A spiking neural network training method based on adaptive threshold integer activation neurons, characterized in that: The following steps are involved: S1: Input neuromorphic data stream or static image, generate pulse train through time encoding, as input of the model; S2: Using adaptive threshold integer activation in the backbone network The SNN module of neurons performs downsampling and extracts multi-scale features; S3: Divide the visual tasks into image recognition, image detection and image segmentation, and connect them to different processing modules respectively.

2. The method for training a spiking neural network based on adaptive threshold integer activated neurons according to claim 1, characterized in that: The input of the model specifically includes: The spiking neural network takes as input a neuromorphic data stream or image in time steps; After time encoding, a four-dimensional tensor in the format of [T, C, H, W] is output to obtain a pulse sequence; After normalization, the backbone network input is obtained.

3. The method for training a spiking neural network based on adaptive threshold integer activated neurons according to claim 1, characterized in that: The S2 specifically includes: Adopting adaptive threshold integer activation The backbone network of the SNN module of neurons; Downsampling and feature extraction are performed sequentially; Capture edge, texture and semantic features layer by layer; Form a multi-level feature representation.

4. The method for training a spiking neural network based on adaptive threshold integer activated neurons according to claim 3, characterized in that: The SNN module is represented as: ； ； ； ； in, It is a point-by-point convolution, which is used to convolve the input feature map in the channel dimension using a convolution kernel of size 1×1; It is a depth convolution, which is used to perform spatial convolution on each channel of the input; It is a standard convolution, which is used in other branches or modules. is a spiking neuron layer for activation with adaptive threshold integer values Neurons, which map continuous convolution outputs to discrete spike activations; is a separable convolution, which is used to and To combine, It is an optional channel mixer used to further enhance the information fusion between channels.

5. The method for training a spiking neural network based on adaptive threshold integer activated neurons according to claim 4, characterized in that: Said The LIF neuron activated by adaptive threshold integer value is used as the basic unit. Neurons are activated by adaptive threshold integer values Neuron.

6. A spiking neural network training method based on adaptive threshold integer activated neurons according to claim 5, characterized in that: Said The membrane potential update formula of a neuron is: ； ； in, is the time step The membrane potential, is the time step The pulse signal, is the weight matrix, connecting the input and membrane potential, is the leak factor of neurons, is the threshold value, It is a Heaviside step function. When the accumulated membrane potential is greater than the threshold, a step signal is generated.

7. The method for training a spiking neural network based on adaptive threshold integer activated neurons according to claim 4, characterized in that: Said Neurons also include: making the pulse threshold of neurons inversely proportional to the rate of change of membrane potential, so that the more important and active neurons have lower thresholds, and its continuous time form is: ； ； in, are trainable parameters, is the membrane potential change rate, is a hyperparameter, set to 0.1 during training. is the time difference between two membrane potential changes, It is the difference between the membrane potential changes before and after. Based on the characteristics of exponential decay, the increase of membrane potential will lead to a decrease in the threshold, thus achieving a sensitive response to the input intensity. The pulse signal generation function and membrane potential update function are further modified as follows: ； ； Based on the above modified pulses capable of integer-valued activation, more detailed information can be retained through integer-valued activation during the training stage.

8. The method for training a spiking neural network based on adaptive threshold integer activated neurons according to claim 3, characterized in that: The sequential downsampling and feature extraction specifically include: Four downsampling modules and four SNN modules are used in the backbone network; The downsampling module and the SNN module are stacked alternately in 4 groups, so that the size of the feature map is gradually reduced; Extract multi-scale feature information from the first shallow feature to the fourth deep feature.

9. The method for training a spiking neural network based on adaptive threshold integer activated neurons according to claim 1, characterized in that: The step S3 specifically includes: For image classification tasks, access the classification head; Mapping the spatial dimensions of the terminal feature map to classification categories for image classification; For image detection tasks, the neck module is used to perform upsampling so that the feature maps are aligned in spatial size; Concatenate high-level features with low-level features to combine details and semantics; Finally, the detection head is connected to output the final detection frame and category; For image segmentation tasks, a feature pyramid module is used to capture contextual information; Combine high and low features to perform mask embedding; Finally, the segmentation head is connected to generate the final segmentation mask.

Images