Pulse neural network construction method based on ANN-SNN conversion

Abstract

The invention belongs to the field of artificial intelligence and neural network calculation, and discloses a pulse neural network construction method based on ANN-SNN conversion. According to the method, ANN model training is optimized by adopting a clipping function capable of learning a threshold value, so that the output of the ANN model can be accurately mapped to the pulse distribution rate of the SNN model; according to the method, through weight normalization and threshold mapping, the conversion precision from the ANN model to the SNN model is improved; according to the invention, a dual-threshold neuron mechanism is adopted, neuron excitation and inhibition are dynamically adjusted, and quantization errors and non-uniform errors are reduced; according to the invention, an optimized membrane potential initialization method is designed, so that the SNN model can still maintain high-precision reasoning capability under extremely low time steps; the method can be widely applied to the fields of low-power-consumption embedded equipment, neural mimicry chips, target detection, automatic driving and the like, and the SNN is promoted to land in practical application.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence and neural network computing, and relates to a method for constructing a spiking neural network based on ANN-SNN conversion to improve the computing performance of neural networks in low-power environments. Background Art

[0002] Artificial neural networks (ANNs) have achieved great success in fields such as computer vision and natural language processing, but their high computational cost and high energy consumption limit their application in embedded and low-power devices. In contrast, spiking neural networks (SNNs) can effectively reduce power consumption through event-driven computing and are suitable for low-power real-time processing tasks. However, directly training SNNs has difficulties in gradient calculation, so the ANN-SNN conversion method has become a mainstream solution.

[0003] The existing ANN-SNN conversion methods mainly have the following problems: 1) Bu et al. analyzed in the literature "Optimal ann-snn conversion for high-accuracy and ultra-low-latency spiking neural networks" that clipping errors, quantization errors, and non-uniform errors are likely to occur during the conversion process, resulting in a decline in the performance of SNNs; 2) Sengupta et al. pointed out in the literature "Going deeper in spiking neural networks: Vgg and residual architectures" that existing methods usually require a long inference time step (T>16) to obtain accuracy comparable to that of ANNs, reducing the real-time processing ability of SNNs; 3) Hubara et al. pointed out in the literature "Binarized neural networks" that traditional conversion methods have low hardware adaptability for SNNs and do not fully utilize the advantages of SNNs in low-power environments. Summary of the Invention

[0004] The present invention proposes an efficient ANN-SNN conversion method, which significantly reduces conversion errors and improves the accuracy of SNNs in the case of low time steps through a learnable clipping function, a dual-threshold neuron design, and an optimized membrane potential initialization strategy.

[0005] A method for constructing a spiking neural network based on ANN-SNN conversion comprises the following steps:

[0006] (1) Construct and train a source artificial neural network ANN model

[0007] 1) Select a deep convolutional neural network architecture (such as VGG-16, ResNet-18) as the ANN model, and pre-train the ANN model using standard datasets (such as CIFAR-10, ImageNet);

[0008] 2) During the training process, use a clipping function with a learnable threshold to replace the traditional ReLU activation function, aiming to improve the adaptability of the ANN model after being converted into an SNN model, so as to ensure that the output of the ANN model can be directly mapped to the spike firing pattern of the SNN model. The clipping function with a learnable threshold is based on the following formula:

[0009]

[0010] where a l represents the activation value of the artificial neural network, represents the activation function, z l represents the weighted input of the l-1 layer, the clip function is used to limit the activation range, L represents the quantization step of the ANN, represents the bit shift term, and λ represents the learnable threshold.

[0011] (2) Conversion from ANN model to SNN model

[0012] 1) After the ANN model is trained, normalize the network weights, and map the threshold parameters obtained during the training of the ANN model to the spike firing thresholds of the corresponding SNN neurons. The activation value mapping from the ANN model to the SNN model satisfies the following formula:

[0013]

[0014] where θ l is the spike firing threshold of the SNN model, T is the time step, a l is the activation value of the ANN model, and φ l (T) is the spike output of the SNN model within T steps.

[0015] 2) Introduce a membrane potential initialization strategy, and set the initial membrane potential of each layer of SNN neurons to half of their firing thresholds to reduce the non-uniform error caused by inconsistent initial states. It is characterized in that the membrane potential initialization satisfies the expectation of minimizing the error, and the theoretical derivation result is:

[0016]

[0017] where E z represents the conversion error, f(z) represents the output value of the ANN model, represents the output value of the SNN model, that is, the optimal initialization value is

[0018] (3) Adopt a dual-threshold neuron mechanism, set positive and negative thresholds, and stipulate that the neuron emits a positive pulse when the membrane potential exceeds the positive threshold, and emits a negative pulse when it is lower than the negative threshold and has emitted a positive pulse before, so as to reduce the influence of quantization error and membrane potential residual on the inference result. The formula of the dual-threshold neuron is as follows:

[0019]

[0020] Among them, θ is the positive threshold, and θ' is the negative threshold. is the membrane potential of neuron i at time step t. is the number of pulses emitted by neuron i at time step t. Then rewrite the membrane dynamics update rule of the neuron as:

[0021]

[0022] Among them, represents the weighted input of layer l-1, and this mechanism ensures that the neuron can recover quickly after excitation, thus reducing quantization error and information loss.

[0023] (3) Low-latency inference and hardware deployment

[0024] It is applicable to low-power embedded devices and neuromorphic chips (such as Loihi, TrueNorth), and shows excellent energy efficiency ratio and real-time performance in hardware deployment.

[0025] Advantages of the present invention:

[0026] (1) The present invention optimizes the ANN training by using a learnable threshold pruning function to ensure that the ANN output can be accurately mapped to the pulse firing rate of the SNN;

[0027] (2) The present invention improves the conversion accuracy from ANN to SNN through weight normalization and threshold mapping;

[0028] (3) The present invention adopts a dual-threshold neuron mechanism to dynamically adjust neuron excitation and inhibition, reducing quantization error and non-uniform error;

[0029] (4) The present invention designs an optimized membrane potential initialization method, enabling the SNN to still maintain high-precision inference ability at extremely low time steps (T≤2);

[0030] The method of the present invention can be widely applied to fields such as low-power embedded devices, neuromorphic chips, target detection, and autonomous driving, promoting the implementation of SNN in practical applications. Description of the drawings

[0031] Figure 1For the improvement effect of the efficient ANN-SNN conversion method proposed by the present invention, where (a) is VGG-16 on CIFAR-10, (b) is ResNet-20 on CIFAR-10, and (c) is ResNet-18 on CIFAR-10.

[0032] Figure 2 For VGG-16 on CIFAR-10, the performance of the converted SNN using different quantization step sizes L and time step sizes T.

[0033] Figure 3 For ResNet-20 on CIFAR-10, the performance of the converted SNN using different quantization step sizes L and time step sizes T.

[0034] Figure 4 For ResNet-18 on CIFAR-10, the performance of the converted SNN using different quantization step sizes L and time step sizes T. Detailed implementation manners

[0035] The following further illustrates the detailed implementation manners of the present invention in combination with the accompanying drawings and technical solutions.

[0036] I. Training an artificial neural network (ANN)

[0037] 1) Select a deep convolutional neural network architecture (such as VGG-16, ResNet-18), and perform pre-training using a standard dataset (such as CIFAR-10, ImageNet).

[0038] 2) Replace ReLU with a clipping function with a learnable threshold to ensure that the output of the ANN model can be directly mapped to the spike firing pattern of the SNN model. The clipping function with a learnable threshold described by this method is based on the following formula:

[0039]

[0040] where a l represents the activation value of the ANN model, represents the activation function, z l represents the weighted input of the l-1 layer, the hyperparameter L represents the quantization step size of the ANN model, represents the bit shift term, and λ represents the learnable threshold.

[0041] II. Conversion from the ANN model to the SNN model

[0042] Perform weight normalization and threshold mapping to make the weights and thresholds of the ANN model applicable to the SNN model.

[0043] Adopt an optimized membrane potential initialization strategy to set the initial membrane potential of the SNN model to half of its threshold, reducing non-uniform errors.

[0044] III. Adopt dual-threshold neurons

[0045] By setting positive and negative dual thresholds, the SNN model can more accurately simulate the behavior of the ANN model within a short time step (T ≤ 2). The formula for the dual-threshold neuron is as follows:

[0046]

[0047] where θ' is the negative threshold, is the number of spikes emitted by neuron i at time step t. To improve the sensitivity of negative spikes, the negative threshold is set to a small negative value (set to -1e-3 according to experience) in this paper. Then the membrane dynamics update rule of the IF neuron is rewritten as:

[0048]

[0049] This mechanism ensures that the neuron can quickly recover after firing, thus reducing quantization errors and information loss.

[0050] IV. Low-latency inference and hardware deployment

[0051] Suitable for low-power embedded devices and neuromorphic chips (such as Loihi, TrueNorth).

[0052] Adopt an event-driven hardware architecture to reduce the energy consumption of the SNN model inference process by an order of magnitude compared with the traditional ANN model.

[0053] Embodiment

[0054] As shown in Table 1, for the pulse neural network construction method based on ANN-SNN conversion, CIFAR-10 is used as the training set, which consists of 60,000 32×32 images in 10 categories. There are 50,000 training images and 10,000 test images. Figure 1 Demonstrates the improvement effect of the efficient ANN-SNN conversion method proposed by the present invention. Figures 2 - 4 Demonstrates the performance of the converted SNN model using different quantization step sizes L and time step sizes T.

[0055] Table 1 Test results on CIFAR-10

[0056]

Claims

1. A method for constructing a spiking neural network based on ANN-SNN conversion, characterized in that The steps are as follows: (1) Construct and train an ANN model 1) Select a deep convolutional neural network architecture as the ANN model and pre-train the ANN model using a standard dataset; 2) During the training process, use a clipping function with a learnable threshold to replace the ReLU activation function to ensure that the output of the ANN model is directly mapped to the spike firing pattern of the SNN model; The clipping function with a learnable threshold is based on the following formula: Among them, a l represents the activation value of the ANN model, represents the activation function, z l represents the weighted input of the l-1 layer, the clip function is used to limit the activation range, L represents the quantization step of the ANN, represents the bit shift term, and λ represents the learnable threshold; (2) Conversion from the ANN model to the SNN model 1) After the ANN model is trained, normalize the network weights and map the threshold parameters obtained during the training of the ANN model to the spike firing thresholds of the corresponding SNN neurons. The activation value mapping from the ANN model to the SNN model satisfies the following formula: where, θ l is the spike firing threshold of the SNN model, T is the time step, and φ l (T) is the spike output of the SNN model within T steps; 2) Introduce a membrane potential initialization strategy and set the initial membrane potential of each layer of SNN neurons to half of their firing thresholds to reduce the non-uniform error caused by inconsistent initial states; the membrane potential initialization satisfies the expectation of minimizing the error, which is: Among them, E z represents the conversion error, and f(z) represents the output value of the ANN model. represents the output value of the SNN model, that is, the optimal initialization value is 3) Adopt a two-threshold neuron mechanism, set positive and negative thresholds, and stipulate that the neuron fires a positive spike when the membrane potential exceeds the positive threshold and fires a negative spike when it is lower than the negative threshold and has fired a positive spike before, so as to reduce the impact of quantization error and membrane potential residual on the inference result; the formula for the two-threshold neuron is as follows: where θ is the positive threshold and θ' is the negative threshold, is the membrane potential of neuron i at time step t, is the number of spikes emitted by neuron i at time step t; then the membrane dynamics update rule of the neuron is rewritten as: Among them, represents the weighted input of the l-1 layer, ensuring that the neuron can quickly recover after excitation, thereby reducing quantization error and information loss.

Images