High-performance and low-delay deep pulse neural network conversion method

Abstract

The invention discloses a high-performance and low-delay deep pulse neural network conversion method. A corresponding SNN neural network is converted with the minimum time step length. Inspired by quantitative perceptual training, a QReLU activation function is introduced in the training process, coding errors are eliminated, in addition, a multi-bit pulse sequence is integrated in the QReLU activation function for training, conversion performance can be improved, and finally, a non-leakage IF (Integrating-and-Fire) neuron with a cache is designed to reduce reasoning errors. Experiments of classification and detection tasks prove that the method can achieve the performance of the artificial neural network level by only using 16 time steps.

Description

Technical Field

[0001] The present invention relates to the field of pulse neural networks and the field of conversion from artificial neural networks to pulse neural networks, and specifically to a high-performance and low-latency deep pulse neural network conversion method. Background Art

[0002] In recent years, spiking neural networks (SNNs) have attracted widespread attention from researchers due to their spatiotemporal information processing capabilities and low power consumption. Unlike traditional artificial neural networks (ANNs) that communicate with each neuron between layers at each reasoning step, SNNs only transmit sparse signals when specific neurons are triggered. The on-demand computing characteristics of SNNs are extremely friendly to low-power hardware, making them very suitable for application on neuromorphic platforms, such as event-based cameras and brain-like chips. This dedicated bionic system can significantly reduce memory usage and energy consumption, which is very attractive for many application areas including the Internet of Things (IoT), autonomous robotic systems, and self-driving vehicles. Despite its excellent properties, training high-performance deep SNNs remains a challenging problem because the non-differentiable nature of spike events makes it impossible to directly apply backpropagation methods.

[0003] One class of methods converts ANNs to SNNs by demonstrating the relationship between spike neurons and rectified linear unit (ReLU) functions through rate-based coding methods. Rate-based coding refers to using the rate of spike trains transmitted through each neuron in the SNN to measure the output of the network. The goal of this type of conversion method is generally to maintain the high performance of artificial neural networks in the SNN framework. Recent work has studied the errors introduced during the encoding process and proposed an error-aware neural network training method to obtain better SNN performance, but this only considers the errors caused by rounding. Many studies have also proposed inference errors and balanced encoding errors and inference errors by selecting better conversion parameters, but adjusting the conversion parameters introduces additional encoding errors, further limiting the performance of the converted SNN.

[0004] Another class of methods attempts to train SNNs by approximating the gradient by replacing the threshold function with other functions. Some works have proposed back-propagation methods for superimposed gradients. This training method shows different ranges of gradient functions and encoding schemes, which can significantly reduce the number of time steps compared to the conversion method. Although SNNs trained by back-propagation show the potential to achieve ANN-level performance, accurately training SNNs still requires additional overhead, which makes SNNs difficult to train and limits their application in real-world tasks.

[0005] Recently, the emergence of combination methods has alleviated the disadvantage of conversion methods that require larger time steps to achieve competitive SNN performance. These methods usually consist of two steps: conversion and fine-tuning. In the first step, ANN is first converted to SNN, and then the converted SNN is further fine-tuned. However, compared with the conversion method, additional overhead is required to fine-tune the SNN. In addition, backpropagation imposes additional constraints on the hybrid method, which further limits its application scope. Summary of the invention

[0006] The purpose of the present invention is to solve the following technical problems:

[0007] 1) Inefficiency: The performance of SNN converted by ANN needs to rely on a large time step to compensate for the conversion error, which greatly reduces the efficiency in practice.

[0008] 2) Unstable methods and high overhead: Existing methods are prone to introduce additional coding errors, which further limits the performance of SNNs. Approximate gradient and combination-based methods require additional overhead in training SNNs, limiting the application of SNNs in practical scenarios.

[0009] 3) Ignoring the inference error: Coding error and inference error will reduce the performance of SNN during the entire conversion process, but existing methods only focus on how to reduce the coding error, which limits the performance of the algorithm and leads to longer simulation time to achieve results comparable to the original ANN.

[0010] In view of the problems of related technologies, the present invention provides a high-performance and low-latency deep pulse neural network conversion method, which aims to solve the limitations of the existing technology in terms of efficiency, overhead, and solving inference errors. Therefore, the technical solution of the present invention includes the following necessary technical features: the method first proposes a quantized ReLU activation function to minimize the encoding error in the conversion process, and proposes a non-leaky IF (Integrate-and-Fire) neuron with cache to reduce the inference error, and integrates these two methods into the conversion framework from ANN to SNN, reducing the gap between the converted SNN and ANN.

[0011] The specific technical solution for achieving the purpose of the present invention is:

[0012] A high-performance and low-latency deep spiking neural network conversion method is used to solve encoding errors and inference errors, and can have the characteristics of high efficiency and low latency in various tasks. The method includes the following specific steps:

[0013] Step 1: Construct a quantized activation function

[0014] First, define the membrane potential formula as follows:

[0015]

[0016] Among them, V j (t) represents the membrane potential of neuron j at time t, V th Measure whether the neuron has generated a spike, Represents the weight between the input i-th neuron and the current j-th neuron, represents the set of all input neurons in the last layer connected to the jth neuron, S i (t) and S j (t) represents the input pulse and output pulse value of the current neuron; let the above formula be divided by V th T, the following definition of pulse rate is obtained:

[0017]

[0018] Among them, r i represents the rate of pulse potential of the i-th neuron in a period of time, let represents the accumulated membrane potential in the time period T, R * It represents the ratio of the accumulated membrane potential to time over a period of time, so R * is defined as follows:

[0019]

[0020] From the above formula, we can get that when R * When it approaches 0, a simple IF neuron will be more similar to the rectified linear unit ReLU function in the process of processing information, so R * This is considered as a coding error; when V a >0, R * It will gradually decrease as time T increases. Since more and more coding errors will accumulate between the layers of the neural network, the SNN converted from the ANN with more layers requires a large time step to maintain high fidelity and performance. On the other hand, a larger time step will damage the efficiency of the SNN. Therefore, the pulse sequence rate is rewritten as a lower bound form so that it is not affected by time T:

[0021]

[0022] in, is the weighted sum of the input pulse rates in the SNN, based on the maximum pulse rate r max , the IF neuron is further defined using a step function:

[0023]

[0024] By mapping the pulse rate in SNN and the activation value a=λr in ANN, the corresponding quantized rectified linear unit QReLU function is obtained, namely:

[0025]

[0026] Because the quantized corrected linear unit QReLU function eliminates the influence of time T, it is used to replace the original ReLU function in the ANN, so that the SNN converted from the ANN eliminates the coding error R * ; Where λ represents the mapping coefficient between the SNN activation value and the ANN activation value, and r represents the SNN activation value;

[0027] Step 2: Integrated multi-bit pulse training

[0028] According to the QReLU function obtained in step 1, the QReLU function is transformed into a multi-bit pulse form; first, the original pulse definition is expanded to It is beneficial to directly perform shift operations during the conversion process, where S max represents the maximum value of the pulse signal; at the same time, an adjustment function A(·) with log2 is used, which is defined as follows:

[0029]

[0030] Therefore, the functional form of the multi-bit pulse is transformed into the following:

[0031]

[0032] For the QReLU function, since the maximum rate encoding is equal to the maximum number of pulses Therefore, based on the multi-bit pulse sequence, the quantization boundary λ is introduced b =λS max and characterization capability P=TS max , finally, the QReLU function is modified as follows:

[0033]

[0034] Finally, the QReLU function is embedded in ANN for training to further improve the conversion performance;

[0035] Step 3: Set up a non-leaky IF neuron with cache

[0036] In the process of SNN parameter transmission, a non-leaky IF neuron with a cache mechanism is set to alleviate the "over-firing" problem existing in the inference error; specifically, a new non-leaky IF neuron with a cache is proposed to alleviate the "over-firing" situation to compensate for the inference error. The neuron uses a cyclic mechanism to adjust the pulse, the cache potential V b The definition is as follows:

[0037]

[0038] in, represents the cache potential at time t, V th is a threshold that determines whether a neuron generates a spike; the initial buffer potential is set to 0, and the pulse generation process is re-described as:

[0039]

[0040] In the case of “over-issuance”, V′ j (t)<0, and the above formula is in V′ j When (t)<0, a negative pulse will be generated. By adding the cache potential to the judgment condition of pulse generation, the pulse generation can be controlled to -1 when over-discharging occurs to reduce the total pulse count, so that over-discharging is alleviated, thereby greatly reducing the reasoning error. Like the typical IF neuron, this newly designed neuron only involves addition operations and the application of threshold functions, which helps to maintain efficient reasoning performance. At the same time, it has been proved through experiments that only alleviating the "over-discharging" phenomenon can greatly improve the conversion performance of SNN.

[0041] Beneficial Effects

[0042] 1) The present invention proposes a QReLU function, which can theoretically completely eliminate the coding error and achieve a balance between accuracy and efficiency in the ANN to SNN conversion process.

[0043] 2) Through a detailed analysis of the inference error, a non-leaky IF neuron with a caching mechanism was implemented, which alleviated the inference error to a great extent.

[0044] 3) In order to further improve the computational efficiency in the conversion process, the present invention introduces an integrated design of multi-bit pulses in training, thereby improving the performance of the ANN-SNN conversion framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Flow chart of the ANN to SNN conversion process of the present invention;

[0046] Figure 2This is a flow chart of SNN parameter transfer of non-leaky IF neurons based on a cache mechanism in the present invention. DETAILED DESCRIPTION

[0047] The present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0048] A high-performance and low-latency deep pulse neural network conversion method of the present invention is used to convert ANN into a corresponding SNN with a smaller time step, and the method includes the following specific steps:

[0049] Step 1: Construct a quantized ReLU activation function

[0050] In order to deal with coding errors, the present invention proposes a quantized ReLU activation function in the training stage of artificial neural network to accurately convert weight values ​​to SNN under limited time steps. There are two essential differences between SNN network and ANN network. First, neurons in SNN communicate with each other through pulse sequences S(t), where for t∈{1,...,T}, S(t)∈{0,1}, while the activation function in ANN uses real values; second, in SNN, non-leaky IF neurons are generally used to replace the activation function in traditional ANN. The membrane potential equation of non-leaky IF neuron j can be described as:

[0051]

[0052] Where Vj(t) represents the membrane potential of the jth neuron, t is the time step, is the set of all input neurons connected to the jth neuron in the last layer, Represents the weight between the input i-th neuron and the current j-th neuron, V th Represents the threshold for the neuron to emit a signal. i (t) and S j (t) are the input sequence of the current neuron and the resulting output pulse sequence, respectively. The generation of the emission signal is as follows:

[0053]

[0054] in, Represents the membrane potential before the signal trigger and potential reset operation. Because the conversion process from ANN to SNN is based on the rate encoding method, for a period of time T, the rate can be obtained The goal of the transformation process is to derive a function w s =f(w a ), convert the parameters of ANN into the parameters of SNN, so that after a period of time T, the pulse rate of SNN is proportional to the value of ANN activation function, that is, r∝a.

[0055] Thus, the total number of output pulse trains after a period of time T can be obtained as follows:

[0056]

[0057] Dividing the above formula by time T, we can get the calculation formula for the rate, as shown below:

[0058]

[0059] in, Under ideal conditions, the output peak count should satisfy the following equation, which is:

[0060]

[0061] in, Represents the accumulated membrane potential. Similarly, the ideal output pulse rate can be obtained through the above formula:

[0062]

[0063] And R * is defined as follows:

[0064]

[0065] In the ideal ANN-SNN conversion process, R * will be considered a coding error because when V a >0, the coding error R * <1 / T will gradually decrease as T increases. Since more and more coding errors will accumulate between the layers of the neural network, the SNN with more layers after conversion requires a large time step T to maintain high fidelity and performance. On the other hand, a large time step impairs the efficiency of the SNN. Therefore, in step 1, the present invention expresses the ideal pulse sequence rate as:

[0066]

[0067] in, is the weighted sum of the input pulse rates in the SNN, taking into account the maximum pulse rate r max , the IF neuron can be represented by a step function as:

[0068]

[0069] The proposed function is strictly equivalent to the IF neuron, which is different from the ReLU function that increases the total time step T. By mapping the pulse rate and the activation value a = λr, the corresponding quantized activation function can be obtained, namely:

[0070]

[0071] The approximate gradient is given by the PACT algorithm:

[0072]

[0073] The scaling factor λ here is also called the quantization margin and is learned during the ANN training process. The successful demonstration of quantization techniques shows that a neural network composed of QReLU can achieve similar performance to the original neural network, so by mapping the weights of the neural network equipped with QReLU activations, the encoding error R can be theoretically eliminated. * .

[0074] Step 2: Integrated multi-bit pulse training

[0075] In order to further improve the performance of the proposed conversion method, the present invention integrates shift-based multi-bit pulse training into the proposed method. The definition of pulse is expanded to In order to directly perform the shift operation during the conversion process, the functional form of the multi-bit pulse is expressed as follows:

[0076]

[0077] The adjustment function A(·) is defined as:

[0078]

[0079] Multi-bit pulse sequences can carry more information than previous pulse sequences. However, a large number of multiplications are required in the integration process using non-uniform pulses. Therefore, the present invention uses a log2 adjustment function and a shift operation to replace the multiplication to achieve better efficiency. Therefore, for the QReLU function proposed in step 1, since the maximum rate encoding is equal to the maximum number of pulses

[0080] Therefore, based on the multi-bit pulse sequence, the QReLU function can be modified as follows:

[0081]

[0082] Among them, λ b =λS max Indicates the quantization boundary, P = TS max represents “representational capability”, S max represents the maximum value of the pulse signal, λ represents the mapping coefficient between the SNN activation value and the ANN activation value, and T represents the time step. As described in the definition of “representational capability”, as long as S max The product of and the time step T is constant, the performance of the implemented neural network will remain unchanged.

[0083] Step 3: Set up a non-leaky IF neuron with cache

[0084] In addition to the encoding error, during SNN inference, the total number of spike trains within a period of time T does not always follow the ideal Intuitively, the reasoning process of SNN is quite dynamic, and the spiking neurons cannot accumulate membrane potential uniformly over the time step, which leads to differences in the overall output pulse counts. This type of error is called reasoning error. Generally speaking, there are two types of reasoning errors: "over-firing" and "under-firing". The "over-firing" phenomenon refers to the situation where the number of generated pulses is greater than the expected number of pulses, that is, At this time V′ j (t)≥V th Lead to S j (t) = 1, so that the number of pulses generated exceeds the expected number of pulses; conversely, the "under-firing" phenomenon refers to the situation where insufficient pulses are generated.

[0085] For the inference error, based on experiments, only reducing the "over-firing" problem can significantly improve the performance. In order to further improve the conversion method, this paper proposes a new non-leaky IF neuron with cache to alleviate the "over-firing" situation to compensate for the inference error. The proposed cache neuron uses a cyclic mechanism to adjust the pulse, and the cache potential V b The definition is as follows:

[0086]

[0087] Similar to the membrane potential, the initial buffer potential is set to 0, the pulse generation process is described as:

[0088]

[0089] Like ordinary IF neurons, the proposed neuron only involves addition operations and threshold functions, and the extra power required to transmit the sign bit between neurons is negligible, thus maintaining the inference efficiency. This is done because in the case of "over-firing", that is, the number of pulses generated is greater than the expected number of pulses V′ j (t)<0, and the above formula will generate a negative pulse in this case to reduce the total pulse count, so that the "over-release" problem can be alleviated.

[0090] Example

[0091] See also Figure 1 , this embodiment includes the following specific steps:

[0092] Step 1: Train ANN using quantized ReLU function

[0093] In the process of training ANN, the present invention uses a QReLU activation function to eliminate coding errors. Since QReLU is applied in the ANN training stage, no additional resources are consumed in the conversion process. In general, the ideal output pulse rate is defined as follows:

[0094]

[0095] Among them, R * is defined as follows:

[0096]

[0097] in, Represents the accumulated membrane potential. * Definition, in the ANN-SNN conversion process, when R * When V approaches 0, a simple IF neuron is expressed in a similar way to a ReLU function in processing information. At this point, the weights of an SNN can be directly converted from its corresponding neural network. a >0, R * As T increases, it will gradually decrease, so that the coding error can be gradually reduced. Since more and more coding errors will accumulate between the layers of the neural network, the SNN with more layers after conversion requires a large time step T to maintain high fidelity and performance. On the other hand, a large time step impairs the efficiency of the SNN. Therefore, in step 1, the present invention expresses the ideal pulse sequence rate as:

[0098]

[0099] in, is the weighted sum of the input pulse rates in the SNN, using the maximum pulse rate r max , the IF neuron is further expressed by the step function:

[0100]

[0101] The proposed function is strictly equivalent to the IF neuron. By mapping the pulse rate and the activation value a=λr, the corresponding quantized activation function can be obtained, namely:

[0102]

[0103] Because the QReLU function eliminates the influence of time T, the coding error R can be eliminated in theory. * .

[0104] Step 2. Integrated multi-bit pulse training

[0105] In order to further improve the performance of the conversion framework, the present invention integrates multi-bit pulse sequences into the QReLU function. In order to achieve better efficiency, the present invention uses an adjustment function A(·) with log2 and a shift operation to replace the large number of multiplication operations required when using non-uniform pulse integration, and expands the pulse definition to In order to directly perform the shift operation during the conversion process, the functional form of the multi-bit pulse is expressed as follows:

[0106]

[0107] Based on the above multi-bit pulse sequence and the maximum rate encoding equal to the maximum number of pulses The QReLU function can be redefined as:

[0108]

[0109] Among them, λ b =λS max Indicates the quantization boundary, P = TS max Indicates "representational capability". In general, the QReLU function updated by a multi-bit pulse sequence can carry more information and further improve the conversion efficiency.

[0110] Since the pulse rate r is bounded, a linear mapping function is generally required to transform the weights of the artificial neural network, that is, where λ i is the scaling factor of the i-th layer. Since the present invention integrates multi-bit pulse training, where λ b It will be learned during the ANN training process.

[0111] Step 3. SNN parameter transfer based on non-leaky IF neurons with caching mechanism

[0112] After obtaining the parameters of SNN, such as Figure 2 As shown, the SNN parameters begin to be transferred. First, during the transfer process, the membrane potential before the signal trigger and potential reset operation is generated, as shown below:

[0113]

[0114] Where Vj(t) represents the membrane potential of the jth neuron, t is the time step, is the set of all input neurons connected to the jth neuron in the last layer, Represents the weight between the input i-th neuron and the current j-th neuron, Vth Represents the threshold for the neuron to send a signal. Then, based on the current moment Vj(t) and the cached membrane potential at the previous moment The size of the pulse at the current moment is determined by the following judgment conditions:

[0115]

[0116] The adjustment function A(·) is defined as:

[0117]

[0118] Where S max Indicates the maximum value that a pulse can achieve in a multi-bit pulse definition. j (t), reset the membrane potential and cache membrane potential at the current moment. The membrane potential calculation formula at the current moment is expressed as follows:

[0119]

[0120] The calculation formula of the cache membrane potential at the current moment is expressed as follows:

[0121]

[0122] After the update, the SNN parameters are transferred. After that, the rate r of the SNN within a period of time T is taken as the final output, which can be used for target classification, target detection and target segmentation tasks.

Claims

1. A high-performance and low-latency deep spiking neural network conversion method, characterized in that: The method comprises the following specific steps: Step 1: Construct a quantized activation function First, define the membrane potential formula as follows: Among them, V j (t) represents the membrane potential of neuron j at time t, V th Measure whether the neuron has generated a spike, Represents the weight between the input i-th neuron and the current j-th neuron, represents the set of all input neurons in the last layer connected to the jth neuron, S i (t) and S j (t) represents the input pulse and output pulse value of the current neuron; let the above formula be divided by V th T, the following definition of pulse rate is obtained: Among them, r i represents the rate of pulse potential of the i-th neuron in a period of time, let represents the accumulated membrane potential in the time period T, R * It represents the ratio of the accumulated membrane potential to time over a period of time, so R * is defined as follows: From the above formula, we can get that when R * When it approaches 0, a simple IF neuron will be more similar to the rectified linear unit ReLU function in the process of processing information, so R * This is considered as a coding error; when V a >0, R * It will gradually decrease as time T increases. On the other hand, a larger time step will damage the efficiency of SNN. Therefore, the pulse sequence rate is rewritten as a lower bound so that it is not affected by time T: in, is the weighted sum of the input pulse rates in the SNN, based on the maximum pulse rate r max , the IF neuron is further defined using a step function: By mapping the pulse rate in SNN and the activation value a=λr in ANN, the corresponding quantized rectified linear unit QReLU function is obtained, namely: Because the quantized corrected linear unit QReLU function eliminates the influence of time T, it is used to replace the original ReLU function in the ANN, so that the SNN converted from the ANN eliminates the coding error R * ; Where λ represents the mapping coefficient between the SNN activation value and the ANN activation value, and r represents the SNN activation value; Step 2: Integrated multi-bit pulse training According to the QReLU function obtained in step 1, the QReLU function is transformed into a multi-bit pulse form; first, the original pulse definition is expanded to It is beneficial to directly perform shift operations during the conversion process, where S max represents the maximum value of the pulse signal; at the same time, an adjustment function A(·) with log2 is used, which is defined as follows: Therefore, the functional form of the multi-bit pulse is transformed into the following: For the QReLU function, since the maximum rate encoding is equal to the maximum number of pulses Therefore, based on the multi-bit pulse sequence, the quantization boundary λ is introduced b =λS max and characterization capability P = TS max , finally, the QReLU function is modified as follows: Finally, the QReLU function is embedded in ANN for training to further improve the conversion performance; Step 3: Set up a non-leaky IF neuron with cache In the process of SNN parameter transmission, a non-leakage IF neuron with a cache mechanism is set. The neuron uses a loop mechanism to adjust the pulse and cache the potential V b The definition is as follows: in, represents the cache potential at time t, V th is a threshold that determines whether a neuron generates a spike; the initial buffer potential is set to 0, and the pulse generation process is re-described as: In the case of "over-issuance", V' j (t)<0, and the above formula is in V′ j When (t)<0, a negative pulse will be generated. By adding the cache potential into the judgment condition of pulse generation, the pulse generation can be controlled to -1 when over-discharge occurs, so as to reduce the total pulse count. In this way, over-discharge can be alleviated, thereby greatly reducing the inference error.

Images