Quaternary pulse-based pulse neural network accelerator and calculation method thereof

Abstract

The invention relates to the technical field of neural networks, in particular to a pulse neural network accelerator based on quaternary pulses and a calculation method thereof, and the specific method comprises the steps: constructing a neuron model, presetting three membrane thresholds, carrying out quaternary pulse generation trigger judgment according to the real-time membrane potential of the neuron and the three membrane thresholds, and calculating the pulse neural network accelerator based on the quaternary pulses. Meanwhile, executing a zero-skipping strategy in a quaternary pulse generation process, and screening to obtain an effective quaternary pulse; after neurons in the neuron model generate and give out effective quaternary pulses, resetting membrane potentials of the neurons through a membrane potential resetting strategy; synchronously using the input data with the quaternary pulse to train the deep residual pulse neural network with the quaternary pulse, and generating a final output membrane potential and the quaternary pulse through an additional summator; according to the method, the processing time window of the SNN is remarkably shortened, the calculation efficiency and energy efficiency are improved, and the method is suitable for various neural network calculation tasks.

Description

Technical Field

[0001] The invention relates to the technical field of neural network, and is a pulse neural network accelerator based on quaternary pulses and a calculation method thereof. Background Art

[0002] Spiking Neural Networks (SNNs) are a computational model that simulates the behavior of biological neurons, transmitting information between neurons through discrete binary pulses. Compared with traditional artificial neural networks (ANNs), SNNs have event-based characteristics and can more efficiently process sparse data in time and space, thus showing potential advantages in computing power and energy efficiency. However, the discrete pulse characteristics of SNNs also bring about the problems of long processing time windows and high latency, which limits its performance in practical applications.

[0003] In traditional ANNs, information transmission between neurons is achieved through continuous-valued activation functions, which usually requires a lot of computing resources and energy consumption. In order to improve computing efficiency and energy efficiency, researchers began to explore event-driven computing models such as SNNs. SNNs transmit information through pulse signals. The sparsity and discreteness of pulses make it possible to reduce the use of multipliers in hardware implementation, thereby reducing energy consumption. However, since SNNs need to process a large number of time steps, the computing delay increases, which is a significant challenge in practical applications.

[0004] In order to solve the high latency problem in SNNs, researchers have proposed a variety of methods, including improving coding schemes, optimizing network structures, and designing efficient hardware accelerators. Among them, rate coding and time coding are the two main coding schemes. Rate coding encodes information as the frequency of pulses, while time coding encodes information as the time of pulses. Although time coding has higher information density in theory, rate coding is more commonly used in practical applications because it is easier to implement in hardware.

[0005] In recent years, Residual Networks (ResNets) have achieved remarkable success in ANNs. By introducing skip connections, the network can train deeper architectures, thereby improving performance. In order to apply this technology to SNNs, researchers have proposed a variety of Spiking Residual Networks (SResNets), which further improve the performance and computational efficiency of the network by introducing skip connections in SNNs. Summary of the invention

[0006] The technical problem to be solved by the present invention is that SNNs in the prior art usually require a large number of time steps to achieve reliable performance, resulting in increased computational delays. In addition, existing hardware accelerators often require a large amount of on-chip memory and computing resources when processing SNNs, which limits their scalability in practical applications. A quaternary pulse-based pulse neural network accelerator and a computational method thereof are proposed.

[0007] In order to achieve the above object, the technical solution of the calculation method of the pulse neural network accelerator based on quaternary pulses of the present invention comprises the following steps:

[0008] S1: Construct a neuron model, preset three membrane thresholds, and perform trigger judgment for quaternary pulse generation based on the real-time membrane potential of the neuron and the three membrane thresholds;

[0009] S2: while executing the zero skipping strategy in the process of generating the quaternary pulse, a valid quaternary pulse is obtained by screening;

[0010] S3: After the neuron in the neuron model generates and emits a valid quaternary pulse, the membrane potential of the neuron is reset by using a membrane potential reset strategy;

[0011] S4: The deep residual spiking neural network with quaternary pulses is trained synchronously using the input data with quaternary pulses, and the final output membrane potential and quaternary pulses are generated through an additional adder;

[0012] S5: Perform performance evaluation on the deep residual spike neural network that has completed training and optimization.

[0013] Specifically, S1 includes the following specific steps:

[0014] S11: A constructed neuron model, wherein the neuron model quaternary pulse is a leaky integrate-and-discharge (LIF) neuron model, and the neuron model is used to respond to an input pulse;

[0015] S12: presetting three membrane thresholds, the membrane thresholds including: a first membrane threshold θ1, a second membrane threshold θ2 and a third membrane threshold θ3;

[0016] S13: extracting three preset membrane thresholds, and performing pulse emission triggering judgment according to the real-time membrane potential of the neuron and the three membrane thresholds, wherein the pulse emission triggering judgment includes:

[0017] When the real-time membrane potential V(t) of the neuron is less than or equal to the first membrane threshold θ1, it is judged that the neuron does not meet the pulse emission triggering condition, and the generated pulse value is 0;

[0018] When the neuron's real-time membrane potential V(t) is greater than the first membrane threshold θ1 but less than or equal to the second membrane threshold θ2, the neuron is judged to meet the pulse emission triggering condition and the generated pulse value is 1;

[0019] When the real-time membrane potential V(t) of the neuron is greater than the second membrane threshold θ2 but less than or equal to the second membrane threshold θ3, the neuron is judged to meet the pulse emission triggering condition and the generated pulse value is 2.

[0020] When the real-time membrane potential V(t) of the neuron is greater than the third membrane threshold θ3, the neuron is judged to meet the pulse emission triggering condition and the generated pulse value is 3.

[0021] Specifically, S2 also includes the following specific steps:

[0022] S21: simultaneously executing a zero skipping strategy in the quaternary pulse encoding process, screening and obtaining a valid quaternary pulse, wherein the zero skipping strategy includes: when an input pulse is loaded into an input buffer, indicating the sparseness of the pulse by an index, and performing a flag bit judgment of the index, wherein the input pulse is loaded in a ping-pong form; the index includes three register files of 0.5KB;

[0023] S22: When the eight input pulse groups of the two parts at the same position in the input feature map are both equal to zero, the flag bit of the marking index is 0; otherwise, the flag bit of the marking index is 1;

[0024] S23: Repeat step S1, each time selecting a non-zero input for calculation, and skipping input pulses with a 0 flag through a filtering mechanism.

[0025] Specifically, S3 includes:

[0026] When the neuron in the neuron model generates and emits a valid quaternary pulse, the membrane potential of the neuron is reset by a membrane potential reset strategy, wherein the membrane potential reset strategy specifically includes: a first membrane potential reset mode, a second membrane potential reset mode, and a third membrane potential reset mode;

[0027] The first membrane potential resetting mode includes: sorting the first membrane thresholds of each neuron in the quaternary pulse pulse neural network from small to large, and resetting the first membrane potential by removing the first M membrane thresholds in the sorting sequence;

[0028] The second membrane potential resetting mode includes: according to the pulse value generated by the neuron, the neuron performs a second membrane potential resetting by subtracting different membrane thresholds.

[0029] The third membrane potential reset mode includes: according to the pulse value generated by the neuron, the neuron performs a third membrane potential reset by subtracting different membrane thresholds.

[0030] Specifically, in S4, the deep residual pulse neural network specifically includes multiple residual blocks, each residual block is composed of multiple convolutional layers, in each residual block, the membrane potential of the previous layer is added to the membrane potential of the current layer to realize M2M connection, wherein the deep residual pulse neural network is a deep residual network architecture.

[0031] Specifically, in S4, training a deep residual spike neural network with quaternary pulses using input data with quaternary pulses includes: using a batch normalization over time (BNTT) technique to learn different statistics for each time step.

[0032] Specifically, in S4, generating the final output membrane potential and quaternary pulse through an additional adder includes: calculating the sum of the two input membrane potentials of the residual connection layer, and adding leak_mem to the sum through an accumulator to generate the final output membrane potential and quaternary pulse.

[0033] In addition, the pulse neural network accelerator based on quaternary pulses of the present invention includes the following modules:

[0034] The system includes the following modules: a data buffer, a computing array, a system controller, a zero skip strategy execution module, and an adder module;

[0035] The data buffer is used to store input data, intermediate calculation results and output data, ensuring efficient data transmission between different layers and supporting parallel computing;

[0036] The computing array includes a plurality of computing units, wherein the computing array is used to support the calculation of a pulse convolution layer, a normal convolution layer, a residual connection layer, an average pooling layer, and a fully connected layer;

[0037] The computing array is used to support the calculation of the pulse convolution layer, the ordinary convolution layer, the residual connection layer, the average pooling layer and the fully connected layer, wherein the computing array is composed of 16 computing units;

[0038] The system controller is used to control the communication between the data buffer and the computing array;

[0039] The zero skip strategy execution module is used to skip the calculation of zero-value input pulses by utilizing the sparsity of the input pulses;

[0040] The adder module is used to calculate the sum of the two input membrane potentials of the residual connection layer, and then the accumulator adds leak_mem to the sum and generates the final output membrane potential and pulse.

[0041] A storage medium stores instructions, and when a computer reads the instructions, the computer executes the calculation method of the pulse neural network accelerator based on quaternary pulses.

[0042] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the calculation method of the pulse neural network accelerator based on quaternary pulses when executing the computer program.

[0043] Compared with the prior art, the technical effects of the present invention are as follows: The present invention significantly shortens the processing time window of SNN by introducing quaternary pulse and residual connection technology, and improves the computing efficiency and energy efficiency. The proposed accelerator includes a data buffer, a computing array and a system controller. The computing array consists of multiple computing units and supports the calculation of pulse convolution layer, ordinary convolution layer, residual connection layer, average pooling layer and full connection layer. A layer-by-layer data flow scheme is adopted to calculate the output of all time steps of a layer, and then the output is calculated layer by layer, which reduces the pressure of on-chip memory and avoids repeated loading of weights. Utilizing the sparsity of input pulses, zero input pulses are skipped through a zero skip scheme to reduce the amount of calculation. The residual connection layer is calculated by adding the membrane potential of the previous layer to the membrane potential of the current layer, which reduces memory consumption. Tested on the CIFAR-10 and CIFAR-100 datasets, the best accuracy rates of 93.21% and 93.54% were achieved at 10 and 20 time steps, respectively. When using only 6 time steps, the accuracy rates of 92.73% and 92.14% were achieved, and the repeated calculations were reduced by about 70%. The present invention significantly shortens the processing time window of SNN, improves the computing efficiency and energy efficiency, and is suitable for various neural network computing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0045] in:

[0046] Figure 1 It is a flow chart of a calculation method of a pulse neural network accelerator based on quaternary pulses of the present invention;

[0047] Figure 2 It is a schematic diagram of the structure of the pulse neural network accelerator based on quaternary pulses of the present invention;

[0048] Figure 3It is a schematic diagram of the overall architecture of a pulse neural network accelerator based on quaternary pulses of the present invention;

[0049] Figure 4 A schematic diagram of the structure of a computing unit of the present invention;

[0050] Figure 5 A simplified structural diagram of a computing unit of the present invention;

[0051] Figure 6 A data flow diagram of the present invention;

[0052] Figure 7 It is a schematic diagram of performance comparison between a pulse neural network accelerator model based on quaternary pulses of the present invention and a baseline model;

[0053] Figure 8 This is a layout photo of a pulse neural network accelerator based on quaternary pulses according to the present invention. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0056] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0057] Embodiment 1:

[0058] Exemplarily, in this embodiment, a layer-by-layer data flow scheme is adopted to calculate the output of all time steps of a layer, and then calculate the output layer by layer, which reduces the pressure on the on-chip memory and avoids repeated loading of weights.

[0059] like Figure 1 and Figure 3 As shown, the pulse neural network accelerator based on quaternary pulses according to the embodiment of the present invention is as follows: Figure 1 As shown, the specific steps are as follows:

[0060] S1: Construct a neuron model, preset three membrane thresholds, and perform trigger judgment for quaternary pulse generation based on the real-time membrane potential of the neuron and the three membrane thresholds;

[0061] S1 includes the following specific steps:

[0062] S11: A constructed neuron model, wherein the neuron model quaternary pulse is a leaky integrate-and-discharge (LIF) neuron model, and the neuron model is used to respond to an input pulse;

[0063] S12: presetting three membrane thresholds, the membrane thresholds including: a first membrane threshold θ1, a second membrane threshold θ2 and a third membrane threshold θ3;

[0064] S13: extracting three preset membrane thresholds, and performing pulse emission triggering judgment according to the real-time membrane potential of the neuron and the three membrane thresholds, wherein the pulse emission triggering judgment includes:

[0065] When the real-time membrane potential V(t) of the neuron is less than or equal to the first membrane threshold θ1, it is judged that the neuron does not meet the pulse emission triggering condition, and the generated pulse value is 0;

[0066] When the neuron's real-time membrane potential V(t) is greater than the first membrane threshold θ1 but less than or equal to the second membrane threshold θ2, the neuron is judged to meet the pulse emission triggering condition and the generated pulse value is 1;

[0067] When the real-time membrane potential V(t) of the neuron is greater than the second membrane threshold θ2 but less than or equal to the third membrane threshold θ3, the neuron is judged to meet the pulse emission triggering condition and the generated pulse value is 2.

[0068] When the real-time membrane potential V(t) of the neuron is greater than the third membrane threshold θ3, the neuron is judged to meet the pulse emission triggering condition and the generated pulse value is 3.

[0069] Exemplarily, in this embodiment, a specific description formula of the membrane potential of the leaky integral-discharge (LIF) neuron model is provided, which specifically includes:

[0070]

[0071] in, is the membrane potential of neuron j at time step t; is the membrane potential at time step t-1; τ is the attenuation factor used to scale the leakage of the membrane potential;

[0072] represents the synaptic weight, Indicates the occurrence of a pulse at time step t.

[0073] S2: while executing the zero skipping strategy in the process of generating the quaternary pulse, a valid quaternary pulse is obtained by screening;

[0074] Exemplarily, in this embodiment, the neuron generates quaternary pulses (0, 1, 2, 3) instead of binary pulses (0, 1); this quaternary pulse coding scheme enhances the ability of information transmission, making it possible to achieve competitive performance in a shorter time window.

[0075] S2 also includes the following specific steps:

[0076] S21: simultaneously executing a zero skipping strategy in the quaternary pulse encoding process, screening and obtaining a valid quaternary pulse, wherein the zero skipping strategy includes: when an input pulse is loaded into an input buffer, indicating the sparseness of the pulse by an index, and performing a flag bit judgment of the index, wherein the input pulse is loaded in a ping-pong form; the index includes three register files of 0.5KB;

[0077] S22: When the eight input pulse groups of the two parts at the same position in the input feature map are both equal to zero, the flag bit of the marking index is 0; otherwise, the flag bit of the marking index is 1;

[0078] S23: Repeat step S1, select non-zero input for calculation each time, and skip input pulses with 0 flags through a filtering mechanism, so as to reduce the amount of calculation.

[0079] Exemplarily, in this embodiment, when calculating the pulse convolution layer, during the execution of the zero skip strategy, the input feature map is divided into two parts, upper and lower, with the same height, and each part is calculated by a calculation array.

[0080] Furthermore, in this embodiment, it has been tested that the above zero skip strategy can skip about 44.7% of the input pulses in SResNet38 and 36.8% in SResNet20.

[0081] S3: After the neuron in the neuron model generates and emits a valid quaternary pulse, the membrane potential of the neuron is reset by using a membrane potential reset strategy;

[0082] S3 includes: when the neuron in the neuron model generates and issues a valid quaternary pulse, resetting the membrane potential of the neuron through a membrane potential reset strategy, wherein the membrane potential reset strategy specifically includes: a first membrane potential reset mode, a second membrane potential reset mode, and a third membrane potential reset mode;

[0083] The first membrane potential resetting mode includes: sorting the first membrane thresholds of each neuron in the quaternary pulse pulse neural network from small to large, and resetting the first membrane potential by removing the first M membrane thresholds in the sorting sequence;

[0084] The second membrane potential resetting mode includes: according to the pulse value generated by the neuron, the neuron performs a second membrane potential resetting by subtracting different membrane thresholds.

[0085] The third membrane potential reset mode includes: according to the pulse value generated by the neuron, the neuron performs a third membrane potential reset by subtracting different membrane thresholds.

[0086] S4: The deep residual spiking neural network with quaternary pulses is trained synchronously using the input data with quaternary pulses, and the final output membrane potential and quaternary pulses are generated through an additional adder;

[0087] Exemplarily, in this embodiment, the residual connection technology is adopted to realize the pulse residual network. Figure 3 An example pulse residual network architecture is shown. The present invention uses batch normalization over time (BNTT), which is a time-varying batch normalization (BN) that learns different statistics for each time step to achieve better performance. The input image is not converted into pulses, but directly fed into the network for better performance. The residual network architectures of SResNet38 and SResNet20 are selected in the experiment. All three networks consist of 3 cascaded residual blocks, each residual block in SResNet38 has 12 convolutional layers, while there are 6 convolutional layers in SResNet20; the number of output channels of the 3 residual blocks is 32, 64, and 128, respectively.

[0088] In S4, the deep residual pulse neural network specifically includes multiple residual blocks, each residual block is composed of multiple convolutional layers, in each residual block, the membrane potential of the previous layer is added to the membrane potential of the current layer to achieve M2M connection, wherein the deep residual pulse neural network is a deep residual network architecture, and the residual connection layer is calculated by adding the membrane potential of the previous layer to the membrane potential of the current layer, thereby reducing memory consumption.

[0089] In S4, training a deep residual spiking neural network with quaternary pulses using input data with quaternary pulses includes: using batch normalization over time (BNTT) technology to learn different statistics for each time step to achieve better performance.

[0090] In S4, generating the final output membrane potential and quaternary pulse by the additional adder includes: calculating the sum of the two input membrane potentials of the residual connection layer, and adding leak_mem to the sum through the accumulator to generate the final output membrane potential and quaternary pulse;

[0091] Exemplarily, in this embodiment, the adder tree of the adder calculates the sum of 8 weighted pulses each time;

[0092] Exemplarily, in this embodiment, a calculation process of a pulse convolution layer is also provided, which specifically includes the following steps:

[0093] Step S41: When the first input pulse of the sliding window is calculated, the reset signal is raised. The register V_mem is set to zero by the reset signal, and the register leak_mem records the membrane potential of the previous step after reset and leakage.

[0094] Step S42: All weighted pulses in the sliding window are calculated and accumulated in sequence.

[0095] Step S43: BNTT operation is calculated. Usually, BN operation is fused into convolution layer by preprocessing convolution weights. However, BNTT introduces different BN layers in different time steps, so BNTT layer cannot be fused into convolution layer. We implement BNTT operation in 2 clock cycles using multipliers and accumulators after step 2. Multipliers can also be reused in the calculation process of ordinary convolution layer or fully connected layer, so BNTT operation does not introduce additional hardware cost.

[0096] Step S44: If the convolutional layer has a residual connection, the calculation process is completed and returns to step 1. The residual connection will be implemented in the residual connection layer. Since the final membrane potential is obtained in the residual connection layer, the leakage of the membrane potential is not implemented in this step.

[0097] Step S45: If the convolutional layer has no residual connection, the nxt_sign signal is raised and leak_mem is added to V_mem to generate an output pulse.

[0098] Step S46: When calculating a normal convolutional layer or the last fully connected layer, the calculation process is similar to the pulse convolutional layer, but only the multiplier is used for the calculation of the weighted input. When calculating the average pooling layer, only the accumulator of V_mem works.

[0099] In this embodiment, in order to speed up the calculation of the residual connection layer, an additional adder is introduced. The adder is used to calculate the sum of the two input membrane potentials of the residual connection layer, and then the accumulator adds leak_mem to the sum to generate the final output membrane potential and pulse.

[0100] S5: Perform performance evaluation on the deep residual spike neural network that has completed training and optimization.

[0101] Embodiment 2:

[0102] like Figure 2 , 4 As shown in , 5, the pulse neural network accelerator based on quaternary pulses of the embodiment of the present invention is as follows Figure 2 As shown, it includes the following modules:

[0103] The system includes the following modules: a data buffer, a computing array, a system controller, a zero skip strategy execution module, and an adder module;

[0104] The data buffer is used to store input data, intermediate calculation results and output data, ensuring efficient data transmission between different layers and supporting parallel computing;

[0105] The computing array includes a plurality of computing units, wherein the computing array is used to support the calculation of a pulse convolution layer, a normal convolution layer, a residual connection layer, an average pooling layer, and a fully connected layer;

[0106] The computing array is used to support the calculation of the pulse convolution layer, the ordinary convolution layer, the residual connection layer, the average pooling layer and the fully connected layer, wherein the computing array is composed of 16 computing units;

[0107] Exemplarily, in this embodiment, the computing array is a reconfigurable computing array;

[0108] like Figure 5 As shown, the quaternary pulse-based pulse neural network accelerator provided in this embodiment has three computing arrays, each of which consists of 16 computing units. The computing units can be reconfigured to support different layers in the proposed residual SNN. For example, in Figure 5 In the figure, the calculation process of BNTT is not shown for brevity. The output pulses or membrane potentials of all time steps will be calculated continuously and do not need to be stored into SRAM and then accessed for the next step of calculation.

[0109] The system controller is used to control the communication between the data buffer and the computing array;

[0110] Exemplarily, in this embodiment, the system controller is responsible for coordinating and managing the operation of the entire accelerator, scheduling the work of the data buffer and the computing array according to the network structure and computing requirements, and ensuring that computing tasks are executed sequentially and efficiently.

[0111] The zero skip strategy execution module is used to skip the calculation of zero-value input pulses by utilizing the sparsity of the input pulses;

[0112] The adder module is used to calculate the sum of the two input membrane potentials of the residual connection layer, and then the accumulator adds leak_mem to the sum and generates the final output membrane potential and pulse.

[0113] Exemplarily, in this embodiment, the computing unit architecture includes: the computing array is composed of multiple computing units (CUs), and each CU can calculate the sum of weighted pulses. The computing unit supports calculations of different layers, including pulse convolution layers, ordinary convolution layers, residual connection layers, average pooling layers, and fully connected layers. The computing unit calculates the sum of weighted pulses through an addition tree and supports BNTT operations. The BNTT operation is calculated by multiplying the cumulative sum of weighted pulses with the BN weight and adding the BN bias.

[0114] Embodiment three:

[0115] Exemplarily, in this embodiment, the test steps of the above-mentioned quaternary pulse-based spiking neural network (SNN) accelerator are provided:

[0116] First, the performance of the proposed SNN model under different β values ​​is tested, as shown in Table I. For the CIFAR-10 / CIFAR-100 datasets, reset mode 1 under β = 3 and β = 4 and reset mode 2 under β = 2 and β = 3 achieve competitive performance and are selected for performance evaluation. Figure 4 As shown in Figure 2, the proposed SNN model achieves significantly better performance than the baseline SNN model [8] when the time window does not exceed 30. In particular, the proposed SNN model achieves the best accuracy of 93.21% (72.06%) and 93.54% (71.77%) on the CIFAR-10 (CIFAR-100) dataset at 10 and 20 time steps, respectively, while the accuracy of the baseline model is only 85.87% (61.52%) and 91.28% (67.7%).

[0117] When the time window becomes longer, the performance of the proposed SNN model tends to decline relative to the baseline model, especially when the β value is large. However, when the time window is short, a larger β also leads to better performance. We tested the performance of the proposed SNN model using only 6 time steps, and reset mode 1 was applied under β = 4. On the CIFAR-10 (CIFAR-100) dataset, SResNet38 and SResNet20 achieved accuracies of 92.73% (70.82%) and 92.14% (69.53%), respectively. The baseline model requires more than 20 time steps to achieve similar performance, so the proposed SNN model reduces repeated calculations by about 70%.

[0118] In addition to residual SNNs, non-residual SNNs with quaternary pulses are also tested in this embodiment. The Fashion-MNIST dataset is tested using the network architecture of Input-64C3-MP2-64C3-MP2-256-10, and the audio dataset FSDD is tested using the network architecture of Input-256-10. The inputs of the three datasets are converted into pulse sequences, and reset mode 2 is applied to the three networks at 4 time steps and β = 2. The accuracy rates of 91.84% and 97% were achieved respectively. Overall, the SNN based on quaternary pulses also achieves competitive performance under the non-residual architecture.

[0119] In this embodiment, a layer-by-layer data flow scheme is adopted because the time window required for the proposed SNN model reasoning is short. Figure 6 As shown in Figure 2, there are two different computational data flows that can be used for SNN inference: (1) t-by-t: first calculate the outputs of all layers within a time step, and then repeat the calculation process time by time step; (2) layer-by-layer: calculate the outputs of all time steps of a layer, and then calculate the output layer by layer.

[0120] Further, in the present embodiment, for the pulse convolution layer, the input feature map is divided into two parts, upper and lower, with the same height. Each part is calculated by a computing array. The computing array consists of 16 computing units (CUs). Each CU can calculate the sum of 8 weighted pulses at a time, and the input pulses are shared by different CUs in the computing array. Each CU in the computing array calculates different channels of the output feature map. Therefore, one computing array can calculate 8 input channels and 16 output channels at the same time, and the weights are shared by three computing arrays. When calculating other types of layers, only computing array 1 is in working state, and 16 output channels are calculated each time. The residual connection is not calculated together with the convolution layer, but is calculated separately to reduce memory consumption. When calculating the residual connection layer, the weight memory is reused as input and output memory to save memory.

[0121] For example, in this embodiment, Figure 8 As shown, a processor layout photo of a quaternary pulse-based pulse neural network accelerator is provided;

[0122] The quaternary pulse-based pulse neural network accelerator is implemented using 28 nanometer CMOS technology with a core area of ​​0.63 square millimeters; the design is synthesized by Synopsys Design Compiler and automatically placed and routed using Synopsys IC Compiler.

[0123] Using SResNet20 on the CIFAR-10 dataset achieved a classification accuracy of 92.07%, requiring only 6 time steps;

[0124] When running at 500MHz, the accelerator consumes 0.39 millijoules of energy per frame and has a throughput of 165.7FPS.

[0125] Furthermore, if Figure 7 As shown, this embodiment uses the quaternary pulse-based pulse neural network accelerator to test on the CIFAR-10 and CIFAR-100 datasets, and achieves the best accuracy of 93.21% and 93.54% at 10 and 20 time steps, respectively.

[0126] Furthermore, 92.07% classification accuracy was achieved using SResNet20 on the CIFAR-10 dataset, requiring only 6 time steps;

[0127] Furthermore, the non-residual spiking neural network with quaternary pulse-based learning was tested on the Fashion-MNIST dataset and the audio dataset FSDD, achieving accuracies of 91.84% and 97%, respectively.

[0128] Using only 6 time steps, we achieve 92.73% and 92.14% accuracy, reducing repeated computations by about 70%.

[0129] Furthermore, in this embodiment,

[0130] The network architecture of Input-64C3-MP2-64C3-MP2-256-10 is tested on the Fashion-MNIST dataset;

[0131] Use the network architecture of Input-256-10 to test the audio dataset FSDD;

[0132] Reset mode 2 is applied to the three networks at 4 time steps and β = 2.

[0133] In this embodiment, after testing, it can be found that: the proposed quaternary pulse-based spiking neural network (SNN) accelerator achieves the highest accuracy in the reference work on the CIFAR-10 dataset;

[0134] The proposed design achieves 155.1FPS / mm under SResNet38 2 The area efficiency and energy efficiency of 0.66mJ / Frame achieve 263FPS / mm under SResNet20. 2 area efficiency and energy efficiency of 0.39mJ / Frame.

[0135] Embodiment 4:

[0136] This embodiment provides an electronic device, including: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;

[0137] The processor executes the above-mentioned quaternary pulse-based pulse neural network accelerator by calling the computer program stored in the memory.

[0138] The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) and one or more memories, wherein the memory stores at least one computer program, and the computer program is loaded and executed by the processor to implement the pulse neural network accelerator based on quaternary pulses provided by the above method embodiment. The electronic device may also include other components for implementing the functions of the device. For example, the electronic device may also have components such as a wired or wireless network interface and an input and output interface to input and output data. This embodiment will not be described in detail here.

[0139] Embodiment five:

[0140] This embodiment provides a computer-readable storage medium having a rewritable computer program stored thereon;

[0141] When the computer program runs on a computer device, the computer device executes the above-mentioned quaternary pulse-based pulse neural network accelerator.

[0142] For example, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0143] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0144] It should be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0145] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through a wired network or / and a wireless network. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0146] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0148] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or three or more units may be integrated into one unit.

[0151] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0152] In summary, compared with the prior art, the technical effects of the present invention are as follows:

[0153] The present invention significantly shortens the processing time window of SNN and improves the computational efficiency and energy efficiency by introducing quaternary pulse and residual connection technology. The proposed accelerator includes a data buffer, a computing array and a system controller. The computing array consists of multiple computing units and supports the calculation of pulse convolution layer, ordinary convolution layer, residual connection layer, average pooling layer and fully connected layer. A layer-by-layer data flow scheme is adopted to calculate the output of all time steps of a layer, and then the output is calculated layer by layer, which reduces the pressure of on-chip memory and avoids repeated loading of weights. The sparsity of input pulses is utilized to skip zero input pulses through the zero skip scheme to reduce the amount of calculation. The residual connection layer is calculated by adding the membrane potential of the previous layer to the membrane potential of the current layer, which reduces memory consumption. Tested on the CIFAR-10 and CIFAR-100 datasets, the best accuracy rates of 93.21% and 93.54% were achieved at 10 and 20 time steps, respectively. When using only 6 time steps, the accuracy rates of 92.73% and 92.14% were achieved, and the repeated calculations were reduced by about 70%. The present invention significantly shortens the processing time window of SNN, improves the computing efficiency and energy efficiency, and is suitable for various neural network computing tasks.

[0154] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A calculation method for a pulse neural network accelerator based on quaternary pulses, characterized in that: The method comprises the following specific steps: S1: Construct a neuron model, preset three membrane thresholds, and perform trigger judgment for quaternary pulse generation based on the real-time membrane potential of the neuron and the three membrane thresholds; S2: while executing the zero skipping strategy in the process of generating the quaternary pulse, a valid quaternary pulse is obtained by screening; S3: After the neuron in the neuron model generates and emits a valid quaternary pulse, the membrane potential of the neuron is reset by using a membrane potential reset strategy; S4: The deep residual spiking neural network with quaternary pulses is trained synchronously using the input data with quaternary pulses, and the final output membrane potential and quaternary pulses are generated through an additional adder; S5: Perform performance evaluation on the deep residual spike neural network that has completed training and optimization.

2. The computing method based on the pulse neural network accelerator according to claim 1 is characterized in that: S1 includes the following specific steps: S11: A neuron model is constructed, wherein the quaternary pulse of the neuron model is a leakage integrate-discharge neuron model, and the neuron model is used to respond to an input pulse; S12: presetting three membrane thresholds, the membrane thresholds including: a first membrane threshold θ1, a second membrane threshold θ2 and a third membrane threshold θ3; S13: extracting three preset membrane thresholds, and performing pulse emission triggering judgment according to the real-time membrane potential of the neuron and the three membrane thresholds, wherein the pulse emission triggering judgment includes: When the real-time membrane potential V(t) of the neuron is less than or equal to the first membrane threshold θ1, it is judged that the neuron does not meet the pulse emission triggering condition, and the generated pulse value is 0; When the neuron's real-time membrane potential V(t) is greater than the first membrane threshold θ1 but less than or equal to the second membrane threshold θ2, the neuron is judged to meet the pulse emission triggering condition and the generated pulse value is 1; When the real-time membrane potential V(t) of the neuron is greater than the second membrane threshold θ2 but less than or equal to the third membrane threshold θ3, the neuron is judged to meet the pulse emission triggering condition and the generated pulse value is 2. When the real-time membrane potential V(t) of the neuron is greater than the third membrane threshold θ3, the neuron is judged to meet the pulse emission triggering condition and the generated pulse value is 3.

3. The calculation method of the pulse neural network accelerator based on quaternary pulses according to claim 2 is characterized in that: S2 also includes the following specific steps: S21: simultaneously executing a zero skipping strategy in the quaternary pulse encoding process, screening and obtaining a valid quaternary pulse, wherein the zero skipping strategy includes: when an input pulse is loaded into an input buffer, indicating the sparseness of the pulse by an index, and performing a flag bit judgment of the index, wherein the input pulse is loaded in a ping-pong form; the index includes three register files of 0.5KB; S22: When the eight input pulse groups of the two parts at the same position in the input feature map are both equal to zero, the flag bit of the marking index is 0; otherwise, the flag bit of the marking index is 1; S23: Repeat step S1, each time selecting a non-zero input for calculation, and skipping input pulses with a 0 flag through a filtering mechanism.

4. The calculation method of the pulse neural network accelerator based on quaternary pulses according to claim 3, characterized in that S3 include: When the neuron in the neuron model generates and emits a valid quaternary pulse, the membrane potential of the neuron is reset by a membrane potential reset strategy, wherein the membrane potential reset strategy specifically includes: a first membrane potential reset mode, a second membrane potential reset mode, and a third membrane potential reset mode; The first membrane potential resetting mode includes: sorting the first membrane thresholds of each neuron in the quaternary pulse pulse neural network from small to large, and resetting the first membrane potential by removing the first M membrane thresholds in the sorting sequence; The second membrane potential resetting mode includes: according to the pulse value generated by the neuron, the neuron performs a second membrane potential resetting by subtracting different membrane thresholds. The third membrane potential reset mode includes: according to the pulse value generated by the neuron, the neuron performs a third membrane potential reset by subtracting different membrane thresholds.

5. The calculation method of the pulse neural network accelerator based on quaternary pulses according to claim 4 is characterized in that: In S4, the deep residual pulse neural network specifically includes multiple residual blocks, each residual block is composed of multiple convolutional layers, in each residual block, the membrane potential of the previous layer is added to the membrane potential of the current layer to achieve M2M connection, wherein the deep residual pulse neural network is a deep residual network architecture.

6. The calculation method of the pulse neural network accelerator based on quaternary pulses according to claim 5 is characterized in that: In S4, training a deep residual spiking neural network with quaternary spikes using input data with quaternary spikes includes: using a time-varying batch normalization technique to learn different statistics for each time step.

7. The calculation method of the pulse neural network accelerator based on quaternary pulses according to claim 7, characterized in that: In S4, generating the final output membrane potential and quaternary pulse by an additional adder includes: calculating the sum of the two input membrane potentials of the residual connection layer, and adding leak_mem to the sum through the accumulator to generate the final output membrane potential and quaternary pulse.

8. A pulse neural network accelerator based on quaternary pulses, which is implemented based on the calculation method of the pulse neural network accelerator based on quaternary pulses as described in any one of claims 1 to 7, characterized in that: The system includes the following modules: a data buffer, a computing array, a system controller, a zero skip strategy execution module, and an adder module; The data buffer is used to store input data, intermediate calculation results and output data, ensuring efficient data transmission between different layers and supporting parallel computing; The computing array includes a plurality of computing units, wherein the computing array is used to support the calculation of a pulse convolution layer, a normal convolution layer, a residual connection layer, an average pooling layer, and a fully connected layer; The computing array is used to support the calculation of the pulse convolution layer, the ordinary convolution layer, the residual connection layer, the average pooling layer and the fully connected layer, wherein the computing array is composed of 16 computing units; The system controller is used to control the communication between the data buffer and the computing array; The zero skip strategy execution module is used to skip the calculation of zero-value input pulses by utilizing the sparsity of the input pulses; The adder module is used to calculate the sum of the two input membrane potentials of the residual connection layer, and then the accumulator adds leak_mem to the sum and generates the final output membrane potential and pulse.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the calculation method of the pulse neural network accelerator based on quaternary pulses as described in any one of claims 1-7.

10. An electronic device, characterized in that: include: A memory for storing instructions; A processor is used to execute the instructions so that the device performs operations to implement the computing method of the quaternary pulse-based pulse neural network accelerator as described in any one of claims 1-7.

Images