ANN and SNN heterogeneous fusion processor and heterogeneous fusion processing method
By designing a heterogeneous fusion processor for ANN and SNN, and utilizing the heterogeneous fusion processing method, the operating state of the computational cores is selectively limited according to the weight accuracy and sparsity. This solves the problems of high power consumption of deep learning processors and low accuracy of spiking neural network processors, and achieves high energy efficiency, low power consumption inference computation and deep network adaptability.
Patent Information
- Application Number
- CN202410827106.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-25
- Publication Date
- 2024-11-12
AI Technical Summary
Existing deep learning processors are power-hungry and lack biological interpretability, while spiking neural network processors have low accuracy and cannot adapt to deep network models, failing to fully leverage the advantages of ANNs and SNNs to achieve high-efficiency, low-power inference computing.
Design a heterogeneous fusion processor for ANN and SNN. Through heterogeneous fusion processing, selectively limit the running state of ANN and SNN computation kernels according to weight precision and sparsity to realize different heterogeneous fusion modes, including inter-layer and intra-layer fusion, and use asynchronous FIFO for data transmission.
It achieves high-efficiency, low-power inference computing, improves the system's throughput and energy efficiency, fully leverages the advantages of ANN and SNN, and adapts to deep network models.
Smart Images

Figure CN118940203A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an ANN and SNN heterogeneous fusion processor and a heterogeneous fusion processing method. Background Art
[0002] Artificial Neural Network (ANN) is the most widely used artificial intelligence algorithm. It abstracts the human brain neural network from the perspective of information processing and has achieved remarkable results in fields such as image classification, speech recognition, and target detection. As the third generation of neural networks, Spiking Neural Network (SNN) has both biological rationality and event-driven characteristics, providing the potential for low-power reasoning at the edge.
[0003] There are currently two types of processors for neural network reasoning. One is the deep learning processor. This type of accelerator is based on the topological structure of the ANN model and can complete the reasoning of the ANN model. The other is the pulse neural network processor. This type of accelerator is based on the topological structure of the SNN model and can complete the reasoning of the SNN model.
[0004] However, both of the above processors have certain defects when working alone. For example, deep learning processors usually consume high power at the edge and lack biological interpretability. Pulsing neural network processors usually have low accuracy and cannot adapt to deep network models. Their training algorithms are not as mature as ANN. Deep neural network accelerators are usually not used for inference calculations of pulsing neural network models, and pulsing neural network processors are usually not used for inference calculations of artificial neural networks. Therefore, for the inference of a neural network, it is impossible to give full play to the advantages of ANN and SNN to achieve high-efficiency and low-power inference calculations. Summary of the invention
[0005] The present invention provides an ANN and SNN heterogeneous fusion processor and a heterogeneous fusion processing method, which are used to selectively limit the operating states of CNN computing cores and SNN computing cores according to the requirements of weight accuracy and weight sparsity, realize different heterogeneous fusion modes of ANN and SNN, give full play to the advantages of ANN and SNN, and realize high-energy-efficiency and low-power reasoning calculation.
[0006] The present invention provides an ANN and SNN heterogeneous fusion processor, including an ANN computing core, an SNN computing core, a first asynchronous FIFO and a second asynchronous FIFO; The ANN computing core is used for dense computing of a first weight precision, and the SNN computing core is used for sparse computing of a second weight precision, wherein the first weight precision and the second weight precision are used to characterize the bit length of the weight, and the first weight precision is lower than the second weight precision; The input end of the first asynchronous FIFO is connected to the ANN computing core, and the output end is connected to the SNN computing core. The first asynchronous FIFO is used to transmit the calculation result of the ANN computing core to the SNN computing core in the inter-layer fusion mode; The input end of the second asynchronous FIFO is connected to the SNN computing core, and the output end is connected to the ANN computing core. The second asynchronous FIFO is used to transmit the calculation result of the SNN computing core to the ANN computing core in the intra-layer fusion mode; The ANN and SNN heterogeneous fusion processor is used to selectively limit the operating states of the ANN computing core and the SNN computing core according to the weight accuracy and sparsity of the input data to be inferred, so as to realize different heterogeneous fusion modes of the ANN and the SNN.
[0007] According to the ANN and SNN heterogeneous fusion processor provided by the present invention, the bit width of the first asynchronous FIFO is 8 bits, and the bit width of the second asynchronous FIFO is 24 bits.
[0008] According to the ANN and SNN heterogeneous fusion processor provided by the present invention, the heterogeneous fusion mode includes mode 1, mode 2, mode 3, mode 4 and mode 5; Wherein, the mode 1 is that the ANN computing core independently runs the reasoning task; Mode 2 is that the SNN computing core independently runs the reasoning task; The third mode is to encode the calculation result of the ANN calculation core as input and transmit it to the SNN calculation core to achieve inter-layer fusion; The fourth mode is to transmit the calculation result of the SNN calculation core to the ANN calculation core, and add the sparse calculation result of the high weight precision of the SNN calculation core and the dense calculation result of the low weight precision of the ANN calculation core as the final inference result to achieve intra-layer fusion; The mode five is to divide the ANN computing core and the SNN computing core, and inputs of different input value sizes are respectively input into the ANN computing core or the SNN computing core for reasoning.
[0009] According to the ANN and SNN heterogeneous fusion processor provided by the present invention, the mode 5 is specifically: An input threshold is set, and the ANN computing core and the SNN computing core are divided according to the input threshold, and inputs greater than the input threshold are sent to the ANN computing core for reasoning, and inputs less than or equal to the input threshold are sent to the SNN computing core for reasoning.
[0010] According to the ANN and SNN heterogeneous fusion processor provided by the present invention, the ANN computing core includes an input interface module, a first cache module, a systolic array module, an input rearrangement module and a weight rearrangement module; The input interface module configures the input activation value, weight, register parameter, and neuron parameter of the ANN computing core to the first cache module through an asynchronous two-phase handshake protocol; The input rearrangement module and the weight rearrangement module are used to respectively arrange the input activation values and weights read from the first cache module; The systolic array module includes 4 groups of 8×8 fixed-output processing unit arrays, which are used to calculate input activation values and weights to obtain output activation values; the processing unit array can share the same input feature map or the same convolution kernel weights to improve resource utilization of the processing unit array.
[0011] According to the ANN and SNN heterogeneous fusion processor provided by the present invention, the ANN computing core also includes a channel addition module and an ANN neuron module: The channel addition module is used to add the output activation values output by each of the systolic array modules, and transmit the addition result to the ANN neuron module; The ANN neuron module is used to receive the addition result, and after truncating the high-bit output activation value in the addition result, perform operation with multiple activation value functions in a pre-configured lookup table to obtain an operation result; When the heterogeneous fusion mode is mode 3 to realize inter-layer fusion, the operation result is encoded as a pulse and transmitted to the SNN computing core; When the heterogeneous fusion mode is mode four to realize intra-layer fusion, the SNN calculation core transmits multiple 24-bit membrane potential values to the ANN calculation core, and the calculation result and the multiple 24-bit membrane potential values are added in the ANN neuron module.
[0012] According to the ANN and SNN heterogeneous fusion processor provided by the present invention, the SNN computing core includes an input interface module, a second cache module, an 8-bit CSR computing module, and a 1-bit CSR computing module: The input interface module configures the input activation value, weight, register parameter, and neuron parameter of the SNN computing core to the second cache module through an asynchronous two-phase handshake protocol; The 8-bit CSR calculation module is used to complete the decompression calculation of the CSR storage weight with 8-bit input and 8-bit weight; the 1-bit CSR calculation module is used to complete the decompression calculation of the CSR storage weight with 1-bit input and 8-bit weight.
[0013] According to the ANN and SNN heterogeneous fusion processor provided by the present invention, when the heterogeneous fusion mode is mode two, the SNN computing core runs independently, completes weight accumulation through the 1-bit CSR computing module, transmits the weight accumulation result to the SNN neuron module to complete the corresponding neural dynamics behavior, and outputs it through the output interface module; when the heterogeneous fusion mode is mode three, the 8-bit CSR computing module is used to complete weight accumulation, transmits the weight accumulation result to the SNN neuron module to complete the corresponding neural dynamics behavior, and outputs it through the output interface module.
[0014] According to the ANN and SNN heterogeneous fusion processor provided by the present invention, the SNN neuron module includes LI neurons, LIF neurons and IF neurons, and the neural dynamics behavior includes threshold comparison and pulse emission.
[0015] According to the ANN and SNN heterogeneous fusion processor provided by the present invention, before transmitting the weight accumulation result to the SNN neuron module to complete the corresponding neural dynamics behavior, it also includes: Determine whether the current calculation layer is the last layer of the SNN. If so, use LI neurons; if not, use LIF neurons or IF neurons.
[0016] According to the ANN and SNN heterogeneous fusion processor provided by the present invention, the weight accumulation result is transmitted to the SNN neuron module to complete the corresponding neural dynamics behavior, specifically including:
[0017] If it is determined that the current calculation layer is the last layer of the SNN, the weight accumulation result is transmitted to the LI neuron for threshold comparison; If it is determined that the current calculation layer is not the last layer of the SNN, the weight accumulation result is transmitted to the LIF neuron or the IF neuron for threshold comparison and pulse emission.
[0018] The present invention also provides an ANN and SNN heterogeneous fusion processing method, the method is implemented by the ANN and SNN heterogeneous fusion processor as described in any one of the above items, and the method includes: Obtaining weight accuracy and sparsity of the data to be inferred, wherein the weight accuracy is used to characterize the weight bit length of the data to be inferred; Based on the weight accuracy and sparsity of the data to be inferred, the operating states of the ANN computing core and the SNN computing core are selectively limited to achieve different heterogeneous fusion modes of the ANN and the SNN.
[0019] According to the ANN and SNN heterogeneous fusion processing method provided by the present invention, based on the weight accuracy and sparsity of the data to be inferred, the operation states of the ANN computing core and the SNN computing core are selectively limited to realize different heterogeneous fusion modes of the ANN and the SNN, specifically including: If all the data to be inferred are dense computing data of the first weight precision, the ANN computing core runs independently to realize the ANN independent working mode; If all the data to be inferred are sparse calculations of the second weight accuracy, the SNN computing core runs independently to realize the SNN independent working mode; If the data to be inferred includes dense calculation data of the first weight precision and sparse calculation data of the first weight precision, the data to be inferred is selected to be calculated by the ANN calculation core, and the calculation result of the ANN calculation core is encoded as input and transmitted to the SNN calculation core for secondary calculation to achieve inter-layer fusion; If the data to be inferred includes sparse calculation data of the second weight precision and dense calculation data of the second weight precision, the data to be inferred is selected to be calculated by the SNN calculation core, and the calculation result of the SNN calculation core is transmitted to the ANN calculation core, and the sparse calculation result of the second weight precision of the SNN calculation core is added to the dense calculation result of the low weight precision of the ANN calculation core as the final reasoning result, so as to achieve intra-layer fusion; If the data to be inferred includes dense calculation data of a first weight accuracy and sparse calculation data of a second weight accuracy, the data to be inferred is selected according to the weight accuracy and sparsity and is input into the ANN calculation core and the SNN calculation core for reasoning respectively; The first weight accuracy is lower than the second weight accuracy. The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the ANN and SNN heterogeneous fusion processing method described above is implemented.
[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described ANN and SNN heterogeneous fusion processing methods.
[0021] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned ANN and SNN heterogeneous fusion processing methods.
[0022] The ANN and SNN heterogeneous fusion processor and heterogeneous fusion processing method provided by the present invention, the ANN computing core and the SNN computing core are connected through the first asynchronous FIFO and the second asynchronous FIFO, and are adapted to selectively limit the operating states of the ANN computing core and the SNN computing core according to the weight accuracy and sparsity of the data to be inferred, so as to realize different heterogeneous fusion modes of the ANN and the SNN; compared with the deep learning processor, the present invention introduces a pulse neural network for decision classification, which can realize inter-layer pipelining; compared with the pulse neural network accelerator, the present invention introduces an artificial neural network for feature extraction, which can realize inter-layer pipelining; the same task can be decomposed into dense calculations with low bit weights and sparse calculations with high bit weights, further improving the throughput and energy efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0024] Figure 1 It is a structure diagram of the pulse neural network and artificial neural network algorithm provided by the present invention.
[0025] Figure 2 It is a structural schematic diagram of the ANN and SNN heterogeneous fusion processor provided by the present invention.
[0026] Figure 3 It is a structural schematic diagram of the ANN computing core provided by the present invention.
[0027] Figure 4 It is a structural schematic diagram of the SNN computing core provided by the present invention.
[0028] Figure 5 This is the data stream of the output fixed PE array provided by the present invention.
[0029] Figure 6 Schematic diagrams of three dynamic behaviors of neurons provided by the present invention.
[0030] Figure 7 It is a structural schematic diagram of five ANN and SNN heterogeneous fusion modes provided by the present invention.
[0031] Figure 8 It is a flow chart of the ANN and SNN heterogeneous fusion method provided by the present invention.
[0032] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] The fusion reasoning of artificial neural networks and pulse neural networks is a trend. For the reasoning of a neural network, a mixture of ANN and SNN can achieve higher energy efficiency. For example, ANN is used for feature extraction, and then it is sent to SNN for decision classification after being encoded into a pulse train. This is inter-layer fusion. For the same layer of the network, since the circuit of ANNCore is a pulsating array structure, it is suitable for low-bit dense calculations. The circuit of SNN Core is a synaptic cross array structure with compressed storage, so it is suitable for high-bit sparse calculations. The calculation results of the two are added together as the final result, which is intra-layer fusion. Therefore, a hardware platform is needed to complete the above-mentioned inter-layer fusion and intra-layer fusion of ANN and SNN.
[0035] In view of this, the present invention proposes an ANN and SNN heterogeneous fusion processor. Combining the computing characteristics of artificial neural networks and pulse neural networks, the present invention proposes inter-layer fusion and intra-layer fusion mechanisms, which can give full play to the advantages of ANN and SNN and realize high-efficiency and low-power reasoning calculations.
[0036] The present invention will be described in detail below in conjunction with specific implementation modes.
[0037] In order to further understand the heterogeneous fusion mechanism of the two neural network models, the algorithm structures of the pulse neural network and the artificial neural network are first explained.
[0038] Specifically, in order to simplify the circuit hardware design implementation of the two types of neural network algorithms, the algorithms should be simplified accordingly, such as Figure 1 As shown, Figure 1 The principle of the simplified processing scheme of the above algorithm is: requiring the circuit to use fewer resources when implementing it, so as to achieve lower circuit power consumption and faster operation process.
[0039] In a possible embodiment, the neurons of the spiking neural network supported by the present invention are linear LIF models. After the accumulated pulse timing sequence is input, linear leakage operation, threshold comparison and pulse emission are performed. The membrane potential dynamics equation of the LIF neuron model is as follows: V j (t+1)=V j (t)+∑x i w ij -λ j (1)
[0040] Among them, V j (t) is the membrane potential of neuron j at time t, w ij is the weight of the ith synaptic connection, x i is the input of the i-th synaptic connection, λ j is the linear leakage of neuron j.
[0041] Through the above-mentioned settings, the present invention further strengthens the detailed function design based on the LIF model to support most of the pulse neural network algorithms.
[0042] In a possible embodiment, the output activation function of the artificial neural network supported by the present invention is selected as the ReLU function to facilitate the integrated design of the neuron circuit. The dynamic equation of the neuron model of the implemented artificial neural network is as follows: y j =f(∑x i w ij +b j ) (2)
[0043] Among them, y j is the activation value output, x i is the activation value input, w ij is the weight, b j is the bias value, and f(x) is the ReLU type activation function.
[0044] On the basis of the above embodiments, the present invention proposes a processor for heterogeneous fusion of artificial neural network and pulse neural network, which adopts the architecture of on-chip near-memory operation and uses an asynchronous two-phase handshake protocol for input and output.
[0045] In some specific embodiments of the present invention, Figure 2 As shown, Figure 2 It is a structural schematic diagram of the ANN and SNN heterogeneous fusion processor provided by the present invention, and the ANN and SNN heterogeneous fusion processor provided by the present invention includes: an ANN computing core, an SNN computing core, a first asynchronous FIFO and a second asynchronous FIFO; The ANN computing core is used for dense computing of a first weight precision, and the SNN computing core is used for sparse computing of a second weight precision, wherein the first weight precision and the second weight precision are used to characterize the bit length of the weight, and the first weight precision is lower than the second weight precision; The input end of the first asynchronous FIFO is connected to the ANN computing core, and the output end is connected to the SNN computing core. The first asynchronous FIFO is used to transmit the calculation result of the ANN computing core to the SNN computing core in the inter-layer fusion mode; The input end of the second asynchronous FIFO is connected to the SNN computing core, and the output end is connected to the ANN computing core. The second asynchronous FIFO is used to transmit the calculation result of the SNN computing core to the ANN computing core in the intra-layer fusion mode; The ANN and SNN heterogeneous fusion processor is used to selectively limit the operating states of the ANN computing core and the SNN computing core according to the weight accuracy and sparsity of the input data to be inferred, so as to realize different heterogeneous fusion modes of the ANN and the SNN.
[0046] It should be noted that the existing deep learning processors are based on the topology of the ANN model and can complete the reasoning of the ANN model, but their power consumption at the edge is usually high and they lack biological interpretability; the pulse neural network processor is based on the topology of the SNN model and can complete the reasoning of the SNN model, but its accuracy is usually low, it cannot adapt to deep network models, and its training algorithm is not as mature as ANN.
[0047] Therefore, the ANN and SNN heterogeneous fusion processor proposed in the present invention is composed of two cores, which are used to process the reasoning tasks of artificial neural networks (especially convolutional neural networks) and pulse neural networks respectively. The two cores transmit data through two asynchronous FIFOs to complete the fusion mode of various artificial neural networks and pulse neural networks, and both cores can be designed with fine-grained gating units to further reduce system power consumption.
[0048] In some possible embodiments of the present invention, Figure 3 , 4 As shown, Figure 3 is a schematic diagram of the structure of the ANN computing core provided by the present invention, Figure 4 The schematic diagram of the structure of the SNN computing core provided by the present invention is shown in FIG. The ANN computing core and the SNN computing core can run independently in their respective clock domains, and the two transmit data through two asynchronous FIFOs to complete inter-layer fusion and intra-layer fusion.
[0049] In a possible embodiment, the ANN computing core includes an input interface module, a first cache module, a systolic array module, an input rearrangement module, and a weight rearrangement module; The input interface module configures the input activation value, weight, register parameter, and neuron parameter of the ANN computing core to the first cache module through an asynchronous two-phase handshake protocol; The input rearrangement module and the weight rearrangement module are used to respectively arrange the input activation values and weights read from the first cache module; The systolic array module includes 4 groups of 8×8 fixed-output processing unit arrays, which are used to calculate input activation values and weights to obtain output activation values; the processing unit array can share the same input feature map or the same convolution kernel weights to improve resource utilization of the processing unit array.
[0050] In some possible implementations of the present invention, the ANN computing core further includes a channel addition module and an ANN neuron module: The channel addition module is used to add the output activation values output by each of the systolic array modules, and transmit the addition result to the ANN neuron module; The ANN neuron module is used to receive the addition result, and after truncating the high-bit output activation value in the addition result, perform operation with multiple activation value functions in a pre-configured lookup table to obtain an operation result; Specifically, this embodiment provides an implementation method of an ANN computing kernel, which can complete the reasoning task of the ANN model. The input interface module configures the input activation value, weight, register parameter, and neuron parameter to the corresponding cache module through an asynchronous two-phase handshake protocol. The pulsating array module includes 4 groups of 8×8 output fixed-type PE arrays, which can share the same input feature map or share the same convolution kernel weights to maximize the resource utilization of the array. The input rearrangement module and the output weight rearrangement module are used to arrange the inputs and weights read from the cache module to meet the following requirements: Figure 5 The data flow shown. The results calculated by the pulsating array module are sent to the channel addition module and the ANN neuron module. The ANN neuron module will truncate the high-bit activation value and implement various activation functions according to the pre-configured lookup table. The final output result will be sent to the outside of the chip, or encoded into pulses and sent to the SNN computing core (inter-layer fusion mode). In the intra-layer fusion mode, the SNN computing core will send multiple 24-bit membrane potential values to the ANN computing core, and then the ANN neuron module will complete the addition of the results of the two clock domains.
[0051] In some possible implementations of the present invention, the SNN computing core includes an input interface module, a second cache module, an 8-bit CSR computing module, and a 1-bit CSR computing module: The input interface module configures the input activation value, weight, register parameter, and neuron parameter of the SNN computing core to the second cache module through an asynchronous two-phase handshake protocol; The 8-bit CSR calculation module is used to complete the decompression calculation of the CSR storage weight with 8-bit input and 8-bit weight; the 1-bit CSR calculation module is used to complete the decompression calculation of the CSR storage weight with 1-bit input and 8-bit weight.
[0052] In some possible embodiments of the present invention, the SNN neuron module includes LI neurons, LIF neurons and IF neurons, and the neural dynamics behaviors include threshold comparison and pulse emission.
[0053] Specifically, this embodiment provides an implementation method of an SNN computing core, which can complete the reasoning task of the SNN model. The input interface module configures the input activation value, weight, register parameter, and neuron parameter to the corresponding cache module through an asynchronous two-phase handshake protocol. The 8-bit CSR calculation module can complete the decompression calculation of the CSR storage weight with an input of 8 bits and a weight of 8 bits. Similarly, the 1-bit CSR module can complete the decompression calculation of the CSR storage weight with an input of 1 bit and a weight of 8 bits. When the SNN Core module runs independently, the 1-bit CSR module is used to complete the weight accumulation, and then sent to the SNN neuron module to complete the corresponding neural dynamics behavior, and finally transmitted to the outside of the chip through the output interface module. In the inter-layer fusion mode, the 8-bit CSR module is used to complete the weight accumulation (the 8-bit asynchronous FIFO will transmit the calculation results from the ANN Core as the input of the 8-bit CSR module), and then sent to the SNN neuron module to complete the corresponding neural dynamics behavior, and finally transmitted to the outside of the chip through the output interface module. If the current layer is the last layer of the network, LI neurons are usually used, that is, no firing is performed, only threshold comparison is performed. Otherwise, LIF neurons or IF neurons are used. The schematic diagram of the dynamic behavior of these three types of neurons is shown in Figure 6 shown.
[0054] In some possible implementations of the present invention, the bit width of the first asynchronous FIFO is 8 bits, and the bit width of the second asynchronous FIFO is 24 bits.
[0055] Specifically, this embodiment provides an implementation method of an asynchronous FIFO. In a possible embodiment, the ANN and SNN heterogeneous fusion processor provided by the present invention includes two asynchronous FIFO modules. The 8-bit wide asynchronous FIFO is used in the inter-layer fusion mode to transmit the result of the ANN Core to the SNN Core through the 8-bit asynchronous FIFO. The 24-bit wide asynchronous FIFO is used in the intra-layer fusion mode to transmit the result of the SNN Core to the ANN Core through the 24-bit asynchronous FIFO.
[0056] On the basis of the above-mentioned embodiment, according to the weight accuracy and sparsity of the data to be inferred, the operating states of the ANN computing core and the SNN computing core are selectively limited to determine the heterogeneous fusion mode of the ANN and the SNN.
[0057] In some possible implementations of the present invention, the heterogeneous fusion modes include mode 1, mode 2, mode 3, mode 4, and mode 5; Wherein, the mode 1 is that the ANN computing core independently runs the reasoning task; Mode 2 is that the SNN computing core independently runs the reasoning task; The third mode is to encode the calculation result of the ANN calculation core as input and transmit it to the SNN calculation core to achieve inter-layer fusion; The fourth mode is to transmit the calculation result of the SNN calculation core to the ANN calculation core, and add the sparse calculation result of the high weight precision of the SNN calculation core and the dense calculation result of the low weight precision of the ANN calculation core as the final inference result to achieve intra-layer fusion; The mode five is to divide the ANN computing core and the SNN computing core, and inputs of different input value sizes are respectively input into the ANN computing core or the SNN computing core for reasoning.
[0058] In a possible embodiment, mode five is specifically: setting an input threshold, dividing the ANN computing core and the SNN computing core according to the input threshold, sending inputs greater than the input threshold to the ANN computing core for reasoning, and sending inputs less than or equal to the input threshold to the SNN computing core for reasoning.
[0059] Specifically, this embodiment provides an implementation method for the heterogeneous fusion mode of ANN and SNN. By introducing a pulse neural network for decision-making and classification, inter-layer pipelining can be achieved. By introducing an artificial neural network for feature extraction, inter-layer pipelining can be achieved. The same task can be decomposed into dense calculations with low bit weights and sparse calculations with high bit weights, further improving the system's throughput and energy efficiency.
[0060] In a possible embodiment, Figure 7 As shown, Figure 7 It is a schematic diagram of the structure of five ANN and SNN heterogeneous fusion modes provided by the present invention. Figure 7 The processor has five working modes. Mode 1 and Mode 2 can independently run ANN or SNN reasoning tasks. Mode 3 encodes the results of ANN Core into input and transmits them to SNN Core to realize inter-layer pipeline operation. Mode 4 transmits the results of SNN Core to ANN Core, and adds the sparse calculation results of SNN Core with high weight precision and the dense calculation results of ANN Core with low weight precision as the final reasoning results. Mode 5 divides the neural network model, sends the larger input values to ANN Core for reasoning, and sends the smaller input values to SNN Core for reasoning to maximize energy efficiency.
[0061] In a possible embodiment, when the heterogeneous fusion mode is mode three to realize inter-layer fusion, the operation result is encoded as a pulse and transmitted to the SNN computing core; When the heterogeneous fusion mode is mode four to implement intra-layer fusion, the SNN calculation core transmits multiple 24-bit membrane potential values to the ANN calculation core, and the calculation result and the multiple 24-bit membrane potential values are added in the ANN neuron module.
[0062] In a possible embodiment, when the heterogeneous fusion mode is the second mode, the SNN computing core runs independently, completes weight accumulation through the 1-bit CSR computing module, transmits the weight accumulation result to the SNN neuron module to complete the corresponding neural dynamics behavior, and outputs it through the output interface module; When the heterogeneous fusion mode is mode three, the 8-bit CSR calculation module is used to complete weight accumulation, and the weight accumulation result is transmitted to the SNN neuron module to complete the corresponding neural dynamics behavior, and output through the output interface module.
[0063] In a possible embodiment, before transmitting the weight accumulation result to the SNN neuron module to complete the corresponding neural dynamics behavior, it also includes: Determine whether the current calculation layer is the last layer of the SNN. If so, use LI neurons; if not, use LIF neurons or IF neurons.
[0064] In some possible implementations of the present invention, the weight accumulation result is transmitted to the SNN neuron module to complete the corresponding neural dynamics behavior, specifically including
[0065] If it is determined that the current calculation layer is the last layer of the SNN, the weight accumulation result is transmitted to the LI neuron for threshold comparison; If it is determined that the current calculation layer is not the last layer of the SNN, the weight accumulation result is transmitted to the LIF neuron or the IF neuron for threshold comparison and pulse emission.
[0066] It can be seen from the above embodiments that the present invention aims to explore a processor that heterogeneously integrates an artificial neural network model and a pulse neural network model from the perspective of digital circuit design. Artificial neural networks and pulse neural networks each have their own areas of expertise. Artificial neural networks such as convolutional neural networks are good at feature extraction, while pulse neural networks are better at decision classification. Artificial neural networks use multipliers to complete the multiplication and addition operations of input feature values and weights, while pulse neural networks only need adders because their inputs are encoded into pulse trains of multiple time steps (timestep), and thus have event-driven characteristics. Therefore, the fusion reasoning of artificial neural networks and pulse neural networks can give full play to their respective advantages.
[0067] The present invention has achieved beneficial effects through the above settings. Compared with the deep learning processor, the processor designed by the present invention, which is a heterogeneous fusion of the artificial neural network and the pulse neural network, introduces the pulse neural network for decision classification, and can achieve inter-layer pipelining; compared with the pulse neural network accelerator, the processor designed by the present invention, which is a heterogeneous fusion of the artificial neural network and the pulse neural network, introduces the artificial neural network for feature extraction, and can achieve inter-layer pipelining. At the same time, by heterogeneously fusing the two neural network models, the same task can be decomposed into dense calculations of low bit weights and sparse calculations of high bit weights, further improving the throughput and energy efficiency of the system.
[0068] Based on the above embodiments, in some possible implementations of the present invention, such as Figure 8 As shown, Figure 8 : is a flow chart of the ANN and SNN heterogeneous fusion method provided by the present invention, the method is implemented by the ANN and SNN heterogeneous fusion processor as described in any one of the above items, and the method includes: Step 810: Obtain weight accuracy and sparsity of the data to be inferred, wherein the weight accuracy is used to characterize the weight bit length of the data to be inferred; Step 820 selectively limits the operating states of the ANN computing core and the SNN computing core based on the weight accuracy and sparsity of the data to be inferred, so as to realize different heterogeneous fusion modes of the ANN and the SNN.
[0069] In a possible embodiment, based on the weight accuracy and sparsity of the data to be inferred, the operation states of the ANN computing core and the SNN computing core are selectively limited to achieve different heterogeneous fusion modes of the ANN and the SNN, specifically including: If all the data to be inferred are dense computing data of the first weight precision, the ANN computing core runs independently to realize the ANN independent working mode; If all the data to be inferred are sparse calculations of the second weight accuracy, the SNN computing core runs independently to realize the SNN independent working mode; If the data to be inferred includes dense calculation data of the first weight precision and sparse calculation data of the first weight precision, the data to be inferred is selected to be calculated by the ANN calculation core, and the calculation result of the ANN calculation core is encoded as input and transmitted to the SNN calculation core for secondary calculation to achieve inter-layer fusion; If the data to be inferred includes sparse calculation data of the second weight precision and dense calculation data of the second weight precision, the data to be inferred is selected to be calculated by the SNN calculation core, and the calculation result of the SNN calculation core is transmitted to the ANN calculation core, and the sparse calculation result of the second weight precision of the SNN calculation core is added to the dense calculation result of the low weight precision of the ANN calculation core as the final reasoning result, so as to achieve intra-layer fusion; If the data to be inferred includes dense calculation data of a first weight accuracy and sparse calculation data of a second weight accuracy, the data to be inferred is selected according to the weight accuracy and sparsity and is input into the ANN calculation core and the SNN calculation core for reasoning respectively; The first weight accuracy is lower than the second weight accuracy.
[0070] It is worth noting that the implementation principle and beneficial effects of the ANN and SNN heterogeneous fusion processing method provided in the embodiment of the present invention are similar to the implementation principle and beneficial effects of the ANN and SNN heterogeneous fusion processor shown in the above embodiment. Please refer to the implementation principle and beneficial effects of the ANN and SNN heterogeneous fusion processor shown in the above embodiment, and no further details will be given here.
[0071] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 complete mutual communication through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the ANN and SNN heterogeneous fusion processing method, the method comprising: obtaining the weight accuracy and sparsity of the data to be inferred, wherein the weight accuracy is used to characterize the weight bit length of the data to be inferred; based on the weight accuracy and sparsity of the data to be inferred, selectively limiting the operating states of the ANN computing core and the SNN computing core to achieve different heterogeneous fusion modes of the ANN and the SNN.
[0072] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0073] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the ANN and SNN heterogeneous fusion processing method provided by the above methods, and the method includes: obtaining the weight accuracy and sparsity of the data to be inferred, wherein the weight accuracy is used to characterize the weight bit length of the data to be inferred; based on the weight accuracy and sparsity of the data to be inferred, selectively limiting the operating states of the ANN computing core and the SNN computing core to achieve different heterogeneous fusion modes of the ANN and the SNN.
[0074] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the ANN and SNN heterogeneous fusion processing methods provided by the above-mentioned methods, the method comprising: obtaining the weight accuracy and sparsity of the data to be inferred, wherein the weight accuracy is used to characterize the weight bit length of the data to be inferred; based on the weight accuracy and sparsity of the data to be inferred, selectively limiting the operating states of the ANN computing core and the SNN computing core to achieve different heterogeneous fusion modes of the ANN and the SNN.
[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor. Through the description of the above implementation methods, a person of ordinary skill in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.
[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An ANN and SNN heterogeneous fusion processor, characterized in that: It includes an ANN computing core, an SNN computing core, a first asynchronous FIFO and a second asynchronous FIFO; The ANN computing core is used for dense computing of a first weight precision, and the SNN computing core is used for sparse computing of a second weight precision, wherein the first weight precision and the second weight precision are used to characterize the bit length of the weight, and the first weight precision is lower than the second weight precision; The input end of the first asynchronous FIFO is connected to the ANN computing core, and the output end is connected to the SNN computing core. The first asynchronous FIFO is used to transmit the calculation result of the ANN computing core to the SNN computing core in the inter-layer fusion mode; The input end of the second asynchronous FIFO is connected to the SNN computing core, and the output end is connected to the ANN computing core. The second asynchronous FIFO is used to transmit the calculation result of the SNN computing core to the ANN computing core in the intra-layer fusion mode; The ANN and SNN heterogeneous fusion processor is used to selectively limit the operating states of the ANN computing core and the SNN computing core according to the weight accuracy and sparsity of the input data to be inferred, so as to realize different heterogeneous fusion modes of the ANN and the SNN.
2. The ANN and SNN heterogeneous fusion processor according to claim 1, characterized in that: The bit width of the first asynchronous FIFO is 8 bits, and the bit width of the second asynchronous FIFO is 24 bits.
3. The ANN and SNN heterogeneous fusion processor according to claim 2, characterized in that: The heterogeneous fusion modes include mode 1, mode 2, mode 3, mode 4 and mode 5; Wherein, the mode 1 is that the ANN computing core independently runs the reasoning task; Mode 2 is that the SNN computing core independently runs the reasoning task; The third mode is to encode the calculation result of the ANN calculation core as input and transmit it to the SNN calculation core to achieve inter-layer fusion; The fourth mode is to transmit the calculation result of the SNN calculation core to the ANN calculation core, and add the sparse calculation result of the high weight precision of the SNN calculation core and the dense calculation result of the low weight precision of the ANN calculation core as the final inference result to achieve intra-layer fusion; The mode five is to divide the ANN computing core and the SNN computing core, and inputs of different input value sizes are respectively input into the ANN computing core or the SNN computing core for reasoning.
4. The ANN and SNN heterogeneous fusion processor according to claim 3, characterized in that: The mode five is specifically as follows: setting an input threshold, dividing the ANN computing core and the SNN computing core according to the input threshold, sending inputs greater than the input threshold to the ANN computing core for reasoning, and sending inputs less than or equal to the input threshold to the SNN computing core for reasoning.
5. The ANN and SNN heterogeneous fusion processor according to claim 3, characterized in that: The ANN computing core includes an input interface module, a first cache module, a systolic array module, an input rearrangement module and a weight rearrangement module; The input interface module configures the input activation value, weight, register parameter, and neuron parameter of the ANN computing core to the first cache module through an asynchronous two-phase handshake protocol; The input rearrangement module and the weight rearrangement module are used to respectively arrange the input activation values and weights read from the first cache module; The systolic array module includes 4 groups of 8×8 fixed-output processing unit arrays, which are used to calculate input activation values and weights to obtain output activation values; the processing unit array can share the same input feature map or the same convolution kernel weights to improve resource utilization of the processing unit array.
6. The ANN and SNN heterogeneous fusion processor according to claim 5, characterized in that: The ANN computing core also includes a channel addition module and an ANN neuron module; The channel addition module is used to add the output activation values output by each of the systolic array modules, and transmit the addition result to the ANN neuron module; The ANN neuron module is used to receive the addition result, and after truncating the high-bit output activation value in the addition result, perform operation with multiple activation value functions in a pre-configured lookup table to obtain an operation result; When the heterogeneous fusion mode is mode 3 to realize inter-layer fusion, the operation result is encoded as a pulse and transmitted to the SNN computing core; When the heterogeneous fusion mode is mode four to realize intra-layer fusion, the SNN calculation core transmits multiple 24-bit membrane potential values to the ANN calculation core, and the calculation result and the multiple 24-bit membrane potential values are added in the ANN neuron module.
7. The ANN and SNN heterogeneous fusion processor according to claim 3, characterized in that: The SNN computing core includes an input interface module, a second cache module, an 8-bit CSR computing module, and a 1-bit CSR computing module: The input interface module configures the input activation value, weight, register parameter, and neuron parameter of the SNN computing core to the second cache module through an asynchronous two-phase handshake protocol; The 8-bit CSR calculation module is used to complete the decompression calculation of the CSR storage weight with 8-bit input and 8-bit weight; the 1-bit CSR calculation module is used to complete the decompression calculation of the CSR storage weight with 1-bit input and 8-bit weight.
8. The ANN and SNN heterogeneous fusion processor according to claim 7, characterized in that: When the heterogeneous fusion mode is the second mode, the SNN computing core runs independently, completes weight accumulation through the 1-bit CSR computing module, transmits the weight accumulation result to the SNN neuron module to complete the corresponding neural dynamics behavior, and outputs it through the output interface module; When the heterogeneous fusion mode is mode three, the 8-bit CSR calculation module is used to complete weight accumulation, and the weight accumulation result is transmitted to the SNN neuron module to complete the corresponding neural dynamics behavior, and output through the output interface module.
9. The ANN and SNN heterogeneous fusion processor according to claim 8, characterized in that: The SNN neuron module includes LI neurons, LIF neurons and IF neurons, and the neural dynamics behavior includes threshold comparison and pulse emission.
10. The ANN and SNN heterogeneous fusion processor according to claim 9, characterized in that: Before transmitting the weight accumulation result to the SNN neuron module to complete the corresponding neural dynamics behavior, it also includes: Determine whether the current calculation layer is the last layer of the SNN. If so, use LI neurons; if not, use LIF neurons or IF neurons.
11. The ANN and SNN heterogeneous fusion processor according to claim 10, characterized in that: The weight accumulation result is transmitted to the SNN neuron module to complete the corresponding neural dynamics behavior, specifically including: If it is determined that the current calculation layer is the last layer of the SNN, the weight accumulation result is transmitted to the LI neuron for threshold comparison; if it is determined that the current calculation layer is not the last layer of the SNN, the weight accumulation result is transmitted to the LIF neuron or the IF neuron for threshold comparison and pulse emission.
12. A method for heterogeneous fusion of ANN and SNN, characterized in that: The method is implemented by the ANN and SNN heterogeneous fusion processor according to any one of claims 1 to 11, and the method includes: Obtaining weight accuracy and sparsity of the data to be inferred, wherein the weight accuracy is used to characterize the weight bit length of the data to be inferred; Based on the weight accuracy and sparsity of the data to be inferred, the operating states of the CNN computing core and the SNN computing core are selectively limited to achieve different heterogeneous fusion modes of the ANN and the SNN.
13. The ANN and SNN heterogeneous fusion processing method according to claim 12, characterized in that: Based on the weight accuracy and sparsity of the data to be inferred, the operation states of the ANN computing core and the SNN computing core are selectively limited to achieve different heterogeneous fusion modes of the ANN and the SNN, specifically including: If all the data to be inferred are dense computing data of the first weight precision, the ANN computing core runs independently to realize the ANN independent working mode; If all the data to be inferred are sparse calculations of the second weight accuracy, the SNN computing core runs independently to realize the SNN independent working mode; If the data to be inferred includes dense calculation data of the first weight precision and sparse calculation data of the first weight precision, the data to be inferred is selected to be calculated by the ANN calculation core, and the calculation result of the ANN calculation core is encoded as input and transmitted to the SNN calculation core for secondary calculation to achieve inter-layer fusion; If the data to be inferred includes sparse calculation data of the second weight precision and dense calculation data of the second weight precision, the data to be inferred is selected to be calculated by the SNN calculation core, and the calculation result of the SNN calculation core is transmitted to the ANN calculation core, and the sparse calculation result of the second weight precision of the SNN calculation core is added to the dense calculation result of the low weight precision of the ANN calculation core as the final reasoning result, so as to achieve intra-layer fusion; If the data to be inferred includes dense calculation data of a first weight accuracy and sparse calculation data of a second weight accuracy, the data to be inferred is selected according to the weight accuracy and sparsity and is input into the ANN calculation core and the SNN calculation core for reasoning respectively; The first weight accuracy is lower than the second weight accuracy.
Citation Information
Patent Citations
Processor integrating artificial neural network and spiking neural network
CN114781633A
Cross-clock domain communication transmission method and system based on asynchronous handshake protocol
CN116132552A
ANN / SNN accelerator and ANN / SNN neuron acceleration calculation method
CN118627562A
