A pulse neural network processor with unified architecture
By designing a unified architecture spiking neural network processor that combines sparse and dense architectures, it is possible to efficiently process sparse and dense spiking neural networks on the same chip, solving the problem of insufficient diversity support in existing technologies and achieving low-power and high-efficiency computing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2023-01-18
- Publication Date
- 2026-05-29
AI Technical Summary
Existing spiking neural network processors lack a unified architecture and cannot simultaneously support sparse and dense architectures, resulting in insufficient support for the diversity of spiking neural networks.
Design a spiking neural network processor with a unified architecture, including a LIF neuron module, a spiking collector module, an SRAM controller module, and a configuration module. It achieves the fusion of sparse and dense architectures by using spiking-driven computation in a sparse architecture and matrix multiplication in a dense architecture, and utilizes a three-level indexed storage and pipelined computation method.
Under the same chip architecture, it can support both sparse and dense architectures, reduce task energy consumption, flexibly configure to adapt to different network requirements, sparse architecture has low power consumption and dense architecture has fast computing, achieving more efficient energy utilization.
Smart Images
Figure CN116306835B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of spiking neural network processing technology, and in particular to a spiking neural network processor with a unified architecture. Background Technology
[0002] A spiking neural network processor is a processor specifically designed for processing spiking neural network calculations. Compared to general-purpose processors, it offers advantages such as higher efficiency and lower power consumption.
[0003] There has been a great deal of research in academia regarding processor design for spiking neural networks. Current research can be broadly categorized into two main types:
[0004] (1) A pulse-driven computing paradigm processor designed for the sparse characteristics of spiking neural networks;
[0005] (2) A matrix computation processor designed for densely connected spiking neural networks.
[0006] In the study of (1), the chip often uses SRAM memory to store the weights and target neuron addresses of the spiking neural network. The pulse finds the corresponding weight and target neuron address through the index, obtains the membrane potential of the target neuron, and completes the neuron's calculation. The calculation at this time is pulse-driven, that is, the calculation is only performed when the neuron receives a pulse, so it can achieve better results in sparse spiking neural networks.
[0007] In the study (2), the connections between neurons are very dense, and the chip defaults to a fully connected network. The chip often uses SRAM memory to store only the weights of the spiking neural network, without storing the address of the target neuron. In this case, the computation of the neuron is achieved by performing matrix operations on the pulses and weights.
[0008] However, research on spiking neural network processors for these two computing paradigms is independent. Currently, there is no unified architecture for spiking neural network processors that unifies sparse and dense architectures, which also leads to insufficient support for the diversity of spiking neural networks. Summary of the Invention
[0009] The purpose of this invention is to overcome the problems of the prior art and provide a spiking neural network processor with a unified architecture to solve the technical problem that traditional technical solutions do not have a unified architecture for spiking neural network processors to unify sparse and dense architectures, and that this results in insufficient support for the diversity of spiking neural networks.
[0010] The above objectives are achieved through the following technical solutions:
[0011] A spiking neural network processor with a unified architecture includes a LIF neuron module, a spiking collector module, an SRAM controller module, an SRAM module, and a configuration module;
[0012] The LIF neuron module is used to perform calculations for LIF neurons;
[0013] The pulse collector module is used to store pulses;
[0014] The SRAM controller module is used to enable communication between the LIF neuron module, the pulse collector module, and the SRAM module; the SRAM module includes a weight SRAM, an entry SRAM, and a neuron SRAM.
[0015] The configuration module is used to configure the chip architecture, which includes sparse architecture and dense architecture.
[0016] Furthermore, in the sparse architecture, the computation of LIF neurons is driven by pulses; in the dense architecture, the computation of LIF neurons is completed using matrix multiplication.
[0017] Furthermore, in the sparse architecture, both the weights and destination addresses of neurons are stored in the weight SRAM; in the dense architecture, all weights of neurons are stored in the weight SRAM, but the destination addresses are not stored.
[0018] Furthermore, the sparse architecture includes a FIFO, the entry SRAM, the weight SRAM, the neuron SRAM, a decoder, a register, an adder, a selector, an integrator, a comparator, a multiplier, and a pulse generator.
[0019] Furthermore, the sparse architecture employs a three-level index storage, using the pulse ID as an address index to the entry SRAM to obtain the starting address and length of the weight SRAM. The decoder then decodes the weight SRAM to obtain its address and finds the corresponding weight and the address of the target neuron. The membrane potential and bias of the corresponding neuron are then found through the address of the target neuron, thus completing the three-level indexing.
[0020] Furthermore, in the sparse architecture, the computation of the LIF neuron module is divided into a J-stage and a V-stage by a state machine, wherein the J-stage is used to accumulate weights, and the V-stage is used to accumulate membrane potential and bias, and to determine whether to fire a pulse or decay.
[0021] Furthermore, in the dense architecture, the computation of LIF neurons is performed using matrix multiplication, specifically as follows:
[0022] The register is set to a length of 64 bits, the weight SRAM has a bit width of 64 bits, and the pulse is stored in the register; the calculation process includes:
[0023] Starting with the first neuron, a 64-bit pulse and a 64-bit fan-in weight of the current neuron are read out each clock cycle; the pulse and the fan-in weight of the neuron are then logically ANDed to obtain the effective weight (the weight used for calculation).
[0024] Sum all the effective weights to get the weight sum;
[0025] Then add the weighted sum to the membrane potential of the current neuron;
[0026] If the fan-in weights of the current neuron have been accumulated, then the sum of the membrane potentials just mentioned is added to the bias to obtain the final membrane potential.
[0027] Otherwise, return it to the adder to continue accumulating it with other valid fan-in weights of the current neuron until all fan-in weights of the current neuron have been accumulated;
[0028] Finally, it is determined whether the membrane potential of the neuron exceeds the threshold to decide whether to fire a pulse or perform decay.
[0029] Once the current neuron has finished its calculation, the next neuron is calculated, and so on, until all neurons have been calculated.
[0030] Furthermore, the threshold is a custom threshold.
[0031] Furthermore, the entire computation process of the dense architecture is pipelined, which means that it can process 64-bit pulses per clock cycle.
[0032] Beneficial effects
[0033] The present invention provides a spiking neural network processor with a unified architecture, which can simultaneously support sparse and dense architectures within the same chip architecture. Using the storage and pulse-driven computation of the sparse architecture, it achieves lower energy consumption when processing sparse spiking neural networks; using the storage and matrix operations of the dense architecture, it achieves lower energy consumption when processing dense spiking neural networks. It integrates the advantages of both architectures into a single chip, and also has the following advantages:
[0034] 1. Users can choose the architecture configuration according to different network requirements;
[0035] 2. When one architecture is used, the modules dedicated to the other architecture will be in a dormant state to maintain the lowest power consumption;
[0036] 3. Sparse architecture consumes less power, while dense architecture has faster computing speed, and these characteristics can be selected for different applications. Attached Figure Description
[0037] Figure 1 This is a schematic diagram of the weighted SRAM in a sparse storage configuration of a spiking neural network processor with a unified architecture as described in this invention.
[0038] Figure 2 This is a schematic diagram of the weight SRAM in a dense storage configuration in a spiking neural network processor with a unified architecture as described in this invention.
[0039] Figure 3 This is an architecture diagram of a spiking neural network processor with a unified architecture as described in this invention;
[0040] Figure 4 This is a flowchart illustrating the computation process in the sparse architecture of a spiking neural network processor with a unified architecture as described in this invention.
[0041] Figure 5 This is a flowchart of the computation process in the dense architecture of a spiking neural network processor with a unified architecture as described in this invention. Detailed Implementation
[0042] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. The described embodiments are merely some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] like Figure 3 As shown, a spiking neural network processor with a unified architecture effectively supplements the current lack of a fusion architecture that combines sparse and dense architectures in spiking neural network processor research. This spiking neural network processor is based on sparse and dense storage and is designed with a unified structure that can support both sparse and dense architectures.
[0044] It includes a LIF neuron module, a spike collector module, an SRAM controller module, an SRAM module, and a configuration module; among which:
[0045] The LIF neuron module is used to perform calculations for LIF neurons;
[0046] The pulse collector module is used to store pulses;
[0047] The SRAM controller module is used to enable communication between the LIF neuron module, the pulse collector module, and the SRAM module; the SRAM module includes a weight SRAM, an entry SRAM, and a neuron SRAM.
[0048] The configuration module is used to configure the chip architecture, which includes sparse architecture and dense architecture.
[0049] Specifically, this processor includes sparse and dense architectures, wherein:
[0050] Sparse architecture corresponds to the processing of sparse spiking neural network computation, using spiking to drive the computation of LIF neurons;
[0051] Dense architecture corresponds to the processing of dense spiking neural network calculations, which uses matrix multiplication to complete the calculation of LIF neurons.
[0052] like Figure 1 As shown, in a sparse architecture, the connections between neurons are sparse. In this case, a sparse storage configuration is used, and the weights and destination addresses of neurons are stored in the weight SRAM. The corresponding weights and the ID of the destination neuron can be found through the pulse index.
[0053] like Figure 2 As shown, in a dense architecture, the connections between neurons are fully connected by default. In this case, when using a dense storage configuration, all the weights of the neurons are stored in the weight SRAM, while the destination address is not stored.
[0054] At the same time, the storage method has also changed. Now, the weight corresponding to a certain address is the fan-in weight of a certain neuron. Every 3 addresses in the weight SRAM store all the fan-in weights of a neuron, and the fan-in weights of each neuron are stored in the weight SRAM in this way.
[0055] like Figure 4 As shown, the sparse architecture includes FIFO, the entry SRAM, the weight SRAM, the neuron SRAM, a decoder, a register, an adder, a selector, an integrator, a comparator, a multiplier, and a pulse generator.
[0056] The sparse architecture employs a three-level indexed storage. First, the pulse ID is used as an address index to the entry SRAM, obtaining the starting address and length of the weight SRAM. Decoding yields the address of the weight SRAM, which in turn locates the corresponding weight and the address of the target neuron. Finally, using the target neuron's address, the membrane potential and bias of the corresponding neuron are found, completing the three-level indexing.
[0057] In the sparse architecture, the computation of LIF neurons is divided into two stages by a state machine: the J stage and the V stage. The J stage accumulates the weights, while the V stage accumulates the membrane potential and bias, determining whether to fire a pulse or decay.
[0058] All pulses are stored in a FIFO (First-In-First-Out) buffer. At startup, the pulse is read first, used as an address to access the entry SRAM, obtaining the entry address and address length of the weight SRAM. This is then decoded to obtain the address, and subsequently, the weight SRAM is accessed to obtain the weights and the target neuron address. After decoding, the weights and address are sent to registers and selectors, respectively. The target neuron address accesses the neuron SRAM to obtain the membrane potential and bias. The membrane potential and weights are added together in an adder to obtain the weighted membrane potential. The updated membrane potential and bias are then stored back into the neuron SRAM.
[0059] In phase V, the membrane potential and bias are retrieved from the neuron's SRAM and accumulated to obtain the updated membrane potential for phase V. This membrane potential then passes through a register and is compared with a threshold to determine whether to fire a pulse. If the membrane potential is greater than or equal to the threshold, the pulse generator is alerted to generate a pulse; if the membrane potential is less than the threshold, it is sent to a multiplier and multiplied by a decay constant to obtain the decayed membrane potential. Finally, depending on whether a pulse is fired, a decision is made whether to update the membrane potential to 0 or to the decayed value, and write it back to the neuron's SRAM.
[0060] When the current chip needs to process dense spiking neural networks, the user configures the chip to a dense architecture through the configuration module.
[0061] In dense architectures, chips perform calculations using matrix operations.
[0062] Figure 5 It demonstrates how computation is performed in a dense architecture, assuming that the register length is 64 bits and the bit width of the weighted SRAM is also 64 bits.
[0063] First, the pulses are stored in registers instead of FIFOs because this allows for greater flexibility in pulse operation.
[0064] The calculation will start from the first neuron, and 64-bit pulses and 64-bit fan-in weights of the current neuron will be read out each clock cycle.
[0065] The effective weight (the weight to be calculated) is obtained by performing a logical AND operation between the impulse and the fan-in weight of the neuron.
[0066] Sum all the effective weights to get the weight sum;
[0067] Then add the weighted sum to the membrane potential of the current neuron;
[0068] If the fan-in weights of the current neuron have been accumulated, then the sum of the membrane potentials just accumulated is added to the bias to obtain the final membrane potential.
[0069] Otherwise, return it to the adder to continue accumulating it with other valid fan-in weights of the current neuron until all fan-in weights of the current neuron have been accumulated;
[0070] Finally, it is determined whether the membrane potential of the neuron exceeds the threshold to decide whether to fire a pulse or perform decay.
[0071] Once the current neuron has finished its calculation, the next neuron is calculated, and so on, until all neurons have been calculated.
[0072] The entire computation process of the dense architecture is pipelined, which means that it can process 64-bit pulses per clock cycle.
[0073] The spiking neural network processor with a unified architecture provided by the present invention can simultaneously support sparse and dense architectures under the same chip architecture. Using the storage and pulse-driven computation of the sparse architecture, lower task energy consumption can be achieved when processing sparse spiking neural networks; using the storage and matrix operations of the dense architecture, lower task energy consumption can be achieved when processing dense spiking neural networks. The advantages of the two architectures can be integrated into a single chip.
[0074] The above description is merely illustrative of the embodiments of the present invention and is not intended to limit the present invention. For those skilled in the art, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A spiking neural network processor with a unified architecture, characterized in that, It includes a LIF neuron module, a pulse collector module, an SRAM controller module, an SRAM module, and a configuration module; The LIF neuron module is used to perform calculations for LIF neurons; The pulse collector module is used to store pulses; The SRAM controller module is used to enable communication between the LIF neuron module, the pulse collector module, and the SRAM module; the SRAM module includes a weight SRAM, an entry SRAM, and a neuron SRAM. The configuration module is used to configure the chip architecture, which includes sparse architecture and dense architecture. The sparse architecture includes a FIFO, the entry SRAM, the weight SRAM, the neuron SRAM, a decoder, a register, an adder, a selector, an integrator, a comparator, a multiplier, and a pulse generator; The sparse architecture employs a three-level index storage. The pulse ID is used as an address index to the entry SRAM to obtain the starting address and length of the weight SRAM. The decoder decodes the weight SRAM to obtain its address and finds the corresponding weight and the address of the target neuron. The membrane potential and bias of the corresponding neuron are found through the address of the target neuron, thus completing the three-level index. In the dense architecture, matrix multiplication is used to compute LIF neurons, specifically: The register is set to a length of 64 bits, the weight SRAM has a bit width of 64 bits, and the pulse is stored in the register; the calculation process includes: Starting from the first neuron, 64-bit pulses and 64-bit fan-in weights of the current neuron are read out each clock cycle; The effective weight is obtained by performing a logical AND operation between the pulse and the fan-in weight of the neuron. Sum all the effective weights to get the weight sum; Then add the weighted sum to the membrane potential of the current neuron; If the fan-in weights of the current neuron have been accumulated, then the sum of the membrane potentials just mentioned is added to the bias to obtain the final membrane potential. Otherwise, return it to the adder to continue accumulating it with other valid fan-in weights of the current neuron until all fan-in weights of the current neuron have been accumulated; Finally, it is determined whether the membrane potential of the neuron exceeds the threshold to decide whether to fire a pulse or perform decay. Once the current neuron has finished its calculation, the next neuron is calculated, and so on, until all neurons have been calculated.
2. The spiking neural network processor with a unified architecture according to claim 1, characterized in that, In the sparse architecture, computation of LIF neurons is driven by spiking.
3. A spiking neural network processor with a unified architecture according to claim 1, characterized in that, In the sparse architecture, both the weights and destination addresses of neurons are stored in the weight SRAM; in the dense architecture, all weights of neurons are stored in the weight SRAM, but the destination addresses are not stored.
4. A spiking neural network processor with a unified architecture according to claim 1, characterized in that, In the sparse architecture, the computation of the LIF neuron module is divided into a J-stage and a V-stage by a state machine. The J-stage is used to accumulate weights, and the V-stage is used to accumulate membrane potential and bias, and to determine whether to fire a pulse or decay.
5. A spiking neural network processor with a unified architecture according to claim 1, characterized in that, The threshold is a custom threshold.
6. A spiking neural network processor with a unified architecture according to claim 1, characterized in that, The entire computation process of the dense architecture is pipelined, meaning that each clock cycle can process a 64-bit pulse.