Style frame chip applied to spiking neural network
By designing a style frame chip in a pulsed neural network, using sparseness and locality characteristics for speculative rapid forwarding and conditional recovery, dynamically adjusting the time window, the problems of low energy efficiency and high latency of SNN on edge devices are solved, and efficient calculations are achieved and low latency inference process.
Patent Information
- Application Number
- CN202411171784.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-26
AI Technical Summary
The deployment of existing pulsed neural networks (SNNs) on edge devices is plagued by problems of low energy efficiency and high latency, especially due to increased inference latency due to computations across multiple time steps.
A style frame chip is designed, including a slice unit array, a pooling unit, an accumulation unit and a global buffer. Through components such as synaptic array groups, dynamic schedulers and multiplexers, speculation is used to utilize the characteristics of sparse pulse data. Fast forwarding and conditional recovery, dynamically adjust the time window, and optimize computing efficiency.
It realizes high-energy-efficient and low-latency computing, while reducing unnecessary computing overhead, improving computing efficiency, and greatly reducing storage and transmission resources through sparseness and local compression methods, significantly improving the computing efficiency of SNN and Energy efficiency.
Smart Images

Figure CN118940813A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of pulse neural networks, and in particular to a pattern framework chip applied to pulse neural networks. Background Art
[0002] Spiking Neural Networks (SNNs) are considered as an energy-efficient alternative to Deep Neural Networks (DNNs). By adopting event-driven information processing, SNNs can significantly reduce the computational requirements associated with DNNs while still achieving comparable performance. However, current SNNs mainly prioritize high accuracy and large sparsity by building complex neuron models that generate sparse spikes. This approach results in low energy efficiency and high latency, posing significant challenges to the deployment of SNNs on edge devices.
[0003] Existing research focuses on two types of solutions: dedicated asynchronous processors and sparse-aware accelerators. Typical dedicated asynchronous processors include IBM's TrueNorth and Intel's Loihi. TrueNorth has 4096 parallel dedicated neuron blocks, designed to efficiently process millions of neurons while taking advantage of the sparsity of pulse operations. It uses synchronous circuits for computation and asynchronous circuits for interconnection, optimizing power consumption. Loihi integrates neuromorphic cores with many spiking neural units for parallel processing, and an asynchronous network-on-chip (NoC) facilitates communication between cores, further improving efficiency. However, these designs are still limited in parallel processing within individual cores, and usually assume a large amount of on-chip memory to store the weights of the entire network, which may not be practical in storage-constrained situations.
[0004] However, the above solutions usually have the following technical problems: SNNs need to perform calculations across multiple time steps, which offsets their inherent advantages in energy efficiency and leads to higher inference latency.
[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention
[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.
[0007] Some embodiments of the present disclosure propose a pattern framework chip applied to a pulse neural network to solve one or more of the technical problems mentioned in the above background technology section.
[0008] In a first aspect, some embodiments of the present disclosure provide a style framework chip applied to a pulse neural network, wherein the style framework chip comprises: a slice unit array, a pooling unit, an accumulation unit, an activation unit and a global buffer, wherein the slice unit in the slice unit array comprises: a processing unit array, a slice input buffer, a slice output buffer and a slice activation unit; the processing unit in the processing unit array comprises: a synapse array group, a processing input buffer, a processing output buffer and a processing accumulation unit; each synapse array comprises: a static random access memory array, a synapse input buffer, a synapse output buffer, an image to column scheduling The image-to-column scheduler is connected to the synaptic input buffer, the synaptic output buffer, the bit counter, and the reducer respectively; the reducer is connected to the dynamic scheduler and the multiplexer respectively; each slice unit in the slice unit array is connected to each other, and each slice unit in the slice unit array is connected to each other with the pooling unit, the accumulation unit, the activation unit, and the global buffer respectively; each processing unit in the processing unit array is connected to each other, and each processing unit is connected to each other with the slice input buffer, the slice output buffer, and the slice activation unit respectively; the synaptic array Each synaptic array in the column group is communicatively connected, and each synaptic array is communicatively connected with a processing input buffer, a processing output buffer, and a processing accumulation unit respectively; the synaptic input buffer is configured to: cache compressed input pulses; the image to column scheduler is configured to: read input pulses from the corresponding synaptic input buffer in time sequence, decompress the read input pulses according to the row index, and input the decompressed input pulses to the corresponding bit counter and reducer; the bit counter and reducer are configured to: count the number of pulses within a preset time window, and perform weighted activation on the received input pulses, and weight the decompressed input pulses. The weighted input pulse is input into the corresponding static random access memory array; the static random access memory array is configured to: perform speculative fast forwarding processing on the input pulse; the dynamic scheduler is configured to: in response to determining that the speculative fast forwarding of the input pulse fails, trigger the conditional recovery mode, return the input pulse to the original pulse sequence to determine the pulse emission time and final membrane potential of the input pulse, and store the final membrane potential of the input pulse into the membrane potential of the current neuron of the pulse neural network; the multiplexer is configured to: select between the unweighted input pulse and the weighted input pulse for storage calculation.
[0009] Optionally, the dynamic scheduler is further configured to: dynamically adjust the time window.
[0010] Optionally, the synaptic output buffer is configured to: cache the membrane potential and output pulse corresponding to the input pulse.
[0011] Optionally, the dynamic scheduler is further configured to: initialize the time window to a maximum value; in response to determining that speculative fast forwarding of an input pulse is successful, update a success counter; in response to determining that speculative fast forwarding of an input pulse fails, update a failure counter and shorten the time window.
[0012] Optionally, the static random access memory array is further configured to: in response to determining that a neuron does not emit any pulse signal within a time window, determine the cumulative membrane potential of the neuron at the end of the time window, and sum the product of the pulse input within the time window and the corresponding weight value; in response to the cumulative membrane potential being less than or equal to the sum of the product of the positive weight within the time window and the corresponding pulse input, determine the cumulative membrane potential as a maximum cumulative membrane potential; in response to determining that the maximum cumulative membrane potential is less than or equal to a preset threshold, determine that the neuron does not emit a pulse within the time window.
[0013] Optionally, the synapse array further includes: a switch matrix, a precharger, a write driver, a sense amplifier, a first adder, a second adder, a first shift register, a second shift register, and an adder-accumulator. The sense amplifier includes: a first read latch and a second read latch; the first read latch is communicatively connected to the first adder, and the second read latch is communicatively connected to the second adder; the first shift register is communicatively connected to the first read latch and the first shift register respectively; the second shift register is communicatively connected to the second read latch and the second shift register respectively; the adder-accumulator is communicatively connected to the first shift register, the second shift register, the dynamic scheduler, the image to column scheduler, and the synapse output buffer respectively; the switch matrix, the precharger, and the write driver are connected in parallel and in series; the multiplexer is connected to the first adder and the second adder through a bus.
[0014] Optionally, the pooling unit is configured to: expand the inter-layer interaction data in the pulse neural network from four dimensions to a two-dimensional bitmap; perform byte compression on the two-dimensional bitmap along the T dimension to obtain a compressed sparse byte matrix; and convert the compressed sparse byte matrix into a target storage format.
[0015] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the style framework chip applied to the pulse neural network of some embodiments of the present disclosure, the sparsity in the SNN pulse activity is utilized to achieve high-energy efficiency and low-latency calculations while ensuring accuracy. In addition, speculative fast forwarding and conditional recovery work together to save unnecessary computing overhead and improve computing efficiency. By dynamically adjusting the size of the time window, it is not only possible to flexibly respond to the pulse emission patterns of different neurons in each time step, but also to optimize the accuracy of the speculation mechanism, thereby reducing computing delays and improving overall computing efficiency. By utilizing the sparsity and temporal locality of pulse data, this compression method can greatly reduce the storage and transmission resources of inter-layer interaction data in SNN. Through hierarchical and modular design, the style framework chip can efficiently process sparse pulse data in the SNN model, significantly improving computing efficiency and energy efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.
[0017] Figure 1 is a schematic diagram of the structure of some embodiments of a pattern framework chip applied to a pulse neural network according to the present disclosure;
[0018] Figure 2 is a schematic structural diagram showing a slicing unit of a pattern framework chip applied to a pulse neural network of the present disclosure;
[0019] Figure 3 is a schematic diagram showing the structure of a processing unit of a pattern framework chip applied to a pulse neural network of the present disclosure;
[0020] Figure 4 is a schematic diagram showing the structure of a synaptic array of a pattern framework chip applied to a pulse neural network of the present disclosure;
[0021] Figure 5 It is a schematic diagram showing the transformation of the inter-layer interaction data in the pulse neural network of the present disclosure from four dimensions to a sparse byte matrix of a target storage format. DETAILED DESCRIPTION
[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0023] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.
[0024] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0027] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.
[0028] Figure 1 1 is a schematic diagram of the structure of some embodiments of the style framework chip applied to the pulse neural network according to the present disclosure. 100 is a schematic diagram of the structure of some embodiments of the style framework chip applied to the pulse neural network according to the present disclosure. The style framework chip applied to the pulse neural network includes a slice unit array, a pooling unit 101, an accumulation unit 102, an activation unit 103 and a global buffer 104.
[0029] like Figure 1As shown in the example, the slice unit array may include: slice unit 1, slice unit 2, slice unit 3, slice unit 4, slice unit 5, slice unit 6, slice unit 7, and slice unit 8. The pooling unit 101 may refer to an arithmetic logic unit (ALU) with compression and pooling functions. The accumulation unit 102 may refer to an accumulator (ACC). The activation unit 103 may refer to a hardware-connected activation function (AU). The global buffer 104 may refer to a cache. The slice unit may refer to an independent functional unit or core in a system-on-chip (SoC) design, which may include one or more processor cores, caches, memory controllers, input and output devices, peripheral interfaces or other functional modules. Multiple tiles can be combined and connected to form a complete system. The style framework may refer to a COMPASS framework.
[0030] like Figure 2 The example shows a schematic diagram of the structure of a slice unit, including: a processing unit array, a slice input buffer 201, a slice output buffer 202 and a slice activation unit 203. The processing unit array may include: processing unit 1, processing unit 2, processing unit 3, and processing unit 4.
[0031] In some embodiments, the slice unit in the slice unit array includes: a processing unit array, a slice input buffer, a slice output buffer and a slice activation unit. Each slice unit in the above-mentioned slice unit array is communicatively connected, and each slice unit in the above-mentioned slice unit array is communicatively connected with a pooling unit, an accumulation unit, an activation unit, and a global buffer, respectively. The slice input buffer and the slice output buffer may refer to the input buffer and the input buffer included in a slice unit. Among them, the input buffer is used to store input pulses. The output buffer is used to store output pulses. The slice activation unit may refer to an activation function (AU) in a slice unit.
[0032] like Figure 3 The example shows a schematic diagram of the structure of a processing unit, including: a synapse array group, a processing input buffer 301, a processing output buffer 302 and a processing accumulation unit 303, wherein the synapse array group may include: synapse array 1, synapse array 2, synapse array 3, and synapse array 4.
[0033] In some embodiments, the processing units in the processing unit array include: a synapse array group, a processing input buffer, a processing output buffer and a processing accumulation unit. The processing units in the above-mentioned processing unit array are communicatively connected, and each processing unit is communicatively connected to a slice input buffer, a slice output buffer, and a slice activation unit, respectively. The synapse arrays in the synapse array group are communicatively connected, and each synapse array is communicatively connected to a processing input buffer, a processing output buffer, and a processing accumulation unit, respectively. The processing input buffer and the processing output buffer may refer to the input buffer and the input buffer included in a processing unit. Among them, the processing input buffer is used to store the input pulse of the current processing unit. The processing output buffer is used to store the output pulse of the current processing unit. The processing accumulation unit may refer to the accumulator (ACC) of the current processing unit.
[0034] like Figure 4 The example shows a schematic diagram of the structure of a synapse array, including: a synapse input buffer, a synapse output buffer, an image to column scheduler, a bit counter, a reducer, a dynamic scheduler, and a multiplexer; in addition, a switch matrix, a precharger, a write driver, a sense amplifier, a first adder, a second adder, a first shift register, a second shift register, and an adder accumulator are also shown, and the sense amplifier includes: a first read latch and a second read latch.
[0035] In some embodiments, each synapse array includes: a static random access memory array, a synapse input buffer, a synapse output buffer, an image to column scheduler, a bit counter, a reducer, a dynamic scheduler, and a multiplexer; the image to column scheduler is respectively connected to the synapse input buffer, the synapse output buffer, the bit counter, and the reducer; the reducer is respectively connected to the dynamic scheduler and the multiplexer. In addition, the synapse array also includes: a switch matrix, a precharger, a write driver, a sense amplifier, a first adder, a second adder, a first shift register, a second shift register, and an adder accumulator, and the sense amplifier includes: a first read latch and a second read latch. The static random access memory array may refer to an SRAM array. The synapse input buffer and the synapse output buffer may refer to the input buffer and the input buffer included in the synapse array. The image to column scheduler may refer to an Img2Col convolution layer. The bit counter may refer to a POPCNT bit counter, which is a hardware instruction used to count the number of 1s in the binary representation of a number, also known as a bit 1 count. A reducer may refer to a reducer, which is a pure function that calculates a new state based on the current state and action. A reducer receives the previous state and action as parameters and returns a new state object. A dynamic scheduler may refer to an adaptive time window scheduler. A multiplexer may refer to a MUX multiplexer. A switch matrix may refer to a matrix that controls the opening and closing of a circuit. A precharger may refer to a first-order RC series circuit, the main function of which is to reduce the impact current during power-on and protect devices such as motor controllers, batteries, and main relays. A write driver may refer to a process of writing data or instructions into a device to write to a chip (IC), also known as "burning" a chip. A sense amplifier may refer to a sense amplifier used in an integrated circuit (IC) design, especially in a static random access memory (SRAM) design, for detecting and amplifying data in a storage unit. A first read latch and a second read latch may be a storage unit in a digital circuit that is sensitive to a pulse level and can change state under the action of a specific input pulse level. The first adder and the second adder can be a digital circuit component for performing addition operations. It is one of the core components of the ALU in the CPU, namely the full adder. The first shift register and the second shift register can be an electronic device for storing and manipulating data. It consists of a series of storage cells, each of which can store a binary digit. The main function of the shift register is to move data from one storage cell to another, thereby realizing the storage and transmission of data. This device is usually composed of flip-flops, which are connected in such a way that the output of one flip-flop can be used as the input of another flip-flop, depending on the type of shift register created.The shift register can work in a digital circuit with several identical time pulses. The data is input into the device in parallel or serial mode, and then each time pulse is shifted one bit to the left or right in turn and output at the output end. This shift register is one-dimensional, but there are also multi-dimensional shift registers, that is, the input and output data itself is a series of bits.
[0036] The data input mode of the shift register is divided into serial input and parallel input. Serial input is to send the data to be input into the register one bit at a time from one input end under the action of the clock pulse; parallel input is to send the input data into the register from several input ends at the same time. The shift direction of the shift register is divided into right shift and left shift. Right shift means that the data is input from the lowest bit on the left and output from the highest bit on the right in turn; when shifting to the left, the data is input from the right side of the low bit and output from the left side of the high bit. The output of the shift register is also divided into serial and parallel. Serial output means that under the action of the clock pulse, the last output end of the register outputs the data of the register one bit at a time; parallel output means that each register unit of the register has an output.
[0037] An adder accumulator is a register that is used to store an operand and result of an arithmetic or logical operation.
[0038] In some embodiments, the synaptic input buffer is configured to: cache compressed input pulses. The input pulse may refer to a pulse signal of a neuron in a spiking neural network (SNN). The compressed input pulse may refer to a pulse in a matrix obtained by expanding the inter-layer interaction data in the SNN from four dimensions to a two-dimensional bitmap, and then byte packing the two-dimensional bitmap along the T dimension to obtain a sparse byte matrix, and then converting the sparse byte matrix into a CSR (Compressed Sparse Row) format.
[0039] In some embodiments, the image-to-column scheduler is configured to: read input pulses from the corresponding synaptic input buffer in time order, decompress the read input pulses according to the row index, and input the decompressed input pulses to the corresponding bit counter and reducer. The row index may refer to a row index in a sparse byte matrix in a CSR format.
[0040] In some embodiments, the bit counter and reducer are configured to: count the number of pulses within a preset time window, perform weighted activation on the received input pulses, and input the weighted input pulses into the corresponding static random access memory array.
[0041] In some embodiments, the static random access memory array is configured to perform speculative fast forwarding processing on input pulses. Speculative fast forwarding (SFF) may refer to a network technology that aims to improve the forwarding efficiency of messages. It uses a cache to process messages and identifies a data flow based on information in the message (such as source IP address, destination IP address, source port, destination port, IP protocol number, etc.) through a data flow-based technology. When the first message of a data flow is forwarded by looking up the routing table, the corresponding forwarding information is generated in the cache.
[0042] The core idea of Speculative Fast Forwarding (SFF) is to use the sparsity of output pulses in SNN to quickly infer the results of non-emitted pulses by merging the calculations of multiple time steps. The specific steps are as follows: 1) Merge and calculate the cumulative membrane potential: Assuming that the neuron does not emit any pulse signals in a certain time window, the cumulative membrane potential at the end of the time window can be merged and calculated, that is, the product of the pulse input and the weight value in this time window; 2) Determine whether a pulse occurs: Since the weight has a negative value, it is easy to know that the cumulative membrane potential is always less than or equal to the product of the only positive weight and the corresponding pulse input in this time window, which is recorded as the maximum value of the cumulative membrane potential. If the maximum value of the cumulative membrane potential does not exceed the threshold, the actual cumulative membrane potential will not exceed the threshold, and the inference is successful, that is, the neuron will not emit pulses in this time window, and the calculation of this time window can be skipped, thereby significantly reducing the delay.
[0043] In some embodiments, the dynamic scheduler is configured to: in response to determining that the speculative fast forwarding of the input pulse fails, trigger the conditional recovery mode, rewind the input pulse to the original pulse sequence to determine the pulse emission time and final membrane potential of the input pulse, and store the final membrane potential of the input pulse in the membrane potential of the current neuron of the pulse neural network. The conditional recovery mode can refer to conditional reversion (CR), which refers to a specific neural process or phenomenon in the pulse network model, usually involving a recurrent network composed of granule cells and Golgi cells. These cells are an important component of the primary cortex of the cerebellum and are involved in the encoding, integration and transmission of motor information. Buonomano and Mauk first studied the use of a recurrent network composed of granule and Golgi cells to represent conditional recovery (CR). For example, conditional recovery: if the cumulative membrane potential maximum exceeds a threshold, it is necessary to revert to the original pulse sequence and process it in chronological order to determine the exact pulse emission time and final membrane potential.
[0044] Therefore, speculative fast forwarding (SFF) and conditional recovery (CR) work together to save unnecessary computing overhead and improve computing efficiency.
[0045] In some embodiments, a multiplexer is configured to select between unweighted input pulses and weighted input pulses for storage calculations.
[0046] In some embodiments, the dynamic scheduler is further configured to: dynamically adjust the time window. Dynamically adjusting the time window may refer to adaptive time window scheduling.
[0047] In some embodiments, the synaptic output buffer is configured to: cache the membrane potential corresponding to the input pulse and the output pulse.
[0048] In some embodiments, the dynamic scheduler is further configured to: initialize the time window to the maximum value; in response to determining that the input pulse speculative fast forwarding is successful, update the success counter; in response to determining that the input pulse speculative fast forwarding fails, update the failure counter and shorten the time window. For example, 1) initialize the time window to the maximum value; 2) if the speculation is successful, update the success counter; if the speculation fails, update the failure counter and reduce the time window; 3) subsequently dynamically adjust the size of the time window according to the success or failure: if successful, increase the time window, if failed, reduce the time window; 4) repeat the above steps until all time steps are processed.
[0049] As a result, the size of the time window can be dynamically adjusted according to the real-time changes in neuronal pulse activity, which can not only flexibly respond to the pulse emission patterns of different neurons in each time step, but also optimize the accuracy of the inference mechanism, thereby reducing computing delays and improving overall computing efficiency. This enables the system to allocate computing resources more intelligently, significantly improving the real-time processing capabilities of spiking neural networks (SNNs).
[0050] In some embodiments, the static random access memory array is further configured to: in response to determining that a neuron does not emit any pulse signal within a time window, determine the cumulative membrane potential of the neuron at the end of the time window, and sum the product of the pulse input within the time window and the corresponding weight value; in response to the cumulative membrane potential being less than or equal to the sum of the product of the positive weight within the time window and the corresponding pulse input, determine the cumulative membrane potential as a cumulative membrane potential maximum value; in response to determining that the cumulative membrane potential maximum value is less than or equal to a preset threshold value, determine that the neuron does not emit a pulse within the time window.
[0051] The core idea of Speculative Fast Forwarding (SFF) is to use the sparsity of output pulses in SNN to quickly infer the results of non-emitted pulses by merging the calculations of multiple time steps. The specific steps are as follows: 1) Merge and calculate the cumulative membrane potential: Assuming that the neuron does not emit any pulse signals in a certain time window, the cumulative membrane potential at the end of the time window can be merged and calculated, that is, the product of the pulse input and the weight value in this time window; 2) Determine whether a pulse occurs: Since the weight has a negative value, it is easy to know that the cumulative membrane potential is always less than or equal to the product of the only positive weight and the corresponding pulse input in this time window, which is recorded as the maximum value of the cumulative membrane potential. If the maximum value of the cumulative membrane potential does not exceed the threshold, the actual cumulative membrane potential will not exceed the threshold, and the inference is successful, that is, the neuron will not emit pulses in this time window, and the calculation of this time window can be skipped, thereby significantly reducing the delay.
[0052] In some embodiments, the pooling unit is configured to: expand the inter-layer interaction data in the spiking neural network from four dimensions to a two-dimensional bitmap; perform byte compression on the two-dimensional bitmap along the T dimension to obtain a compressed sparse byte matrix; and convert the compressed sparse byte matrix into a target storage format. The target storage format may refer to a CSR (Compressed Sparse Row) format.
[0053] Therefore, by exploiting the sparsity and temporal locality of spike data, this compression method can significantly reduce the storage and transmission resources of inter-layer interaction data in SNNs.
[0054] like Figure 5 The example shows a schematic diagram of the transformation of inter-layer interaction data in a pulse neural network from four dimensions to a sparse byte matrix in a target storage format, including: inter-layer interaction data in a pulse neural network (a), an expanded two-dimensional bitmap (T&C span), a sparse byte matrix after T-dimensional byte compression, and a sparse byte matrix in CSR format.
[0055] For example, the inter-layer interaction data (a) may include: T=1; T=2; wherein the length of the inter-layer interaction data (a) is C, the width is W, and the height is H.
[0056] For example, the expanded two-dimensional bitmap (T&C span) may be a 4×4 two-dimensional bitmap:
[0057] A[1,1,1,1];A[2,1,1,1];A[1,2,1,1];A[2,2,1,1];
[0058] A[1,1,1,2];A[2,1,1,2];A[1,2,1,2];A[2,2,1,2];
[0059] A[1,1,2,1];A[2,1,2,1];A[1,2,2,1];A[2,2,2,1];
[0060] A[1,1,2,2];A[2,1,2,2];A[1,2,2,2];A[2,2,2,2].
[0061] For example, a T-dimensional byte-compressed sparse byte matrix can be:
[0062] A[:,1,1,1]; A[:,2,1,1];
[0063] A[:,1,1,2]; A[:,2,1,2];
[0064] A[:,1,2,1]; A[:,2,2,1];
[0065] A[:,1,2,2]; A[:,2,2,2].
[0066] As an example, see further Figure 5 , the specific process from input pulse to output pulse is as follows:
[0067] First, the compressed input pulses are buffered in the synaptic input buffer, and the image-to-column scheduler (Img2Col) reads the input pulses in time order and decompresses and compresses them according to the row index.
[0068] Second, the bit counter (POPCNT) and reducer (Reducer) count the number of pulses in time and input the weighted activation into the SRAM-CIM (static random access memory array) for speculative execution and fast speculative forwarding (SFF).
[0069] Third, if SFF speculation fails, the dynamic scheduler triggers conditional regression (CR) to revert to the original spike train to determine the precise spike firing times and final membrane potential.
[0070] Fourth, a multiplexer (MUX) will select between unweighted and weighted pulses for storage calculations.
[0071] Fifth, to effectively deal with unpredictable sparsity levels, the scheduler dynamically adjusts the time window (TW) to adapt to the sparsity utilization in SFF.
[0072] Sixth, registers with shifting capability (first shift register, second shift register) cache partial sums (representing contributions from different presynaptic neurons).
[0073] Seventh, it is finally accumulated into the membrane potential (MBP) of the current neuron, and the membrane potential and output pulse (OSP) are updated in the synaptic output buffer.
[0074] Eighth, loop execution: the SRAM-CIM core loads the next set of encoded SNN model weights and repeats the above steps until all input pulses are processed.
[0075] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0076] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.
Claims
1. A pattern framework chip applied to a pulse neural network, the pattern framework chip comprising: Slice unit array, pooling unit, accumulation unit, activation unit and global buffer, where The slice unit in the slice unit array includes: a processing unit array, a slice input buffer, a slice output buffer and a slice activation unit; The processing units in the processing unit array include: a synaptic array group, a processing input buffer, a processing output buffer and a processing accumulation unit; Each synapse array includes: a static random access memory array, a synapse input buffer, a synapse output buffer, an image-to-column scheduler, a bit counter, a reducer, a dynamic scheduler, and a multiplexer; the image-to-column scheduler is respectively connected to the synapse input buffer, the synapse output buffer, the bit counter, and the reducer; the reducer is respectively connected to the dynamic scheduler and the multiplexer; Each slice unit in the slice unit array is communicatively connected, and each slice unit in the slice unit array is communicatively connected with a pooling unit, an accumulation unit, an activation unit, and a global buffer respectively; Each processing unit in the processing unit array is communicatively connected, and each processing unit is communicatively connected to a slice input buffer, a slice output buffer, and a slice activation unit respectively; Each synaptic array in the synaptic array group is communicatively connected, and each synaptic array is communicatively connected to a processing input buffer, a processing output buffer, and a processing accumulation unit respectively; The synaptic input buffer is configured to: buffer compressed input pulses; The image-to-column scheduler is configured to: read input pulses from the corresponding synaptic input buffer in time order, decompress the read input pulses according to the row index, and input the decompressed input pulses to the corresponding bit counter and reducer; The bit counter and reducer are configured to: count the number of pulses within a preset time window, and perform weighted activation on the received input pulses, and input the weighted input pulses into the corresponding static random access memory array; A static random access memory array is configured to: perform speculative fast forwarding processing on input pulses; A dynamic scheduler configured to: in response to determining that speculative fast forwarding of an input pulse fails, trigger a conditional recovery mode, rewind the input pulse to the original pulse train to determine a pulse firing time and a final membrane potential of the input pulse, and store the final membrane potential of the input pulse into the membrane potential of a current neuron of the spiking neural network; A multiplexer is configured to select between unweighted input pulses and weighted input pulses for storage calculations.
2. The pattern framework chip applied to a pulse neural network according to claim 1, wherein: The dynamic scheduler is also configured to: dynamically adjust the time window.
3. The pattern framework chip applied to a pulse neural network according to claim 1, wherein: The synaptic output buffer is configured to cache the membrane potential corresponding to the input pulse and the output pulse.
4. The pattern framework chip applied to a pulse neural network according to claim 2, wherein: The dynamic scheduler is also configured to: Initialize the time window to the maximum value; In response to determining that the speculative fast forwarding of the input pulse is successful, updating a success counter; In response to determining that speculative fast forwarding of the input pulse has failed, a failure counter is updated and the time window is shortened.
5. The pattern framework chip applied to a pulse neural network according to claim 1, wherein: The static random access memory array is further configured as: In response to determining that the neuron does not emit any pulse signal within a certain time window, determining the accumulated membrane potential of the neuron at the end of the time window, and summing the product of the pulse input within the time window and the corresponding weight value; In response to the cumulative membrane potential being less than or equal to the sum of the products of the positive weights and the corresponding pulse inputs within the time window, the cumulative membrane potential is determined as a maximum cumulative membrane potential; In response to determining that the cumulative membrane potential maximum value is less than or equal to a preset threshold, it is determined that the neuron does not emit a pulse within the time window.
6. The pattern framework chip applied to a pulse neural network according to claim 1, wherein: The synapse array further comprises: a switch matrix, a precharger, a write driver, a sense amplifier, a first adder, a second adder, a first shift register, a second shift register, and an adder accumulator, wherein the sense amplifier comprises: a first read latch, a second read latch; The first read latch is communicatively connected to the first adder, and the second read latch is communicatively connected to the second adder; The first shift register is communicatively connected to the first read latch and the first shift register respectively; The second shift register is respectively communicatively connected with the second read latch and the second shift register; The adder accumulator is respectively connected to the first shift register, the second shift register, the dynamic scheduler, the image to column scheduler, and the synaptic output buffer for communication; The switch matrix, the precharger, and the write driver are connected in parallel and in series; The multiplexer is connected to the first adder and the second adder through a bus.
7. The pattern framework chip applied to a pulse neural network according to claim 1, wherein: The pooling unit is configured as follows: Expand the inter-layer interaction data in the spiking neural network from four dimensions to a two-dimensional bitmap; Compress the two-dimensional bitmap along the T dimension to obtain a compressed sparse byte matrix; Convert a compressed sparse byte matrix to the target storage format.
Citation Information
Patent Citations
Parallel processing in spiking neural networks
CN115409162A
Pulse neural network processor with unified architecture
CN116306835A
Sparse pulse neural network accelerator based on ping-pong architecture
CN116663626A
Step-ahead spiking neural network
US20240086689A1
