A hybrid neural network accelerator and a hybrid neural network neuron acceleration calculation method

Abstract

This invention provides a hybrid neural network accelerator and a method for accelerating computation using hybrid neural network neurons. The hybrid neural network accelerator has both ANN and SNN acceleration computation capabilities, including M neural computation kernels, a register file, SRAM, a membrane potential accumulator, an transmitter, and a neuron encapsulation module. Each neural computation kernel includes Q neurons. The neuron encapsulation module is used when a neural computation kernel needs to receive more inputs than the fan-in threshold of a single neuron simultaneously. It encapsulates N neurons into a neuron encapsulation group, extracts W inputs through the first neuron in the encapsulation group, and obtains the weight parameters of each synapse in the first neuron based on its storage location in the SRAM. The W inputs are divided into multiple input batches at preset intervals, and these batches are sent to the membrane potential accumulator for computation with the weight parameters. The resulting membrane potentials are then passed to the next neuron in the encapsulation group for computation on the next group of W inputs, until all N neurons have completed the computation. This invention utilizes neuron encapsulation technology to significantly expand the number of fan-ins that each neuron can receive.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a hybrid neural network accelerator and a method for accelerating computation using hybrid neural network neurons. Background Technology

[0002] Traditional von Neumann computer architecture separates processing and storage units. Because the training and inference processes of Artificial Neural Networks (ANNs) and Spiking Neural Networks (SNNs) require significant computation and storage, traditional central processing units (CPUs) often cannot meet their high-performance and low-latency requirements. Therefore, to overcome this limitation, dedicated hardware accelerators—neighborhood accelerators—are needed. Compared to general-purpose processors, neural network accelerators can accelerate core operations such as convolution, pooling, and nonlinear activation through model-hardware co-optimization, thereby providing faster computation speeds and lower power consumption.

[0003] Among existing accelerator internal structures, the crossbar structure is one of the most commonly used computational architectures. In this architecture, the horizontal axis is used as the input axon, the vertical axis as the output dendrite, and the intersection of the horizontal and vertical axes is a synapse that stores different weights. Each MAC operation in the neural network model is executed only when the weight of that synapse is valid and the axon at that location emits a pulse. This is equivalent to combining storage and processing modules, thus reducing the number of data moves to save energy and enhance the parallelism of the operation.

[0004] However, the processing kernel size in the aforementioned accelerator structures is fixed, making it difficult to adapt to convolutional kernels of different sizes. Furthermore, when designing fixed parameters in the accelerator, the fan-in of its neurons is limited, and a fixed fan-in is difficult to be compatible with the complex and varied topology of neural network models. This necessitates the splitting of the network, resulting in the unavailability of certain hardware resources and potentially additional inter-kernel communication overhead, thus affecting the hardware acceleration effect. Summary of the Invention

[0005] This invention provides a hybrid neural network accelerator and a method for accelerating computation of hybrid neural network neurons, which solves the defects of existing accelerators in that the processing kernel size is fixed, making it difficult to adapt to convolution kernels of different sizes, and the fixed fan-in parameters in existing accelerators make it difficult to be compatible with complex and variable neural network model topologies, thus bringing additional inter-kernel communication overhead and affecting hardware acceleration performance.

[0006] This invention provides a hybrid neural network accelerator with ANN and SNN accelerated computing capabilities. The hybrid neural network accelerator includes M neural computing cores, each of which includes Q neurons, and also includes a register file, SRAM, membrane potential accumulator, transmitter, neuron encapsulation module, and synaptic compression module.

[0007] The register file is used to store the operating state and intrinsic parameters of the neuron. Each neuron integrates W synapses, and the W synapses are used to receive W input parameters of the neuron. The intrinsic parameters include the fan-in threshold W of each neuron.

[0008] The SRAM is used to store the location information of each synapse of the neuron and the weight parameters of each synapse;

[0009] The membrane potential accumulator is used to perform corresponding calculations on the input parameters and the weight parameters;

[0010] The transmitter is used to activate the membrane voltage and transmit voltage pulse of the membrane potential accumulator;

[0011] The neuron encapsulation module is used when the neural computing core needs to receive inputs greater than the fan-in threshold W of a single neuron simultaneously. It encapsulates N neurons into a neuron encapsulation group, extracts W inputs through the first neuron in the encapsulation group, obtains the weight parameters of each synapse in the first neuron based on its storage location in SRAM, divides the W inputs into multiple input batches at preset intervals, sends the input batches to a membrane potential accumulator for calculation with the weight parameters, and then passes the calculated membrane potential to the next neuron in the encapsulation group for the next group of W inputs, until all N neurons have completed the calculation.

[0012] According to the hybrid neural network accelerator provided by the present invention, the hybrid neural network accelerator further includes a synapse compression module, wherein the synapse compression module is used to compress the storage of SRAM to store only effective synapses and the weight parameters of the effective synapses, which is adapted to select only neurons containing the effective synapses to participate in the operation during neuron operation, and neurons not containing effective synapses do not participate in the operation, thereby avoiding the waste of neurons, wherein the effective synapse is the synapse that participates in the operation.

[0013] According to the hybrid neural network accelerator provided by the present invention, the neuron encapsulation module is further configured to, when the last neuron in the neuron encapsulation group performs a computation, the SNN emits a pulse by comparing the membrane level with a threshold voltage, or the ANN emits a pulse by activating a function;

[0014] Update the membrane voltage and the neuron's operating state, and write the updated membrane voltage and the neuron's operating state back to the register file and SRAM, respectively.

[0015] According to the hybrid neural network accelerator provided by the present invention, the storage of the SRAM is compressed to store only the effective synapses within the effective weight window and the weight parameters of the effective synapses, specifically including:

[0016] The synaptic region where each neuron actually performs calculations is defined as the effective weight window, and synapses that do not participate in the calculations are defined as invalid synapses.

[0017] Based on the effective weight window, the storage of the SRAM is adjusted so that it only stores synapses within the effective weight window and deletes the weight parameters of invalid synapses outside the effective weight window.

[0018] According to the hybrid neural network accelerator provided by the present invention, the step of selecting only neurons containing effective synapses to participate in the operation, and excluding neurons without effective synapses from the operation, specifically includes the following steps:

[0019] S1. Define neurons containing effective synapses as effective neurons and neurons not containing effective synapses as ineffective neurons.

[0020] S2. Starting from the first neuron in the neuron encapsulation group, search for effective neurons sequentially;

[0021] S3. Determine whether the currently searched neuron has reached the effective weight window;

[0022] S4. If not, then it is determined to be an invalid neuron, and the process jumps to S2;

[0023] S5. If so, sequentially search for the starting synapse position of the effective weight window within the neuron according to the interval between the input batch data, and start executing the neuron's operation at the starting synapse position.

[0024] According to the hybrid neural network accelerator provided by the present invention, the operation of the neuron is initiated at the starting synapse location, specifically including:

[0025] Starting at the initial synapse position, neuron computation is performed on the effective synapses within the effective weight window;

[0026] If the effective weight window spans multiple neurons, two calculations are performed at the boundaries of each neuron;

[0027] After the calculation is completed in the effective weight window, the remaining invalid synapses in the neuron encapsulation group where the effective weight window is located are skipped, and the calculation is performed in the effective weight window of the next neuron encapsulation group.

[0028] According to the hybrid neural network accelerator provided by the present invention, the neuron includes a neuron model and a neuron operator.

[0029] In the hybrid neural network accelerator provided by the present invention, the value of N ranges from 2 to 64, and the maximum value of W is 1152.

[0030] According to the hybrid neural network accelerator provided by the present invention, the hybrid neural network accelerator includes 64 neural computing cores, each of which includes 512 neurons; the hybrid neural network accelerator also includes an L1 router;

[0031] The hybrid neural network accelerator connects to one L1 router for every four neural computing cores, forming a quadtree structure. The 16 L1 routers form a two-dimensional grid structure.

[0032] Each L1 router includes four pairs of asynchronous first transmitter-receiver pairs, which are used for data interaction with adjacent L1 routers in four directions of the two-dimensional mesh structure.

[0033] Each L1 router also includes four pairs of asynchronous second transmitter-receiver pairs for data interaction between the L1 router and each neural computing core;

[0034] The L1 router also includes a distributor;

[0035] The distributor is used to determine the transmission direction of data frames within the L1 router, wherein the data frames are data or pulses that conform to the address event representation rules.

[0036] This invention also provides a method for accelerating computation using hybrid neural network neurons. The method is implemented using a hybrid neural network accelerator as described above. The hybrid neural network accelerator has both ANN and SNN acceleration computation capabilities, includes M neural computation kernels, and each neural computation kernel includes Q neurons. The method includes:

[0037] If the input to the neural computation kernel is greater than the fan-in threshold W of a single neuron in the neural computation kernel, determine the number N of neurons participating in the computation within the neural computation kernel;

[0038] The N neurons are encapsulated into a neuron encapsulation group. W inputs are extracted through the first neuron in the neuron encapsulation group. The weight parameters of each synapse in the first neuron are obtained according to the storage location of the first neuron in the SRAM.

[0039] The W inputs of a single neuron are divided into multiple input batches at preset intervals. The input batches are sent to a membrane potential accumulator for calculation with the weight parameters. The calculated membrane potentials are then passed to the next neuron in the neuron encapsulation group for the calculation of the next group of W inputs, until N neurons have completed the calculation.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the hybrid neural network neuron accelerated computation method as described above.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hybrid neural network neuron accelerated computation method as described above.

[0042] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the hybrid neural network neuron accelerated computation method as described above.

[0043] The present invention provides a hybrid neural network accelerator and a hybrid neural network neuron acceleration computation method. The hybrid neural network has ANN and SNN acceleration computation functions. Through neuron encapsulation technology, it can significantly expand the number of fan-ins that each neuron can receive. It can increase the number of fan-ins with less fixed resources in a single neuron and computation kernel, respectively, thereby improving the hardware utilization rate when deploying the model. At the same time, the present invention can support neuron models with both ANN and SNN computation operations and can execute ANN and SNN hybrid intelligence. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the structure of the hybrid neural network accelerator provided in an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram of the diagonal line phenomenon when deploying a crossbar structure in existing technologies.

[0047] Figure 3 This is a block diagram illustrating the working principle of the synaptic compression module provided by the present invention for synaptic compression.

[0048] Figure 4 This is a diagram showing the overall structure of the hybrid routing network system and the internal structure of L1 routing provided by the present invention.

[0049] Figure 5 This is a flowchart illustrating the accelerated computation method for hybrid neural network neurons provided in an embodiment of the present invention.

[0050] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] With the gradual development of artificial intelligence, related applications such as image classification, object detection, and natural language processing are playing an increasingly important role in people's lives and work. Traditional artificial neural networks (ANNs), such as convolutional neural networks (CNNs), mainly achieve high accuracy and performance on different tasks through deep local feature extraction and spatial translation invariance. Neuromorphic computing, such as third-generation neural networks—spiking neural networks (SNNs)—attempts to mimic the operation of the human brain's nervous system, imitating its biosensing and information processing system architecture. It utilizes the spatiotemporal sparsity of pulses to achieve higher speed and energy efficiency, and is considered to have the potential to unleash the greater potential of neuromorphic computing, making it more conducive to further exploration of neurobehavior and brain science.

[0053] To deploy ANN or SNN models to mobile or edge devices for tasks, dedicated neural network accelerators have been proposed and extensively studied in recent years. Neural network accelerator designs typically employ two approaches: ASIC (Application-Specific Integrated Circuit) and FPGA (Field-Programmable Gate Array). ASICs are custom designs that directly construct computing circuits using physical transistors, generally offering higher performance and energy efficiency. However, their design and manufacturing processes are complex and expensive, resulting in high tape-out costs. This often necessitates consideration of generality in their design, making it difficult to thoroughly optimize for specific networks, and limiting their versatility for deploying a wide variety of emerging computational operations. FPGAs, on the other hand, are programmable logic chips with high flexibility and configurability, allowing for custom design based on specific application requirements. Unlike ASICs, FPGAs require lookup tables to construct circuits, incurring a layer of hardware overhead due to reconfigurability. Their energy and area efficiency are lower than ASICs, but the reconfigurability of FPGAs determines their advantage in neural network accelerator design. Due to the rapid development of neural networks, there is a constant need to design and update neural network accelerators with new architectures. FPGAs can verify or directly deploy new hardware architectures without requiring tape-out. This also allows for thorough optimization of a single model within a hardware design, giving FPGAs a significant advantage over ASICs in co-optimization.

[0054] Among the existing internal structures of accelerators, the crossbar structure is the most commonly used computing architecture. In this architecture, the horizontal axis is used as the input axon, the vertical axis as the output dendrite, and the intersection of the horizontal and vertical axes is the synapse, which stores different weights. Each MAC operation in the neural network model is only executed when the weight of that synapse is valid and the axon at that point emits a pulse, which is equivalent to combining the storage and processing modules. Therefore, it can reduce the number of data moves to save energy and enhance the parallelism of the operation. However, there are several key problems with this structure: (1) The size of the processing kernels in the accelerator is fixed, so they are difficult to adapt to convolution kernels of different sizes; (2) When designing fixed parameters in the accelerator, the fan-in and fan-out of its neurons will be restricted. A fixed fan-in and fan-out will inevitably be difficult to be compatible with the complex and ever-changing topology of the neural network model. This leads to the network having to be split, which results in some hardware resources being unusable and may bring additional inter-core communication overhead, thus affecting the hardware acceleration effect.

[0055] In view of this, this invention proposes to combine the characteristics of ANN and SNN hybrid neural network models with hardware accelerators to explore efficient neuromorphic computing acceleration circuit design schemes and data flow design and deployment schemes.

[0056] In some specific embodiments of the present invention, this solution provides a hybrid neural network accelerator with ANN and SNN accelerated computing functions, including M neural computing kernels, each of which includes Q neurons. Figure 1 This is a schematic diagram of the hybrid neural network accelerator provided in an embodiment of the present invention. This diagram can also be seen as a principle block diagram of neuron encapsulation technology used in crossbar structures, such as... Figure 1 As shown, the hybrid neural network accelerator also includes:

[0057] Register file 10, SRAM 20, membrane potential accumulator 30, transmitter 40, neuron packaging module 50;

[0058] The register file 10 is used to store the operating state and intrinsic parameters of the neuron. Each neuron integrates W synapses, and the W synapses are used to receive W input parameters of the neuron. The intrinsic parameters include the fan-in threshold W of each neuron.

[0059] SRAM 20 is used to store the location information of each synapse of the neuron and the weight parameters of each synapse;

[0060] The membrane potential accumulator 30 is used to perform corresponding calculations on the input parameters and the weighting parameters;

[0061] The transmitter 40 is used to activate the membrane voltage and transmit voltage pulse of the membrane potential accumulator;

[0062] The neuron encapsulation module 50 is used when the neural computing core needs to receive more than the fan-in threshold of a single neuron simultaneously. It encapsulates N neurons into a neuron encapsulation group, extracts W inputs through the first neuron in the neuron encapsulation group, obtains the weight parameters of each synapse in the first neuron according to the storage location of the first neuron in SRAM, divides the W inputs into multiple input batches according to a preset interval, sends the input batches to the membrane potential accumulator to perform calculations with the weight parameters, and passes the calculated membrane potentials to the next neuron in the neuron encapsulation group for the calculation of the next group of W inputs, until N neurons have completed the calculations.

[0063] In some possible embodiments of the present invention, the neuron includes a neuron model and a neuron operator.

[0064] In a possible embodiment, N ranges from 2 to 64, and the maximum value of W is 1152.

[0065] In a possible embodiment, the hybrid neural network accelerator includes 64 neural computational cores, each of which includes 512 neurons.

[0066] It should be noted that existing crossbar neural network accelerators cannot simultaneously receive more inputs than the number of neurons fan-in. This fan-in limitation usually comes from parameter limitations in hardware design, which also limits the number of synapses and lacks configurability and flexibility.

[0067] This invention utilizes neuron encapsulation technology to configure the fan-in of neurons according to requirements. Compared to the traditional method of adapting accelerator fan-in by stacking the number of neurons or splitting the neural network model, this invention can achieve a maximum of 73,724 fan-ins.

[0068] In some possible embodiments of the present invention, such as Figure 1 As shown, Figure 1 This can be seen as a principle block diagram of neuron encapsulation technology used in crossbar structures. The diagram illustrates the principle framework of how the embodiment of the present invention uses neuron encapsulation technology to improve fan-in.

[0069] Specifically, this embodiment provides an implementation of neuron encapsulation technology that significantly expands the number of fan-ins that each neuron can receive.

[0070] In possible embodiments, the neuron developed in this invention comprises the following components: a register file storing the neuron's operational state (resting state, depolarized state, or hyperpolarized state) and intrinsic neuron parameters such as threshold and leakage; a configurable, kernel-wide shared SRAM storing all synaptic connections and weights; a membrane potential accumulator and an emitter capable of activating membrane levels and firing pulses. Each neuron integrates up to 1152 (128×9) synapses to better accommodate the most common 3×3 convolutional kernels. In practical use, users can reconfigure the neuron parameters or weights in the synapses to fine-tune the network according to the model's requirements. This neuron not only supports a large number of neuron models, such as the Accumulation-Fire (IF) model, the Leakage-Accumulation-Fire (LIF) model, and the Degenerate Izhikevich model, but also supports most operators in convolutional neural networks, such as multiply-accumulate, pooling, and ReLU activation.

[0071] In some possible embodiments of the present invention, the neuron encapsulation module is further configured to, when the last neuron in the neuron encapsulation group performs an operation, the SNN emits a pulse by comparing the membrane level with the threshold voltage, or the ANN emits a pulse by activating the activation function; update the membrane voltage and the operating state of the neuron, and write the updated membrane voltage and the operating state of the neuron back to the register file and SRAM, respectively.

[0072] Specifically, to support situations where neurons may need to receive more than 1152 fan-ins, neuron encapsulation technology can "encapsulate" neurons together to form larger neurons and modify the data flow connections within the computation kernel. The parameter N specifies the number of neurons in a set of encapsulations, with a valid range of 1 to 64. N is pre-calculated when the acceptable fan-ins of a single neuron cannot meet the model's requirements. Neuron encapsulation is enabled when the computation kernel detects N > 1 at the start of computation. The synapses and weights of the encapsulated neurons are independent of each other, but other parameters are preset to be the same. During computation, the controller first decodes the received impulses (SNN mode) or the input 8-b data (ANN mode) and assigns them to the input buffer stage of the corresponding neuron. Simultaneously, a set of encapsulated neurons reads its state and intrinsic parameters (REGs of the same color within the same set of encapsulations in the figure) and searches the storage location of the first neuron in SRAM to obtain the weights of all its synapses. Next, the first neuron simultaneously extracts its 1152 inputs in parallel and divides them into different batches at certain intervals. Then, these input data are batched and sent to the accumulator for pulse accumulation or multiply-accumulate operations, and the acquired membrane voltage is passed to the next neuron in the same encapsulation for the next set of 1152 inputs. Since the computation of neurons within the same encapsulation is performed sequentially, access to the weight SRAM can also be performed sequentially, directly accessing the next address each time. After all N neurons have completed their work within one time step, the membrane level in SNN mode is compared with the threshold voltage and pulses are emitted, or in ANN mode, an activation function is activated. This operation only occurs in the last neuron; the previous N-1 neurons do not perform this operation. Finally, the membrane voltage and neuron state are updated according to the operating mode and written back to the corresponding registers and SRAM. In this way, the fan-in of the neuron is amplified to 1152×N by NPM, and a maximum of 73728 (N=64) input values ​​can be received simultaneously for computation.

[0073] In some possible embodiments of the present invention, the hybrid neural network is finely tuned by configuring neuron parameters or weight parameters in the synapse.

[0074] In a possible implementation, based on the basic LIF neuron model, the neuron package mechanism (NPM) is used to increase the fan-in of each neuron in the computational kernel to N×1152, where N ranges from 1 to 64.

[0075] In some possible embodiments of the present invention, the hybrid neural network accelerator further includes a synapse compression module, wherein the synapse compression module is used to compress the storage of SRAM to store only effective synapses and the weight parameters of the effective synapses, which is adapted to select only neurons containing the effective synapses to participate in the operation during neuron operation, and neurons not containing effective synapses do not participate in the operation, thereby avoiding the waste of neurons, wherein the effective synapse is the synapse that participates in the operation.

[0076] It is worth noting that in existing crossbar neural network accelerators, because each neuron selects a different input during computation, not all synapses in the neuromorphic core are used. Figure 2 This is a schematic diagram illustrating the oblique line phenomenon in the existing crossbar structure deployment model, such as... Figure 2 As shown in the example in (a), the 2×2 convolutional kernel slides across the feature map with a stride of 1, flattening the input into a 16×1 vector. However, each output neuron only takes 4 products as the accumulated weight window, ignoring the rest of the input; that is, this portion of the weights is ineffective. When all neurons are arranged together, a diagonal shape is formed, completely wasting synaptic and silicon area in other regions, resulting in extremely low synaptic resource utilization (only 25% in the figure). Furthermore, if the slope of this diagonal line is large, the shape will not be as... Figure 2 As shown in (a), the neurons extend diagonally but reach the bottom of the neuron array much earlier, resulting in the right-side neurons not being actually used, causing wasted fan-out. Figure 2 As shown in (b), this deployment phenomenon will greatly reduce the number of available fan-out neurons.

[0077] Therefore, the synaptic compression technology used in this invention can greatly increase the number of fan-outs of the computing core with less fixed resources, supporting up to 4096 fan-outs, and improving the resource utilization efficiency of synapses.

[0078] In some possible embodiments of the present invention, the step of compressing the storage of the SRAM to store only the effective synapses within the effective weight window and the weight parameters of the effective synapses specifically includes:

[0079] The synaptic region where each neuron actually performs calculations is defined as the effective weight window, and synapses that do not participate in the calculations are defined as invalid synapses.

[0080] Based on the effective weight window, the storage of the SRAM is adjusted so that it only stores synapses within the effective weight window and deletes the weight parameters of invalid synapses outside the effective weight window.

[0081] In possible embodiments, such as Figure 3 As shown, Figure 3This is a block diagram illustrating the working principle of the synaptic compression module provided by the present invention, showing how the present invention uses synaptic compression technology for improvement.

[0082] Specifically, the synaptic compression mechanism (SCM) used in this embodiment of the invention defines the synaptic region where each neuron actually performs calculations as the effective weight window (hereinafter referred to as the "window"). Based on this, the storage of the weight SRAM can be adjusted so that it only stores the synapses within the window. Figure 3 (The orange section in the middle), and at the same time remove invalid weights ( Figure 3 (Middle blue section). Based on this, by calculating the parameters indicating the specific location of the weight window, the composition of a fully encapsulated neuron can be described as follows:

[0083] (1)

[0084] in, This indicates how many complete invalid neurons were in front of the window. This indicates how many inactive synapses precede the window in the first effective neuron. The length of the weight window is defined, and U represents invalid synapses. If an invalid synapse is located after the window, it can be ignored after the window is calculated.

[0085] In some possible embodiments of the present invention, the step of selecting only neurons containing effective synapses to participate in the operation, and excluding neurons not containing effective synapses from the operation, specifically includes:

[0086] S1. Define neurons containing effective synapses as effective neurons and neurons not containing effective synapses as ineffective neurons.

[0087] S2. Starting from the first neuron in the neuron encapsulation group, search for effective neurons sequentially;

[0088] S3. Determine whether the currently searched neuron has reached the effective weight window;

[0089] S4. If not, then it is determined to be an invalid neuron, and the process jumps to S2;

[0090] S5. If so, sequentially search for the starting synapse position of the effective weight window within the neuron according to the interval between the input batch data, and start executing the neuron's operation at the starting synapse position.

[0091] In some possible embodiments of the present invention, the step of initiating neuronal operations at the initial synaptic location specifically includes:

[0092] Starting at the initial synapse position, neuron computation is performed on the effective synapses within the effective weight window;

[0093] If the effective weight window spans multiple neurons, two calculations are performed at the boundaries of each neuron;

[0094] After the calculation is completed in the effective weight window, the remaining invalid synapses in the neuron encapsulation group where the effective weight window is located are skipped, and the calculation is performed in the effective weight window of the next neuron encapsulation group.

[0095] Specifically, such as Figure 3 For example, starting with the first neuron in the encapsulation packet, first determine whether that neuron has reached the window (and...). off N If the size comparison is not found, the neuron is invalid, and the search proceeds to the next neuron; if it is found, the search begins by detecting which synapse of the current neuron the window starts from, sequentially searching at intervals equal to the size of the input batches defined in NPM. off index Compare sizes. Once a window is found, perform computations as described in NPM, accumulating input impulses or performing multiply-accumulate operations. Since the window may span multiple neurons ( Figure 3 (As shown in the example on the right), it may be necessary to perform the calculations at the boundaries of the two neurons in stages. After completing the calculations in the weight window, the remaining synapses can be skipped, the pointer identifying the neuron points to the next encapsulated neuron, and the same process is performed on it.

[0096] This embodiment utilizes Synaptic Compression (SCM) technology, which prunes invalid synaptic connections during model deployment. This eliminates the aforementioned diagonal line phenomenon (the diagonal line can be compressed into a rectangle because the two invalid regions above and below are removed). Calculations are performed only when the synapse is valid, making full use of the neurons wasted in the diagonal line shape. This greatly improves hardware resource efficiency and the number of fan-outs of the neuromorphic core, thus significantly reducing the total number of computational cores required to deploy a model.

[0097] In a possible embodiment, each core of the present invention can provide up to 4096 neurons fan-out.

[0098] This invention uses synaptic compression technology to eliminate the diagonal line phenomenon when the model is deployed on hardware, and no longer stores invalid synapses and weights, so that all intranuclear synapses can be effectively utilized and all intranuclear neurons can be activated and output.

[0099] In practical applications of ANNs and SNNs, multi-core processors improve computing power, but the communication scheme between internal processing cores has a significant impact on processor efficiency and latency. Various proposed routing communication networks struggle to balance data transmission speed and congestion handling. Specifically, existing multi-core neural network accelerators typically use tree structures or 2D-mesh structures for inter-core data communication, each with its own distinct advantages and disadvantages. For large-scale designs, tree structures can reduce latency but have very low throughput because data traffic concentrates at the root node, leading to congestion. 2D-mesh structures, on the other hand, offer higher throughput but relatively higher latency. This invention designs a hybrid routing network that combines the characteristics of both, improving data transmission speed while mitigating internal network congestion to some extent.

[0100] To address the aforementioned issues, this invention proposes a hybrid routing network that combines tree structure and 2D-mesh structure, which can improve data transmission speed while alleviating network congestion to some extent.

[0101] In some possible embodiments of the present invention, the hybrid neural network accelerator further includes an L1 router;

[0102] The hybrid neural network accelerator connects to one L1 router for every four neural computing cores, forming a quadtree structure. The 16 L1 routers form a two-dimensional grid structure.

[0103] Each L1 router includes four pairs of asynchronous first transmitter-receiver pairs, which are used for data interaction with adjacent L1 routers in four directions of the two-dimensional mesh structure.

[0104] Each L1 router also includes four pairs of asynchronous second transmitter-receivers for data interaction between the L1 router and each neural computing core.

[0105] Specifically, this embodiment provides an implementation of a hybrid routing network that combines the characteristics of tree structures and 2D-mesh structures. This hybrid routing network can improve data transmission speed and reduce latency while alleviating network congestion to some extent.

[0106] In a possible embodiment, the L1 router further includes a distributor; the distributor is used to determine the transmission direction of data frames within the L1 router, the data frames being data or pulses carrying address event representation rules.

[0107] In possible embodiments, such as Figure 4 As shown, Figure 4 The overall multi-core structure, hybrid routing network system, and internal structure of L1 routing of the present invention are demonstrated.

[0108] In a possible embodiment, the hybrid neural network accelerator provided by this invention consists of 64 neural computation cores. In SNN mode, each core is equipped with 512 neurons as output, while in ANN mode there are 4096 neurons. Every four computation cores are connected to an L1 router, forming a quadtree structure. Sixteen L1 routers form a two-dimensional grid structure. All twelve L1 routers on the perimeter can interact with the outside world, and the entire system can achieve a throughput of 8.9 Gbps. This hybrid routing network system also supports multicast from the L1 routers to the four cores based on multicast addresses. Each L1 router has four pairs of asynchronous transmitter-receivers for interacting with adjacent routers in four directions at the two-dimensional grid level, and four additional pairs for interaction between the L1 routers and computation cores. Each computation core also has one pair of transmitter-receivers to connect the routers and controllers (including crossbar structures) within the core. The L1 routers also have a distributor to determine which direction a data frame should be sent in. The data frame is 64 bits and carries data or pulses conforming to Address Event Representation (AER) rules.

[0109] In a possible embodiment, to measure the internal data transmission distance, a step is defined as sending a data frame from one computational core to an adjacent computational core. Thus, the longest path required for data transmission within this hybrid network is 8 steps (green dashed line), which is significantly smaller than the 14 steps (red dashed line) of a pure two-dimensional mesh structure. According to calculations, the average distance between any two computational cores in the network is 4.46875 steps, compared to 5.25 steps for a pure two-dimensional mesh structure and 5.34375 steps for a pure quadtree structure. It can be seen that this invention successfully reduces communication latency and also alleviates data congestion at higher-level nodes.

[0110] Based on the above embodiments, the neuron encapsulation technology (NPM), synaptic compression technology (SCM), and hybrid routing network system proposed in this invention are applicable to both ASICs and FPGAs.

[0111] Table 1 shows a performance comparison of different FPGA-based hybrid neural network accelerator design schemes. The table mainly lists the test results comparison of similar technical schemes on FPGA.

[0112] Table 1

[0113] This invention Comparison Option 1 Comparison Option 2 Comparison Option 3 Comparison Option 4 Comparison Option 5 platform XCVU9P XCKU115 XCKU115 VC7VX485T XC7VX690T ASIC Operating frequency (MHz) 120 120 140 50 - 100 Number of neurons Max. 4096 256 2048 512 1024 162 Weight precision (bit) 1 / 2 / 4 / 8 8 4 / 8 8 8 4 SNN computing power (GSOP / s) 260.2 2.56 68.2 0.136 0.73 1.55 SNN Energy Efficiency (GSOP / W) 74.34 1.53 16.5 0.544 1.04 14.49 Maximum fan-in (per neuron) 73728 512 256 <![CDATA[820 c ]]> 1024 113 Maximum fan-out (per core) 4096 256 128 512 64 162 MNIST accuracy 98.7% 96.9% 96.9% 96% 98% - Fashion MNIST accuracy 90.65% - - - - - CIFAR10 accuracy 90.98% - 85.7% - - -

[0114] As shown in Table 1, this invention achieves high computing power and energy efficiency in FPGA design, successfully improves the fan-in and fan-out of the accelerator, and maintains high model accuracy. The specific comparison results can be summarized as follows:

[0115] First, very few existing neural network accelerators support both ANN and SNN computation modes simultaneously. This invention designs a neuron model that can support multiple computational operations, enabling it to perform hybrid intelligence of ANN and SNN.

[0116] Second, existing neural network accelerators typically improve the fan-in of neurons and the fan-out of computational kernels by directly stacking the number of the two types of computational units, which has low configurability and resource utilization efficiency. This invention, by designing and applying neuron encapsulation technology and synaptic compression technology, can directly improve the fan-in and fan-out of individual neurons and computational kernels with less fixed resources, thereby improving the hardware utilization rate when deploying the model.

[0117] Thirdly, the publicly available accelerators typically use a limited number of test datasets. This invention was deployed and tested on three publicly available datasets (MNIST, Fashion MNIST, and CIFAR10), and the results demonstrate the effectiveness of this invention.

[0118] The hybrid neural network accelerator provided by this invention employs neuron encapsulation technology to significantly increase neuron fan-in, and synaptic compression technology to significantly increase computational core fan-out, thereby improving hardware resource utilization efficiency during model deployment. The use of a hybrid on-chip routing system achieves a balance between communication latency and data congestion, with good results in both aspects. Compared to traditional methods of adapting accelerator fan-in by stacking the number of neurons or splitting the neural network model, this invention uses neuron encapsulation technology to make the neuron fan-in size configurable, providing a considerable maximum of 73,724 fan-ins. Compared to traditional model deployment schemes, the synaptic compression technology used in this invention eliminates the storage and computation of invalid weights, operating only on valid weights. Therefore, this invention can increase the number of computational core fan-outs (maximum 4096), improving hardware resource utilization efficiency. Test results from the on-chip network show that this invention successfully reduces communication latency within the accelerator, and based on the internal two-dimensional mesh structure, it can also alleviate data congestion at higher-level nodes.

[0119] In a possible embodiment, the present invention also provides a method for accelerating computation using hybrid neural network neurons, the method being implemented using a hybrid neural network accelerator as described above, the hybrid neural network accelerator comprising M neural computation kernels, each of the neural computation kernels comprising Q neurons. Figure 5 This is a flowchart illustrating the accelerated computation method for hybrid neural network neurons provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the method includes:

[0120] Step 510: If the input to the neural computation kernel is greater than the fan-in threshold W of a single neuron in the neural computation kernel, determine the number N of neurons participating in the computation within the neural computation kernel.

[0121] Step 520: Encapsulate the N neurons into a neuron encapsulation group, extract W inputs through the first neuron in the neuron encapsulation group, and obtain the weight parameters of each synapse in the first neuron according to the storage location of the first neuron in the SRAM.

[0122] Step 530: Divide the W inputs of a single neuron into multiple input batches according to a preset interval, send the input batches to the membrane potential accumulator for calculation with the weight parameters, and pass the calculated membrane potential to the next neuron in the neuron encapsulation group for the calculation of the next group of W inputs, until N neurons have completed the calculation.

[0123] Based on the above embodiments, the synaptic compression module in the hybrid neural network accelerator can be used to compress the SRAM storage to store only the effective synapses and their weight parameters. This is suitable for selecting only neurons containing the effective synapses to participate in the computation during neuron operation, while neurons without effective synapses do not participate in the computation, thus avoiding the waste of neurons. Here, the effective synapse is the synapse that participates in the computation.

[0124] It is understood that the hybrid neural network accelerator provided in any of the above embodiments can be applied to the hybrid neural network neuron acceleration calculation method provided in this embodiment. The implementation principle and beneficial effects of the hybrid neural network neuron acceleration calculation method provided in this embodiment are similar to the implementation principle and beneficial effects of the hybrid neural network accelerator shown in the above embodiments. Therefore, the implementation principle and beneficial effects of the hybrid neural network accelerator shown in the above embodiments can be referred to, and will not be repeated here.

[0125] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logic instructions in the memory 630 to execute a hybrid neural network neuron acceleration computation method. The method includes: determining the number N of neurons participating in the computation within the neural computation kernel when the input of the neural computation kernel is greater than the fan-in threshold W of a single neuron of the neural computation kernel; encapsulating the N neurons into a neuron encapsulation group; extracting W inputs through the first neuron in the neuron encapsulation group; and obtaining the weight parameters of each synapse in the first neuron according to the storage location of the first neuron in the SRAM; dividing the W inputs of a single neuron into multiple input batches according to a preset interval; sending the input batches to a membrane potential accumulator for computation with the weight parameters; and transferring the computed membrane potential to the next neuron in the neuron encapsulation group for computation of the next group of W inputs, until N neurons have completed the computation.

[0126] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the hybrid neural network neuron acceleration computation method provided by the above methods. The method includes: determining the number N of neurons participating in the computation within the neural computation kernel when the input of the neural computation kernel is greater than the fan-in threshold W of a single neuron of the neural computation kernel; encapsulating the N neurons into a neuron encapsulation group; extracting W inputs through the first neuron in the neuron encapsulation group; and obtaining the weight parameters of each synapse in the first neuron according to the storage location of the first neuron in SRAM; dividing the W inputs of a single neuron into multiple input batches according to a preset interval; sending the input batches to a membrane potential accumulator for computation with the weight parameters; and transferring the computed membrane potential to the next neuron in the neuron encapsulation group for computation of the next group of W inputs, until N neurons have completed the computation.

[0128] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for accelerating computation of hybrid neural network neurons provided by the above methods. The method includes: determining the number N of neurons participating in the computation within the neural computation kernel when the input of the neural computation kernel is greater than the fan-in threshold W of a single neuron of the neural computation kernel; encapsulating the N neurons into a neuron encapsulation group; extracting W inputs through the first neuron in the neuron encapsulation group; and obtaining the weight parameters of each synapse in the first neuron according to the storage location of the first neuron in SRAM; dividing the W inputs of a single neuron into multiple input batches according to a preset interval; sending the input batches to a membrane potential accumulator for computation with the weight parameters; and transferring the computed membrane potential to the next neuron in the neuron encapsulation group for computation of the next group of W inputs, until N neurons have completed the computation.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort. Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hybrid neural network accelerator, having ANN and SNN accelerated computation capabilities, wherein the hybrid neural network accelerator comprises M neural computation kernels, each of the neural computation kernels comprising Q neurons, characterized in that, Also includes: Register file, SRAM, membrane potential accumulator, transmitter, neuron encapsulation module, synaptic compression module; The register file is used to store the operating state and intrinsic parameters of the neuron. Each neuron integrates W synapses, and the W synapses are used to receive W input parameters of the neuron. The intrinsic parameters include the fan-in threshold W of each neuron. The SRAM is used to store the location information of each synapse of the neuron and the weight parameters of each synapse; The membrane potential accumulator is used to perform corresponding calculations on the input parameters and the weight parameters; The transmitter is used to activate the membrane voltage and transmit voltage pulse of the membrane potential accumulator; The neuron encapsulation module is used when the neural computing core needs to receive inputs greater than the fan-in threshold W of a single neuron simultaneously. It encapsulates N neurons into a neuron encapsulation group, extracts W inputs through the first neuron in the neuron encapsulation group, obtains the weight parameters of each synapse in the first neuron according to the storage location of the first neuron in SRAM, divides the W inputs into multiple input batches according to a preset interval, sends the input batches to the membrane potential accumulator to perform calculations with the weight parameters, and transmits the calculated membrane potentials to the next neuron in the neuron encapsulation group for the calculation of the next group of W inputs, until N neurons have completed the calculations. The synaptic compression module is used to compress the SRAM storage to store only the effective synapses and their weight parameters. This is suitable for selecting only neurons containing the effective synapses to participate in the operation during neuron computation, while neurons without effective synapses do not participate in the operation, thus avoiding the waste of neurons. The effective synapse is the synapse that participates in the operation. The step of selecting only neurons containing effective synapses to participate in the computation, and excluding neurons without effective synapses from the computation, specifically includes the following steps: S1. Define neurons containing effective synapses as effective neurons and neurons not containing effective synapses as ineffective neurons. S2. Starting from the first neuron in the neuron encapsulation group, search for effective neurons sequentially; S3. Determine whether the currently searched neuron has reached the effective weight window, where the effective weight window is the synaptic region where each neuron is actually calculated. S4. If not, then it is determined to be an invalid neuron, and the process jumps to S2; S5. If so, sequentially search for the starting synapse position of the effective weight window within the neuron according to the interval between the input batch data, and start executing the neuron's operation at the starting synapse position.

2. The hybrid neural network accelerator according to claim 1, characterized in that, The neuron encapsulation module is also used to, when the last neuron in the neuron encapsulation group performs a computation, the SNN emits a pulse by comparing the membrane level with the threshold voltage, or the ANN emits a pulse by activating a function; and, Update the membrane voltage and the neuron's operating state, and write the updated membrane voltage and the neuron's operating state back to the register file and SRAM, respectively.

3. The hybrid neural network accelerator according to claim 1, characterized in that, Compressing the SRAM storage to store only the effective synapses and their weight parameters specifically includes: Synapses that do not participate in computation for each neuron are defined as invalid synapses; based on the effective weight window, the storage of the SRAM is adjusted so that it only stores synapses within the effective weight window, and the weight parameters of invalid synapses outside the effective weight window are deleted.

4. The hybrid neural network accelerator according to claim 3, characterized in that, The neuron begins its computation at the initiation synapse location, specifically including: Starting at the initial synapse position, neuron computation is performed on the effective synapses within the effective weight window; If the effective weight window spans multiple neurons, two calculations are performed at the boundaries of each neuron; After the calculation is completed in the effective weight window, the remaining invalid synapses in the neuron encapsulation group where the effective weight window is located are skipped, and the calculation is performed in the effective weight window of the next neuron encapsulation group.

5. The hybrid neural network accelerator according to any one of claims 1-4, characterized in that, The neuron includes a neuron model and a neuron operator.

6. The hybrid neural network accelerator according to claim 1, characterized in that, The value of N ranges from 2 to 64, and the maximum value of W is 1152.

7. The hybrid neural network accelerator according to claim 1, characterized in that, The hybrid neural network accelerator includes 64 neural computation cores, each of which includes 512 neurons; the hybrid neural network accelerator also includes an L1 router. The hybrid neural network accelerator connects to one L1 router for every four neural computing cores, forming a quadtree structure. The 16 L1 routers form a two-dimensional grid structure. Each L1 router includes four pairs of asynchronous first transmitter-receiver pairs, which are used for data interaction with adjacent L1 routers in four directions of the two-dimensional mesh structure. Each L1 router also includes four pairs of asynchronous second transmitter-receiver pairs for data interaction between the L1 router and each neural computing core; The L1 router also includes a distributor; The distributor is used to determine the transmission direction of data frames within the L1 router, wherein the data frames are data or pulses that conform to the address event representation rules.

8. A method for accelerating computation using hybrid neural network neurons, characterized in that, The method is implemented using the hybrid neural network accelerator according to any one of claims 1-7, wherein the hybrid neural network accelerator has ANN and SNN accelerated computation functions, includes M neural computation kernels, each of the neural computation kernels including Q neurons, and the method includes: If the input to the neural computation kernel is greater than the fan-in threshold W of a single neuron in the neural computation kernel, determine the number N of neurons participating in the computation within the neural computation kernel; The N neurons are encapsulated into a neuron encapsulation group. W inputs are extracted through the first neuron in the neuron encapsulation group. The weight parameters of each synapse in the first neuron are obtained according to the storage location of the first neuron in the SRAM. The W inputs of a single neuron are divided into multiple input batches at preset intervals. The input batches are sent to a membrane potential accumulator for calculation with the weight parameters. The calculated membrane potentials are then passed to the next neuron in the neuron encapsulation group for the calculation of the next group of W inputs, until N neurons have completed the calculation.

Images