Multi-model fusion neural network accelerator supporting out-of-order sparse output

Abstract

The application provides a multi-model fusion neural network accelerator supporting out-of-order sparse output, comprising a data cache module for storing weights, input data and calculation parameters, and a dynamic allocator for determining the data stored to be sent to idle calculation units; each calculation unit performs a multiply-accumulate operation; each prediction module predicts whether to terminate the calculation of a subsequent time step according to the multiply-accumulate operation result; the result output time of each calculation unit is different under the action of the prediction module; a serial-in-serial-out register receives and caches the multiply-accumulate operation results of each calculation unit; a post-processing module extracts the multiply-accumulate operation results of each calculation unit from the serial-in-serial-out register according to the type of the neural network calculated; and an output module outputs the processing result of the post-processing module based on an asynchronous handshake protocol, which, in combination with the characteristics of various neural network models, decouples the calculation units and outputs sparse prediction, compresses the waiting time of the calculation units to the shortest, and achieves the maximum hardware utilization.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-model fusion neural network accelerator that supports out-of-order sparse output. Background Technology

[0002] The computation of neural network models requires hardware support. Because the training and inference processes of neural networks involve significant computational and storage demands, traditional central processing units (CPUs) often cannot meet their high-performance and low-latency requirements, while large GPUs cannot meet the power consumption and latency limitations of edge devices. Therefore, to overcome this deficiency and deploy neural network models to mobile or edge devices for task execution, dedicated hardware accelerators for neural network computation need to be designed. At the data flow level, due to variations in data sparsity and scale, the effective load of each computational unit in a neural network accelerator typically differs greatly, resulting in inconsistent computation progress. This inevitably leads to situations where units that finish computation earlier wait for units that finish later, reducing hardware utilization. In related technologies, accelerators designed for different computational methods and input data models are usually dedicated. For example, Spiking Neural Networks (SNNs) have an additional time dimension compared to Convolutional Neural Networks (CNNs), requiring continuous computation across multiple time steps to obtain a sufficiently reliable result. Furthermore, the data format is single-bit pulses rather than multi-bit numerical values, leading to significant differences in both data storage and computational data flow between the two types of accelerators. Therefore, accelerator fusion is needed for different models. Traditional fusion methods include heterogeneous and homogeneous approaches. The heterogeneous approach designs multiple dedicated accelerators that can exchange data with each other, reducing the cost of data interaction, but it still cannot unify different accelerators and incurs significant area overhead. While the homogeneous approach can provide support for multiple models at the computational unit level or at the functional module and data flow level, these methods are currently limited to convolutional models such as CNN and SCNN, and are powerless for models with completely different computational mechanisms, resulting in low versatility.

[0003] On the other hand, within an accelerator, a large number of computing units are typically configured to enhance its computing power. During computation, these units may receive different computational tasks. For example, some units might be configured to compute the first layer of a neural network, while others compute the second layer. This results in different data dimensions and quantities for each unit, leading to varying computation speeds. Since the accelerator typically reads data from its cache uniformly and sends data synchronously to all computing units, some faster units may complete their computations first and then wait for the later-completing units. During this waiting period, the earlier-completing units remain idle, wasting both time and energy resources. Data sparsity can also contribute to this phenomenon: data like 0, which multiplies by any number, has no impact on the result and is skipped during computation. Units with more 0 data may compute faster than others, also leading to lower utilization. Summary of the Invention

[0004] This invention provides a multi-model fusion neural network accelerator that supports out-of-order sparse output, in order to solve the shortcomings of traditional multi-model fusion neural network accelerators, which cannot unify different accelerators, have large area overhead, are powerless to compute models with completely different mechanisms, and waste hardware resources.

[0005] This invention provides a multi-model fusion neural network accelerator that supports out-of-order sparse output, comprising:

[0006] The data caching module is used to store weight data, input data and calculation parameters. The data caching module determines which idle computing unit to send the stored data to through a dynamic allocator.

[0007] Multiple computing units are used to obtain weight data, input data and computing parameters from the data caching module. Each computing unit is configured to perform a multiply-accumulate operation based on the weight data, input data and computing parameters.

[0008] Multiple prediction modules are connected to a corresponding computing unit. Based on the result of the multiply-accumulate operation, they predict whether to terminate the calculation of subsequent time steps. Under the action of the prediction modules, the output time of the multiply-accumulate operation results of each computing unit is different.

[0009] The serial-in-serial-out register is used to receive and buffer the results of multiplication and accumulation operations from various computational units;

[0010] The post-processing module is used to extract the multiplication and accumulation operation results of each computing unit from the serial-in-serial-out register according to the type of the computed neural network, and process the multiplication and accumulation operation results of each computing unit.

[0011] The output module is used to output the processing results of the post-processing module based on the asynchronous handshake protocol.

[0012] The multi-model fusion neural network accelerator supporting out-of-order sparse output provided by the present invention further includes a preprocessing module for detecting the data in the data caching module and removing zero values ​​or invalid time step data from the input data.

[0013] According to the multi-model fusion neural network accelerator supporting out-of-order sparse output provided by the present invention, the dynamic allocator uses a polling algorithm to allocate preprocessed data to idle computing units in order to balance the computing load.

[0014] According to the multi-model fusion neural network accelerator supporting out-of-order sparse output provided by the present invention, the data caching module is further configured to:

[0015] For convolutional neural networks, the data width is 8 bits;

[0016] For spiking neural networks with a data width of 1 bit, supporting neural network computation of no more than 8 time steps, the data of these 8 time steps are stored sequentially in 8-bit data according to the time step from first to last, and reused with the data space of the convolutional neural network.

[0017] According to the multi-model fusion neural network accelerator supporting out-of-order sparse output provided by the present invention, the data caching module supports a multi-mode storage structure, and switches between convolution sliding window block storage and matrix multiplication row storage by control signal.

[0018] According to the multi-model fusion neural network accelerator supporting out-of-order sparse output provided by the present invention, the data caching module is further configured to:

[0019] For convolutional neural networks, the input feature map is split into blocks by the convolutional sliding window, and the data points in these blocks from top left to bottom right are stored together according to their corresponding positions and stored in different storage areas.

[0020] A single row in the storage area can store data from multiple channels at the same location. During calculation, the position after sliding is compared with the position before sliding to update the changed area.

[0021] For network models that use matrix multiplication, we use matrix multiplication row storage and read data row by row to ensure a one-to-one correspondence between input data and stored data.

[0022] According to the multi-model fusion neural network accelerator supporting out-of-order sparse output provided by the present invention, the plurality of computing units are further configured to:

[0023] For convolutional neural networks or Transformer models, multi-bit data will be directly fed into the corresponding computation unit;

[0024] For spiking neural networks or Spiking-Transformer models, computation can be performed either sequentially or in parallel over time.

[0025] The rule for time serial processing includes retrieving pulse data from only one time step, while ignoring other time steps; the rule for time parallel processing includes loading pulse data from multiple time steps, retrieving pulse data from one time step at a time step in sequence, and sending it to the computing unit until enough time steps for the required calculation are retrieved.

[0026] According to the multi-model fusion neural network accelerator supporting out-of-order sparse output provided by the present invention, the post-processing module includes an activation module;

[0027] The activation module uses a lookup table to perform non-linear mapping for convolutional neural networks and Transformer models in order to process the data output by the computing unit.

[0028] According to the multi-model fusion neural network accelerator supporting out-of-order sparse output provided by the present invention, the post-processing module includes an early termination module.

[0029] The early termination module uses an early termination algorithm to determine whether to terminate the network model calculation prematurely for spiking neural networks and Spiking-Transformer models.

[0030] According to the multi-model fusion neural network accelerator supporting out-of-order sparse output provided by the present invention, the output module is further configured to:

[0031] For spiking neural networks, update the membrane level data at each time step, and calculate the address at which the calculation result is output for each set of data based on the number of times the input data is read.

[0032] This invention provides a multi-model fusion neural network accelerator supporting out-of-order sparse output, comprising: a data caching module for storing weight data, input data, and computational parameters, wherein the data caching module determines which idle computational unit to send the stored data to through a dynamic allocator; multiple computational units for obtaining weight data, input data, and computational parameters from the data caching module, each configured to perform a multiply-accumulate operation based on the weight data, input data, and computational parameters; and multiple prediction modules, each connected to a corresponding computational unit, predicting whether to terminate the computation of subsequent time steps based on the result of the multiply-accumulate operation; and the results of the multiply-accumulate operation of each computational unit under the action of the prediction modules. The output times differ; a serial-in / serial-out register is used to receive and buffer the multiplication and accumulation operation results of each computational unit; a post-processing module is used to extract the multiplication and accumulation operation results of each computational unit from the serial-in / serial-out register according to the type of neural network being computed, and process the multiplication and accumulation operation results of each computational unit; an output module is used to output the processing results of the post-processing module based on an asynchronous handshake protocol. This invention combines the characteristics of various neural network models, decouples the computation process of each computational unit, makes them independent, and performs sparse prediction of output through a prediction module to provide better dynamics, which can compress the waiting time of the computational unit to the shortest possible time, thereby achieving maximum hardware utilization. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0034] Figure 1 This is a functional structure diagram of a multi-model fusion neural network accelerator that supports out-of-order sparse output, provided in an embodiment of the present invention.

[0035] Figure 2 These are comparison diagrams of embodiments of the present invention and existing technologies;

[0036] Figure 3 This is an example diagram of a data organization method that can reduce the number of data reads when the convolution window slides, provided by an embodiment of the present invention;

[0037] Figure 4 This is a neural network accelerated data flow graph that supports out-of-order sparse output, provided by an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0039] Figure 1 A functional block diagram of a multi-model fusion neural network accelerator supporting out-of-order sparse output provided in an embodiment of the present invention is shown below. Figure 1 As shown, the multi-model fusion neural network accelerator supporting out-of-order sparse output provided in this embodiment of the invention includes:

[0040] The data caching module is used to store weight data, input data and calculation parameters. The data caching module determines which idle computing unit to send the stored data to through a dynamic allocator.

[0041] Multiple computing units are used to obtain weight data, input data and computing parameters from the data caching module. Each computing unit is configured to perform a multiply-accumulate operation based on the weight data, input data and computing parameters.

[0042] In this embodiment of the invention, there are 16 computing units, each capable of receiving up to 144 pairs of 8-bit input and 8-bit weight data (16-channel 3×3 convolution kernels), with each pair constituting a synapse. The computing units primarily multiply these 144 pairs of data accordingly, and then accumulate the product results.

[0043] Multiple prediction modules are connected to a corresponding computing unit. Based on the result of the multiply-accumulate operation, they predict whether to terminate the calculation of subsequent time steps. Under the action of the prediction modules, the output time of the multiply-accumulate operation results of each computing unit is different.

[0044] The prediction module can predict whether the received data is positive, negative, or zero. It uses algorithms such as bit-by-bit product calculation and newly encoded product to achieve spatially sparse output. For example, it can terminate the calculation of subsequent bits based on the result. Each computation unit is equipped with a prediction module, supporting the prediction of the multiplication and accumulation results of up to 16 channels of data. To provide prediction capabilities for a larger number of channels, every 2, 4, 8, and 16 computation units are also equipped with a shared prediction module, thus supporting output prediction capabilities for up to 256 channels.

[0045] The serial-in-serial-out register is used to receive and buffer the results of multiplication and accumulation operations from various computational units;

[0046] In this embodiment of the invention, the output results of each calculation unit will not be output simultaneously under the action of the prediction module. Therefore, a serial-in-serial-out register (FIFO) is designed to save all the calculation results to prevent loss and pass them through the subsequent processing in sequence.

[0047] The post-processing module is used to extract the multiplication and accumulation operation results of each computing unit from the serial-in-serial-out register according to the type of the computed neural network, and process the multiplication and accumulation operation results of each computing unit.

[0048] The output module is used to output the processing results of the post-processing module based on the asynchronous handshake protocol.

[0049] In this embodiment of the invention, the output module sequentially sends the result data and its corresponding coordinates out of the accelerator.

[0050] Neural network models, including CNNs, SNNs, and Transformers, can function effectively in various applications and environments. However, common neural network accelerators are typically designed for a specific application or network model, usually undergoing specific optimizations and preprocessing based on the model's unique conditions. Because different models have significantly different computational methods and input data, accelerator designs are often dedicated, limiting their development. For example... Figure 2 As shown in (1), these methods are basically limited to convolutional models such as CNN and SCNN, and are powerless for models with completely different computational mechanisms. Their versatility is still not high. Related technologies are usually implemented through software or hardware methods, such as Figure 2 As shown in (2), some backlogged tasks are redistributed from the original computing units to other units to balance the task load between units as much as possible, so that the computing units maintain a similar computing speed in each round of computing, thereby reducing the waiting time between them. However, this method can only reduce, but cannot completely eliminate, the waiting time, and there is still room for improvement in hardware utilization.

[0051] To reduce idle time of computing resources, after each computing unit completes a round of tasks and outputs the multiplication-accumulation result, it checks whether all data written to the accelerator cache SRAM has been calculated. If so, no more data is read; otherwise, the control logic reads a set of data from the cache, preprocesses it, and sends it to an idle computing unit. In other words, the reading process is sequential, while the calculation process is out-of-order. Therefore, not all computing units need to be idle; as long as there are available hardware resources, data can be read and calculated. This decouples the calculation processes between different computing units, compressing the idle time of computing units to a minimum cache read time.

[0052] This invention provides a multi-model fusion neural network accelerator supporting out-of-order sparse output, comprising: a data caching module for storing weight data, input data, and computational parameters, wherein the data caching module uses a dynamic allocator to determine which idle computational unit to send the stored data to; multiple computational units for obtaining weight data, input data, and computational parameters from the data caching module, each configured to perform a multiply-accumulate operation based on the weight data, input data, and computational parameters; and multiple prediction modules, each connected to a corresponding computational unit, predicting whether to terminate the computation of subsequent time steps based on the result of the multiply-accumulate operation; and the multiply-accumulate operation of each computational unit under the action of the prediction modules. The output times differ; a serial-in / serial-out register is used to receive and buffer the multiplication and accumulation operation results of each computing unit; a post-processing module is used to extract the multiplication and accumulation operation results of each computing unit from the serial-in / serial-out register according to the type of neural network being computed, and process the multiplication and accumulation operation results of each computing unit; an output module is used to output the processing results of the post-processing module based on an asynchronous handshake protocol. This invention combines the characteristics of various neural network models, decouples the computation process of each computing unit, makes them independent, and performs sparse prediction of output through a prediction module to provide better dynamics, which can compress the waiting time of computing units to the shortest possible time, thereby achieving maximum hardware utilization.

[0053] Based on any of the above embodiments, the multi-model fusion neural network accelerator that supports out-of-order sparse output further includes a preprocessing module for detecting the data in the data caching module and removing zero values ​​or invalid time step data from the input data.

[0054] In this embodiment of the invention, the preprocessing module is also used to generate data that may be needed in the output sparse prediction process.

[0055] In this embodiment of the invention, the dynamic allocator uses a round-robin algorithm to distribute the preprocessed data to idle computing units in order to balance the computing load.

[0056] In this embodiment of the invention, after the prediction module processes the data, some units may still be calculating the value of a certain output point, while others have moved to other output locations. The output address is used to identify the output direction of each calculation unit's result, preventing order confusion and reducing idle time of computing resources. To ensure a more balanced distribution of computing load among the units, the allocator uses a round-robin algorithm when sending data after preprocessing. The calculation unit that most recently received data has the lowest priority when data is sent next, with the priorities of the remaining units increasing sequentially. Data is sent only to the highest-priority idle unit each time, thus minimizing the possibility of some units continuously calculating while others remain idle for extended periods.

[0057] Based on any of the above embodiments, the data caching module is further configured to:

[0058] For convolutional neural networks, the data width is 8 bits;

[0059] For spiking neural networks with a data width of 1 bit, supporting neural network computation of no more than 8 time steps, the data of these 8 time steps are stored sequentially in 8-bit data according to the time step from first to last, and reused with the data space of the convolutional neural network.

[0060] In this embodiment of the invention, calculation parameters such as neuron membrane potential, electrical parameter values, and some thresholds used in the calculation, as well as input data, weights, and calculation parameters are all stored in SRAM.

[0061] In this embodiment of the invention, the multi-model fusion neural network accelerator that supports out-of-order sparse output also includes a global register, in which some important indicative parameters such as the model category being calculated, the current time step, the current network layer number, and data coordinates are stored.

[0062] The parameters or weights of neurons can be reconfigured for fine-tuning of the network, supporting a large number of neuron models, such as the Accumulation-Fire (IF) model, the Leakage-Accumulation-Fire (LIF) model, and the Degenerate Izhikevich model. The number of synapses supported by each neuron is configurable, up to a maximum of 2304. To support different data storage types simultaneously, the CNN data width is specified as 8 bits, and the SNN spike width is 1 bit. It supports a maximum of 8 time steps of neural network computation. The data from these 8 time steps is stored sequentially in 8-bit data blocks, thus achieving data space reuse with the CNN and improving storage efficiency.

[0063] In this embodiment of the invention, the data caching module supports a multi-mode storage structure, switching between convolution sliding window storage and matrix multiplication row storage via control signals.

[0064] In this embodiment of the invention, the data caching module is further configured to:

[0065] (1) For convolutional neural networks, the input feature map is split into blocks by convolutional sliding window storage, and the data points from the top left to the bottom right inside these blocks are stored together according to their corresponding positions and stored in different storage areas.

[0066] A single row in the storage area can store data from multiple channels at the same location. During calculation, the position after sliding is compared with the position before sliding to update the changed area.

[0067] In this embodiment of the invention, the input feature map is split into blocks using a convolutional sliding window. Data points within these blocks, from top left to bottom right, are stored together according to their corresponding positions, with different colors stored in different storage areas. To improve data parallelism, a single row in the storage area can store data from multiple channels at the same location. During computation, it is only necessary to compare the position after sliding with the position before sliding and update the changed area accordingly.

[0068] like Figure 3As shown, taking a 3×3 convolution kernel as an example, the input feature map is divided into 3×3 blocks. Within each block, the nine data points from top left to bottom right are stored together according to their corresponding positions. Points of the same color represent data points at the same position within each block, and different colors are stored in nine separate storage areas. To improve data parallelism, each row in a storage area can store data from 16 channels at the same position, totaling 144 bits. During computation, since the convolution window needs to slide, each time the position after sliding is compared with the position before sliding, and the changed area is updated. For example, with a sliding stride of 1, initially all nine storage areas need to be opened simultaneously, reading out the first row of data (including the corresponding channel), corresponding to the first convolution window: A0, A1, A2, A3, A4, A5, A6, A7, A8. The second convolution window slides horizontally to A1, A2, A9, A4, A5, A12, A7, A8, A15. Compared to the previous data, only three points need to be updated: A0→A9, A3→A12, and A6→A15. Therefore, only storage areas 0, 3, and 6 need to be opened. If all data needs to be retrieved again, all nine storage areas need to be opened, which greatly reduces the overhead of reading the cache. This process continues, opening storage areas 1, 4, and 7 next time, then 2, 5, and 8 the following time, and finally returning to storage areas 0, 3, and 6. If vertical sliding is required, such as A0, A1, A2, A3, A4, A5, A6, A7, A8 sliding to A3, A4, A5, A6, A7, A8, A21, A22, A23, then storage areas 0, 1, and 2 need to be opened, again requiring only updating 1 / 3 of the data, and so on. To reduce the frequency of sliding window data updates, when the sliding window reaches the end of the input feature map, it will not move to a new line to start again, but will slide directly down one line to maintain the characteristic that 2 / 3 of the data remains unchanged. If the calculation is a pulse network with time steps, then as described above, 1 bit of data is taken sequentially according to the time step for calculation until all bits are taken.

[0069] (2) For network models that use matrix multiplication, data is read row by row through matrix multiplication to make the input data and the stored data correspond one-to-one.

[0070] Based on any of the above embodiments, the plurality of computing units are further configured to:

[0071] For convolutional neural networks or Transformer models, multi-bit data will be directly fed into the corresponding computation unit;

[0072] For spiking neural networks or Spiking-Transformer models, computation can be performed either sequentially or in parallel over time.

[0073] The rule for time serial processing includes retrieving pulse data from only one time step and ignoring other time steps; the rule for time parallel processing includes loading pulse data from multiple time steps, retrieving pulse data from one time step at a time step in sequence and sending it to the computing unit until enough time steps for calculation are retrieved, without waiting for the current time step to be calculated before retrieving the next time step.

[0074] Based on any of the above embodiments, the post-processing module includes an activation module;

[0075] The activation module uses a lookup table to perform non-linear mapping for convolutional neural networks and Transformer models in order to process the data output by the computing unit.

[0076] In this embodiment of the invention, the lookup table supports most nonlinear functions such as ReLU, Sigmoid, and Softmax to process the results calculated by the computation unit. The allocator can select from which unit and its corresponding serial-in-serial-out register to receive data for processing.

[0077] In this embodiment of the invention, the post-processing module further includes an early termination module;

[0078] The early termination module uses an early termination algorithm to determine whether to terminate the network model calculation prematurely for spiking neural networks and Spiking-Transformer models.

[0079] In this embodiment of the invention, the early termination algorithm determines whether to terminate the calculation of subsequent time steps by comparing the pulse firing rate of the current time step with a preset threshold, thereby achieving sparse output in the time dimension.

[0080] Based on any of the above embodiments, the output module is further configured to:

[0081] For spiking neural networks, update the membrane level data at each time step, and calculate the address at which the calculation result is output for each set of data based on the number of times the input data is read.

[0082] Based on any of the above embodiments, the control data storage, distribution, and output address calculation processes of the multi-model fusion neural network accelerator supporting out-of-order sparse output, as well as the generation of other necessary control signals, are handled. In output address calculation, CNNs and SNNs can directly calculate based on the order of the input data. Transformers and Spiking-Transformers need to determine whether transposition is required based on the properties of the matrix being calculated. If transposition is required, the horizontal and vertical coordinates in space can be directly swapped, while other dimensions remain unchanged.

[0083] The operation of this multi-model fusion neural network accelerator, which supports out-of-order sparse output, is divided into five stages: data acquisition, allocation, computation, post-processing, and output. Before computation, the data to be processed is written into each data buffer using the input frame, and the key parameters required for the accelerator's operation are configured. After computation begins:

[0084] ① First, retrieve the required data from the cache. For models such as CNN and Transformer, multi-bit data will directly enter the next step. For models such as SNN and Spiking-Transformer, the accelerator can calculate according to the rule of time serialization (only one time step, i.e., 1 bit of pulse data, is retrieved, and other time steps are not processed. The time step value in the global register is determined, which is more in line with the biological nature of spiking neural networks) or according to the rule of time parallelization (the retrieved 8 bits of data are sequentially retrieved as one time step, i.e., 1 bit of pulse data, and sent to the subsequent modules until enough time steps are retrieved for calculation. There is no need to wait for the current time step to be calculated before retrieving the next time step, which reduces the repeated reading of weights).

[0085] ② In the preprocessing module, preprocessing is performed according to different model requirements, and the allocator determines which idle computing unit to send the data to through an arbitration algorithm.

[0086] ③ After receiving the data, the calculation unit combines the prediction algorithm to complete the multiplication and accumulation operation, and the result is sent to the serial-in-serial-out register.

[0087] ④ Depending on the type of model processed by this accelerator, the activation module and early termination module will be activated as needed, reading the calculation results from the 16 serial-in and serial-out registers and performing corresponding processing.

[0088] ⑤ The output is sent outside the accelerator. If it is a spiking neural network, the membrane level data of the time step will also be updated. At the same time, based on the number of times the input data is read, the control logic will calculate the address at which the calculation result of each set of data is output, and connect it to the corresponding data frame at the output port for processing by other accelerators or modules.

[0089] After processing the data stream as described above, the multi-model fusion neural network accelerator with out-of-order sparse output provided by the embodiments of the present invention unifies the computational paradigm of different models and can support mainstream models such as CNN, SNN, Transformer, and Spiking-Transformer. It can support both spatial and temporal computation and sparse data output in different dimensions.

[0090] like Figure 4As shown, after each calculation, the result is sent to the subsequent processing module, at which point the computing unit is idle. To reduce idle time, after each computing unit completes a round of tasks and outputs the multiplication-accumulation result, the control logic checks whether all data written to the accelerator cache SRAM has been calculated. If so, no more data is read; otherwise, the control logic reads a set of data from the cache again, preprocesses it, and sends it to the idle computing unit. The reading process is sequential, while the calculation process is out of order. Not all computing units need to be idle; as long as there are available hardware resources, data can be read and calculated. This decouples the calculation processes between different computing units, compressing the idle time of the computing units to a minimum cache read time.

[0091] Based on any of the above embodiments, the specific structure of the multi-model fusion neural network accelerator supporting out-of-order sparse output includes: (1) Data cache: including weight SRAM, input data SRAM, computational parameter SRAM, and important parameter global register. The number of synapses and models supported by each neuron can be configured. SRAM supports different data storage types at the same time, realizing the reuse of CNN data space and improving storage efficiency. (2) Preprocessing module: detects and removes input data or SNN time steps that are 0, and generates data that may be needed in the output sparse prediction process. (3) Computation unit: the most important computation module, which completes multiplication and accumulation operations together with the prediction unit. (4) Prediction unit: uses prediction algorithms to predict whether the result of the received data is positive, negative, or 0, supports output prediction function of up to 256 channels, and realizes spatial sparsity. (5) Serial-in-serial-out register: saves the calculation results of all computation units to prevent loss, and passes them through the subsequent processing in sequence. (6) Activation module: for models such as CNN and Transformer, a lookup table is used to support nonlinear function operators. (7) Early termination module: For models such as SNN and Spiking-Transformer, an early termination algorithm can be used to determine whether to terminate the model calculation in advance, so as to achieve sparse output in the time dimension. (8) Output module: Using an asynchronous handshake protocol, the data and its output coordinates are sent out of the accelerator in sequence. (9) Control logic: Controls the data storage, distribution, output address calculation and other processes, as well as generating other necessary control signals. The working process is divided into five stages: data acquisition, distribution, calculation, post-processing and output. The data acquisition stage supports the reading of multi-bit data, serial time step pulses and parallel time step pulses. The distributor determines the corresponding idle computing unit through an arbitration algorithm. After receiving the data, the computing unit combines the prediction algorithm to complete the multiplication and accumulation operation, and the result is sent to the serial-in-serial-out register. In the post-processing stage, the activation module and the early termination module are started according to the type of model processed by the accelerator this time, read the calculation results from 16 serial-in-serial-out registers and perform corresponding processing. Then it is output to the outside of the accelerator. If it is a spiking neural network, the membrane level data of the time step will also be updated. This data flow and architecture design unifies the computing paradigms of different models, providing support for all mainstream models and achieving high computing power and energy efficiency benefits.

[0092] Table 1 lists a comparison of test results for similar neural network accelerator hardware solutions. As can be seen from Table 1, the technology proposed in this invention achieves high computing power and energy efficiency while maintaining high model accuracy.

[0093] Table 1 Comparison of different design schemes

[0094]

[0095] This invention is applicable to both ASICs and FPGAs. It has been implemented in ASICs using tools from Synopsys such as DesignCompiler and PrimeTime, and has passed comprehensive testing. The neuron scale reaches 35 K, the synapse scale reaches 4.5 M, and the peak energy efficiency (90% output sparsity) at 500 MHz is 61.19 TOPs / W. This ensures the feasibility and high performance of the algorithm and hardware design.

[0096] Relatively few existing neural network accelerators simultaneously support multiple computational modes such as CNN, SNN, Transformer, and Spiking-Transformer. The dynamic nature of workloads within accelerators leads to varying computational speeds across computing units, typically limiting hardware resource utilization. Using output sparsity techniques further complicates data processing within computing units, making the progress even more unpredictable. If synchronous computation modes are still used, it will result in long waiting times for computing units, leading to wasted hardware resources. Existing accelerators generally use software or hardware methods to prioritize computational tasks and then schedule them to achieve a more balanced computational process. While these approaches reduce potential waiting times, they are still insufficient for optimizing accelerator hardware resource utilization and may introduce other hardware overhead.

[0097] The multi-model fusion neural network accelerator supporting out-of-order sparse output provided in this invention model and extracts commonalities from the data processing and computation processes of different models, designing an accelerator capable of supporting computational operations for multiple models. It unifies the data access methods and computational paradigms of different models, using the same computing circuitry and different data scheduling commands to complete the computational processes required by different models. It can support hybrid intelligence of multiple models such as CNN / SNN / Transformer / Spiking-Transformer, and achieves high computing power and energy efficiency benefits. The invention also studies the process of distributing data to computing units, proposing a neural network accelerated data stream supporting out-of-order sparse output. It uses inference algorithms to predict the sparsity of output data and distributes new workload data to idle resources, successfully compressing useless waiting time during computation and maximizing the utilization of hardware computing resources. In dataflow design, this invention also proposes a data organization method that can reduce the number of data reads when the convolution window slides. The input data is divided into 9 categories according to the relative position. Each time the input data is allocated to the computing unit, only 1 / 3 needs to be updated. The weight data originally stored in the unit only needs to be shifted, which greatly reduces the high-consumption interaction process between the computing module and the storage module.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-model fusion neural network accelerator supporting out-of-order sparse output, characterized in that, include: The data caching module is used to store weight data, input data and calculation parameters. The data caching module determines which idle computing unit to send the stored data to through a dynamic allocator. Multiple computing units are used to obtain weight data, input data and computing parameters from the data caching module. Each computing unit is configured to perform a multiply-accumulate operation based on the weight data, input data and computing parameters. Multiple prediction modules are connected to a corresponding computing unit. Based on the result of the multiply-accumulate operation, the prediction module determines whether to terminate the calculation of the subsequent time step. Under the action of the prediction module, the output time of the multiply-accumulate operation result of each computing unit is different. The serial-in-serial-out register is used to receive and buffer the results of multiplication and accumulation operations from various computational units; The post-processing module is used to extract the multiplication and accumulation operation results of each computing unit from the serial-in-serial-out register according to the type of the computed neural network, and process the multiplication and accumulation operation results of each computing unit. The output module is used to output the processing results of the post-processing module based on the asynchronous handshake protocol.

2. The multi-model fusion neural network accelerator supporting out-of-order sparse output according to claim 1, characterized in that, It also includes a preprocessing module for detecting the data in the data caching module and removing zero values ​​or invalid time step data from the input data.

3. The multi-model fusion neural network accelerator supporting out-of-order sparse output according to claim 2, characterized in that, The dynamic allocator uses a round-robin algorithm to distribute preprocessed data to idle computing units in order to balance the computing load.

4. The multi-model fusion neural network accelerator supporting out-of-order sparse output according to claim 1, characterized in that, The data caching module is also used for: For convolutional neural networks, the data width is 8 bits; For spiking neural networks with a data width of 1 bit, supporting neural network computation of no more than 8 time steps, the data of these 8 time steps are stored sequentially in 8-bit data according to the time step from first to last, and reused with the data space of the convolutional neural network.

5. The multi-model fusion neural network accelerator supporting out-of-order sparse output according to claim 1, characterized in that, The data caching module supports a multi-mode storage structure, switching between convolution sliding window block storage and matrix multiplication row storage via control signals.

6. The multi-model fusion neural network accelerator supporting out-of-order sparse output according to claim 5, characterized in that, The data caching module is also used for: For convolutional neural networks, the input feature map is split into blocks by the convolutional sliding window, and the data points in these blocks from top left to bottom right are stored together according to their corresponding positions and stored in different storage areas. A single row in the storage area can store data from multiple channels at the same location. During calculation, the position after sliding is compared with the position before sliding to update the changed area. For network models that use matrix multiplication, we use row-wise storage for matrix multiplication and read data row by row to ensure a one-to-one correspondence between input data and stored data.

7. The multi-model fusion neural network accelerator supporting out-of-order sparse output according to claim 1, characterized in that, The plurality of computing units are also used for: For convolutional neural networks or Transformer models, multi-bit data will be directly fed into the corresponding computation unit; For spiking neural networks or Spiking-Transformer models, computation can be performed either sequentially or in parallel over time. The rule for time serial processing includes retrieving pulse data from only one time step, while ignoring other time steps; the rule for time parallel processing includes loading pulse data from multiple time steps, retrieving pulse data from one time step at a time step in sequence, and sending it to the computing unit until enough time steps for the required calculation are retrieved.

8. The multi-model fusion neural network accelerator supporting out-of-order sparse output according to claim 1, characterized in that, The post-processing module includes an activation module; The activation module uses a lookup table to perform non-linear mapping for convolutional neural networks and Transformer models in order to process the data output by the computing unit.

9. The multi-model fusion neural network accelerator supporting out-of-order sparse output according to claim 1, characterized in that, The post-processing module includes an early termination module; The early termination module uses an early termination algorithm to determine whether to terminate the network model calculation prematurely for spiking neural networks and Spiking-Transformer models.

10. The multi-model fusion neural network accelerator supporting out-of-order sparse output according to claim 1, characterized in that, The output module is also used for: For spiking neural networks, update the membrane level data at each time step, and calculate the address at which the calculation result is output for each set of data based on the number of times the input data is read.

Images