A bit-level computing model architecture system for sparse neural network acceleration

Abstract

The application discloses a bit-level calculation model architecture system for sparse neural network acceleration, and the number of 1 in the weight bit is pruned to not exceed a specific value through training, so that the number of weight bit calculation is effectively reduced, and more information is retained compared with directly pruning the weight bit width to a low bit. In cooperation with the skip '0' bit operation of the hardware architecture, the hardware acceleration of the neural network is realized. Therefore, at least partially, the problem that the number of weight bit calculation is directly reduced in the prior art, thereby causing the problem of less pruned bit width or greater precision loss is overcome.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sparse neural network computing, and particularly relates to a bit-level computing model architecture system for sparse neural network acceleration. BACKGROUND

[0002] With the rapid development of artificial intelligence technology based on deep learning, the system design of high energy efficiency deep learning is particularly important, and the requirement for computing efficiency is also higher and higher. In order to improve the performance of neural networks, researchers reduce the computing demand of neural networks through model quantization.

[0003] The weights of neural networks can be quantized from high-precision floating-point numbers to low-precision fixed-point numbers, and the bit width of weight quantization is different for different networks. In order to improve the performance by weight quantization, a large number of researchers are committed to bit-level computing hardware acceleration. At present, the main method for bit-level computing acceleration is to directly prune the weights to low bits to achieve the effect of acceleration, however, this method will cause a large amount of information loss, resulting in less pruning bit width or greater precision loss. SUMMARY

[0004] The purpose of the present application is to provide a bit-level computing model architecture system for sparse neural network acceleration, which can reduce the number of bit computations while maintaining a certain amount of information, greatly improving the performance.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A bit-level computing model architecture system for sparse neural network acceleration, comprising: a controller, a computing array, an input data cache, a post-processing unit, an output data cache, a compression module and an output feature map clustering module; wherein:

[0007] The controller is configured to control other parts of the model architecture;

[0008] The input data cache is configured to read input pictures from an off-chip cache or read input feature maps from the off-chip cache in combination with clustering results in the off-chip cache, and weight data corresponding to the computing array, and cache for use by the computing array; wherein the weight data is pre-pruned to prune the number of bits equal to 1 in each weight to not more than a set number W;

[0009] The computing array is configured to read input pictures or input feature maps and weight data from the input data cache, and perform convolution operations of neural networks;

[0010] The post-processing unit is configured to post-process the convolution operation results output by the computing array to obtain output feature maps.

[0011] The output data cache is used to cache the output feature map;

[0012] The compression module is used to convert the output feature map into a compressed format and store it in the off-chip cache;

[0013] The output feature map clustering module is used to cluster all output feature maps and store the clustering results in an off-chip cache.

[0014] As can be seen from the technical solution provided by this invention, by pruning one number in the weight bits to no more than a specific value during training, the number of weight bit calculations is effectively reduced. At the same time, compared to directly pruning the weight bit width to a lower bit, more information is preserved. Combined with the model architecture's jump '0' bit operation, hardware acceleration of the neural network is achieved. Therefore, this at least partially overcomes the problem in existing technologies where directly reducing the number of weight bit calculations results in a smaller pruned bit width or greater accuracy loss. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram of a model architecture system for accelerating bit-level computation via sparse neural networks, provided as an embodiment of the present invention;

[0017] Figure 2 A flowchart of a model pruning method provided in an embodiment of the present invention;

[0018] Figure 3 This is a flowchart of weight pruning, compression, and subsequent calculations provided in an embodiment of the present invention.

[0019] Figure 4 A schematic diagram of a computing unit provided in an embodiment of the present invention;

[0020] Figure 5 This is a clustering flowchart provided for an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0022] First, the following explanations are provided for the terms that may be used in this article:

[0023] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0024] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0025] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0026] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0027] The following is a detailed description of a model architecture system for accelerating bit-level computation in sparse neural networks provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0028] This invention provides a model architecture system for accelerating bit-level computation in sparse neural networks, such as... Figure 1 As shown, it includes: a TOP Controller, a computing array, an input data buffer, a post-processing unit, an output data buffer, a compression module, and an output feature map clustering module.

[0029] The controller is used to control other parts of the model architecture, thereby completing the entire sparse matrix calculation process.

[0030] The input data cache is used to read input images from the off-chip cache or to read input feature maps from the off-chip cache in combination with clustering results in the off-chip cache, as well as to calculate the weight data corresponding to the calculation array, and cache them for use by the calculation array; wherein, the weight data has been pruned in advance, and the number of bits with 1 in each weight has been pruned to no more than a set number W.

[0031] The computing array is used to read input images or input feature maps and weight data from the input data cache, and to perform convolution operations on the neural network.

[0032] The post-processing unit is used to perform post-processing (activation function, pooling, etc.) on the convolution operation results output by the computing array to obtain the output feature map.

[0033] The output data cache is used to cache the output feature map.

[0034] The compression module is used to convert the output feature map into a compressed format and store it in the off-chip cache.

[0035] The output feature map clustering module is used to cluster all output feature maps and store the clustering results in an off-chip cache.

[0036] To facilitate understanding, the following section will introduce the various parts of the system and the pruning methods involved.

[0037] I. Model pruning method.

[0038] In this embodiment of the invention, a model pruning method is provided to prune the number of bits with a value of 1 in each weight to no more than a set number W. The main process is as follows: Figure 2 As shown, it includes:

[0039] Step 1: Set the pruning target, i.e., the number W.

[0040] Step 2: Count the number of bits that are 1 in each weight. If the number exceeds W, prune the excess bits in ascending order of bit position, then retrain and check the accuracy. Example: Assume the pruning target is 4 (i.e., W = 4). Given the weight '10101111', which has 6 bits that are '1', we need to prune the excess bits by removing bits 0 and 1 in ascending order of bit position.

[0041] Step 3: If the accuracy loss range is exceeded, stop pruning and save the current weights and the number of 1-bit weights N.

[0042] Step 4: If the accuracy loss range is not exceeded, return to step 2.

[0043] Let the initial weight's bit width (number of bits) be M, then its initial information content is 2^M. M , that is, 2 M The number of '1' bits after pruning is W, meaning no more than W '1' bits in M ​​positions, therefore its information content is... Compared to directly pruning the weight bit width to W, its information content is increased. times, of which, This represents the number of combinations of selecting i numbers from M. Therefore, this invention can partially overcome the problem in the prior art where directly reducing the number of weight bits for calculation results in a smaller pruning bit width or a greater loss of accuracy.

[0044] like Figure 3 The diagram illustrates a flowchart of weight pruning, compression, and calculation. For example, the initial weights are represented in binary as W0 (10101011) and W1 (00101001). After pruning, they become W0 (10100000) and W1 (00101000). Subsequently, the weights are compressed into the position information of the bits with a value of '1', which are the sequences {7, 5} and {5, 3}, respectively. These are then fed into the PE for calculation, thus skipping the calculation of the bits with a value of '0' in the weights and shortening the calculation time.

[0045] II. Description of each part of the system.

[0046] 1. Computational array.

[0047] like Figure 1 As shown, the smallest unit of the computing array is a computing element (PE). It is assumed that the computing array contains M rows and N columns of computing elements, where M and N are both integers greater than 1.

[0048] In this embodiment of the invention, each row of computing units can share the same input image or input feature map; each column of computing units calculates the output feature map of the same channel. Specifically: the input of the first column of computing units is the input image or input feature map, and weight data; the convolution result of the current column of computing units serves as the input of the computing unit in the previous row of the same column; the convolution result of the N computing units in the first row is the convolution result output by the computing array.

[0049] Figure 1 In the diagram, the arrows IFM&Weight, filled with diagonal lines, encompass the input image, input feature map, and weight data.

[0050] In this embodiment of the invention, the computation unit includes: a shift-add unit, which calculates the convolution result through shift and / or addition operations (bit-level multiplication operations); an address calculation subunit, used to calculate the address of the partial sum of the output feature map (an intermediate result during convolution calculation) based on the convolution result; and a partial sum caching subunit, used to cache the address of the partial sum of the output feature map and accumulate the partial sums based on the partial sums of adjacent computation units, outputting the complete partial sum as the operation result of the computation unit, such as... Figure 1 As shown, the arrow Psum (Partial sum) filled with dots represents the partial sum. The sum of the partial sums of each column's calculation unit is accumulated and propagated from the bottommost calculation unit to the topmost unit. During the propagation process, the sums are continuously accumulated, and finally the final result of each column is obtained.

[0051] like Figure 4 The diagram shows the main structure of the computing unit. The computing unit includes four input ports, which respectively input: non-zero input feature map I. nz Weight W nz_bit Input feature map and row and column addresses of weights (I row I col W row W col ), parts of adjacent computing units and Psum in , (W row W col ) represents the row and column addresses of the input feature map, (I row I col ) represents the row and column addresses of the weights, where non-zero values ​​represent the input feature map I. nz and weight W nz_bit Input to shift-add unit ( Figure 3 The weights of the input feature map are input row-by-row to the address calculation subunit (AddrCmp), and the partial sums of adjacent calculation units are input to the partial sum buffer subunit (Psum Buffer).

[0052] In this embodiment of the invention, considering that there are zero or non-zero elements in the initial feature map, in order to shorten the calculation time, only the non-zero elements in the feature map are input into the calculation unit to participate in the calculation.

[0053] The internal computation of the computing unit is mainly divided into two stages: the local computation stage and the adjacent PE accumulation stage. During the local computation stage, the computing unit receives a non-zero input feature map I. nz With weight W nz_bit Bit-level multiplication is performed in the shift-add unit, and at the same time, according to I row I col W row and W col The partial sum address of the output feature map is calculated and stored in the partial sum cache. During the adjacent computation unit accumulation phase, the current computation unit performs partial sum accumulation with its adjacent computation units, ultimately obtaining the complete partial sum through the output port Psum. out Output.

[0054] 2. Input data caching.

[0055] like Figure 1 As shown, the number of input data buffers is the same as the number of rows M of computing units in the computing array, and each input data buffer is individually connected to the first computing unit in each row of the computing array.

[0056] 3. Post-processing unit.

[0057] like Figure 1 As shown, the number of post-processing units is the same as the number of columns N of computing units in the computing array. The N computing units in the first row are each connected to a post-processing unit. The post-processing unit calculates the activation function and pooling operation on the convolution operation results of the corresponding computing unit to obtain the output feature map of a single channel.

[0058] 4. Output data caching.

[0059] like Figure 1 As shown, the number of output data buffers is the same as the number of post-processing units, and the output feature maps of a single channel of the number of post-processing units are buffered.

[0060] 5. Compression module.

[0061] like Figure 1 As shown, the number of compression modules is the same as the number of output data buffers, and they are used to compress the output feature maps in the output data buffers.

[0062] 6. Output feature map clustering module.

[0063] The output feature map clustering module is primarily responsible for grouping feature maps with similar sparsity together. For example, a single output feature map may have 64 elements, including 40 zero elements and 24 non-zero elements. The module counts the number of non-zero elements in each output feature map and then groups feature maps with similar sparsity (number of non-zero elements) together. The output feature map clustering module includes: a data length cache subunit for recording the number of non-zero elements in all output feature maps in the output data cache; a sorting subunit for sorting channel numbers based on the number of non-zero elements in the output feature maps; a channel number cache subunit for caching the channel numbers of the clustered output feature maps (i.e., the clustering results); and a selector subunit for connecting the data channels of the output data cache and the off-chip cache, such as... Figure 1 As shown, the clustering results will be stored in off-chip cache (DRAM) via DDR AXI (DDR interface based on bus protocol). The clustering process is as follows: Figure 5 As shown, assuming there are four input feature maps I0 to I3, they need to be imported into a 1x2 computational unit array in two separate steps. First, record the number of non-zero elements in each feature map, which are 8, 4, 7, and 3 respectively. Then, sort them by size. Finally, cluster the feature maps by size and import them into the computational array in the order of I0 and I2, and I1 and I3 respectively.

[0064] III. System Workflow.

[0065] In this embodiment of the invention, the system workflow is controlled by a controller, and the main processes include: importing weights and initial input images from an external source into an off-chip cache (DRAM). The network processed by this invention contains multiple layers. The first layer processes the input image, and the output result is stored in the off-chip cache. The output of the previous layer serves as the input for the next layer. The input feature map for the next layer is read from the aforementioned off-chip cache. Then, the input cache unit reads a portion of the weight data (partial weight data of the current layer). Subsequently, the weight data is decompressed and transmitted to the computation array along with the input feature map. The computation array mainly performs the multiplication and accumulation operations in the convolution calculation. The post-processing unit is responsible for performing activation, pooling, and other operations on the multiplication and accumulation results to obtain the output feature map. The output cache is responsible for storing the output feature map. Subsequently, the output feature map is sent to the compression module to be converted into a compressed format (which can reduce the storage amount of the off-chip cache and reduce the computation time). The output feature map clustering module sorts the output feature maps according to the number of non-zero elements in each output feature map, and clusters feature maps with similar sparsity together to balance the data load of the next layer. Finally, the output of the compression module is stored in the off-chip cache and loaded into the input cache unit as the input of the next layer. This process is repeated until the last layer. The clustering results are cached in the channel number cache sub-unit and then stored in the off-chip cache through DDR AXI. The next layer reads the feature map in the off-chip cache according to the channel number in the clustering results.

[0066] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A model architecture system for bit-level computation oriented to sparse neural network acceleration, characterized in that, The application relates to a neural network processing system, which comprises a controller, a calculation array, an input data cache, a post-processing unit, an output data cache, a compression module and an output feature map clustering module. The controller is used for controlling other parts of the model architecture. The input data cache is used for reading input pictures from an off-chip cache or reading input feature maps from the off-chip cache in combination with clustering results in the off-chip cache and weight data corresponding to the calculation array, and the input data cache is used for caching the data for the calculation array; wherein the weight data is pruned in advance, and the number of bits equal to 1 in each weight is pruned to not more than a set number W, and the steps comprise the following steps: step 1, setting the number W; step 2, counting the number of bits equal to 1 in each weight, and if the number exceeds the number W, the part exceeding the number is arranged in the order of bits from low to high, then retraining is carried out, and the accuracy is checked; step 3, if the accuracy loss range is exceeded, the pruning is stopped, and the current weight and the number W of bits equal to 1 are saved; step 4, if the accuracy loss range is not exceeded, step 2 is returned. The calculation array is used for reading input pictures or input feature maps and weight data from the input data cache, and performing convolution operation of the neural network. The post-processing unit is used for post-processing the convolution operation result output by the calculation array to obtain an output feature map. The output data cache is used for caching the output feature map. The compression module is used for converting the output feature map into a compressed format and storing the output feature map into the off-chip cache. The output feature map clustering module is used for clustering all output feature maps and storing the clustering results into the off-chip cache, and the output feature map clustering module comprises a data length cache subunit, an ordering subunit, a channel serial number cache subunit and a selector subunit. The calculation array comprises M rows and N columns of calculation units; the input of the calculation units in the first column is input pictures or input feature maps and weight data, and the convolution operation result of the current column of calculation units is used as the input of the calculation unit in the same column in the previous row; the convolution operation result of the N calculation units in the first row is the convolution operation result output by the calculation array; wherein M and N are both integers greater than 1.

2. The bit-level computing model architecture system oriented to sparse neural network acceleration according to claim 1, wherein, The calculation unit comprises a shift and addition unit, an address calculation subunit, a partial sum cache subunit and a selector subunit.

3. The bit-level computing model architecture system oriented to sparse neural network acceleration according to claim 2, wherein, The shift and addition unit is used for calculating the convolution result through shift and / or addition operation. The address calculation subunit is used for calculating the address of the partial sum of the output feature map in combination with the convolution result. The partial sum cache subunit is used for caching the address of the partial sum of the output feature map and combining the partial sum of the adjacent calculation unit to accumulate the partial sum, and outputting the complete partial sum as the operation result of the calculation unit. The calculation unit comprises four input ports, which respectively input the non-zero input feature map, the weight, the row and column address of the input feature map and the weight and the partial sum of the adjacent calculation unit.

4. The bit-level computing model architecture system oriented to sparse neural network acceleration according to claim 3, wherein, ​ The non-zero input feature map and the weight are input to the shift-add unit, and the input feature map and the weight are input to the address calculation subunit in a row-column manner, and the partial sum of adjacent calculation units is input to the partial sum cache subunit.

5. The bit-level computing model architecture system for sparse neural network acceleration according to claim 1 or 2, wherein, The number of the input data caches is the same as the number of rows M of the calculation units in the calculation array, and each input data cache is connected to the first calculation unit in each row of the calculation array.

6. The bit-level computing model architecture system for sparse neural network acceleration according to claim 1 or 2, wherein, The number of the post-processing units is the same as the number of columns N of the calculation units in the calculation array, and the first row of N calculation units is connected to one post-processing unit respectively, and the post-processing unit calculates the activation function calculation and the pooling operation on the convolution operation result of the corresponding calculation unit to obtain the output feature map of a single channel.

7. The bit-level computing model architecture system for sparse neural network acceleration of claim 1, wherein, The number of the output data caches is the same as the number of the post-processing units, and the output feature map of a single channel of the post-processing unit is cached.

Images