Neural network accelerator and acceleration method

Abstract

The invention provides a neural network accelerator and an acceleration method. The neural network accelerator comprises an on-chip cache module and a calculation module, the on-chip cache module is used for storing sparse weight parameters of the input feature map, the output feature map and the compression format; the calculation module comprises N processing unit groups, each processing unit group is used for decompressing the sparse weight parameter group and screening out sparse weight parameters with non-zero weights, and the processing unit groups acquire input data corresponding to the sparse weight parameters in the sparse weight parameter group and send the input data to the processing unit groups; and carrying out multiply-accumulate calculation on all sparse weight parameters in the sparse weight parameter group and the input data.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a neural network accelerator and an acceleration method. Background Art

[0002] Neural network (NN) algorithms have been widely used in many fields such as computer vision and speech recognition. Neural network algorithms are characterized by large amount of computation and large amount of data. The computing circuits based on central processing unit (CPU) can no longer meet the performance and energy efficiency requirements of neural network algorithms. This is especially true in mobile devices with limited computing resources and battery life. Therefore, various accelerator circuits for neural networks have emerged, which optimize the multiply-accumulate (MUL) operator of neural networks, and optimize the architecture and data flow for different neural network model structures, so as to achieve the goal of high energy efficiency.

[0003] For neural network accelerators, in addition to the energy consumption caused by calculation, the movement of weight parameters between storage units and computing units also causes energy consumption overhead that cannot be ignored. With the increase in the cost of accessing off-chip memory and the continuous increase in model size, the memory access energy consumption overhead, especially the memory access energy consumption overhead for model weights, has become the energy consumption bottleneck of neural network accelerators, and the development of energy-efficient neural network accelerators faces major challenges. The existing technology prunes the model weights without losing the model accuracy, improves the model sparsity, and compresses the model on this basis to reduce the model scale and model calculation memory access overhead. This allows the accelerator to load sparse weights stored in a compressed format from the main memory outside the chip, and decompress them on the chip before calculation, thereby greatly reducing the energy consumption overhead of loading weight parameters from the main memory outside the chip. However, the sparse weight compression method has problems such as low compression ratio or slow decompression speed. The former limits the further reduction of weight loading overhead, and the latter does not utilize the large-scale parallel computing of the neural network accelerator.

[0004] In the prior art, for sparse models after pruning, sparse neural network accelerators usually skip the multiplication and accumulation calculations related to zero-valued weights to save energy and delay. However, in the prior art, the sparsity of different convolution kernels varies. For the processing elements (PE) of convolution kernel-level parallel processing, this means that different processing elements need to complete different numbers of non-zero multiplication and accumulation, that is, the workload is unbalanced, and the overall throughput of the accelerator will be limited by the processing element with the heaviest load. Summary of the invention

[0005] The purpose of the present invention is to provide a neural network accelerator and acceleration method, which can make the workload of all processing units more balanced, thereby improving the overall throughput of the accelerator. In addition, it can also reduce the area overhead occupied by sparse weight decompression.

[0006] In order to achieve the above object, the present invention provides a neural network acceleration method, comprising:

[0007] Dividing the sparse weight parameters into a plurality of sparse weight parameter groups so that the sparsity of the sparse weight parameters contained in each of the sparse weight parameter groups is balanced;

[0008] Compressing the sparse weight parameter group using a static key;

[0009] Distributing the plurality of compressed sparse weight parameter groups to a plurality of processing unit groups respectively, each of the processing unit groups dynamically calculating a key, and decompressing the sparse weight parameter groups to screen out the sparse weight parameters with non-zero weights;

[0010] The processing unit group obtains input data corresponding to the sparse weight parameters in the sparse weight parameter group, and performs multiplication and accumulation calculations on all the sparse weight parameters in the group and the input data.

[0011] Optionally, in the neural network acceleration method, after the processing unit group obtains the input data corresponding to the sparse weight parameters in the sparse weight parameter group and performs multiplication and accumulation calculations on all the sparse weight parameters in the group and the input data, it also includes:

[0012] Output the result of the multiplication and accumulation calculation.

[0013] Optionally, in the neural network acceleration method, the method of compressing the sparse weight parameter group using a static key includes:

[0014] Taking the vector of the sparse weight parameter group as the first vector;

[0015] Mapping into a second vector through a key matrix, wherein the length of the second vector is smaller than the length of the first vector;

[0016] A bitmap is used to record positions of non-zero values ​​in the first vector.

[0017] Optionally, in the neural network acceleration method, the method for decompressing the sparse weight parameters includes:

[0018] Using a key matrix, the second vector is restored into a third vector, wherein the length of the third vector is the same as the length of the first vector;

[0019] Recording different bits in the first vector and the third vector as error codes;

[0020] The erroneous encoding is corrected.

[0021] Optionally, in the neural network acceleration method, the method of dividing all the sparse weight parameters into a plurality of sparse weight parameter groups so that the sparsity of the sparse weight parameters contained in each of the sparse weight parameter groups is balanced includes:

[0022] Arrange the sparse weight parameters from small to large according to the size of sparsity;

[0023] A sparse weight parameter with a smaller sparsity is paired with a sparse weight parameter with a larger sparsity to form a sparse weight parameter group, thereby forming a plurality of sparse weight parameter groups, and the sparsity of the sparse weight parameters contained in the plurality of sparse weight parameter groups is balanced.

[0024] Accordingly, the present invention also provides a neural network accelerator applied to the neural network acceleration method, comprising:

[0025] On-chip cache module and computing module;

[0026] The on-chip cache module is used to store input feature maps, output feature maps and sparse weight parameters in compressed format;

[0027] The calculation module includes N processing unit groups, each of which is used to decompress the sparse weight parameter group and filter out the sparse weight parameters with non-zero weights. The processing unit group obtains the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and performs multiplication and accumulation calculations on all the sparse weight parameters in the sparse weight parameter group and the input data.

[0028] Optionally, the neural network accelerator further includes: a control module, wherein the control module is used to provide control signals to the on-chip cache module and the computing module.

[0029] Optionally, in the neural network accelerator, the on-chip cache module includes a feature map storage unit and N network weight storage units, the feature map storage unit is used to store input feature maps and output feature maps, and the N network weight storage units are all used to store sparse weight parameters in a compressed format.

[0030] Optionally, in the neural network accelerator, the computing module further includes:

[0031] Feature map data distribution unit and activation unit;

[0032] The feature map data distribution unit is used to distribute the input feature map to multiple processing unit groups respectively;

[0033] The activation unit is used to perform nonlinear activation operation on the output data after the multiplication and accumulation calculation, and rewrite it back to the on-chip cache module.

[0034] Optionally, in the neural network accelerator, the processing unit group includes:

[0035] A sparse weight decompression unit, an input feature map selection unit, and a multiply-accumulate array group;

[0036] The sparse weight decompression unit is used to decompress the sparse weight parameter group;

[0037] The input feature map selection unit is used to select the input data corresponding to the sparse weight parameter;

[0038] The multiplication-accumulation array group is used to perform multiplication-accumulation calculations on all sparse weight parameters in the sparse weight parameter group and the input data.

[0039] Optionally, in the neural network accelerator, the sparse weight decompression unit includes:

[0040] Priority encoder, XOR decoder and error correction unit;

[0041] The priority encoder encodes the bitmap data and outputs the positions of non-zero weights in the sparse weight parameters;

[0042] The XOR decoder is used to decompress the compressed format sparse weight parameters;

[0043] The error correction unit is used to flip the erroneously encoded bits in the decompressed sparse weight parameters to complete error correction.

[0044] Optionally, in the neural network accelerator, the XOR decoder includes:

[0045] Dynamic decoding key generation logic and XOR decoding array;

[0046] The dynamic decoding key construction logic generates a dynamic decoding key using the position data of non-zero weights in the sparse weight parameters generated by the priority encoder;

[0047] The XOR decoding array decompresses the compressed sparse weight parameters using the dynamic decoding key.

[0048] In the neural network accelerator and acceleration method provided by the present invention, all sparse weight parameters are divided into multiple sparse weight parameter groups, so that the sparsity of the sparse weight parameters contained in each sparse weight parameter group is balanced, which improves the balance of the workload of all processing units, thereby improving the overall throughput of the accelerator. In addition, the present invention uses a key to encrypt the sparse weight parameter group, and uses a dynamic decoding key to generate a logical calculation key in the accelerator. Compared with the prior art method of searching for a decoding key stored in a table, the storage area occupied by the decoding key is reduced, and the storage area overhead occupied by the sparse weight decompression decoding key is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a neural network acceleration method according to an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of an algorithm for compressing a sparse weight parameter group using a key according to an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of an algorithm for decompressing a sparse weight parameter group using a key according to an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of compressing and decompressing a sparse weight parameter group using a key according to an embodiment of the present invention;

[0053] Figure 5 is a schematic diagram of a load balancing method for pairing, grouping and compressing sparse weight parameters according to an embodiment of the present invention;

[0054] Figure 6 is a schematic diagram of a neural network accelerator according to an embodiment of the present invention;

[0055] Figure 7 is a schematic diagram of a priority encoder according to an embodiment of the present invention;

[0056] Figure 8 is a schematic diagram of an XOR decoder according to an embodiment of the present invention;

[0057] Fig. 9 Schematic diagram of the dynamic decoding key construction logic of an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0059] Hereinafter, the terms "first", "second", etc. are used to distinguish between similar elements and are not necessarily used to describe a particular order or chronological sequence. It is to be understood that these terms used in this manner are interchangeable where appropriate. Similarly, if the method described herein includes a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0060] Furthermore, it should be understood that when a layer (or film), region, pattern or structure is referred to as being "on" a substrate, layer (or film), region and / or pattern, it can be directly on another layer or substrate, and / or there can be intervening layers. In addition, it should be understood that when a layer is referred to as being "under" another layer, it can be directly under another layer, and / or there can be one or more intervening layers. In addition, references to being "on" and "under" each layer can be made based on the accompanying drawings.

[0061] Please refer to Figure 1 The present invention provides a neural network acceleration method, comprising:

[0062] Step S1: dividing the sparse weight parameters into a plurality of sparse weight parameter groups so that the sparsity of the sparse weight parameters contained in each sparse weight parameter group is balanced;

[0063] Step S2: compressing the sparse weight parameter group using a static key;

[0064] Step S3: distributing the multiple compressed sparse weight parameter groups to the multiple processing unit groups respectively, each processing unit group decompresses the sparse weight parameter group and screens out the sparse weight parameters with non-zero weights;

[0065] Step S4: the processing unit group obtains the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and performs multiplication and accumulation calculations on all the sparse weight parameters and the input data in the group.

[0066] Preferably, the processing unit group obtains input data corresponding to the sparse weight parameters in the sparse weight parameter group, and after performing multiplication and accumulation calculations on all the sparse weight parameters and the input data in the group, further comprises: outputting the results after the multiplication and accumulation calculations.

[0067] Preferably, the method for compressing the sparse weight parameter using a static key includes: using the vector of the sparse weight parameter group as the first vector; mapping it into a second vector through a key matrix, the length of the second vector is less than the length of the first vector; using a bitmap to record the position of non-zero values ​​in the first vector. The method for decompressing the sparse weight parameter group includes: using the key matrix to restore the second vector to a third vector, the length of the third vector is the same as the length of the first vector; recording the different bits in the first vector and the third vector as error codes; and correcting the error codes. Figure 2 , Figure 2 It is a schematic diagram of the algorithm for compressing sparse weight parameter groups using a key. The XORNet compression process first flattens the first vector Ws (n bits) of the input sparse weight data, converts the sparse weights from the original matrix format to a one-dimensional vector format, and then performs bit decomposition to decompose the multi-bit sparse weights of the first vector into n multi-bit vectors, that is, to obtain n vectors of length Nsparse. Then, the encoding compression is performed by decoding the key matrix, such as Figure 3 , Figure 3 It is a schematic diagram of the algorithm for decompressing the sparse weight parameter group using a key. The decoding obtains the second vector We of the compressed shorter (Nencoded) encoded weight data. The XORNet decompression process first decodes and decompresses the encoded weight data We by decoding the key matrix, and obtains the third vector Wd of the decoded weight data with the same length as the first vector Ws. Finally, through the shaping process, the weight data is restored to the original data format.

[0068] like Figure 4 , Figure 4 This is a specific implementation of the present invention of using a key to compress a sparse weight parameter group. In this example, Nsparse=8, Nencoded=3 is taken as an example, which represents a sparse weight compression ratio of 62.5%. The 8 input sparse weight data Ws are first flattened and bit-decomposed to obtain a vector of length Nsparse=8. Subsequently, XORNet compression is performed, and the compressed encoded weight data We is obtained by multiplying it with the decoding key matrix in the GF(2) domain. The XOR decompression process is to multiply the decoding key matrix and We in the GF(2) domain to obtain the decoded weight Wd. In this example, there is an error in the encoding of the decoded weight data, and the encoding error information needs to be recorded to achieve lossless compression in hardware.

[0069] Preferably, all sparse weight parameters are divided into multiple sparse weight parameter groups so that the sum of the values ​​of the sparse weight parameters contained in each sparse weight parameter group is balanced, including: arranging the sparse weight parameters from small to large according to the size of the sparsity; combining a smaller sparse weight parameter and a larger sparse weight parameter into a sparse weight parameter group, thereby forming multiple sparse weight parameter groups, and the sum of the values ​​of the sparse weight parameters contained in the multiple sparse weight parameter groups is balanced. Figure 5 , Figure 5 A specific embodiment for dividing all sparse weight parameters into multiple sparse weight parameter groups. In this example, 32 convolution kernels and 4 convolution kernels are paired as an example. For the sparse weight data of the 32 convolution kernels in the neural network layer, each convolution kernel is first sorted by sparsity to obtain the sparsity sequence relationship of the 32 convolution kernels. On the basis of the sparsity sorting, according to the sparsity sequence relationship, the two convolution kernels with the highest sparsity and the two convolution kernels with the lowest sparsity are paired, and the process is completed in sequence until 8 groups of pairings are completed. The paired convolution kernels are first cross-flattened, and the sparse weight data after cross-flattening can be subjected to the XORNet compression and storage process. In other embodiments of the present invention, it can be other numbers of convolution kernels or other numbers of pairings.

[0070] Accordingly, please refer to Figure 6 The present invention also provides a neural network accelerator applied to the neural network acceleration method, including: an on-chip cache module and a computing module; the on-chip cache module is used to store input feature maps, output feature maps and sparse weight parameters in compressed format; the computing module includes N processing unit groups, each processing unit group is used to decompress the sparse weight parameter group, filter out the sparse weight parameters with non-zero weights, the processing unit group obtains the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and performs multiplication and accumulation calculations on all sparse weight parameters and input data in the sparse weight parameter group. It also includes: a control module, which is used to provide control signals to the on-chip cache module and the computing module.

[0071] The on-chip cache module includes a feature map storage unit and N network weight storage units. The feature map storage unit is used to store input feature maps and output feature maps. The N network weight storage units are used to store sparse weight parameters in a compressed format. The sparse weight parameters in a compressed format include encoded weights, bitmaps, and encoding error information.

[0072] The computing module also includes: a feature map data distribution unit and an activation unit; the feature map data distribution unit is used to distribute the input feature map to multiple processing unit groups respectively; the activation unit is used to perform nonlinear activation operations on the output data after the multiplication and accumulation calculations, and rewrite them back to the on-chip cache module.

[0073] The processing unit group includes: a sparse weight decompression unit, an input feature map selection unit and a multiplication-accumulation array group (MAC array); the sparse weight decompression unit is used to decompress the sparse weight parameter group; the input feature map selection unit is used to select the input data corresponding to the sparse weight parameters; the multiplication-accumulation array group is used to perform multiplication-accumulation calculations on all sparse weight parameters and input data in the sparse weight parameter group.

[0074] The sparse weight decompression unit includes: a priority encoder, an XOR decoder and an error correction unit; the priority encoder encodes the bitmap data and outputs the position of the non-zero weight in the sparse weight parameter; the XOR decoder is used to decompress the sparse weight parameter group in a compressed format; the error correction unit is used to flip the wrongly encoded bits in the decompressed sparse weight parameter to complete the error correction. The XOR decoder includes: dynamic decoding key generation logic and an XOR decoding array; the dynamic decoding key construction logic generates a dynamic decoding key using the position data of the non-zero weight in the sparse weight parameter generated by the priority encoder; the XOR decoding array decompresses the sparse weight parameter in a compressed format using the dynamic decoding key.

[0075] like Figure 7 , Figure 7 This is a specific embodiment of the priority encoder of the present invention. Taking a 32-bit bitmap as an example, four bitmap non-zero coding positions are generated in each cycle, and in this example, the four non-zero coding positions are P1=0, P2=4, P3=8, and P4=11.

[0076] like Figure 8 , Figure 8 This is a specific embodiment of the XOR decoder of the present invention. In this example, Nsparse=32, Nencoded=4 is used as an example, and the decoding process can be completed by the XOR array in the XOR decoder. Fig. 9 , Fig. 9 This is an embodiment of the dynamic decoding key construction logic. The decoding key matrix is ​​generated by the dynamic decoding key construction logic, wherein the dynamic decoding key construction logic receives the 5-bit non-zero encoding position P generated by the priority decoder, and then generates KP data, i.e., the decoding key matrix, for the decoding process according to the corresponding logic.

[0077] In summary, in the neural network accelerator and acceleration method provided by the embodiments of the present invention, all sparse weight parameters are divided into multiple sparse weight parameter groups, so that the sparsity of the sparse weight parameters contained in each sparse weight parameter group is balanced, which improves the balance of the workload of all processing units, thereby improving the overall throughput of the accelerator. In addition, the present invention uses a key to encrypt the sparse weight parameter group, and uses a dynamic decoding key to generate a logical calculation key in the accelerator. Compared with the prior art method of searching for a decoding key stored in a table, the storage area occupied by the decoding key is reduced, and the storage area overhead occupied by the sparse weight decompression decoding key is reduced.

[0078] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A neural network acceleration method, characterized in that: include: Dividing the sparse weight parameters into a plurality of sparse weight parameter groups so that the sparsity of the sparse weight parameters contained in each of the sparse weight parameter groups is balanced; Compressing the sparse weight parameter group using a static key; Distributing the plurality of compressed sparse weight parameter groups to a plurality of processing unit groups respectively, each of the processing unit groups dynamically calculating a key, and decompressing the sparse weight parameter groups to screen out the sparse weight parameters with non-zero weights; The processing unit group obtains input data corresponding to the sparse weight parameters in the sparse weight parameter group, and performs multiplication and accumulation calculations on all the sparse weight parameters in the group and the input data.

2. The neural network acceleration method according to claim 1, characterized in that: The processing unit group obtains the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and after performing multiplication and accumulation calculations on all the sparse weight parameters in the group and the input data, further includes: Output the result of the multiplication and accumulation calculation.

3. The neural network acceleration method according to claim 1, characterized in that: Methods for compressing sparse weight parameter groups using static keys include: Taking the vector of the sparse weight parameter group as the first vector; Mapping into a second vector through a key matrix, wherein the length of the second vector is smaller than the length of the first vector; A bitmap is used to record positions of non-zero values ​​in the first vector.

4. The neural network acceleration method according to claim 3, characterized in that: The method for decompressing the sparse weight parameter group includes: Using a static key matrix, the second vector is restored into a third vector, wherein the length of the third vector is the same as the length of the first vector; Recording different bits in the first vector and the third vector as error codes; The erroneous encoding is corrected.

5. The neural network acceleration method according to claim 1, characterized in that: The method of dividing all the sparse weight parameters into a plurality of sparse weight parameter groups so that the sparsity of the sparse weight parameters contained in each of the sparse weight parameter groups is balanced comprises: Arrange the sparse weight parameters from small to large according to the size of sparsity; A sparse weight parameter with a smaller sparsity is paired with a sparse weight parameter with a larger sparsity to form a sparse weight parameter group, thereby forming a plurality of sparse weight parameter groups, and the sparsity of the sparse weight parameters contained in the plurality of sparse weight parameter groups is balanced.

6. A neural network accelerator applied to the neural network acceleration method according to any one of claims 1 to 4, characterized in that: include: On-chip cache module and computing module; The on-chip cache module is used to store input feature maps, output feature maps and sparse weight parameters in compressed format; The calculation module includes N processing unit groups, each of which is used to decompress the sparse weight parameter group and filter out the sparse weight parameters with non-zero weights. The processing unit group obtains the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and performs multiplication and accumulation calculations on all the sparse weight parameters in the sparse weight parameter group and the input data.

7. The neural network accelerator according to claim 6, characterized in that: Also includes: A control module is used to provide control signals to the on-chip cache module and the computing module.

8. The neural network accelerator according to claim 6, characterized in that: The on-chip cache module includes a feature map storage unit and N network weight storage units. The feature map storage unit is used to store input feature maps and output feature maps, and the N network weight storage units are used to store sparse weight parameters in a compressed format.

9. The neural network accelerator according to claim 6, characterized in that: The calculation module also includes: Feature map data distribution unit and activation unit; The feature map data distribution unit is used to distribute the input feature map to multiple processing unit groups respectively; The activation unit is used to perform nonlinear activation operation on the output data after the multiplication and accumulation calculation, and rewrite it back to the on-chip cache module.

10. The neural network accelerator according to claim 6, characterized in that: The processing unit group includes: A sparse weight decompression unit, an input feature map selection unit, and a multiply-accumulate array group; The sparse weight decompression unit is used to decompress the sparse weight parameter group; The input feature map selection unit is used to select the input data corresponding to the sparse weight parameter; The multiplication-accumulation array group is used to perform multiplication-accumulation calculations on all sparse weight parameters in the sparse weight parameter group and the input data.

11. The neural network accelerator according to claim 10, characterized in that: The sparse weight decompression unit comprises: Priority encoder, XOR decoder and error correction unit; The priority encoder encodes the bitmap data and outputs the positions of non-zero weights in the sparse weight parameters; The XOR decoder is used to decompress the sparse weight parameters in the compressed format; The error correction unit is used to flip the erroneously encoded bits in the decompressed sparse weight parameters to complete error correction.

12. The neural network accelerator according to claim 11, characterized in that: The XOR decoder comprises: Dynamic decoding key generation logic and XOR decoding array; The dynamic decoding key construction logic generates a dynamic decoding key using the position data of non-zero weights in the sparse weight parameters generated by the priority encoder; The XOR decoding array decompresses the compressed sparse weight parameters using the dynamic decoding key.

Images