Neural network accelerators and acceleration methods

Abstract

This invention provides a neural network accelerator and acceleration method, comprising: an on-chip cache module and a computation module; the on-chip cache module is used to store input feature maps, output feature maps, and sparse weight parameters in compressed format; the computation module includes N processing unit groups, each processing unit group is used to decompress the sparse weight parameter group, filter out sparse weight parameters with non-zero weights, obtain the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and perform multiplication and accumulation calculations on all sparse weight parameters in the sparse weight parameter group and the input data.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a neural network accelerator and acceleration method. Background Technology

[0002] Neural Network (NN) algorithms have been widely applied in various fields such as computer vision and speech recognition. NN algorithms are characterized by high computational and data volume, and computing circuits primarily based on Central Processing Units (CPUs) can no longer meet the performance and energy efficiency requirements of NN algorithms. This is especially true in mobile devices with limited computing resources and battery life. Therefore, various accelerator circuits designed for neural networks have emerged, optimizing for the main operator of neural networks, Multiply Accumulate, and optimizing the architecture and data flow for different neural network model structures to achieve high energy efficiency.

[0003] For neural network accelerators, in addition to the energy consumption caused by computation, the movement of weight parameters between storage and computation units also incurs significant energy costs. With the increasing cost of accessing external memory and the continuous growth of model size, memory access energy costs, especially for model weights, have become a bottleneck for neural network accelerators, posing a major challenge to the development of energy-efficient neural network accelerators. Existing technologies prune model weights to increase model sparsity without sacrificing model accuracy, and then compress the model to reduce model size and computational memory access costs. This allows the accelerator to load sparse weights stored in compressed format from external main memory and decompress them on the pre-computation chip, thus greatly reducing the energy cost of loading weight parameters from external main memory. However, sparse weight compression methods suffer from low compression ratios or slow decompression speeds. The former limits further reductions in weight loading costs, while the latter hinders the large-scale parallel computation of neural network accelerators.

[0004] Existing technologies for sparse models after pruning typically skip multiply-accumulate calculations related to zero-value weights in sparse neural network accelerators to save energy and latency. However, in existing technologies, the sparsity of different convolutional kernels varies. For processing elements (PEs) that perform parallel processing at the convolutional kernel level, this means that different processing elements need to complete different numbers of non-zero multiply-accumulate operations, resulting in an unbalanced workload. Consequently, the overall throughput of the accelerator is limited by the most heavily loaded processing element. Summary of the Invention

[0005] The purpose of this invention is to provide a neural network accelerator and acceleration method that can more evenly distribute the workload of all processing units, thereby improving the overall throughput of the accelerator. Furthermore, it can reduce the area overhead occupied by sparse weight decompression.

[0006] To achieve the above objectives, the present invention provides a neural network acceleration method, comprising:

[0007] The sparse weight parameters are divided into multiple sparse weight parameter groups so that the sparsity of the sparse weight parameters contained in each sparse weight parameter group is balanced.

[0008] The sparse weight parameter set is compressed using a static key;

[0009] The multiple compressed sparse weight parameter groups are distributed to multiple processing unit groups. Each processing unit group calculates the key in a dynamic manner, decompresses the sparse weight parameter groups, and filters out the sparse weight parameters with non-zero weights.

[0010] The processing unit group obtains the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and performs multiplication and accumulation calculations on all sparse weight parameters in the group and the input data.

[0011] Optionally, in the neural network acceleration method, after the processing unit group obtains the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and performs multiplication and accumulation calculations on all sparse weight parameters in the group and the input data, the method further includes:

[0012] Output the result after multiplication and summation.

[0013] Optionally, in the neural network acceleration method described above, the method of compressing the sparse weight parameter set using a static key includes:

[0014] Use the vector of the sparse weight parameter group as the first vector;

[0015] The key matrix is ​​used to map the vector into a second vector, the length of which is less than the length of the first vector.

[0016] The positions of non-zero values ​​in the first vector are recorded using a bitmap.

[0017] Optionally, in the neural network acceleration method, the method for decompressing the sparse weight parameters includes:

[0018] The second vector is recovered into a third vector using a key matrix, the length of which is the same as the length of the first vector;

[0019] The different bits in the first and third vectors are recorded as error codes;

[0020] The erroneous codes are corrected.

[0021] Optionally, in the neural network acceleration method, the method of dividing all the sparse weight parameters into multiple sparse weight parameter groups to balance the sparsity of the sparse weight parameters contained in each sparse weight parameter group includes:

[0022] Arrange the sparse weight parameters in ascending order of sparsity;

[0023] Sparse weight parameters with smaller sparsity are paired with sparse weight parameters with larger sparsity to form a sparse weight parameter group, thereby forming multiple sparse weight parameter groups, and the sparsity of the sparse weight parameters contained in the multiple sparse weight parameter groups is balanced.

[0024] Accordingly, the present invention also provides a neural network accelerator applied to a neural network acceleration method, comprising:

[0025] On-chip cache module and computing module;

[0026] The on-chip cache module is used to store the input feature map, the output feature map, and the sparse weight parameters in the compressed format;

[0027] The calculation module includes N processing unit groups. Each processing unit group is used to decompress the sparse weight parameter group, filter out the sparse weight parameters with non-zero weights, obtain the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and perform multiplication and accumulation calculation on all sparse weight parameters in the sparse weight parameter group and the input data.

[0028] Optionally, the neural network accelerator further includes a control module, which provides control signals to the on-chip cache module and the computing module.

[0029] Optionally, in the neural network accelerator, the on-chip cache module includes a feature map storage unit and N network weight storage units. The feature map storage unit is used to store the input feature map and the output feature map, and the N network weight storage units are all used to store sparse weight parameters in compressed format.

[0030] Optionally, in the neural network accelerator, the computing module further includes:

[0031] Feature map data distribution unit and activation unit;

[0032] The feature map data distribution unit is used to distribute the input feature map to multiple processing unit groups respectively;

[0033] The activation unit is used to perform nonlinear activation operations on the output data after multiplication and accumulation calculations, and then write it back to the on-chip cache module.

[0034] Optionally, in the neural network accelerator, the processing unit group includes:

[0035] Sparse weight decompression unit, input feature map selection unit, and multiply-accumulate array group;

[0036] The sparse weight decompression unit is used to decompress the sparse weight parameter set;

[0037] The input feature map selection unit is used to select the input data corresponding to the sparse weight parameters;

[0038] The multiply-accumulate array group is used to perform multiply-accumulate calculations on all sparse weight parameters in the sparse weight parameter group and the input data.

[0039] Optionally, in the neural network accelerator, the sparse weight decompression unit includes:

[0040] Priority encoder, XOR decoder and error correction unit;

[0041] The priority encoder encodes the bitmap data and outputs the positions of non-zero weights in the sparse weight parameters.

[0042] The XOR decoder is used to decompress the sparse weight parameters of the compressed format;

[0043] The error correction unit is used to flip the erroneously encoded bits in the decompressed sparse weight parameters to complete the error correction.

[0044] Optionally, in the neural network accelerator, the XOR decoder includes:

[0045] Dynamic decoding key generation logic and XOR decoding array;

[0046] The dynamic decoding key construction logic uses the position data of non-zero weights in the sparse weight parameters generated by the priority encoder to generate a dynamic decoding key;

[0047] The XOR decoding array uses the dynamic decoding key to decompress the sparse weight parameters of the compressed format.

[0048] In the neural network accelerator and acceleration method provided by this invention, all sparse weight parameters are divided into multiple sparse weight parameter groups to ensure that the sparsity of the sparse weight parameters contained in each sparse weight parameter group is balanced, thereby improving the workload balance of all processing units and thus improving the overall throughput of the accelerator. Furthermore, this invention uses a key to encrypt the sparse weight parameter groups and uses a dynamically decoded key to generate a logical computation key in the accelerator. Compared with the existing technology that uses a decoded key stored in a table, this reduces the storage area occupied by the decoded key and the storage overhead of decompressing the sparse weights and decoding the key. Attached Figure Description

[0049] Figure 1 This is a flowchart of a neural network acceleration method according to an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of an algorithm for compressing sparse weight parameter groups using a key, according to an embodiment of the present invention.

[0051] Figure 3 This is a schematic diagram of an algorithm for decompressing sparse weight parameter groups using a key, according to an embodiment of the present invention.

[0052] Figure 4 This is a schematic diagram illustrating the use of a key to compress and decompress a sparse weight parameter group according to an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of a load balancing method for pairing, grouping, and compressing sparse weight parameters according to an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of a neural network accelerator according to an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of a priority encoder according to an embodiment of the present invention;

[0056] Figure 8 This is a schematic diagram of an XOR decoder according to an embodiment of the present invention;

[0057] Figure 9 This is a schematic diagram of the dynamic decoding key construction logic in an embodiment of the present invention. Detailed Implementation

[0058] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0059] In the following text, the terms “first,” “second,” etc., are used to distinguish between similar elements and are not necessarily used to describe a specific order or chronological sequence. It should be understood that these terms, as used herein, may be replaced where appropriate. Similarly, if the methods described herein comprise a series of steps, and the order of these steps presented herein is not necessarily the only possible order in which they can be performed, and some described steps may be omitted and / or other steps not described herein may be added to the method.

[0060] Furthermore, it should be understood that when a layer (or film), region, pattern, or structure is referred to as being "on" a substrate, layer (or film), region, and / or pattern, it can be located directly on another layer or substrate, and / or intercalation layers may also be present. Additionally, it should be understood that when a layer is referred to as being "under" another layer, it can be located directly under that layer, and / or one or more intercalation layers may also be present. Furthermore, references to "on" and "under" the layers may be made based on the accompanying drawings.

[0061] Please refer to Figure 1 This invention provides a neural network acceleration method, comprising:

[0062] Step S1: Divide the sparse weight parameters into multiple sparse weight parameter groups so that the sparsity of the sparse weight parameters contained in each sparse weight parameter group is balanced.

[0063] Step S2: Compress the sparse weight parameter set using a static key;

[0064] Step S3: Distribute the multiple compressed sparse weight parameter groups to multiple processing unit groups. Each processing unit group decompresses the sparse weight parameter group and filters out the sparse weight parameters with non-zero weights.

[0065] Step S4: The processing unit group obtains the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and performs multiplication and accumulation calculations on all sparse weight parameters and input data in the group.

[0066] Preferably, after the processing unit group obtains the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and performs multiplication and accumulation calculations on all sparse weight parameters and input data in the group, it further includes: outputting the result after multiplication and accumulation calculations.

[0067] Preferably, the method for compressing sparse weight parameters using a static key includes: using the vector of the sparse weight parameter group as a first vector; mapping it to a second vector through a key matrix, the length of the second vector being less than the length of the first vector; and using a bitmap to record the positions of non-zero values ​​in the first vector. The method for decompressing the sparse weight parameter group includes: using a key matrix to recover a third vector from the second vector, the length of the third vector being the same as the length of the first vector; recording the different bits in the first and third vectors as error codes; and correcting the error codes. Figure 2 , Figure 2 This is a schematic diagram of an algorithm that uses a key to compress sparse weight parameter sets. The XORNet compression process first flattens the first vector Ws (n bits) of the input sparse weight data, converting the sparse weights from the original matrix format to a one-dimensional vector format. Then, it performs bit decomposition, decomposing the multi-bit sparse weights of the first vector into n multi-bit vectors, resulting in n vectors of length Nsparse. Finally, it encodes and compresses the data using a decoding key matrix, as shown below. Figure 3 , Figure 3 This is a schematic diagram of an algorithm that uses a key to decompress a sparse weight parameter set. Decoding yields a second vector We, which is a shorter (Nencoded) weight data after compression. The XORNet decompression process first decodes and decompresses the encoded weight data We using a decoding key matrix, resulting in a third vector Wd, which is the same length as the first vector Ws. Finally, a reshaping process restores the original weight data format.

[0068] like Figure 4 , Figure 4 This is a specific implementation of the present invention using a key to compress a sparse weight parameter set. In this example, Nsparse = 8 and Nencoded = 3 represent a sparse weight compression ratio of 62.5%. The input 8 sparse weight data Ws are first flattened and bit-decomposed to obtain a vector of length Nsparse = 8. Then, XORNet compression is performed by multiplying the decoded key matrix in the GF(2) domain to obtain the compressed encoded weight data We. The XOR decompression process involves multiplying the decoded key matrix and We in the GF(2) domain to obtain the decoded weight Wd. In this example, the decoded weight data contains encoding errors, requiring error recording to achieve lossless compression in hardware.

[0069] Preferably, the method of dividing all sparse weight parameters into multiple sparse weight parameter groups, such that the sum of the values ​​of the sparse weight parameters contained in each sparse weight parameter group is balanced, includes: arranging the sparse weight parameters in ascending order of sparsity; combining smaller sparse weight parameters with larger sparse weight parameters into a sparse weight parameter group, thereby forming multiple sparse weight parameter groups, and ensuring that the sum of the values ​​of the sparse weight parameters contained in the multiple sparse weight parameter groups is balanced. For example... Figure 5 , Figure 5 This is a specific implementation example of dividing all sparse weight parameters into multiple sparse weight parameter groups. In this example, 32 convolutional kernels are paired with 4 convolutional kernels. For the sparse weight data of the 32 convolutional kernels in the neural network layer, each convolutional kernel is first sorted by sparsity, resulting in a sparsity sequence relationship among the 32 kernels. Based on the sparsity sorting, according to the sparsity sequence relationship, the two convolutional kernels with the highest sparsity and the two convolutional kernels with the lowest sparsity are paired, and this process is repeated sequentially until 8 pairs are completed. The paired convolutional kernels are then cross-flattened, and the cross-flattened sparse weight data can then be compressed and stored using XORNet. In other embodiments of this invention, other numbers of convolutional kernels or other numbers of pairs can be used.

[0070] Accordingly, please refer to Figure 6 The present invention also provides a neural network accelerator applied to a neural network acceleration method, comprising: an on-chip cache module and a computation module; the on-chip cache module is used to store input feature maps, output feature maps, and sparse weight parameters in compressed format; the computation module includes N processing unit groups, each processing unit group being used to decompress the sparse weight parameter group, filter out sparse weight parameters with non-zero weights, obtain the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and perform multiplication and accumulation calculations on all sparse weight parameters in the sparse weight parameter group and the input data. It also includes: a control module, which is used to provide control signals to the on-chip cache module and the computation module.

[0071] The on-chip cache module includes a feature map storage unit and N network weight storage units. The feature map storage unit is used to store the input feature map and the output feature map. The N network weight storage units are used to store sparse weight parameters in compressed format. The sparse weight parameters in compressed format include the encoded weights, the bitmap, and the encoding error information.

[0072] The computation module also includes a feature map data distribution unit and an activation unit. The feature map data distribution unit is used to distribute the input feature map to multiple processing unit groups respectively. The activation unit is used to perform nonlinear activation operation on the output data after multiplication and accumulation calculation and write it back to the on-chip cache module.

[0073] The processing unit group includes: a sparse weight decompression unit, an input feature map selection unit, and a multiply-accumulate array group (MAC array); the sparse weight decompression unit is used to decompress the sparse weight parameter group; the input feature map selection unit is used to select the input data corresponding to the sparse weight parameters; the multiply-accumulate array group is used to perform multiply-accumulate calculations on all sparse weight parameters and input data in the sparse weight parameter group.

[0074] The sparse weight decompression unit includes a priority encoder, an XOR decoder, and an error correction unit. The priority encoder encodes the bitmap data and outputs the positions of non-zero weights in the sparse weight parameters. The XOR decoder decompresses the compressed sparse weight parameter group. The error correction unit flips the erroneously encoded bits in the decompressed sparse weight parameters to complete error correction. The XOR decoder includes dynamic decoding key generation logic and an XOR decoding array. The dynamic decoding key generation logic uses the position data of non-zero weights in the sparse weight parameters generated by the priority encoder to generate a dynamic decoding key. The XOR decoding array uses the dynamic decoding key to decompress the compressed sparse weight parameters.

[0075] like Figure 7 , Figure 7 This is a specific embodiment of the priority encoder of the present invention. Taking a 32-bit bitmap as an example, four non-zero encoded positions are generated in each cycle. In this example, the four non-zero encoded positions are P1=0, P2=4, P3=8, and P4=11.

[0076] like Figure 8 , Figure 8 This is a specific embodiment of the XOR decoder of the present invention. This example uses Nsparse=32 and Nencoded=4; the decoding process can be completed using the XOR array in the XOR decoder. Figure 9 , Figure 9 This is one embodiment of dynamic decoding key construction logic. The decoding key matrix is ​​generated by the dynamic decoding key construction logic, which receives the 5-bit non-zero encoding position P generated by the priority decoder, and then generates KP data, i.e., the decoding key matrix, according to the corresponding logic for use in the decoding process.

[0077] In summary, the neural network accelerator and acceleration method provided in this invention divides all sparse weight parameters into multiple sparse weight parameter groups, ensuring a balanced sparsity of sparse weight parameters within each group. This improves the workload balance of all processing units, thereby increasing the overall throughput of the accelerator. Furthermore, this invention uses a key to encrypt the sparse weight parameter groups and generates a logical computation key in the accelerator using a dynamically decoded key. Compared to existing technologies that rely on searching for the decoded key in a table, this reduces the storage area occupied by the decoded key and the storage overhead associated with decompressing the sparse weights and decoding the key.

[0078] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A neural network acceleration method, characterized in that, include: The sparse weight parameters are divided into multiple sparse weight parameter groups so that the sparsity of the sparse weight parameters contained in each sparse weight parameter group is balanced. The sparse weight parameter set is compressed using a static key; The multiple compressed sparse weight parameter sets are distributed to the processing unit groups of multiple neural network accelerators. Each processing unit group calculates the key in a dynamic manner, decompresses the sparse weight parameter sets, and filters out the sparse weight parameters with non-zero weights. The processing unit group obtains the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and performs multiplication and accumulation calculations on all sparse weight parameters in the group and the input data; Methods for compressing sparse weight parameter sets using static keys include: Use the vector of the sparse weight parameter group as the first vector; The key matrix is ​​used to map the vector into a second vector, the length of which is less than the length of the first vector. Use a bitmap to record the positions of non-zero values ​​in the first vector; The method for decompressing the sparse weight parameter set includes: The second vector is restored to a third vector using a static key matrix, the length of the third vector being the same as the length of the first vector. The different bits in the first and third vectors are recorded as error codes; Correct the error codes; A method for dividing all the sparse weight parameters into multiple sparse weight parameter groups, such that the sparsity of the sparse weight parameters contained in each sparse weight parameter group is balanced, includes: Arrange the sparse weight parameters in ascending order of sparsity; Sparse weight parameters with smaller sparsity are paired with sparse weight parameters with larger sparsity to form a sparse weight parameter group, thereby forming multiple sparse weight parameter groups, and the sparsity of the sparse weight parameters contained in the multiple sparse weight parameter groups is balanced.

2. The neural network acceleration method as described in claim 1, characterized in that, The processing unit group acquires the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and after performing multiplication and accumulation calculations on all sparse weight parameters in the group and the input data, it further includes: Output the result after multiplication and summation.

3. A neural network accelerator applied to the neural network acceleration method according to any one of claims 1 to 2, characterized in that, include: On-chip cache module and computing module; The on-chip cache module is used to store the input feature map, the output feature map, and the sparse weight parameters in the compressed format; The calculation module includes N processing unit groups. Each processing unit group is used to decompress the sparse weight parameter group, filter out the sparse weight parameters with non-zero weights, obtain the input data corresponding to the sparse weight parameters in the sparse weight parameter group, and perform multiplication and accumulation calculation on all sparse weight parameters in the sparse weight parameter group and the input data.

4. The neural network accelerator as described in claim 3, characterized in that, Also includes: A control module is provided to the on-chip cache module and the computing module.

5. The neural network accelerator as described in claim 3, characterized in that, The on-chip cache module includes a feature map storage unit and N network weight storage units. The feature map storage unit is used to store the input feature map and the output feature map, and the N network weight storage units are all used to store sparse weight parameters in compressed format.

6. The neural network accelerator as described in claim 3, characterized in that, The computing module also includes: Feature map data distribution unit and activation unit; The feature map data distribution unit is used to distribute the input feature map to multiple processing unit groups respectively; The activation unit is used to perform nonlinear activation operations on the output data after multiplication and accumulation calculations, and then write it back to the on-chip cache module.

7. The neural network accelerator as described in claim 3, characterized in that, The processing unit group includes: Sparse weight decompression unit, input feature map selection unit, and multiply-accumulate array group; The sparse weight decompression unit is used to decompress the sparse weight parameter set; The input feature map selection unit is used to select the input data corresponding to the sparse weight parameters; The multiply-accumulate array group is used to perform multiply-accumulate calculations on all sparse weight parameters in the sparse weight parameter group and the input data.

8. The neural network accelerator as described in claim 7, characterized in that, The sparse weighted decompression unit includes: Priority encoder, XOR decoder and error correction unit; The priority encoder encodes the bitmap data and outputs the positions of non-zero weights in the sparse weight parameters. The XOR decoder is used to decompress the sparse weight parameters of the compressed format; The error correction unit is used to flip the erroneously encoded bits in the decompressed sparse weight parameters to complete the error correction.

9. The neural network accelerator as described in claim 8, characterized in that, The XOR decoder includes: Dynamic decoding key generation logic and XOR decoding array; The dynamic decoding key construction logic uses the position data of non-zero weights in the sparse weight parameters generated by the priority encoder to generate a dynamic decoding key; The XOR decoding array uses the dynamic decoding key to decompress the sparse weight parameters of the compressed format.

Images