Design method and use method of variable precision calculation unit applied to quantized neural network convolution layer

Abstract

The application provides a design method and use method of a variable precision calculation unit applied to a quantized neural network convolution layer, shift operation is performed on a vector with a precision of 1, and the shift bit number, dimension and number of an operation basic block are determined. After the operation basic blocks are arranged, an addition tree is connected to form an array, and a variable precision fusion calculation unit is obtained. According to the actual precision of the current convolution layer activation value and the weight value, the variable precision fusion calculation unit array is configured, and each column of the fusion calculation unit is calculated in parallel. After the current convolution layer operation is completed, the two-dimensional array is reconfigured according to the data precision of the next convolution layer, and the operation of all convolution layers is completed in this way. The problems that the operation unit in the prior art is prone to causing waste of calculation resources in convolution operation less than 4 bits, and cannot dynamically adopt different quantization bit widths to process data according to the actual situation of the convolution layer are solved, and the resource utilization rate and the calculation efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence hardware acceleration, and particularly relates to a design method and use method of a variable precision calculation unit applied to a quantized neural network convolution layer, and particularly to an architecture of a special hardware acceleration unit for a convolution layer. BACKGROUND

[0002] With the continuous development of neural network models, the depth and number of neurons increase dramatically, and the parameter quantity of some models has broken through the level of one hundred billion. Generally, the trained model is deployed on edge devices (such as smart phones, embedded sensors and smart cameras) to perform tasks such as target recognition and speech recognition. However, such devices are limited by low power design and limited storage resources, and it is difficult to meet the needs of deploying large neural network models. Specifically, the high computational load of model inference and frequent memory access requirements significantly restrict the actual application performance. Under this background, the collaborative optimization of model compression technology and hardware acceleration has become a research hotspot.

[0003] Quantization technology, as a core means of model compression, can effectively reduce the computational load and memory occupation by reducing the bit precision of weights and activation values, while maintaining the model inference accuracy. In recent years, quantization technology has developed rapidly. Studies have shown that a low-precision quantization model with less than 4 bits can achieve results comparable to a 32-bit floating-point model. In addition, the mixed precision quantization strategy further alleviates the precision loss problem caused by uniform low-precision quantization by dynamically allocating different quantization bit widths for different convolution layers.

[0004] The existing technical solutions for neural network model inference hardware optimization are as follows:

[0005] TPU is based on systolic array structure, supports 8 / 16-bit wide calculation, and significantly improves the efficiency of large-scale matrix operation. However, it has insufficient support for operations less than 4 bits, which will cause waste of operation resources. DaDianNao adopts fixed bit width design (DaDianNao: A Machine-Learning Supercomputer," in MICRO, 2014.), and realizes 256 times of multiplication operation per cycle, which optimizes the efficiency of convolution calculation. However, due to its fixed bit width design, it cannot provide good support for mixed precision quantization. Stripes activation value adopts variable precision, while weight data adopts fixed bit width of 16 bits (Stripes: Bit-serial deep neural network computing," in MICRO, 2016). This is not friendly to neural network model inference operation with weight quantization to very low precision. Bit Fusion architecture adopts dynamic bit width adjustment mechanism (Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Network," in ISCA, 2018), which reduces hardware power consumption while maintaining precision. However, its minimum granularity is 2 bits, which has insufficient support for binary neural network inference operation, and will cause waste of operation resources.

[0006] In summary, the existing hardware has the following two problems for very low precision quantization neural network model inference operation:

[0007] 1. The data precision supported by the operation unit is high, and in the low precision convolution operation less than 4 bits, it is easy to cause waste of calculation resources.

[0008] 2. Insufficient support for mixed precision quantization operation.

[0009] Therefore, how to provide a mixed precision quantization calculation method that can adapt to different operation requirements of high precision (32 bits and above) and low precision (less than 32 bits, such as 4 bits, 2 bits, 1 bit) has become a problem to be solved. SUMMARY

[0010] The application provides a design method and use method of a variable precision calculation unit applied to a quantized neural network convolution layer, to solve the problem that the data precision supported by an operation unit is high, and calculation resources are easily wasted in convolution operation less than 4 bits, and to solve the problem that the prior art has insufficient support for mixed precision quantization, and cannot dynamically adopt different quantization bit widths to process data according to the actual situation of the convolution layer, to improve resource utilization and calculation efficiency.

[0011] In order to achieve the above-mentioned purpose, the technical scheme of the application provides a design method of a variable precision calculation unit applied to a quantized neural network convolution layer, comprising: counting the number of bits 1 in a binary, performing a shift operation on a vector with a precision of 1, and determining the shift bit number of each operation basic block. The dimension and number of operation basic blocks are calculated according to the dimension and precision of the current vector to be calculated. Each operation basic block is arranged according to the physical array structure of the current hardware and the shift bit number of each operation basic block. After arrangement, an addition tree is connected to form an array to obtain a variable precision fusion calculation unit. The fusion calculation unit can output corresponding calculation results according to the precision of different vectors to be calculated.

[0012] As a preferred embodiment of the above technical scheme, preferably, counting the number of bits 1 in a binary, performing a shift operation on a vector with a precision of 1, and determining the shift bit number of each operation basic block, comprises: cutting a vector by bit according to the precision of the vector to obtain a plurality of sub-vectors; performing a bitwise AND operation on the split sub-vectors based on a bit stream algorithm, and counting the number of bits 1 in the vector result after the AND operation; performing a shift operation on a vector with a precision of 1 according to the bit weight of the sub-vector, thereby determining the shift bit number of each operation basic block.

[0013] As a preferred embodiment of the above technical scheme, preferably, the dimension and number of operation basic blocks are calculated according to the dimension and precision of the current vector to be calculated, comprising: presetting the dimension of the operand vector, splitting the dimension of the current vector to be calculated according to the preset dimension to obtain a plurality of independent low-dimensional vectors, thereby determining the dimension of the operation basic block; and determining the number of operation basic blocks required according to the precision of the current vector to be calculated.

[0014] As the preferred technical scheme, preferably, the dimension of the operand vector is preset, the current vector dimension to be calculated is split according to the preset dimension to obtain a plurality of low-dimensional vectors independent of each other, so as to determine the dimension of the operation basic block, comprising: when the preset dimension of the operand vector is equal to the current vector dimension, no splitting is needed, and the current vector is directly sliced according to the dimension to obtain the plurality of low-dimensional vectors independent of each other; when the preset dimension of the operand vector is greater than the current vector dimension, the number of operand vectors is supplemented; when the preset dimension of the operand vector is less than the current vector dimension, the current vector dimension is split according to the preset dimension of the operand vector to obtain a plurality of low-dimensional vectors independent of each other.

[0015] As the preferred technical scheme, preferably, according to the physical array structure of the current hardware and the shift bit number of each operation basic block, each operation basic block is arranged, and after the arrangement, an addition tree is connected to form an array to obtain a variable-precision fusion computing unit, and further comprising: a weight cache and an input cache capable of providing operands for the operation basic block are arranged outside the array, and each operation basic block, the weight cache and the input cache constitute the variable-precision fusion computing unit; wherein the hardware unit in which the operation basic block is fixed can meet the shift of the operation basic block.

[0016] As the preferred technical scheme, preferably, the fusion computing unit can output corresponding calculation results according to the precision of different vectors to be calculated, comprising: according to the precision, a plurality of operation basic blocks are respectively arranged to form a plurality of processing units, the lower the precision, the fewer the operation basic blocks contained in a single processing unit, the more the processing units in the current fusion computing unit, and the higher the parallelism degree of the fusion computing unit.

[0017] The application also provides a use method of a variable-precision computing unit applied to a quantized neural network convolution layer and designed by using the above design method, comprising: according to the actual precision of the activation value and the weight value of the current convolution layer, the shift bit number of all operation basic blocks in a plurality of variable-precision fusion computing units arranged in a two-dimensional array is configured; each column of fusion computing units is calculated in parallel, and different feature maps are respectively output, so as to complete the operation of the entire convolution layer and obtain all output feature maps; after the operation of the current convolution layer is completed, the two-dimensional array is reconfigured according to the data precision of the next convolution layer, and then the next convolution layer is calculated by using the newly configured two-dimensional array, and the operation of all convolution layers is completed in this way.

[0018] As the preferred technical solution of the above, preferably, according to the actual precision of the current convolution layer activation value and weight value, the shift bit number of all operation basic blocks in the plurality of variable precision fusion calculation units in the two-dimensional array is configured, and then the method comprises: determining the current fusion logic according to the dimension and precision of the convolution layer, fusing the operation basic blocks according to the fusion logic, and constituting the processing units in the variable precision fusion calculation unit; wherein the shift bit numbers of the fused operation basic blocks are the same / different.

[0019] As the preferred technical solution of the above, preferably, the different column fusion calculation units calculate in parallel, and output different feature maps respectively, and the method comprises: the processing units at the same position in the different column fusion calculation units output different feature values at the same position of the same feature map in the process of parallel calculation.

[0020] As the preferred technical solution of the above, preferably, the plurality of variable precision fusion calculation units in the two-dimensional array specifically comprises a plurality of variable precision fusion calculation units arranged in rows and columns, each row of variable precision fusion calculation units shares an input cache to realize that the input cache simultaneously provides data to a row of variable precision fusion calculation units, each variable precision fusion calculation unit has a respective independent weight cache, which is used to accumulate the calculation results of all variable precision fusion calculation units in each column and use the accumulated results as output results, which are saved in an output cache.

[0021] The application provides a design method and use method of a variable precision calculation unit applied to a quantized neural network convolution layer, the number of bits 1 in a binary is counted, a vector with a precision of 1 is subjected to a shift operation, and the shift bit number of each operation basic block is determined. The dimension and number of operation basic blocks are calculated according to the dimension and precision of the current vector to be calculated. Each operation basic block is arranged according to the physical array structure of the current hardware and the shift bit number of each operation basic block, and after the arrangement, an addition tree is used for connection to constitute an array, thereby obtaining a variable precision fusion calculation unit. The fusion calculation unit can output corresponding calculation results according to the precision of different vectors to be calculated. According to the actual precision of the current convolution layer activation value and weight value, the shift bit number of all operation basic blocks in the plurality of variable precision fusion calculation units in the two-dimensional array is configured; each column fusion calculation unit calculates in parallel, and outputs different feature maps respectively, thereby completing the operation of the entire convolution layer and obtaining all output feature maps; after the operation of the current convolution layer is completed, the two-dimensional array is reconfigured according to the data precision of the next convolution layer, and then the next convolution layer is calculated using the newly configured two-dimensional array, and the operation of all convolution layers is completed in this way.

[0022] The application solves the problem of waste of computing resources in less than 4-bit convolution operation caused by high data precision supported by the operation unit in the prior art, and solves the problem of insufficient support for mixed precision quantization in the prior art, and the problem of inability to dynamically use different quantization bit widths to process data according to the actual situation of the convolution layer to improve resource utilization and computing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0024] Figure 1 The flowchart of the design method of the variable precision computing unit applied to the quantized neural network convolution layer provided by the present application.

[0025] Figure 2 The flowchart of the use method of the variable precision computing unit obtained based on the design method provided by the present application.

[0026] Figure 3 The schematic diagram of vector slicing and packing in the design method of the variable precision computing unit applied to the quantized neural network convolution layer provided by the present application.

[0027] Figure 4 The schematic diagram of the operation basic block of the I-dimensional operand and the operation basic block of the Is-dimensional operand in the design method of the variable precision computing unit applied to the quantized neural network convolution layer provided by the present application.

[0028] Figure 5 The schematic diagram of the dot product operation process using the operation basic block in the design method of the variable precision computing unit applied to the quantized neural network convolution layer provided by the present application.

[0029] Figure 6 The schematic diagram of the variable precision fusion computing unit with 16 operation basic blocks after the operation basic blocks obtained based on the design method of the present application are arranged.

[0030] Figure 7A The schematic diagram of the processing unit logically composed of different numbers of basic blocks in the variable precision fusion computing unit Figure 1 .

[0031] Figure 7B The schematic diagram of the processing unit logically composed of different numbers of basic blocks in the variable precision fusion computing unit Figure 2 .

[0032] Figure 7C Schematic diagram of using different numbers of basic blocks to logically constitute a processing unit in a variable-precision fusion calculation unit Figure 3 .

[0033] Figure 7D Schematic diagram of using different numbers of basic blocks to logically constitute a processing unit in a variable-precision fusion calculation unit Figure 4 .

[0034] Figure 8A Schematic diagram of the shift of basic blocks in different logical fusions of a processing unit using different numbers of basic blocks to logically constitute the processing unit in a variable-precision fusion calculation unit Figure 1 , wherein the numbers in the basic blocks represent the number of shift bits.

[0035] Figure 8B Schematic diagram of the shift of basic blocks in different logical fusions of a processing unit using different numbers of basic blocks to logically constitute the processing unit in a variable-precision fusion calculation unit Figure 2 , wherein the numbers in the basic blocks represent the number of shift bits.

[0036] Figure 8C Schematic diagram of the shift of basic blocks in different logical fusions of a processing unit using different numbers of basic blocks to logically constitute the processing unit in a variable-precision fusion calculation unit Figure 3 , wherein the numbers in the basic blocks represent the number of shift bits.

[0037] Figure 8D Schematic diagram of the shift of basic blocks in different logical fusions of a processing unit using different numbers of basic blocks to logically constitute the processing unit in a variable-precision fusion calculation unit Figure 4 , wherein the numbers in the basic blocks represent the number of shift bits.

[0038] Figure 9 Schematic diagram of a convolution layer.

[0039] Figure 10 Array structure of a variable-precision fusion unit in a method for using a variable-precision calculation unit provided by the present application. DETAILED DESCRIPTION

[0040] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0041] First, the design method of the variable precision calculation unit applied to the quantization of the neural network convolution layer in the application is briefly described, as shown in Figure 1

[0042] Step 101, count the number of bits 1 in the binary, perform a shift operation on the vector with a precision of 1, and determine the shift bit number of each operation basic block.

[0043] Specifically, according to the precision of the vector, the vector is cut by bit to obtain a plurality of sub-vectors. According to the Bit-serial algorithm (bit stream algorithm), the sub-vectors are cut by bit and the number of bits 1 in the binary is counted, that is, the number of bits 1 in the vector after the AND operation is counted. According to the bit weight of each of the sub-vectors with a precision of 1 bit, a shift operation is performed on each of the sub-vectors to determine the shift bit number of each operation basic block.

[0044] Step 102, calculate the dimension and number of operation basic blocks according to the dimension and precision of the current vector to be calculated.

[0045] The dimension of the operand vector is preset, the current vector dimension to be calculated is split according to the preset dimension to obtain a plurality of low-dimensional vectors independent of each other, so as to determine the dimension of the operation basic block; and the number of operation basic blocks required is determined according to the precision of the current vector to be calculated.

[0046] For the above splitting process, specifically including:

[0047] When the preset operand vector dimension is equal to the current vector dimension, no splitting is needed, and the current vector is directly sliced according to the dimension to obtain the plurality of low-dimensional vectors independent of each other.

[0048] When the preset operand vector dimension is greater than the current vector dimension, the number of operand vectors is supplemented.

[0049] When the preset operand vector dimension is less than the current vector dimension, the current vector dimension is split according to the preset operand vector dimension to obtain a plurality of low-dimensional vectors independent of each other.

[0050] Step 103, arrange each operation basic block according to the physical array structure of the current hardware and the shift bit number of each operation basic block, and arrange the array after the addition tree connection to form a variable precision fusion calculation unit.

[0051] Further, the weight cache and input cache capable of providing operands for the operation basic blocks are arranged outside the array, and the operation basic blocks, the weight cache and the input cache constitute the variable precision fusion calculation unit; wherein the hardware unit of the operation basic block can meet the shift of the operation basic block.

[0052] ​Step 104, the fusion computing unit can output the corresponding calculation result according to the accuracy of different to-be-calculated vectors.

[0053] According to the accuracy of the to-be-calculated vector, the fusion computing unit recombines different numbers of operation basic blocks therein, thereby forming a plurality of processing units. The lower the accuracy of the to-be-calculated vector, the fewer operation basic blocks included in a single processing unit, the more processing units in the current fusion computing unit, and the higher the parallelism of the fusion computing unit.

[0054] Secondly, the use method of the variable-precision computing unit applied to the quantized neural network convolution layer obtained by the design method is briefly described, as shown in Figure 2

[0055] Step 201, according to the actual accuracy of the activation value and the weight value of the current convolution layer, the shift bit number of all operation basic blocks in the plurality of variable-precision fusion computing units arranged in a two-dimensional array is configured.

[0056] The determination of the shift bit number of the operation basic block is realized by step 101; step 102 is performed according to the actual accuracy of the activation value and the weight value of the current convolution layer to determine the number of operation basic blocks; after obtaining the shift bit number and the number of operation basic blocks, step 103 is performed to solidify operation basic blocks with different shift bit numbers on the physical array structure of the current hardware, so that each operation basic block can meet the bit shift requirement in the operation process, and the array is connected by an addition tree after arrangement to obtain a variable-precision fusion computing unit.

[0057] Step 202, according to the dimension and accuracy of the convolution layer, the current fusion logic is determined to fuse the operation basic blocks to form processing units in the variable-precision fusion computing unit; wherein the shift bit numbers of the fused operation basic blocks are the same / different.

[0058] This step aims to determine the fusion logic according to the dimension and accuracy of the convolution layer and the output requirement, then group each operation basic block in the variable-precision fusion computing unit, and then obtain the processing unit.

[0059] A variable-precision fusion computing unit includes a plurality of processing units, and the plurality of processing units include at least one operation basic block.

[0060] ​The several variable precision fusion calculation units are arranged in a two-dimensional array to perform data processing on the convolution layer. The several variable precision fusion calculation units in the two-dimensional array include several variable precision fusion calculation units arranged in rows and columns. Each row of variable precision fusion calculation units shares an input buffer to simultaneously provide data to the variable precision fusion calculation units in the row. Each variable precision fusion calculation unit has an independent weight buffer for accumulating the calculation results of all the variable precision fusion calculation units in each column as output results and storing the output results in an output buffer.

[0061] In step 203, the columns of fusion calculation units perform parallel calculation and output different feature maps respectively, thereby completing the operation of the entire convolution layer and obtaining all the output feature maps.

[0062] Specifically, in the array of variable precision fusion calculation units arranged in a two-dimensional array, the processing units at the same position in different columns of fusion calculation units respectively output different feature values at the same position in the same feature map in the process of parallel calculation.

[0063] In step 204, after the operation of the current convolution layer is completed, the two-dimensional array is reconfigured according to the data precision of the next convolution layer, and then the next convolution layer is calculated using the new two-dimensional array. This process is repeated to complete the operation of all convolution layers.

[0064] The design method of the variable precision calculation unit applied to the quantized neural network convolution layer provided by the application is described through specific examples.

[0065] Suppose that the vector a and the vector w need to be calculated by point multiplication at present, the dimension of the vector a and the vector w is I, the precision of the vector w is N, and the precision of the vector a is M. It is assumed that I=4, N=2, and M=2. The vector w and the vector a are cut by bits and recombined by bits to form sub-vectors, as shown in the following table. The popcount function is used to count the number of 1s in binary. It can be understood that the vector w and the vector a are both I-dimensional vectors with an element precision of 1 bit. According to the Bit-serial algorithm, the number of operation basic blocks is calculated using Bit-serial calculation. Specifically, the point multiplication of the vector w with a precision of N and the vector a with a precision of M requires Figure 3

[0066]

[0067] ​​​​​​​​​​​one such basic block. Figure 3 For example, the operation of the dot product of and requires 4 operation basic blocks, and the operation process can be represented as Figure 4 .

[0068]

[0069] According to the further calculation of formula (1) according to the specific numerical values of the above examples, the following results are obtained:

[0070]

[0071] Therefore, it can be known that the dot product of the operation precision of N and the precision of M requires 4 operation basic blocks. Further, as shown in Figure 5 , 4 basic blocks are used to respectively operate the sliced vectors of and vector. Each group of operations is respectively subjected to bitwise AND operation, popcount operation and shift operation, and finally the correct result is obtained by accumulating.

[0072] It should be noted that the shift number of each group of operations is determined by the subscript of the sliced vector. According to formula (2), the shift numbers of the 4 basic blocks are 0, 1, 1 and 2 respectively.

[0073] In actual convolution operation, the number of multiplication and addition operation groups required to calculate one output feature value is often large, and directly using the operand with a length of I as the input of the basic block cannot achieve the purpose of the present application.

[0074] In this regard, the present application adopts the operation of splitting the multiplication and addition operation:

[0075] The I-dimensional vector dot product is split into k+1 groups of vector dot products. Except for the last group, the vector dimension in each group of vector dot products is Is, and the value of Is is much smaller than the value of I. In this way, each group of vector dot product operations can be respectively operated using the operation basic block, as shown in Figure 6 .

[0076]

[0077] wherein k+1 is the split group number of splitting the I-dimensional vector into Is-dimensional vector, and k is a natural number.

[0078] The I-dimensional vector dot product is split into a set of summation of Is-dimensional vector dot products for operation on the operation basic block with operand bit width of Is, which can be divided into three split cases. In the following example case, the current vector is an I-dimensional vector, the operand vector dimension is Is, and the low-dimensional vector is Is equal to the operand vector dimension.

[0079] 1. When I = Is, no split is needed, and the vector is directly sliced by bit according to the precision of the current vector to obtain a set of Is-dimensional vectors with element precision of 1 bit, and the operation is performed on the basic block.

[0080] 2. When I < Is, the Is-dimensional vector should be supplemented. It is assumed that I = 3 and Is = 4, and the bit-serial basic block is used for dot product operation, which needs to be supplemented. As shown in equation (4), wherein, .

[0081]

[0082] 3. When I > Is, the split should be performed, and the current vector dimension I is split according to the preset operand vector dimension Is to obtain a set of independent low-dimensional vectors. For example, when I = 8 and Is = 4, as shown in equation (5), 、 、 and are independent low-dimensional vectors after splitting.

[0083]

[0084] wherein, MAC is the value of the vector dot product.

[0085] As described above, using M × N operation basic blocks, the dot product operation of two Is-dimensional vectors with bit widths of M bits and N bits, i.e., Is group multiplication and addition operation, can be realized. If a fixed number of operation basic blocks is given, the set of operation basic blocks can realize variable precision multiplication and addition operation through internal logic fusion.

[0086] Accordingly, the technical scheme of the present application provides a design method of a variable precision fusion calculation unit. Further, as Figures 7A to 8DAs shown, the embodiment can realize the dot product calculation of the vector w and the vector a through 16 operation basic blocks, arranged in a physical structure of 4x4, connected by an addition tree, and the operation basic array outside contains two buffers, namely the weight buffer and the input buffer, which provide operands for all basic blocks in the array composed of operation basic blocks. Specifically, according to the input precision and the weight precision, the slicing mode of the input buffer and the weight buffer is determined to obtain a vector with a precision of 1, which provides data for the variable precision fusion calculation unit.

[0087] The operation basic block array together with the two buffers forms a variable precision fusion calculation unit (VPFU). Further, within the variable precision fusion calculation unit, a certain number of basic blocks are logically combined to form a processing element (PE). The fewer basic blocks the PE contains, the lower the precision of the data supported by the operation, and the more PEs the VPFU contains, and the higher the parallelism. Conversely, the more basic blocks the PE contains, the higher the precision of the data supported by the operation, but the fewer PEs the VPFU contains, and the lower the parallelism.

[0088] Based on the above embodiment, the use method of the variable precision fusion calculation unit obtained based on the above design method is described.

[0089] The general convolution layer is as shown in Figure 9 As shown, the input feature map is usually a three-dimensional structure (length x width x channel), and the convolution kernel group is a four-dimensional structure (number x channel number x length x width). Each convolution kernel performs convolution calculation with the overlapping part of the input feature map through sliding operation, generates a feature value of the corresponding output feature map, and generates an output feature map from it. A plurality of independent convolution kernels are respectively convolved with the input feature map, and finally a plurality of output feature maps (the number is consistent with the number of convolution kernels) are generated. Since there is no data dependency between the calculation processes of different convolution kernels, the operation process can be completely parallelized.

[0090] According to this characteristic, the variable precision fusion calculation unit is combined into a two-dimensional array, as shown in Figures 7A to 7D .

[0091] In the process of combining the two-dimensional array, the fusion logic is determined according to the dimension and precision of the convolution layer and the output demand, and then each operation basic block in the variable precision fusion calculation unit is grouped to obtain a processing element.

[0092] Figures 7A to 7DFour different ways of fusing basic blocks are presented, each of which shows different trade-off between operation precision and parallelism:

[0093] In Figure 7A , a single basic block is used to construct a processing element (PE), each PE supports Is group multiplication-addition operation with 1-bit input data and 1-bit weight. VPFU integrates 16 PEs, which realizes the maximum parallelism, and each operation cycle completes 16 Is group multiplication-addition operations. However, the input / weight bit width is the lowest, which is 1 bit, and VPFU finally outputs a partial sum, Figure 7A The bit shift number of each operation block in Figure 8A is shown.

[0094] In Figure 7B , 2x2 basic blocks are logically fused to construct a PE, which supports multiplication-addition operation with 2-bit input data and 2-bit weight. VPFU integrates 4 such PEs, which realizes 4 Is group multiplication-addition operations per calculation cycle, and finally generates a partial sum, Figure 7B The bit shift number of each operation block in Figure 8B is shown.

[0095] In Figure 7C , 1x4 basic blocks are logically fused to construct a PE, which supports multiplication-addition operation with 2-bit input data and 2-bit weight. Similarly, in VPFU, there are 4 such PEs, which can calculate 4 Is group multiplication-addition operations in total and generate a partial sum output, Figure 7C The bit shift number of each operation block in Figure 8C is shown.

[0096] Finally, Figure 7D , 16 basic blocks are fused into a PE, which can operate 1 Is group multiplication-addition operation with 4-bit input data precision and 4-bit weight data precision. A VPFU only contains one such PE, and in this case, the parallelism between PEs is the lowest, but the bit width of input data and weight data is the largest, which is 4 bits, Figure 7D The bit shift number of each operation block in Figure 8D is shown.

[0097] The concept of logical fusion is realized by the shift of basic blocks. Figures 8A to 8D When operating different precision data, the shift of each basic block in the PE in Figures 7A to 7D is shown. For example, Figure 5 detailedly shows that when using 4 basic blocks, operating input data precision of 2 bits and weight data precision of 2 bits, the shift of each basic block is 0, 1, 1, and 2, respectively; that is Figure 8B .

[0098] The array is composed of H rows and W columns of VPFUs (Variable-Precision Fusion Units). Figure 10 As shown, the input buffer is located at the edge of each variable precision fusion unit of each row of the array, and the VPFUs of each row share an input buffer to provide data to the VPFUs of the row simultaneously. Each VPFUs has a separate weight buffer. The output buffer is located at the bottom of the array, and the calculation results of all VPFUs of each column are accumulated as output and saved in the output buffer.

[0099] The calculation capability of the VPFU is related to the input data bit width N and the weight data bit width M. As known from the above, MN basic blocks constitute a PE, and each PE can perform Is group multiplication and addition operations. Therefore, in the 16 basic block VPFU, the maximum operation amount Imax is:

[0100]

[0101] The technical scheme of the present application is based on the Scale-sim software simulator to implement the data flow of convolution operation on the VPFU array. The convolution layers of the GooleNet and MobileNet models are selected to evaluate the performance of the data flow. The VPFU is also implemented in RTL, and its area and power consumption are evaluated by DC synthesis. Experiments show that compared with the advanced multiplication and addition operation unit Fusion Unit, the VPFU has advantages in area and power consumption.

[0102] The technical scheme of the present application will be described in combination with specific application fields:

[0103] Image processing technology is widely used in target recognition, image recognition and other application scenarios. In the field of computers, the essence of an image is a multi-channel two-dimensional pixel array, and each pixel point is encoded as a digital number in the RGB format. Essentially, image processing is processing and operation on these three-dimensional arrays. Convolutional neural networks are widely used in target recognition in image processing, and common convolutional neural networks include GoogleNet, MobileNet, etc. The VPFU array designed by the present application is used to speed up the inference operation of these neural network models.

[0104] In actual processing, the pixel points in the image are processed and binaryzed as input data of the convolution layer, and then convolution operation is performed with the weight data of the current convolution layer.

[0105] First, the precisions of the input data and the weight data used by the current convolution layer are determined, and then the fusion unit is configured, i.e., the shift bit number of each basic block is configured to realize precision matching. According to the number of variable precision fusion units, an array is formed, and is connected by an addition tree.

[0106] The input data and the weight data are placed into the input cache and the weight cache respectively, and finally the array is used to operate the current convolution layer to obtain output data. The output data is used as the input data of the next convolution layer for convolution operation. And so on to obtain the result of model inference. Compared with the prior art BitFusion, the variable precision fusion unit has smaller area and lower power consumption. The variable precision fusion array can realize a maximum acceleration of 2.27 times in the convolution layer, effectively improving the speed of picture recognition.

[0107] In the field of large language models, the application of the present application: for example, GPT-3, Wenxin Yanyan, Tongyiqian, etc. Large model applications, the VPFU array designed by the present application can accelerate the general matrix multiplication with the largest operation amount in large language model inference.

[0108] For example, for the quantization block of the 4bit and 2bit Transformer model in the Transformer model, the VPFU array proposed in the present application can effectively shorten the time of generating an answer by a large language model. Specifically, for the general matrix multiplication operation of the Transformer quantization block, first, the precisions (4 bits or 2 bits) of the input matrix and the weight matrix are determined, and the fusion unit is configured according to the precisions, i.e., the shift bit number of all basic blocks is set to realize precision matching.

[0109] According to the number of variable precision fusion units, an array is formed, and is connected by an addition tree. The input matrix is input into the input cache, and the weight matrix is input into the weight cache. Then the VPFU array is used to perform matrix multiplication operation to obtain an output matrix. All general matrix multiplications in the Transformer inference operation can be accelerated by using the VPFU array. Using the present application can improve the output answer time compared with the prior art.

[0110] Under the same configuration, the data stream of the pulsatile array designed by Scale-Sim (System Level Convolutional Neural Network Accelerator Simulator) is compared with the data stream of the VPFU array. Compared with the traditional pulsatile array, the VPFU array realizes an average acceleration of 7.74 times in the convolution layer of the GoogleNet model; realizes an average acceleration of 3.81 times in the convolution layer of the MobileNet model.

[0111] Compared with Bit Fusion, the RTL (Register Transfer Level) hardware implementation and synthesis of the two are performed using the DC tool (Design Compiler). Experimental data shows that the VPFU has a 33.46% reduction in area and a 28.30% reduction in power consumption compared with the Fusion Unit. Under the condition that the areas of the Bit Fusion array and the VPFU array are basically the same, the performance of the two is evaluated using scale-sim. In the operation scenario where the input data and the weight data are both 4-bit precision, the VPFU array exhibits a certain performance improvement in the partial convolution layers of the GoogleNet and MobileNet models. Experimental data shows that, compared with the Bit Fusion architecture, the VPFU array achieves an average speedup of 1.42 times and 1.48 times, respectively. When the operation precision is further reduced to 1 bit, the performance difference is more significant, and the VPFU array achieves an average speedup of 2.18 times (GoogleNet) and 2.27 times (MobileNet) in the same network layer.

[0112] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A design method of a variable precision computing unit applied to quantized neural network convolution layers, characterized in that, The method comprises the following steps: counting the number of bits 1 in the binary, performing a shift operation on the vector with precision 1, and determining the shift bit number of each operation basic block; configuring the shift bit number of all operation basic blocks in the variable precision fusion calculation unit arranged in a two-dimensional array according to the actual precision of the current convolution layer activation value and weight value; calculating the dimension and number of operation basic blocks according to the dimension and precision of the current vector to be calculated; determining the current fusion logic according to the dimension and precision of the convolution layer, fusing the operation basic blocks, and forming the processing units in the variable precision fusion calculation unit; wherein the shift bit numbers of the fused operation basic blocks are the same or different; arranging each operation basic block according to the physical array structure of the current hardware and the shift bit number of each operation basic block, connecting to form an array after arrangement, obtaining a variable precision fusion calculation unit, and setting a weight cache and an input cache outside the array which can provide operands for the operation basic blocks; each operation basic block, weight cache and input cache together form a variable precision fusion calculation unit; wherein the hardware unit fixed with the operation basic block can meet the shift of the operation basic block; The fusion calculation unit can output corresponding calculation results according to the precision of different vectors to be calculated; specifically, each column of fusion calculation units calculates in parallel and outputs different feature maps respectively, thereby completing the operation of the entire convolution layer and obtaining all output feature maps; wherein the processing units at the same position in different columns of fusion calculation units output different feature values at the same position of the same feature map during parallel calculation; After the current convolution layer operation is completed, the two-dimensional array is reconfigured according to the data precision of the next convolution layer, and then the next convolution layer is calculated using the newly configured two-dimensional array, and the operation of all convolution layers is completed in this way.

2. The design method of claim 1, wherein counting the number of bits 1 in the binary, performing a shift operation on the vector with precision 1, and determining the shift bit number of each operation basic block, comprising: cutting the vector by bit according to the precision of the vector to obtain a plurality of sub-vectors; performing a bitwise AND operation on the split sub-vectors based on the bit stream algorithm, and counting the number of bits 1 in the AND operation result vector; performing a shift operation on the vector with precision 1 according to the bit weight of the sub-vector, thereby determining the shift bit number of each operation basic block.

3. The method of claim 1, wherein, The method comprises the following steps: presetting the dimension of the operand vector, splitting the dimension of the current vector to be calculated according to the preset dimension to obtain a plurality of low-dimensional vectors independent of each other, thereby determining the dimension of the operation basic block; determining the number of operation basic blocks required according to the precision of the current vector to be calculated.

4. The method of designing according to claim 3, wherein, The method comprises the following steps: when the preset operand vector dimension is equal to the current vector dimension, no splitting is required, and the current vector is directly sliced according to the dimension to obtain the plurality of low-dimensional vectors independent of each other; When the preset operation number vector dimension is greater than the current vector dimension, the number of the operation number vectors is supplemented; When the preset operation number vector dimension is less than the current vector dimension, the current vector dimension is split according to the preset operation number vector dimension to obtain a plurality of low-dimensional vectors independent of each other.

5. The method of claim 1, wherein, The fusion calculation unit can output corresponding calculation results according to the accuracy of different to-be-calculated vectors, including: According to the accuracy, a plurality of operation basic blocks are respectively constituted into a plurality of processing units, the lower the accuracy, the fewer operation basic blocks included in a single processing unit, the more processing units in the current fusion calculation unit, and the higher the parallelism of the fusion calculation unit.

6. A method of using a variable precision computing unit for quantized neural network convolutional layers designed by the method of any one of claims 1-5, wherein, The plurality of variable-precision fusion calculation units in the two-dimensional array specifically include a plurality of variable-precision fusion calculation units arranged in rows and columns, each row of variable-precision fusion calculation units shares an input cache to realize that the input cache simultaneously provides data to a row of variable-precision fusion calculation units, and each variable-precision fusion calculation unit has a separate weight cache for accumulating the calculation results of all variable-precision fusion calculation units in each column as output results and saving in an output cache.

Images