Design method and use method of variable precision calculation unit applied to quantization neural network convolutional layer

Abstract

The invention provides a design method and a use method of a variable precision calculation unit applied to a quantization neural network convolutional layer, and the method comprises the steps: carrying out the shift operation of a vector with the precision of 1, and determining the shift digit, dimension and number of an operation basic block; and arranging operation basic blocks and then connecting the operation basic blocks by using an adder tree to form an array to obtain a fusion calculation unit with variable precision. According to the actual precision of the activation value and the weight value of the current convolutional layer, a variable-precision fusion calculation unit array is configured, and all columns of fusion calculation units carry out parallel calculation; and after the operation of the current convolutional layer is completed, according to the data precision of the next convolutional layer, the two-dimensional array is reconfigured, and the operation of all the convolutional layers is completed by parity of reasoning. The problems that in the prior art, an operation unit is prone to causing waste of calculation resources in convolution operation smaller than 4 bits, different quantization bit widths cannot be flexibly and dynamically adopted to process data according to the actual situation of a convolution layer, and the resource utilization rate and the calculation efficiency cannot be improved are solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence hardware acceleration, and more particularly to a design method and usage method of a variable precision computing unit applied to the convolutional layer of a quantized neural network, especially the architecture of a dedicated hardware acceleration unit for convolutional layers. Background Technology

[0002] With the continuous development of neural network models, their depth and number of neurons have increased dramatically, with some models exceeding hundreds of billions of parameters. Trained models are typically deployed on edge devices (such as smartphones, embedded sensors, and smart cameras) to perform tasks like object recognition and speech recognition. However, these devices, limited by low-power design and limited storage resources, struggle to meet the demands of deploying massive neural network models. Specifically, the high computational demands of model inference and frequent memory accesses significantly restrict the performance of practical applications. Against this backdrop, the synergistic optimization of model compression techniques and hardware acceleration has become a research hotspot.

[0003] Quantization, as a core method of model compression, effectively reduces computational load and memory consumption while maintaining model inference accuracy by lowering the bit precision of weights and activation values. In recent years, quantization techniques have developed rapidly. Research shows that using extremely low-precision quantization (less than 4 bits) can achieve results comparable to 32-bit floating-point models. Furthermore, hybrid precision quantization strategies further mitigate the precision loss caused by uniformly applying extremely low precision quantization by dynamically allocating the quantization bit widths of different convolutional layers.

[0004] The following are existing technical solutions for optimizing inference hardware for neural network models: TPUs, based on a systolic array structure, support 8 / 16-bit wide computation, significantly improving the efficiency of large-scale matrix operations. However, they lack support for operations smaller than 4 bits, leading to wasted computational resources. DaDianNao, with its fixed-bit-width sampling design (DaDianNao: A Machine-Learning Supercomputer, in MICRO, 2014), achieves 256 multiplications per cycle, optimizing convolution computation performance. However, its fixed-bit-width design limits its support for mixed-precision quantization. Stripes uses variable precision for activation values ​​but a fixed 16-bit width for weight data (Stripes: Bit-serial deep neural network computing, in MICRO, 2016), which is unfriendly to inference operations in neural network models with extremely low-precision weight quantization. The Bit Fusion architecture employs a dynamic bit-width adjustment mechanism (Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Network," in ISCA, 2018), which reduces hardware power consumption while maintaining accuracy. However, its minimum granularity is 2 bits, which is insufficient for inference operations in binary neural networks, resulting in a waste of computing resources.

[0005] In summary, existing hardware has the following two problems for inference operations on extremely low-precision quantized neural network models: 1. The computation unit supports high data precision, which can easily lead to a waste of computing resources in low-precision convolution operations with less than 4 bits.

[0006] 2. Insufficient support for mixed-precision quantization operations.

[0007] Therefore, how to provide a mixed-precision quantization calculation method that can adapt to different computational needs of high precision (32 bits and above) and low precision (below 32 bits, such as 4 bits, 2 bits, 1 bit) has become an urgent problem to be solved. Summary of the Invention

[0008] This invention provides a design method and usage method for a variable precision computing unit applied to the convolutional layer of a quantized neural network. This addresses the problem that existing technologies, where computing units support high data precision, can easily lead to wasted computing resources in convolution operations with less than 4 bits. It also solves the problem that existing technologies lack sufficient support for mixed-precision quantization, making it impossible to flexibly and dynamically adopt different quantization bit widths to process data according to the actual situation of the convolutional layer, thereby improving resource utilization and computing efficiency.

[0009] To achieve the above objectives, the present invention provides a design method for a variable-precision computation unit applied to the convolutional layer of a quantized neural network, comprising: counting the number of 1 bits in a binary representation; performing a shift operation on a vector with a precision of 1; and determining the shift bits for different basic computational blocks. The dimension and number of basic computational blocks are calculated based on the dimension and precision of the vector to be computed. The basic computational blocks are arranged according to the physical array structure of the current hardware and the shift bits of each basic computational block. After arrangement, they are connected using an addition tree to form an array, resulting in a variable-precision fusion computation unit. This fusion computation unit can output corresponding computation results based on the precision of different vectors to be computed.

[0010] As a preferred embodiment of the above technical solution, preferably, the following steps are taken: counting the number of 1 bits in the binary representation, performing a shift operation on a vector with a precision of 1, and determining the shift bits for each of the different basic operational blocks, including: dividing the vector bitwise according to its precision to obtain several sub-vectors; performing a bitwise AND operation on the divided sub-vectors based on a bitstream algorithm, and counting the number of 1 bits in the vector result after the AND operation; and performing a shift operation on the vector with a precision of 1 according to the bit weights of the sub-vectors, thereby determining the shift bits for each of the different basic operational blocks.

[0011] As a preferred embodiment of the above technical solution, preferably, the dimension and number of computational basic blocks are calculated based on the dimension and precision of the current vector to be calculated, including: presetting the dimension of the operand vector, splitting the dimension of the current vector to be calculated according to the preset dimension to obtain several mutually independent low-dimensional vectors, thereby determining the dimension of the computational basic blocks; and determining the required number of computational basic blocks based on the precision of the current vector to be calculated.

[0012] As a preferred embodiment of the above technical solution, preferably, the dimension of the operand vector is preset, and the dimension of the current vector to be calculated is split according to the preset dimension to obtain several independent low-dimensional vectors, thereby determining the dimension of the basic computation block. This includes: when the preset operand vector dimension is equal to the current vector dimension, no splitting is required, and the current vector is directly sliced ​​according to the dimension to obtain the several independent low-dimensional vectors; when the preset operand vector dimension is greater than the current vector dimension, the number of operand vectors is supplemented; when the preset operand vector dimension is less than the current vector dimension, the current vector dimension is split according to the preset operand vector dimension to obtain several independent low-dimensional vectors.

[0013] As a preferred embodiment of the above technical solution, preferably, the basic operational blocks are arranged according to the physical array structure of the current hardware and the shift bits of each basic operational block, and then connected by an addition tree to form an array to obtain a variable precision fusion computing unit. It also includes: a weight cache and an input cache are set outside the array to provide operands for the basic operational blocks respectively, and each basic operational block, the weight cache and the input cache constitute a variable precision fusion computing unit; wherein, the hardware unit with the basic operational blocks is able to meet the shift of the basic operational blocks.

[0014] As a preferred embodiment of the above technical solution, the fusion computing unit can output corresponding calculation results according to the precision of different vectors to be calculated, including: according to the precision, different numbers of basic operation blocks are respectively formed into several processing units. The lower the precision, the fewer basic operation blocks are contained in a single processing unit, the more processing units are in the current fusion computing unit, and the higher the parallelism of the fusion computing unit.

[0015] This invention also provides a method for using a variable precision computing unit designed using the above-described method for use in a quantized neural network convolutional layer, comprising: configuring the shift bits of all basic operational blocks within a plurality of variable precision fusion computing units arranged in a two-dimensional array according to the actual precision of the activation and weight values ​​of the current convolutional layer; performing parallel computations in each column of fusion computing units to output different feature maps, thereby completing the computation of the entire convolutional layer and obtaining all output feature maps; after the current convolutional layer is completed, reconfiguring the two-dimensional array according to the data precision of the next convolutional layer, and then using the newly configured two-dimensional array to compute the next convolutional layer, and so on to complete the computation of all convolutional layers.

[0016] As a preferred embodiment of the above technical solution, preferably, based on the actual precision of the current convolutional layer activation value and weight value, the shift bits of all basic operational blocks in a plurality of variable precision fusion computing units arranged in a two-dimensional array are configured, and then the process includes: determining the current fusion logic based on the dimension and precision of the convolutional layer, and fusing the basic operational blocks according to the fusion logic to form a processing unit within the variable precision fusion computing unit; wherein the shift bits of the fused basic operational blocks are the same or different.

[0017] As a preferred embodiment of the above technical solution, the parallel computation of different column fusion computing units and the output of different feature maps respectively include: the processing units at the same position within the different column fusion computing units output different feature values ​​at the same position of the same feature map during the parallel computation process.

[0018] As a preferred embodiment of the above technical solution, the variable precision fusion computing units arranged in a two-dimensional array specifically include several variable precision fusion computing units arranged in rows and columns. Each row of variable precision fusion computing units shares an input buffer so that the input buffer can simultaneously provide data to a row of variable precision fusion computing units. Each variable precision fusion computing unit has its own independent weight buffer, which is used to accumulate the calculation results of all variable precision fusion computing units in each column and use them as the output result, which is stored in the output buffer.

[0019] This invention provides a design and usage method for variable-precision computation units applied to convolutional layers of quantized neural networks. The method involves counting the number of 1 bits in a binary representation, performing shift operations on vectors with a precision of 1, and determining the shift bits for different computational blocks. The dimension and number of computational blocks are calculated based on the dimension and precision of the vector to be computed. The computational blocks are arranged according to the physical array structure of the current hardware and the shift bits for each computational block. After arrangement, they are connected using an addition tree to form an array, resulting in variable-precision fusion computation units. These fusion computation units can output corresponding computation results based on the precision of different vectors to be computed. Based on the actual precision of the activation and weight values ​​of the current convolutional layer, the shift bits for all computational blocks within several variable-precision fusion computation units arranged in a two-dimensional array are determined. Each column of fusion computation units performs parallel computation, outputting different feature maps, thereby completing the computation of the entire convolutional layer and obtaining all output feature maps. After the current convolutional layer is completed, the two-dimensional array is reconfigured according to the data precision of the next convolutional layer. The newly configured two-dimensional array is then used to compute the next convolutional layer, and so on, completing the computation of all convolutional layers.

[0020] This invention addresses the problem that existing technologies, which support high data precision in their computational units, can easily lead to wasted computational resources in convolution operations with precision less than 4 bits. It also solves the problem that existing technologies lack sufficient support for mixed-precision quantization, making it impossible to flexibly and dynamically adopt different quantization bit widths to process data based on the actual situation of the convolutional layer, thereby improving resource utilization and computational efficiency. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating the design method of a variable-precision computation unit applied to the convolutional layer of a quantized neural network, provided by this invention.

[0023] Figure 2 A flowchart illustrating the usage method of the variable precision calculation unit obtained based on the design method provided by this invention.

[0024] Figure 3 This is a schematic diagram of vector slicing and packing in the design method of variable precision computing unit applied to the convolutional layer of a quantized neural network provided by the present invention.

[0025] Figure 4 This diagram illustrates the basic operational blocks for I-dimensional operands and the basic operational blocks for Is-dimensional operands in the design method of the variable precision computation unit applied to the convolutional layer of a quantized neural network provided by the present invention.

[0026] Figure 5 This is a schematic diagram illustrating the dot product operation process using basic operational blocks in the design method of the variable precision computation unit applied to the convolutional layer of a quantized neural network provided by the present invention.

[0027] Figure 6 This is a schematic diagram of a variable precision fusion computing unit having 16 computational basic blocks after arranging the computational basic blocks obtained by the design method according to the present invention.

[0028] Figure 7A This is a schematic diagram of a processing unit composed of different numbers of basic block logic in a variable-precision fusion computing unit. Figure 1 .

[0029] Figure 7B This is a schematic diagram of a processing unit composed of different numbers of basic block logic in a variable-precision fusion computing unit. Figure 2 .

[0030] Figure 7C This is a schematic diagram of a processing unit composed of different numbers of basic block logic in a variable-precision fusion computing unit. Figure 3 .

[0031] Figure 7D This is a schematic diagram of a processing unit composed of different numbers of basic block logic in a variable-precision fusion computing unit. Figure 4 .

[0032] Figure 8A This diagram illustrates the shifting of basic blocks in a variable-precision fusion computing unit, using different numbers of basic block logic components to form the processing unit under different logic fusion scenarios. Figure 1 In this context, the numbers in the basic block represent the number of shift bits.

[0033] Figure 8B This diagram illustrates the shifting of basic blocks in a variable-precision fusion computing unit, using different numbers of basic block logic components to form the processing unit under different logic fusion scenarios. Figure 2In this context, the numbers in the basic block represent the number of shift bits.

[0034] Figure 8C This diagram illustrates the shifting of basic blocks in a variable-precision fusion computing unit, using different numbers of basic block logic components to form the processing unit under different logic fusion scenarios. Figure 3 In this context, the numbers in the basic block represent the number of shift bits.

[0035] Figure 8D This diagram illustrates the shifting of basic blocks in a variable-precision fusion computing unit, using different numbers of basic block logic components to form the processing unit under different logic fusion scenarios. Figure 4 In this context, the numbers in the basic block represent the number of shift bits.

[0036] Figure 9 This is a schematic diagram of a convolutional layer.

[0037] Figure 10 The array structure for using the variable precision fusion unit in the method of using the variable precision calculation unit provided by the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] First, the design method of the variable-precision computation unit applied to the convolutional layer of a quantized neural network in this invention will be briefly described, such as... Figure 1 As shown: Step 101: Count the number of 1 bits in the binary representation, perform shift operations on the vector with a precision of 1, and determine the shift bits for each of the different basic operation blocks.

[0040] Specifically, based on the precision of the vector, it is bitwise divided to obtain several sub-vectors. Using the Bit-serial algorithm, a bitwise AND operation is performed on the split sub-vectors, and the number of 1 bits in the binary representation is counted, i.e., the number of 1 bits in the resulting vector. Based on the bit weights of these 1-bit precision sub-vectors, shift operations are performed on them to determine the shift bits for different basic operational blocks.

[0041] Step 102: Calculate the dimension and number of basic operational blocks based on the dimension and precision of the vector to be calculated.

[0042] The dimensions of the operand vector are preset, and the dimensions of the current vector to be calculated are split according to the preset dimensions to obtain several independent low-dimensional vectors, thereby determining the dimensions of the basic operation blocks; the number of basic operation blocks required is determined according to the precision of the current vector to be calculated.

[0043] The above splitting process specifically includes: When the preset operand vector dimension is equal to the current vector dimension, there is no need to split it. The current vector is directly sliced ​​according to the dimension to obtain the several mutually independent low-dimensional vectors.

[0044] When the preset operand vector dimension is greater than the current vector dimension, the number of operand vectors is increased.

[0045] When the preset operand vector dimension is smaller than the current vector dimension, the current vector dimension is split according to the preset operand vector dimension to obtain several independent low-dimensional vectors.

[0046] Step 103: Arrange the basic operational blocks according to the physical array structure of the current hardware and the shift bits of each basic operational block. After arrangement, connect them with an addition tree to form an array, and obtain a variable precision fusion computing unit.

[0047] Furthermore, a weight cache and an input cache are provided outside the array to provide operands for the basic operation blocks respectively. Each basic operation block, weight cache, and input cache constitute a variable precision fusion computing unit. Among them, the hardware unit with the basic operation blocks is able to meet the shifting of the basic operation blocks.

[0048] Step 104: The fusion computing unit can output corresponding calculation results according to the precision of different vectors to be calculated.

[0049] Depending on the precision of the vector to be computed, the fusion computing unit recombines different numbers of basic computational blocks to form several processing units. The lower the precision of the vector to be computed, the fewer basic computational blocks a single processing unit contains, resulting in more processing units within the current fusion computing unit and higher parallelism.

[0050] Next, a brief description will be given of the method for using the variable-precision computation unit applied to the convolutional layer of a quantized neural network, obtained according to the above design method, as follows: Figure 2 As shown: Step 201: Based on the actual precision of the current convolutional layer activation and weight values, configure the shift bits of all basic operational blocks in several variable precision fusion computing units arranged in a two-dimensional array.

[0051] The determination of the shift bits of the basic computational blocks is achieved through step 101; step 102 is executed based on the actual precision of the current convolutional layer activation value and weight value to determine the number of basic computational blocks; after obtaining the shift bits and number of basic computational blocks, step 103 is executed to solidify basic computational blocks with different shift bits onto the physical array structure of the current hardware, so that each basic computational block can meet the displacement requirements during the computation process. After arrangement, they are connected by an addition tree to form an array, resulting in a variable precision fusion computing unit.

[0052] Step 202: Determine the current fusion logic based on the dimension and precision of the convolutional layer, fuse the basic operation blocks to form a processing unit within the variable precision fusion calculation unit; wherein the number of shift bits of the fused basic operation blocks are the same or different.

[0053] This step aims to determine the fusion logic based on the dimension and precision of the convolutional layer and the output requirements, then group the basic operational blocks within the variable precision fusion computation unit to obtain the processing unit.

[0054] A variable precision fusion computing unit includes several processing units, and each processing unit includes at least one basic computational block.

[0055] Several variable-precision fusion computation units are arranged in a two-dimensional array to process data from the convolutional layer. These variable-precision fusion computation units in the two-dimensional array include those arranged in rows and columns. Each row of variable-precision fusion computation units shares an input buffer, allowing the input buffer to simultaneously provide data to a row of variable-precision fusion computation units. Each variable-precision fusion computation unit has its own independent weight buffer, used to accumulate the computation results of all variable-precision fusion computation units in each column, using this as the output result, which is stored in the output buffer.

[0056] Step 203: The fusion computing units in each column perform parallel calculations and output different feature maps respectively, thereby completing the operation of the entire convolutional layer and obtaining all output feature maps.

[0057] Specifically, in a two-dimensional array of variable precision fusion computing units, processing units at the same position within different columns of fusion computing units output different feature values ​​at the same position of the same feature map during parallel computing.

[0058] Step 204: After the current convolutional layer operation is completed, the two-dimensional array is reconfigured according to the data precision of the next convolutional layer. Then, the new two-dimensional array is used to calculate the next convolutional layer, and so on to complete the operation of all convolutional layers.

[0059] The design method of the variable precision computing unit applied to the convolutional layer of a quantized neural network provided by the present invention will now be described through specific embodiments.

[0060] Suppose that the vector to be processed is currently... with vector Perform dot product calculations; their dimension is I, and they are vectors. The precision is N, and the vector Let the precision be M, and let I=4, N=2, M=2; , The vector w and vector a are cut positionally and then recombined positionally to form sub-vectors, such as... Figure 3 As shown.

[0061] The popcount function is used to count the number of 1s in a binary number. .

[0062] It is understandable. and All are I-dimensional vectors with element precision of 1 bit. According to the Bit-serial algorithm, the number of basic operational blocks is calculated using Bit-serial, specifically with an operational precision of N. With a precision of M The dot product requires These are basic blocks. Figure 3 For example, the operation and The dot product of requires four basic operational blocks, and its operation can be represented as follows: Figure 4 .

[0063] Based on the specific numerical values ​​in the examples above, formula (1) is further calculated to obtain: Therefore, it can be seen that the calculation precision is N. With a precision of M The dot product requires four basic operational blocks. Furthermore, such as... Figure 5 As shown, four basic blocks are used to perform operations respectively. and A vector is a sliced ​​vector. Each operation is performed by bitwise AND, popcount, and shift operations, and finally the slices are summed to obtain the correct result.

[0064] It should be noted that the number of shifts for each operation is determined by the index of the slice vector. According to formula (2), the number of shifts for the four basic blocks are 0, 1, 1, and 2, respectively.

[0065] In actual convolution operations, the number of multiplication and addition operation groups required to calculate an output eigenvalue is often very large. Directly using an operand with a length of I as the input of the basic block cannot achieve the purpose of this invention.

[0066] For this reason, this invention adopts an operation of splitting the multiplication and addition operations: The dot product of the I-dimensional vector is split, specifically split into k + 1 groups of vector dot products. Except for the last group, the vector dimension in each group of vector dot products is Is, and the value of Is is much smaller than the value of I. In this way, each group of vector dot product operations can be respectively performed using the operation basic block, as Figure 6 shown.

[0067] Among them, k + 1 is the number of split groups for splitting the I-dimensional vector into Is-dimensional vectors, and k is a natural number.

[0068] The dot product of the I-dimensional vector is split into the accumulation of a number of Is-dimensional vector dot products for operation on the operation basic block with an operand bit width of Is, which can be divided into three splitting cases. In the following example cases, the current vector is an I-dimensional vector, the operand vector dimension is Is, and the low-dimensional vector has the same dimension as the operand vector, which is Is.

[0069] 1. When I = Is, there is no need to split. Directly according to the precision of the current vector, perform vector slicing bit by bit to obtain several Is-dimensional vectors with an element precision of 1 bit, and perform operations on the basic block.

[0070] 2. When I < Is, it should be supplemented to an Is-dimensional vector. Let's assume I = 3 and Is = 4. At this time, when using the bit-serial basic block for dot product operation, it needs to be supplemented. As shown in formula (4), where .

[0071] 3. When I > Is, it should be split. According to the preset operand vector Is dimension, split the current vector dimension I to obtain several independent low-dimensional vectors. For example, when I = 8 and Is = 4, then as shown in formula (5), , , and are all independent low-dimensional vectors after splitting.

[0072] Among them, MAC is the value of the vector dot product.

[0073] As shown above, using M×N basic operation blocks, the dot product operation of two Is-dimensional vectors with bit widths of M bits and N bits respectively can be implemented, i.e., Is-group multiplication and addition operation. Given a fixed number of basic operation blocks, through internal logic fusion, this set of basic operation blocks can implement variable-precision multiplication and addition operations.

[0074] Accordingly, the present invention provides a design method for a variable precision fusion computing unit. Further, as... Figures 7A to 8D As shown, this specific embodiment can realize the dot product calculation of vector w and vector a through 16 basic operation blocks, arranged in a 4×4 physical structure and connected by an addition tree. The basic operation array contains two buffers outside, namely the weight buffer and the input buffer. These two buffers provide operands for all basic blocks in the array composed of basic operation blocks. Specifically, the sharding mode of the input buffer and the weight buffer is determined according to the input precision and the weight precision, respectively, to obtain a vector with a precision of 1, which provides data for the variable precision fusion calculation unit.

[0075] The aforementioned array of basic computational blocks, together with the two caches, forms a Variable-Precision Fusion Unit (VPFU). Further, within the VPFU, a certain number of basic computational blocks are logically combined to form Processing Elements (PEs). The fewer basic blocks a PE contains, the lower the precision of the data it supports for computation; relatively speaking, the VPFU contains more PEs, resulting in higher parallelism. Conversely, the more basic blocks a PE contains, the higher the precision of the data it supports for computation, but the VPFU contains fewer PEs, resulting in lower parallelism.

[0076] The method of using the variable precision fusion computing unit obtained by the above design method, provided by the present invention, will be described based on the above specific embodiments.

[0077] General convolutional layers such as Figure 9 As shown, the input feature map is typically a three-dimensional structure (length × width × channels), while the convolutional kernel group is a four-dimensional structure (number of kernels × number of channels × length × width). Each convolutional kernel performs convolution calculations with the overlapping portion of the input feature map through a sliding operation, generating a feature value for the corresponding output feature map, thus producing one output feature map. Multiple independent convolutional kernels perform convolution operations with the input feature map respectively, ultimately producing multiple output feature maps (the number of which is the same as the number of convolutional kernels). Since the calculation process between different convolutional kernels has no data dependency, its operation process can be fully parallelized.

[0078] Based on this characteristic, variable-precision fusion computing units are combined into a two-dimensional array, such as... Figures 7A to 7D As shown.

[0079] In the process of combining two-dimensional arrays, the fusion logic is determined according to the dimension and precision of the convolutional layer and the output requirements. Then, the basic operation blocks in the variable precision fusion calculation unit are grouped to obtain the processing unit.

[0080] Figures 7A to 7D Four different ways of merging basic logic blocks are shown, each exhibiting different trade-offs between computational precision and parallelism: exist Figure 7A In this approach, a single basic block is used to construct the processing unit (PE). Each PE supports Is groups, performing multiply-add operations on 1-bit input data and 1-bit weights. VPFU integrates 16 PEs, achieving maximum parallelism by completing 16 Is group multiply-add operations per operation cycle. However, its input / weight bit width is the lowest, at 1 bit. VPFU ultimately outputs a partial sum. Figure 7A The number of displacements in each operation block is as follows Figure 8A As shown.

[0081] exist Figure 7B In this model, 2×2 basic blocks of logic are fused together to form a PE (Programmable Element), supporting multiply-add operations with 2-bit input data and 2-bit weights. The VPFU (Vertical Component Fusion Unit) integrates four such PEs, implementing four sets of multiply-add operations (Is) per computation cycle, ultimately producing a partial sum. Figure 7B The number of displacements in each operation block is as follows Figure 8B As shown.

[0082] exist Figure 7C In this model, 1×4 basic block logic fusions constitute a PE, which supports multiply-add operations on 2-bit input data and 2-bit weights. Similarly, in VPFU, there are 4 such PEs, which can compute a total of 4 sets of multiply-add operations and produce a partial sum output. Figure 7C The number of displacements in each operation block is as follows Figure 8C As shown.

[0083] at last, Figure 7D In this approach, 16 basic blocks are merged into a single PE (Preset Entity). This PE can perform multiply-add operations on one set of Is, with input data precision of 4 bits and weight data precision of 4 bits. A VPFU (Vertical Power Activation Unit) contains only one such PE. In this case, the parallelism between the PEs is the lowest, but the bit width of the input and weight data that can be processed is the largest, at 4 bits. Figure 7D The number of displacements in each operation block is as follows Figure 8D As shown.

[0084] The concept of logical fusion is achieved by shifting basic blocks. Figures 8A to 8D When displaying data with different precisions, the corresponding Figures 7A to 7DThe shifting of each basic block within the PE in the diagram. For example, Figure 5 The diagram shows in detail the shifting of each basic block when using four basic blocks, with input data precision of 2 bits and weight data precision of 2 bits. Specifically, the four basic blocks are shifted by 0, 1, 1, and 2 bits respectively. Figure 8B The situation.

[0085] The array consists of H rows and W columns of VPFUs (Variable-Precision Fusion Units). Figure 10 As shown, the input buffer is located at the edge of each variable-precision fusion computation unit in each row of the array. Each row of VPFUs shares an input buffer, allowing the input buffer to provide data to a row of VPFUs simultaneously. Each VPFU has its own independent weight buffer. The output buffer is located at the bottom of the array, where the computation results of all VPFUs in each column are accumulated and stored as the output.

[0086] The computational power of a VPFU is related to the input data bit width N and the weight data bit width M. As mentioned above, MN basic blocks constitute a PE, and each PE can perform Is groups of multiplication and addition operations. Therefore, in a VPFU with 16 basic blocks, its maximum computational complexity Imax is: This invention implements the data flow of convolution operations on a VPFU array using the Scale-sim software simulator. Performance of the data flow is evaluated using selected convolutional layers from GoogleNet and MobileNet models. The invention also implements VPFU using RTL and performs DC synthesis to evaluate its area and power consumption. Experiments show that VPFU has advantages in both area and power consumption compared to the more advanced multiply-accumulate fusion unit.

[0087] The technical solution of this invention will now be described in conjunction with specific application fields: Image processing technology is widely used in applications such as object recognition and image recognition. In the field of computer science, an image is essentially a multi-channel two-dimensional pixel array, with each pixel encoded as a number using formats such as RGB. Essentially, image processing involves processing and operating on these three-dimensional arrays. Convolutional neural networks (CNNs) are widely used in object recognition within image processing; common CNNs include GoogleNet and MobileNet. The VPFU array designed in this invention accelerates inference operations for these neural network models.

[0088] In actual processing, the pixels in the image are processed and converted into binary data, which is then used as input data for the convolutional layer. Subsequently, a convolution operation is performed with the weight data of the current convolutional layer.

[0089] First, the precision of the input and weight data used in the current convolutional layer is determined. Then, the fusion units are configured, specifically the shift bits for each basic block, to achieve precision matching. Based on the number of variable-precision fusion units, an array is formed and connected using an addition tree.

[0090] Input data and weight data are placed into input buffers and weight buffers respectively. Finally, this array is used to perform operations on the current convolutional layer to obtain output data. The output data is then used as input data for the next convolutional layer for convolution operations. This process continues to obtain the model's inference result. Compared to the existing BitFusion technology, the variable precision fusion unit has a smaller area and lower power consumption. The variable precision fusion array can achieve up to 2.27 times speedup in convolutional layers, effectively improving the speed of image recognition.

[0091] Applications of this application in the field of large language models: For example, in the application of large models such as GPT-3, Wenxin Yiyan, and Tongyi Qianwen, the VPFU array designed by this invention can accelerate the general matrix multiplication, which has the largest computational load in the reasoning of large language models.

[0092] For example, this invention accelerates computations for quantized blocks of 4-bit and 2-bit Transformer models. The proposed VPFU array can effectively shorten the time for generating answers in large language models. Specifically, for general matrix multiplication operations on Transformer quantized blocks, the precision (4-bit or 2-bit) of the input matrix and weight matrix is ​​first determined. The fusion unit is then configured according to the precision, i.e., the shift bits of all basic blocks are set to achieve precision matching.

[0093] An array is formed based on the number of variable precision fusion units (VPFUs) and connected using an addition tree. The input matrix is ​​input into an input buffer, and the weight matrix is ​​input into a weight buffer. Then, matrix multiplication is performed using the VPFU array to obtain the output matrix. All general matrix multiplication operations in Transformer inference can be accelerated using the VPFU array. Using this invention improves the time for outputting the answer compared to existing technologies.

[0094] Under the same configuration, the systolic array data stream designed by Scale-Sim (system-level convolutional neural network accelerator simulator) was compared with the VPFU array data stream. The VPFU array achieved an average speedup of 7.74 times in some convolutional layers of the GoogleNet (neural network) model and an average speedup of 3.81 times in some convolutional layers of the MobileNet (micro neural network) model compared with the traditional systolic array.

[0095] Compared to Bit Fusion, VPFU was implemented and synthesized using DC tools (Design Compiler electronic design automation software) at the RTL (Register Transfer Level). Experimental data shows that VPFU reduces the area by 33.46% and power consumption by 28.30% compared to Fusion Unit. With the Bit Fusion array and VPFU array having roughly the same area, their performance was evaluated using scale-sim. In a computation scenario where both input and weight data have 4-bit precision, the VPFU array showed a performance improvement in some convolutional layers of GoogleNet and MobileNet models. Experimental data shows that compared to the Bit Fusion architecture, the VPFU array achieved an average speedup of 1.42x and 1.48x, respectively. When the computation precision is further reduced to 1 bit, the performance difference becomes more significant, with the VPFU array achieving an average speedup of 2.18x (GoogleNet) and 2.27x (MobileNet) in the same network layers.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A design method for a variable-precision computation unit applied to the convolutional layer of a quantized neural network, characterized in that, include: Count the number of 1 bits in the binary representation, perform shift operations on a vector with a precision of 1, and determine the shift bits for different basic operation blocks; Calculate the dimension and number of basic operational blocks based on the dimension and precision of the vector to be computed; Based on the physical array structure of the current hardware and the shift bits of each basic operation block, the basic operation blocks are arranged and connected by an addition tree to form an array, resulting in a variable precision fusion computing unit. The fusion computing unit can output corresponding calculation results according to the precision of different vectors to be calculated.

2. The design method according to claim 1, characterized in that, Count the number of 1 bits in a binary representation, perform shift operations on a vector with a precision of 1, and determine the shift bits for different basic operation blocks, including: Based on the precision of the vector, the vector is cut bit by bit to obtain several sub-vectors; Based on the bitstream algorithm, a bitwise AND operation is performed on the split sub-vectors, and the number of 1 bits in the vector result after the AND operation is counted. Based on the position weights of the sub-vectors, a shift operation is performed on the vector with a precision of 1 to determine the shift number of each of the different basic operation blocks.

3. The design method according to claim 1, characterized in that, The calculation of the dimension and number of basic operational blocks based on the dimension and precision of the current vector to be calculated includes: The dimensions of the operand vector are preset, and the dimensions of the current vector to be calculated are split according to the preset dimensions to obtain several independent low-dimensional vectors, thereby determining the dimensions of the basic operational block. The number of the required basic operational blocks is determined based on the precision of the vector to be computed.

4. The design method according to claim 3, characterized in that, The process of pre-setting the dimension of the operand vector, splitting the dimension of the current vector to be calculated according to the pre-set dimension to obtain several independent low-dimensional vectors, thereby determining the dimension of the basic operational block, includes: When the preset operand vector dimension is equal to the current vector dimension, there is no need to split it. The current vector is directly sliced ​​according to the dimension to obtain the several mutually independent low-dimensional vectors. When the preset operand vector dimension is greater than the current vector dimension, the number of operand vectors is increased; When the preset operand vector dimension is smaller than the current vector dimension, the current vector dimension is split according to the preset operand vector dimension to obtain several independent low-dimensional vectors.

5. The design method according to claim 1, characterized in that, The step of arranging the basic operational blocks according to the physical array structure of the current hardware and the shift bits of each operational block, and then connecting them with an addition tree to form an array to obtain a variable-precision fusion computing unit, further includes: The array is equipped with weight cache and input cache that can provide operands for the basic operation blocks respectively. Each basic operation block, weight cache and input cache constitutes a variable precision fusion computing unit; among them, the hardware unit with the basic operation blocks can meet the shifting of the basic operation blocks.

6. The method according to claim 1, characterized in that, The fusion computing unit can output corresponding calculation results based on the precision of different vectors to be calculated, including: Based on the precision, different numbers of basic computational blocks are used to form several processing units. The lower the precision, the fewer basic computational blocks a single processing unit contains, the more processing units are in the current fusion computing unit, and the higher the parallelism of the fusion computing unit.

7. A method for using a variable-precision computational unit designed according to any one of claims 1-6 and applied to a quantized neural network convolutional layer, characterized in that, include: Based on the actual precision of the current convolutional layer activation and weight values, configure the shift bits of all basic operational blocks within several variable precision fusion computing units arranged in a two-dimensional array; Each column of fusion computing units performs parallel computations and outputs different feature maps, thereby completing the operation of the entire convolutional layer and obtaining all output feature maps. After the current convolutional layer operation is completed, the two-dimensional array is reconfigured according to the data precision of the next convolutional layer. Then, the newly configured two-dimensional array is used to calculate the next convolutional layer, and so on to complete the operation of all convolutional layers.

8. The method of use according to claim 7, characterized in that, Based on the actual precision of the current convolutional layer activation and weight values, the shift bits of all basic operational blocks within several variable-precision fusion computation units arranged in a two-dimensional array are configured, followed by: The current fusion logic is determined based on the dimension and precision of the convolutional layer, and the basic operational blocks are fused to form a processing unit within the variable precision fusion computation unit; wherein the number of shift bits of the fused basic operational blocks may be the same or different.

9. The method of use according to claim 7, characterized in that, The fusion computing units in each column perform parallel computations and output different feature maps, including: Processing units at the same position within different column fusion computing units output different feature values ​​at the same position of the same feature map during parallel computing.

10. The method of use according to claim 7, characterized in that, The aforementioned two-dimensional array of variable precision fusion computing units specifically includes several variable precision fusion computing units arranged in rows and columns. Each row of variable precision fusion computing units shares an input buffer, so that the input buffer can simultaneously provide data to a row of variable precision fusion computing units. Each variable precision fusion computing unit has its own independent weight buffer, which is used to accumulate the calculation results of all variable precision fusion computing units in each column and use them as the output result, which is stored in the output buffer.

Images