Memory access calculation method, calculation system and storage medium for 4-bit quantization

Abstract

The present invention discloses a memory access calculation method, a computing system and a storage medium for 4-bit quantization. The method is used for a tensor core computing unit. In the process of storing a weight matrix B: the 4-bit quantized weight matrix B is adjusted in data arrangement so that the thread cluster of the tensor core computing unit reads continuous 128 bytes of data each time; the weight matrix B is quantized by 4 bits in groups according to the adjusted scaling process so that two adjacent weight data can correspond to two adjacent quantization groups; the scaling coefficients obtained by group quantization form a scaling coefficient matrix; the scaling coefficient matrix is ​​adjusted in data arrangement so that the scaling coefficients read by the tensor core computing unit correspond to the weight data. The method utilizes a special weight and scaling coefficient matrix data arrangement to achieve 100% memory access merging, and redesigns the NF4 data group quantization form so that two adjacent data are quantized in two adjacent groups.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a memory access calculation method, a computing system and a storage medium for 4-bit quantization. Background Art

[0002] In large language model inference calculations, matrix multiplication is usually the operation with the highest proportion of inference time. NF4 (4-bit normalized floating point) weight quantization is one of the main means to improve the memory access performance of matrix multiplication. It can load and calculate more weight data per unit time, reducing the inference latency and video memory usage of large models. Traditional NF4 weights are grouped into a number of continuous data, and the scaling coefficient of each group is calculated and then grouped and quantized, thereby achieving the effect of improving quantization accuracy. At the same time, traditional NF4 weights are stored in n×m 1-dimensional continuous memory. When using the tensor core computing unit for calculation, it is usually necessary to first read the weight matrix block into the shared memory, and then each thread in the thread cluster reads the corresponding data from the shared memory, thereby realizing the matrix multiplication acceleration of the NF4 weight using the tensor core.

[0003] The traditional technique of computing NF4 weight matrices by relying on Tensor Cores has the following disadvantages:

[0004] 1. Low matrix reading efficiency: Since the Tensor Core hardware acceleration unit has a specific data distribution form, to read a 1-dimensional linearly stored matrix, the data in the global memory must be merged and read in units of 128 bytes and written to the shared memory of the thread block, and each thread then reads the corresponding data from the shared memory. This method requires the use of limited shared memory resources and increases the time consumption of shared memory reading and writing.

[0005] 2. Low computational efficiency: After reading the NF4 weights, the scaling factor of each group of weights also needs to be read, and the inverse quantized floating point value is obtained by multiplying the NF4 format data. When the scaling factor is half-precision, each thread needs to read 4 bytes of data to read the data with the highest efficiency, that is, two scaling factors. However, two adjacent data of the weights are a group, and only one scaling factor is needed. Therefore, the two scaling factors need to be split and calculated, which reduces the computational efficiency. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the present invention provides a memory access calculation method, a computing system and a storage medium for 4-bit quantization. The memory access calculation method for 4-bit quantization adjusts the NF4 data quantization and designs a special data reading form to improve the reading efficiency of the weight matrix and the scaling factor matrix, and improve the calculation efficiency of the scaling factor and the weight matrix. To achieve the above purpose, the technical solution of the present invention includes:

[0007] A memory access calculation method for 4-bit quantization, used in a tensor core computing unit, in the process of storing a weight matrix B:

[0008] The data arrangement of the 4-bit quantized weight matrix B is adjusted so that the thread cluster of the tensor core computing unit reads continuous 128 bytes of data each time;

[0009] According to the adjusted scaling process, the weight matrix B is quantized by 4 bits in groups, so that two adjacent weight data can correspond to two adjacent quantization groups; the scaling coefficients obtained by the group quantization form a scaling coefficient matrix;

[0010] The scaling factor matrix is ​​adjusted in data arrangement so that the scaling factors read by the tensor core computing unit correspond to the weight data.

[0011] A further improvement of the present invention is that The process of adjusting the data arrangement based on the weight matrix B includes:

[0012] Transpose weight matrix B: k×m→m×k

[0013] Decompose the weight matrix B into 6 dimensions:

[0014]

[0015] Among them: m is an integer multiple of 8, k is an integer multiple of 32; 41 and 42 are two dimensions with values ​​of 4 respectively;

[0016] Swap the 6 dimensions:

[0017]

[0018] A further improvement of the present invention is that the adjusted scaling process includes:

[0019]

[0020] Among them: weight matrix B NF4 is the 4-bit quantized weight matrix B; GroupSize is the size of the group quantization;

[0021] The resulting scaling factor matrix

[0022] A further improvement of the present invention is that the process of adjusting the data arrangement of the scaling factor matrix includes:

[0023] Combine and transpose the scaling factor matrix S:

[0024]

[0025] Decompose the scaling factor matrix S into 4 dimensions:

[0026]

[0027] To swap dimensions:

[0028]

[0029] Among them: 81 and 82 are two dimensions with values ​​of 8 respectively.

[0030] The present invention also provides a computing system, the computing system comprising:

[0031] a memory for storing computer-executable instructions; and

[0032] A processor, wherein the processor is used to execute the computer executable instructions to implement the above-mentioned memory access calculation method for 4-bit quantization.

[0033] A further improvement of the present invention is that the processor includes a graphics processing unit GPU, a neural network processing unit NPU and a field programmable gate array FPGA chip.

[0034] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, the above-mentioned memory access calculation method for 4-bit quantization can be implemented.

[0035] The technical solution provided by the present invention has the following technical effects:

[0036] 1. Utilize special weight and scaling factor matrix data arrangement to achieve 100% memory access merging, without the need for shared memory intervention or expensive thread block synchronization technology, thereby reducing the use of additional resources such as shared memory.

[0037] 2. Redesign the NF4 data grouping quantization format so that two adjacent data are quantized in two adjacent groups, which can fully utilize half2 (half-precision floating point number) data to accelerate calculation; enable the read 4-byte data to be directly and synchronously calculated without splitting, simplify calculation instructions, and improve calculation efficiency.

[0038] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The figure shows the distribution diagram of weight matrix data in the thread cluster. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0041] like Figure 1 As shown, an embodiment of the present invention provides a memory access calculation method for 4-bit quantization, which is used to achieve efficient data memory access and calculation in a tensor core computing unit. The smallest computing unit of a tensor core is a thread cluster, that is, 32 consecutive threads. In the m16n8k16 computing instruction, the data distribution of weight data in 32 threads is as follows Figure 1 As shown. In order to make the data read by 32 threads be continuously distributed in the global memory, the data arrangement of the weight matrix needs to be designed, and each thread needs to read 4 bytes of data to enable the thread cluster to read 128 bytes of continuous data. For 4-bit weights, each thread needs to read 8 4-bit data (32 bits = 4 bytes).

[0042] In order to achieve the above purpose, the method of this embodiment adjusts the data arrangement of the weight matrix. Assume that for matrix multiplication C=AB, where the input matrix Weight Matrix

[0043] The process of adjusting the original weight matrix arrangement B includes:

[0044] Transpose weight matrix B: k×m→m×k

[0045] Decompose the weight matrix B into 6 dimensions:

[0046]

[0047] Among them: m is an integer multiple of 8, k is an integer multiple of 32; 41 and 42 are two dimensions with values ​​of 4 respectively. Since the values ​​are the same, they are distinguished by subscripts;

[0048] Swap the 6 dimensions:

[0049]

[0050] Through the data arrangement conversion of the weight matrix B, each thread cluster of the tensor core computing unit reads 128 bytes of data, that is, 8×4×4×2 4-bit weights, and each thread reads 4 bytes of data, that is, 4×2 4-bit weight values. By reading the weight matrix in this way, the continuous 128 bytes of data read from the global memory can be stored in the register and stored in an array of length 4, so that four rounds of tensor core calculations can be performed.

[0051] The weight matrix B needs to be stored in the form of 4-bit group quantization, so the scaling factor matrix also needs to be processed accordingly.

[0052] Assume that the scaling process of the weight matrix is ​​B half =NF4Code(B NF )×S

[0053] The weight Scaling factor matrix NF4Code(·) is the conversion function from NF4 data to half data. At this time, since the size of the register is 4 bytes, if the calculation is performed in this way, then according to the data distribution of the tensor core, each thread will process two half-precision calculations, and these two half-precision values ​​will be in the same group, that is, only one half-precision scaling factor is needed, which will lead to a decrease in calculation utilization. It is necessary to first split the 4-byte scaling factor into two half-precision values, and only use one half-precision in this calculation, and the other half-precision scaling factor will be used in the next group of calculations. In order to increase the calculation efficiency of the scaling factor, the quantization process needs to be adjusted so that the two adjacent weight data can correspond to the two adjacent groups. Thereby, the two adjacent half-precision weights can be multiplied with the scaling factors of the two adjacent groups using the half2 (half-precision floating point) data structure.

[0054] The adjusted scaling process includes:

[0055]

[0056] Among them: weight matrix B NF4is the weight matrix B of 4-bit quantization; GroupSize is the size of group quantization; scaling factor matrix

[0057] Through the above adjustment of the data arrangement of the weight matrix, in order to enable the scaling factor read by each thread to be multiplied by the corresponding weight data read, the data arrangement of the scaling factor matrix needs to be adjusted: the process of adjusting the data arrangement of the scaling factor matrix includes:

[0058] Combine and transpose the scaling factor matrix S:

[0059]

[0060] Decompose the scaling factor matrix S into 4 dimensions:

[0061]

[0062] To swap dimensions:

[0063]

[0064] Among them: 81 and 82 are two dimensions with values ​​of 8 respectively.

[0065] By converting the data arrangement of the scaling factor matrix, each thread cluster also reads 128 bytes of data, that is, 8×8 data. However, the data read at this time is redundant, because each Tensor Core calculation only requires 16 half-precision values, that is, 8 half2 (half-precision floating point) data. Therefore, each read will read the scaling factors of 8 consecutive groups additionally, and the thread bundle shuffle strategy needs to be used to share the data of every 4 threads in the thread cluster, which only involves thread cluster synchronization, and the synchronization cost is extremely low.

[0066] An embodiment of the present invention further provides a computing system, the computing system comprising:

[0067] a memory for storing computer-executable instructions; and

[0068] A processor is used to execute the computer executable instructions to implement the above-mentioned memory access calculation method for 4-bit quantization. In this embodiment, the processor includes a graphics processing unit GPU, a neural network processing unit NPU and a field programmable gate array FPGA chip.

[0069] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, the above-mentioned memory access calculation method for 4-bit quantization can be implemented.

[0070] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A memory access calculation method for 4-bit quantization, used in a tensor core computing unit, characterized in that: In the process of storing the weight matrix B: The data arrangement of the 4-bit quantized weight matrix B is adjusted so that the thread cluster of the tensor core computing unit reads continuous 128 bytes of data each time; According to the adjusted scaling process, the weight matrix B is quantized by 4 bits in groups, so that two adjacent weight data can correspond to two adjacent quantization groups; The scaling coefficients obtained by group quantization form a scaling coefficient matrix; Adjust the data arrangement of the scaling factor matrix so that the scaling factors read by the tensor core computing unit correspond to the weight data; For the form The process of adjusting the data arrangement based on the weight matrix B includes: Transpose weight matrix B: k×m→m×k Decompose the weight matrix B into 6 dimensions: Among them: m is an integer multiple of 8, k is an integer multiple of 32; 41 and 42 are two dimensions with values ​​of 4 respectively; Swap the 6 dimensions:

2. The memory access calculation method for 4-bit quantization according to claim 1, characterized in that: The adjusted scaling process includes: Among them: weight matrix B NF4 is the 4-bit quantized weight matrix B; GroupSize is the size of the group quantization; The resulting scaling factor matrix 3. The memory access calculation method for 4-bit quantization according to claim 2, characterized in that: The process of adjusting the data arrangement of the scaling factor matrix includes: Combine and transpose the scaling factor matrix S: Decompose the scaling factor matrix S into 4 dimensions: To swap dimensions: Among them: 81 and 82 are two dimensions with values ​​of 8 respectively.

4. A computing system, characterized in that: The computing system comprises: a memory for storing computer-executable instructions; and A processor, wherein the processor is used to execute the computer executable instructions to implement the memory access calculation method for 4-bit quantization according to any one of claims 1-3.

5. A computing system according to claim 4, characterized in that: The processor includes a graphics processing unit GPU, a neural network processing unit NPU and a field programmable gate array FPGA chip.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, the memory access calculation method for 4-bit quantization according to any one of claims 1 to 3 can be implemented.

Images