A matrix-vector multiplication calculation method, calculation system and storage medium

Abstract

The present invention discloses a matrix-vector multiplication calculation method, a calculation system and a storage medium, in which the calculation process is distributed to multiple threads, each thread pre-reads the input data of vector A and the weight data of matrix B from a storage device for calculation; the weight data is stored and transmitted in the storage device in a 4-bit compressed manner; before calculation, the 4-bit compressed weight data is dequantized; during the dequantization process, the 4-bit compressed weight data is converted into a 16-bit floating point number form by a magic number-based logical operation, and is operated with a scaling factor and zero-point data to obtain the dequantized weight data. The method adopts an improved calculation logic, the thread allocation flexibility is improved, and it has a high calculation efficiency for the application scenario of vector and matrix multiplication; the weight data is stored and transmitted in the storage device in a 4-bit compressed manner; and the bandwidth required for transmitting weight data is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a matrix-vector multiplication calculation method, a calculation system and a storage medium. Background Art

[0002] Generative large language models have demonstrated superior performance in various language tasks such as dialogue question answering, text generation, and language translation. However, in the reasoning process of generative large language models, token / s (number of tokens per second) in the decoding stage is one of the important indicators for evaluating the reasoning performance of large models.

[0003] The reasoning process of the generative large language model includes two stages, namely the prefill stage and the decoding stage. In the prefill stage, after the input text is processed by the model, a token will be generated as the input of the decode stage. The subsequent decode stage will also continuously splice the keys and values ​​of the newly generated tokens after the existing KV cache. According to the timeline of decode_layer in the llama2-7b model, the calculation time of vector-matrix multiplication (gemv) accounts for as much as 60% of the total time.

[0004] At present, Huggingface's solution is to implement Gemv by calling torch.linear based on Torch's Transformers. The underlying interface is the Cublas computing interface, which is very mature in processing matrix multiplication and has the best performance in various matrix shapes. However, because it is necessary to be compatible with matrix multiplication of various shapes (m, n, k) and to use all computing resources, the computing model is constrained by design templates such as TensorCore and three-level pipeline asynchronous copy. Its mechanism is too complicated and is not very efficient for application scenarios of vector and matrix multiplication. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention generally provides a matrix-vector multiplication calculation method, a calculation system and a storage medium. The matrix-vector multiplication calculation method converts 4-bit compressed weight data into 16-bit floating point numbers by improving the calculation mode and adopting the ground-order number operation in the dequantization process, thereby achieving the purpose of improving the efficiency of matrix-vector multiplication calculation. To achieve the above purpose, the technical solution of the present invention includes:

[0006] A matrix-vector multiplication calculation method for a large language model distributes a calculation process to multiple threads, each thread pre-reads input data of a vector A and weight data of a matrix B from a storage device for calculation; the weight data is stored and transmitted in the storage device in a 4-bit compressed manner; before calculation, the 4-bit compressed weight data is dequantized; during the dequantization process, the 4-bit compressed weight data is converted into a 16-bit floating point number form by using a magic number-based logical operation method, and is operated with a scaling factor and zero point data to obtain the dequantized weight data.

[0007] A further improvement of the present invention is that each thread pre-reads 32 bytes of data from vector A and matrix B for calculation each time.

[0008] A further improvement of the present invention is that: the weight data is quantized in batches; during the quantization process, the compressed data of each weight data is calculated, and the scaling factor and the zero point of the corresponding batch are calculated, and the expression is:

[0009]

[0010]

[0011]

[0012] Where: n = 4; Max 4bit 、Min 4bit They are the upper and lower limits of the range represented by 4 bits; scaling is the scaling factor; zero is the zero point, that is, the value corresponding to the quantization of the original data.

[0013] x org Represents the original weight data, x 4bit Represents the compressed data corresponding to the original weight data, and round() is the rounding function.

[0014] A further improvement of the present invention is that: in the dequantization process, the expression for calculating the dequantized weight data is:

[0015] x 16bit =(x 4bit -zero)×scaling

[0016] Where: x 16bit is the weight data after dequantization.

[0017] A further improvement of the present invention is that: in the process of converting the 4-bit compressed weight data into a 16-bit floating point number using a magic number-based logical operation: the 4-bit compressed weight data are grouped in two and saved in the form of an 8-bit integer; the two 8-bit integers are spliced ​​into a 32-bit integer T1:

[0018] T1=OOABOOCD

[0019] Among them, OO represents eight bits of 0; A, B, C, and D are the weights of 4-bit compression respectively;

[0020] Perform an AND operation on T1 and the low-order template 0x000f000f to obtain T2:

[0021] T2=OOOBOOOD

[0022] OR T2 with the magic number 0x05400540 and shift the result left by four bits to get T3:

[0023] T3=54BO54DO

[0024] The high 16 bits 54BO of T3 are the 16-bit floating point number of weight B; the low 16 bits 54DO of T3 are the 16-bit floating point number of weight D;

[0025] Perform an AND operation on T1 and the high-order template 0x00f000f0 to obtain T4:

[0026] T4=OOAOOOCO

[0027] ORing T4 with the magic number 0x54005400 yields T5:

[0028] T3=54AO54CO

[0029] The high 16 bits 54AO of T5 are the 16-bit floating point form of weight A; the low 16 bits 54CO of T5 are the 16-bit floating point form of weight C.

[0030] The present invention also provides a computing system, the computing system comprising:

[0031] a memory for storing computer-executable instructions; and

[0032] A processor is used to execute the computer executable instructions to implement the above-mentioned matrix-vector multiplication calculation method.

[0033] A further improvement of the present invention is that the processor includes a graphics processing unit GPU, a neural network processing unit NPU and a field programmable gate array FPGA chip.

[0034] The present invention also provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, the above-mentioned matrix-vector multiplication calculation method can be implemented.

[0035] The technical solution provided by the present invention has the following technical effects:

[0036] 1. Improved computing logic is adopted, thread allocation flexibility is improved, and the computing efficiency is higher for vector and matrix multiplication application scenarios;

[0037] 2. The weight data is stored and transmitted in the storage device in a 4-bit compressed manner; the bandwidth required for transmitting the weight data is reduced, and the utilization rate of the video memory bandwidth is improved; the inference speed is improved, and the number of tokens generated per second increases;

[0038] 3. The 4-bit compressed weight data is converted into a 16-bit floating point number using logic operations based on magic numbers. This process only uses bitwise logic operations and shift operations, with a small amount of calculation. It can effectively improve the computational efficiency of vector-matrix multiplication operations, thereby improving the efficiency of the large language model inference stage and reducing the time required to output each word.

[0039] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Shown is a schematic diagram of the calculation logic of the matrix-vector multiplication in the present invention;

[0041] Figure 2 The figure shows the magic number in FP 16 (16-bit floating point number) form;

[0042] Figure 3 , Figure 4 Schematic diagrams of 4-bit compressed weight data converted to FP16 format. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0044] An embodiment of the present invention provides a matrix-vector multiplication calculation method for a large language model. The matrix-vector multiplication calculation method of this embodiment adopts an improved calculation logic: the calculation process is distributed to multiple threads, and each thread pre-reads the input data of vector A and the weight data of matrix B from a storage device for calculation.

[0045] like Figure 1 As shown, in this embodiment, the dimension of vector A is m*k, and the dimension of matrix B is k*n; where (m=1, k=1024, n=2048). Threads T0, T1, and T2 calculate the inner product of vector A and a column in matrix B respectively. During the calculation process, each thread pre-reads (prefetch) 32 bytes of float4 data from vector A and matrix B for calculation each time; for each column of matrix B, the number of iterations required by the corresponding thread is:

[0046]

[0047] By adopting the above simple calculation logic, the complex mechanism of tensorcore with double buffer in the cublas library is avoided; it has high calculation efficiency for application scenarios of vector and matrix multiplication.

[0048] In order to reduce the bandwidth required for transmitting weight data and improve the utilization rate of video memory bandwidth, the weight data is stored and transmitted in the storage device in a 4-bit compressed manner. The weight data is quantized in batches; during the quantization process, the compressed data of each weight data is calculated, and the scaling factor and zero point of the corresponding batch are calculated.

[0049] Compressing 128 data from a 16-bit half (half-precision floating point) type to 4 bits involves the calculation of scaling and zero. 4 bits of compression can represent 16 different values. This involves mapping the original data to one of these 16 possible values. Scaling is usually used to determine the proportion of each original value in the new representation range, while zero is used to determine the position of zero values ​​in the original data in the new representation. Assume that the range of the original data is [Min. org ,Max org ], where Min org Indicates the minimum value in the original data, Max org Indicates the maximum value in the original data; the new 4-bit range is [Min 4bit ,Max 4bit ], where Min 4bit Represents the minimum value in the quantized data, Max 4bit Indicates the maximum value in the quantized data.

[0050] 1.1 Calculation scaling:

[0051] 1.2 Calculate zero:

[0052] Here, n is the number of digits in the new representation, which is 4, so 2 n =16. 4 bits are unsigned, that is, they only represent non-negative numbers. Then, for each original data point x org , which can be mapped to a new 4-bit representation by the following formula: Such 4096×4096×2Byte weights will become after 4-bit quantization compression It becomes more than 1 / 4 of the original. We can calculate that based on the theoretical bandwidth of the RTX 3090 graphics card, 936.2GB / s, copying 4k×4k=16M of data requires After quantization, the copy time is reduced to 3.5ms, thereby significantly reducing the weight data transmission time.

[0053] The 4-bit quantized weight data needs to be dequantized after being read from the video memory into the computing unit. In the video memory, the weight data is stored and transmitted in 4-bit form, while the scaling factor and zero point are both in 16-bit floating point form (fp16), so the 4-bit quantized weight data needs to be converted into 16-bit floating point form in the computing unit.

[0054] The traditional method of this conversion process is to store two uint4_t (unsigned 4-bit integer) in an 8-bit integer (uint8_t), put the upper four bits in A through >>4, store the lower four bits in B through &0x0f, and then convert it to half for subsequent calculations. However, this conversion to half calculation consumes a lot of instructions and has low calculation efficiency.

[0055] In this embodiment, in the process of converting the 4-bit compressed weight data into a 16-bit floating point number (FP16) by using a magic number-based logical operation: the 4-bit compressed weight data is grouped in two and saved in the form of an 8-bit integer; the two 8-bit integers are concatenated into a 32-bit integer T1:

[0056] T1=OOABOOCD

[0057] Among them, OO represents eight bits of 0; A, B, C, and D are the weights of 4-bit compression respectively;

[0058] Perform an AND operation on T1 and the low-order template 0x000f000f to obtain T2:

[0059] T2=OOOBOOOD

[0060] OR T2 with the magic number 0x05400540 and shift the result left by four bits to get T3:

[0061] T3=54BO54DO

[0062] The high 16 bits 54BO of T3 are the 16-bit floating point number of weight B; the low 16 bits 54DO of T3 are the 16-bit floating point number of weight D;

[0063] Perform an AND operation on T1 and the high-order template 0x00f000f0 to obtain T4:

[0064] T4=OOAOOOCO

[0065] ORing T4 with the magic number 0x54005400 yields T5:

[0066] T3=54AO54CO

[0067] The high 16 bits 54AO of T5 are the 16-bit floating point form of weight A; the low 16 bits 54CO of T5 are the 16-bit floating point form of weight C.

[0068] The following explains the conversion process. In the 16-bit floating point format (FP16), the highest bit (16 bits) is the sign bit, bits 15 to 11 are the exponent bits exp, and bits 10 to 1 represent the fractional part fraction; the decimal representation of FP16 format is:

[0069]

[0070] like Figure 2 As shown in the figure, when the high 16 bits and low 16 bits of the magic number 0x54005400 used in this embodiment are in FP16 format, the decimal value is 64. If 1 is added to bits 8-5 in the figure, the value will be added by 1 on the basis of 64 until 64+16, and the data is continuous, that is, this position is the intersection of 4bit and fp16 types. No matter how these four numbers change, they can be reflected in the changes in 4bit and in fp16, and the effect is the same. In this way, the replacement from 4bit to fp16 can be completed.

[0071] like Figure 3 and Figure 4 As shown, they are schematic diagrams of 4-bit compressed weights converted to FP16. Figure 3 In , the weight of 4-bit compression is between 8 and 5 bits, the binary value is 1010, and the decimal value is 10; after conversion to FP16, Figure 3 The decimal representation of the FP16 shown is 74, which is equal to the compressed weight plus 64. Figure 4 Similarly, the compressed weight is 1110, which is 14 in decimal; after conversion to FP16 using the above method, the value is 78, which is also equal to the compressed weight plus 64.

[0072] The above-mentioned method of performing logical operations based on magic numbers can efficiently convert the compressed weights from 4-bit form to fp form. The conversion result is linear and uniform, and only bitwise logical operations and shift operations are used. The amount of calculation is small, which can effectively improve the computational efficiency of vector-matrix multiplication operations, thereby improving the efficiency of the inference stage of large language models and reducing the time required to output each word.

[0073] Table 1 compares the reasoning time (ms) of the technical solution of the present application and other solutions (columns 2-8), where the last column represents the acceleration ratio of the solution of the present application compared to the Torch solution.

[0074] An embodiment of the present invention further provides a computing system, the computing system comprising:

[0075] a memory for storing computer-executable instructions; and

[0076] A processor is used to execute the computer executable instructions to implement the above-mentioned matrix vector multiplication calculation method. The processor includes a graphics processing unit GPU, a neural network processing unit NPU and a field programmable gate array FPGA chip. The above-mentioned method can improve the overall computing efficiency of the computing system and reduce the operating cost of a large language model.

[0077] Table-1 Comparison of the time consumption of reasoning between the technical solution of this application and other solutions

[0078]

[0079] An embodiment of the present invention further provides a computer-readable storage medium storing executable instructions for executing the above-mentioned matrix-vector multiplication calculation method.

[0080] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A matrix-vector multiplication calculation method, characterized in that: The calculation process is distributed to multiple threads, each thread pre-reads the input data of the vector A and the weight data of the matrix B from the storage device for calculation; the weight data is stored and transmitted in the storage device in a 4-bit compressed manner; before the calculation, the 4-bit compressed weight data is dequantized; during the dequantization process, the 4-bit compressed weight data is converted into a 16-bit floating point number form by a logic operation based on a magic number, and is operated with a scaling factor and zero point data to obtain the dequantized weight data; In the process of converting the 4-bit compressed weight data into 16-bit floating point format by using logic operations based on magic numbers: the 4-bit compressed weight data are grouped in two and saved in the form of 8-bit integers; the two 8-bit integers are concatenated into a 32-bit integer T1: T1=OOABOOCD Among them, OO represents eight bits of 0; A, B, C, and D are the weights of 4-bit compression respectively; Perform an AND operation on T1 and the low-order template 0x000f000f to obtain T2: T2=OOOBOOOD OR T2 with the magic number 0x05400540 and shift the result left by four bits to get T3: T3=54BO54DO The high 16 bits 54BO of T3 are the 16-bit floating point number of weight B; the low 16 bits 54DO of T3 are the 16-bit floating point number of weight D; Perform an AND operation on T1 and the high-order template 0x00f000f0 to obtain T4: T4=OOAOOOCO ORing T4 with the magic number 0x54005400 yields T5: T3=54AO54CO The high 16 bits 54AO of T5 are the 16-bit floating point form of weight A; the low 16 bits 54CO of T5 are the 16-bit floating point form of weight C.

2. A matrix-vector multiplication calculation method according to claim 1, characterized in that: Each thread pre-reads 32 bytes of data from vector A and matrix B for calculation each time.

3. A matrix-vector multiplication calculation method according to claim 1, characterized in that: The weight data is quantized in batches; during the quantization process, the compressed data of each weight data is calculated, and the scaling factor and zero point of the corresponding batch are calculated, and the expression is: Where: n = 4; Max 4bit 、Min 4bit They are the upper and lower limits of the range represented by 4 bits; scaling is the scaling factor; zero is the zero point, that is, the value corresponding to the quantization of the original data; x org Represents the original weight data, x 4bit Represents the compressed data corresponding to the original weight data, round() is the rounding function; Max org Indicates the maximum value in the original data; Min org Indicates the minimum value in the original data.

4. A matrix-vector multiplication calculation method according to claim 3, characterized in that: In the dequantization process, the expression for calculating the dequantized weight data is x 16bit =(x 4bit -zero)×scaling Where: x 16bit is the weight data after dequantization.

5. A computing system, characterized in that: The computing system comprises: a memory for storing computer-executable instructions; and A processor, wherein the processor is used to execute the computer executable instructions to implement the matrix-vector multiplication calculation method according to any one of claims 1-4.

6. A computing system according to claim 5, characterized in that: The processor includes a graphics processing unit GPU, a neural network processing unit NPU and a field programmable gate array FPGA chip.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, it can implement the matrix-vector multiplication calculation method according to any one of claims 1 to 4.

Images