A large language model quantization method and system based on FP8

Abstract

The present invention proposes a large language model quantization method and system based on FP8, which belongs to the technical field of artificial intelligence technology, including: using a general FP8 quantization method to quantize the weights of a large model, representing the quantized data in binary format, converting the binary format data into INT8 data, using the INT8 data to load on hardware, and during calculation, converting the INT8 data into FP8 data type through binary representation, and fusing the above conversions to reduce the efficiency delay problem caused by the conversion. Through the above method, the FP8 quantization method can be used, which is suitable for hardware that does not support the FP8 data type, and can be used in large model tasks such as question answering and generation, and has high practical value and innovative value.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a large language model quantization method and system based on FP8. Background Art

[0002] Benefiting from the explosive growth of electronic data and hardware computing power, the number of parameters of large language models is getting larger and larger, and the model effect is getting better and better. Large language models have replaced small models dedicated to tasks and fields in most NLP tasks, and have been widely used, and have been quickly implemented and promoted in many fields. Due to the high computing power requirements of large models, many business scenarios cannot provide sufficiently powerful computing power support, resulting in reasoning efficiency becoming one of the bottlenecks for implementation. Large model compression and various reasoning technologies have been proposed to solve the problem of low reasoning efficiency, among which model quantization is basically a necessary option for optimization. However, in some real scenarios and tasks, even the mainstream INT8 quantization will lead to reduced model performance. FP8 data accuracy is a data accuracy recently proposed to accelerate model training and reasoning, and its effect in some model quantization tasks is better than INT8 data accuracy. However, currently only some GPUs of NVIDIA support FP8 data types, and most computing cards do not support FP8 data types, so FP8 quantization cannot be used. Summary of the invention

[0003] The present invention aims at solving the problems in the prior art, and the technical solution adopted in this application is as follows:

[0004] In a first aspect, the present invention provides a large language model quantization method based on FP8, comprising:

[0005] Get FP8 data, use the quantization method to quantize the weight of the FP32 type large model, and get the model weight and quantization parameter scale of the FP8 data type;

[0006] For each element of the model weight, the FP8 data of the element is represented as binary format, and the binary format data is converted into INT8 data;

[0007] Load the converted INT8 weight data into static random access memory;

[0008] For each element of the loaded INT8 weight data, it is represented in binary format and converted into FP8 type data;

[0009] Using the obtained quantization parameter scale, the FP8 type data is inversely quantized using the FP8 general inverse quantization method to obtain the FP32 data type weight;

[0010] The FP32 weight data obtained by inverse quantization and the FP32 input data are used to perform GEMM calculation to obtain the calculation result;

[0011] The input data is inferred through the large language model to obtain the output result of the large language model.

[0012] Furthermore, the quantization method adopts the FP8 data type quantization method.

[0013] Furthermore, the FP8 data type precision includes three representations: E5M2, E4M3, and E3M4.

[0014] Furthermore, the inverse quantization operation and the GEMM calculation are fused, and the fused GEMM calculation replaces the conventional GEMM calculation.

[0015] In a second aspect, the present invention provides a large language model quantization system based on FP8, comprising:

[0016] The first processing module is configured to: obtain FP8 data, quantize the weight of the FP32 type large model using the FP8 data type, and obtain the model weight and quantization parameter scale of the FP8 data type;

[0017] The second processing module is configured to: for each element of the model weight, represent the FP8 data of the element in binary format, and convert the binary format data into INT8 data;

[0018] A third processing module is configured to: load the converted INT8 weight data into SRAM;

[0019] The fourth processing module is configured to: represent each element of the loaded INT8 weight data in binary format, and convert the binary format data into FP8 type data;

[0020] A fifth processing module is configured to: use the obtained quantization parameter scale and a general inverse quantization method of FP8 to inverse quantize the FP8 type data to obtain a FP32 data type weight;

[0021] A sixth processing module is configured to: perform GEMM calculation using the FP32 type weight data obtained by inverse quantization and the FP32 type input data to obtain a calculation result;

[0022] The output module is configured to: infer the input data through the large language model to obtain the output result of the large language model.

[0023] In a third aspect, the present invention provides a computer-readable storage medium, the computer-readable storage medium comprising a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the FP8-based large language model quantization method described in the first aspect.

[0024] In a fourth aspect, the present invention provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the FP8-based large language model quantization method described in the first aspect.

[0025] Compared with the prior art, the advantages and positive effects of the present invention are:

[0026] The present invention provides a large language model quantization method and system based on FP8, which is used to solve the problem of using FP8 quantization on hardware that does not support FP8. First, the large model weights are quantized using a general FP8 quantization method, and then the quantized data is represented in binary format, and then the binary format data is converted into INT8 data, and the INT8 data is loaded on the hardware. During calculation, the INT8 data is converted into the FP8 data type through binary representation, and the above conversions are fused at the same time to reduce the efficiency delay problem caused by the conversion. In the above manner, the FP8 quantization method can be used, which is suitable for hardware that does not support the FP8 data type, and can be used in large model tasks such as question answering and generation, and has high practical value and innovation value. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 It is a flow chart of the large language model quantization method based on FP8 of the present invention;

[0029] Figure 2 This is a schematic diagram of the FP8 data of the present invention represented in binary format. DETAILED DESCRIPTION

[0030] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0031] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0032] Embodiment 1, as Figure 1-2 As shown, this embodiment provides a large language model quantization method based on FP8, including: quantizing large model weights using FP8 quantization method; converting model weight data from FP8 to INT8 data type; loading INT8 weight data into SRAM; converting model weight data from INT8 to FP8 data type; inverse quantization of FP8 type weight data to FP32 type; GEMM calculation; large language model inference. The specific steps are as follows:

[0033] First, obtain FP8 data and use the FP8 quantization method to quantize the large model weights. Use the quantization method to quantize the FP32 type large model weights. The quantization method can use the FP8 data type quantization method: scale the FP32 data by the scaling factor, that is, divide the FP32 data by the scaling factor to obtain the scaled data. The scaling factor can be obtained by the maximum value or percentile of the tensor data; round the scaled data to FP8 type data. After the model is quantized, the model weight of the FP8 data type is obtained, and the quantization parameter is the scaling factor scale. The FP8 data type precision includes three representations: E5M2, E4M3, and E3M4, which can be supported by the method of this embodiment.

[0034] Second, the model weight data is converted from FP8 to INT8 data type. For each element of the model weight, the FP8 data of the element is represented in binary format, and then the binary format data is converted to INT8 data. Assuming that the FP8 E5M2 data type element is 0.375, the schematic diagram of FP8 data represented in binary format is as follows: Figure 2 As shown in the figure, after conversion to INT8 type, the value is 54. In order to improve the inference efficiency, this step is performed offline and integrated with the previous quantization operation. That is, after quantizing each element of the weight data, the quantized FP8 data element is immediately converted to INT8 data, and the INT8 data type result is directly saved.

[0035] Third, the INT8 weight data is loaded into SRAM. The converted INT8 weight data is loaded into the static random access memory, which can be SRAM. Since it has been converted to the INT8 data type, the hardware can load it.

[0036] Fourth, the model weight data is converted from INT8 to FP8 data type. Each element of the loaded INT8 weight data is represented in binary format, and then the binary format data is converted to FP8 type data. For example, if the INT8 type data value is 54, the value after conversion to FP8 is 0.375.

[0037] Fifth, inverse quantize the FP8 type weight data to FP32 type. Use the quantization parameter scale obtained in the first step to inverse quantize the FP8 type data, that is, multiply the FP8 data by the scaling factor to obtain the FP32 data type weight.

[0038] Sixth, GEMM calculation. The FP32 type weight data obtained by inverse quantization is used to perform GEMM calculations with the input data. GEMM calculations refer to the multiplication of two matrices, which is the main calculation unit of the Transformer architecture commonly used in large models. In order to improve the efficiency of inference, the conversion operation in the fourth step, the inverse quantization operation in the fifth step, and the GEMM calculation are fused. Replace the conventional GEMM calculation with the fused GEMM calculation. Specifically, the fused GEMM calculation includes the following steps: assuming that the GEMM input matrices are weight and input, which represent weight data and input data respectively, the weight data is of INT8 type, and the input data is of FP32 type; convert each element of the weight data of INT8 data type to FP8 data type; inverse quantize each element of the converted weight data of FP8 data type to FP32 data type; use FP32 type weight and FP32 data type input to perform matrix multiplication to obtain the calculation result.

[0039] Seventh, large language model reasoning. Based on the above-mentioned improved module, the input data is accepted, and the improved model is used to reason about the data to obtain the output result of the large language model. The improved model refers to a model in which the weight data of the model is quantized FP8 type data. Reasoning is a special term for deep learning, which refers to the process of using a trained deep learning model to predict new data. The data type is different for different tasks. In the question-answering task, the input data is a question. In the retrieval task, the input data is a question and a retrieval document. The method of this embodiment can be applied to various tasks of the large language model.

[0040] Embodiment 2: This embodiment provides a large language model quantization system based on FP8, including:

[0041] The first processing module is configured to: obtain FP8 data, quantize the weight of the FP32 type large model using the FP8 data type, and obtain the model weight and quantization parameter scale of the FP8 data type;

[0042] The second processing module is configured to: for each element of the model weight, represent the FP8 data of the element in binary format, and convert the binary format data into INT8 data;

[0043] A third processing module is configured to: load the converted INT8 weight data into SRAM;

[0044] The fourth processing module is configured to: represent each element of the loaded INT8 weight data in binary format, and convert the binary format data into FP8 type data;

[0045] A fifth processing module is configured to: use the obtained quantization parameter scale and a general inverse quantization method of FP8 to inverse quantize the FP8 type data to obtain a FP32 data type weight;

[0046] A sixth processing module is configured to: perform GEMM calculation using the FP32 type weight data obtained by inverse quantization and the FP32 type input data to obtain a calculation result;

[0047] The output module is configured to: infer the input data through the large language model to obtain the output result of the large language model.

[0048] Embodiment 3, this embodiment provides a computer-readable storage medium, the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the FP8-based large language model quantization method described in Embodiment 1.

[0049] Embodiment 4. This embodiment provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the FP8-based large language model quantization method described in Embodiment 1.

[0050] The electronic device may include: a processor, a memory and a communication unit. These components communicate via one or more buses. Those skilled in the art will appreciate that the structure of the electronic device does not limit the embodiments of the present invention, and it may be a bus structure or a star structure, or a combination of certain components, or different component arrangements.

[0051] The communication unit is used to establish a communication channel so that the electronic device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0052] The processor is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0053] The memory is used to store the execution instructions of the processor. The memory can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0054] When the execution instructions in the memory are executed by the processor, the electronic device is enabled to execute part or all of the steps of Embodiment 1.

[0055] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A large language model quantization method based on FP8, characterized in that: include: Get FP8 data, use the quantization method to quantize the weight of the FP32 type large model, and get the model weight and quantization parameter scale of the FP8 data type; For each element of the model weight, the FP8 data of the element is represented as binary format, and the binary format data is converted into INT8 data; Load the converted INT8 weight data into static random access memory; For each element of the loaded INT8 weight data, it is represented in binary format and converted into FP8 type data; Using the obtained quantization parameter scale, the FP8 type data is inversely quantized using the FP8 general inverse quantization method to obtain the FP32 data type weight; The FP32 weight data obtained by inverse quantization and the FP32 input data are used to perform GEMM calculation to obtain the calculation result; The input data is inferred through the large language model to obtain the output result of the large language model; Accept input data, use the improved model to infer the data, and obtain the output result of the large language model. The improved model refers to a model whose weight data is quantized FP8 type data. Inference is a special term for deep learning, which refers to the process of using a trained deep learning model to predict new data. For different tasks, the data type is different. In question-answering tasks, the input data is questions, and in retrieval tasks, the input data is questions and retrieval documents.

2. The FP8-based large language model quantization method according to claim 1, characterized in that: The quantization method uses the FP8 data type quantization method.

3. The FP8-based large language model quantization method according to claim 1, characterized in that: The FP8 data type precision includes three representations: E5M2, E4M3, and E3M4.

4. The large language model quantization method based on FP8 according to claim 1, characterized in that: The inverse quantization operation and the GEMM calculation are fused, and the fused GEMM calculation replaces the conventional GEMM calculation.

5. A large language model quantization system based on FP8, characterized in that: include: The first processing module is configured to: obtain FP8 data, quantize the weight of the FP32 type large model using the FP8 data type, and obtain the model weight and quantization parameter scale of the FP8 data type; The second processing module is configured to: for each element of the model weight, represent the FP8 data of the element in binary format, and convert the binary format data into INT8 data; A third processing module is configured to: load the converted INT8 weight data into a static random access memory; The fourth processing module is configured to: represent each element of the loaded INT8 weight data in binary format, and convert the binary format data into FP8 type data; A fifth processing module is configured to: use the obtained quantization parameter scale and a general inverse quantization method of FP8 to inverse quantize the FP8 type data to obtain a FP32 data type weight; A sixth processing module is configured to: perform GEMM calculation using the FP32 type weight data obtained by inverse quantization and the FP32 type input data to obtain a calculation result; The output module is configured to: infer the input data through the large language model to obtain the output result of the large language model; Accept input data, use the improved model to infer the data, and obtain the output result of the large language model. The improved model refers to a model whose weight data is quantized FP8 type data. Inference is a special term for deep learning, which refers to the process of using a trained deep learning model to predict new data. For different tasks, the data type is different. In question-answering tasks, the input data is questions, and in retrieval tasks, the input data is questions and retrieval documents.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the FP8-based large language model quantization method described in any one of claims 1-4.

7. An electronic device, characterized in that: It includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the FP8-based large language model quantization method described in any one of claims 1 to 4.

Images