Model quantization method, terminal device, and computer-readable storage medium

Abstract

This application discloses a model quantization method, a terminal device, and a computer-readable storage medium. The method includes: performing a first quantization operation on the initial weight data of a target network layer in the model to obtain first weight data, wherein the bit width of each element in the first weight data is the first bit width; performing a second quantization operation on the first weight data to obtain second weight data, wherein the bit width of each element in the second weight data is the second bit width, and the second bit width is smaller than the first bit width; storing the weight quantized data of the target network layer, wherein the weight quantized data includes the second weight data; and determining the output data of the target network layer based on the weight quantized data during the model's inference process. This achieves an effective balance between reducing storage space, improving inference efficiency, and improving inference accuracy.

Description

Technical Field

[0001] This application belongs to the field of model quantization technology, and in particular relates to a model quantization method, terminal device and computer-readable storage medium. Background Technology

[0002] Neural network models, especially large language models, have achieved remarkable success in the field of natural language processing (NLP). These models, through deep learning techniques, are able to quickly process and understand the complexity of human language, demonstrating superior capabilities in a variety of NLP tasks. However, these models typically require substantial computational resources and storage space, limiting their application in resource-constrained environments. Summary of the Invention

[0003] This application provides a model quantization method, apparatus, terminal device, and computer-readable storage medium. By performing secondary quantization on model weights, the accuracy and efficiency of overall model quantization are improved while saving storage space.

[0004] A first aspect of this application provides a model quantization method, comprising: performing a first quantization operation on initial weight data of a target network layer in a model to obtain first weight data, wherein the bit width of each element in the first weight data is the first bit width; performing a second quantization operation on the first weight data to obtain second weight data, wherein the bit width of each element in the second weight data is the second bit width, and the second bit width is smaller than the first bit width; storing the weight quantization data of the target network layer, wherein the weight quantization data includes the second weight data; and determining the output data of the target network layer based on the weight quantization data during the inference process of the model.

[0005] A second aspect of this application provides a model quantization apparatus, comprising: a first quantization module, configured to perform a first quantization operation on initial weight data of a target network layer in a model to obtain first weight data, wherein the bit width of each element in the first weight data is the first bit width; a second quantization module, configured to perform a second quantization operation on the first weight data to obtain second weight data, wherein the bit width of each element in the second weight data is the second bit width, and the second bit width is smaller than the first bit width; a storage module, configured to store the weight quantization data of the target network layer, wherein the weight quantization data includes the second weight data; and a determination module, configured to determine the output data of the target network layer based on the weight quantization data during the inference process of the model.

[0006] A third aspect of this application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the model quantization method described above.

[0007] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the model quantization method described above.

[0008] The model quantization method provided in the first aspect of this application first performs a first quantization operation on the initial weight data of the target network layer in the model to obtain integer first weight data. Then, a second quantization operation is performed on the first weight data to obtain second weight data with a further reduced bit width, and the quantized weight data, including the second weight data, is stored. Subsequently, during the model's inference process, the output data of the target network layer is quickly and accurately determined based on the stored quantized weight data. The above-mentioned progressive two-stage weight quantization method quantizes the initial weight data into second weight data with a second bit width step by step for storage, which can not only effectively save the storage space occupied by model weights and improve the efficiency of model inference, but also reduce the quantization error introduced by the one-time weight quantization scheme in the prior art. The first quantization operation, as an intermediate step, helps to maintain the accuracy of the weights. When performing the second quantization operation, it can be based on the already relatively accurate first weight data, which can better control the overall quantization error and improve the quantization accuracy, thereby further improving the accuracy of model inference. Thus, the model quantization method of this application can achieve an effective balance between reducing storage space, improving inference efficiency, and improving inference accuracy.

[0009] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the model quantization method in the existing technology;

[0012] Figure 2 This is a flowchart illustrating a model quantization method provided in one embodiment of this application;

[0013] Figure 3 This is a flowchart illustrating a model quantization method provided in another embodiment of this application;

[0014] Figure 4This is a schematic diagram of the structure of a model quantization device provided in one embodiment of this application;

[0015] Figure 5 This is a schematic diagram of the structure of a terminal device provided in one embodiment of this application. Detailed Implementation

[0016] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0017] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0018] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0019] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0020] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0021] The collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information involved in this application embodiment all comply with the provisions of relevant laws and regulations, have obtained the user's authorization or consent, have taken necessary confidentiality measures, and do not violate public order and good morals.

[0022] As mentioned earlier, neural network models, especially large language models, typically require significant computational resources and storage space, limiting their application in resource-constrained environments. Therefore, effectively compressing and optimizing large language models to reduce computational and storage requirements is crucial for their deployment and application.

[0023] Model quantization techniques reduce model storage space and computational complexity by decreasing the precision of model parameter representation. Currently, the most widely used model quantization schemes include: Weight 4-bit Activation 16-bit Quantization (W4A16), Activation-aware Weight Quantization (AWA), and SmoothQuantization. Figure 1 As shown, these model quantization schemes typically first smooth the weight values ​​of the original large model, then use standard quantization methods to quantize the model weights to the target precision, and finally use the quantized weight values ​​for model inference. While these quantization schemes can reduce the storage space of large model parameters and improve inference efficiency to some extent, they are all one-time weight quantization schemes. That is, they directly quantize the model weights from floating-point types to discrete integer types of the target bit width. For example, they directly quantize the original model weight values ​​from 16-bit floating-point types to 4-bit integer types (int4). Although these schemes reduce the storage space of weights, they are likely to introduce large quantization errors, leading to a decrease in the model's inference accuracy.

[0024] To at least partially address the aforementioned technical problems, embodiments of this application provide a model quantization method, apparatus, terminal device, and computer-readable storage medium. It can be applied to scenarios requiring model quantization in the field of deep learning, including but not limited to natural language processing (such as intelligent chatbots using ChatGPT), healthcare, autonomous driving, and finance. It is particularly suitable for memory-constrained hardware devices such as embedded devices, mobile devices, and IoT devices. The model quantization scheme of this application employs progressive two-level weight quantization to accurately quantize weight data into easily storable integer data levels for storage. During inference, the output data of the target network layer is quickly and accurately determined based on the stored weight quantization data. This enables neurons in the target network layer to perform accurate and efficient inference calculations on the output data, achieving the effect of saving storage space while improving inference efficiency and accuracy.

[0025] like Figure 2 As shown, the model quantization method provided in this application includes the following steps S210, S220, S230 and S240.

[0026] Step S210: Perform a first quantization operation on the initial weight data of the target network layer in the model to obtain first weight data, wherein the bit width of the elements in the first weight data is the first bit width.

[0027] The models in this application embodiment can be various types of neural network models. For example, various existing or future large language models. For simplicity, the ChatGPT model in the field of natural language processing will be used as an example for the following description.

[0028] For example, the target network layer can be any of the fully connected (Linear) layers in the ChatGPT model. In some special examples, the target network layer may also include other network layers of the ChatGPT model, such as convolutional layers. In this embodiment, the original weight data of each fully connected layer of the ChatGPT model can be read, and the original weight data can be smoothed to obtain initial weight data. This initial weight data can be a matrix composed of multiple smoothed initial weight values, which can be called the initial weight matrix. The size of the initial weight matrix of the fully connected layer depends on the number of input features (in_features) and the number of output features (out_features) of that layer. The size of the initial weight matrix can be represented as [out_features, in_features], where each row of the initial weight matrix corresponds to the weight of an output feature, and each column corresponds to the weight of an input feature. For example, if a fully connected layer has C1 input features and C0 output features, then the size of the initial weight matrix of that layer is C0xC1, and the initial weight matrix consists of C0xC1 initial weight values.

[0029] In this embodiment, the first width can be set according to actual needs. For example, the first width can be a positive integer less than or equal to 16. In a specific example, the first width can be 8, and the first quantization operation can perform an int8 quantization operation on the initial weight data. For example, quantizing each weight value in the initial weight matrix from the original 32-bit floating-point number (FP32) to an 8-bit signed integer can limit the numerical range of the original weight values ​​to an integer range of -128 to 127.

[0030] In this embodiment, various suitable weight quantization methods can be used to perform the first quantization operation on the initial weight data. For example, a channel-by-channel quantization method can be used to perform the first quantization operation, where each channel corresponds to each column of the initial weight matrix. Specifically, the quantization scaling factor (referred to as the initial scaling factor for ease of distinction) of each column (along the C1 dimension) of the initial weight matrix can be calculated first, resulting in C0 initial scaling factors. Then, based on the initial scaling factors, the initial weight matrix can be quantized into a first weight matrix (i.e., the first weight data) in int8 format. The C0 initial scaling factors can be used as weight quantization parameters in the first quantization operation, referred to as the first weight quantization parameters.

[0031] Step S220: Perform a second quantization operation on the first weight data to obtain the second weight data; wherein the bit width of the elements in the second weight data is the second bit width, which is smaller than the first bit width.

[0032] In this embodiment, the second bit width can be any positive integer smaller than the first bit width. In the aforementioned example where the first bit width is 8, the second bit width can be 4, and the second quantization operation can be performing a uint4 quantization operation on the first weight data. For example, converting each weight value in the first weight matrix from an 8-bit signed integer after the first quantization to a 4-bit unsigned integer further restricts the numerical range of the weight values ​​after the first quantization to the integer range of 0 to 15. The first weight matrix can be further quantized into a second weight matrix in uint4 format.

[0033] In this embodiment, various suitable weight quantization methods can be used to perform a second quantization operation on the first weight data. For example, a group-by-group quantization method can be used to perform the second quantization operation. For example, each column of weights can be divided into multiple groups, and the number of weights in each group, size1, can be set according to actual needs. A quantization scaling factor (referred to as the first scaling factor for easy distinction) can be calculated for each group of weights, resulting in (C1 / size1)*C0 first scaling factors. A zero point (referred to as the first zero point element for easy distinction) can also be calculated for each group of weights. Then, based on the first scaling factor and the first zero point element, the first weight matrix can be quantized into a first weight matrix (i.e., the first weight data) with an int8 data format. The first scaling factor and the first zero point element can be used as weight quantization parameters in the first quantization operation, referred to as the second weight quantization parameters.

[0034] Step S230: Store the weight quantization data of the target network layer, wherein the weight quantization data includes the second weight data.

[0035] In this embodiment, the weight quantization data includes at least the second weight data for each target network layer, such as at least the second weight matrix for each fully connected layer. In some examples, the weight quantization data also includes quantization parameters involved in the two quantization operations. For example, it includes the aforementioned first weight quantization parameters and / or second weight quantization parameters. Exemplarily, the weight quantization data can be directly stored in a preset location. Alternatively, some or all of the weight quantization data can be converted into a data format that is easy to store.

[0036] Step S240: During the inference process of the model, the output data of the target network layer is determined based on the weight quantization data.

[0037] In this embodiment, during model inference, the stored weight quantization data can be retrieved, and then the output data of the target network layer can be determined based on the weight quantization data and the initial activation data output by the previous network layer. Specifically, various suitable methods can be used to process the weight quantization data and the initial activation data output by the previous network layer. After processing, a matrix multiplication operation is performed, and then the output data of the target network layer can be further inferred based on the result of the matrix multiplication operation.

[0038] In a specific example, the initial activation data output from the previous network layer of the current fully connected layer can be quantized using a suitable quantization method to obtain quantized activation data. The quantized activation data can be in 8-bit, 12-bit, or 16-bit formats. Then, matrix multiplication can be performed using the quantized activation data and the weight quantization data, and the intermediate sums obtained from the matrix multiplication can be processed. For example, these intermediate sums can be dequantized into 16-bit floating-point data. The processed intermediate sums can be summed to determine the output data of the current fully connected layer. Exemplarily, before performing the matrix multiplication, the second weight data can be quantized again based on the weight quantization data. For example, based on the second weight quantization parameters in the second quantization operation, the uint4 type second weight matrix can be quickly and accurately weighted into an int8 type first weight matrix. For example, the quantization of the initial activation data can be int8 quantization. The subsequent matrix multiplication operation can be a regular int8 matrix multiplication operation. This not only improves inference efficiency but also ensures good inference and computational accuracy. Of course, in other examples, the initial activation data may not need to be quantized before the matrix multiplication operation.

[0039] As mentioned earlier, existing model quantization schemes are all one-time weight quantization schemes. While these schemes reduce the storage space of weights, they are likely to introduce large quantization errors, leading to a decrease in the model's inference accuracy. However, the model quantization method described in this application first performs a first quantization operation on the initial weight data of the target network layer in the model to obtain integer first weight data. Then, a second quantization operation is performed on the first weight data to obtain second weight data with a further reduced bit width, and the weight quantization data, including the second weight data, is stored. Subsequently, during model inference, the output data of the target network layer is quickly and accurately determined based on the stored weight quantization data. This progressive two-stage weight quantization method quantizes the initial weight data into second weight data with a second bit width step by step for storage, which not only effectively saves the storage space occupied by model weights and improves the efficiency of model inference, but also reduces the quantization error introduced by one-time quantization compared to existing one-time weight quantization schemes. The first quantization operation serves as an intermediate step, helping to maintain the accuracy of the weights. The second quantization operation can then be performed based on the already relatively accurate first weight data, allowing for better control of the overall quantization error and improving quantization accuracy, thereby further enhancing the accuracy of model inference. Therefore, the model quantization method of this application embodiment achieves an effective balance between reducing storage space, improving inference efficiency, and increasing inference accuracy.

[0040] In one implementation, step S230, storing the weight quantization data of the target network layer, may include the following steps:

[0041] Step S231: Divide each second weight element in the second weight data into multiple second weight element groups, wherein the number of second weight elements in each second weight element group is a second preset number.

[0042] Step S232: Concatenate the second weight elements in each second weight element group to obtain the concatenated element corresponding to each second weight element group.

[0043] Step S233: Store the splicing elements.

[0044] In this embodiment, the second preset number can be set according to actual needs and can be greater than or equal to 2. For example, the elements in the second weight data can be unsigned integers. For example, the second quantization operation can be a uint4 quantization operation, and each second weight element in the second weight data is of type uint4. The second preset number can be 2, 4, 8, etc. Taking a second preset number of 8 as an example, the second weight matrix (e.g., denoted as W) can be... q2In this algorithm, each column of elements is grouped, with eight adjacent elements forming a second weight element group. Then, the eight uint4 type weight values ​​from each second weight element group are concatenated together to obtain an int32 type weight value, called a concatenation element. The matrix formed by these concatenation elements can be denoted as W. q3 , store W q3 This is to facilitate its use in subsequent reasoning processes.

[0045] The model quantization method provided in this application embodiment may further include the following steps: Step S250, during the inference process of the model, the spliced ​​elements are decoded according to the second preset number and the second bit width to obtain the second weight data.

[0046] For example, you can first read the saved W from storage. q3 Each int32 integer weight value is then processed, and each int32 integer can be decomposed into eight individual uint4 type weight values. For example, each uint4 type weight value can be extracted using bitwise operations such as bit shifting and masking.

[0047] In the above scheme, by merging a second preset number of second weight elements into a single concatenation element, the storage space requirement can be effectively reduced, allowing the weight data to occupy fewer storage resources, facilitating model saving and loading. It also improves memory access efficiency, reduces the number of memory accesses, and increases cache utilization. Furthermore, it enhances compatibility with existing hardware and software frameworks. During the model loading phase, converting the weights back to their original format simplifies the weight processing. By reducing the steps in the storage and loading process, the risk of data conversion errors can be reduced, improving model reliability. This quantization scheme provides a flexible way to balance model accuracy and efficiency; weights can be quantized to different degrees as needed to adapt to different application scenarios and hardware limitations. Therefore, it can further improve model storage and computation efficiency, facilitating the deployment of large neural network models in resource-constrained environments.

[0048] In one implementation, step S240 determines the output data of the target network layer based on the weight quantization data, including the following steps:

[0049] Step S241: Perform a third quantization operation on the second weight data to obtain the third weight data, wherein the bit width of the elements in the third weight data is equal to the bit width of the first element.

[0050] Step S242: Determine the output data of the target network layer based on the third weight data and the initial activation data output by the first network layer, wherein the first network layer is the network layer preceding the target network layer.

[0051] The third quantization operation can be understood as a process of weighting the second weight data. That is, before inference, to facilitate storage and save storage space, the initial weight data is quantized into second weight data with a smaller data volume. During inference, to improve inference accuracy, the second weight data can be requantized into third weight data with a higher precision first-order width. For example, if the third quantization operation is int8 quantization, the uint4 type second weight matrix can be requantized into an int8 type third weight matrix. Exemplarily, the third weight matrix can be equal to the first weight matrix obtained from the first quantization operation. Then, based on the third weight matrix and the initial activation data output by the first network layer (e.g., called the initial activation matrix), the input data for each neuron in the current fully connected layer can be determined. For example, the initial activation matrix can be quantized into an int8 type quantized activation matrix. Then, an int8 matrix multiplication operation can be performed on the third weight matrix and the quantized activation matrix. The input data for each neuron in the current fully connected layer can then be determined based on the result of the matrix multiplication operation.

[0052] This converts the floating-point matrix multiplication in the original model to int8 type, reducing computational complexity and effectively improving the model's inference efficiency. This allows for high-precision model inference even in hardware scenarios with limited floating-point computing power and storage. Therefore, the above model quantization scheme further improves inference accuracy while effectively reducing model weight storage space and improving inference efficiency.

[0053] In one implementation, the weighted quantization data further includes second quantization parameter data. The second quantization parameter data may include various second weighted quantization parameters from the second quantization operation.

[0054] Step S220 performs a second quantization operation on the first weighted data, including the following steps:

[0055] Step S221: Divide each first weight element in the first weight data into multiple first weight element groups to obtain first grouping information, wherein the number of first weight elements in each first weight element group is a first preset number.

[0056] Step S222: Determine the second quantization parameter corresponding to the first weight element based on each first weight element, the first grouping information, and the second bit width, and obtain the second quantization parameter data, wherein the second quantization parameter corresponding to the first weight element in the same first weight element group is the same.

[0057] Step S223: Based on the second quantization parameter data, the first weight data is quantized using a group-by-group quantization method to obtain the second weight data;

[0058] In this embodiment, a per-group quantization method can be used to quantize the first weight data. The first preset number (e.g., denoted as size1) can be arbitrarily set according to actual needs. The specific implementation process of the second quantization operation is described below using size1 = 128 as an example. The first weight data is, for example, the first weight matrix W of type int8 obtained by quantizing the initial weight matrix of size C0xC1 mentioned above. q1 The first weight matrix W q1 The size remains C0xC1. Next, the first weight matrix of type int8 can be grouped along the C1 dimension by size1 = 128. Then, the second weight quantization parameters in the second quantization operation can be further determined based on the grouping. W q1 Each column of 128 first weight elements (elements in each first weight element group) can share a scaling factor (such as the first scaling factor) and a zero-point element (such as the first zero-point element). The second weight quantization parameter can include the first scaling factor and the first zero-point element corresponding to each group. For example, the first scaling factor Scale2 can be calculated using the following formula. i1,j1 and the first zero-point element Zeros i1,j1 .

[0059]

[0060] Where i1 and j1 are the index values ​​of each first weight element group in the C1 and C0 dimensions of the first weight matrix, respectively.

[0061] In this embodiment of the application, Scales2 can be composed of various Scale2 i1,j1 The matrix formed (e.g., called the first scaling factor matrix), Zeros can be composed of individual Zeros i1,j1 The matrix formed (for example, called the first zero-point element matrix).

[0062] As can be seen from the above formula, the sizes of Scales2 and Zeros are both... Scales2 is an int8 matrix and Zeros is a uint4 matrix.

[0063] After calculating Scales2 and Zeros, the following formula can be used to apply the first weight matrix W. q1 Performing a uint4 quantization operation yields a second weight matrix W of type uint4. q2 Each of the second weighted elements

[0064]

[0065] Where i2 and j2 are the index values ​​of each second weight element in the C1 and C0 dimensions of the second weight matrix, respectively.

[0066] Step S241 performs a third quantization operation on the second weight data, which may include: step S2411, quantizing the second weight data according to the second quantization data to obtain the third weight data, wherein the elements in the third weight data are the same as the corresponding elements in the first weight data.

[0067] For example, the second weight data may include the first scaling factor matrix Scales2 and the first zero-point element matrix Zeros. The third weight data is, for example, a third weight matrix W of type int8. q3 In this step, the third weight matrix W can be calculated using the following formula. q3 :

[0068] W q3 =(W q2 -Zeros)×Scales2

[0069] It is understandable that, in this example, since the first scaling factor matrix Scales2 and the first zero-point element matrix Zeros are used to weight the first weight matrix W... q1 Quantized into a second weight matrix W q2 The quantization parameters are obtained using the third weight matrix W obtained by the above formula. q3 With the first weight matrix W q1 Each element in the dataset corresponds to the same element. Therefore, the above scheme quickly and accurately achieves the weighting of the second-weighted data.

[0070] In one implementation, step S242 determines the output data of the target network layer based on the third weight data and the initial activation data output by the first network layer, including the following steps:

[0071] Step S2421: Perform the fourth quantization operation on the initial activation data to obtain quantized activation data, wherein the bit width of the elements in the quantized activation data can be equal to the bit width of the first element.

[0072] Step S2422: Perform matrix multiplication on the third weight data and the quantization activation data to obtain the first intermediate data consisting of multiple intermediate elements. Each intermediate element is obtained by multiplying a row of elements in the quantization activation data with a column of elements in the third weight data and then accumulating the product terms. The bit width of the intermediate element can be equal to 4 times the bit width of the first element.

[0073] Step S2423: Perform dequantization on the first intermediate data to obtain floating-point second intermediate data, wherein the bit width of the elements in the second intermediate data can be equal to twice the bit width of the first bit;

[0074] Step S2424: Determine the output data of the target network layer based on the second intermediate data.

[0075] Taking a width of 8 as an example, the fourth quantization operation can be an int8 quantization operation. The initial activation data is, for example, an initial activation matrix. In this step, various suitable quantization methods can be used to perform int8 quantization on the initial activation matrix. For example, a group-by-group quantization method can be used. Specifically, per-token quantization or per-tensor quantization can be used to convert the float16 type initial activation matrix into an int8 type quantized activation matrix X. 1 The data type of each activation element in the quantization activation matrix is ​​int8. Then, the third weight matrix W obtained in the above steps can be... q3 Quantization activation matrix X 1 Performing int8 matrix multiplication operations yields the intermediate sums of each matrix multiplication, resulting in an intermediate sum matrix (i.e., a matrix composed of the aforementioned intermediate elements), denoted as PSUM, which serves as the first intermediate data. Each intermediate element in the intermediate sum matrix PSUM is stored as 32 bits. Then, a dequantization operation can be performed on the intermediate sum matrix PSUM to convert the data type of each intermediate element to float16, ensuring that the values ​​of each element in the converted second intermediate data correspond to the data range of the initial weight elements in the initial weight data. For example, the dequantization operation on the intermediate sum matrix PSUM can be performed quickly based on the weight quantization parameters in the first quantization operation and the activation quantization parameters in the fourth quantization operation.

[0076] In the above scheme, the initial weight data of the target network layer undergoes progressive two-stage weight quantization, converting the weights from floating-point operations to integer calculations, thus saving storage space for the weight data. Then, during inference, the initial activation data is first quantized, and the second weight data is quickly requantized. Afterward, an efficient integer matrix multiplication operation is performed based on the quantized, higher-precision weight data and activation data. Furthermore, the intermediate results of the matrix multiplication operation are dequantized to obtain floating-point intermediate matrix multiplication data, ultimately yielding high-precision output data for the target network layer. This model quantization method not only reduces the storage consumption of large model deployments but also significantly reduces the computational load of matrix multiplication operations while improving quantization accuracy, thereby increasing inference efficiency and accuracy. This effectively enhances the throughput of large model deployments and expands the application of large models in hardware environments with limited floating-point computing resources.

[0077] In one implementation, the weight quantization data further includes first quantization parameter data, which includes the quantization parameters involved in the first quantization operation. Step S210 performs a first quantization operation on the initial weight data of the target network layer in the model, including the following steps:

[0078] Step S211: Based on each initial weight element in the initial weight data, the channel in which each initial weight element is located, and the first bit width, determine the first quantization parameter corresponding to the initial weight element to obtain the first quantization parameter data, wherein the first quantization parameter corresponding to the initial weight elements located in the same channel is the same.

[0079] Step S212: Based on the first quantization parameter data, the initial weight data is quantized using a channel-by-channel quantization method to obtain the first weight data.

[0080] In this embodiment of the application, the initial weight data can be quantized using a per-channel quantization method.

[0081] The initial weight data is an initial weight matrix of size C0xC1 (denoted as W). q0 Taking this as an example, each channel can correspond to each column of the initial weight matrix. First, the quantization scaling factor (referred to as the initial scaling factor, denoted as Scale1) of each column of the initial weight matrix (along the C1 dimension) can be calculated. i0,j0 For example, Scale1 can be calculated using the following formula. j0 :

[0082]

[0083] Here, j0 is the index value of each column of the initial weight matrix. The matrix composed of all the initial scaling factors can be called the initial scaling factor matrix, denoted as Scale1.

[0084] Then, the initial weight matrix W can be calculated using the following formula. q0 The first weight matrix W, quantized to int8 format q1 .

[0085]

[0086] In the above formula, W represents the first weight matrix. q1 In the first weight element, i0 and j0 are the index values ​​of each first weight element in the C1 and C0 dimensions of the first weight matrix, respectively. It can be seen that every C1 weight elements share an initial scaling factor, and there are a total of C0 initial scaling factors.

[0087] In one implementation, the weighted quantization data further includes fourth quantization parameter data, which includes the quantization parameters involved in the fourth quantization operation. Step S2421 performs the fourth quantization operation on the initial activation data, including the following steps:

[0088] Step S2421.1: Dynamically divide each initial activation element in the initial activation data into multiple activation element groups to obtain the second grouping information, wherein the initial activation elements in each activation element group are located in the same sheet or correspond to the same unit of language information.

[0089] Step S2421.2: Based on each initial activation element, the second group information, and the first bit width, determine the fourth quantization parameter corresponding to the initial activation element to obtain the fourth quantization parameter data, wherein the fourth quantization parameter corresponding to the initial activation element in the same activation element group is the same.

[0090] Step S2421.3: Based on the fourth quantization parameter data, the initial activation data is quantized using a group-by-group quantization method to obtain quantized activation data.

[0091] For example, the initial activation data can be the initial activation matrix X. f The initial activation matrix X of type float16 can be obtained by using per-token quantization or per-tensor quantization. f Convert the quantized activation matrix X to int8 type 1 Taking the per-token quantization method for quantizing the initial activation matrix as an example, the initial activation elements corresponding to each token can be divided into activation element groups. The number of elements in different activation element groups can be different. The fourth quantization parameter includes a dynamic scaling factor matrix composed of dynamic scaling factors corresponding to each activation element group. The following formula can be used to calculate it.

[0092]

[0093] Then, the initial activation matrix X can be calculated using the following formula. f The quantized activation matrix X is quantized to type int8. 1 :

[0094]

[0095] The above-mentioned methods for quantizing initial weight data and initial activation data have low computational cost and high quantization efficiency and accuracy.

[0096] In one implementation, step S2423 performs an inverse quantization operation on the first intermediate data, including the following steps:

[0097] Step S2423.1: Perform right shift, saturation and data type conversion operations on each intermediate element in the first intermediate data to obtain the third intermediate data, wherein the data type of the elements in the third intermediate data is floating point.

[0098] Step S2423.2: Determine the second intermediate data based on the third intermediate data.

[0099] Let's take converting a 32-bit portion of PSUM to float16 data as an example. We can perform a right shift, a clamping operation, and a type conversion operation on PSUM sequentially. First, the right shift operation converts the bit mode of PSUM from integer to floating-point format. The number of bits shifted depends on the exponent width of the target floating-point format. For example, for float16, the exponent width is 5 bits, so the right shift can be set to 13 bits. The clamping operation ensures that the value is within the range that float16 can represent. The representation range of float16 is approximately ±6.5e4 (i.e., ±65504). The clamping operation restricts the data to between -65504 and 65504. Afterward, the clamped integer value can be converted to float16 type.

[0100] The above scheme can reduce the bit width of intermediate data in matrix multiplication operations, thereby improving computational efficiency and reducing memory usage.

[0101] In one implementation, the weighted quantization data further includes first quantization parameter data and fourth quantization parameter data. The first quantization parameter data includes the quantization parameters involved in the first quantization operation, and the fourth quantization parameter data includes the quantization parameters involved in the fourth quantization operation. Step S2423.2 Determining the second intermediate data based on the third intermediate data includes: determining the second intermediate data based on the third intermediate data, the first quantization parameter data, and the fourth quantization parameter data.

[0102] For example, the third intermediate data can be a third intermediate matrix, denoted as Psum. f16 The second intermediate data can be a second intermediate matrix, denoted as Y. f The first quantization parameter data can be the aforementioned initial scaling factor matrix Scale1, and the fourth quantization parameter data can be the aforementioned dynamic scaling factor matrix. The third intermediate data of type float16 can be dequantized using the following formula to obtain the second intermediate matrix Y. f :

[0103]

[0104] The above scheme quickly and accurately determines the second intermediate data based on the third intermediate data, the first quantization parameter data, and the fourth quantization parameter data. The inverse quantization method for these intermediate data further improves the accuracy and efficiency of model inference.

[0105] Figure 3 This is a schematic flowchart illustrating a model quantization method provided in another embodiment of this application. Figure 3 As shown, the model quantization process mainly includes: the secondary quantization process of model weights and the quantization process of model inference operators.

[0106] First, the model weight quantization process mainly involves performing secondary quantization on the weights of the fully connected (linear) layers in the original model and packing them into int32. Specifically, after obtaining the original weight matrix of the fully connected layers, the original weights of the model can be smoothed to obtain the initial weight matrix W. q0 Then, the initial weight matrix can be quantized into a first weight matrix W in int8 format using the per-channel quantization method. q1 Specifically, the initial scaling factor for each channel can be calculated first, resulting in an initial scaling factor matrix Scale1. Then, the initial weight matrix is ​​quantized into a first weight matrix based on the initial scaling factor matrix Scale1. Next, the first weight matrix in int8 format is grouped along the ci dimension by group_size = 128, meaning every 128 weight elements share a first scaling factor and a first zero element. The first scaling factor and first zero element are calculated for each group, resulting in a first scaling factor matrix Scale2 and a first zero element matrix Zeros. Finally, based on the first scaling factor matrix and the first zero element matrix, the first weight matrix is ​​quantized into a second weight matrix W in int4 format using a per-group quantization method. q2 .

[0107] Subsequently, to facilitate model saving and loading, the calculated uint4 type second weight matrix was concatenated into int32 type integers in groups of 8 along the Co dimension and saved, resulting in W. q3 It simultaneously stores Scale1, Scale2, and Zeros. That is, the stored weight quantization data includes: Scale1, Scale2, Zeros, and W. q3 .

[0108] It's understandable that, assuming the initial weight matrix of the fully connected layer is K*M, taking group_size = 128 as an example, then W q3 The total memory usage of Scale1, Scale2, and Zeros is approximately

[0109]

[0110] The memory usage for the corresponding float16 floating-point model is M. f :

[0111] M f =2*M*K

[0112] Therefore, the memory usage ratio of the quantized model compared to the unquantized model is S1:

[0113]

[0114] Therefore, for the linear operator, the two-level weight quantization scheme of this application embodiment saves nearly 74.4% of the weight storage space compared to before quantization.

[0115] Furthermore, the memory usage of the scheme that directly stores the int8 quantization result is...

[0116]

[0117] Therefore, for the linear operator, compared to the scheme of directly performing one int8 weight quantization, the two-stage weight quantization scheme of this application embodiment still saves nearly 49% of weight storage space.

[0118] During model inference, the inference operator quantization process can be executed. To improve model efficiency, linear operators can be converted from floating-point operations to int8 calculations, involving four parts: dynamic quantization of initial activation data, requantization of weights, implementation of matrix multiplication operators, intermediate and PSUM dequantization. First, the original activation matrix of the linear layer is obtained. Then, a per-token method is used to count the dynamic scaling factor corresponding to each token, resulting in a dynamic scaling factor matrix. Next, the initial activation matrix can be quantized into an int8 format quantized activation matrix based on the dynamic scaling factor matrix. Finally, the W values ​​in the stored weight quantization data can be... q3The first weight matrix is ​​decoded into a second weight matrix. Then, it is weighted using the first scaling factor matrix Scale2 and the first zero-point element matrix Zeros to obtain a first weight matrix in int8 format. Next, an int8 matrix multiplication operation is performed on the first weight matrix and the quantization activation matrix to obtain the first intermediate data P0. P0 is retained as 32 bits. Then, P0 is converted to float16 type. For the 32-bit P0, it is first right-shifted and then saturated to between -65504 and 65504, and then cast to float16 type third intermediate data P1. Then, P1 is dequantized according to the initial scaling factor matrix and the dynamic scaling factor matrix to obtain the second intermediate data P2. Finally, the output data of the linear layer can be determined based on the second intermediate data P2.

[0119] The aforementioned model quantization scheme, by introducing a two-stage quantization process for weights, not only effectively reduces the storage footprint of weights but also converts float-type matrix multiplication in inference to int8 type, significantly improving the inference efficiency of large language models and successfully deploying large models in hardware environments with limited float computing power and storage. Simultaneously, the presence of the weight smoothing module suppresses the impact of activation outliers on quantization accuracy. This balances model inference efficiency and accuracy, improving the user experience for deploying large models.

[0120] This application also provides a model quantization device. For example... Figure 4 As shown, the model quantization device 400 includes:

[0121] The first quantization module 410 is used to perform a first quantization operation on the initial weight data of the target network layer in the model to obtain the first weight data, wherein the bit width of the element in the first weight data is the first bit width;

[0122] The second quantization module 420 is used to perform a second quantization operation on the first weight data to obtain the second weight data; wherein the bit width of the elements in the second weight data is the second bit width, which is smaller than the first bit width;

[0123] Storage module 430 is used to store the weight quantization data of the target network layer, wherein the weight quantization data includes the second weight data;

[0124] The determination module 440 is used to determine the output data of the target network layer based on the weight quantization data during the inference process of the model.

[0125] In one implementation, the determining module 440 includes:

[0126] The third quantization submodule is used to perform a third quantization operation on the second weight data to obtain the third weight data, wherein the bit width of the elements in the third weight data is equal to the bit width of the first element.

[0127] The first determining submodule is used to determine the output data of the target network layer based on the third weight data and the initial activation data output by the first network layer, wherein the first network layer is the network layer preceding the target network layer.

[0128] In one implementation, the weight quantization data further includes second quantization parameter data, and the second quantization module 420 includes:

[0129] The first grouping submodule is used to divide each first weight element in the first weight data into multiple first weight element groups to obtain first grouping information, wherein the number of first weight elements in each first weight element group is a first preset number.

[0130] The second determining submodule is used to determine the second quantization parameter corresponding to the first weight element based on each first weight element, the first grouping information, and the second bit width, and to obtain the second quantization parameter data, wherein the second quantization parameter corresponding to the first weight element in the same first weight element group is the same.

[0131] The first weight quantization submodule is used to quantize the first weight data according to the second quantization parameter data using a group-by-group quantization method to obtain the second weight data;

[0132] The third quantization submodule includes:

[0133] The second weight quantization unit is used to quantize the second weight data according to the second quantization parameter data to obtain the third weight data, wherein the elements in the third weight data are the same as the corresponding elements in the first weight data.

[0134] In one implementation, the first determining submodule includes:

[0135] The fourth quantization unit is used to perform the fourth quantization operation on the initial activation data to obtain quantized activation data, wherein the bit width of the elements in the quantized activation data can be equal to the bit width of the first element.

[0136] The matrix multiplication unit is used to perform matrix multiplication on the third weight data and the quantized activation data to obtain the first intermediate data consisting of multiple intermediate elements. Each intermediate element is obtained by multiplying a row of elements in the quantized activation data with a column of elements in the third weight data and then accumulating the product terms. The bit width of the intermediate element can be equal to 4 times the bit width of the first element.

[0137] The dequantization unit is used to perform a dequantization operation on the first intermediate data to obtain a floating-point second intermediate data, wherein the bit width of the elements in the second intermediate data can be equal to twice the bit width of the first bit;

[0138] The first determining unit is used to determine the output data of the target network layer based on the second intermediate data.

[0139] In one implementation, the dequantization unit includes:

[0140] The data conversion subunit is used to perform right shift, saturation and data type conversion operations on each intermediate element in the first intermediate data to obtain the third intermediate data, wherein the data type of the elements in the third intermediate data is floating point.

[0141] The first determining subunit is used to determine the second intermediate data based on the third intermediate data.

[0142] In one implementation, the weighted quantization data further includes first quantization parameter data and fourth quantization parameter data. The first quantization parameter data includes quantization parameters involved in the first quantization operation, and the fourth quantization parameter data includes quantization parameters involved in the fourth quantization operation. The first determining subunit is further configured to: determine second intermediate data based on the third intermediate data, the first quantization parameter data, and the fourth quantization parameter data.

[0143] In one implementation, the first quantization module 410 includes:

[0144] The first quantization parameter determination submodule is used to determine the first quantization parameter corresponding to the initial weight element based on each initial weight element in the initial weight data, the channel in which each initial weight element is located, and the first bit width, so as to obtain the first quantization parameter data. The first quantization parameter corresponding to the initial weight elements located in the same channel is the same.

[0145] The initial weight quantization submodule is used to quantize the initial weight data according to the first quantization parameter data using a channel-by-channel quantization method to obtain the first weight data;

[0146] and / or

[0147] The fourth quantization unit includes:

[0148] The second grouping subunit is used to dynamically divide each initial activation element in the initial activation data into multiple activation element groups to obtain the second grouping information, wherein the initial activation elements in each activation element group are located in the same sheet or correspond to the same unit of language information.

[0149] The fourth quantization parameter determination subunit is used to determine the fourth quantization parameter corresponding to the initial activation element based on each initial activation element, the second group information and the first bit width, and to obtain the fourth quantization parameter data. The fourth quantization parameter corresponding to the initial activation element in the same activation element group is the same.

[0150] The activation quantization subunit is used to quantize the initial activation data according to the fourth quantization parameter data, using a group-by-group quantization method, to obtain quantized activation data.

[0151] In one implementation, the elements in the second weight data can be unsigned integers, and the storage module 430 includes:

[0152] The third grouping unit is used to divide each second weight element in the second weight data into multiple second weight element groups, wherein the number of second weight elements in each second weight element group is a second preset number.

[0153] The splicing unit is used to splice the individual second weight elements in each second weight element group to obtain the splicing element corresponding to each second weight element group;

[0154] Storage unit, used to store spliced ​​elements;

[0155] The model quantization device 400 also includes:

[0156] The decoding module is used to decode the concatenated elements during the inference process of the model, based on the second preset number and the second bit width, to obtain the second weight data.

[0157] like Figure 5 As shown, this application embodiment also provides a terminal device 500, including: at least one processor 510 ( Figure 5 The diagram shows only one processor, memory 520, and computer program 530 stored in memory 520 and executable on at least one processor 510. When processor 510 executes computer program 530, it implements the steps of the above-described model quantization method.

[0158] Terminal devices may include, but are not limited to, processors and memory. Figure 5This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than illustrated, or combine certain components, or use different components. The processor may be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0159] It should be noted that the information interaction and execution process between the above-mentioned devices / modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0160] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The functional modules in the embodiments can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules can be implemented in hardware or as software functional modules. Furthermore, the specific names of the functional modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0161] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described model quantization method.

[0162] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps in the above-described model quantization method.

[0163] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A model quantization method, characterized in that, include: Perform a first quantization operation on the initial weight data of the target network layer in the model to obtain first weight data, wherein the bit width of the elements in the first weight data is the first bit width; Perform a second quantization operation on the first weighted data to obtain the second weighted data; The bit width of the element in the second weight data is the second bit width, which is smaller than the first bit width; Store the weight quantization data of the target network layer, wherein the weight quantization data includes the second weight data; During the inference process of the model, the output data of the target network layer is determined based on the weight quantization data.

2. The model quantization method as described in claim 1, characterized in that, The step of determining the output data of the target network layer based on the weighted quantization data includes: Perform a third quantization operation on the second weight data to obtain third weight data, wherein the bit width of the elements in the third weight data is equal to the first bit width; Based on the third weight data and the initial activation data output by the first network layer, the output data of the target network layer is determined, wherein the first network layer is the network layer preceding the target network layer.

3. The model quantization method as described in claim 2, characterized in that, The weighted quantization data further includes second quantization parameter data, and the second quantization operation on the first weighted data includes: The first weight element in the first weight data is divided into multiple first weight element groups to obtain the first grouping information, wherein the number of first weight elements in each first weight element group is a first preset number. Based on each first weight element, the first grouping information and the second bit width, the second quantization parameter corresponding to the first weight element is determined, and the second quantization parameter data is obtained, wherein the second quantization parameter corresponding to the first weight element in the same first weight element group is the same. Based on the second quantization parameter data, the first weight data is quantized using a group-by-group quantization method to obtain the second weight data; The third quantization operation performed on the second weighted data includes: The second weight data is quantized based on the second quantization parameter data to obtain the third weight data, wherein the elements in the third weight data are the same as the corresponding elements in the first weight data.

4. The model quantization method as described in claim 2, wherein determining the output data of the target network layer based on the third weight data and the initial activation data output by the first network layer includes: Perform a fourth quantization operation on the initial activation data to obtain quantized activation data; Perform a matrix multiplication operation on the third weight data and the quantization activation data to obtain first intermediate data composed of multiple intermediate elements, wherein each intermediate element is obtained by multiplying a row of elements in the quantization activation data and a column of elements in the third weight data and then accumulating the product terms. Perform a dequantization operation on the first intermediate data to obtain a floating-point second intermediate data; Based on the second intermediate data, the output data of the target network layer is determined.

5. The model quantization method as described in claim 4, characterized in that, The step of performing inverse quantization on the first intermediate data includes: Perform right shift, saturation, and data type conversion operations on each intermediate element in the first intermediate data to obtain the third intermediate data, wherein the data type of the elements in the third intermediate data is floating point. The second intermediate data is determined based on the third intermediate data.

6. The model quantization method as described in claim 5, characterized in that, The weighted quantization data further includes first quantization parameter data and fourth quantization parameter data. The first quantization parameter data includes the quantization parameters involved in the first quantization operation, and the fourth quantization parameter data includes the quantization parameters involved in the fourth quantization operation. Determining the second intermediate data based on the third intermediate data includes: The second intermediate data is determined based on the third intermediate data, the first quantization parameter data, and the fourth quantization parameter data.

7. The model quantization method as described in claim 6, characterized in that, The first quantization operation performed on the initial weight data of the target network layer in the model includes: Based on each initial weight element in the initial weight data, the channel in which each initial weight element is located, and the first bit width, the first quantization parameter corresponding to the initial weight element is determined, and the first quantization parameter data is obtained, wherein the first quantization parameter corresponding to the initial weight elements located in the same channel is the same. Based on the first quantization parameter data, the initial weight data is quantized using a channel-by-channel quantization method to obtain the first weight data; and / or The fourth quantization operation on the initial activation data includes: The initial activation elements in the initial activation data are dynamically divided into multiple activation element groups to obtain the second grouping information, wherein the initial activation elements in each activation element group are located in the same sheet or correspond to the same unit of language information. Based on each initial activation element, the second grouping information, and the first bit width, the fourth quantization parameter corresponding to the initial activation element is determined, and the fourth quantization parameter data is obtained, wherein the fourth quantization parameter corresponding to the initial activation elements in the same activation element group is the same. Based on the fourth quantization parameter data, the initial activation data is quantized using a group-by-group quantization method to obtain the quantized activation data.

8. The model quantization method according to any one of claims 1-7, characterized in that, The storage of the weight quantization data of the target network layer includes: Each second weight element in the second weight data is divided into multiple second weight element groups, wherein the number of second weight elements in each second weight element group is a second preset number; Concatenate the individual second weight elements in each second weight element group to obtain the concatenated element corresponding to each second weight element group; Store the splicing elements; The method further includes: During the inference process of the model, the splicing elements are decoded according to the second preset number and the second bit width to obtain the second weight data.

9. A terminal device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the model quantization method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the model quantization method as described in any one of claims 1 to 8.

Images