Model quantification method and device, storage medium and program product

Abstract

The invention relates to the technical field of computers, and discloses a model quantification method and device, a storage medium and a program product. The method comprises the following steps: in an off-line quantization stage, calculating to obtain a leveling coefficient according to an input matrix and a weight matrix of each linear layer operator, and calculating to obtain an input quantization parameter and a weight quantization parameter according to the leveling coefficient; performing 8-bit floating point quantization, grouping and 4-bit unsigned shaping asymmetric quantization on the weight matrix according to the weight quantization parameter to obtain a scaling coefficient, a zero reference value and a 4-bit unsigned shaping quantization weight; and in the online reasoning stage, performing online reasoning on the reasoning task based on the leveling coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value and the 4-bit unsigned shaping quantization weight to obtain a reasoning result. According to the scheme of the embodiment, the hardware memory occupied by the large language model can be reduced, the model precision loss can be reduced, and the model reasoning efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model quantization method, device, storage medium and program product. Background Art

[0002] The performance of large language models in many natural language processing tasks such as reasoning, translation, and conversation can match or even surpass that of humans. However, the high bandwidth and huge storage required for model reasoning have brought great challenges to actual engineering deployment. Among them, for linear layer operators, the matrix calculation volume is large and the weights occupy a large amount of storage, which brings huge time and computing costs to users and has become the main bottleneck for the deployment and reasoning of large language models.

[0003] By quantizing the weights of linear layer operators in the offline quantization stage, the deployment requirements of large language models can be effectively reduced. At present, the existing model quantization methods usually quantize the weights of linear layer operators into 8-bit floating point types, but they still occupy a large amount of hardware memory, so model deployment still has great limitations. Summary of the invention

[0004] The present invention provides a model quantization method, device, storage medium and program product, which can reduce the hardware memory occupied by a large language model, reduce the loss of model accuracy, and improve the model reasoning efficiency.

[0005] According to one aspect of the present invention, a model quantization method is provided, comprising:

[0006] In the offline quantization stage, a smoothing coefficient is calculated according to the input matrix and the weight matrix of each linear layer operator, and an input quantization parameter and a weight quantization parameter are calculated according to the smoothing coefficient;

[0007] Performing 8-bit floating-point quantization on the weight matrix according to the weight quantization parameter to obtain 8-bit floating-point quantization weights, and grouping the 8-bit floating-point quantization weights based on a preset grouping number to obtain grouped quantization weights;

[0008] Perform 4-bit unsigned integer asymmetric quantization on the grouped quantization weights to obtain a scaling factor, a zero reference value, and a 4-bit unsigned integer quantization weight;

[0009] In the online reasoning stage, the reasoning task is reasoned online based on the smoothing coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value and the 4-bit unsigned integer quantization weight to obtain a reasoning result.

[0010] According to another aspect of the present invention, there is provided a model quantization device, comprising:

[0011] A smoothing coefficient calculation module, used to calculate the smoothing coefficient according to the input matrix and weight matrix of each linear layer operator in the offline quantization stage, and calculate the input quantization parameter and the weight quantization parameter according to the smoothing coefficient;

[0012] A first weight quantization module is used to perform 8-bit floating-point quantization on the weight matrix according to the weight quantization parameter to obtain 8-bit floating-point quantization weights, and group the 8-bit floating-point quantization weights based on a preset grouping number to obtain grouped quantization weights;

[0013] A second weight quantization module is used to perform 4-bit unsigned integer asymmetric quantization on the grouped quantization weight to obtain a scaling factor, a zero reference value and a 4-bit unsigned integer quantization weight;

[0014] The online reasoning module is used to perform online reasoning on the reasoning task based on the smoothing coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value and the 4-bit unsigned integer quantization weight in the online reasoning stage to obtain the reasoning result.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor, and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model quantization method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a processor to implement the model quantization method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the model quantization method according to any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention is as follows: in the offline quantization stage, according to the input matrix and weight matrix of each linear layer operator, a smoothing coefficient is calculated, and according to the smoothing coefficient, an input quantization parameter and a weight quantization parameter are calculated; according to the weight quantization parameter, the weight matrix is ​​quantized by 8-bit floating point to obtain an 8-bit floating point quantization weight, and the 8-bit floating point quantization weight is grouped based on a preset grouping number to obtain the grouped quantization weight; the grouped quantization weight is asymmetric quantized by 4-bit unsigned integer to obtain a scaling coefficient, a zero reference value and a 4-bit unsigned integer quantization weight; in the online inference stage, the inference task is online inferenced based on the smoothing coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value and the 4-bit unsigned integer quantization weight to obtain an inference result; by adjusting the input matrix and the weight matrix by using the smoothing coefficient, and sequentially performing 8-bit floating point quantization and 4-bit unsigned integer asymmetric quantization on the weight matrix, the hardware memory occupied by the large language model can be reduced, the model accuracy loss can be reduced, and the model inference efficiency can be improved.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a model quantization method provided according to Embodiment 1 of the present invention;

[0025] Figure 2 This is an offline quantization flow chart provided according to the first embodiment of the present invention;

[0026] Figure 3 is an online reasoning flow chart provided according to the first embodiment of the present invention;

[0027] Figure 4 is a structural schematic diagram of a model quantization device provided according to Embodiment 2 of the present invention;

[0028] Figure 5 It is a structural schematic diagram of an electronic device for implementing the model quantization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 A flowchart of a model quantization method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of quantizing a large language model. The method can be executed by a model quantization device. The model quantization device can be implemented in the form of hardware and / or software. Typically, the model quantization device can be configured in an electronic device, such as a computer device or a server. Figure 1 As shown, the method includes:

[0033] S110 . In the offline quantization stage, a smoothing coefficient is calculated according to an input matrix and a weight matrix of each linear layer operator, and an input quantization parameter and a weight quantization parameter are calculated according to the smoothing coefficient.

[0034] Among them, the quantization of a large language model can include two stages: offline quantization and online inference. In the offline quantization stage, the corresponding smoothing coefficient can be determined according to the dynamic range of the input matrix and weight matrix of each linear layer operator in the specified dimension, and the input matrix and weight matrix can be smoothed based on the smoothing coefficient to obtain the input quantization parameter and weight quantization parameter corresponding to each linear layer operator.

[0035] Among them, the input quantization parameter can be a parameter vector required for 8-bit floating point (FP8) quantization of the input matrix; the weight quantization parameter can be a parameter vector required for 8-bit floating point quantization of the weight matrix.

[0036] Optionally, the smoothing coefficients are calculated based on the input matrix and weight matrix of each linear layer operator, which may include:

[0037] Obtaining the maximum value of the input matrix in each input channel and the maximum value of the weight matrix in each input channel;

[0038] The smoothing coefficient is calculated according to the maximum value of the input matrix in each input channel and the maximum value of the weight matrix in each input channel.

[0039] In a specific example, we can use the formula According to the input matrix X and the weight matrix W, the smoothing coefficient θ is calculated, where the dimensions of the input matrix X and the weight matrix W are [N,C in ] and [C in ,C out ], X i and W i Respectively represent the i-th input channel C in The input vector and weight vector of dimension θ can be [1,C in ].

[0040] Optionally, calculating the input quantization parameter and the weight quantization parameter according to the smoothing coefficient may include:

[0041] Dividing the input matrix by the smoothing coefficient to obtain a smoothing input matrix, and calculating the input quantization parameter according to the smoothing input matrix;

[0042] The leveling coefficient is multiplied by the weight matrix to obtain a leveling weight matrix, and the weight quantization parameter is calculated based on the leveling weight matrix.

[0043] In a specific example, the formula Calculate the input quantization parameter The flattened weight matrix W′ can be calculated according to the formula W′=θ×W; then, the flattened weight matrix can be quantized by FP8 channel by channel, and W′ can be taken out at the i-th input channel C in Data W i ′, and based on the formula Calculate the weight quantization parameters of each input channel, and then obtain the complete weight quantization parameters

[0044] S120. Perform 8-bit floating-point quantization on the weight matrix according to the weight quantization parameter to obtain 8-bit floating-point quantization weights, and group the 8-bit floating-point quantization weights based on a preset grouping number to obtain grouped quantization weights.

[0045] In this embodiment, the flattened weight matrix of each input channel can be divided by the corresponding weight quantization parameter, and the calculation result can be FP8 quantized to obtain the FP8 quantized weight corresponding to each input channel. After that, by traversing all input channels, the complete FP8 quantized weight can be obtained. Furthermore, the FP8 quantized weight can be grouped on the input channel dimension using a preset number of groups to obtain the grouped quantized weight. For example, the dimension of the FP8 quantized weight is [C in ,C out ], the preset number of groups is g, then the dimension of the group quantization weight is adjusted to [C in ×g,C out / g].

[0046] In this embodiment, by grouping the FP8 quantization weights to achieve dimensionality adjustment, business accuracy can be improved, the memory occupied by the large language model can be further reduced, and computing efficiency can be improved.

[0047] Optionally, performing 8-bit floating-point quantization on the weight matrix according to the weight quantization parameter to obtain an 8-bit floating-point quantized weight may include:

[0048] Through the hardware data type conversion instruction, 8-bit floating-point quantization is performed channel by channel according to the flattened weight matrix and the weight quantization parameter to obtain an 8-bit floating-point quantization weight.

[0049] Specifically, you can use the hardware data type conversion instruction cast in the E4M3 (E stands for exponent and M stands for mantissa) data format. fp16_to_fp8 Perform FP8 quantization on the flattened weight matrix channel by channel to obtain FP8 quantized weights. Typically, this can be based on the formula Calculate the FP8 quantization weight of the i-th input channel.

[0050] S130 , performing 4-bit unsigned integer asymmetric quantization on the grouped quantization weights to obtain a scaling factor, a zero reference value, and a 4-bit unsigned integer quantization weight.

[0051] In this embodiment, a second stage of uint4 asymmetric quantization can be performed for the grouped quantization weights. To minimize the loss of precision, during the second stage of quantization, a stochastic gradient descent algorithm can be used to learn the scaling factor scale and the zero reference value zero point to obtain the optimal scale and zero point.

[0052] Optionally, performing 4-bit unsigned integer asymmetric quantization on the grouped quantization weights to obtain a scaling factor, a zero reference value, and a 4-bit unsigned integer quantization weight may include:

[0053] Calculate a first quantization parameter and a second quantization parameter according to a preset loss function by using a gradient descent algorithm, and calculate a scaling factor and a zero reference value according to the first quantization parameter, the second quantization parameter and the group quantization weight;

[0054] According to the scaling factor and the zero reference value, the grouped quantization weight is subjected to 4-bit unsigned integer asymmetric quantization to obtain a 4-bit unsigned integer quantization weight.

[0055] Among them, the preset loss function can be defined as In this embodiment, based on the preset loss function, the first quantization parameter β and the second quantization parameter α can be learned using a gradient descent algorithm to continuously optimize the scaling factor and zero reference value z, thereby reducing the loss of quantization accuracy. Scaling factor and a zero reference value z, determined based on the first quantization parameter β and the second quantization parameter α.

[0056] In a specific example, we can use the formula According to the zoom factor Zero reference value z and group quantization weight W fp8 , calculate the 4-bit unsigned integer quantization weight W uint4 , clip means limiting the value to a specified range, the minimum value of the specified range is 0 and the maximum value is 2 N -1, if the value is less than the minimum value, the value is adjusted to 0, if the value is greater than the maximum value, the value is adjusted to 2 N -1.

[0057] Optionally, calculating a scaling factor and a zero reference value according to the first quantization parameter, the second quantization parameter, and the group quantization weight may include:

[0058] By formula According to the first quantization parameter β, the second quantization parameter α and the group quantization weight W fp8 , calculate the scaling factor and a zero reference value z, where round means rounding off, and N=4.

[0059] The first quantization parameter β and the second quantization parameter α are quantization parameters that need to be learned using a gradient descent algorithm. Typically, the gradient descent algorithm can be used to continuously adjust the values ​​of β and α to continuously adjust the scaling factor. and the value of the zero reference value z. For example, for the current β and α, we can calculate and z, and then calculate W uint4 ; Then, the calculated parameter values ​​can be brought into the preset loss function to calculate the current loss value; if it is detected that the current loss value is less than or equal to the preset loss threshold, the current β and α can be determined to be the optimal value; and if the current loss value is greater than the preset loss threshold, the gradient descent algorithm can be used to obtain the β and α of the next learning round, and re-judge based on the preset loss function until the optimal β and α are obtained. Finally, the optimal β and α can be calculated based on the optimal and z, and can be based on the Perform 4-bit unsigned integer asymmetric quantization with z to obtain W uint4 .

[0060] Optionally, in this embodiment, after obtaining W uint4 After that, you can put the two W uint4 Compressed and saved in an INT8 data format, typically, two W uint4 Placed in the high 4 bits and low 4 bits of an INT8 data respectively.

[0061] In a specific implementation of this embodiment, the calculation process of the offline quantization stage can be as follows: Figure 2 As shown in the figure, the W4A8 (4-bit weight 8-bit input) quantization model is used. First, for the linear layer operator, the semantic features are extracted to perform data dynamic range analysis, and the θ required by the subsequent smooth module is calculated. Then, the input matrix is ​​pseudo-quantized to FP8 based on the θ to obtain And save; secondly, use the smooth module to adjust the weight matrix according to θ, and perform staged quantization of FP8 and UINT4 on the adjusted weight matrix to obtain the quantization parameters.

[0062] The advantage of the above setting is that it can effectively reduce weight storage while greatly reducing the loss of model accuracy.

[0063] S140. In the online reasoning stage, perform online reasoning on the reasoning task based on the smoothing coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value and the 4-bit unsigned integer quantization weight to obtain a reasoning result.

[0064] In this embodiment, after the quantization of the weights is completed in the offline quantization step, the quantization parameters, such as the smoothing coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value and the 4-bit unsigned integer quantization weight can be solidified; then, when performing online inference on the current inference task, for each linear layer operator, the Runtime inference module can be used to load the fixed quantization parameters, build a basic inference environment, and call the smooth module to smooth the current input activation distribution according to the smoothing coefficient; further, the unint4Tfp8 module can be used to quickly convert the model weights quantized into the UINT4 data type into the FP8 data format by hardware instructions; finally, the FP8_GEMM module can be used in the hardware to perform efficient matrix operations on the input and weights of the linear layer operator in the FP8 data format.

[0065] The reasoning task may include a question-answering task, a translation task, a text generation task, a speech recognition task, etc. This embodiment does not specifically limit the type of reasoning task. For example, when the reasoning task is a question-answering task, the reasoning result may be the answer text corresponding to the input question.

[0066] Optionally, in the online reasoning stage, performing online reasoning on the reasoning task based on the smoothing coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value, and the 4-bit unsigned integer quantization weight to obtain the reasoning result may include:

[0067] Obtaining a current input matrix of a current linear layer operator, and performing 8-bit floating-point quantization on the current input matrix according to a flattening coefficient and an input quantization parameter corresponding to the current linear layer operator, to obtain an 8-bit floating-point quantized input corresponding to the current linear layer operator;

[0068] According to the scaling factor and the zero reference value corresponding to the current linear layer operator, the corresponding 4-bit unsigned integer quantization weight is dequantized to 8-bit floating point to obtain the 8-bit floating point quantization weight corresponding to the current linear layer operator;

[0069] The output matrix of the current linear layer operator is calculated based on the 8-bit floating-point quantized input, 8-bit floating-point quantized weight, input quantization parameter and weight quantization parameter corresponding to the current linear layer operator.

[0070] In a specific example, the computational flow of the online inference phase can be as follows: Figure 3First, for the current linear layer operator, the corresponding solidified quantization parameters are loaded through the Runtime inference module, including the flattening coefficient θ and the input quantization parameter Weight quantization parameter Scaling factor Zero reference value z and 4-bit unsigned integer quantization weight W uint4 ; Secondly, based on the formula Adjust the data distribution of the current input matrix to reduce the activation quantization error; then, cast the adjusted input matrix fp16_to_fp8 Hardware instructions are quantized to FP8. For example, the formula Perform FP8 quantization; after that, you can load the unint4Tfp8 module and use hardware instructions to quickly dequantize the model weights in the UINT4 data format back to the FP8 data format. The unint4Tfp8 module requires z and W uint4 Three inputs, the output result is a weight matrix in FP8 data format, that is, FP8 quantized weights; further, a general matrix multiplication (GEMM) operation can be performed between the weight matrix in FP8 data format and the input matrix on hardware that supports FP8 data format to obtain the calculation result in FP8 data format; finally, the calculation result in FP8 data format can be used to calculate the weight matrix in FP8 data format. and The final calculation result of the linear layer operator in FP16 format is calculated, that is, the output matrix.

[0071] In this embodiment, by constructing a quantization model in an offline manner before the inference operation, and providing a method for quickly converting FP8 from UINT4, efficient inference operations of large language models can be guaranteed; moreover, matrix operations can be completed by hardware that supports FP8 operations, and by configuring reasonable FP8 floating-point operations, the efficiency of inference operations can be significantly improved; finally, by providing hardware instructions for converting FP16 to FP8 data format, the FP8 quantization speed can be accelerated.

[0072] The technical solution of the embodiment of the present invention is as follows: in the offline quantization stage, according to the input matrix and weight matrix of each linear layer operator, a smoothing coefficient is calculated, and according to the smoothing coefficient, an input quantization parameter and a weight quantization parameter are calculated; according to the weight quantization parameter, the weight matrix is ​​quantized by 8-bit floating point to obtain an 8-bit floating point quantization weight, and the 8-bit floating point quantization weight is grouped based on a preset grouping number to obtain the grouped quantization weight; the grouped quantization weight is asymmetric quantized by 4-bit unsigned integer to obtain a scaling coefficient, a zero reference value and a 4-bit unsigned integer quantization weight; in the online inference stage, the inference task is online inferenced based on the smoothing coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value and the 4-bit unsigned integer quantization weight to obtain an inference result; by adjusting the input matrix and the weight matrix by using the smoothing coefficient, and sequentially performing 8-bit floating point quantization and 4-bit unsigned integer asymmetric quantization on the weight matrix, the hardware memory occupied by the large language model can be reduced, the model accuracy loss can be reduced, and the model inference efficiency can be improved.

[0073] Embodiment 2

[0074] Figure 4 This is a schematic diagram of the structure of a model quantization device provided in Embodiment 2 of the present invention. Figure 4 As shown, the device includes: a smoothing coefficient calculation module 210, a first weight quantization module 220, a second weight quantization module 230 and an online reasoning module 240; wherein,

[0075] The smoothing coefficient calculation module 210 is used to calculate the smoothing coefficient according to the input matrix and weight matrix of each linear layer operator in the offline quantization stage, and calculate the input quantization parameter and the weight quantization parameter according to the smoothing coefficient;

[0076] A first weight quantization module 220 is used to perform 8-bit floating-point quantization on the weight matrix according to the weight quantization parameter to obtain 8-bit floating-point quantization weights, and group the 8-bit floating-point quantization weights based on a preset grouping number to obtain grouped quantization weights;

[0077] A second weight quantization module 230 is used to perform 4-bit unsigned integer asymmetric quantization on the grouped quantization weights to obtain a scaling factor, a zero reference value and a 4-bit unsigned integer quantization weight;

[0078] The online reasoning module 240 is used to perform online reasoning on the reasoning task based on the smoothing coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value and the 4-bit unsigned integer quantization weight in the online reasoning stage to obtain the reasoning result.

[0079] The technical solution of the embodiment of the present invention is as follows: in the offline quantization stage, according to the input matrix and weight matrix of each linear layer operator, a smoothing coefficient is calculated, and according to the smoothing coefficient, an input quantization parameter and a weight quantization parameter are calculated; according to the weight quantization parameter, the weight matrix is ​​quantized by 8-bit floating point to obtain an 8-bit floating point quantization weight, and the 8-bit floating point quantization weight is grouped based on a preset grouping number to obtain the grouped quantization weight; the grouped quantization weight is asymmetric quantized by 4-bit unsigned integer to obtain a scaling coefficient, a zero reference value and a 4-bit unsigned integer quantization weight; in the online inference stage, the inference task is online inferenced based on the smoothing coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value and the 4-bit unsigned integer quantization weight to obtain an inference result; by adjusting the input matrix and the weight matrix by using the smoothing coefficient, and sequentially performing 8-bit floating point quantization and 4-bit unsigned integer asymmetric quantization on the weight matrix, the hardware memory occupied by the large language model can be reduced, the model accuracy loss can be reduced, and the model inference efficiency can be improved.

[0080] Optionally, the smoothing coefficient calculation module 210 is specifically used to obtain the maximum value of the input matrix in each input channel and the maximum value of the weight matrix in each input channel;

[0081] The smoothing coefficient is calculated according to the maximum value of the input matrix in each input channel and the maximum value of the weight matrix in each input channel.

[0082] Optionally, the smoothing coefficient calculation module 210 is specifically configured to obtain a smoothing input matrix by dividing the input matrix by the smoothing coefficient, and calculate the input quantization parameter according to the smoothing input matrix;

[0083] The leveling coefficient is multiplied by the weight matrix to obtain a leveling weight matrix, and the weight quantization parameter is calculated based on the leveling weight matrix.

[0084] Optionally, the first weight quantization module 220 is specifically configured to perform 8-bit floating-point quantization on a channel-by-channel basis according to the flattened weight matrix and the weight quantization parameter through a hardware data type conversion instruction to obtain an 8-bit floating-point quantization weight.

[0085] Optionally, a second weight quantization module 230 is specifically configured to calculate a first quantization parameter and a second quantization parameter according to a preset loss function by using a gradient descent algorithm, and calculate a scaling factor and a zero reference value according to the first quantization parameter, the second quantization parameter and the group quantization weight;

[0086] According to the scaling factor and the zero reference value, the grouped quantization weight is subjected to 4-bit unsigned integer asymmetric quantization to obtain a 4-bit unsigned integer quantization weight.

[0087] Optionally, the second weight quantization module 230 is specifically configured to use the formula According to the first quantization parameter β, the second quantization parameter α and the group quantization weight W fp8 , calculate the scaling factor and a zero reference value z, where round means rounding off, and N=4.

[0088] Optionally, the online reasoning module 240 is specifically used to obtain a current input matrix of the current linear layer operator, and perform 8-bit floating-point quantization on the current input matrix according to a flattening coefficient and an input quantization parameter corresponding to the current linear layer operator to obtain an 8-bit floating-point quantized input corresponding to the current linear layer operator;

[0089] According to the scaling factor and the zero reference value corresponding to the current linear layer operator, the corresponding 4-bit unsigned integer quantization weight is dequantized to 8-bit floating point to obtain the 8-bit floating point quantization weight corresponding to the current linear layer operator;

[0090] The output matrix of the current linear layer operator is calculated based on the 8-bit floating-point quantized input, 8-bit floating-point quantized weight, input quantization parameter and weight quantization parameter corresponding to the current linear layer operator.

[0091] The model quantization device provided in the embodiment of the present invention can execute the model quantization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0092] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0093] Embodiment 3

[0094] Figure 5 A schematic diagram of the structure of an electronic device 30 that can be used to implement an embodiment of the present invention is shown. The electronic device 30 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 30 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0095] like Figure 5As shown, the electronic device 30 includes at least one processor 31, and a memory connected to the at least one processor 31, such as a read-only memory (ROM) 32, a random access memory (RAM) 33, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 31 can perform various appropriate actions and processes according to the computer program stored in the read-only memory 32 or the computer program loaded from the storage unit 38 to the random access memory 33. In the RAM 33, various programs and data required for the operation of the electronic device 30 can also be stored. The processor 31, the ROM 32 and the RAM 33 are connected to each other through a bus 34. The input / output (I / O) interface 35 is also connected to the bus 34.

[0096] A number of components in the electronic device 30 are connected to the I / O interface 35, including: an input unit 36, such as a keyboard, a mouse, etc.; an output unit 37, such as various types of displays, speakers, etc.; a storage unit 38, such as a disk, an optical disk, etc.; and a communication unit 39, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 39 allows the electronic device 30 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0097] The processor 31 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 31 include, but are not limited to, a central processing unit, a graphics processing unit, various special artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any appropriate processors, controllers, microcontrollers, etc. The processor 31 performs the various methods and processes described above, such as the model quantization method.

[0098] In some embodiments, the model quantization method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 38. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 30 via the ROM 32 and / or the communication unit 39. When the computer program is loaded into the RAM 33 and executed by the processor 31, one or more steps of the model quantization method described above may be performed. Alternatively, in other embodiments, the processor 31 may be configured to perform the model quantization method in any other suitable manner (e.g., by means of firmware).

[0099] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays, application specific integrated circuits, application specific standard products, systems on a chip, load programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0100] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0101] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0102] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device 30 having: a display device (e.g., a cathode ray tube or a liquid crystal display) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device 30. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0103] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area networks, wide area networks, blockchain networks, and the Internet.

[0104] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server.

[0105] This embodiment may also include a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the model quantization method provided by any embodiment of the present invention.

[0106] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0107] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A model quantization method, characterized in that: include: In the offline quantization stage, a smoothing coefficient is calculated according to the input matrix and the weight matrix of each linear layer operator, and an input quantization parameter and a weight quantization parameter are calculated according to the smoothing coefficient; Performing 8-bit floating-point quantization on the weight matrix according to the weight quantization parameter to obtain 8-bit floating-point quantization weights, and grouping the 8-bit floating-point quantization weights based on a preset grouping number to obtain grouped quantization weights; Perform 4-bit unsigned integer asymmetric quantization on the grouped quantization weights to obtain a scaling factor, a zero reference value, and a 4-bit unsigned integer quantization weight; In the online reasoning stage, the reasoning task is reasoned online based on the smoothing coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value and the 4-bit unsigned integer quantization weight to obtain a reasoning result.

2. The method according to claim 1, characterized in that According to the input matrix and weight matrix of each linear layer operator, the flattening coefficient is calculated, including: Obtaining the maximum value of the input matrix in each input channel and the maximum value of the weight matrix in each input channel; The smoothing coefficient is calculated according to the maximum value of the input matrix in each input channel and the maximum value of the weight matrix in each input channel.

3. The method according to claim 1, characterized in that According to the smoothing coefficient, an input quantization parameter and a weight quantization parameter are calculated, including: Dividing the input matrix by the smoothing coefficient to obtain a smoothing input matrix, and calculating the input quantization parameter according to the smoothing input matrix; The leveling coefficient is multiplied by the weight matrix to obtain a leveling weight matrix, and the weight quantization parameter is calculated based on the leveling weight matrix.

4. The method according to claim 3, characterized in that The weight matrix is ​​subjected to 8-bit floating point quantization according to the weight quantization parameter to obtain an 8-bit floating point quantization weight, including: Through the hardware data type conversion instruction, 8-bit floating-point quantization is performed channel by channel according to the flattened weight matrix and the weight quantization parameter to obtain an 8-bit floating-point quantization weight.

5. The method according to claim 1, characterized in that The grouped quantization weights are subjected to 4-bit unsigned integer asymmetric quantization to obtain a scaling factor, a zero reference value, and a 4-bit unsigned integer quantization weight, including: Calculate a first quantization parameter and a second quantization parameter according to a preset loss function by using a gradient descent algorithm, and calculate a scaling factor and a zero reference value according to the first quantization parameter, the second quantization parameter and the group quantization weight; According to the scaling factor and the zero reference value, the grouped quantization weight is subjected to 4-bit unsigned integer asymmetric quantization to obtain a 4-bit unsigned integer quantization weight.

6. The method according to claim 5, characterized in that Calculating a scaling factor and a zero reference value according to the first quantization parameter, the second quantization parameter, and the group quantization weight includes: By formula and According to the first quantization parameter β, the second quantization parameter α and the group quantization weight W fp8 , calculate the scaling factor and a zero reference value z, where round means rounding off, and N=4.

7. The method according to claim 1, characterized in that In the online reasoning stage, the reasoning task is reasoned online based on the smoothing coefficient, the input quantization parameter, the weight quantization parameter, the scaling coefficient, the zero reference value and the 4-bit unsigned integer quantization weight to obtain a reasoning result, including: Obtaining a current input matrix of a current linear layer operator, and performing 8-bit floating-point quantization on the current input matrix according to a flattening coefficient and an input quantization parameter corresponding to the current linear layer operator, to obtain an 8-bit floating-point quantized input corresponding to the current linear layer operator; According to the scaling factor and the zero reference value corresponding to the current linear layer operator, the corresponding 4-bit unsigned integer quantization weight is dequantized to 8-bit floating point to obtain the 8-bit floating point quantization weight corresponding to the current linear layer operator; The output matrix of the current linear layer operator is calculated based on the 8-bit floating-point quantized input, 8-bit floating-point quantized weight, input quantization parameter and weight quantization parameter corresponding to the current linear layer operator.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor, and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the model quantization method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to enable a processor to implement the model quantization method according to any one of claims 1 to 7 when executed.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the model quantization method according to any one of claims 1 to 7 when being executed by a processor.

Images