Model quantization method and device, electronic device and storage medium
By setting the initial hyperparameters in the model and adaptively adjusting it, the block quantization method is adopted to solve the accuracy and speed problems of complex artificial intelligence models under limited hardware, and efficient model quantization is achieved.
Patent Information
- Application Number
- CN202410433346.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-04-10
AI Technical Summary
In the prior art, model quantization methods are difficult to ensure model accuracy on hardware devices with limited computing power, while not reducing model inference speed. Especially in complex artificial intelligence models, quantization methods have problems of accuracy loss and hardware dependence.
By setting the initial hyperparameters for the model to be quantized and adjusting the hyperparameters of each layer to be quantized according to the calibration data set, the block quantization method is adopted to enable multiple weight elements in the weight matrix of the layer to be quantized to share a quantization parameter, adaptively Adjust hyperparameters to suit different models and datasets, enabling efficient quantization at low bits.
It realizes that the model is quantized to low bits without reducing the accuracy of the model, improves the inference efficiency, avoids the hardware dependence of pruning and sparse matrix calculation, and is suitable for various quantization solutions.
Smart Images

Figure CN118428453B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a model quantization method and device, an electronic device, and a storage medium. Background Art
[0002] With the continuous development of artificial intelligence technology, the model structure related to artificial intelligence has been designed to be more and more complex, the corresponding model weight parameters are getting larger and larger, the model reasoning time is getting longer and longer, and the computing power requirements of hardware devices are getting higher and higher, which brings greater challenges to hardware devices. The quantization methods in related technologies are difficult to guarantee model accuracy. How to meet the model accuracy without reducing the model reasoning speed on hardware devices with limited computing power will become increasingly important. Summary of the invention
[0003] The present disclosure proposes a model quantization technology solution.
[0004] According to one aspect of the present disclosure, a model quantization method is provided, comprising: obtaining a model to be quantized, the model to be quantized comprising at least one layer to be quantized; determining an initial hyperparameter corresponding to at least one layer to be quantized of the model to be quantized, the hyperparameter representing the number of weight elements in a weight matrix of the layer to be quantized that share the same quantization parameter, the quantization parameter being used to convert a weight matrix represented by high precision into a low precision representation; adjusting the hyperparameter of each layer to be quantized in the model to be quantized according to a calibration data set; and quantizing the model to be quantized according to the adjusted hyperparameters of each layer to be quantized to obtain a target quantization model.
[0005] In a possible implementation, the adjustment process of the hyperparameters of any layer to be quantized in the model to be quantized includes: for the 1st to Tth calibration data in the calibration data set, performing the following steps: determining a first result based on the i-th calibration data and the weight matrix of the layer to be quantized, where i is a positive integer less than or equal to T; determining a second result based on the i-th calibration data in the calibration data set and the quantization weight matrix of the layer to be quantized, where the quantization weight matrix is determined by a preset number of quantization bits, the hyperparameters adjusted for the i-1th time, and the weight matrix of the layer to be quantized; determining a third result based on the deviation between the first result and the second result; adjusting the hyperparameters adjusted for the i-1th time based on the third result to obtain the hyperparameters adjusted for the i-th time, wherein the hyperparameters adjusted for the Tth time are the adjusted hyperparameters of the layer to be quantized.
[0006] In a possible implementation, the hyperparameter adjusted for the i-1th time is adjusted according to the third result to obtain the hyperparameter adjusted for the i-th time, including: when the third result is greater than a first preset threshold, subtracting a preset value from the hyperparameter adjusted for the i-1th time to obtain the hyperparameter adjusted for the i-th time; when the third result is less than a second preset threshold, adding the preset value to the hyperparameter adjusted for the i-1th time to obtain the hyperparameter adjusted for the i-th time; when the third result is less than or equal to the first preset threshold and greater than or equal to the second preset threshold, using the hyperparameter adjusted for the i-1th time as the hyperparameter adjusted for the i-th time.
[0007] In a possible implementation, a second result is determined based on the i-th calibration data in the calibration data set and the quantization weight matrix of the layer to be quantized, including: performing block processing on the weight matrix of the layer to be quantized according to the hyperparameter adjusted for the i-1th time to obtain at least one block matrix; performing quantization processing on each block matrix according to a preset number of quantization bits to obtain a block quantization matrix corresponding to each block matrix; splicing each block quantization matrix to obtain a quantization weight matrix; and determining the second result based on the i-th calibration data in the calibration data set and the quantization weight matrix.
[0008] In a possible implementation, each block matrix is quantized according to a preset number of quantization bits to obtain a block quantization matrix corresponding to each block matrix, including: determining a quantization parameter of each block matrix according to a preset number of quantization bits and a maximum value and a minimum value of each block matrix; and quantizing the block matrix according to the quantization parameter of each block matrix to obtain a block quantization matrix corresponding to each block matrix.
[0009] In a possible implementation, the layer to be quantized includes a linear layer, and the quantization parameter includes a first parameter from a first matrix, a second parameter from a second matrix, and a third parameter from a third matrix, and the sizes of the first matrix, the second matrix, and the third matrix are the same and are determined based on the size of the weight matrix and the hyperparameter; wherein, when the value of the hyperparameter is K, it indicates that the number of weight elements sharing the same quantization parameter in the weight matrix of the layer to be quantized is K, and the third parameter at any position in the third matrix is used to represent the position index of the K weight elements in the weight matrix that share the first parameter at the same position in the first matrix and the second parameter at the same position in the second matrix, the first parameter at the same position in the first matrix is used to characterize the scaling ratio of the K weight elements, and the second parameter at the same position in the second matrix is used to characterize the zero position of the K weight elements, and K is greater than or equal to 1 and less than or equal to the total number of weight elements in the weight matrix.
[0010] In one possible implementation, the calibration data set matches a target task, and the target task includes at least one of image classification, target detection, image segmentation, and speech recognition. The method also includes: inputting the data to be processed corresponding to the target task into the target quantization model for processing to obtain a processing result, wherein the data to be processed includes at least one of image data, speech data, and text data that matches the task type of the target task.
[0011] According to one aspect of the present disclosure, a model quantization device is provided, comprising: an acquisition module, used to acquire a model to be quantized, the model to be quantized comprising at least one layer to be quantized; a determination module, used to determine an initial hyperparameter corresponding to at least one layer to be quantized of the model to be quantized, the hyperparameter representing the number of weight elements in a weight matrix of the layer to be quantized that share the same quantization parameter, the quantization parameter being used to convert a weight matrix represented by high precision into a low precision representation; an adjustment module, used to adjust the hyperparameters of each layer to be quantized in the model to be quantized according to a calibration data set; and a quantization module, used to quantize the model to be quantized according to the adjusted hyperparameters of each layer to be quantized to obtain a target quantization model.
[0012] In one possible implementation, the adjustment module is used to: for the 1st to Tth calibration data in the calibration data set, perform the following steps: determine a first result based on the i-th calibration data and the weight matrix of the layer to be quantized, where i is a positive integer less than or equal to T; determine a second result based on the i-th calibration data in the calibration data set and the quantization weight matrix of the layer to be quantized, where the quantization weight matrix is determined by a preset number of quantization bits, the hyperparameter adjusted for the i-1th time, and the weight matrix of the layer to be quantized; determine a third result based on the deviation between the first result and the second result; adjust the hyperparameter adjusted for the i-1th time based on the third result to obtain the hyperparameter adjusted for the i-th time, wherein the hyperparameter adjusted for the Tth time is the adjusted hyperparameter of the layer to be quantized.
[0013] In a possible implementation, the hyperparameter adjusted for the i-1th time is adjusted according to the third result to obtain the hyperparameter adjusted for the i-th time, including: when the third result is greater than a first preset threshold, subtracting a preset value from the hyperparameter adjusted for the i-1th time to obtain the hyperparameter adjusted for the i-th time; when the third result is less than a second preset threshold, adding the preset value to the hyperparameter adjusted for the i-1th time to obtain the hyperparameter adjusted for the i-th time; when the third result is less than or equal to the first preset threshold and greater than or equal to the second preset threshold, using the hyperparameter adjusted for the i-1th time as the hyperparameter adjusted for the i-th time.
[0014] In a possible implementation, a second result is determined based on the i-th calibration data in the calibration data set and the quantization weight matrix of the layer to be quantized, including: performing block processing on the weight matrix of the layer to be quantized according to the hyperparameter adjusted for the i-1th time to obtain at least one block matrix; performing quantization processing on each block matrix according to a preset number of quantization bits to obtain a block quantization matrix corresponding to each block matrix; splicing each block quantization matrix to obtain a quantization weight matrix; and determining the second result based on the i-th calibration data in the calibration data set and the quantization weight matrix.
[0015] In a possible implementation, each block matrix is quantized according to a preset number of quantization bits to obtain a block quantization matrix corresponding to each block matrix, including: determining a quantization parameter of each block matrix according to a preset number of quantization bits and a maximum value and a minimum value of each block matrix; and quantizing the block matrix according to the quantization parameter of each block matrix to obtain a block quantization matrix corresponding to each block matrix.
[0016] In a possible implementation, the layer to be quantized includes a linear layer, and the quantization parameter includes a first parameter from a first matrix, a second parameter from a second matrix, and a third parameter from a third matrix, and the sizes of the first matrix, the second matrix, and the third matrix are the same and are determined based on the size of the weight matrix and the hyperparameter; wherein, when the value of the hyperparameter is K, it indicates that the number of weight elements sharing the same quantization parameter in the weight matrix of the layer to be quantized is K, and the third parameter at any position in the third matrix is used to represent the position index of the K weight elements in the weight matrix that share the first parameter at the same position in the first matrix and the second parameter at the same position in the second matrix, the first parameter at the same position in the first matrix is used to characterize the scaling ratio of the K weight elements, and the second parameter at the same position in the second matrix is used to characterize the zero position of the K weight elements, and K is greater than or equal to 1 and less than or equal to the total number of weight elements in the weight matrix.
[0017] In one possible implementation, the calibration data set matches a target task, and the target task includes at least one of image classification, target detection, image segmentation, and speech recognition. The device also includes a processing module for: inputting the data to be processed corresponding to the target task into the target quantization model for processing to obtain a processing result, wherein the data to be processed includes at least one of image data, speech data, and text data that matches the task type of the target task.
[0018] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.
[0019] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.
[0020] In the disclosed embodiment, a model to be quantized can be obtained, the model to be quantized includes at least one layer to be quantized; the initial hyperparameter corresponding to at least one layer to be quantized of the model to be quantized is determined, the hyperparameter represents the number of weight elements in the weight matrix of the layer to be quantized that share the same quantization parameter, and the quantization parameter is used to convert the weight matrix represented by high precision into a low precision representation; according to the calibration data set, the hyperparameter of each layer to be quantized in the model to be quantized is adjusted; according to the adjusted hyperparameter of each layer to be quantized, the model to be quantized is quantized to obtain a target quantization model. In this way, by setting the initial hyperparameter, multiple weight elements in the weight matrix to be quantized share a quantization parameter, and the hyperparameter is adaptively adjusted using different calibration data in the calibration data set, which is conducive to automatically adapting the adjusted hyperparameter to different models to be quantized and different calibration data sets, so that the target quantization model obtained based on the adjusted hyperparameter can show higher efficiency and accuracy in the reasoning process.
[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.
[0023] Figure 1 A flowchart of a model quantization method according to an embodiment of the present disclosure is shown.
[0024] Figure 2 A schematic diagram showing a hyperparameter adjustment method according to an embodiment of the present disclosure.
[0025] Figure 3 A block diagram of a model quantization device according to an embodiment of the present disclosure is shown.
[0026] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0027] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0028] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0029] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.
[0030] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.
[0031] In order to better illustrate the model quantization method of the embodiment of the present disclosure, the hierarchical quantization and channel-by-channel quantization in the related technology are first explained. Hierarchical quantization and channel-by-channel quantization are to quantize the weight parameters of the model to be quantized (such as a deep learning model) at different granularities. For example, hierarchical quantization is that the weight matrix of a layer to be quantized (such as a fully connected layer) of the model to be quantized shares a quantization parameter, while channel-by-channel quantization is to split the weight matrix of the layer to be quantized into channels in a certain dimension. After the split, each channel shares a quantization parameter. For example, if the weight matrix has 64 channels in the channel dimension, the weight matrix will correspond to 64 quantization parameters.
[0032] Among them, each quantization parameter can be composed of two parts: scaling ratio and zero point position. The original weight matrix and the quantized weight matrix of the model to be quantized can be converted to each other through the quantization parameter. The conversion formula is:
[0033] weight = (Qweight - zeropoints) * scales (1)
[0034] Qweight = clamp(weight / scales, max) + zeropoints (2)
[0035] Among them, scales represents the scaling ratio, zeropoints represents the zero point position, weight represents the original weight matrix to be quantized, Qweight represents the weight matrix after quantization, clamp() represents the clamping operation function, and clamp(weight / scales, max) represents the maximum value of the quotient of each weight element in the weight matrix weight and the scaling ratio scale is clamped to max. For example, assuming that the maximum cutoff value max is set to 255, clamp(260, 255) = 255, where the maximum value of 8-bit integer is 255, and 260 is truncated to 255, that is, quantized to 8 bits. For another example, assuming that the maximum cutoff value max is set to 15, clamp(260, 15) = 15, where the maximum value of 4-bit integer is 15, and 260 is truncated to 15, that is, quantized to 4 bits.
[0036] In actual application, if hierarchical quantization is used, the scales and zeropoints are both real numbers. Formulas (1) and (2) can complete the transformation of the weight matrix through element-wise addition and subtraction and multiplication and division. If channel-wise quantization is used, assuming that the original weight matrix is a 128×64 matrix, it can be channel-wise quantized along the next dimension, and the corresponding scales and zeropoints are both 128×1 matrices. When calculating multiplication and division and addition and subtraction, the method of broadcasting can be used to repeatedly stack them to the same size as the original weight matrix and then calculate them. For example, when calculating weights / scales in formula (2), the scales of 128×1 can be expanded to 128×64, for example: (X i ,0)=(X i ,1)=...=(X i ,63), where X i It is any element in the 128×1 matrix representing the scaling factor scales, i=0,1,...,127. Then, the division between the 128×64 weight matrix weights and the 128×64 scaling factor matrix scales is calculated.
[0037] However, when the model to be quantized is quantized to a low bit (e.g., 4 bits), the hierarchical quantization or channel-by-channel quantization methods in the related art are difficult to guarantee the accuracy of the model after quantization. In the related art, the solution of quantizing to a lower bit (e.g., 4 bits) requires the model to be pruned to ensure the accuracy in order to ensure the accuracy of the model after quantization. However, the pruned model usually relies on the hardware to provide sparse matrix calculations to achieve higher performance, and cannot be accelerated on hardware that does not support sparse matrix calculations.
[0038] In view of this, the embodiment of the present disclosure sets initial hyperparameters for the to-be-quantized layer of the to-be-quantized model, adjusts the hyperparameters of each to-be-quantized layer in the to-be-quantized model according to the calibration data set, and uses the adjusted hyperparameters of each to-be-quantized layer to quantize the to-be-quantized model to obtain a target quantized model. The hyperparameter represents the number of weight elements that share the same quantization parameter in the weight matrix of the to-be-quantized layer, and the hyperparameter of each to-be-quantized layer is used to quantize the to-be-quantized layer, that is, the weight matrix of the to-be-quantized layer is block-quantized with the value of the hyperparameter as the granularity.
[0039] Compare the hierarchical quantization and channel-wise quantization in the related art. For example, hierarchical quantization is based on the granularity of the layer to be quantized, and all weight elements of the layer to be quantized share a quantization parameter. Channel-wise quantization is based on the granularity of the channel, and all weight elements of each channel of the layer to be quantized share a quantization parameter. The embodiment of the present disclosure is a block quantization with the value of a hyperparameter as the granularity. Some weight elements in the weight matrix of the layer to be quantized can share a quantization parameter, and the weight parameters of the layer to be quantized can be quantized at a finer granularity than hierarchical quantization and channel-wise quantization. Among them, the smaller the value of the adjusted hyperparameter, the finer the granularity of the block quantization, and the quantization method with a finer granularity will have a higher accuracy.
[0040] Furthermore, the model quantization method of the embodiment of the present disclosure adjusts the hyperparameters of each layer to be quantized in the model to be quantized according to different calibration data in the calibration data set, so that the adjusted hyperparameters automatically adapt to different models to be quantized and different calibration data sets, so that the model to be quantized can be subjected to block quantization processing with an adaptive granularity based on the adjusted hyperparameters, and the model can be quantized to a low bit without reducing the accuracy of the model. Compared with the solution in the related art that prunes the model after quantization in order to ensure the accuracy of the quantized model, the model quantization method of the embodiment of the present disclosure does not involve a pruning process, does not require sparse matrix calculations, and does not have the hardware dependence caused by sparse matrix calculations, so it is easy to integrate into various quantization schemes to achieve better reasoning acceleration.
[0041] Figure 1 A flow chart of a model quantization method according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the model quantization method includes:
[0042] In step S11, a model to be quantized is obtained, where the model to be quantized includes at least one layer to be quantized;
[0043] In step S12, an initial hyperparameter corresponding to at least one layer to be quantized of the model to be quantized is determined, wherein the hyperparameter represents the number of weight elements in the weight matrix of the layer to be quantized that share the same quantization parameter, and the quantization parameter is used to convert the weight matrix represented by high precision into a low precision representation form;
[0044] In step S13, according to the calibration data set, the hyper parameters of each layer to be quantized in the model to be quantized are adjusted;
[0045] In step S14, the model to be quantized is quantized according to the adjusted hyper parameters of each layer to be quantized to obtain a target quantized model.
[0046] In a possible implementation, the model quantization method can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the method can be executed by a server.
[0047] In one possible implementation, in step S11, the model to be quantized obtained may be a preliminarily trained neural network model, such as convolutional neural networks (CNN), deep learning neural networks (DNN), recurrent neural networks (RNN), residual networks (ResNets), back propagation neural networks (BP), backbone neural networks (Backbone Neural Network), large language models (LLM), etc. The present disclosure does not impose any specific restrictions on the type of the model to be quantized.
[0048] Among them, the obtained model to be quantized includes at least one layer to be quantized, and the layer to be quantized may include a linear layer (also called a fully connected layer). Optionally, each network layer of the model to be quantized may be a layer to be quantized (for example, including a linear layer), and the model to be quantized may also have some network layers that are not quantifiable, and another part that are quantifiable layers to be quantized (for example, including a linear layer). The present disclosure does not specifically limit the network structure of the model to be quantized. Optionally, the obtained model to be quantized may be a neural network model for any task in image classification, target detection, image segmentation, speech recognition, natural language processing, machine translation, and question-answering system, and the present disclosure does not limit this.
[0049] In step S11, the model to be quantized is obtained, and in step S12, the initial hyperparameter corresponding to at least one layer to be quantized of the model to be quantized can be determined. Optionally, the same hyperparameter can be set for multiple layers to be quantized in the model to be quantized, or different hyperparameters can be set for different layers to be quantized, and the present disclosure does not limit this. Among them, the initial hyperparameter can be set in a numerical range of [1, S], S represents the number of weight elements in the weight matrix of the layer to be quantized, and when the numerical range is met, the initial hyperparameter can be set to 1, or the initial hyperparameter can be set to S, or the initial hyperparameter can be set to S / 2, and the present disclosure does not limit the initial value of the set hyperparameter.
[0050] For example, assuming that the weight matrix of a layer to be quantized is The initial value of the hyperparameter can be set to 2, indicating that every two weight elements in the weight matrix of the layer to be quantized share a quantization parameter. For example, if the weight matrix is stored in a row-major manner, weight element W1 and weight element W2 can share a quantization parameter, and weight element W3 and weight element W4 can share another quantization parameter; for example, if the weight matrix is stored in a column-major manner, weight element W1 and weight element W3 can share a quantization parameter, and weight element W2 and weight element W4 can share another quantization parameter.
[0051] Alternatively, the initial value of the hyperparameter may be set to 4, indicating that every four weight elements in the weight matrix of the layer to be quantized share a quantization parameter, and weight elements W1 to W4 correspond to the same quantization parameter.
[0052] Alternatively, the initial value of the hyperparameter may be set to 1, indicating that each weight element in the weight matrix of the layer to be quantized uses a quantization parameter, and the weight elements W1 to W4 correspond to different quantization parameters.
[0053] Among them, the quantization parameter is used to convert the weight matrix represented by high precision into a low-precision representation. For example, for any weight element in the weight matrix, the quantization parameter can convert the weight parameter represented by float32 (32-bit floating point number) into a weight parameter represented by int8 (signed 8-bit integer). It should be understood that the number of bits occupied by the weight parameter represented by high precision is higher than the number of bits occupied by the weight parameter represented by low precision. The present disclosure only takes float32 and int8 as examples, and does not limit the specific number of bits occupied by specific high-precision representation forms and low-precision representation forms.
[0054] In step S12, initial hyperparameters are set for at least one layer to be quantized of the model to be quantized, and in step S13, the hyperparameters of each layer to be quantized in the model to be quantized can be adjusted according to the calibration data set. For example, the calibration data in the calibration data set can be input into the model to be quantized, and the layer to be quantized of the model to be quantized will output a first result without quantizing the weight parameters, and output a second result when quantizing the weight parameters based on the hyperparameters, and the hyperparameters are adjusted by comparing the deviations between the first result and the second result. In this way, the process is repeated until all the calibration data in the calibration data set are traversed, and the hyperparameters of each layer to be quantized in the model to be quantized can be adaptively obtained.
[0055] The calibration data set may be selected according to the specific target task. For example, in an image classification task, the calibration data set may include images of various categories, each category having a corresponding label. In a speech recognition task, the calibration data set may include speech recordings of different people and corresponding text transcriptions. The embodiments of the present disclosure do not limit the form of the calibration data set.
[0056] It should be understood that the process of adaptively adjusting the hyperparameters is an iterative process. If the hyperparameters adjusted in step S13 do not enable the target quantization model to achieve the expected accuracy or efficiency, you can return to step S12 to reset the initial hyperparameters, and then execute step S13 again until the optimal hyperparameters are found.
[0057] In step S13, the adjusted hyper parameters of each layer to be quantized are determined, and in step S14, the model to be quantized is quantized according to the adjusted hyper parameters of each layer to be quantized to obtain a target quantized model.
[0058] Optionally, the weight parameters of each layer to be quantized can be directly quantized according to the adjusted hyperparameters of each layer to be quantized, and the weight parameters represented by high precision can be converted into quantized weight parameters represented by low precision to obtain the target quantization model after quantization. Alternatively, the hyperparameters of each layer to be quantized can be fixed after adjustment, and the quantization-aware training can be performed on the model to be quantized by setting a small batch of training data sets that match the target task, and the weight parameters of each layer to be quantized in the model to be quantized can be adjusted until a trained target quantization model is obtained. The embodiments of the present disclosure do not limit the specific processing process of obtaining the target quantization model based on the hyperparameters.
[0059] In the model quantization method of the embodiment of the present disclosure, the initial hyperparameters can be set so that multiple weight elements in the weight matrix to be quantized share one quantization parameter, and the hyperparameters can be adaptively adjusted using a small amount of calibration data sets, which is conducive to making the adjusted hyperparameters automatically adapt to different models to be quantized and different calibration data sets, so that the target quantization model obtained based on the adjusted hyperparameters can show higher efficiency and accuracy in the reasoning process.
[0060] The model quantization method of the embodiment of the present disclosure is described in detail below.
[0061] In step S11, the model to be quantized is obtained, and in step S12, initial hyperparameters may be set for at least one layer to be quantized of the model to be quantized, wherein the hyperparameters represent the number of weight elements in the weight matrix of the layer to be quantized that share the same quantization parameter.
[0062] In a possible implementation, the quantization parameter includes a first parameter from a first matrix, a second parameter from a second matrix, and a third parameter from a third matrix, and the sizes of the first matrix, the second matrix, and the third matrix are the same, which are determined by the size of the weight matrix and the hyperparameter; wherein, when the value of the hyperparameter is K, it indicates that the number of weight elements sharing the same quantization parameter in the weight matrix of the layer to be quantized is K, and the third parameter at any position in the third matrix is used to indicate the position index of the K weight elements in the weight matrix that share the first parameter at the same position in the first matrix and the second parameter at the same position in the second matrix, the first parameter at the same position in the first matrix is used to characterize the scaling ratio of the K weight elements, and the second parameter at the same position in the second matrix is used to characterize the zero position of the K weight elements, and K is greater than or equal to 1 and less than or equal to the total number of weight elements in the weight matrix.
[0063] In this way, corresponding to hyperparameters of different sizes, the model to be quantized is block quantized at different granularities. Compared with component quantization and channel quantization in related technologies, block quantization with hyperparameters as the granularity is more flexible (for example, the weight parameters of the quantized layer can be quantized at a finer granularity) and has stronger applicability.
[0064] Exemplarily, the quantization parameters can be expressed as (S[m][n], Z[m][n], k_idx[m][n]), where the first parameter S[m][n] representing the scaling ratio comes from the mth row and nth column of the first matrix S, the second parameter Z[m][n] representing the zero point position comes from the mth row and nth column of the second matrix, and the third parameter k_idx[m][n] representing the position index of the weight element in the weight matrix comes from the mth row and nth column of the third matrix. The sizes of these three matrices are the same and can be determined by the size of the weight matrix and the hyperparameters.
[0065] For example, assuming that the value of the hyperparameter is K, the size of the weight matrix W of the layer to be quantized is M×N, where M and N are both positive integers, M represents the number of rows of the weight matrix W, and N represents the number of columns of the weight matrix W. According to the size of the weight matrix W and the value of the hyperparameter K, it can be determined that the sizes of the first matrix S, the second matrix Z, and the third matrix k_idx are M×N / K (or M / K×N).
[0066] Among them, the third parameter k_idx[m][n] in the mth row and nth column of k_idx in the third matrix is used to represent the position index of K weight elements in the weight matrix W that share the first parameter S[m][n] in the mth row and nth column of the first matrix S and the second parameter Z[m][n] in the mth row and nth column of the second matrix Z. The parameter value of the first parameter S[m][n] in the mth row and nth column of the first matrix S is used to characterize the scaling ratio of the K weight elements, and the second parameter Z[m][n] in the mth row and nth column of the second matrix Z is used to characterize the zero position of the K weight elements, 1≤K≤M×N.
[0067] For example, the size of the weight matrix W is 128×64, the value of the hyperparameter K is 16, and every 16 weight elements in the weight matrix W share a quantization parameter, which is equivalent to the weight matrix W being divided into 512 small blocks containing 16 weight elements, and the weight elements in each small block share the same quantization parameter.
[0068] The first matrix S, the second matrix Z, and the third matrix k_idx corresponding to the weight matrix W of size 128×64 may be matrices of size 128×4. In this case, the memory may store the 128×64 (i.e., 8192) weight elements contained in the weight matrix W in a row-major or column-major manner, and then the third parameter k_idx[0][0]=0, the third parameter k_idx[0][1]=16, the third parameter k_idx[0][2]=32, and so on, the third parameter k_idx
[127] [3]=8176.
[0069] The third parameter k_idx[0][0] indicates that the first parameter S[0][0] and the second parameter Z[0][0] correspond to the 0th to 15th weight elements of the weight matrix W. The third parameter k_idx[0][1] indicates that the first parameter S[0][1] and the second parameter Z[0][1] correspond to the 16th to 31st weight elements of the weight matrix W. The third parameter k_idx[0][2] indicates that the first parameter S[0][2] and the second parameter Z[0][2] correspond to the 32nd to 47th weight elements of the weight matrix W. And so on. The third parameter k_idx
[127] [3] indicates that the first parameter S
[127] [3] and the second parameter Z
[127] [3] correspond to the 8176th to 8191th weight elements of the weight matrix W.
[0070] In this case, if the 0th to 15th elements of the weight matrix W are separated to form a new weight matrix and substituted into weight in formula (2); the first parameter S[0][0] and the second parameter Z[0][0] are separated to form a new quantization parameter, and the first parameter S[0][0] is substituted into scales, and the second parameter Z[0][0] is substituted into zeropoints, then the new weight matrix and the new quantization parameter can be hierarchically quantized by formula (2).
[0071] In the example, if the value of the hyperparameter K is the same as the number of columns (or rows) of the weight matrix, block quantization and channel quantization with K as the granularity are equivalent. For example, for a weight matrix of size 128×64, block quantization with K=64 as the granularity is equivalent to channel quantization; or, block quantization with K=128 as the granularity is also equivalent to channel quantization.
[0072] In the example, if the value of the hyperparameter K is the same as the size of the weight matrix, block quantization and hierarchical quantization with K as the granularity are equivalent. For example, for a weight matrix of size 128×64, block quantization and hierarchical quantization with K=8192=128×64 as the granularity are equivalent.
[0073] In the example, if the value of the hyperparameter K is 1, it means that each weight element in the weight matrix has a separate quantization parameter, and the first matrix and the second matrix are the same size as the weight matrix.
[0074] It can be seen that the larger the value of the hyperparameter K, the larger the block granularity of the weight matrix, the more weight elements in the weight matrix can share the same quantization parameter, the smaller the size of the corresponding quantization parameter matrix (such as the first matrix to the third matrix), the lower the quantization accuracy of the model to be quantized, and the faster the model runs; among them, the largest block of the weight matrix is the size of the entire weight matrix, and at this time, the block quantization degenerates into hierarchical quantization;
[0075] On the contrary, the smaller the value of the hyperparameter K is, the smaller the block granularity of the weight matrix is, the fewer weight elements in the weight matrix share the same quantization parameter, the larger the size of the corresponding quantization parameter matrix (such as the first matrix to the third matrix), the higher the quantization accuracy of the model to be quantized, and the slower the model runs; among them, the minimum hyperparameter value K is 1, each block consists of 1 weight element, and each weight element in the weight matrix has a separate quantization parameter.
[0076] In step S12, initial hyperparameters are set for at least one layer to be quantized of the model to be quantized. In step S13, the hyperparameters of each layer to be quantized in the model to be quantized may be adjusted according to the calibration data set.
[0077] Figure 2 A schematic diagram showing a method for adjusting hyperparameters according to an embodiment of the present disclosure is shown as follows: Figure 2 As shown, for the 1st to Tth calibration data in the calibration data set, the following steps are performed: a first result is determined based on the i-th calibration data and the weight matrix of the layer to be quantized, where i is a positive integer less than or equal to T; a second result is determined based on the i-th calibration data in the calibration data set and the quantization weight matrix of the layer to be quantized, where the quantization weight matrix is determined by a preset number of quantization bits, the hyperparameter adjusted for the i-1th time, and the weight matrix of the layer to be quantized; a third result is determined based on the deviation between the first result and the second result; and the hyperparameter adjusted for the i-1th time is adjusted based on the third result to obtain the hyperparameter adjusted for the i-th time, where the hyperparameter adjusted for the Tth time is the adjusted hyperparameter of the layer to be quantized.
[0078] For example, it is assumed that the layer to be quantized is a linear layer, the weight matrix of the layer to be quantized is W, and the calibration data set includes T calibration data, for example: calibration data 1 to calibration data T. In step S12, the initial value K of the hyperparameter value K is set. (0) , hereinafter referred to as the hyperparameter K (0) , indicating the value of the hyperparameter K = K (0)In step S13, calibration data 1 to calibration data T may be input one by one until each calibration data is traversed to obtain the adjusted hyperparameter value K=K (T) .
[0079] For the first (for example, i=1) calibration data 1, the first calibration data 1 can be input into the model to be quantized to obtain the input data X1 of the layer to be quantized, wherein, if the layer to be quantized is the first layer of the model to be quantized, the calibration data 1 in the calibration data set can be directly used as the input data X1, otherwise, the input data X1 can be determined by the layers before the layer to be quantized based on the calibration data 1, for example, the calibration data 1 can be input into the model to be quantized, and the output data of the previous layer of the layer to be quantized can be used as the input data X1. Then, the input data X1 can be multiplied by the weight matrix W of the layer to be quantized to obtain the first result O1=X1·W.
[0080] Furthermore, the input data X1 can be multiplied by the quantization weight matrix QW1 of the layer to be quantized to determine the second result O2=X1·QW1. The quantization weight matrix QW1 is determined by the initial hyperparameter K (0) , and each K in the weight matrix W (0) The shared quantization parameter is determined by the K of the weight matrix W. (0) The maximum and minimum values among the weight elements and the preset number of quantization bits are determined.
[0081] According to the deviation between the first result O1 and the second result O2, for example, the second norm of the difference between the first result O1 and the second result O2, a third result O3 is determined, and the third result O3 is determined based on the first calibration data 1. The initially set hyperparameter K can be adjusted according to the third result O3. (0) , get the hyperparameter K after the first adjustment (1) , indicating the value of the hyperparameter after the first adjustment K = K (1) .
[0082] For the second calibration data 2, the second (for example, i=2) calibration data 2 can be input into the model to be quantized to obtain the input data X2 of the layer to be quantized, and the input data X2 can be multiplied by the weight matrix W of the layer to be quantized to obtain the first result O1=X2·W. And, the input data X2 is multiplied by the quantization weight matrix QW2 of the layer to be quantized to determine the second result O2=X1·QW2. Among them, the quantization weight matrix QW2 is the hyperparameter K after the first adjustment. (1) , and each K in the weight matrix W (1) The shared quantization parameter is determined by the K of the weight matrix W. (1) The maximum and minimum values of the weight elements and the preset number of quantization bits are determined.
[0083] According to the deviation between the first result O1 and the second result O2, for example, the second norm of the difference between the first result O1 and the second result O2, a third result O3 is determined, and the third result O3 is determined based on the calibration data 2. The hyperparameter K adjusted for the first time can be adjusted according to the third result O3. (1) , get the hyperparameter K after the second adjustment (2) , indicating the value of the hyperparameter after the second adjustment K = K (2) .
[0084] Similarly, for the ith calibration data i, the ith calibration data i can be input into the model to be quantized to obtain the input data X of the layer to be quantized i , the input data X i Multiply it by the weight matrix W of the layer to be quantized to obtain the first result O1=X i W. And, the input data X i and the quantization weight matrix QW of the layer to be quantized i Multiply and determine the second result O2 = X1·QW i Among them, the quantization weight matrix QW i is the hyperparameter K adjusted by the i-1th time (i-1) , and each K in the weight matrix W (i-1) The quantization parameters shared by the weight elements are determined by the hyperparameter K (i-1) represents the value of the hyperparameter after the i-1th adjustment K = K (i -1) The shared quantization parameter is K of the weight matrix W (i-1) The maximum and minimum values of the weight elements and the preset number of quantization bits are determined.
[0085] According to the deviation between the first result O1 and the second result O2, for example, the second norm of the difference between the first result O1 and the second result O2, a third result O3 is determined, and the third result O3 is determined based on the calibration data i. The hyperparameter K adjusted for the i-1th time can be adjusted according to the third result O3. (i-1) , get the hyperparameter K after the i-th adjustment (i) , represents the value of the hyperparameter after the i-th adjustment K = K (i) .
[0086] Repeat this process until all calibration data 1 to calibration data T in the calibration data set are traversed once, and the hyperparameter K after the Tth adjustment is determined based on calibration data T. (T) , the hyperparameter K obtained for the Tth time (T) As the hyperparameter K of the layer to be quantized after adjustment. Among them, the hyperparameter K (T) K = K, which represents the value of the hyperparameter after the Tth adjustment (T)
[0087] In this way, the corresponding hyperparameters can be adaptively adjusted for different types of models to be quantized and different calibration data sets. The target quantization model obtained based on the hyperparameters can achieve higher accuracy and inference speed on the target task.
[0088] In a possible implementation, the hyperparameter adjusted for the i-1th time is adjusted according to the third result to obtain the hyperparameter adjusted for the i-th time, including: when the third result is greater than a first preset threshold, subtracting a preset value from the hyperparameter adjusted for the i-1th time to obtain the hyperparameter adjusted for the i-th time; when the third result is less than a second preset threshold, adding the preset value to the hyperparameter adjusted for the i-1th time to obtain the hyperparameter adjusted for the i-th time; when the third result is less than or equal to the first preset threshold and greater than or equal to the second preset threshold, using the hyperparameter adjusted for the i-1th time as the hyperparameter adjusted for the i-th time.
[0089] For example, Figure 2 As shown, for the third result O3 determined for the first time by the first calibration data 1 (for example, i=1), if the third result O3 is greater than the first preset threshold (for example, 3), the initial hyperparameter K (0) Decrement the preset value (e.g. 16) to get the hyperparameter K after the first adjustment (1) , that is: hyperparameter K (1) = hyperparameter K (0) -Preset value. Among them, the hyperparameter K (1) The minimum can be reduced to 1.
[0090] If the third result O3 is less than the second preset threshold (for example, 0.5), the initial hyperparameter K (0) Add a preset value (e.g. 16) to get the hyperparameter K after the first adjustment (1) , that is: hyperparameter K (1) = hyperparameter K (0) +Preset value. Among them, the hyperparameter K (1) The maximum amount of weight elements added to the weight matrix of this layer.
[0091] If the third result O3 is between the second preset threshold (e.g., 0.5) and the first preset threshold (e.g., 3), that is, the second preset threshold ≤ the third result O3 ≤ the first preset threshold, the initial hyperparameter K (0) As the first tuned hyperparameter K (1) , that is: hyperparameter K (1) = hyperparameter K (0) .
[0092] Similarly, for the third result O3 determined by the i-th calibration data i for the i-th time, if the third result O3 is greater than the first preset threshold (for example, 3), the hyperparameter K adjusted for the i-1th time is (i-1) Decrement the preset value (e.g. 16) to get the hyperparameter K after the i-th adjustment (i) , that is: hyperparameter K (i) = hyperparameter K (i-1) -Preset value. Among them, the hyperparameter K (i) The minimum is reduced to 1.
[0093] If the third result O3 is less than the second preset threshold (for example, 0.5), the hyperparameter K adjusted for the i-1th time (i-1) Add a preset value (e.g. 16) to get the hyperparameter K after the i-th adjustment (i) , that is: hyperparameter K (i) = hyperparameter K (i-1) +Preset value. Among them, the hyperparameter K (i) The maximum amount of weight elements added to the weight matrix of this layer.
[0094] If the third result O3 is between the second preset threshold (e.g., 0.5) and the first preset threshold (e.g., 3), that is, the second preset threshold ≤ the third result O3 ≤ the first preset threshold, the hyperparameter K adjusted for the i-1th time is (i-1) As the hyperparameter K after the i-th adjustment (i) , that is: hyperparameter K (i) = hyperparameter K (i-1) .
[0095] It should be understood that the first preset threshold and the second preset threshold are adjustable parameters. Setting different values will result in different adjusted hyperparameters, which can be set according to actual application scenarios. The present disclosure does not limit the specific values of the first preset threshold and the second preset threshold.
[0096] In this way, adaptive adjustment of hyperparameters can be achieved to adapt to different types of models to be quantized and different calibration data sets. Each pair of third results is compared with a preset threshold (e.g., a first preset threshold and a second preset threshold) once, and the hyperparameters can be intelligently adjusted once to obtain better hyperparameters.
[0097] In a possible implementation, a second result is determined based on the i-th calibration data in the calibration data set and the quantization weight matrix of the layer to be quantized, including: performing block processing on the weight matrix of the layer to be quantized according to the hyperparameter adjusted for the i-1th time to obtain at least one block matrix; performing quantization processing on each block matrix according to a preset number of quantization bits to obtain a block quantization matrix corresponding to each block matrix; splicing each block quantization matrix to obtain a quantization weight matrix; and determining the second result based on the i-th calibration data in the calibration data set and the quantization weight matrix.
[0098] For example, assume that the value of the hyperparameter after the i-1th adjustment is K (i-1) , the weight matrix W of the layer to be quantized is a matrix with M rows and N columns. The weight matrix W has M×N weight elements. The memory can store the M×N weight elements contained in the weight matrix W in a row-major or column-major manner. The order in which the weight elements are stored in the memory can be (i-1) The weight elements are a block matrix, and we get (M×N) / K (i-1) A block matrix.
[0099] For example, if the address of the first weight element in the memory is Addr0, the address corresponding to the first block matrix is Addr0~Addr0+K (i-1) -1, the address corresponding to the second block matrix is Addr0+K (i-1) ~Addr0+2×K (i-1) -1, the address corresponding to the third block matrix is Addr0+2×K (i-1) ~Addr0+3×K (i-1) -1, and so on, until the (M×N) / Kth (i-1) The address corresponding to the block matrix is Addr0+M×NK (i-1) ~Addr0+M×N-1.
[0100] Each block matrix is quantized according to a preset number of quantization bits (e.g., 4). (i-1) weight elements can share the same quantization parameter, (M×N) / K (i-1) The block matrices can correspond to (M×N) / K (i -1) quantization parameters. According to the preset number of quantization bits (for example, 4), and (M×N) / K (i-1) The quantization parameters of the block matrices are (M×N) / K (i-1) The block matrices are quantized respectively to determine the block quantization matrix of each block matrix.
[0101] Then, (M×N) / K can be (i-1) The quantization matrices of each block are concatenated to obtain the quantization weight matrix of the weight matrix W; the second result O2 can be obtained by multiplying the i-th calibration data with the quantization weight matrix.
[0102] In this way, the weight matrix can be dynamically divided and quantized, which is conducive to making the subsequently obtained target quantization model have better accuracy and computational efficiency in the target task, and can reduce the computational complexity of the target quantization model while maintaining the accuracy and performance of the target quantization model.
[0103] In a possible implementation, each block matrix is quantized according to a preset number of quantization bits to obtain a block quantization matrix corresponding to each block matrix, including: determining a quantization parameter of each block matrix according to a preset number of quantization bits, a maximum value and a minimum value of each block matrix; and quantizing the block matrix according to the quantization parameter of each block matrix to obtain a block quantization matrix corresponding to each block matrix. In this way, a block quantization matrix corresponding to each block matrix can be quickly obtained.
[0104] For example, assume that each block matrix can have a hyperparameter K (i) The weight elements can determine the quantization parameters of each block matrix according to the preset number of quantization bits, the maximum value and the minimum value of each block matrix, for example:
[0105] S=(Lmax-Lmin) / 2 quant_bit , Z = 2 quant_bit -Lmax / S (3)
[0106] Among them, S is the K of the block matrix (i) The first parameter shared by the weight elements is used to represent the scaling ratio, and Z is the K of the block matrix (i) The second parameter shared by the weight elements is used to indicate the zero point position, quant_bit indicates the preset quantization position, and Lmax indicates the K of the block matrix. (i) The maximum value of the weight elements, Lmin represents the K (i) The minimum value of the weight elements.
[0107] While determining the first parameter S and the second parameter Z of each block matrix according to formula (3), the first matrix can be constructed according to the first parameter S of each block matrix, the second matrix can be constructed according to the second parameter Z, and the third matrix can be constructed according to the position index of each block matrix.
[0108] According to formula (3), the first parameter S and the second parameter Z of each block matrix are determined. Formula (2) can be used to convert the hyperparameter K(i) Substitute the block matrix of weight elements into weight, substitute the first parameter S into scales, substitute the second parameter Z into zeropoints, and quantize the block matrix to obtain a block quantization matrix corresponding to each block matrix.
[0109] In this way, in step S12, the initial value of the hyperparameter is set for the model to be quantized, indicating the initial granularity of block quantization. After that, a small amount of test set data can be selected in step S13 as the calibration data set of the model to be quantized, and input into the model to be quantized one by one. The model to be quantized continuously infers and optimizes the value of the hyperparameter through the calibration data to achieve adaptive adjustment of the quantization granularity. After traversing the calibration set, each layer to be quantized of each model to be quantized can obtain its own adjusted hyperparameter value K, and take K as the quantization granularity. Subsequently, quantization processing can be performed based on this granularity to obtain the target quantization model.
[0110] In a possible implementation, the calibration data set matches the target task, and the target task includes at least one of image classification, target detection, image segmentation, and speech recognition. Before adjusting the hyperparameters of each layer to be quantized in the model to be quantized according to the calibration data set in step S13, the method further includes: obtaining a calibration data set matching the target task according to the target task; and in step S14, quantizing the model to be quantized according to the adjusted hyperparameters of each layer to be quantized to obtain a target quantized model. After obtaining the target quantized model in step S14, the method further includes: inputting the data to be processed corresponding to the target task into the target quantized model for processing to obtain a processing result, wherein the data to be processed includes at least one of image data, speech data, and text data that matches the task type of the target task.
[0111] In the example, assuming that the model to be quantized is a pre-trained large language model, the initial value of the hyperparameter used to represent the quantization granularity can be set according to its model size, for example, 256. After the model to be quantized and the initial value are selected, a calibration data set can be selected according to the specific task. For example, for a text classification task, a data set consisting of text and category labels can be selected, and a part (for example, 200 cases) can be randomly selected from the data to form a calibration set, which is input into the model to be quantized one by one to obtain the adjusted hyperparameters. The model to be quantized is quantized according to the adjusted hyperparameters to obtain a target quantized model. The target quantized model can be used to perform text classification tasks.
[0112] In this way, for different types of models to be quantized and different calibration datasets, the hyperparameters can be adaptively adjusted to achieve higher accuracy and performance on the target task.
[0113] In summary, the quantization method with adaptive granularity proposed in the embodiment of the present disclosure can set the value K of the hyperparameter so that each K weight elements in the weight matrix of the model to be quantized share a quantization parameter, and can use a small amount of calibration data set to calibrate the model to be quantized. During the calibration process, the value K of the hyperparameter is adaptively adjusted to make it adaptively adjusted for different models and different data to achieve higher quantization accuracy. Through this method, the model to be quantized can be quantized to a lower bit position, and no hardware is required to support sparse computing. It can be integrated into any category of quantization schemes to achieve better reasoning acceleration. Among them, the adaptive process of the hyperparameter K can adapt to different calibration data sets, so that it has higher accuracy and performance on the target task.
[0114] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.
[0115] In addition, the present disclosure also provides a model quantization device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any model quantization method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.
[0116] Figure 3 A block diagram of a model quantization device according to an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the device comprises:
[0117] An acquisition module 31 is used to acquire a model to be quantized, where the model to be quantized includes at least one layer to be quantized;
[0118] A determination module 32, used to determine an initial hyperparameter corresponding to at least one layer to be quantized of the model to be quantized, wherein the hyperparameter represents the number of weight elements in the weight matrix of the layer to be quantized that share the same quantization parameter, and the quantization parameter is used to convert the weight matrix represented by high precision into a low precision representation form;
[0119] An adjustment module 33, used for adjusting the hyper parameters of each to-be-quantized layer in the to-be-quantized model according to the calibration data set;
[0120] The quantization module 34 is used to quantize the model to be quantized according to the adjusted hyper parameters of each layer to be quantized to obtain a target quantized model.
[0121] In one possible implementation, the adjustment module 33 is used to: for the 1st to Tth calibration data in the calibration data set, perform the following steps: determine a first result based on the i-th calibration data and the weight matrix of the layer to be quantized, where i is a positive integer less than or equal to T; determine a second result based on the i-th calibration data in the calibration data set and the quantization weight matrix of the layer to be quantized, where the quantization weight matrix is determined by a preset number of quantization bits, the hyperparameter adjusted for the i-1th time, and the weight matrix of the layer to be quantized; determine a third result based on the deviation between the first result and the second result; adjust the hyperparameter adjusted for the i-1th time based on the third result to obtain the hyperparameter adjusted for the i-th time, wherein the hyperparameter adjusted for the Tth time is the adjusted hyperparameter of the layer to be quantized.
[0122] In a possible implementation, the hyperparameter adjusted for the i-1th time is adjusted according to the third result to obtain the hyperparameter adjusted for the i-th time, including: when the third result is greater than a first preset threshold, subtracting a preset value from the hyperparameter adjusted for the i-1th time to obtain the hyperparameter adjusted for the i-th time; when the third result is less than a second preset threshold, adding the preset value to the hyperparameter adjusted for the i-1th time to obtain the hyperparameter adjusted for the i-th time; when the third result is less than or equal to the first preset threshold and greater than or equal to the second preset threshold, using the hyperparameter adjusted for the i-1th time as the hyperparameter adjusted for the i-th time.
[0123] In a possible implementation, a second result is determined based on the i-th calibration data in the calibration data set and the quantization weight matrix of the layer to be quantized, including: performing block processing on the weight matrix of the layer to be quantized according to the hyperparameter adjusted for the i-1th time to obtain at least one block matrix; performing quantization processing on each block matrix according to a preset number of quantization bits to obtain a block quantization matrix corresponding to each block matrix; splicing each block quantization matrix to obtain a quantization weight matrix; and determining the second result based on the i-th calibration data in the calibration data set and the quantization weight matrix.
[0124] In a possible implementation, each block matrix is quantized according to a preset number of quantization bits to obtain a block quantization matrix corresponding to each block matrix, including: determining a quantization parameter of each block matrix according to a preset number of quantization bits and a maximum value and a minimum value of each block matrix; and quantizing the block matrix according to the quantization parameter of each block matrix to obtain a block quantization matrix corresponding to each block matrix.
[0125] In a possible implementation, the layer to be quantized includes a linear layer, and the quantization parameter includes a first parameter from a first matrix, a second parameter from a second matrix, and a third parameter from a third matrix, and the sizes of the first matrix, the second matrix, and the third matrix are the same and are determined based on the size of the weight matrix and the hyperparameter; wherein, when the value of the hyperparameter is K, it indicates that the number of weight elements sharing the same quantization parameter in the weight matrix of the layer to be quantized is K, and the third parameter at any position in the third matrix is used to represent the position index of the K weight elements in the weight matrix that share the first parameter at the same position in the first matrix and the second parameter at the same position in the second matrix, the first parameter at the same position in the first matrix is used to characterize the scaling ratio of the K weight elements, and the second parameter at the same position in the second matrix is used to characterize the zero position of the K weight elements, and K is greater than or equal to 1 and less than or equal to the total number of weight elements in the weight matrix.
[0126] In one possible implementation, the calibration data set matches a target task, and the target task includes at least one of image classification, target detection, image segmentation, and speech recognition. The device also includes a processing module for: inputting the data to be processed corresponding to the target task into the target quantization model for processing to obtain a processing result, wherein the data to be processed includes at least one of image data, speech data, and text data that matches the task type of the target task.
[0127] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0128] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.
[0129] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.
[0130] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.
[0131] The electronic device may be provided as a terminal, a server, or a device in other forms.
[0132] Figure 4 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 4 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.
[0133] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OS X TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.
[0134] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.
[0135] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.
[0136] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.
[0137] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0138] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0139] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0140] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0141] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0142] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0143] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.
[0144] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0145] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0146] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.
[0147] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A model quantization method, characterized in that: The method comprises: Acquire a model to be quantized, wherein the model to be quantized includes at least one layer to be quantized; Determine an initial hyperparameter corresponding to at least one layer to be quantized of the model to be quantized, wherein the hyperparameter represents the number of weight elements in the weight matrix of the layer to be quantized that share the same quantization parameter, and the hyperparameter is used to perform block quantization on the weight matrix of the layer to be quantized, and the quantization parameter is used to convert a weight matrix represented by high precision into a low precision representation form; According to a calibration data set, adjusting the hyperparameters of each layer to be quantized in the model to be quantized, wherein the calibration data set matches a target task, and the target task includes at least one of image classification, target detection, image segmentation, and speech recognition; According to the adjusted hyper parameters of each layer to be quantized, the model to be quantized is quantized to obtain a target quantized model.
2. The method according to claim 1, characterized in that The process of adjusting the hyper parameters of any layer to be quantized in the model to be quantized includes: For the 1st to Tth calibration data in the calibration data set, perform the following steps: Determine a first result according to the i-th calibration data and the weight matrix of the layer to be quantized, where i is a positive integer less than or equal to T; Determine a second result according to the i-th calibration data in the calibration data set and the quantization weight matrix of the layer to be quantized, wherein the quantization weight matrix is determined by a preset number of quantization bits, the hyperparameter adjusted for the i-1th time, and the weight matrix of the layer to be quantized; Determining a third result according to a deviation between the first result and the second result; The hyperparameter adjusted for the (i-1)th time is adjusted according to the third result to obtain the hyperparameter adjusted for the i-th time, wherein the hyperparameter adjusted for the Tth time is the adjusted hyperparameter of the layer to be quantized.
3. The method according to claim 2, characterized in that Adjusting the hyperparameter adjusted for the (i-1)th time according to the third result to obtain the hyperparameter adjusted for the i-th time includes: When the third result is greater than the first preset threshold, the hyperparameter adjusted for the (i-1)th time is decremented by a preset value to obtain the hyperparameter adjusted for the i-th time; When the third result is less than the second preset threshold, the preset value is added to the hyperparameter adjusted for the (i-1)th time to obtain the hyperparameter adjusted for the i-th time; When the third result is less than or equal to the first preset threshold and greater than or equal to the second preset threshold, the hyperparameter adjusted for the (i-1)th time is used as the hyperparameter adjusted for the i-th time.
4. The method according to claim 2, characterized in that: Determining a second result according to the i-th calibration data in the calibration data set and the quantization weight matrix of the layer to be quantized includes: Performing block processing on the weight matrix of the layer to be quantized according to the hyperparameter adjusted for the (i-1)th time to obtain at least one block matrix; Quantize each block matrix according to a preset number of quantization bits to obtain a block quantization matrix corresponding to each block matrix; Each block quantization matrix is concatenated to obtain a quantization weight matrix; A second result is determined according to the i-th calibration data in the calibration data set and the quantization weight matrix.
5. The method according to claim 4, characterized in that Each block matrix is quantized according to a preset number of quantization bits to obtain a block quantization matrix corresponding to each block matrix, including: Determine a quantization parameter of each block matrix according to a preset number of quantization bits, a maximum value and a minimum value of each block matrix; The block matrix is quantized according to the quantization parameter of each block matrix to obtain a block quantization matrix corresponding to each block matrix.
6. The method according to any one of claims 1 to 5, characterized in that The layer to be quantized includes a linear layer, the quantization parameter includes a first parameter from a first matrix, a second parameter from a second matrix, and a third parameter from a third matrix, the first matrix, the second matrix, and the third matrix have the same size and are determined based on the size of the weight matrix and the hyperparameter; Among them, when the value of the hyperparameter is K, it means that the number of weight elements sharing the same quantization parameter in the weight matrix of the layer to be quantized is K, the third parameter at any position in the third matrix is used to represent the position index of the K weight elements in the weight matrix that share the first parameter at the same position in the first matrix and the second parameter at the same position in the second matrix, the first parameter at the same position in the first matrix is used to characterize the scaling ratio of the K weight elements, the second parameter at the same position in the second matrix is used to characterize the zero position of the K weight elements, and K is greater than or equal to 1 and less than or equal to the total number of weight elements in the weight matrix.
7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: The data to be processed corresponding to the target task is input into the target quantization model for processing to obtain a processing result, wherein the data to be processed includes at least one of image data, voice data, and text data that matches the task type of the target task.
8. A model quantization device, characterized in that: include: An acquisition module, used for acquiring a model to be quantized, wherein the model to be quantized includes at least one layer to be quantized; A determination module, used to determine an initial hyperparameter corresponding to at least one to-be-quantized layer of the to-be-quantized model, wherein the hyperparameter represents the number of weight elements in the weight matrix of the to-be-quantized layer that share the same quantization parameter, and the hyperparameter is used to perform block quantization on the weight matrix of the to-be-quantized layer, and the quantization parameter is used to convert a weight matrix represented by high precision into a low precision representation form; An adjustment module, configured to adjust the hyperparameters of each layer to be quantized in the model to be quantized according to a calibration data set, wherein the calibration data set matches a target task, and the target task includes at least one of image classification, target detection, image segmentation, and speech recognition; The quantization module is used to quantize the model to be quantized according to the adjusted hyper parameters of each layer to be quantized to obtain a target quantized model.
9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.
10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Convolutional neural network quantification method and device, computer and storage medium
CN110363281A
Quantification method and device of neural network model
CN111563589A
Model quantification method and device, electronic equipment and storage medium
CN113554097A
On-chip adaptation method, device and equipment of quantization neural network model and medium
CN115392441A
Neural network model quantification method and device, storage medium and electronic equipment
CN115936092A
