Prediction method, device, electronic device, storage medium and computer program product based on low-rank quantized large model
By performing low-rank decomposition and quantization processing on the linear layer of the large model, the quantization low-rank module is constructed, which solves the quantization error problem caused by excessive parameter quantity of large model, realizes the low-bit representation and acceleration of the model, and improves the prediction speed and Accuracy.
Patent Information
- Application Number
- CN202411376160.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-29
AI Technical Summary
When deploying the equilateral end scene of physical robots and robotic arms, the large model has too high parameters, which affects the learning of quantization parameters, resulting in large quantization errors, affecting the generalization ability and inference speed of the model, and is low in the existing technology. There are fewer bit quantization methods, and it is urgent to quantify the activation and weight simultaneously.
By performing low-rank decomposition of the linear layer parameters of the large model, a quantized low-rank module is constructed, including activation of the quantization layer and the linear layer, parameter quantization and inverse quantization, combined with the scale scaling layer, the low-bit representation of the model is realized, and knowledge distillation is carried out through Adjust parameters to reduce quantization error.
The low-rank quantization acceleration of the model is achieved, the prediction and inference speed is improved, the storage demand is reduced, and the original accuracy level is gradually restored.
Smart Images

Figure CN118886453B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a prediction method, device, electronic device, storage medium and computer program product based on a low-rank quantized large model. Background Art
[0002] Due to the huge amount of computation and parameters of large models, they face great difficulties when deployed in edge scenarios such as physical robots and robotic arms. In related technologies, large model reasoning acceleration methods represented by model quantization mainly focus on model parameter compression, which can compress model parameters to 4 bits almost losslessly, but the activation value usually maintains a high bit width of 16 or 8 bits, and cannot effectively utilize low-bit computing instruction sets (Int4xInt4).
[0003] Since the number of parameters in large models is too high, there are a small number of outliers in both the activation and parameter parts. These outliers affect the learning of quantization parameters, which can easily lead to excessive quantization steps and high rounding errors. On the other hand, when these outliers are quantized to low-bit representation, they can easily cause large truncation errors, which seriously affect the generalization and emergence capabilities of large models. It is precisely because of the influence of outliers that there is little research on the simultaneous low-bit quantization of activations and weights. In related technologies, the quantization error of low bits (int4) is high and the precision loss is large. There is an urgent need to study the low-bit representation method for simultaneous quantization of activations and weights. Summary of the invention
[0004] The prediction method, device, electronic device, storage medium and computer program product based on the low-rank quantized large model provided by the exemplary embodiments of the present disclosure can at least solve the above-mentioned technical problems and other technical problems not mentioned above.
[0005] According to one aspect of the present disclosure, a prediction method based on a low-rank quantized large model is provided, and the prediction method based on the low-rank quantized large model includes: obtaining a prediction model, wherein the prediction model is a large model; performing low-rank decomposition on the parameters of the linear layer in the prediction model to obtain a first low-rank matrix, a second low-rank matrix, and a third low-rank matrix; based on the first low-rank matrix, the second low-rank matrix, and the third low-rank matrix, constructing a quantized low-rank module corresponding to the linear layer, wherein the quantized low-rank module sequentially includes a first activated quantization layer, a first linear layer, a scale scaling layer, a second activated quantization layer, and a second linear layer, and the first The parameters of the linear layer and the second linear layer are obtained by quantizing and dequantizing the first low-rank matrix and the third low-rank matrix respectively, the parameter of the scale scaling layer is the second low-rank matrix, the first activated quantization layer is used to quantize the input of the linear layer, and the second activated quantization layer is used to quantize the output of the scale scaling layer; by replacing the linear layer in the prediction model with the quantized low-rank module, a compressed prediction model is obtained; input data is input into the compressed prediction model to obtain a prediction result corresponding to the input data, wherein the input data includes image data and / or text data.
[0006] Optionally, the method of obtaining a compressed prediction model by replacing the linear layer in the prediction model with the quantized low-rank module includes: obtaining a calibration data set from a training data set of the prediction model; inputting the calibration data set into a teacher network and a student network, respectively, to obtain a first output and a second output, respectively, wherein the teacher network is a Transformer model block in the prediction model, and the student network is obtained by replacing the linear layer in the teacher network with the corresponding quantized low-rank module; calculating the loss based on the first output and the second output; and adjusting the parameters of the quantized low-rank module in the student network based on the loss to obtain the compressed prediction model composed of the student network with adjusted parameters.
[0007] Optionally, the method further includes: after acquiring a calibration data set from the training data set of the prediction model, performing data augmentation on the calibration data set.
[0008] Optionally, the parameters of the first linear layer and the second linear layer are obtained by the following steps: determining the quantization step size of the first linear layer based on the first low-rank matrix and the maximum integer value after quantization; determining the quantization step size of the second linear layer based on the second low-rank matrix and the maximum integer value after quantization; obtaining the first low-bit integer matrix after quantization based on the first low-rank matrix, the quantization step size of the first linear layer, the maximum integer value after quantization, and the minimum integer value after quantization; obtaining the second low-bit integer matrix after quantization based on the second low-rank matrix, the quantization step size of the second linear layer, the maximum integer value after quantization, and the minimum integer value after quantization; dequantizing the first low-bit integer matrix based on the quantization step size of the first linear layer to obtain a first floating-point matrix, wherein the first floating-point matrix is the parameter of the first linear layer; dequantizing the second low-bit integer matrix based on the quantization step size of the second linear layer to obtain a second floating-point matrix, wherein the second floating-point matrix is the parameter of the second linear layer.
[0009] Optionally, the method further includes: determining a quantization step size of the first activated quantization layer based on input data of the linear layer and a maximum integer value after quantization.
[0010] Optionally, the method also includes: determining the quantization step size of the second activated quantization layer based on the input data of the second activated quantization layer and the maximum integer value after quantization.
[0011] According to another aspect of the present disclosure, a prediction device based on a low-rank quantized large model is also provided, the device comprising: a model acquisition unit, configured to: acquire a prediction model, wherein the prediction model is a large model; a parameter decomposition unit, configured to: obtain a first low-rank matrix, a second low-rank matrix, and a third low-rank matrix by performing low-rank decomposition on the parameters of the linear layer in the prediction model; a module construction unit, configured to: construct a quantized low-rank module corresponding to the linear layer based on the first low-rank matrix, the second low-rank matrix, and the third low-rank matrix, wherein the quantized low-rank module sequentially comprises a first activated quantization layer, a first linear layer, a scale scaling layer, a second activated quantization layer, and a second linear layer, the first The parameters of the linear layer and the second linear layer are obtained by quantizing and dequantizing the first low-rank matrix and the third low-rank matrix respectively, the parameter of the scale scaling layer is the second low-rank matrix, the first activated quantization layer is used to quantize the input of the linear layer, and the second activated quantization layer is used to quantize the output of the scale scaling layer; the model compression unit is configured to: obtain a compressed prediction model by replacing the linear layer in the prediction model with the quantized low-rank module; the result prediction unit is configured to: input input data into the compressed prediction model to obtain a prediction result corresponding to the input data, wherein the input data includes image data and / or text data.
[0012] According to another aspect of an embodiment of the present disclosure, an electronic device is also provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute any of the prediction methods based on the low-rank quantized large model as described above.
[0013] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is also provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute any of the prediction methods based on the low-rank quantized large model as described above.
[0014] According to another aspect of an embodiment of the present disclosure, a system is also provided, comprising at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute any of the prediction methods based on the low-rank quantized large model as described above.
[0015] According to another aspect of an embodiment of the present disclosure, a computer program product is also provided, including a computer program / instruction, which, when executed by a processor, implements the prediction method based on a low-rank quantized large model as described in any one of the above.
[0016] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0017] According to the prediction method, device, electronic device, storage medium and computer program product based on the low-rank quantized large model disclosed in the present invention, the linear layer parameters (weights) in the model can be low-rank decomposed and the activation and weight can be jointly quantized, so that the low-bit representation can be performed while reducing the number of parameters. In addition, the intermediate activations after calculation of a low-rank factor (matrix) are also quantized into low-bit representation, which can achieve hardware-friendly model low-rank quantization acceleration, improve the speed of model prediction reasoning, and reduce the storage requirements for model deployment.
[0018] In addition, by adjusting the parameters in the quantized low-rank modules obtained based on each linear layer in the model, block-level knowledge distillation can be achieved, which can enable the compressed model to gradually restore its original accuracy level. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0020] Figure 1 A schematic diagram showing a flow chart of a prediction method based on a low-rank quantized large model in an exemplary embodiment of the present disclosure;
[0021] Figure 2 A schematic diagram showing a quantized low-rank module constructed in an exemplary embodiment of the present disclosure;
[0022] Figure 3 A schematic diagram showing block-level knowledge distillation based on quantized low-rank decomposition in an exemplary embodiment of the present disclosure is shown;
[0023] Figure 4 A block diagram of a prediction device based on a low-rank quantized large model in an exemplary embodiment of the present disclosure is shown;
[0024] Figure 5 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.
[0027] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0028] In order to solve the above problems, the present disclosure provides a prediction method, device, electronic device, storage medium and computer program product based on a low-rank quantized large model, which can perform low-rank decomposition on the linear layer parameters (weights) in the model and jointly quantize the activation and weights, thereby reducing the number of parameters and performing low-bit representation. In addition, the intermediate activations after calculation of a low-rank factor (matrix) are also quantized into low-bit representation, which can achieve hardware-friendly model low-rank quantization acceleration, improve the speed of model prediction reasoning, and reduce the storage requirements for model deployment.
[0029] Next, we will refer to Figures 1 to 5 The prediction method, device, electronic device, storage medium and computer program product based on the low-rank quantized large model disclosed in the present invention are specifically described.
[0030] Figure 1 A schematic flow chart of a prediction method based on a low-rank quantized large model in an exemplary embodiment of the present disclosure is shown.
[0031] Reference Figure 1 In step 101, a prediction model is obtained, wherein the prediction model is a large model.
[0032] According to an exemplary embodiment of the present disclosure, the prediction model may be a large model (also called a foundation model) including a multi-head self-attention unit (MHSA) and a feed-forward-network (FFN), such as a large language model (Large Language Model), etc. It is understandable that the prediction model may also be other models including a linear layer.
[0033] According to an exemplary embodiment of the present disclosure, the prediction model can be applied to the following fields including but not limited to: text classification, machine translation, text generation, language understanding (such as question-answering system, semantic role labeling, etc.) in Natural Language Processing (NLP); image classification, target detection, image generation (such as style transfer, image super-resolution, etc.), video processing (video classification, action recognition, video generation, etc.) in Computer Vision (CV); speech recognition and synthesis; multimodal learning (combining data in multiple modalities such as text, image, audio, etc. for learning and reasoning, such as visual question-answering, video description, etc.); reinforcement learning, etc.
[0034] In step 102, a first low-rank matrix, a second low-rank matrix, and a third low-rank matrix are obtained by performing low-rank decomposition on the parameters of the linear layer in the prediction model.
[0035] According to an exemplary embodiment of the present disclosure, the weights (parameters) of each linear layer (Linear Layer) in the multi-head self-attention unit and the feedforward propagation unit in the large model can be low-rank decomposed; for example, the parameter W of the linear layer can be decomposed into low-rank factors (low-rank matrices) U (first low-rank matrix), S (second low-rank matrix) and V (third low-rank matrix).
[0036] In step 103, based on the first low-rank matrix, the second low-rank matrix, and the third low-rank matrix, a quantized low-rank module corresponding to the linear layer is constructed, wherein the quantized low-rank module sequentially includes a first activated quantized layer, a first linear layer, a scale scaling layer, a second activated quantized layer, and a second linear layer, the parameters of the first linear layer and the second linear layer are obtained by quantizing and dequantizing the first low-rank matrix and the third low-rank matrix, respectively, the parameter of the scale scaling layer is the second low-rank matrix, the first activated quantized layer is used to quantize the input of the linear layer, and the second activated quantized layer is used to quantize the output of the scale scaling layer. According to an exemplary embodiment of the present disclosure, U and V can be low-bit quantized to obtain a quantization step size and , and perform inverse quantization to obtain a floating-point matrix represented by quantization accuracy and , to convert the floating point matrix and as parameters of the first and second linear layers, respectively.
[0037] According to an exemplary embodiment of the present disclosure, a quantized low-rank calculation module may be constructed first. A quantization layer Quant1 (the first activated quantization layer) may be added to the input X of the current linear layer in the prediction model to perform low-bit quantization on the input X. The quantization step size may be expressed as ; Next, you can add a linear calculation layer Linear1 (the first linear layer), the parameters of this layer Can be initialized as a quantized low-rank matrix , the quantization step size can be initialized as ; Next, you can add a scale layer (Scale Layer) Scale1, the parameters of this layer can be initialized to the diagonal value of the S matrix; then, a quantization layer Quant2 (the second activation quantization layer) can be added to quantize the output of the scale scaling layer, and the quantization step size can be expressed as ; Next, you can add a linear calculation layer Linear2 (the second linear layer), the parameters of this layer Can be initialized as a quantized low-rank matrix , the quantization step size can be initialized as The calculation order of the entire low-rank calculation module is Quant1, Linear1, Scale1, Quant2, Linear2.
[0038] Figure 2 A schematic diagram showing a quantized low-rank module constructed in an exemplary embodiment of the present disclosure.
[0039] Reference Figure 2 According to an exemplary embodiment of the present disclosure, Figure 2 A linear layer (i layer) included in the above prediction model can obtain a quantized low-rank factor U, a factor S, and a quantized low-rank factor V based on the parameter W of the linear layer, and add an activation quantization layer for quantizing the input activation X and an activation quantization layer for quantizing the output processed by the factor S, so as to replace the linear layer with the corresponding quantized low-rank module 201, so that the calculation architecture of inputting the activation X into the linear layer to obtain the output of the current layer in the original prediction model is replaced with Figure 2The computing architecture below: After the input activation X is activated and quantized (the first activation quantization layer) to obtain the first output, the first output is calculated based on the quantized low-rank factor U (the first linear layer) to obtain the second output, and the second output is calculated based on the factor S (scaling layer) to obtain the third output, and the third output is activated and quantized (the second activation quantization layer) to obtain the fourth output, and the output calculated based on the quantized low-rank factor V (the second linear layer) is used as the current layer output of the corresponding linear layer. The output of the previous linear layer can also be used as the input of the current layer (activation X).
[0040] According to an exemplary embodiment of the present disclosure, the parameters of the first linear layer and the second linear layer can be obtained by the following steps: determining the quantization step size of the first linear layer based on the first low-rank matrix and the maximum integer value after quantization; determining the quantization step size of the second linear layer based on the second low-rank matrix and the maximum integer value after quantization; obtaining the quantized first low-bit integer matrix based on the first low-rank matrix, the quantization step size of the first linear layer, the maximum integer value after quantization, and the minimum integer value after quantization; obtaining the quantized second low-bit integer matrix based on the second low-rank matrix, the quantization step size of the second linear layer, the maximum integer value after quantization, and the minimum integer value after quantization; dequantizing the first low-bit integer matrix based on the quantization step size of the first linear layer to obtain a first floating-point matrix, wherein the first floating-point matrix is a parameter of the first linear layer; dequantizing the second low-bit integer matrix based on the quantization step size of the second linear layer to obtain a second floating-point matrix, wherein the second floating-point matrix is a parameter of the second linear layer.
[0041] According to an exemplary embodiment of the present disclosure, in the inference acceleration method of a large model, it can be mainly achieved by compressing the model parameters. Although the model parameters can be compressed to 4 bits almost losslessly, the activation value usually maintains a high bit width of 16 or 8 bits, and the low-bit calculation instruction set (Int4xInt4) cannot be effectively utilized. Therefore, the present disclosure proposes a joint quantization method of activation-weight based on quantized low-rank matrix decomposition, which can effectively utilize the low-bit calculation instruction set, and the calculation steps may include but are not limited to the following:
[0042] The linear layer weight W of the prediction model can be decomposed into three low-rank matrices U, S, and V by singular value decomposition (SVD), where W , U , , , where S is a diagonal matrix, only the matrix diagonals have values and the rest are zero.
[0043] You can do low-bit fixed-point quantization on U and V:
[0044] = ) /
[0045] = ) /
[0046] , )
[0047] , )
[0048] in, Represents the maximum integer value after quantization, Represents the minimum integer value after quantization, () represents the absolute value function, () represents the maximum value function, and Indicates the quantization step size of U and V, and Represent the low-bit integer matrices after quantization of U and V respectively.
[0049] Can and Perform inverse quantization to obtain a floating-point matrix under quantization accuracy and :
[0050] =
[0051] =
[0052] You can use floating point matrices and as parameters of the first and second linear layers, respectively.
[0053] According to an exemplary embodiment of the present disclosure, since the intermediate calculation results after low-rank decomposition are of floating-point type, fixed-point calculations may not be performed. Therefore, when constructing a quantized low-rank calculation module, quantization of the input data and the intermediate calculation results can be introduced to achieve quantized fixed-point representations of both the input and intermediate layer representations, so as to better utilize the hardware fixed-point calculation instruction set and achieve the acceleration effect of low-rank quantization and fixed-point quantization.
[0054] For a linear layer in a prediction model, assume that O is the model output, X is the input, and W is the linear layer weight.
[0055] According to an exemplary embodiment of the present disclosure, when constructing a quantized low-rank calculation module, a quantization layer Quant1 (a first activated quantization layer) may be first added to the input X to perform low-bit quantization on the input X.
[0056] According to an exemplary embodiment of the present disclosure, a quantization step size of a first activated quantization layer may be determined based on input data of a linear layer and a maximum integer value after quantization.
[0057] For example, the quantization step size of the quantization layer Quant1 can be expressed as , and can be initialized to = / , the calculation process of this layer can be expressed as:
[0058] Quant1(X)= , )
[0059] Next, we can add a linear calculation layer Linear1 (the first linear layer), the parameters of which are Can be initialized as a quantized low-rank matrix , the quantization factor of its parameters (weights) is initialized as , the forward calculation process of this layer can be expressed as:
[0060] Linear1( , )= , )
[0061] in, Represents the input of this calculation layer and is also the output of the previous quantization layer Quant1.
[0062] Next, you can add a scale layer (Scale Layer) Scale1, the parameters of this layer It can be initialized to the diagonal value of the S matrix. The forward calculation process of this layer can be expressed as:
[0063] Scale1 ( , )=
[0064] in, It can represent the input of this calculation layer and also the output of the previous Linear1 layer.
[0065] Next, a quantization layer Quant2 (second activation quantization layer) can be added to quantize the output of the scale scaling layer.
[0066] According to an exemplary embodiment of the present disclosure, a quantization step size of the second activated quantization layer may be determined based on input data of the second activated quantization layer and a quantized maximum integer value.
[0067] For example, the quantization step size of the quantization layer Quant2 can be expressed as , and can be initialized to = ) / , the calculation process of this layer can be expressed as:
[0068] Quant2( , )= , )
[0069] in, Represents the input of this calculation layer and is also the output of the previous scaling layer Scale1.
[0070] Next, we can add a linear calculation layer Linear2 (the second linear layer), the parameters of which are Can be initialized as a quantized low-rank matrix , the quantization factor of its weight can be initialized as , the forward calculation process of this layer can be expressed as:
[0071] Linear2( , )= , )
[0072] in, Represents the input of this calculation layer and is also the output of the previous quantization layer Quant2.
[0073] The calculation order of the entire quantized low-rank calculation module can be Quant1, Linear1, Scale1, Quant2, Linear2; the forward calculation can be expressed as:
[0074] Linear2(Quant2(Scale1 (Linear1(Quant1(X), ), ), ), )
[0075] Return to reference Figure 1 In step 104, a compressed prediction model is obtained by replacing the linear layer in the prediction model with a quantized low-rank module.
[0076] According to an exemplary embodiment of the present disclosure, replacing each linear layer in the prediction model with a corresponding quantized low-rank module can compress the prediction model to accelerate model reasoning, reduce storage requirements during model deployment, etc. According to an exemplary embodiment of the present disclosure, after constructing a quantized low-rank module (quantized low-rank calculation module), the activation and weight can be jointly quantized while performing weight low-rank calculation, and hardware-friendly low-rank quantization acceleration can be achieved. However, the introduction of intermediate activation quantization may also increase the model error caused by the low-rank quantization factor. Due to the nonlinear characteristics of the intermediate layer quantization operation, the closed-form analytical solution of the quantized low-rank factor may not be obtained. To this end, the idea of local knowledge distillation can be introduced, and a Transformer model block (Block) in the prediction model is used as a knowledge distillation unit, and each block contains a multi-head attention unit and a feedforward network unit. For example, specifically, the Transformer model block (Block) of the original floating-point parameters can be used as a teacher network; then the teacher network can be copied as a student network, and each linear layer (Linear) in the student network can be replaced with a corresponding quantized low-rank calculation module, and each quantized low-rank module can form a block-level knowledge distillation unit. Therefore, by constructing a block-level knowledge distillation unit, the quantized low-rank weights (parameters) within the block can be learned under the local supervision of the output results of the block-level knowledge distillation unit.
[0077] According to an exemplary embodiment of the present disclosure, a compressed prediction model is obtained by replacing a linear layer in a prediction model with a corresponding quantized low-rank module, which may include but is not limited to the following steps: obtaining a calibration data set from a training data set of the prediction model; inputting the calibration data set into a teacher network and a student network, respectively, to obtain a first output and a second output, respectively, wherein the teacher network is a Transformer model block in the prediction model, and the student network is obtained by replacing a linear layer in the teacher network with a corresponding quantized low-rank module; calculating the loss based on the first output and the second output; adjusting the parameters of the quantized low-rank module in the student network based on the loss to obtain a compressed prediction model composed of the student network with adjusted parameters.
[0078] According to an exemplary embodiment of the present disclosure, the learning of quantization parameters and low-rank parameters can be performed based on the Adam (Adaptive Moment Estimation) optimizer. First, the calibration data set can be input into the teacher network and the student network to obtain the block-level outputs Y1 (first output) and Y2 (second output), respectively. The Frobenius norm between the two can be calculated as the loss. Based on the loss, the gradient backpropagation at the block level can be performed, and the Adam optimizer can be used to optimize the quantization low-rank calculation module. , , , and the weights and The quantization step size and , quantization step size of quantization layer (Quant1 and Quant2) and Fine tune the parameters.
[0079] This process can gradually restore the model to its original accuracy level by making slight adjustments to the quantized low-rank factors U, V and parameter S. In the block-level knowledge distillation unit, each parameter in each block of the model can be gradient updated through back propagation to achieve fine-tuning of the model.
[0080] For example, after inputting the calibration dataset into the teacher network and the student network and obtaining the block-level outputs Y1 and Y2 respectively, the Frobenius norm between the two can be calculated as the loss, and the block-level gradient backpropagation can be performed based on the loss. The Adam optimizer can also be used to learn the quantization of the low-rank computing module. , , , and the weights and The quantization step size and , quantization step size of quantization layer (Quant1 and Quant2) and Fine tune the parameters.
[0081] Figure 3 A schematic diagram showing block-level knowledge distillation based on quantized low-rank decomposition in an exemplary embodiment of the present disclosure is shown.
[0082] Reference Figure 3 According to an exemplary embodiment of the present disclosure, Figure 3The above prediction model may contain a Transformer model block with multiple linear layers (i layer, i+1 layer, etc.), namely the teacher network 301, in which each linear layer can be replaced with a corresponding quantized low-rank module 201 to obtain a student network 302. By inputting the calibration data set into the teacher network and the student network respectively, the parameters in the student network can be fine-tuned based on the loss. Each Transformer model block can be individually parameterized to achieve block-level knowledge distillation.
[0083] According to an exemplary embodiment of the present disclosure, after a calibration data set is acquired from a training data set of a prediction model, data augmentation may be performed on the calibration data set.
[0084] According to an exemplary embodiment of the present disclosure, a small amount of unlabeled data may be collected as a calibration data set, and data augmentation may be performed through methods such as random cropping.
[0085] According to an exemplary embodiment of the present disclosure, a small amount of data can be obtained from the original training set of the prediction model as a calibration data set. For the text data, data augmentation can be performed on the text data through methods such as synonym replacement, back translation (translation into another language and then translation back), keyboard enhancer (random insertion of wrong characters to simulate keyboard errors), etc.; for the image data, data augmentation can be performed through methods such as random scaling, random cropping, random horizontal flipping, random vertical inversion, etc., to obtain a calibration data set after data augmentation.
[0086] Return to reference Figure 1 In step 105, the input data is input into the compressed prediction model to obtain a prediction result corresponding to the input data, wherein the input data includes image data and / or text data.
[0087] According to an exemplary embodiment of the present disclosure, the compressed prediction model can reduce the storage space requirement for deployment and improve the prediction speed.
[0088] According to the exemplary embodiments of the present disclosure, the present disclosure can refer to the low-rank representation and knowledge distillation method, decompose the model parameters into quantized low-rank factors, perform low-bit representation while reducing the amount of parameters, and construct a low-bit joint quantization of activation-weight, and quantize the intermediate activation after a low-rank factor calculation to a low-bit representation. Since low-bit quantization can be considered as a nonlinear transformation, the product of multiple low-rank factors is equivalent to deepening the network depth, and also provides a higher representation capability for low-rank low-bit representation. Therefore, constructing multiple quantized low-rank factors has a lower quantization error than directly quantizing the original parameters at a low bit, and also provides the possibility of quantizing activations and weights to low bits (Int4) at the same time. However, the introduction of intermediate activation quantization may also increase the difficulty of learning low-rank quantization factors. Due to the nonlinear characteristics of the intermediate layer quantization operation, it may be impossible to obtain a closed-form analytical solution for the quantized low-rank factor. To this end, the present disclosure can introduce the idea of local knowledge distillation, take a Transformer model block (Block) as a knowledge distillation unit, and learn the low-rank quantization weights within the block under the local supervision of the block-level output results.
[0089] Figure 4 A block diagram of a prediction device based on a low-rank quantized large model in an exemplary embodiment of the present disclosure is shown.
[0090] Reference Figure 4 An exemplary embodiment of the present disclosure also provides a prediction device 400 based on a low-rank quantized large model, which may include but is not limited to a model acquisition unit 401, a parameter decomposition unit 402, a module construction unit 403, a model compression unit 404, and a result prediction unit 405.
[0091] The model acquisition unit 401 can acquire a prediction model, wherein the prediction model is a large model.
[0092] The parameter decomposition unit 402 may obtain a first low-rank matrix, a second low-rank matrix, and a third low-rank matrix by performing low-rank decomposition on the parameters of the linear layer in the prediction model.
[0093] The module construction unit 403 can construct a quantized low-rank module corresponding to the linear layer based on the first low-rank matrix, the second low-rank matrix, and the third low-rank matrix, wherein the quantized low-rank module sequentially includes a first activated quantization layer, a first linear layer, a scale scaling layer, a second activated quantization layer, and a second linear layer, the parameters of the first linear layer and the second linear layer are obtained by quantizing and dequantizing the first low-rank matrix and the third low-rank matrix, respectively, the parameter of the scale scaling layer is the second low-rank matrix, the first activated quantization layer is used to quantize the input of the linear layer, and the second activated quantization layer is used to quantize the output of the scale scaling layer.
[0094] The model compression unit 404 may obtain a compressed prediction model by replacing the linear layer in the prediction model with a quantized low-rank module.
[0095] The result prediction unit 405 may input input data into the compressed prediction model to obtain a prediction result corresponding to the input data, wherein the input data includes image data and / or text data.
[0096] It can be understood that in the exemplary embodiment of the prediction device 400 based on the low-rank quantized large model, the specific implementation process is roughly the same as the exemplary embodiment of the prediction method based on the low-rank quantized large model, which will not be repeated here. The prediction device 400 based on the low-rank quantized large model can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, these devices may correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. In addition, one or more functions implemented by these devices may also be uniformly executed by components in physical entity devices (for example, processors, clients, or servers, etc.).
[0097] Figure 5 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.
[0098] Reference Figure 5 The electronic device 500 includes at least one memory 501 and at least one processor 502, wherein the at least one memory 501 stores a set of computer executable instructions. When the computer executable instruction set is executed by the at least one processor 502, a prediction method based on a low-rank quantized large model according to an exemplary embodiment of the present disclosure is performed.
[0099] As an example, the electronic device 500 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device 500 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 500 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.
[0100] In the electronic device 500, the processor 502 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.
[0101] The processor 502 may execute instructions or codes stored in the memory 501, wherein the memory 501 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.
[0102] The memory 501 may be integrated with the processor 502, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 501 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 501 and the processor 502 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 502 can read files stored in the memory.
[0103] In addition, the electronic device 500 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 500 may be connected to each other via a bus and / or a network.
[0104] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the above-mentioned prediction method based on the low-rank quantized large model.
[0105] Examples of computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers. It should be noted that the instructions can also be used to execute additional steps in addition to the above steps or to perform more specific processing when executing the above steps. These additional steps and further processing contents have been mentioned in the description of the relevant methods, so they will not be repeated here to avoid repetition.
[0106] Another embodiment of the present disclosure relates to a system comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute the above-mentioned prediction method based on the low-rank quantized large model.
[0107] It should be noted that the system according to the exemplary embodiments of the present disclosure can completely rely on the execution of computer programs or instructions to realize corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., lib library) to realize the corresponding functions.
[0108] On the other hand, when the above-mentioned system is implemented in software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.
[0109] According to an exemplary embodiment of the present disclosure, the storage device may be integrated with the computing device, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage device may include an independent device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage device and the computing device may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the computing device can read instructions stored in the storage device.
[0110] Another embodiment of the present disclosure relates to a computer program product, including a computer program / instruction, which, when executed by a processor, implements the prediction method based on a low-rank quantized large model as described in any one of the above.
[0111] According to the prediction method, device, electronic device, storage medium and computer program product based on the low-rank quantized large model provided by the present invention, the linear layer parameters (weights) in the model can be low-rank decomposed and the activation and weight can be jointly quantized, so that the low-bit representation can be performed while reducing the number of parameters. In addition, the intermediate activations after calculation of a low-rank factor (matrix) are also quantized into low-bit representation, which can achieve hardware-friendly model low-rank quantization acceleration, improve the speed of model prediction reasoning, and reduce the storage requirements for model deployment.
[0112] In addition, by adjusting the parameters in the quantized low-rank modules obtained based on each linear layer in the model, block-level knowledge distillation can be achieved, which can enable the compressed model to gradually restore its original accuracy level.
[0113] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.
Claims
1. A prediction method based on a low-rank quantized large model, characterized in that: The prediction method based on the low-rank quantized large model includes: Obtaining a prediction model, wherein the prediction model is a large model; By performing low-rank decomposition on the parameters of the linear layer in the prediction model, a first low-rank matrix, a second low-rank matrix, and a third low-rank matrix are obtained; Based on the first low-rank matrix, the second low-rank matrix, and the third low-rank matrix, a quantized low-rank module corresponding to the linear layer is constructed, wherein the quantized low-rank module sequentially includes a first activated quantization layer, a first linear layer, a scale scaling layer, a second activated quantization layer, and a second linear layer, the parameters of the first linear layer and the second linear layer are obtained by quantizing and dequantizing the first low-rank matrix and the third low-rank matrix, respectively, the parameter of the scale scaling layer is the second low-rank matrix, the first activated quantization layer is used to quantize the input of the linear layer, and the second activated quantization layer is used to quantize the output of the scale scaling layer; Obtaining a compressed prediction model by replacing the linear layer in the prediction model with the quantized low-rank module; Input data is input into the compressed prediction model to obtain a prediction result corresponding to the input data, wherein the input data includes image data and / or text data.
2. The prediction method based on a low-rank quantized large model according to claim 1, characterized in that: The method of obtaining a compressed prediction model by replacing the linear layer in the prediction model with the quantized low-rank module comprises: Acquire a calibration data set from the training data set of the prediction model; Inputting the calibration data set into a teacher network and a student network respectively, and obtaining a first output and a second output respectively, wherein the teacher network is a Transformer model block in the prediction model, and the student network is obtained by replacing the linear layer in the teacher network with the corresponding quantized low-rank module; calculating a loss based on the first output and the second output; The parameters of the quantized low-rank module in the student network are adjusted based on the loss to obtain the compressed prediction model composed of the student network with adjusted parameters.
3. The prediction method based on a low-rank quantized large model according to claim 1, characterized in that: The method further comprises: After obtaining the calibration data set from the training data set of the prediction model, data augmentation is performed on the calibration data set.
4. The prediction method based on a low-rank quantized large model according to claim 1, characterized in that: The parameters of the first linear layer and the second linear layer are obtained by the following steps: Determine a quantization step size of a first linear layer based on the first low-rank matrix and the quantized maximum integer value; Determine a quantization step size of a second linear layer based on the second low-rank matrix and the quantized maximum integer value; Obtaining a quantized first low-bit integer matrix based on the first low-rank matrix, the quantization step size of the first linear layer, the maximum integer value after quantization, and the minimum integer value after quantization; Obtaining a quantized second low-bit integer matrix based on the second low-rank matrix, the quantization step size of the second linear layer, the maximum integer value after quantization, and the minimum integer value after quantization; Dequantizing the first low-bit integer matrix based on the quantization step size of the first linear layer to obtain a first floating-point matrix, wherein the first floating-point matrix is a parameter of the first linear layer; The second low-bit integer matrix is dequantized based on the quantization step size of the second linear layer to obtain a second floating-point matrix, wherein the second floating-point matrix is a parameter of the second linear layer.
5. The prediction method based on a low-rank quantized large model according to claim 1, characterized in that: The method further comprises: The quantization step size of the first activated quantization layer is determined based on the input data of the linear layer and the quantized maximum integer value.
6. The prediction method based on a low-rank quantized large model according to claim 1, characterized in that: The method further comprises: A quantization step size of the second activated quantization layer is determined based on the input data of the second activated quantization layer and the quantized maximum integer value.
7. A prediction device based on a low-rank quantized large model, characterized in that: The device based on the low-rank quantized large model includes: The model acquisition unit is configured to: acquire a prediction model, wherein the prediction model is a large model; A parameter decomposition unit is configured to: obtain a first low-rank matrix, a second low-rank matrix, and a third low-rank matrix by performing low-rank decomposition on the parameters of the linear layer in the prediction model; A module construction unit is configured to: construct a quantized low-rank module corresponding to the linear layer based on the first low-rank matrix, the second low-rank matrix, and the third low-rank matrix, wherein the quantized low-rank module sequentially comprises a first activated quantization layer, a first linear layer, a scale scaling layer, a second activated quantization layer, and a second linear layer, the parameters of the first linear layer and the second linear layer are obtained by quantizing and dequantizing the first low-rank matrix and the third low-rank matrix, respectively, the parameter of the scale scaling layer is the second low-rank matrix, the first activated quantization layer is used to quantize the input of the linear layer, and the second activated quantization layer is used to quantize the output of the scale scaling layer; A model compression unit is configured to: obtain a compressed prediction model by replacing the linear layer in the prediction model with the quantized low-rank module; The result prediction unit is configured to: input input data into the compressed prediction model to obtain a prediction result corresponding to the input data, wherein the input data includes image data and / or text data.
8. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the prediction method based on the low-rank quantized large model as described in any one of claims 1-6.
9. A computer-readable storage medium storing instructions, characterized in that: When the instruction is executed by at least one processor, the at least one processor is prompted to perform the prediction method based on the low-rank quantized large model as described in any one of claims 1-6.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by the processor, the prediction method based on the low-rank quantized large model as described in any one of claims 1-6 is implemented.
Citation Information
Patent Citations
Large language model compression method and system based on low-rank quantization
CN118657180A
Low-Rank Adaptation of Neural Network Models
US20220383126A1
