Big language model binarization quantification method and system based on instructive alternate optimization
By employing a binarization quantization method for large language models based on guided alternating optimization, and utilizing singular value decomposition and Hessian matrix to evaluate weight sensitivity, important regions are divided and iterative row-column alternating optimization is performed. This solves the problems of low quantization accuracy and poor compatibility of large language models when deployed on edge devices, achieving high-precision and high-compatibility quantization results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGDONG UNIV OF TECH
- Filing Date
- 2026-01-15
- Publication Date
- 2026-05-19
AI Technical Summary
When existing large language models are deployed on edge devices, they suffer from low quantization accuracy and poor compatibility. Traditional quantization strategies cannot handle the differences in weight distribution across different layers and channels with fine granularity, resulting in accuracy loss in low-bit representation and instability in cross-device deployment.
A binary quantization method for large language models based on guided alternating optimization is adopted. The weight sensitivity is evaluated by singular value decomposition and Hessian matrix. The weight matrix is divided into important and unimportant regions. The scaling factor of the weight matrix is optimized by alternating first-order and second-order row and column optimization to reduce quantization error.
It significantly improves the accuracy of large language models after low-bit quantization, making it suitable for edge deployment scenarios and enhancing the model's cross-hardware adaptability and quantization accuracy.
Smart Images
Figure CN122065892A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of large language model deployment, and more specifically, to a method and system for binarization and quantization of large language models based on guided alternation optimization. Background Technology
[0002] Large Language Models (LLMs) have seen rapid development in the field of natural language processing in recent years, demonstrating outstanding performance in tasks such as text generation, code assistance, knowledge-based question answering, and complex reasoning. As the model size continues to expand to billions or even trillions of parameters, while their algorithmic capabilities have significantly improved, the resulting demand for GPU memory, memory consumption, and computational bandwidth has also increased dramatically, making it difficult to deploy the models directly on edge devices, local terminals, and in scenarios with limited computing power.
[0003] Currently, mainstream large language models generally use 32-bit or 16-bit floating-point formats to represent weights and activation values. While this high-precision representation ensures good inference accuracy, it significantly increases model size and inference cost on resource-constrained hardware. To address this contradiction, low-bit quantization technology has gradually become an important direction for model compression. By converting floating-point parameters into 8-bit, 4-bit, or even 2-bit fixed-point representations, low-bit quantization technology can significantly reduce storage requirements and improve inference efficiency, making it a key technology driving the deployment of LLMs to edge devices.
[0004] However, directly using an initial uniform quantization strategy often weakens model performance because there are significant differences in the weight distribution, activation dynamic range, and sensitivity to inference results across different layers and channels. Traditional methods cannot address these differences with fine-grained processing, resulting in non-negligible accuracy loss under low-bit constraints. Although improved schemes such as dynamic range quantization, symmetric / asymmetric quantization, and channel-by-channel quantization have emerged in recent years, which have improved the usability of quantized models to some extent, the following shortcomings still exist: (1) There is a lack of systematic evaluation of the importance of parameters, and the quantization strategy is difficult to match with the characteristics of the model structure; (2) The error generated by low-bit quantization is difficult to be effectively compensated in key areas; (3) There are large differences in the computing units between different edge hardware, and the stability of quantized models deployed across devices is poor. Summary of the Invention
[0005] To address the issues of low quantization accuracy and poor compatibility with edge deployment in existing technologies for large language models, this invention proposes a binarization quantization method and system for large language models based on guided alternating optimization. This method ensures that the quantized large language model maintains high accuracy while guaranteeing a high compression ratio, and also possesses good cross-hardware adaptability.
[0006] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: Firstly, this application proposes a binarization and quantization method for large language models based on guided alternation optimization, comprising the following steps: S1: Obtain the weight parameters of the large language model to be quantized, where the weight parameters correspond to the weight matrix of the large language model; S2: Decompose the weight matrix to obtain row guidance vectors and column guidance vectors; the row guidance vectors correspond to the initial values of the row vector scaling factors, and the column guidance vectors correspond to the initial values of the column vector scaling factors. S3: Based on the sensitivity of each weight element in the weight matrix, divide the weight matrix into important and unimportant regions; S4: Based on the initial values of the row vector scaling factor and the column vector scaling factor, obtain the analytical solutions of the row vector scaling factor and the column vector scaling factor respectively, and obtain the first-order binary matrix of all regions of the weight matrix. S5: Based on the analytical solutions of the row vector scaling factor, the analytical solutions of the column vector scaling factor, and the first-order binarized matrix, perform first-order row-column alternation optimization iteration on the row vector scaling factor and the column vector scaling factor to obtain the first-order reconstruction weight matrix. S6: Based on the first-order reconstruction weight matrix, obtain the residual matrix corresponding to the important region, and binarize the residual matrix to obtain the second-order binarized matrix; S7: Based on the second-order binary matrix, perform first- and second-order row and column alternation optimization iterations on important regions to obtain the second-order reconstruction weight matrix, and use the second-order reconstruction weight matrix as the weight parameters after the large language model is quantized.
[0007] Preferably, in S2, the weight matrix is decomposed to obtain row guidance vectors and column guidance vectors, the process of which is as follows: S21: For the weight matrix Perform singular value decomposition, wherein the singular value decomposition satisfies the expression: ;in, Represents a left singular matrix; Represents a right singular matrix; Represents a real matrix; Given a diagonal matrix, sort the singular values in the diagonal matrix in descending order: , represents the largest singular value, and r represents the number of singular values; S22: Truncate the matrix after singular value decomposition to obtain a rank-1 approximate matrix. , ;in, This represents the first left singular vector in the left singular matrix. This represents the first right singular vector in the right singular matrix; S23: With the first left singular vector As row guidance vector ,Right now ; with the first right singular vector As column guide vector ,Right now ; S24: Use the row guidance vector as the initial value of the row vector scaling factor. Use the column guide vector as the initial value of the column vector scaling factor. .
[0008] Preferably, in S3, based on the sensitivity of each weight element in the weight matrix, the weight matrix is divided into important and unimportant regions. The process is as follows: S31: Calculate the sensitivity of each weight element in the weight matrix. The expression is:
[0009] in, The Hessian matrix representing the weight matrices of each layer in a large language model satisfies the expression: ,in, These are the activation values of each layer in the large language model. Represents the inverse matrix. This represents the element in the i-th row and i-th column; Represents the original value of each weight element; S32: Sensitivity based on each weight element in the weight matrix Calculate the sensitivity of each column j. Sort each column by sensitivity Sort in descending order; sensitivity of each column j The calculation expression is: ; S33: Sensitivity to pressure Searching each column in descending order to obtain the most sensitive element yields the k value with the smallest binarization error; S34: Sensitivity of the weight matrix The top k columns are considered important regions, while the rest are considered unimportant regions.
[0010] Preferably, in S4, based on the initial values of the row vector scaling factor and the column vector scaling factor, analytical solutions for the row vector scaling factor and the column vector scaling factor are obtained respectively, and the first-order binary matrix of all regions of the weight matrix is obtained. The process is as follows: S41: For the weight matrix Binarize all regions to obtain a first-order binary matrix. The expression is: ; S42: Define first-order quantization error Based on first-order quantization error Solving for row vector scaling factor Analytical solution and column vector scaling factor The analytical solution.
[0011] Preferably, in S5, based on the analytical solutions of the row vector scaling factors, the analytical solutions of the column vector scaling factors, and the first-order binarized matrix, a first-order row-column alternating optimization iteration is performed on the row vector scaling factors and the column vector scaling factors to obtain the first-order reconstructed weight matrix. The process is as follows: S51: Initialize the number of iterations; S52: Fixed column vector scaling factor and first-order binary matrix Update the row vector scaling factor based on the analytical solution of the row vector scaling factor. ; S53: Fixed row vector scaling factor and first-order binary matrix Update the column vector scaling factor based on the analytical solution of the column vector scaling factor. ; S54: Determine if the maximum number of iterations has been reached. If not, return to S52; otherwise, end the iteration and obtain the first-order reconstruction weight matrix. Satisfies the expression: .
[0012] Preferably, in S6, based on the first-order reconstruction weight matrix, the residual matrix corresponding to the important region is obtained, and the residual matrix is binarized to obtain a second-order binarized matrix. The process is as follows: S61: Based on weight matrix and the first-order reconstruction weight matrix The initial residual matrix is obtained. Satisfies the expression: ; S62: For the initial residual matrix Binarization yields a second-order residual binarization matrix. ; S63: For the initial residual matrix The rows are averaged to obtain the initial values of the second-order row vector scaling factor. Eliminate the initial residual matrix The energy difference of the row vectors in the middle, and the initial residual matrix after eliminating the difference. The columns are averaged to obtain the initial values of the scaling factor for the second-order column vector. ; S64: Based on the second-order residual binarization matrix Initial value of second-order row vector scaling factor and the initial value of the scaling factor for the second-order column vector Initialize the first-order residual matrix Satisfies the expression: ; S65: For the first-order residual matrix Binarization yields the first-order residual binarization matrix. ; S66: Define second-order quantization error Based on second-order quantization error ,make Update row vector scaling factor Analytical solution; let Update column vector scaling factor Analytical solution; let Calculate the scaling factor of the second-order row vector. Analytical solution; let Calculate the scaling factor of the second-order column vector. The analytical solution; S67: Based on the first-order residual binarization matrix Row vector scaling factor Analytical solution and column vector scaling factor The analytical solution initializes the second-order residual matrix. Satisfies the expression: ; S68: Using the initial residual matrix First-order residual matrix Second-order residual matrix The residual matrix of the important region is represented by the first-order residual binarization matrix. and the second-order residual binarization matrix As a second-order binary matrix.
[0013] Preferably, in S7, based on the second-order binary matrix, first- and second-order row-column alternating optimization iterations are performed on important regions to obtain the second-order reconstructed weight matrix. The second-order reconstructed weight matrix is then used as the weight parameters after quantization of the large language model. The process is as follows: S71: Initialize the number of iterations; S72: Based on the analytical solutions of the first-order residual matrix, the second-order binarized matrix, the row vector scaling factor, and the column vector scaling factor, a first-order row-column alternating optimization iteration is performed on the important region; S73: Based on the analytical solutions of the initial residual matrix, the second-order binarized matrix, the second-order row vector scaling factor, and the second-order column vector scaling factor, perform second-order row and column alternating optimization iterations on important regions; S74: Determine if the maximum number of iterations has been reached. If not, return to S72; otherwise, end the iteration and obtain the second-order reconstruction weight matrix. Satisfies the expression: The second-order reconstructed weight matrix is used as the weight parameters after quantization of the large language model.
[0014] Preferably, in S73, based on the first-order residual matrix The analytical solutions for the second-order binary matrix, row vector scaling factors, and column vector scaling factors are used to perform first-order row-column alternating optimization iterations on important regions. The process is as follows: S731: Initialize the number of iterations; S732: Update and fix the first-order residual matrix Satisfies the expression: ; S733: Fixed column vector scaling factor and the first-order residual binarization matrix Update the row vector scaling factor based on the analytical solution of the row vector scaling factor. ; S734: Fixed row vector scaling factor and the first-order residual binarization matrix Update the row vector scaling factor based on the analytical solution of the column vector scaling factor. ; S735: Based on the first-order residual matrix Update the first-order residual binarization matrix Satisfies the expression: ; S736: Determine if the maximum number of iterations has been reached. If not, return to S732; otherwise, end the iteration.
[0015] Preferably, in S74, based on the analytical solutions of the initial residual matrix, the second-order binarized matrix, the second-order row vector scaling factor, and the second-order column vector scaling factor, a second-order row-column alternating optimization iteration is performed on the important region. The process is as follows: S741: Initialize the number of iterations; S742: Update and fix the second-order residual matrix Satisfies the expression: ; S743: Fixed second-order column vector scaling factor and the second-order residual binarization matrix Update the second-order row vector scaling factor based on the analytical solution of the second-order row vector scaling factor. ; S744: Fixed second-order row vector scaling factor and the second-order residual binarization matrix Update the scaling factor of the second-order column vector based on the analytical solution of the scaling factor. ; S745: Based on the initial residual matrix Update the second-order residual binarization matrix Satisfies the expression: ; S746: Determine if the maximum number of iterations has been reached. If not, return to S742; otherwise, end the iteration.
[0016] Secondly, this application proposes a binarization and quantization system for large language models based on guided alternation optimization, used to implement the method, including: The weight parameter acquisition module is used to acquire the weight parameters of the large language model to be quantized, wherein the weight parameters correspond to the weight matrix of the large language model. The row and column scaling factor acquisition module is used to decompose the weight matrix to obtain row guidance vectors and column guidance vectors; the row guidance vectors correspond to the initial values of the row vector scaling factor, and the column guidance vectors correspond to the initial values of the column vector scaling factor. The region partitioning module is used to divide the weight matrix into important and unimportant regions based on the sensitivity of each weight element in the weight matrix. The first-order row and column iterative initialization module is used to obtain the analytical solutions of the row vector scaling factor and the column vector scaling factor based on the initial values of the row vector scaling factor and the column vector scaling factor, respectively, and to obtain the first-order binary matrix of all regions of the weight matrix. The first-order row and column iteration module is used to perform first-order row and column alternating optimization iteration on the row vector scaling factor and the column vector scaling factor based on the analytical solution of the row vector scaling factor, the analytical solution of the column vector scaling factor and the first-order binarized matrix, so as to obtain the first-order reconstruction weight matrix. The first and second order row and column iteration initialization module is used to obtain the residual matrix corresponding to the important region based on the first order reconstruction weight matrix, and to binarize the residual matrix to obtain the second order binarized matrix. The first and second order row and column iteration module is used to perform alternating first and second order row and column optimization iterations on important regions based on the second order binary matrix to obtain the second order reconstructed weight matrix, which is then used as the weight parameters after the large language model is quantized.
[0017] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention proposes a binarization and quantization method and system for large language models based on guided alternating optimization. By evaluating the impact of each weight parameter on the performance of the large language model, the weight matrix is divided into important and unimportant regions. All regions of the weight matrix are binarized, and a first-order alternating row and column scaling factor is used to alternately optimize the row and column scaling factors to address inter-column bias, resulting in a first-order reconstructed weight matrix. The important regions of the first-order reconstructed weight matrix are binarized again, and optimized using a first- and second-order alternating row and column scaling factor to obtain a second-order reconstructed weight matrix. This significantly improves the accuracy of the large language model after low-bit quantization. The second-order reconstructed weight matrix is used as the weight parameters after quantization of the large language model. This invention effectively reduces the quantization error of key weight parameters, achieves high quantization accuracy, and the quantization process relies entirely on the internal structural information of the large language model. It is suitable for edge deployment scenarios where data is unavailable or data is sensitive, and has high edge deployment compatibility. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating the binarization and quantization method for large language models based on guided alternation optimization proposed in this embodiment of the invention. Figure 2 This is a schematic diagram illustrating the process of performing first-order row-column alternation optimization iteration on all regions of the weight matrix as proposed in this embodiment of the invention; Figure 3 This is a schematic diagram illustrating the process of performing alternating first- and second-order row and column optimization iterations on important regions of the first-order reconstructed weight matrix as proposed in this embodiment of the invention. Figure 4 This diagram illustrates the composition of the binarization and quantization system for a large language model based on guided alternating optimization, as proposed in this embodiment of the invention. Detailed Implementation
[0019] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions; It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.
[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments; The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.
[0021] Example 1 This embodiment provides a binarization and quantization method for large language models based on guided alternation optimization. The flowchart of this method can be found in [link to flowchart]. Figure 1 This includes the following steps: S1: Obtain the weight parameters of the large language model to be quantized, where the weight parameters correspond to the weight matrix of the large language model; S2: Decompose the weight matrix to obtain row guidance vectors and column guidance vectors; the row guidance vectors correspond to the initial values of the row vector scaling factors, and the column guidance vectors correspond to the initial values of the column vector scaling factors. S3: Based on the sensitivity of each weight element in the weight matrix, divide the weight matrix into important and unimportant regions; S4: Based on the initial values of the row vector scaling factor and the column vector scaling factor, obtain the analytical solutions of the row vector scaling factor and the column vector scaling factor respectively, and obtain the first-order binary matrix of all regions of the weight matrix. S5: Based on the analytical solutions of the row vector scaling factor, the analytical solutions of the column vector scaling factor, and the first-order binarized matrix, perform first-order row-column alternation optimization iteration on the row vector scaling factor and the column vector scaling factor to obtain the first-order reconstruction weight matrix. S6: Based on the first-order reconstruction weight matrix, obtain the residual matrix corresponding to the important region, and binarize the residual matrix to obtain the second-order binarized matrix; S7: Based on the second-order binary matrix, perform first- and second-order row and column alternation optimization iterations on important regions to obtain the second-order reconstruction weight matrix, and use the second-order reconstruction weight matrix as the weight parameters after the large language model is quantized.
[0022] In this embodiment, the large language model Llama-2-7b is selected as the model to be quantized. First, the weight parameters of the large language model to be quantized are obtained. These weight parameters correspond to multiple weight matrices of the large language model. Activation values, KV-cache parameters, etc., are also obtained. The weight matrices are decomposed to obtain multiple row-guided vectors and column-guided vectors. The row-guided vectors are used as the initial values for the row-vector scaling factors, and the column-guided vectors are used as the initial values for the column-vector scaling factors. The sensitivity of each weight element in the weight matrix is calculated, and each weight matrix is divided into important and unimportant regions based on the sensitivity. Based on the initial values of the row-vector scaling factors and column-vector scaling factors, the analytical solutions for the row-vector scaling factors and column-vector scaling factors are obtained, respectively. The algorithm analyzes and obtains the first-order binarized matrix of all regions of the weight matrix. It then performs first-order row-column alternating optimization iterations on the row and column scaling factors to obtain the first-order reconstructed weight matrix. Based on the first-order reconstructed weight matrix, it obtains the residual matrix corresponding to its important regions. The residual matrix is then binarized to obtain the second-order binarized matrix. According to the second-order binarized matrix, the analytical solutions of the row and column scaling factors are updated. The initial values and analytical solutions of the second-order row and column scaling factors are obtained. The important regions are then subjected to first- and second-order row-column alternating optimization iterations to obtain the second-order reconstructed weight matrix. This second-order reconstructed weight matrix is then used as the weight parameters after quantization of the large language model.
[0023] Example 2 In this embodiment, the optimization process for any weight matrix in the large language model to be quantized is as follows: In S2, the weight matrix is decomposed to obtain the row guidance vector and column guidance vector, as follows: S21: For the weight matrix Perform singular value decomposition, wherein the singular value decomposition satisfies the expression: ;in, Represents a left singular matrix; Represents a right singular matrix; Represents a real matrix; Given a diagonal matrix, sort the singular values in the diagonal matrix in descending order: , represents the largest singular value, and r represents the number of singular values; S22: Truncate the matrix after singular value decomposition to obtain a rank-1 approximate matrix. , ;in, This represents the first left singular vector in the left singular matrix. This represents the first right singular vector in the right singular matrix; S23: With the first left singular vector As row guidance vector ,Right now ; with the first right singular vector As column guide vector ,Right now ; S24: Use the row guidance vector as the initial value of the row vector scaling factor. Use the column guide vector as the initial value of the column vector scaling factor. .
[0024] Specifically, the rank-1 approximation formula for singular value decomposition of the weight matrix is: .
[0025] Preferably, in S3, based on the sensitivity of each weight element in the weight matrix, the weight matrix is divided into important and unimportant regions. The process is as follows: S31: Calculate the sensitivity of each weight element in the weight matrix. The expression is:
[0026] in, The Hessian matrix representing the weight matrices of each layer in a large language model satisfies the expression: ,in, These are the activation values of each layer in the large language model. Represents the inverse matrix. This represents the element in the i-th row and i-th column; Represents the original value of each weight element; S32: Sensitivity based on each weight element in the weight matrix Calculate the sensitivity of each column j. Sort each column by sensitivity Sort in descending order; sensitivity of each column j The calculation expression is: ; S33: Sensitivity to pressure Searching each column in descending order to obtain the most sensitive element yields the k value with the smallest binarization error; S34: Sensitivity of the weight matrix The top k columns are considered important regions, while the rest are considered unimportant regions.
[0027] In this embodiment, the saliency of parameters in the unbinarized layer is evaluated using the Hessian matrix, and the weight sensitivity is derived, thereby obtaining accurate parameter importance indicators without reducing computational efficiency. Specifically, in S31, the standard Hessian matrix expression is: , among which, if After vectorization ,but .in, Let represent a real matrix, where n is the number of rows and m is the number of columns. In large language models, the Hessian matrix is too large to be calculated. In this embodiment, the Hessian matrix is approximated and expressed as: ,in These are the activation values for each layer of the large language model.
[0028] In S32, the columns are sorted by sensitivity. The column sequence is obtained by sorting in descending order. , , ..., , ..., },in, =1,2,…,m, where m is the column number.
[0029] The search process described in step S33 is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] The columns are designated as important regions of the weight matrix, while the remaining columns are designated as unimportant regions. The binarization error is calculated, and a search is performed to minimize this error. Value. The search expression satisfies:
[0030] in, Represents the union, and These represent the scaling factor and weight matrix of the important regions, respectively. and These represent the scaling factor and weight matrix of the unimportant region, respectively. , Represents the L1 norm; Represents a binary matrix; This indicates taking the option that minimizes the binarization error. value; Let f(x) denote the Frobenius norm, which is the square root of the sum of the squares of the absolute values of each element.
[0031] Preferably, in S4, based on the initial values of the row vector scaling factor and the column vector scaling factor, analytical solutions for the row vector scaling factor and the column vector scaling factor are obtained respectively, and the first-order binary matrix of all regions of the weight matrix is obtained. The process is as follows: S41: For the weight matrix Binarize all regions to obtain a first-order binary matrix. The expression is: ; S42: Define first-order quantization error Based on first-order quantization error Solving for row vector scaling factor Analytical solution and column vector scaling factor The analytical solution.
[0032] Specifically, first-order quantization error The expression is: ;in, This represents the Frobenius norm, which is the square root of the sum of the squares of the absolute values of each element. The binary structure matrix in the iteration process satisfies the expression: ;make The scaling factor of the row vector is obtained by solving. The analytical solution satisfies the expression: ;make The scaling factor of the column vector is obtained by solving. The analytical solution satisfies the expression: ;in, Represents a diagonalized matrix. This indicates the matrix transpose.
[0033] Preferably, in S5, based on the analytical solutions of the row vector scaling factors, the analytical solutions of the column vector scaling factors, and the first-order binarized matrix, a first-order row-column alternation optimization iteration is performed on the row vector scaling factors and the column vector scaling factors to obtain a first-order reconstructed weight matrix. The flowchart for the first-order row-column alternation optimization iteration on all regions of the weight matrix is shown below. Figure 2 The process is as follows: S51: Initialize the number of iterations; S52: Fixed column vector scaling factor and first-order binary matrix Update the row vector scaling factor based on the analytical solution of the row vector scaling factor. ; S53: Fixed row vector scaling factor and first-order binary matrix Update the column vector scaling factor based on the analytical solution of the column vector scaling factor. ; S54: Determine if the maximum number of iterations has been reached. If not, return to S52; otherwise, end the iteration and obtain the first-order reconstruction weight matrix. Satisfies the expression: .
[0034] Preferably, in S6, based on the first-order reconstruction weight matrix, the residual matrix corresponding to the important region is obtained, and the residual matrix is binarized to obtain a second-order binarized matrix. The process is as follows: S61: Based on weight matrix and the first-order reconstruction weight matrix The initial residual matrix is obtained. Satisfies the expression: ; S62: For the initial residual matrix Binarization yields a second-order residual binarization matrix. ; S63: For the initial residual matrix The rows are averaged to obtain the initial values of the second-order row vector scaling factor. Eliminate the initial residual matrix The energy difference of the row vectors in the middle, and the initial residual matrix after eliminating the difference. The columns are averaged to obtain the initial values of the scaling factor for the second-order column vector. ; S64: Based on the second-order residual binarization matrix Initial value of second-order row vector scaling factor and the initial value of the scaling factor for the second-order column vector Initialize the first-order residual matrix Satisfies the expression: ; S65: For the first-order residual matrix Binarization yields the first-order residual binarization matrix. ; S66: Define second-order quantization error Based on second-order quantization error ,make Update row vector scaling factor Analytical solution; let Update column vector scaling factor Analytical solution; let Calculate the scaling factor of the second-order row vector. Analytical solution; let Calculate the scaling factor of the second-order column vector. The analytical solution; S67: Based on the first-order residual binarization matrix Row vector scaling factor Analytical solution and column vector scaling factor The analytical solution initializes the second-order residual matrix. Satisfies the expression: ; S68: Using the initial residual matrix First-order residual matrix Second-order residual matrix The residual matrix of the important region is represented by the first-order residual binarization matrix. and the second-order residual binarization matrix As a second-order binary matrix.
[0035] Specifically, in S62, the second-order residual binarization matrix The expression is: .
[0036] In S63, the initial value of the second-order row vector scaling factor The expression is: Where j represents the j-th row, k represents the k-th column, and m represents the row number. This yields the initial values of the scaling factor for the second-order column vector. The expression is: Where n represents the column number, This represents the second-order row term scaling factor. This represents the second-order column scaling factor.
[0037] In S65, the first-order residual binarization matrix The expression is: .
[0038] In S66, second-order quantization error The expression is:
[0039] in, This represents the Frobenius norm, which is the square root of the sum of the squares of the absolute values of each element. Based on second-order quantization error ,make Update row vector scaling factor The analytical solution is expressed as: ;make Update column vector scaling factor The analytical solution is expressed as: ; make Obtain the second-order row vector scaling factor The analytical solution is expressed as: ;make Obtain the scaling factor of the second-order column vector. The analytical solution satisfies the expression: .
[0040] Preferably, in S7, based on the second-order binary matrix, first- and second-order alternating row and column optimization iterations are performed on the important regions to obtain the second-order reconstructed weight matrix. The second-order reconstructed weight matrix is used as the weight parameters after quantization of the large language model. The flowchart for the first- and second-order alternating row and column optimization iterations on the important regions of the first-order reconstructed weight matrix is shown below. Figure 3 The process is as follows: S71: Initialize the number of iterations; S72: Based on the analytical solutions of the first-order residual matrix, the second-order binarized matrix, the row vector scaling factor, and the column vector scaling factor, a first-order row-column alternating optimization iteration is performed on the important region; S73: Based on the analytical solutions of the initial residual matrix, the second-order binarized matrix, the second-order row vector scaling factor, and the second-order column vector scaling factor, perform second-order row and column alternating optimization iterations on important regions; S74: Determine if the maximum number of iterations has been reached. If not, return to S72; otherwise, end the iteration and obtain the second-order reconstruction weight matrix. Satisfies the expression: The second-order reconstructed weight matrix is used as the weight parameters after quantization of the large language model.
[0041] Preferably, in S73, based on the first-order residual matrix The analytical solutions for the second-order binary matrix, row vector scaling factors, and column vector scaling factors are used to perform first-order row-column alternating optimization iterations on important regions. The process is as follows: S731: Initialize the number of iterations; S732: Update and fix the first-order residual matrix Satisfies the expression: ; S733: Fixed column vector scaling factor and the first-order residual binarization matrix Update the row vector scaling factor based on the analytical solution of the row vector scaling factor. ; S734: Fixed row vector scaling factor and the first-order residual binarization matrix Update the row vector scaling factor based on the analytical solution of the column vector scaling factor. ; S735: Based on the first-order residual matrix Update the first-order residual binarization matrix Satisfies the expression: ; S736: Determine if the maximum number of iterations has been reached. If not, return to S732; otherwise, end the iteration.
[0042] Preferably, in S74, based on the analytical solutions of the initial residual matrix, the second-order binarized matrix, the second-order row vector scaling factor, and the second-order column vector scaling factor, a second-order row-column alternating optimization iteration is performed on the important region. The process is as follows: S741: Initialize the number of iterations; S742: Update and fix the second-order residual matrix Satisfies the expression: ; S743: Fixed second-order column vector scaling factor and the second-order residual binarization matrix Update the second-order row vector scaling factor based on the analytical solution of the second-order row vector scaling factor. ; S744: Fixed second-order row vector scaling factor and the second-order residual binarization matrix Update the scaling factor of the second-order column vector based on the analytical solution of the scaling factor. ; S745: Based on the initial residual matrix Update the second-order residual binarization matrix Satisfies the expression: ; S746: Determine if the maximum number of iterations has been reached. If not, return to S742; otherwise, end the iteration.
[0043] Example 3 This embodiment provides a binarization and quantization system for large language models based on guided alternation optimization. See [link to documentation]. Figure 4 The system is used to implement the aforementioned binarization and quantization method for large language models based on guided alternation optimization, including: The weight parameter acquisition module is used to acquire the weight parameters of the large language model to be quantized, wherein the weight parameters correspond to the weight matrix of the large language model. The row and column scaling factor acquisition module is used to decompose the weight matrix to obtain row guidance vectors and column guidance vectors; the row guidance vectors correspond to the initial values of the row vector scaling factor, and the column guidance vectors correspond to the initial values of the column vector scaling factor. The region partitioning module is used to divide the weight matrix into important and unimportant regions based on the sensitivity of each weight element in the weight matrix. The first-order row and column iterative initialization module is used to obtain the analytical solutions of the row vector scaling factor and the column vector scaling factor based on the initial values of the row vector scaling factor and the column vector scaling factor, respectively, and to obtain the first-order binary matrix of all regions of the weight matrix. The first-order row and column iteration module is used to perform first-order row and column alternating optimization iteration on the row vector scaling factor and the column vector scaling factor based on the analytical solution of the row vector scaling factor, the analytical solution of the column vector scaling factor and the first-order binarized matrix, so as to obtain the first-order reconstruction weight matrix. The first and second order row and column iteration initialization module is used to obtain the residual matrix corresponding to the important region based on the first order reconstruction weight matrix, and to binarize the residual matrix to obtain the second order binarized matrix. The first and second order row and column iteration module is used to perform alternating first and second order row and column optimization iterations on important regions based on the second order binary matrix to obtain the second order reconstructed weight matrix, which is then used as the weight parameters after the large language model is quantized.
[0044] The same or similar labels correspond to the same or similar parts; The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting the invention. Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A binarization and quantization method for large language models based on guided alternation optimization, characterized in that, Includes the following steps: S1: Obtain the weight parameters of the large language model to be quantized, where the weight parameters correspond to the weight matrix of the large language model; S2: Decompose the weight matrix to obtain row guidance vectors and column guidance vectors; the row guidance vectors correspond to the initial values of the row vector scaling factors, and the column guidance vectors correspond to the initial values of the column vector scaling factors. S3: Based on the sensitivity of each weight element in the weight matrix, divide the weight matrix into important and unimportant regions; S4: Based on the initial values of the row vector scaling factor and the column vector scaling factor, obtain the analytical solutions of the row vector scaling factor and the column vector scaling factor respectively, and obtain the first-order binary matrix of all regions of the weight matrix. S5: Based on the analytical solutions of the row vector scaling factor, the analytical solutions of the column vector scaling factor, and the first-order binarized matrix, perform first-order row-column alternation optimization iteration on the row vector scaling factor and the column vector scaling factor to obtain the first-order reconstruction weight matrix. S6: Based on the first-order reconstruction weight matrix, obtain the residual matrix corresponding to the important region, and binarize the residual matrix to obtain the second-order binarized matrix; S7: Based on the second-order binary matrix, perform first- and second-order row and column alternation optimization iterations on important regions to obtain the second-order reconstruction weight matrix, and use the second-order reconstruction weight matrix as the weight parameters after the large language model is quantized.
2. The binarization and quantization method for large language models based on guided alternation optimization according to claim 1, characterized in that, In S2, the weight matrix is decomposed to obtain the row guidance vector and column guidance vector, as follows: S21: For the weight matrix Perform singular value decomposition, wherein the singular value decomposition satisfies the expression: ;in, Represents a left singular matrix; Represents a right singular matrix; Represents a real matrix; Given a diagonal matrix, sort the singular values in the diagonal matrix in descending order: , represents the largest singular value, and r represents the number of singular values; S22: Truncate the matrix after singular value decomposition to obtain a rank-1 approximate matrix. , ;in, This represents the first left singular vector in the left singular matrix. This represents the first right singular vector in the right singular matrix; S23: With the first left singular vector As row guidance vector ,Right now ; with the first right singular vector As column guide vector ,Right now ; S24: Use the row guidance vector as the initial value of the row vector scaling factor. Use the column guide vector as the initial value of the column vector scaling factor. .
3. The binarization and quantization method for large language models based on guided alternation optimization according to claim 1, characterized in that, In S3, based on the sensitivity of each weight element in the weight matrix, the weight matrix is divided into important and unimportant regions. The process is as follows: S31: Calculate the sensitivity of each weight element in the weight matrix. The expression is: in, The Hessian matrix representing the weight matrices of each layer in a large language model satisfies the expression: ,in, These are the activation values of each layer in the large language model. Represents the inverse matrix. This represents the element in the i-th row and i-th column; Represents the original value of each weight element; S32: Sensitivity based on each weight element in the weight matrix Calculate the sensitivity of each column j. Sort each column by sensitivity Sort in descending order; sensitivity of each column j The calculation expression is: ; S33: Sensitivity to pressure Searching each column in descending order to obtain the most sensitive element yields the k value with the smallest binarization error; S34: Sensitivity of the weight matrix The top k columns are considered important regions, while the rest are considered unimportant regions.
4. The binarization and quantization method for large language models based on guided alternation optimization according to claim 2, characterized in that, In S4, based on the initial values of the row vector scaling factor and the column vector scaling factor, the analytical solutions of the row vector scaling factor and the column vector scaling factor are obtained respectively, and the first-order binary matrix of all regions of the weight matrix is obtained. The process is as follows: S41: For the weight matrix Binarize all regions to obtain a first-order binary matrix. The expression is: ; S42: Define first-order quantization error Based on first-order quantization error Solving for row vector scaling factor Analytical solution and column vector scaling factor The analytical solution.
5. The binarization and quantization method for large language models based on guided alternation optimization according to claim 4, characterized in that, In S5, based on the analytical solutions of the row vector scaling factors, the analytical solutions of the column vector scaling factors, and the first-order binarized matrix, a first-order row-column alternation optimization iteration is performed on the row vector scaling factors and the column vector scaling factors to obtain the first-order reconstructed weight matrix. The process is as follows: S51: Initialize the number of iterations; S52: Fixed column vector scaling factor and first-order binary matrix Update the row vector scaling factor based on the analytical solution of the row vector scaling factor. ; S53: Fixed row vector scaling factor and first-order binary matrix Update the column vector scaling factor based on the analytical solution of the column vector scaling factor. ; S54: Determine if the maximum number of iterations has been reached. If not, return to S52; otherwise, end the iteration and obtain the first-order reconstruction weight matrix. Satisfies the expression: .
6. The binarization and quantization method for large language models based on guided alternation optimization according to claim 5, characterized in that, In S6, based on the first-order reconstruction weight matrix, the residual matrix corresponding to the important regions is obtained. The residual matrix is then binarized to obtain a second-order binarized matrix. The process is as follows: S61: Based on weight matrix and the first-order reconstruction weight matrix The initial residual matrix is obtained. Satisfies the expression: ; S62: For the initial residual matrix Binarization yields a second-order residual binarization matrix. ; S63: For the initial residual matrix The rows are averaged to obtain the initial values of the second-order row vector scaling factor. Eliminate the initial residual matrix The energy difference of the row vectors, and the effect on the initial residual matrix after eliminating the difference. The columns are averaged to obtain the initial values of the scaling factor for the second-order column vector. ; S64: Based on the second-order residual binarization matrix Initial value of second-order row vector scaling factor and the initial value of the scaling factor for the second-order column vector Initialize the first-order residual matrix Satisfies the expression: ; S65: For the first-order residual matrix Binarization yields the first-order residual binarization matrix. ; S66: Define second-order quantization error Based on second-order quantization error ,make Update row vector scaling factor Analytical solution; let Update column vector scaling factor Analytical solution; let Calculate the scaling factor of the second-order row vector. Analytical solution; let Calculate the scaling factor of the second-order column vector. The analytical solution; S67: Based on the first-order residual binarization matrix Row vector scaling factor Analytical solution and column vector scaling factor The analytical solution initializes the second-order residual matrix. Satisfies the expression: ; S68: Using the initial residual matrix First-order residual matrix Second-order residual matrix The residual matrix of the important region is represented by the first-order residual binarization matrix. and the second-order residual binarization matrix As a second-order binary matrix.
7. The binarization and quantization method for large language models based on guided alternation optimization according to claim 6, characterized in that, In S7, based on the second-order binary matrix, first- and second-order row-column alternation optimization iterations are performed on important regions to obtain the second-order reconstructed weight matrix. The second-order reconstructed weight matrix is then used as the weight parameters after quantization of the large language model. The process is as follows: S71: Initialize the number of iterations; S72: Based on the analytical solutions of the first-order residual matrix, the second-order binarized matrix, the row vector scaling factor, and the column vector scaling factor, a first-order row-column alternating optimization iteration is performed on the important region; S73: Based on the analytical solutions of the initial residual matrix, the second-order binarized matrix, the second-order row vector scaling factor, and the second-order column vector scaling factor, perform second-order row and column alternating optimization iterations on important regions; S74: Determine if the maximum number of iterations has been reached. If not, return to S72; otherwise, end the iteration and obtain the second-order reconstruction weight matrix. Satisfies the expression: The second-order reconstructed weight matrix is used as the weight parameters after quantization of the large language model.
8. The binarization and quantization method for large language models based on guided alternation optimization according to claim 7, characterized in that, In S73, based on the first-order residual matrix The analytical solutions for the second-order binary matrix, row vector scaling factors, and column vector scaling factors are used to perform first-order row-column alternating optimization iterations on important regions. The process is as follows: S731: Initialize the number of iterations; S732: Update and fix the first-order residual matrix Satisfies the expression: ; S733: Fixed column vector scaling factor and the first-order residual binarization matrix Update the row vector scaling factor based on the analytical solution of the row vector scaling factor. ; S734: Fixed row vector scaling factor and the first-order residual binarization matrix Update the row vector scaling factor based on the analytical solution of the column vector scaling factor. ; S735: Based on the first-order residual matrix Update the first-order residual binarization matrix Satisfies the expression: ; S736: Determine if the maximum number of iterations has been reached. If not, return to S732; otherwise, end the iteration.
9. The binarization and quantization method for large language models based on guided alternation optimization according to claim 7, characterized in that, In S74, based on the analytical solutions of the initial residual matrix, the second-order binarized matrix, the second-order row vector scaling factor, and the second-order column vector scaling factor, a second-order row-column alternating optimization iteration is performed on the important region. The process is as follows: S741: Initialize the number of iterations; S742: Update and fix the second-order residual matrix Satisfies the expression: ; S743: Fixed second-order column vector scaling factor and the second-order residual binarization matrix Update the second-order row vector scaling factor based on the analytical solution of the second-order row vector scaling factor. ; S744: Fixed second-order row vector scaling factor and the second-order residual binarization matrix Update the scaling factor of the second-order column vector based on the analytical solution of the scaling factor. ; S745: Based on the initial residual matrix Update the second-order residual binarization matrix Satisfies the expression: ; S746: Determine if the maximum number of iterations has been reached. If not, return to S742; otherwise, end the iteration.
10. A binarization and quantization system for large language models based on guided alternation optimization, used to implement the binarization and quantization method for large language models based on guided alternation optimization as described in any one of claims 1-9, characterized in that, include: The weight parameter acquisition module is used to acquire the weight parameters of the large language model to be quantized, wherein the weight parameters correspond to the weight matrix of the large language model. The row and column scaling factor acquisition module is used to decompose the weight matrix to obtain row guidance vectors and column guidance vectors; the row guidance vectors correspond to the initial values of the row vector scaling factor, and the column guidance vectors correspond to the initial values of the column vector scaling factor. The region partitioning module is used to divide the weight matrix into important and unimportant regions based on the sensitivity of each weight element in the weight matrix. The first-order row and column iterative initialization module is used to obtain the analytical solutions of the row vector scaling factor and the column vector scaling factor based on the initial values of the row vector scaling factor and the column vector scaling factor, respectively, and to obtain the first-order binary matrix of all regions of the weight matrix. The first-order row and column iteration module is used to perform first-order row and column alternating optimization iteration on the row vector scaling factor and the column vector scaling factor based on the analytical solution of the row vector scaling factor, the analytical solution of the column vector scaling factor and the first-order binarized matrix, so as to obtain the first-order reconstruction weight matrix. The first and second order row and column iteration initialization module is used to obtain the residual matrix corresponding to the important region based on the first order reconstruction weight matrix, and to binarize the residual matrix to obtain the second order binarized matrix. The first and second order row and column iteration module is used to perform alternating first and second order row and column optimization iterations on important regions based on the second order binary matrix to obtain the second order reconstructed weight matrix, which is then used as the weight parameters after the large language model is quantized.