Weight binary quantization method and system for large language model

Abstract

The invention provides a weight binary quantization method and system for a large language model, and the method comprises the steps: carrying out the reordering operation of a calibration data matrix and a weight matrix, obtaining a reordered weight matrix and a reordered calibration data matrix, carrying out the Cholesky decomposition of the reordered calibration data matrix, and obtaining a target calibration data matrix; splicing the quantization results of the weight blocks to obtain the quantization results of the weight blocks; based on a target error matrix, error compensation is carried out on the reordering weight matrix, a reordering weight matrix after error compensation is obtained, and the target error matrix is obtained after discarding operation is carried out on a target error value in the propagation error matrix; the target error value is determined according to index information of the weight blocks; and obtaining a target quantization weight matrix according to the reordering weight matrix after error compensation. While the model is compressed to the maximum extent, the output precision of the model is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a weighted binary quantization method and system for large language models. Background Technology

[0002] Deep learning and large language models have developed rapidly and achieved remarkable results; however, their huge number of parameters has brought storage and computational pressures.

[0003] Quantization techniques, as an effective method for compressing neural networks, can reduce GPU memory usage during the inference process of large language models. The quantization process typically uses affine transformations to map the original tensor to a discrete numerical space with a lower bit width.

[0004] Current ultra-low bit post-training quantization (PTQ) methods primarily rely on error compensation. These methods generally suffer from poor generalization ability and mediocre performance on out-of-calibration datasets. Experiments revealed two key phenomena: first, the weight matrix of large language models after quantization often has larger amplitudes in subsequent input channels, leading to overemphasis on certain channel dimensions and a deviation from the original weight distribution; second, current error-compensated quantization methods generally quantize in ascending order of channel indices, causing each layer to overemphasize channels with larger indices. Furthermore, the Hessian matrix of the next layer is calculated based on the output of the previous layer, and this offset caused by compensation ultimately results in significant model output errors. Therefore, there is an urgent need for a weight binary quantization method and system for large language models to address these issues. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a weighted binary quantization method and system for large language models.

[0006] This invention provides a weight binary quantization method for large language models, comprising: The calibration data matrix and the weight matrix input to the large language model are reordered to obtain the reordered weight matrix and the reordered calibration data matrix, respectively. The reordered calibration data matrix is ​​then subjected to Cholesky decomposition to obtain the target calibration data matrix. The quantization results of all rows in the weight block are concatenated to obtain the weight block quantization result, wherein the weight block is obtained by dividing the reordered weight matrix into blocks; Based on the target error matrix, error compensation is performed on the reordered weight matrix to obtain an error-compensated reordered weight matrix. The target error matrix is ​​obtained by discarding the target error values ​​in the propagation error matrix. The propagation error matrix is ​​calculated based on the weight blocks, the quantization results of the weight blocks, and the target calibration data matrix. The target error values ​​are determined according to the index information of the weight blocks. The target quantization weight matrix is ​​obtained based on the reordered weight matrix after error compensation.

[0007] According to the present invention, a weight binary quantization method for a large language model includes reordering the calibration data matrix and the weight matrix input to the large language model to obtain a reordered weight matrix and a reordered calibration data matrix, comprising: Based on the calibration data, calculate the Hessian matrix corresponding to the weight matrix input to the current layer of the large language model; The Hessian matrix is ​​reordered in ascending order to obtain the reordered calibration data matrix. Summation is performed on each column of the weight matrix, and the summation results are reordered in ascending order to obtain the reordered weight matrix. The step of performing Cholesky decomposition on the reordered calibration data matrix to obtain the target calibration data matrix includes: The reordered calibration data matrix is ​​regularized, and the regularized reordered calibration data matrix is ​​then subjected to Cholesky decomposition to obtain the target calibration data matrix.

[0008] According to the present invention, a binary weight quantization method for a large language model is provided, wherein the quantization results of all rows in the weight block are concatenated to obtain the weight block quantization result, wherein the weight block is obtained by dividing the reordered weight matrix into blocks, including: Based on a preset block size, the reordered weight matrix is ​​divided into blocks to obtain multiple weight blocks; The key rows in the weighted block are subjected to approximate binarization to obtain a first quantization result; wherein, the key row is the row with the highest error sensitivity determined from all rows in the weighted block based on the reordered calibration data matrix; Based on a preset binarization threshold, each non-key row in the weighted block is binarized to obtain the second quantization result corresponding to each non-key row. The first quantization result and the second quantization result are concatenated to obtain the weighted block quantization result.

[0009] According to the present invention, a weight binary quantization method for a large language model is provided, wherein the reordered weight matrix is ​​subjected to error compensation based on a target error matrix to obtain an error-compensated reordered weight matrix, comprising: Repeatedly execute the preset iterative steps to perform error compensation on the reordered weight matrix until all weight blocks in the reordered weight matrix have completed error compensation, and obtain the error-compensated reordered weight matrix; The iterative steps include: The block quantization error of the current round is calculated based on the quantization error between the current round's weight block and the current round's weight block quantization result; wherein, the current round's weight block is obtained after error compensation based on the target error matrix of the previous round; The original error matrix of the current round is obtained based on the ratio between the block quantization error of the current round and the calibration data matrix sub-block of the current round. The calibration data matrix sub-block of the current round is determined in the target calibration data matrix based on the row and column information of the weight block of the current round. Based on the row information of the weight block in the current round and the column information of the weight block in the next round, the weight block association matrix of the current round is obtained from the target calibration data matrix; The propagation error matrix of the current round is obtained by multiplying the original error matrix of the current round and the weighted block correlation matrix of the current round. Based on the tensor values ​​in the propagation error matrix of the current round and the distance decay factor of the current round, the target error values ​​in the propagation error matrix of the current round are discarded to obtain the target error matrix of the current round. The distance decay factor of the current round is determined based on the index information of the weight block of the current round. Based on the target error matrix of the current round, error compensation is performed on the weight blocks of the next round.

[0010] According to the present invention, a weighted binary quantization method for large language models is provided, wherein the target error value in the propagation error matrix of the current round is discarded based on the tensor value in the propagation error matrix of the current round and the distance decay factor of the current round, to obtain the target error matrix of the current round, including: Based on the magnitude of the tensor values ​​in the propagation error matrix of the current round, obtain the tensor value sorting result of the current round; Obtain the difference between the total number of weight blocks in the reordered weight matrix and the index information of the weight blocks in the current round, and obtain the distance decay factor of the current round based on the ratio between the cyclic index information corresponding to the weight blocks in the next round and the difference. Based on the distance decay factor of the current round, the tensor values ​​of the current round are sorted, with the top... k The smallest tensor value is determined as the target error value in the propagation error matrix of the current round; The target error value in the propagation error matrix of the current round is set to zero to obtain the target error matrix of the current round.

[0011] According to the present invention, a binary weight quantization method for a large language model is provided, wherein obtaining the target quantized weight matrix based on the error-compensated reordered weight matrix includes: Based on the reordering operation performed on the weight matrix, the reverse reordering operation is performed on the error-compensated reordered weight matrix to obtain the target quantization weight matrix.

[0012] This invention also provides a weighted binary quantization system for large language models, comprising: The reordering module is used to reorder the calibration data matrix and the weight matrix input to the large language model, respectively, to obtain the reordered weight matrix and the reordered calibration data matrix, and to perform Cholesky decomposition on the reordered calibration data matrix to obtain the target calibration data matrix. The block quantization module is used to concatenate the quantization results of all rows in the weight block to obtain the weight block quantization result. The weight block is obtained by dividing the reordered weight matrix into blocks. The block error compensation module is used to perform error compensation on the reordered weight matrix based on the target error matrix to obtain the error-compensated reordered weight matrix. The target error matrix is ​​obtained by discarding target error values ​​in the propagation error matrix. The propagation error matrix is ​​calculated based on the weight blocks, the weight block quantization results, and the target calibration data matrix. The target error value is determined according to the index information of the weight blocks. The quantization matrix output module is used to obtain the target quantization weight matrix based on the reordered weight matrix after error compensation.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the weighted binary quantization method for large language models as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the weighted binary quantization method for large language models as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the weighted binary quantization method for large language models as described above.

[0016] The present invention provides a weighted binary quantization method and system for large language models. By discarding smaller values ​​in each compensation value in blocks, with a higher discard rate for larger distances between two blocks, the overcompensation of subsequent dimensions is reduced. To make the weight distribution after quantization close to the distribution before quantization, the weight matrix is ​​reordered according to the sum of the magnitudes of each column before quantization, and this order is recorded. Then, the original order is restored after quantization, so that the overcompensated dimensions in each layer are the parts with larger original weight magnitudes. This improves the output accuracy of the model while compressing the model to the maximum extent. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the weighted binary quantization method for large language models provided by the present invention. Figure 2 A schematic diagram of the quantization of the original input weight matrix provided by this invention; Figure 3 This is a schematic diagram of the structure of the weighted binary quantization system for large language models provided by the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Deep learning and large language models (LLMs) have experienced rapid development and achieved remarkable success in recent years. However, their massive number of parameters has also triggered a series of severe storage and computational challenges.

[0021] Taking the GPT-3 model with 175 billion parameters as an example, when stored in FP32 (32-bit floating-point) format, the weights alone require nearly 700GB of memory, a requirement far exceeding the capacity of most devices. Furthermore, the latency and high energy consumption during inference significantly limit its application on edge devices.

[0022] Quantization, as an effective means of compressing neural networks, can significantly reduce the GPU memory usage of large language models during inference. The quantization process typically employs affine transformations to convert the original tensor... X Mapping to a discrete numerical space with a lower bit width, the specific formula is as follows: ; in, s Indicates the scaling factor. z It's midnight. round For the floor function, clamp The purpose of a function is to restrict the range of its values ​​to the low-bit representation space. range This indicates the low bit space range.

[0023] Currently, quantization techniques are mainly divided into two types: Quantization Aware Training (QAT) and Post-Training Quantization (PTQ). PTQ, in particular, eliminates the need for backpropagation, making its computation simpler and faster. For large language models with extremely large structures and parameters, PTQ is especially important, particularly suitable for scenarios with limited time and resources.

[0024] While existing PTQ methods have achieved success with 8-bit and 4-bit quantization, the need for more aggressive quantization schemes, such as quantizing weights to 1-bit binarization, is becoming increasingly urgent as large language models continue to expand. However, current state-of-the-art ultra-low bit PTQ methods mainly rely on error compensation mechanisms. These methods generally suffer from a serious problem: poor generalization ability. On datasets outside the calibration set, their performance is often unsatisfactory and mediocre.

[0025] Through a series of experiments, two key phenomena were discovered in existing technologies. First, after quantization, the amplitude of subsequent input channels in the weight matrix of large language models often becomes larger. This overemphasis on subsequent channels causes the model to overemphasize certain channel dimensions, which is seriously inconsistent with the original weight distribution. Second, current error-compensated quantization methods basically perform quantization operations in ascending order of channel index. This quantization method causes each layer to overemphasize the channel dimensions with larger indices. Moreover, since the Hessian matrix of the next layer is calculated based on the output of the previous layer, the offset caused by this compensation method ultimately leads to a large error in the model output.

[0026] Furthermore, existing error-compensated quantization algorithms are prone to overfitting to the calibration dataset when the quantization bit width reaches approximately 1 bit. To address these issues, this invention proposes a plug-and-play PTQ method, which incorporates two core designs aimed at achieving high-precision weight binarization. To address the current trend of higher attention being paid to subsequent dimensions, this invention discards smaller values ​​in each compensation value in blocks, with a higher discard rate for larger distances between blocks, effectively reducing overcompensation for subsequent dimensions. To ensure the quantized weight distribution closely approximates the unquantized distribution, this invention reorders the weight matrix based on the sum of the magnitudes of each column before quantization and records this order. The original order is restored after quantization, ensuring that the overcompensated dimensions in each layer are those with larger original weight magnitudes.

[0027] Figure 1 This is a flowchart illustrating the weighted binary quantization method for large language models provided by the present invention, as shown below. Figure 1 As shown, this invention provides a weight binary quantization method for large language models, comprising: Step 101: Reorder the calibration data matrix and the weight matrix input to the large language model to obtain the reordered weight matrix and the reordered calibration data matrix, and perform Cholesky decomposition on the reordered calibration data matrix to obtain the target calibration data matrix.

[0028] In this invention, the Hessian matrix is ​​first calculated using calibration data. Then, the sums of each column of the weight matrix are calculated and sorted in ascending order. This is done to ensure that channels with larger original values ​​(corresponding to the columns in the weight matrix) are processed more frequently in subsequent compensation. Because channels with larger original values ​​may play a more important role in the model, this sorting method helps to better maintain the quantized weight distribution close to the original distribution in subsequent processing, reducing the impact of quantization on model performance.

[0029] In deep learning quantization scenarios, calibration data used to calculate the Hessian matrix is ​​used to simulate the input distribution of the model in real-world applications, accurately assessing the sensitivity of weight parameters and guiding the quantization process. Calibration data represents various input scenarios the model might encounter during actual deployment. When calculating the Hessian matrix, calibration data is used to perform forward and backward propagation on the model, obtaining the second derivative information of the model parameters (weight matrix) to construct the Hessian matrix. The Hessian matrix reflects the curvature information of the model's loss function with respect to the weight parameters. By analyzing the Hessian matrix, the degree of influence of different weight parameters on model performance can be understood, i.e., parameter sensitivity. In this invention, summing and sorting the columns of the weight matrix is ​​based on the parameter sensitivity information revealed by the Hessian matrix. This ensures that channels with large original values ​​and critical impacts on model performance are more fully processed in subsequent quantization compensation, maintaining the similarity between the quantized weight distribution and the original distribution, and reducing the damage of quantization errors to model performance.

[0030] In this invention, different types of models and application scenarios use calibration data with different physical meanings. For example, in the field of natural language processing, using text from a corpus as calibration data can simulate various text inputs that the model may encounter in actual natural language processing tasks (such as text classification, sentiment analysis, named entity recognition, etc.). For instance, when training a model for sentiment analysis, the calibration data can include texts with different topics and different sentiment tendencies (positive, negative, neutral). The Hessian matrix calculated in this way can accurately reflect the sensitivity of the model weights under different text features, guide the quantization process, and ensure that the quantized model can still accurately determine sentiment tendencies when processing various real texts.

[0031] For certain domain-specific natural language processing tasks, such as medical text analysis and legal document processing, it is necessary to use domain-specific text data as calibration data. For example, when training a model for medical text classification, calibration data can include various medical reports, medical records, diagnostic recommendations, and other texts. These texts have specific lexical, grammatical, and semantic structures, which can more accurately simulate the model's input in real medical scenarios. This makes the calculated Hessian matrix more consistent with the needs of the specific domain, guides the quantization process, and improves the performance of the quantized model in the specific domain.

[0032] After the above sorting, we obtain the reordered weight matrix and the reordered calibration data matrix. The reordered weight matrix is ​​a weight matrix rearranged according to specific rules, while the reordered calibration data matrix is ​​a calibration data matrix corresponding to the reordered weight matrix and processed according to the same sorting rules.

[0033] Furthermore, after regularizing the reordered calibration data matrix, Cholesky decomposition is performed. Cholesky decomposition is a method that decomposes a symmetric positive definite matrix into the product of a lower triangular matrix and its transpose. In this invention, the target calibration data matrix obtained after Cholesky decomposition of the reordered calibration data matrix will be used for subsequent error propagation calculations, providing a mathematical basis and computational basis for the propagation of error between matrices.

[0034] Step 102: Concatenate the quantization results of all rows in the weight block to obtain the weight block quantization result, wherein the weight block is obtained by dividing the reordered weight matrix into blocks.

[0035] In this invention, the reordering weight matrix is ​​divided into blocks, each block being of size [size missing]. β Then, a series of quantization operations are performed on each weight block Wb. First, the rows with the greatest impact on the error (i.e., key rows) are selected, as these rows contribute significantly to the overall error during quantization and require special attention. Next, the key rows are approximated with binarization, a preset binarization threshold is determined for non-key rows, and then non-key rows in the weight block (such as those above and below the key rows) are binarized according to this preset threshold. Finally, the binarization results of these three parts (key rows, the remaining rows above the key rows, and the remaining rows below the key rows) are combined to obtain the quantization result of the weight block. Figure 2 This is a quantization diagram of the original input weight matrix provided by the present invention, which can be referred to. Figure 2 As shown, red rows are key rows, blue rows are non-key rows, and yellow rows are quantified rows.

[0036] After performing the above quantization operation on all weight blocks, the quantization results of all weight blocks are concatenated to obtain the entire weight block quantization result. This result is a matrix containing the quantization information of all blocks, reflecting the state of the weight matrix after block quantization.

[0037] Step 103: Based on the target error matrix, perform error compensation on the reordered weight matrix to obtain the error-compensated reordered weight matrix. The target error matrix is ​​obtained by discarding the target error values ​​in the propagation error matrix. The propagation error matrix is ​​calculated based on the weight blocks, the weight block quantization results, and the target calibration data matrix. The target error values ​​are determined according to the index information of the weight blocks.

[0038] In this invention, the propagation error matrix is ​​calculated using weighted blocks, the quantization results of the weighted blocks, and the target calibration data matrix. The propagation error matrix reflects the distribution of errors generated during the quantization process at different locations (different weighted blocks and different locations within each block). By using the target calibration data matrix (obtained through Cholesky decomposition), the error can be reasonably propagated between different weighted blocks, making the error calculation more accurate and consistent with reality.

[0039] In this invention, target error values ​​are determined based on the index information of the weighted blocks, and these target error values ​​in the propagation error matrix are discarded to obtain the target error matrix. The discard ratio increases with the block index. This is done to suppress overcompensation for subsequent channels. Because as the block index increases, excessive errors may accumulate during error propagation. If all of these errors were used for compensation, the quantization results of subsequent channels might deviate too much from the original distribution, affecting model performance. By gradually increasing the discard ratio, the effect of error compensation can be balanced, avoiding overcompensation.

[0040] Furthermore, error compensation is performed on the reordered weight matrix based on the target error matrix. The error information in the target error matrix is ​​fed back into the reordered weight matrix to correct the errors generated during quantization, making the error-compensated reordered weight matrix closer to the original weight distribution, thereby improving the performance of the quantized model.

[0041] Step 104: Obtain the target quantization weight matrix based on the reordered weight matrix after error compensation.

[0042] In this invention, after error compensation, the reordered weight matrix has already corrected the errors generated during quantization. At this point, restoring the original order of the reordered weight matrix (because a reordering operation was performed in step 101) yields the target quantized weight matrix, which is the final quantized weight matrix obtained after a series of quantization, error propagation, and compensation operations. This matrix minimizes quantization errors while maintaining a weight distribution close to the original distribution, making it better suited for large language models and reducing the negative impact of quantization on model performance.

[0043] The weight binary quantization method for large language models provided by this invention reduces overcompensation for subsequent dimensions by discarding smaller values ​​in each compensation value in blocks, with a higher discard rate for larger distances between two blocks. To make the weight distribution after quantization close to the distribution before quantization, the weight matrix is ​​reordered according to the sum of the magnitudes of each column before quantization, and this order is recorded. Then, the original order is restored after quantization, so that the overcompensated dimensions in each layer are the parts with larger original weight magnitudes. This improves the output accuracy of the model while compressing the model to the maximum extent.

[0044] Based on the above embodiments, the step of reordering the calibration data matrix and the weight matrix input to the large language model to obtain the reordered weight matrix and the reordered calibration data matrix includes: Based on the calibration data, calculate the Hessian matrix corresponding to the weight matrix input to the current layer of the large language model; The Hessian matrix is ​​reordered in ascending order to obtain the reordered calibration data matrix. Summation is performed on each column of the weight matrix, and the summation results are reordered in ascending order to obtain the reordered weight matrix. The step of performing Cholesky decomposition on the reordered calibration data matrix to obtain the target calibration data matrix includes: The reordered calibration data matrix is ​​regularized, and the regularized reordered calibration data matrix is ​​then subjected to Cholesky decomposition to obtain the target calibration data matrix.

[0045] In this invention, the parameters of each weight matrix in the large language model have varying degrees of influence on the model output (such as predicted text, classification results, etc.). The Hessian matrix is ​​a second-order partial derivative matrix that quantifies the sensitivity of the model output to the weight parameters. By calculating the Hessian matrix, we can understand the degree of second-order influence of changes in each weight parameter on the model output, i.e., the curvature information of the parameters.

[0046] Specifically, given calibration data X Using the formula for the Hessian matrix: ; in, H This represents the desired Hessian matrix. XThis represents the calibration data, and the elements in the calculated Hessian matrix are preserved in float32 format. The calculated Hessian matrix reflects the sensitivity of the model output to each weight parameter under the current layer weight matrix. A highly sensitive parameter means that even a small change in it will have a significant impact on the model output.

[0047] Traditional error compensation mechanisms based on index order systematically assign greater weights to high-index channels, thus distorting the original weight distribution of the model. To address this, this invention requires reordering the Hessian matrix, ensuring that subsequent error compensation targets the truly important channels in the original distribution.

[0048] In this invention, the calculated Hessian matrix is ​​reordered in ascending order along both the column direction (dim=0) and the row direction (dim=1). Specifically, the columns of the Hessian matrix are rearranged first, and then the rows of the rearranged matrix are rearranged to obtain the reordered calibration data matrix. Specifically, it can be expressed as: ; By reordering, the arrangement order of elements in the Hessian matrix is ​​changed, so that the new order can be used to determine which channels (corresponding columns of the weight matrix) need to be processed in error compensation.

[0049] Each column of the weight matrix corresponds to a channel of the model, and different channels contribute differently to the model's performance. By summing and sorting each column of the weight matrix, the original importance of each channel can be quantified, allowing subsequent processing to better maintain the quantized weight distribution close to the original distribution. Specifically, for the input weight matrix... W The summation operation is performed along the input channel dimension (i.e., the column direction, dim=1) to obtain the sum for each column. Then, these sums are sorted in ascending order, and the weight matrix is ​​adjusted according to the sorting order. W The columns are rearranged to obtain the reordering weight matrix. Specifically, it can be expressed as: ; Reordering weight matrix By moving channels with higher original importance to higher index positions, these important channels can receive more frequent and stronger error compensation during subsequent quantization, thereby reducing the impact of quantization on model performance. In this invention, these important channels are precisely those that already occupy an important position in the original distribution, not merely those with later index numbers. Simultaneously, to maintain consistency in mathematical operations, this invention also performs corresponding row and column rearrangements on the Hessian matrix. This preparatory operation fundamentally transforms error compensation from a process of introducing distributional bias into a process of strengthening the original important features of the model.

[0050] Furthermore, the reordered calibration data matrix is ​​regularized to ensure the stability of the subsequent Cholesky decomposition and the accuracy of numerical calculations. Next, the regularized reordered calibration data matrix is ​​subjected to Cholesky decomposition to obtain the target calibration data matrix. Specifically, it can be expressed as: ; in, Represents the identity matrix; This represents the regularization coefficient, used to control the strength of regularization.

[0051] In this invention, Cholesky decomposition can be used for subsequent error propagation calculations, allowing for the reasonable propagation of errors between different weight blocks. This makes the error calculation more accurate and consistent with reality, providing a mathematical foundation for subsequent error propagation calculations. The target calibration data matrix can more accurately calculate the distribution of errors generated during quantization at different locations (different weight blocks and different locations within blocks), thereby achieving more effective error compensation and improving the performance of the quantized model.

[0052] Based on the above embodiments, the quantization results of all rows in the weight block are concatenated to obtain the weight block quantization result. The weight block is obtained by dividing the reordered weight matrix into blocks, including: Based on a preset block size, the reordered weight matrix is ​​divided into blocks to obtain multiple weight blocks; The key rows in the weighted block are subjected to approximate binarization to obtain a first quantization result; wherein, the key row is the row with the highest error sensitivity determined from all rows in the weighted block based on the reordered calibration data matrix; Based on a preset binarization threshold, each non-key row in the weighted block is binarized to obtain the second quantization result corresponding to each non-key row. The first quantization result and the second quantization result are concatenated to obtain the weighted block quantization result.

[0053] When dealing with large-scale weight matrices, directly quantizing the entire matrix can lead to high computational complexity and difficulties in optimization. In this invention, the weight matrix is ​​quantized according to a preset size... β Block processing can break down a large quantization problem into multiple smaller quantization subproblems, reducing computational difficulty, improving quantization efficiency, and also facilitating more refined quantization processing for each small block.

[0054] Specifically, based on the pre-set block size β Along the rows and columns of the weight matrix, the reordered weight matrix is ​​divided into multiple relatively small submatrices according to certain rules (such as sequential division). These submatrices are called weight blocks. By processing these blocks, the complex overall quantization problem is transformed into multiple relatively simple local quantization problems, which facilitates independent quantization operations and analysis for each weight block, improving the flexibility and operability of quantization.

[0055] Within a weighted block, different rows exhibit varying sensitivities to quantization errors. Key rows are the most sensitive to quantization errors, and their quantization results significantly impact model performance. To minimize performance degradation during quantization, key rows undergo approximate binarization—binarization is performed while preserving key row information with high precision—to balance quantization accuracy and computational efficiency. In this invention, key rows are determined using a reordered calibration data matrix, as this matrix reflects the relationship between weight parameters and model output sensitivity, accurately identifying the rows within the weighted block most sensitive to quantization errors.

[0056] Specifically, the error sensitivity index of each row in the weighted block is calculated using the reordered calibration data matrix (for example, it can be determined based on the sensitivity information corresponding to the weight parameters of that row in the reordered calibration data matrix, such as the values ​​of relevant elements in the Hessian matrix). Then, all rows are sorted in descending order of error sensitivity, and the row with the highest error sensitivity is selected as the key row.

[0057] Furthermore, a specific function is invoked to perform a high-precision approximation preservation operation on the key rows. In this invention, this function can, based on the numerical characteristics of the key rows, employ an appropriate method (such as multinomial fitting, neural network approximation, etc.) to approximate the key row information into a form suitable for binarization while preserving its main information. Then, binarization processing is performed to obtain the first quantization result. By performing approximate binarization processing on the key rows, quantization can be achieved while ensuring that key row information is not lost as much as possible, reducing the critical impact of quantization errors on model performance, and improving the accuracy of the quantized model.

[0058] For non-critical rows, since their sensitivity to quantization errors is relatively low, a relatively simple binarization method can be used for processing. In this invention, a preset binarization threshold is used. This is a critical value used to determine whether a weight value is quantized to 0 or 1. The optimal binarization threshold is determined by searching for this threshold. This allows the binarization results of non-critical rows to meet certain quantization accuracy requirements while minimizing quantization errors.

[0059] In this invention, the optimal binarization threshold is searched. The optimal binarization threshold can be searched within a certain range using methods such as traversal and optimization algorithms (e.g., gradient descent, genetic algorithm). The search objective could be to minimize the quantization error after binarizing non-critical rows, or to minimize the performance loss of the quantized model.

[0060] Furthermore, based on the optimal binarization threshold obtained from the search... This involves setting a predefined binarization threshold and performing binarization on each non-critical row in the weighted block. The specific rule is: if the weight value is greater than or equal to... If the weight value is less than 1, then it is quantized to 1; if the weight value is less than 1, then it is quantized to 1. If the threshold is zero, the data is quantized to 0, thus obtaining the second quantization result for each non-critical row. This invention performs binarization processing on non-critical rows by searching for the optimal binarization threshold, which can improve the quantization accuracy of non-critical rows as much as possible while ensuring computational efficiency and reducing the impact of quantization errors on model performance.

[0061] After completing the approximate binarization of key rows and the binarization of non-key rows, their quantization results need to be integrated to restore the overall structure of the weight blocks and obtain the complete weight block quantization result. The concatenation process can recombine the quantization results of different rows according to their positional relationship in the original weight blocks to form a quantized matrix of the same size as the original weight blocks.

[0062] In this invention, the first quantization result (quantization result of the key row) and the second quantization result (quantization result of each non-key row) are concatenated according to their corresponding row positions, based on the row index order of the key rows and non-key rows in the original weight block. After concatenation, a complete weight block quantization result is obtained, which provides a foundation for the subsequent quantization result of combining all weight block quantization results into the entire weight matrix. It also facilitates further evaluation and application of the quantized weight matrix.

[0063] Based on the above embodiments, the step of performing error compensation on the reordering weight matrix based on the target error matrix to obtain the error-compensated reordering weight matrix includes: Repeatedly execute the preset iterative steps to perform error compensation on the reordered weight matrix until all weight blocks in the reordered weight matrix have completed error compensation, and obtain the error-compensated reordered weight matrix; The iterative steps include: The block quantization error of the current round is calculated based on the quantization error between the current round's weight block and the current round's weight block quantization result; wherein, the current round's weight block is obtained after error compensation based on the target error matrix of the previous round; The original error matrix of the current round is obtained based on the ratio between the block quantization error of the current round and the calibration data matrix sub-block of the current round. The calibration data matrix sub-block of the current round is determined in the target calibration data matrix based on the row and column information of the weight block of the current round. Based on the row information of the weight block in the current round and the column information of the weight block in the next round, the weight block association matrix of the current round is obtained from the target calibration data matrix; The propagation error matrix of the current round is obtained by multiplying the original error matrix of the current round and the weighted block correlation matrix of the current round. Based on the tensor values ​​in the propagation error matrix of the current round and the distance decay factor of the current round, the target error values ​​in the propagation error matrix of the current round are discarded to obtain the target error matrix of the current round. The distance decay factor of the current round is determined based on the index information of the weight block of the current round. Based on the target error matrix of the current round, error compensation is performed on the weight blocks of the next round.

[0064] In this invention, each weight block generates quantization error during the quantization process, which affects model performance. By performing error compensation on each weight block, the impact of quantization error on the overall weight matrix is ​​gradually reduced, ultimately resulting in an error-compensated reordered weight matrix, thereby improving model performance and accuracy.

[0065] Specifically, following a pre-defined iterative process, each weight block in the reordered weight matrix is ​​processed sequentially until all weight blocks have undergone error compensation. Each iteration processes one weight block, updating relevant matrices and parameters based on the processing results to prepare for the next weight block. Through multiple iterations of error compensation, quantization errors can be reduced more comprehensively and effectively, making the quantized weight matrix closer to the original weight matrix and improving the model's accuracy and generalization ability.

[0066] In this invention, not only is the quantization error of the current block intelligently compensated for in subsequent unquantized blocks, but an overcompensation mechanism is also used to prevent overcompensation. The formula for calculating the error matrix is ​​as follows: ; in, This indicates the weighted block currently being processed (in the current round). i The b arrive b+β OK. This indicates the above weighted blocks. i The corresponding quantized result is the weighted block quantization result of the current round. This represents the block quantization error of the current round, which includes all the information lost by the weighted blocks of the current round after quantization. This is a sub-block of the target calibration data matrix after Cholesky decomposition, specifically a sub-block of the calibration data matrix for the current round. It corresponds to the rows and columns of the weight block for the current round, characterizing the sensitivity and correlation between the internal weight variables of the weight block. The formula for the above error matrix is ​​the standardized error, i.e., the original error matrix for the current round. In this invention, the block quantization error for the current round is directly divided by the target calibration data matrix. The calibration data matrix sub-block of the current round is equivalent to scaling and rotating the original error matrix to place it in an "optimal" coordinate system defined by Hessian information. The original error matrix is ​​thus processed. E It is more suitable for subsequent compensation and can more effectively reduce overall quantitative losses.

[0067] In this invention, quantization error is the difference between the original weight value and the quantized weight value during the quantization process, reflecting the information lost during quantization. Calculating the block quantization error of the current round is to clarify the error generated by the current weight block during the quantization process, so as to perform error compensation in subsequent rounds. The weight block of the current round is obtained after error compensation based on the target error matrix of the previous round, which means that the weight block of each round is continuously updated and optimized according to the error information.

[0068] In this invention, there are certain interrelationships and sensitivities among the weight blocks. The weight block correlation matrix encodes the interrelationships and sensitivities between the weight of the current block (current round) and the weight of the target block (next round). By obtaining this correlation matrix, the influence of the error of the current weight block on the subsequent unquantized weight blocks can be clearly defined, thereby achieving optimal error allocation based on second-order derivative information.

[0069] Specifically, let the row range of the weighted block in the current round be... The column range for the next round of weighted block partitioning is Obtain the corresponding sub-block from the target calibration data matrix. This sub-block is the weighted block correlation matrix for the current round. In this invention, the weighted block correlation matrix provides crucial information for error propagation, enabling errors to be rationally distributed based on the relationships between weight parameters, thereby improving the effectiveness of error compensation.

[0070] Furthermore, by multiplying the original error matrix of the current round with the weight block correlation matrix of the current round, the extent to which the target block (i.e., the next weight block) needs to be adjusted to compensate for the quantization error of the current block can be calculated. This is an optimal error allocation method based on second-order derivative information, which can more effectively reduce the overall quantization loss.

[0071] Specifically, let the original error matrix of the current round be... E The weighted block association matrix for the current round is: Then the propagation error matrix of the current round The propagation error matrix clarifies the specific impact of the current block error on subsequent target blocks, providing a specific adjustment direction and magnitude for subsequent error compensation.

[0072] Furthermore, during error propagation, not all errors need compensation. Some minor errors may be noise, and compensating for them could lead to overfitting and reduce the model's generalization ability. By introducing a distance decay factor and a discard operation, the number of errors to discard can be dynamically determined based on the distance between the target block and the current block. Only those errors with large magnitudes and significant importance are compensated, while minor errors that may be noise are discarded. Specifically, the distance decay factor is determined by assuming the index of the weighted block in the current round is... i The total number of blocks is N The starting index of the weighted block in the current round is b Then the distance decay factor and i / (Nb) Relatedly, the farther the target block is from the current block, the larger this value becomes, and therefore the higher the proportion of discards, k.

[0073] Then, determine the number of elements to be discarded. ,in, It is the propagation error matrix The total number of elements in the middle. p This is a preset base discard ratio. Next, the function is used. Find the propagation error matrix The smallest middle tensor value k The index of each element, i.e., the propagation error matrix. The element with the smallest absolute value is determined as the target error value.

[0074] For the identified target error values, discard them and set them to zero. The target error matrix of the current round is obtained. In this invention, the discard operation can effectively prevent insignificant errors from being amplified and propagated, suppress overfitting, improve the generalization ability of the model, and at the same time ensure that important errors can be reasonably compensated.

[0075] Finally, error compensation is applied to the next round of weight partitioning to obtain the error-compensated weight partitioning, as shown in the formula. By continuously performing error compensation, the propagation and accumulation of quantization errors in the weight matrix can be gradually reduced, making the quantized weight matrix closer to the original weight matrix and improving the model's performance and accuracy.

[0076] Based on the above embodiments, the step of discarding the target error values ​​in the propagation error matrix of the current round based on the tensor values ​​in the propagation error matrix of the current round and the distance decay factor of the current round to obtain the target error matrix of the current round includes: Based on the magnitude of the tensor values ​​in the propagation error matrix of the current round, obtain the tensor value sorting result of the current round; Obtain the difference between the total number of weight blocks in the reordered weight matrix and the index information of the weight blocks in the current round, and obtain the distance decay factor of the current round based on the ratio between the cyclic index information corresponding to the weight blocks in the next round and the difference. Based on the distance decay factor of the current round, the tensor values ​​of the current round are sorted, with the top... k The smallest tensor value is determined as the target error value in the propagation error matrix of the current round; The target error value in the propagation error matrix of the current round is set to zero to obtain the target error matrix of the current round.

[0077] In this invention, the formulas for the error propagation and discarding steps are specifically as follows: in, The propagation error matrix for the current round (denoted as ) All tensor values ​​(i.e., error values) in the problem are sorted by their absolute values ​​to obtain a sorted result. The less important items (i.e., errors with smaller absolute values) are identified to prepare for subsequent discarding. In this invention, the sorting is based on the absolute value of the tensor values ​​(rather than the original values) because the contribution of the error is determined by its magnitude.

[0078] Then, calculate the distance decay factor for the current round. Among them, the distance decay factor is the core factor for dynamically adjusting the discard ratio. This indicates the current loop index (corresponding to the index of the next round of weighted blocks). N Total number of blocks b This is the starting index of the current block. This can be understood as the distance from the current block to subsequent blocks. It quantifies the impact of the distance between the target block and the current block on error propagation. That is, the greater the distance, the larger the decay factor and the higher the discard ratio.

[0079] Furthermore, determine the number of target error values. Choose the smallest k This value is used as the target error value, thereby discarding more small errors during long-distance propagation and retaining more errors during short-distance propagation, preventing small errors (which may be noise) from accumulating and being amplified in multiple rounds of propagation.

[0080] Furthermore, set the target error value to zero. And generate the target error matrix. This invention will use the current round propagation error matrix. The elements corresponding to the target error values ​​are forcibly set to zero to obtain the target error matrix. This avoids the model from overfitting to noise by discarding small errors. In subsequent error compensation processes, the model weights can be adjusted more precisely, thereby improving the quantization effect and the model's generalization ability.

[0081] Based on the above embodiments, obtaining the target quantization weight matrix according to the error-compensated reordered weight matrix includes: Based on the reordering operation performed on the weight matrix, the reverse reordering operation is performed on the error-compensated reordered weight matrix to obtain the target quantization weight matrix.

[0082] In this invention, to optimize the error compensation effect, the columns (input channel dimensions) of the original weight matrix are first reordered.

[0083] After block quantization and error compensation, the column order of the resulting weight matrix remains the rearranged order. To ensure compatibility with the input of the next layer of the model, the inverse operation of reordering needs to be performed. Specifically, based on the recorded original column order, the column order of the quantized weight matrix is ​​restored to its initial arrangement. In this invention, important channels (channels with larger numerical amplitudes) that are moved to the back after reordering and receive enhanced error compensation are then returned to their original positions through the inverse operation. The core of this step is to eliminate the impact of reordering on the model structure and ensure that the quantized weight matrix can be directly used for the next layer's calculation.

[0084] After the inverse operation, the output weight matrix has the exact same channel order as the original model, ensuring the compatibility of the model structure. The weight values ​​are binarized with high quality through block quantization and error compensation, significantly reducing storage and computational pressure. Ultimately, while compressing the model to the maximum extent (e.g., quantizing the weights to 1 bit), the model's high performance and compatibility are maintained through reordering and inverse operations, solving the problem of poor generalization ability of traditional low-bit quantization methods.

[0085] The weighted binary quantization system for large language models provided by this invention is described below. The weighted binary quantization system for large language models described below can be referred to in correspondence with the weighted binary quantization method for large language models described above.

[0086] Figure 3 This is a schematic diagram of the structure of the weighted binary quantization system for large language models provided by the present invention, as shown below. Figure 3As shown, this invention provides a weighted binary quantization system for large language models, including a reordering module 301, a block quantization module 302, a block error compensation module 303, and a quantization matrix output module 304. The reordering module 301 reorders the calibration data matrix and the weight matrix input to the large language model to obtain a reordered weight matrix and a reordered calibration data matrix, and performs Cholesky decomposition on the reordered calibration data matrix to obtain a target calibration data matrix. The block quantization module 302 concatenates the quantization results of all rows in the weight blocks to obtain the weight block quantization result. The reordered weight matrix is ​​obtained by dividing it into blocks; the block error compensation module 303 is used to perform error compensation on the reordered weight matrix based on the target error matrix to obtain the error-compensated reordered weight matrix, wherein the target error matrix is ​​obtained by discarding the target error values ​​in the propagation error matrix; the propagation error matrix is ​​calculated based on the weight blocks, the weight block quantization results, and the target calibration data matrix; the target error value is determined according to the index information of the weight blocks; the quantization matrix output module 304 is used to obtain the target quantization weight matrix based on the error-compensated reordered weight matrix.

[0087] The present invention provides a weighted binary quantization method and system for large language models. By discarding smaller values ​​in each compensation value in blocks, with a higher discard rate for larger distances between two blocks, the overcompensation of subsequent dimensions is reduced. To make the weight distribution after quantization close to the distribution before quantization, the weight matrix is ​​reordered according to the sum of the magnitudes of each column before quantization, and this order is recorded. Then, the original order is restored after quantization, so that the overcompensated dimensions in each layer are the parts with larger original weight magnitudes. This improves the output accuracy of the model while compressing the model to the maximum extent.

[0088] The system provided in this embodiment of the invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0089] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4As shown, the electronic device may include: a processor 401, a communications interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communications interface 402, and the memory 403 communicate with each other through the communication bus 404. Processor 401 can call logic instructions in memory 403 to execute a weighted binary quantization method for a large language model. This method includes: reordering the calibration data matrix and the weight matrix input to the large language model to obtain a reordered weight matrix and a reordered calibration data matrix; performing Cholesky decomposition on the reordered calibration data matrix to obtain a target calibration data matrix; concatenating the quantization results of all rows in the weight blocks to obtain weight block quantization results, wherein the weight blocks are obtained by dividing the reordered weight matrix into blocks; performing error compensation on the reordered weight matrix based on a target error matrix to obtain an error-compensated reordered weight matrix, wherein the target error matrix is ​​obtained by discarding target error values ​​in the propagation error matrix; the propagation error matrix is ​​calculated based on the weight blocks, the weight block quantization results, and the target calibration data matrix; the target error value is determined based on the index information of the weight blocks; and obtaining the target quantized weight matrix based on the error-compensated reordered weight matrix.

[0090] Furthermore, the logical instructions in the aforementioned memory 403 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is capable of executing the weighted binary quantization method for large language models provided by the above methods, the method comprising: performing a reordering operation on the calibration data matrix and the weight matrix input to the large language model respectively to obtain a reordered weight matrix and a reordered calibration data matrix, and performing Cholesky decomposition on the reordered calibration data matrix to obtain a target calibration data matrix; and performing quantization on the quantization results of all rows in the weight block. The weighted block quantization result is obtained by concatenating the rows, wherein the weight block is obtained by dividing the reordered weight matrix into blocks; based on the target error matrix, error compensation is performed on the reordered weight matrix to obtain the error-compensated reordered weight matrix, wherein the target error matrix is ​​obtained by discarding the target error values ​​in the propagation error matrix; the propagation error matrix is ​​calculated based on the weight blocks, the weight block quantization result, and the target calibration data matrix; the target error value is determined according to the index information of the weight blocks; and the target quantized weight matrix is ​​obtained based on the error-compensated reordered weight matrix.

[0092] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program performs the weighted binary quantization method for large language models provided in the above embodiments. The method includes: reordering a calibration data matrix and a weight matrix input to a large language model to obtain a reordered weight matrix and a reordered calibration data matrix; performing Cholesky decomposition on the reordered calibration data matrix to obtain a target calibration data matrix; concatenating the quantization results of all rows in a weight block to obtain a weight block quantization result, wherein the weight block is obtained by dividing the reordered weight matrix into blocks; performing error compensation on the reordered weight matrix based on a target error matrix to obtain an error-compensated reordered weight matrix, wherein the target error matrix is ​​obtained by discarding target error values ​​in a propagation error matrix; the propagation error matrix is ​​calculated based on the weight blocks, the weight block quantization result, and the target calibration data matrix; the target error value is determined according to the index information of the weight blocks; and obtaining a target quantized weight matrix based on the error-compensated reordered weight matrix.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A weighted binary quantization method for large language models, characterized in that, include: The calibration data matrix and the weight matrix input to the large language model are reordered to obtain the reordered weight matrix and the reordered calibration data matrix, respectively. The reordered calibration data matrix is ​​then subjected to Cholesky decomposition to obtain the target calibration data matrix. The quantization results of all rows in the weight block are concatenated to obtain the weight block quantization result, wherein the weight block is obtained by dividing the reordered weight matrix into blocks; Based on the target error matrix, error compensation is performed on the reordered weight matrix to obtain an error-compensated reordered weight matrix. The target error matrix is ​​obtained by discarding the target error values ​​in the propagation error matrix. The propagation error matrix is ​​calculated based on the weight blocks, the quantization results of the weight blocks, and the target calibration data matrix. The target error values ​​are determined according to the index information of the weight blocks. The target quantization weight matrix is ​​obtained based on the reordered weight matrix after error compensation.

2. The weight binary quantization method for large language models according to claim 1, characterized in that, The process of reordering the calibration data matrix and the weight matrix input to the large language model to obtain a reordered weight matrix and a reordered calibration data matrix includes: Based on the calibration data, calculate the Hessian matrix corresponding to the weight matrix input to the current layer of the large language model; The Hessian matrix is ​​reordered in ascending order to obtain the reordered calibration data matrix. Summation is performed on each column of the weight matrix, and the summation results are reordered in ascending order to obtain the reordered weight matrix. The step of performing Cholesky decomposition on the reordered calibration data matrix to obtain the target calibration data matrix includes: The reordered calibration data matrix is ​​regularized, and the regularized reordered calibration data matrix is ​​then subjected to Cholesky decomposition to obtain the target calibration data matrix.

3. The weight binary quantization method for large language models according to claim 1, characterized in that, The quantization results of all rows in the weight block are concatenated to obtain the weight block quantization result. The weight block is obtained by dividing the reordered weight matrix into blocks, including: Based on a preset block size, the reordered weight matrix is ​​divided into blocks to obtain multiple weight blocks; The key rows in the weighted block are subjected to approximate binarization to obtain a first quantization result; wherein, the key row is the row with the highest error sensitivity determined from all rows in the weighted block based on the reordered calibration data matrix; Based on a preset binarization threshold, each non-key row in the weighted block is binarized to obtain the second quantization result corresponding to each non-key row. The first quantization result and the second quantization result are concatenated to obtain the weighted block quantization result.

4. The weight binary quantization method for large language models according to claim 1, characterized in that, The step of performing error compensation on the reordering weight matrix based on the target error matrix to obtain the error-compensated reordering weight matrix includes: Repeatedly execute the preset iterative steps to perform error compensation on the reordered weight matrix until all weight blocks in the reordered weight matrix have completed error compensation, and obtain the error-compensated reordered weight matrix; The iterative steps include: The block quantization error of the current round is calculated based on the quantization error between the current round's weight block and the current round's weight block quantization result; wherein, the current round's weight block is obtained after error compensation based on the target error matrix of the previous round; The original error matrix of the current round is obtained based on the ratio between the block quantization error of the current round and the calibration data matrix sub-block of the current round. The calibration data matrix sub-block of the current round is determined in the target calibration data matrix based on the row and column information of the weight block of the current round. Based on the row information of the weight block in the current round and the column information of the weight block in the next round, the weight block association matrix of the current round is obtained from the target calibration data matrix; The propagation error matrix of the current round is obtained by multiplying the original error matrix of the current round and the weighted block correlation matrix of the current round. Based on the tensor values ​​in the propagation error matrix of the current round and the distance decay factor of the current round, the target error values ​​in the propagation error matrix of the current round are discarded to obtain the target error matrix of the current round. The distance decay factor of the current round is determined based on the index information of the weight block of the current round. Based on the target error matrix of the current round, error compensation is performed on the weight blocks of the next round.

5. The weight binary quantization method for large language models according to claim 4, characterized in that, The step of discarding target error values ​​in the propagation error matrix of the current round based on the tensor values ​​and the distance decay factor of the current round, to obtain the target error matrix of the current round, includes: Based on the magnitude of the tensor values ​​in the propagation error matrix of the current round, obtain the tensor value sorting result of the current round; Obtain the difference between the total number of weight blocks in the reordered weight matrix and the index information of the weight blocks in the current round, and obtain the distance decay factor of the current round based on the ratio between the cyclic index information corresponding to the weight blocks in the next round and the difference. Based on the distance decay factor of the current round, the tensor values ​​of the current round are sorted, with the top... k The smallest tensor value is determined as the target error value in the propagation error matrix of the current round; The target error value in the propagation error matrix of the current round is set to zero to obtain the target error matrix of the current round.

6. The weight binary quantization method for large language models according to any one of claims 1 to 5, characterized in that, The step of obtaining the target quantization weight matrix based on the error-compensated reordered weight matrix includes: Based on the reordering operation performed on the weight matrix, the reverse reordering operation is performed on the error-compensated reordered weight matrix to obtain the target quantization weight matrix.

7. A weighted binary quantization system for large language models, characterized in that, include: The reordering module is used to reorder the calibration data matrix and the weight matrix input to the large language model, respectively, to obtain the reordered weight matrix and the reordered calibration data matrix, and to perform Cholesky decomposition on the reordered calibration data matrix to obtain the target calibration data matrix. The block quantization module is used to concatenate the quantization results of all rows in the weight block to obtain the weight block quantization result. The weight block is obtained by dividing the reordered weight matrix into blocks. The block error compensation module is used to perform error compensation on the reordered weight matrix based on the target error matrix to obtain the error-compensated reordered weight matrix. The target error matrix is ​​obtained by discarding target error values ​​in the propagation error matrix. The propagation error matrix is ​​calculated based on the weight blocks, the weight block quantization results, and the target calibration data matrix. The target error value is determined according to the index information of the weight blocks. The quantization matrix output module is used to obtain the target quantization weight matrix based on the reordered weight matrix after error compensation.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the weighted binary quantization method for large language models as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the weighted binary quantization method for large language models as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the weighted binary quantization method for large language models as described in any one of claims 1 to 6.

Images