Large language model quantification method based on wavelet transform and BiLLM
By combining wavelet transform and BiLLM algorithm to quantify large language models, the problems of degradation of accuracy after quantization and insufficient computing storage requirements in the prior art are solved, and efficient model compression and performance retention are achieved.
Patent Information
- Application Number
- CN202411702181.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-03-11
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When quantifying large language models, it is difficult for the prior art to effectively reduce computing and storage requirements while maintaining accuracy, especially on resource-constrained devices.
The large language model quantization method based on wavelet transform and BiLLM is adopted to decompose the network weight matrix through Haar wavelet transform, block and quantify the low-frequency coefficient part, and use the BiLLM binary residual approximation method to compress the high-frequency coefficient part.
While greatly compressing amplitude, most of the capabilities of the original model are retained, which significantly reduces storage and computing overhead, and maintains high performance and accuracy during the model inference process.
Smart Images
Figure CN119599083A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of large language models, and in particular relates to a large language model quantization method based on wavelet transform and BiLLM. Background Art
[0002] Large language models (such as GPT-3, OPT, etc.) usually contain billions or even hundreds of billions of parameters, and their weight matrices are huge, resulting in a large amount of computing memory and storage space required when deployed. This requirement makes it very difficult to run models on resource-constrained devices (such as edge devices or mobile devices). Using special quantization methods to compress weights from high precision (such as 32-bit floating point numbers) to low precision (such as 8-bit, 4-bit or even lower precision) can significantly reduce the memory and storage usage of the model.
[0003] Traditional quantization methods, such as fixed-point quantization and uniform quantization, use fixed-point numbers and fixed ranges, are widely used in hardware acceleration, and have high computational efficiency. However, for models with uneven weight distribution or at low bit widths, they may lead to reduced accuracy and large information loss. Therefore, it is very necessary to study new quantization methods that can better retain the original information of the model while maintaining accuracy, improve the performance of the quantized model, and further reduce computing and storage requirements. Summary of the invention
[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to propose a large language model quantization method based on wavelet transform and BiLLM.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] Large language model quantization method based on wavelet transform and BiLLM, including:
[0007] Enter LLM;
[0008] For each network weight matrix of LLM, Haar wavelet transform is applied to decompose the network weight matrix to obtain the wavelet coefficient matrix; the low-frequency coefficient part in the wavelet coefficient matrix is divided into blocks, the optimal threshold is selected, the weights less than the optimal threshold are reset to zero, and the part not less than the optimal threshold is quantized to compress the low-frequency coefficient part of the wavelet coefficient matrix;
[0009] The BiLLM binary residual approximation method is used to compress the high-frequency coefficient part of the wavelet coefficient matrix;
[0010] After each network weight matrix of LLM is processed, the quantization of LLM is completed.
[0011] Furthermore, the low-frequency coefficient part in the wavelet coefficient matrix is divided into blocks, the optimal threshold is selected, the weights less than the optimal threshold are reset to zero, and the part not less than the optimal threshold is quantized, specifically:
[0012] After dividing the low-frequency coefficient part of the wavelet coefficient matrix into blocks, the sub-matrix set obtained by dividing the blocks is {A1 {k,j}};
[0013] Traverse the sub-matrix set {k,j} For each matrix block N0 in}, calculate the matrix block N0 threshold set quan_candidates;
[0014] Traverse each threshold quan in the threshold set quan_candidates, set the elements in the matrix block whose absolute value is less than the threshold to zero, and quantize the elements not less than the threshold to the threshold multiplied by the sign of the element, and obtain the sparsely quantized matrix block N1. The calculation process of the element N1(i,j) in the matrix block N1 is:
[0015]
[0016] Calculate the ratio r of non-zero elements of the sparse matrix block N1:
[0017]
[0018] Among them, count_nonzero(N1) is the number of non-zero elements in the sparse matrix block N1, and (N1).size is the number of elements in the sparse matrix block N1, that is, the matrix block size.
[0019] Furthermore, the quantization threshold of the wavelet low-frequency coefficients adopts linear quantization in the floating-point exponent range, and the calculation method of the threshold set quan_candidates is expressed as:
[0020] quan_candidates={min(N0(i,j))+i·l|i∈{0,1,2,4,8,16}}
[0021] in, That is, the weight value is divided into 6 parts according to the floating point exponent range; min(|N0(i,j)|) and max(|N0(i,j)|) correspond to the minimum and maximum absolute values of the matrix elements in a given matrix block N0.
[0022] Furthermore, before traversing each threshold value quan of the threshold value set quan_candidates, it also includes initializing the optimal threshold value quan0=0, C0=0;
[0023] When traversing each threshold quan in the threshold set quan_candidates, the following steps are also included:
[0024] Calculate the cost function:
[0025]
[0026] Where H represents the Hessian matrix obtained by taking the second-order partial derivative of the weights under the second norm of the difference between the LLM estimate and the observed value, ‖H(N0)-H(N1)‖ F represents the Frobenius-norm of the difference vector of the Hessian matrix H corresponding to the matrix block N0 and the sparse matrix block N1;
[0027] If Cost(quan) is greater than C0, quan0 is updated to the current threshold quan, and C0 is updated to the value of the current cost function Cost(quan).
[0028] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0029] 1. The present invention combines the traditional wavelet transform data compression strategy and the BiLLM binary residual approximation algorithm for LLM, uses Haar wavelet transform to extract model network approximation information and compresses it, and then uses the BiLLM strategy to further compress the remaining high-frequency coefficient matrix, while ensuring a very large compression amplitude, retaining most of the capabilities of the original model; the weight quantization scheme of the wavelet low-frequency coefficients uses linear quantization of the weight value according to the floating-point exponent range instead of linear quantization on the real axis, which can reduce the number of bits required for weight expression while maintaining the expressive power of the large model, and the high-frequency and low-frequency weights use different word lengths to represent data, further reducing the size of the quantized model data. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is the overall flow chart of the method of the present invention;
[0031] Figure 2 It is a schematic diagram of the process of compressing the low-frequency coefficient part of the wavelet coefficient matrix of the present invention;
[0032] Figure 3 It is a schematic diagram of the process of compressing the high-frequency coefficient part of the wavelet coefficient matrix of the present invention. DETAILED DESCRIPTION
[0033] Explanation of related terms:
[0034] LLM: Large Language Model, large language model;
[0035] BiLLM: Binary Large Language Model, a post-training quantization scheme. BiLLM tailors an extreme compression mechanism for pre-trained large language models. For details of the scheme, please refer to the article: Huang, Wei, etc., BiLLM: Pushing the Limit of Post-Training Quantization for LLMs, ICML 2024.
[0036] Haar wavelet transform: Haar wavelet transform was proposed by Alfréd Haar in 1909. It is the simplest transform in wavelet transform and the earliest wavelet transform. It is a special case of Dobesi wavelet with N=2, which can be called D2. It decomposes and reconstructs signals through two basic wavelet functions; the process of Haar wavelet transform includes two stages: signal decomposition and reconstruction. In the decomposition stage, the signal is decomposed into low-frequency coefficients and high-frequency coefficients. In the reconstruction stage, the original signal is restored through the linear combination of low-frequency coefficients and high-frequency coefficients. Haar wavelet transform is often used in image compression, signal denoising and feature extraction, and has the characteristics of simple calculation and fast operation speed.
[0037] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.
[0038] Example
[0039] This embodiment takes the large language model OPT-1.3B as an example; the OPT-1.3B model contains 24 layers (layer 0 to layer 23), and the weight matrix of layer 0 is used as an example for explanation. The key weight matrix of layer 0 includes:
[0040] self_attn.k_proj: key projection matrix in the multi-head self-attention mechanism;
[0041] self_attn.v_proj: value projection matrix in the multi-head self-attention mechanism;
[0042] self_attn.q_proj: query projection matrix in multi-head self-attention mechanism;
[0043] self_attn.out_proj: output projection matrix of the multi-head self-attention mechanism;
[0044] fc1: the first fully connected layer in the feedforward neural network stage;
[0045] fc2: The second fully connected layer in the feedforward neural network stage.
[0046] like Figure 1 As shown, the present invention, a large language model quantization method based on wavelet transform and BiLLM, comprises:
[0047] S1, input large language model LLM;
[0048] S2. For each network weight matrix of LLM, the Haar wavelet transform is applied to decompose the network weight matrix to obtain the wavelet coefficient matrix; the low-frequency coefficient part in the wavelet coefficient matrix is divided into blocks, the optimal threshold is selected, the weights less than the optimal threshold are reset to zero, and the part not less than the optimal threshold is quantized to compress the low-frequency coefficient part of the wavelet coefficient matrix; Figure 2 As shown, specifically:
[0049] Apply Haar wavelet transform to decompose the network weight matrix, specifically:
[0050] For LLM, the set of network weight matrices of each layer is set to L is the number of hidden layers of LLM;
[0051] Haar wavelet transform is used to decompose the LLM network weight matrix and extract the low-frequency matrix A1 k , expressed as:
[0052] A1 k =Y T M k U T
[0053] Among them, U represents the wavelet basis filter matrix for e-th low-frequency decomposition; M k Representing a collection Any matrix element in ; e is the number of wavelet transforms.
[0054] In this embodiment, the self_attn.k_proj matrix of the 0th layer of the OPT-1.3B model is selected for detailed description, and the dimension of the self_attn.k_proj matrix is 2048×2048.
[0055] The Haar wavelet transform is used to perform a secondary decomposition of the network weight matrix of the model to obtain the low-frequency coefficient part of the wavelet coefficient matrix and the high-frequency coefficient part of the wavelet coefficient matrix. This part of the low-frequency component represents the main features in the model weight matrix and retains the important information and structure of the model. The high-frequency part contains more details and noise information, which will be further processed in subsequent steps. By extracting low-frequency approximate information, the number of parameters that need to be processed can be significantly reduced to prepare for the subsequent binarization process. The wavelet decomposition process of the weight matrix is implemented by the function haar_wavelet_decomposition. This function accepts the original weight matrix parameters as input, and uses the wavedec2 function provided by the Python library pywt for secondary decomposition to extract the low-frequency coefficient part of the matrix. In the original model, the coefficients of the weight matrix are represented by 16-bit floating point numbers. After wavelet decomposition, the dimension of the low-frequency coefficient matrix is only the original (reduced from 2048×2048 to 512×512), which is equivalent to each weight parameter of the original self_attn.k_proj matrix being represented by only 1 bit after wavelet decomposition.
[0056] The low-frequency coefficient part in the wavelet coefficient matrix is divided into blocks, specifically:
[0057] For A1 k The matrix is divided into blocks, and the size of each block does not exceed f*f. The set of sub-matrices obtained by scoring the blocks is {A1 {k,j}}, f is the side length of the matrix block. In this embodiment, the low-frequency matrix A1 extracted by the secondary wavelet decomposition is k , the data is divided into blocks with a side length of f=64, and each block is a 64×64 matrix block.
[0058] In this embodiment, an optimal threshold is selected, the weights less than the optimal threshold are reset to zero, and the weights not less than the optimal threshold are quantized, specifically:
[0059] Traverse the sub-matrix set {k,j}}, calculate the absolute minimum value min(|N0(i,j)|) and maximum value max(|N0(i,j)|) of the matrix elements in the matrix block N0, and generate the threshold set quan_candidates:
[0060] quan_candidates={min(N0(i,j))+i·l|i∈{0,1,2,4,8,16}}
[0061] in, That is, the weight value is divided into 6 parts according to the floating point exponent range;
[0062] Initialize the optimal threshold quan0 to 0 and C0 to 0; traverse each threshold quan in the threshold set quan_candidates, set the elements in the matrix block whose absolute value is less than the threshold to zero, and quantize the elements not less than the threshold to the threshold multiplied by the sign of the element, and obtain the sparsely quantized matrix block N1. The calculation process of the element N1(i,j) in the matrix block N1 is:
[0063]
[0064] Calculate the ratio r of non-zero elements of the sparse matrix block N1:
[0065]
[0066] Among them, count_nonzero(N1) is the number of non-zero elements in the sparse matrix block N1, and (N1).size is the number of elements in the sparse matrix block N1, that is, the matrix block size.
[0067] When traversing each threshold quan in the threshold set quan_candidates, the following steps are also included:
[0068] The sparsification matrix block process requires the Hessian matrix H to be updated batch by batch. The calibration data uses the Hugging Face C4 (Colossal Clean Crawled Corpus) dataset, and the data batch size is nsamples = 128. The input inp and output out are used to update the Hessian matrix H through the add_batch method in the SPAGPTQ class. In addition, in order to ensure the stability of the Hessian matrix H in numerical calculations, the damping factor percdamp = 0.01 is added to correct the diagonal elements of the Hessian matrix. After the Hessian matrix H is updated, the cost function Cost(quan) under different thresholds is calculated, and the optimal threshold quan0 is continuously updated iteratively by minimizing the cost function;
[0069] The calculation formula of the cost function Cost(quan) is:
[0070]
[0071] Where H represents the Hessian matrix obtained by taking the second-order partial derivative of the weights under the second norm of the difference between the LLM model estimate and the observed value, ‖H(N0)-H(N1)‖ F represents the Frobenius-norm of the difference vector between the Hessian matrix H corresponding to the matrix block N0 and the sparse matrix block N1; H(N0) and H(N1) represent the performance of the Hessian matrix of the original matrix block and the sparse matrix block on the calibration data set, respectively.
[0072] If Cost(quan) is greater than C0, quan0 is updated to the current threshold quan, and C0 is updated to the value of the current cost function Cost(quan).
[0073] In this embodiment, the low-frequency coefficient part of the compressed wavelet coefficient matrix is specifically:
[0074] After completing the sparseness of the low-frequency part, set the threshold If the ratio of non-zero elements in the low-frequency matrix is r≤r nz , the compressed row storage format is used and marked with 1; the compressed row storage format stores non-zero element values, column indices, and row pointers to efficiently store sparse data and reduce space usage; if r>r nz , all elements are stored and the storage format description value is 0.
[0075] S3, using the BiLLM binary residual approximation method to compress the high-frequency coefficient part of the wavelet coefficient matrix; Figure 3 As shown, specifically:
[0076] The BiLLM binary residual approximation method is applied to compress the difference between the original network weight matrix and the global matrix corresponding to the sparse low-frequency matrix; the binarization formula is:
[0077] W b =α·sign(W f )
[0078]
[0079] Among them, W b and W f They represent the binarized network weight matrix and the original full-precision weight matrix respectively. α is the scaling factor used to adjust the scale of the binarization result.
[0080] Finally, a set of network weight matrices represented by binary parameters is obtained.
[0081] In this embodiment, the residual matrix W represented by the difference between the global matrix corresponding to the sparse low-frequency matrix and the original network weight matrix is difference , the residual matrix W difference According to blocksize=128, each block is a 128×128 matrix block. The initial binarization is performed through the quantize method in the braq_quantizer class to obtain the first part of the binarization result B1, and then the residual between the result and the original significant weight is calculated, and the residual is binarized again to generate the second part of the binarization result B2. Finally, by adding B1 and B2, a more accurate significant weight binarization result B is obtained. salient :
[0082] B salient =B1+B2=Binary(W saliebt )+Binary(W salient -B1)
[0083] S4. After each network weight matrix of LLM is processed, the quantization of the large language model LLM is completed.
[0084] In this embodiment, after the quantization of the OPT-1.3B model is completed, the compressed weight matrix is first decompressed, which specifically includes three parts: a wavelet sparse matrix, the first layer of the BiLLM compressed matrix, and the BiLLM residual matrix. After decompression, these three matrices are merged into a complete weight matrix, most of the weight information of the model is restored, and the merged weight matrix is used to perform the conventional reasoning process. Through the method of the present invention, the storage and computing overhead can be significantly reduced while maintaining high performance and accuracy during the model reasoning process. The high-frequency and low-frequency weights use different word lengths to represent data. In the current example, the high frequency is 1-2 bits and the low frequency is 4 bits, which further reduces the size of the quantized model data.
[0085] It should also be noted that in this specification, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0086] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A large language model quantization method based on wavelet transform and BiLLM, characterized in that: include: Enter LLM; For each network weight matrix of LLM, Haar wavelet transform is applied to decompose the network weight matrix and obtain the wavelet coefficient matrix; The low-frequency coefficient part in the wavelet coefficient matrix is divided into blocks, the optimal threshold is selected, the weights less than the optimal threshold are reset to zero, and the part not less than the optimal threshold is quantized to compress the low-frequency coefficient part of the wavelet coefficient matrix; The BiLLM binary residual approximation method is used to compress the high-frequency coefficient part of the wavelet coefficient matrix; After each network weight matrix of LLM is processed, the quantization of LLM is completed.
2. The large language model quantization method based on wavelet transform and BiLLM according to claim 1, characterized in that: The low-frequency coefficient part in the wavelet coefficient matrix is divided into blocks, the optimal threshold is selected, the weights less than the optimal threshold are reset to zero, and the part not less than the optimal threshold is quantized, specifically: After dividing the low-frequency coefficient part of the wavelet coefficient matrix into blocks, the sub-matrix set obtained by dividing the blocks is {A1 {k，j} }; Traverse the sub-matrix set {k，j} For each matrix block N0 in}, calculate the matrix block N0 threshold set quan_candidates; Traverse each threshold quan of the threshold set quan_candidates, set the elements in the matrix block whose absolute value is less than the threshold to zero, and quantize the elements not less than the threshold to the threshold multiplied by the sign of the element, and obtain the sparsely quantized matrix block N1. The calculation process of the element N1(i, j) in the matrix block N1 is: Calculate the ratio r of non-zero elements of the sparse matrix block N1: Among them, count_nonzero(N1) is the number of non-zero elements in the sparse matrix block N1, and (N1).size is the number of elements in the sparse matrix block N1, that is, the matrix block size.
3. The large language model quantization method based on wavelet transform and BiLLM according to claim 2, characterized in that: The quantization threshold of the wavelet low-frequency coefficient adopts linear quantization in the floating-point exponent range. The calculation method of the threshold set quan_candidates is expressed as: quan_candidates={min(N0(i,j))+i·l|i∈{0, 1, 2, 4, 8, 16}} in, That is, the weight value is divided into 6 parts according to the floating point exponent range; min(|N0(i, j)|) and max(|N0(i, j)|) correspond to the minimum and maximum absolute values of the matrix elements in a given matrix block N0.
4. The large language model quantization method based on wavelet transform and BiLLM according to claim 2, characterized in that: Before traversing each threshold value quan in the threshold value set quan_candidates, it also includes initializing the optimal threshold value quan0=0, C0=0; When traversing each threshold quan in the threshold set quan_candidates, the following steps are also included: Calculate the cost function: Where H represents the Hessian matrix obtained by taking the second-order partial derivative of the weights under the second norm of the difference between the LLM estimated value and the observed value, ||H(N0)-H(N1)|| F represents the Frobenius-norm of the difference vector of the Hessian matrix H corresponding to the matrix block N0 and the sparse matrix block N1; If Cost(quan) is greater than C0, quan0 is updated to the current threshold quan, and C0 is updated to the value of the current cost function Cost(quan).