LLMs combined compression method based on joint optimization, electronic equipment and storage medium

Abstract

The invention discloses an LLMs combined compression method based on joint optimization, an electronic device and a storage medium, a quantization method and a rarefaction method are combined, errors caused by quantization and rarefaction are minimized at the same time, the method comprises the following steps that 128 fragments are sampled from a WikiText2 training corpus, and each fragment comprises 2048 tags; 2, initializing a diagonal transformation matrix by using a statistical index, carrying out preliminary quantization on a weight by using an equivalent transformation operation, determining a quantization step length and a zero point, and calibrating the quantization step length and the zero point through the data sampled in the step 1; calculating a sparse mask, rearranging weights by using an importance index # imgabs0 #, and preferentially sparsening unimportant weights; carrying out joint optimization; and evaluating the compressed model by using different data sets, and verifying the performance improvement of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of compression and acceleration of neural networks, and in particular to an LLMs combination compression method, electronic equipment and storage medium based on joint optimization. Background Art

[0002] Large Language Models (LLMs) play an important role in the field of artificial intelligence, especially in the field of natural language processing (NLP). Large language models learn the structure, meaning, and usage of language by analyzing large amounts of text data, so that they can understand and generate natural language text. These models usually have billions or even hundreds of billions of parameters and are able to process and generate extremely complex text content.

[0003] In the field of deep learning, the Transformer structure has been widely used in various visual tasks, and its performance has been significantly improved as the model size increases. However, the complexity and scale of the Transformer model also lead to a significant increase in computing resource requirements. Therefore, researchers have proposed a variety of model compression techniques to meet this challenge, including quantization and sparsification methods: (1) Quantization is a technique that converts the weights and activation values ​​of a model from high-precision floating-point numbers to low-bit fixed-point numbers. This method can reduce the size of the model and improve memory access efficiency, thereby speeding up the computation process. On hardware that supports low-bit operations, quantized models can significantly improve inference speed. Post-training quantization (PTQ) is a commonly used method that does not require a lot of retraining and can be implemented with only a small amount of calibration data. However, a certain amount of quantization error will be introduced during the quantization process, especially when dealing with outliers, the error is more significant.

[0004] (2) Sparsification is a method of achieving model compression by converting dense weight matrices into sparse matrices. Sparsification can significantly reduce the amount of computation and improve the model inference speed. Similar to quantization, sparsification also faces the problem of how to maintain model performance while reducing computing resources.

[0005] Although both quantization and sparsification techniques have made significant progress in model compression, they have some major drawbacks when applied to LLMs models: ① Increased quantization error: When processing LLMs, the presence of outliers significantly increases the quantization error. The magnitude of outliers tends to increase in subsequent Transformer blocks, which can lead to a large amount of rounding errors. ② Instability of sparsification: If the importance assessment of weights is not effectively handled during the sparsification process, it may lead to the loss of key weights, thus affecting the performance of the model; ③Reconstruction granularity problem: Existing methods often ignore the importance of sparsification when dealing with the quantization of LLMs; considering different compression algorithms simultaneously can effectively reduce the corresponding compression loss, but if compressed sequentially, it will lead to a decrease in model performance.

[0006] In summary, existing quantization and sparsification methods have significant challenges when applied to LLMs models, mainly in terms of outlier processing, sparsification stability, and the combination of quantization and sparsification. These shortcomings need to be addressed through more sophisticated and optimized methods to achieve more efficient model compression and better model performance. Summary of the invention

[0007] The purpose of the present invention is to provide a LLMs combined compression method, electronic device and storage medium based on joint optimization, to solve the problems existing in the prior art, to combine quantization and sparsification methods, and to minimize the errors caused by quantization and sparsification at the same time.

[0008] In order to achieve the above object, the solution of the present invention is: A LLMs combined compression method based on joint optimization includes the following steps: Step 1: Data preparation 128 segments are sampled from the WikiText2 training corpus, each containing 2048 tags; Step 2: Initial weight quantization Initialize the diagonal transformation matrix using statistical indicators, preliminarily quantize the weights using equivalent transformation operations, determine the quantization step size and zero point, and calibrate them using the data sampled in step 1; Step 3: Apply sparse mask Compute sparse masks, using importance metrics Rearrange weights, prioritize sparse weights that are not important; importance index The calculation formula is ,in represents the F-norm, Indicates The activation value of the output channel, Indicates taking the absolute value, Indicates Line The weight value of the column; Step 4: Joint Optimization 4-1. Minimize the errors of quantization and sparsification at the same time, perform gradient updates on the transformation matrix during the optimization process, and ensure that the distribution of weights can be effectively optimized in each iteration; 4-2. In each iteration, the sparse mask is recalculated to ensure that the sparsity of the weights and the quantization error are minimized; Step 5: Model evaluation The compressed model is evaluated using different datasets to verify its performance improvement.

[0009] An electronic device comprises a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the LLMs combination compression method based on joint optimization.

[0010] A computer-readable storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the LLMs combined compression method based on joint optimization.

[0011] After adopting the above technical solution, the present invention has the following technical effects: Through steps 3 and 4, a joint optimization strategy is introduced. By using a learnable transformation matrix, the errors of quantization and sparsification are optimized simultaneously in each iteration, avoiding the error accumulation problem in traditional sequence optimization methods, thereby improving the overall performance of the model; through step 4, a dynamic reordering method is proposed, which rearranges the weights according to the sparsity measure in each iteration, and preferentially sparsifies unimportant weights to ensure the stability and compression effect of the sparsification process; through the joint optimization strategy and the dynamic reordering method, the losses of quantization and sparsification are jointly optimized, which significantly improves the performance of the model under low-bit quantization and high-sparse configuration. For example, in the LLaMA2-13b model, when 2-bit quantization and 75% sparse configuration are used, the average accuracy is 9.03% higher than the most advanced method in the prior art; by avoiding large-scale fine-tuning process, the joint optimization strategy of the present invention significantly reduces the consumption of computing resources, making model compression more efficient; the present invention shows significant performance improvement on different models and data sets, and has wide applicability and versatility. In summary, this patent effectively solves the problems of error accumulation, improper sparsification strategy and local optimality existing in the prior art by introducing joint optimization and dynamic reordering methods, thereby having significant beneficial effects on compression effect and model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The flowchart and performance comparison of the joint optimization of the present invention and the existing sequential optimization are shown. DETAILED DESCRIPTION

[0013] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0014] The main disadvantages of the prior art include the following aspects: (I) Error accumulation of sequence compression methods: In the prior art, quantization and sparsification are usually performed sequentially, which will lead to the accumulation of quantization error and sparsification error, thereby significantly reducing the model performance; for example, performing quantization optimization first and then weight sparsification will amplify the quantization error and increase the overall mean square error loss.

[0015] (II) Improper choice of sparsification strategy: When existing sparsification methods are used to process large-scale language models, traditional pruning algorithms may severely degrade model performance and usually require subsequent fine-tuning to restore performance, which increases the complexity of model compression and the consumption of computing resources.

[0016] (III) Local optimum of optimization strategy: Since quantization and sparsification are performed in stages in the existing technology, each stage can only reach the local optimum, making it difficult to minimize the error globally.

[0017] In view of the above shortcomings, the present invention proposes a LLMs combination compression method based on joint optimization, comprising the following steps: Step 1: Data preparation 128 segments are sampled from the WikiText2 training corpus, each containing 2048 tags (tokens); Step 2: Initial weight quantization Initialize the diagonal transformation matrix using statistical indicators, perform preliminary quantization on the weights using equivalent transformation operations (such as scaling, offset, rotation, and rearrangement), determine the quantization step size and zero point, and calibrate them using the data sampled in step 1; Step 3: Apply sparse mask Compute sparse masks, using importance metrics Rearrange weights, prioritize sparse weights that are not important; importance index The calculation formula is ,in represents the F-norm, Indicates The activation value of the output channel, Indicates taking the absolute value, Indicates Line The weight value of the column; Step 4: Joint Optimization 4-1. Minimize the errors of quantization and sparsification at the same time, perform gradient updates on the transformation matrix during the optimization process, and ensure that the distribution of weights can be effectively optimized in each iteration; 4-2. In each iteration, the sparse mask is recalculated to ensure that the sparsity of the weights and the quantization error are minimized; 5. Model Evaluation The compressed model is evaluated using different datasets to verify its performance improvement.

[0018] Through the above scheme, the present invention has the following technical effects: a. Through steps 3 and 4, a joint optimization strategy is introduced. By using a learnable transformation matrix, the errors of quantization and sparsification are optimized simultaneously in each iteration, avoiding the error accumulation problem in traditional sequential optimization methods, thereby improving the overall performance of the model; b. Through step 4, a dynamic reordering method is proposed to rearrange the weights according to the sparsity metric in each iteration, giving priority to sparsifying unimportant weights to ensure the stability and compression effect of the sparsification process; c. Through the joint optimization strategy and dynamic reordering method, the quantization and sparsification losses are jointly optimized, which significantly improves the performance of the model under low-bit quantization and high sparsity configuration. For example, in the LLaMA2-13b model, when 2-bit quantization and 75% sparsity configuration are used, the average accuracy is 9.03% higher than the most advanced method in the existing technology; d. By avoiding large-scale fine-tuning processes, the joint optimization strategy of the present invention significantly reduces the consumption of computing resources, making model compression more efficient; e. The present invention shows significant performance improvement on different models and data sets, and has wide applicability and versatility.

[0019] In summary, this patent effectively solves the problems of error accumulation, improper sparsification strategy and local optimality existing in the prior art by introducing joint optimization and dynamic reordering methods, thereby having significant beneficial effects on compression effect and model performance.

[0020] The present invention also discloses an electronic device, comprising a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the above method.

[0021] The present invention also discloses a computer-readable storage medium, which stores a computer program; when the computer program is executed in a computer, the computer executes the above method. The storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0022] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A LLMs combined compression method based on joint optimization, characterized in that The following steps are involved: Step 1: Data preparation 128 segments are sampled from the WikiText2 training corpus, each containing 2048 tags; Step 2: Initial weight quantization Initialize the diagonal transformation matrix using statistical indicators, preliminarily quantize the weights using equivalent transformation operations, determine the quantization step size and zero point, and calibrate them using the data sampled in step 1; Step 3: Apply sparse mask Compute sparse masks, using importance metrics Rearrange weights, prioritize sparse weights that are not important; Importance Index The calculation formula is ,in represents the F-norm, Indicates The activation value of the output channel, Indicates taking the absolute value, Indicates Line The weight value of the column; Step 4: Joint Optimization 4-1. Minimize the errors of quantization and sparsification at the same time, perform gradient updates on the transformation matrix during the optimization process, and ensure that the distribution of weights can be effectively optimized in each iteration; 4-2. In each iteration, the sparse mask is recalculated to ensure that the sparsity of the weights and the quantization error are minimized; Step 5: Model evaluation The compressed model is evaluated using different datasets to verify its performance improvement.

2. An electronic device, characterized in that It includes a processor, a memory and an application program; the application program is stored in the memory and is configured to be executed by the processor according to the joint optimization-based LLMs combination compression method as claimed in claim 1.

3. A computer-readable storage medium, characterized in that The storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the LLMs combination compression method based on joint optimization as described in claim 1.

Images