Large model compression method
By preprocessing and dynamic quantization of the weight parameters of the large model, combined with STE algorithm and pruning technology, the problem of poor compression effect of the large model in the existing technology is solved, and the calculation amount and storage are achieved while ensuring performance The reduction in space improves the speed and accuracy of reasoning.
Patent Information
- Application Number
- CN202411045364.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-12-03
Smart Images

Figure CN119067181A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a large model compression method. Background Art
[0002] With the continuous development of deep learning technology, large models (usually refers to various common deep learning frameworks and models with more than one billion parameters) have been widely used in various fields. The computational complexity and storage space requirements of large models have also brought challenges to practical applications. Researchers have proposed various model compression technologies. In existing technologies, large model compression methods mainly include pruning, quantization, knowledge distillation, etc. These methods can reduce the size and complexity of the model to a certain extent, but directly using them often has a greater impact on model performance. Therefore, how to achieve reliable compression of large models while ensuring the performance of large models is an urgent problem to be solved. Summary of the invention
[0003] The present invention proposes a large model compression method to solve the problem of poor model compression effect in the prior art.
[0004] A large model compression method comprises the following steps:
[0005] S1. Preprocess the weight parameters of the large model to obtain the preprocessed weight parameters;
[0006] S2, dynamically quantize the activation value and preprocessed weight parameter of the large model to obtain the quantized activation value and weight parameter;
[0007] S3, preset loss function, training samples and sample labels, perform the mth training on the large model, forward calculate the training sample prediction value according to the training sample, quantized activation value and weight parameter, input the training sample prediction value and sample label into the loss function to calculate the total loss of the training sample after the mth training, and use the STE algorithm to reversely adjust the weight parameters of the large model according to the total loss of the training sample after the mth training;
[0008] S4, preset the loss threshold and the total number of training times. If the total loss of the training samples after the mth training does not reach the loss threshold and the current number of training times does not reach the total number of training times, repeat step S3 and perform the m+1th training on the large model. Otherwise, execute step S5.
[0009] S5. If the total loss of the training samples after the mth training reaches the loss threshold, or the current number of training times reaches the preset total number of training times, the training is terminated to obtain a compressed large model.
[0010] Preferably, S1 specifically includes the following:
[0011] S11, scaling each weight parameter of the large model using a minimum-maximum normalization calculation method to obtain a normalized weight parameter;
[0012] S12, presetting a parameter threshold, and pruning the normalized weight parameters according to the parameter threshold to obtain preprocessed weight parameters.
[0013] Preferably, S11 can be specifically expressed as:
[0014] w′ i =(w i -min_w i ) / (max_w i -min_w i )
[0015] In the formula, w i is the i-th weight parameter of the large model, min_w i and max_w i are the i-th weight parameter w of the large model respectively. i The minimum and maximum values of w′ i is the i-th weight parameter after normalization.
[0016] Preferably, S12 can be specifically expressed by the formula:
[0017] w i _pruned = w′ i *threshold
[0018] Where w′ i is the i-th weight parameter after normalization, threshold is a parameter threshold less than 1, which is used to control the degree of pruning, w i _pruned is the i-th weight parameter after preprocessing.
[0019] Preferably, in S2, the activation value of the large model and the preprocessed weight parameters are dynamically quantized, and the specific formula is as follows:
[0020]
[0021] In the formula, w i _pruned is the i-th weight parameter after preprocessing, a i is the i-th activation value, is the quantized i-th weight parameter, The i-th activation value after quantization, Quantize(.) is the quantization function.
[0022] Preferably, in S3, the training sample prediction value is forward calculated according to the training sample, the quantized activation value and the weight parameter, which can be specifically expressed by the formula:
[0023] f(x j )=acv(S(x j ))
[0024] in,
[0025] In the formula, f(x j ) is the jth sample x in the training sample j The predicted value, acv is the activation function, S(x j ) is the jth sample x in the training sample j The corresponding weight calculation value, w j is the jth sample x j The corresponding set of weight parameters, a j is the jth sample x j The corresponding set of activation values, is the jth sample x j The corresponding quantized i-th weight parameter, is the jth sample x j The corresponding quantized i-th activation value, n is the total number of weight parameters or activation values, and Bitshift(.) is the shift function.
[0026] Preferably, the loss function preset in S3 is specifically:
[0027]
[0028] In the formula, Loss is the total loss of training samples, k is the total number of training samples, and y j is the jth training sample x j The sample label of f(x j ) is the jth training sample x j The predicted value of .
[0029] Preferably, the STE algorithm is used in S3 to reversely adjust the weight parameters of the large model, which can be specifically expressed by the formula:
[0030]
[0031] In the formula, Loss is the total loss of the training sample, w is the set of weight parameters of the large model, It is a set of quantized weight parameters of the large model.
[0032] The above-mentioned large model compression method preprocesses the weight parameters of the large model, performs dynamic quantization operations on the activation values and the preprocessed weight parameters to obtain the quantized activation values and weight parameters, presets the loss function, training samples and sample labels, forward calculates the predicted values of the training samples according to the training samples, the quantized activation values and weight parameters, inputs the predicted values of the training samples and the sample labels into the loss function to calculate the total loss of the training samples after each training, and uses the STE algorithm to reversely adjust the weight parameters of the large model according to the total loss of the training samples after each training; presets the loss threshold and the total number of training times, and if the total loss of the training samples after the current training reaches the loss threshold, or the number of training times after the current training reaches the total number of training times, the training is terminated to obtain the compressed large model. This method compresses the large model by combining pruning with dynamic quantization technology, reduces the computational complexity and storage space of the large model, and improves the reasoning speed and accuracy of the large model to a certain extent, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 It is a flow chart of a large model compression method in one embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.
[0035] See also Figure 1 The present invention provides a large model compression method, comprising the following steps:
[0036] S1. Preprocess the weight parameters of the large model to obtain the preprocessed weight parameters.
[0037] Specifically, before quantizing the weight parameters of the large model, the weight parameters of the large model need to be preprocessed, and the preprocessing includes normalizing and pruning the weight parameters.
[0038] Furthermore, S1 specifically includes the following:
[0039] S11. Each weight parameter of the large model is scaled using the minimum-maximum normalization calculation method to obtain the normalized weight parameter. The specific calculation formula is as follows:
[0040] w′ i =(w i -min_w i ) / (max_w i -min_w i ) (1)
[0041] In the formula, w iis the i-th weight parameter of the large model, min_w i and max_w i are the i-th weight parameter w of the large model respectively. i The minimum and maximum values of w′ i is the i-th weight parameter after normalization.
[0042] The above formula (1) can be used to scale the value of the weight parameter of the large model to between 0 and 1.
[0043] S12, presetting a parameter threshold, and pruning the normalized weight parameters according to the parameter threshold to obtain preprocessed weight parameters.
[0044] Specifically, after completing the normalization calculation of the weight parameters of the large model, it is also necessary to prune the normalized weight parameters of the large model. Pruning is a method of removing unimportant weight parameters in the model. When pruning, a weight parameter threshold needs to be set, and the weight parameter threshold is used to determine which weight parameters in the normalized weight parameters need to be pruned. The calculation is performed according to the following formula:
[0045] w i _pruned = w′ i *threshold (2)
[0046] Where w′ i is the i-th weight parameter after normalization, threshold is a parameter threshold less than 1, which is used to control the degree of pruning, w i _pruned is the i-th weight parameter after preprocessing, that is, the i-th weight parameter after pruning.
[0047] Through the above formula (2), the normalized i-th weight parameter w′ can be i The value of is limited to within the preset parameter threshold, and the weight parameters below the preset parameter threshold will be set to 0. For example, the value of threshold is set to 0.2. Through formula (2), the weight parameters with values lower than 0.2 in the normalized weight parameters will be set to 0. Through the above operation, the structure of the large model network can be guaranteed to remain unchanged, and it is beneficial to the regularization of the large model.
[0048] In the actual pruning process, a weight parameter distribution histogram of multiple weight parameters of a large model can be drawn, and the parameter threshold can be set to between 90% and 95% of the weight parameter distribution histogram proportion, because too small a parameter threshold will result in too many weight parameters being lost, which will significantly reduce the performance of the large model.
[0049] S2. Dynamically quantize the activation value and preprocessed weight parameters of the large model to obtain quantized activation value and weight parameters.
[0050] Specifically, after preprocessing the weight parameters of the large model, the number of weight parameters of the large model will be reduced, and then the weight parameters will be quantized on this basis. If only the weight parameters of the large model are quantized, the storage size of the large model can only be reduced. When the large model is running, it needs to be dequantized and then calculated, which has limited performance improvement for the large model when running. Therefore, considering quantizing the activation values of the large model as well, the performance of the quantized large model can be further improved.
[0051] Furthermore, S2 performs dynamic quantization operations on the activation values and preprocessed weight parameters of the large model. The specific formula is as follows:
[0052]
[0053] In the formula, w i _pruned is the i-th weight parameter after preprocessing, that is, the i-th weight parameter before quantization, a i is the i-th activation value. The weight parameters and activation values before quantization are all floating point numbers. is the quantized i-th weight parameter, The i-th activation value after quantization, the quantized weight parameter and activation value are all integers. Quantize(.) is the quantization function, which uses rounding down to obtain the quantized integer value.
[0054] The multiplication of activation values or weight parameters in the above formulas (3)-1 and (3)-2 can be converted into shift operations to improve the calculation speed, that is:
[0055]
[0056] In the formula, w is the set of weight parameters of the large model, is the quantized weight parameter of the i-th element, a is the set of activation values, is the i-th quantized activation value, n is the weight parameter or the total number of activation values, and Bitshift(.) is the shift function.
[0057] S3, preset loss function, training samples and sample labels, perform the m-th training on the large model, forward calculate the training sample prediction value according to the training samples, quantized activation values and weight parameters, input the training sample prediction value and sample label into the loss function to calculate the total loss of the training sample after the m-th training, and use the STE algorithm to reversely adjust the weight parameters of the large model according to the total loss of the training samples after the m-th training.
[0058] Specifically, the above formulas (3)-1 and (3)-2 give the calculation formulas for the quantization of weight parameters and activation values of the large model. If the weight parameters of a large model are directly converted into quantized values according to the above formulas, it will inevitably lead to a significant decline in the performance of the large model. This is because the quantization of the weight parameter value leads to a loss of precision, and the loss of precision will inevitably lead to result errors caused by calculation errors during inference of the large model. Therefore, after the parameters are quantized, the quantized parameters need to be dynamically adjusted again.
[0059] Furthermore, in S3, the training sample prediction value is forward calculated based on the training sample, the quantized activation value and the weight parameter, which can be specifically expressed by the formula:
[0060] f(x j )=acv(S(x j )) (5)
[0061] in,
[0062]
[0063] In the formula, f(x j ) is the jth sample x in the training sample j The predicted value, acv is the activation function, S(x j ) is the jth sample x in the training sample j The corresponding weight calculation value, w j is the jth sample x j The corresponding set of weight parameters, a j is the jth sample x j The corresponding set of activation values, is the jth sample x j The corresponding quantized i-th weight parameter, is the jth sample x j The corresponding quantized i-th activation value, Bitshift(.) is the shift function.
[0064] It should be noted that the method for dynamically quantizing each weight parameter or activation value given in formula (3)-1 and formula (3)-2 is applicable to the dynamic quantization operation of weight parameters and activation values in each training batch. During the first training, the weight parameters need to be randomly initialized. In each training batch, when a sample is forward calculated, the part of weight parameters used in calculating each activation value is not all weight parameters, but the formula used in the actual calculation and the part of weight parameters used by it.
[0065] As a neural network model with a very large number of parameters, the large model consists of several layers. The activation value of the current layer is the input value of the next layer connected to it. The weight calculation value of the last layer is processed by the activation function acv to obtain the final prediction value. Each sample will go through layers of backward calculations to finally obtain the prediction value of the sample, and then backpropagate and update the weight value according to the prediction value. All samples go through this process for each iterative training.
[0066] For each sample, the above formula (5) is used to calculate backward layer by layer to complete the forward propagation of the large model network, and the forward propagation continues until the predicted value of each sample is calculated.
[0067] Furthermore, the loss function preset in S3 is specifically:
[0068]
[0069] In the formula, Loss is the total loss of training samples, k is the total number of training samples, and y j is the jth training sample x j The sample label of f(x j ) is the jth training sample x j The predicted value of .
[0070] When designing the loss function, the root mean square error function is used as the loss function and the loss is calculated. In each training batch, the prediction value corresponding to each sample is calculated, and the loss value of each sample after each iterative training is calculated based on the prediction value of each sample and the label of the corresponding sample. Then, the total loss is calculated based on the loss value of all samples in each training batch. After each batch of training, all samples will reversely update the weight parameters.
[0071] In each iterative training process, after the large model completes forward propagation and loss calculation, it is necessary to dynamically adjust the weight parameters of the large model in reverse. The gradient descent algorithm is usually used to achieve this. However, after the parameters are converted by the quantization function, they change from continuous values to discrete values, which means that the gradient of the parameters will become 0, resulting in the inability to perform the reverse calculation process. At this time, consider using the STE (Straight Through Estimator) algorithm to implement the back propagation calculation of the gradient and solve the problem of non-differentiable functions. The STE straight-through estimator sets the input gradient to a threshold function equal to its output gradient, regardless of the actual derivative of the threshold function itself.
[0072] Furthermore, S3 uses the STE algorithm to reversely adjust the weight parameters of the large model, which can be specifically expressed by the formula:
[0073]
[0074] In the formula, Loss is the total loss of the training sample, w is the set of weight parameters of the large model, which is a floating point number. It is a set of quantized weight parameters of the large model.
[0075] S4, preset the loss threshold and the total number of training times. If the total loss of the training samples after the mth training does not reach the loss threshold and the current number of training times does not reach the total number of training times, repeat step S3 and perform the m+1th training on the large model. Otherwise, execute step S5.
[0076] S5. If the total loss of the training samples after the mth training reaches the loss threshold, or the current number of training times reaches the preset total number of training times, the training is terminated to obtain a compressed large model.
[0077] Specifically, a loss threshold and a total number of training times M are preset, the total loss of the training samples after the m-th training is calculated, and the total loss of the training samples after the m-th training is compared with the preset loss threshold. If the total loss of the training samples after the m-th training does not reach the preset loss threshold, it is necessary to determine whether the current number of training times reaches the preset total number of training times. If the current number of training times does not reach the preset total number of training times, that is, m<M, the large model is trained for the m+1th time. Otherwise, when the total loss of the training samples after the m-th training reaches the preset loss threshold, or the current number of training times reaches the preset total number of training times, that is, m=M, the training is terminated to obtain the compressed large model.
[0078] The above-mentioned large model compression method preprocesses the weight parameters of the large model, performs dynamic quantization operations on the activation values and the preprocessed weight parameters to obtain the quantized activation values and weight parameters, presets the loss function, training samples and sample labels, forward calculates the predicted values of the training samples according to the training samples, the quantized activation values and weight parameters, inputs the predicted values of the training samples and the sample labels into the loss function to calculate the total loss of the training samples after each training, and uses the STE algorithm to reversely adjust the weight parameters of the large model according to the total loss of the training samples after each training; presets the loss threshold and the total number of training times, and if the total loss of the training samples after the current training reaches the loss threshold, or the number of training times after the current training reaches the total number of training times, the training is terminated to obtain the compressed large model. This method compresses the large model by combining pruning with dynamic quantization technology, reduces the computational complexity and storage space of the large model, and improves the reasoning speed and accuracy of the large model to a certain extent, and has broad application prospects.
[0079] The above is a detailed introduction to a large model compression method provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.
Claims
1. A large model compression method, characterized in that: The method comprises the following: S1. Preprocess the weight parameters of the large model to obtain the preprocessed weight parameters; S2, dynamically quantize the activation value and preprocessed weight parameter of the large model to obtain the quantized activation value and weight parameter; S3, preset loss function, training samples and sample labels, perform the mth training on the large model, forward calculate the training sample prediction value according to the training sample, quantized activation value and weight parameter, input the training sample prediction value and sample label into the loss function to calculate the total loss of the training sample after the mth training, and use the STE algorithm to reversely adjust the weight parameters of the large model according to the total loss of the training sample after the mth training; S4, preset the loss threshold and the total number of training times. If the total loss of the training samples after the mth training does not reach the loss threshold and the current number of training times does not reach the total number of training times, repeat step S3 and perform the m+1th training on the large model. Otherwise, execute step S5. S5. If the total loss of the training samples after the mth training reaches the loss threshold, or the current number of training times reaches the preset total number of training times, the training is terminated to obtain a compressed large model.
2. The large model compression method according to claim 1, characterized in that: S1 specifically includes the following: S11, scaling each weight parameter of the large model using a minimum-maximum normalization calculation method to obtain a normalized weight parameter; S12, presetting a parameter threshold, and pruning the normalized weight parameters according to the parameter threshold to obtain preprocessed weight parameters.
3. The large model compression method according to claim 2, characterized in that: S11 can be specifically expressed as: In i =(in i -min_in i ) / (max_in i -min_in i ) In the formula, w i is the i-th weight parameter of the large model, min_w i and max_w i are the i-th weight parameter w of the large model respectively. i The minimum and maximum values of w′ i is the i-th weight parameter after normalization.
4. The large model compression method according to claim 3, characterized in that: S12 can be specifically expressed by the formula: w i _pruned=w i *threshold Where w′ i is the i-th weight parameter after normalization, threshold is a parameter threshold less than 1, which is used to control the degree of pruning, w i _pruned is the i-th weight parameter after preprocessing.
5. The large model compression method according to claim 4, characterized in that: In S2, the activation values and preprocessed weight parameters of the large model are dynamically quantized. The specific formula is as follows: In the formula, w i _pruned is the i-th weight parameter after preprocessing, a i is the i-th activation value, is the quantized i-th weight parameter, The i-th activation value after quantization, Quantize(.) is the quantization function.
6. The large model compression method according to claim 5, characterized in that: In S3, the training sample prediction value is forward calculated based on the training sample, the quantized activation value and the weight parameter, which can be specifically expressed by the formula: f(x j )=acv(S(x j )) in, In the formula, f(x j ) is the jth sample x in the training sample j The predicted value, acv is the activation function, S(x j ) is the jth sample x in the training sample j The corresponding weight calculation value, w j is the jth sample x j The corresponding set of weight parameters, a j is the jth sample x j The corresponding set of activation values, is the jth sample x j The corresponding quantized i-th weight parameter, is the jth sample x j The corresponding quantized i-th activation value, n is the total number of weight parameters or activation values, and Bitshift(.) is the shift function.
7. The large model compression method according to claim 6, characterized in that: The loss function preset in S3 is: In the formula, Loss is the total loss of training samples, k is the total number of training samples, and y j is the jth training sample x j The sample label of f(x j ) is the jth training sample x j The predicted value of .
8. The large model compression method according to claim 7, characterized in that: S3 uses the STE algorithm to reversely adjust the weight parameters of the large model, which can be expressed as follows: In the formula, Loss is the total loss of the training sample, w is the set of weight parameters of the large model, It is a set of quantized weight parameters of the large model.
