A language model pruning method based on pre-pruning and joint pruning

Abstract

This invention relates to the field of language model pruning, and more particularly to a language model pruning method based on pre-pruning and joint pruning. The method includes calculating the total pruning rate by using the device's reserved storage space and the actual storage space occupied by the language model; obtaining the structural pruning rate and the unstructured pruning rate of each layer of the language model using a pre-pruning method; performing structural and unstructured pruning on the language model; and restoring model accuracy through retraining. This invention, through pre-pruning as a pruning rate allocation method, can obtain a reasonable structural pruning rate and the unstructured pruning rate of each layer of the language model, and the joint structural and unstructured pruning can thoroughly prune the language model.

Description

Technical Field

[0001] This invention relates to the field of language model pruning, and more particularly to a language model pruning method based on pre-pruning and joint pruning. Background Technology

[0002] Language models are a core area of ​​natural language processing. In pursuit of superior performance, the number of parameters in language models has exploded. Google's BERT model with 110 million parameters (proposed in 2018), OpenAI's GPT-2 model with 1.5 billion parameters (released in 2019), and Microsoft and NVIDIA's MT-NLG language model with 530 billion parameters (released in 2021) are all examples of large-scale language models that are difficult to deploy on mobile and embedded devices with limited storage resources. To address this issue, pruning algorithms have been proposed.

[0003] Pruning algorithms are divided into structured pruning and unstructured pruning. Structured pruning reduces the number of parameters by decreasing the number of neurons. Unstructured pruning reduces the number of parameters by decreasing the weights in the weight matrix. In 2017, Narang et al. performed unstructured pruning on language models based on the magnitude of the absolute values ​​of the weights. This method uses the magnitude of the absolute values ​​of the weights as the pruning criterion, but the magnitude of the absolute values ​​of the weights cannot accurately determine the importance of the weights to the language model, and this method arbitrarily sets the pruning rate of each layer in the language model, without a reasonable pruning rate allocation method. In 2018, Wen W et al. proposed using Lasso group regularization to approximate the elements in the ISS group of the language model to zero, achieving structured pruning. This method transforms a large dense matrix into a small dense matrix, resulting in incomplete pruning of the language model. Chinese patent application CN114358257A discloses a method to obtain the pruning rate of each layer in the target model through pre-training, and then prunes the target model based on the pruning rate of each layer. This method is designed for convolutional network models, not language models, and its application to language models also suffers from incomplete pruning. Summary of the Invention

[0004] To address the above problems, this invention proposes a language model pruning method based on pre-pruning and joint pruning, specifically including the following steps:

[0005] The total pruning rate is calculated by combining the device's reserved storage space and the actual storage space occupied by the language model.

[0006] The pre-pruning method is used to obtain the structural pruning rate and the unstructured pruning rate of each layer of the language model;

[0007] The language model is subjected to structural and unstructured pruning, and the model accuracy is restored through retraining.

[0008] Furthermore, a pre-pruning method is used to obtain the structural pruning rate and the unstructured pruning rate of the language model, i.e., to obtain the structural pruning sensitivity and the unstructured pruning sensitivity of the language model. The structural pruning rate and the unstructured pruning rate of the language model are then calculated using the obtained sensitivity, including:

[0009]

[0010] Wherein, SPR is the structural pruning rate of the language model, UPR is the unstructured pruning rate of the language model, SST is the structural pruning sensitivity of the language model, UST is the unstructured pruning sensitivity of the language model, and TPR is the total pruning rate of the language model.

[0011] Furthermore, the pruning rate of each layer of the language model is calculated through the unstructured pruning rate of the language model, including:

[0012]

[0013] Among them, PR i PROP represents the unstructured pruning rate of the i-th layer of the language model. i TW represents the proportion of unstructured pruning in the i-th layer of the language model. i denoted as the number of weight parameters in the i-th layer of the language model.

[0014] Furthermore, the proportion of unstructured pruning in the i-th layer of the language model (PROP) i Represented as:

[0015]

[0016] Among them, PROP i ST represents the proportion of unstructured pruning in the i-th layer of the language model. i denoted as the unstructured pruning sensitivity of the i-th layer in the language model.

[0017] Furthermore, the process of obtaining the sensitivity of structural pruning in the language model includes:

[0018] Set an initial pruning rate to perform structural pruning on the entire language model. After pruning is complete, test the language model and obtain the perplexity under the corresponding pruning.

[0019] Increase the pruning rate and continue to perform structural pruning on the language model to obtain the perplexity under the corresponding pruning until the pruning rate reaches the threshold.

[0020] The structural pruning sensitivity of the language model is obtained by averaging all the obtained perplexity values.

[0021] Furthermore, the process of obtaining the unstructured pruning sensitivity of each layer of the language model includes:

[0022] Set the initial layer pruning rate to perform unstructured pruning on the current layer. After pruning is complete, test the language model and obtain the perplexity under the corresponding pruning.

[0023] Increase the pruning rate, continue pruning the language model, and obtain the perplexity under the corresponding pruning until the pruning rate reaches the threshold.

[0024] The unstructured pruning sensitivity of the current layer is obtained by averaging all the obtained perplexity values.

[0025] Furthermore, performing structural and unstructured pruning on the language model and restoring model accuracy through retraining includes the following steps:

[0026] Set the total number of pruning attempts;

[0027] Structural pruning is performed on the language model. The single-time structural pruning rate of the language model is the language model structural pruning rate divided by the total number of pruning operations.

[0028] After each structural pruning, the language model is retrained until it recovers to the set perplexity.

[0029] After structural pruning is completed, unstructured pruning is performed. During the unstructured pruning process, the unstructured pruning rate of each layer of the language model is obtained based on the proportion of unstructured pruning in each layer of the computational language model.

[0030] The unstructured pruning rate of each layer in a single language model is the unstructured pruning rate of each layer in the language model divided by the total number of pruning operations.

[0031] After each pruning, the language model is retrained until it returns to the set perplexity.

[0032] This invention can obtain a reasonable structural pruning rate and an unstructured pruning rate for each layer of the language model by using a pre-pruning pruning method. Furthermore, by combining structural and unstructured pruning, the language model can be thoroughly pruned. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of a stacked LSTM language model in an embodiment of the present invention;

[0034] Figure 2 This is a flowchart of the pre-pruning method in this invention;

[0035] Figure 3 This is a flowchart of the solution for the sensitivity of structural pruning of the language model in this invention;

[0036] Figure 4 This is a flowchart of the solution process for the unstructured pruning sensitivity of the language model in this invention;

[0037] Figure 5 This is a flowchart of the language model joint pruning and retraining process in this invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] This invention proposes a language model pruning method based on pre-pruning and combined pruning, specifically including the following steps:

[0040] The total pruning rate is calculated by combining the device's reserved storage space and the actual storage space occupied by the language model.

[0041] The pre-pruning method is used to obtain the structural pruning rate and the unstructured pruning rate of each layer of the language model;

[0042] The language model is subjected to structural and unstructured pruning, and the model accuracy is restored through retraining.

[0043] The total pruning rate is calculated by considering the device's reserved storage space and the actual storage space occupied by the language model. This embodiment provides a training method using the `fit` method in TensorFlow. Figure 1 The language model shown can have its weights saved as a weight file, the size of which is the actual storage space occupied by the language model. The total pruning rate of the language model can be obtained by dividing the device's reserved storage space by the actual storage space occupied by the language model.

[0044] A pre-pruning method is used to obtain the structural pruning rate and the unstructured pruning rate of each layer of the language model. The specific process of the pre-pruning method is as follows: Figure 2 As shown, the process consists of four steps:

[0045] The first step is to obtain the structural pruning sensitivity of the language model. Sensitivity is used to allocate the pruning rate; higher sensitivity means a greater impact on the model after pruning, and therefore a lower allocated pruning rate. The process for solving the structural pruning sensitivity of the language model is as follows: Figure 3 As shown, the specific steps include:

[0046] 101. Set the initial pruning rate. In this embodiment, the initial pruning rate is set to 10%.

[0047] 102. Prune the language model according to the set pruning rate;

[0048] 103. After pruning, test the language model and obtain the perplexity under the corresponding pruning.

[0049] 104. The perplexity calculated after each pruning is accumulated, and it is determined when the current pruning rate reaches the threshold. In this embodiment, the pruning rate threshold is set to 90%.

[0050] 105. If the threshold is not reached, the pruning rate is increased. In this embodiment, it is increased by 10% each time. After the increase, the process jumps to step 102.

[0051] 106. If the threshold is reached, the accumulated perplexity is averaged based on the number of prunings, and this average value is used as the sensitivity.

[0052] The second step is to obtain the unstructured pruning sensitivity of the language model. Structured pruning can only be applied to the entire model because it reduces neurons, thus altering the correspondence between the model's input and output. Unstructured pruning, on the other hand, simply transforms a dense matrix into a sparse matrix, without affecting the correspondence between the model's input and output, so it can be applied to all layers in the model. The solution for the unstructured pruning sensitivity of the language model flows as follows: Figure 4 As shown. The process for obtaining the unstructured pruning sensitivity of each layer of the language model is as follows: Figure 3 Similar, will Figure 3 The structural pruning in the language model is replaced with unstructured pruning, and this process is applied to all layers of the language model to obtain the unstructured pruning sensitivity of each layer. Specifically, the steps include:

[0053] 201. Obtain the unstructured pruning sensitivity of each layer of the language model;

[0054] 202. Obtain the proportion of unstructured pruning in each layer of the language model, expressed as:

[0055]

[0056] PROP i ST represents the proportion of unstructured pruning in the i-th layer of the language model. i This represents the unstructured pruning sensitivity of the i-th layer in the language model.

[0057] 203. Set the initial overall unstructured pruning rate of the language model; in this embodiment, it is set to 10%.

[0058] 204. Obtaining the Unstructured Pruning Coefficients of the Language Model: The sensitivity of each layer in the language model to unstructured pruning varies. Therefore, the overall pruning rate needs to be allocated to each layer according to the pruning ratio. However, simply having the pruning ratio is insufficient because the number of weights in each layer is different. If the overall pruning rate is simply multiplied by the pruning ratio, the final pruning rate will differ from the initially given overall pruning rate. Therefore, pruning coefficients are needed to reconcile this. The pruning coefficient multiplied by the pruning ratio of each layer equals the pruning rate of that layer. The formula for calculating the unstructured pruning coefficients of the language model is shown below:

[0059]

[0060] Where OPR represents the overall unstructured pruning rate of the language model, TW i PROP represents the number of weight parameters in the i-th layer of the language model, and PC represents the unstructured pruning coefficients of the language model. i This represents the proportion of unstructured pruning in the i-th layer of the language model obtained by Formula 1.

[0061] 205. Perform unstructured pruning on each layer by multiplying the pruning ratio by the pruning coefficient to obtain the pruning rate;

[0062] 206. Test the language model and obtain the perplexity under the corresponding pruning rate, and accumulate the perplexity calculated each time;

[0063] 207. Determine when the current pruning rate reaches the threshold. In this embodiment, the pruning rate threshold is set to 90%.

[0064] 208. If the threshold is not reached, the pruning rate is increased. In this embodiment, it is increased by 10% each time. After the increase, the process jumps to step 204.

[0065] 209. If the threshold is reached, the accumulated perplexity is averaged based on the number of prunings, and this average value is used as the sensitivity.

[0066] The third step is to obtain the structural pruning rate and the unstructured pruning rate of the language model. For the language model as a whole, the weights for structural and unstructured pruning are the same, so sensitivity alone can be used to determine the pruning rate. The calculation formula is shown below:

[0067]

[0068] Where SPR represents the structural pruning rate of the language model, UPR represents the unstructured pruning rate of the language model, SST represents the structural pruning sensitivity of the language model, UST represents the unstructured pruning sensitivity of the language model, and TPR represents the total pruning rate of the language model.

[0069] The fourth step is to obtain the unstructured pruning rate of each layer of the language model.

[0070] Using the unstructured pruning rate, unstructured pruning ratio, and number of weights in layers 1 through n of the language model obtained in step three, the unstructured pruning rate of each layer of the language model is calculated, as shown in the following formula:

[0071]

[0072] Among them PR i PROP represents the unstructured pruning rate of the i-th layer of the language model. i TW represents the proportion of unstructured pruning in the i-th layer of the language model obtained by Formula 1. i This represents the number of weight parameters in the i-th layer of the language model. UPR represents the unstructured pruning rate of the language model.

[0073] The language model undergoes structural and unstructured pruning, and its accuracy is restored through retraining. The process of performing structural and unstructured pruning and restoring model perplexity through retraining is as follows: Figure 5 As shown, the specific steps include:

[0074] 401. Set the total number of pruning attempts;

[0075] 402. Perform structural pruning on the language model. The single structural pruning rate of the language model is the language model structural pruning rate divided by the total number of pruning operations.

[0076] 403. After each structural pruning, retrain the language model until the language model recovers to the set perplexity.

[0077] 404. After structural pruning is completed, unstructured pruning is performed. During the unstructured pruning process, the unstructured pruning rate of each layer of the language model is obtained based on the proportion of unstructured pruning in each layer of the language model.

[0078] 405. The unstructured pruning rate of each layer of the language model in a single iteration is the unstructured pruning rate of each layer of the language model divided by the total number of pruning iterations.

[0079] 406. After each pruning, retrain the language model until it returns to the set perplexity.

[0080] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A language model pruning method based on pre-pruning and joint pruning, characterized in that, Specifically comprising the following steps: The total pruning rate is calculated by the device reserved storage space and the actual storage space of the language model, that is, the sensitivity of the structural pruning of the language model and the sensitivity of the non-structural pruning of the language model are obtained, the structural pruning rate of the language model and the non-structural pruning rate of the language model are calculated by the obtained sensitivity, comprising: Wherein, SPR is the structural pruning rate of the language model, UPR is the non-structural pruning rate of the language model, SST is the structural pruning sensitivity of the language model, UST is the non-structural pruning sensitivity of the language model, and TPR is the total pruning rate of the language model; The pruning rate of each layer of the language model is calculated by the non-structural pruning rate of the language model as: where PR i is the unstructured pruning rate of the i-th layer of the language model, TW i is the number of weight parameters in the i-th layer of the language model, PROP i is the unstructured pruning proportion in the i-th layer of the language model, and is represented as: wherein PROP i is the proportion of unstructured pruning of the i-th layer in the language model, ST i is the sensitivity of unstructured pruning of the i-th layer in the language model; The process of obtaining the structural pruning sensitivity of the language model comprises: Set the initial pruning rate to perform structural pruning on the entire language model, test the language model after pruning and obtain the perplexity under the corresponding pruning; Increase the pruning rate, continue to perform structural pruning on the language model, obtain the perplexity under the corresponding pruning, and stop until the pruning rate reaches the threshold; Average all obtained perplexities to obtain the structural pruning sensitivity of the language model; The process of obtaining the non-structural pruning sensitivity of each layer of the language model comprises: Set the initial layer pruning rate to perform non-structural pruning on the current layer, test the language model after pruning and obtain the perplexity under the corresponding pruning; Increase the pruning rate, continue to prune the language model, obtain the perplexity under the corresponding pruning, and stop until the pruning rate reaches the threshold; Average all obtained perplexities to obtain the non-structural pruning sensitivity of the current layer; The structural pruning rate of the language model and the non-structural pruning rate of each layer of the language model are obtained by the pre-pruning method; The language model is pruned structurally and non-structurally, and the model accuracy is restored by retraining, comprising the following steps: Set the total number of pruning times; Perform structural pruning on the language model, and the single language model structural pruning rate is the structural pruning rate of the language model divided by the total pruning times; After each structural pruning, the language model is retrained until the language model is restored to the set perplexity; After completing the structural pruning, non-structural pruning is performed, and during the non-structural pruning process, the non-structural pruning rate of each layer of the language model is obtained according to the calculation of the non-structural pruning proportion of each layer of the language model; The single language model non-structural pruning rate of each layer is the non-structural pruning rate of each layer of the language model divided by the total pruning times; After each pruning, the language model is retrained until the language model is restored to the set perplexity.

Images