Long Short-Term Memory Compression Method Based on Knowledge Distillation Recovery Strategy Pruning

Abstract

The present invention discloses a long short-term memory compression method based on pruning with a knowledge distillation recovery strategy. In the implementation of knowledge distillation, in addition to introducing distillation loss for the student model to fit the logits output of the teacher model, the cross-entropy loss between the output probability distribution and the true label is also used to ensure that the output of the student model matches the true label of the sample. These two parts of the loss together constitute the objective function, which helps the student model learn from the "hidden knowledge" of the teacher model and optimize the output probability distribution of the model, thereby improving the accuracy of the pruned model. In addition, the present invention applies knowledge distillation to the pruning process of the LSTM model. By rationally transferring knowledge, the pruned model has stronger representation capabilities.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a long-short term memory compression method based on knowledge distillation recovery strategy pruning. Background Art

[0002] Deep neural network compression and acceleration are a series of technologies and methods for optimizing deep neural network models in resource-constrained environments. These technologies aim to reduce model size, speed up inference, and reduce computing resource consumption, enabling efficient model deployment in scenarios such as mobile devices, embedded systems, and edge devices. Weight pruning is a deep neural network compression method that reduces model size by removing redundant, less-contributing connections or parameters. Pruning can be performed before, during, or after training based on criteria such as weight sensitivity or gradient sensitivity.

[0003] Coarse-grained pruning and fine-grained pruning are two commonly used pruning methods for deep neural network compression. Coarse-grained pruning is a relatively coarse pruning method that typically prunes parameters at the layer or module level. Specifically, this method prunes a subset of parameters across an entire layer or module, rather than pruning each parameter individually. This pruning approach removes some connections within the layer or module, thereby reducing the network size. Because coarse-grained pruning selects a subset of parameters across an entire layer or module, it may result in some information loss and may also reduce model accuracy. However, its advantage lies in its relatively simple and fast operation, making it suitable for hardware acceleration and efficient inference. Fine-grained pruning is a more detailed pruning method that selectively prunes each parameter. Specifically, this method selectively prunes some parameters within the network based on their sensitivity or importance, while retaining others.

[0004] Fine-grained pruning is more refined and precise than coarse-grained pruning because it better preserves important connections and parameters, reduces information loss, and potentially maintains model accuracy. However, using coarse-grained pruning can easily lead to a decrease in model accuracy, especially when high compression rates are required. Even with fine-grained pruning, which has a good compression effect, model accuracy may still be reduced to an insufficient level when the proportion of pruned parameters is too large.

[0005] Knowledge distillation is another method for deep neural network compression. It usually transfers the output (logits) of the teacher network as "dark knowledge" hidden within the model to a smaller student network, allowing the student network to approach the performance of the teacher network. Compared with direct training with one-hot labels, the knowledge distillation strategy performs better in improving the accuracy of small networks. Through knowledge distillation, the student network can obtain more information than direct training with one-hot labels, thereby achieving better accuracy with a relatively small network structure. This makes knowledge distillation a powerful neural network compression method, especially suitable for resource-constrained environments such as mobile devices and embedded systems.

[0006] Typically, an over-parameterized original model has strong learning and representation capabilities. However, after pruning, the network becomes smaller or subject to certain constraints, and its own learning capabilities may be unable to obtain complex representations. Therefore, even fine-tuning can be difficult to fully restore the accuracy of the original model. In traditional iterative pruning methods, the pruned model is fine-tuned to restore the model's accuracy. However, for models with excessively fine-grained or coarse-grained pruning, fine-tuning can be difficult, making it difficult to fully restore their accuracy. Summary of the Invention

[0007] To address the limitations and defects of the existing technology, the present invention provides a long short-term memory compression method based on knowledge distillation recovery strategy pruning, including:

[0008] Step S1: training a long short-term memory model based on the obtained data set to obtain an original model with a preset generalization capability, and saving the original model;

[0009] Step S2: setting pruning parameters, including weight pruning method, initial value of sparsity, and expected value of sparsity;

[0010] Step S3: Evaluate the importance of connections or weight blocks according to the weight pruning method, determine the pruning ratio according to the sparsity after sorting, set the corresponding parameters to zero according to the pruning ratio, and prohibit updating the parameters that have been set to zero, so as to obtain a pruned model;

[0011] Step S4: Use the knowledge distillation method to train the pruned model, using the original model as the teacher and the pruned model as the student. By adding distillation loss to the loss function, the student model fits the logits output of the teacher model. After iterative training for a preset number of times, a model with restored accuracy is obtained.

[0012] Step S5: Evaluate the accuracy of the precision restored model, adjust the sparsity, increase or decrease the sparsity according to the preset precision loss range, and return to step S3 to continue pruning until the expected value of the sparsity is reached or the preset termination condition is met.

[0013] Optionally, also include:

[0014] Get the BERT model that has been fine-tuned on the preset task.

[0015] The BERT model is used as a teacher and the pruned model is used as a student, and the pruned model is trained using a knowledge distillation method.

[0016] Optionally, also include:

[0017] The difference in the results of logits output is directly compared using mean squared error loss to calculate the distillation loss, which is expressed as follows:

[0018]

[0019] Among them, z T is the logits output of the teacher model, z S is the logits output of the student model, and n is the number of predicted categories.

[0020] Optionally, also include:

[0021] Using the cross entropy loss between the output probability distribution and the true label as part of the objective function, the final loss function is expressed as follows:

[0022]

[0023] Among them, y S is the output probability distribution predicted by the student model, which is obtained by passing logits through the softmax function; when the sample comes from the original labeled dataset, t is the labeled true value label; when the sample comes from the dataset generated by data enhancement, the prediction result of the BERT model is used as the true value label; α is the weight hyperparameter.

[0024] The present invention has the following beneficial effects:

[0025] In implementing knowledge distillation, the present invention not only introduces a distillation loss to help the student model fit the logits output of the teacher model, but also uses a cross-entropy loss between the output probability distribution and the true label to ensure that the student model's output matches the true label of the sample. These two losses together constitute the objective function, helping the student model learn from the "hidden knowledge" of the teacher model and optimize the model's output probability distribution, thereby improving the accuracy of the pruned model. Furthermore, the present invention applies knowledge distillation to the pruning process of the LSTM model. By rationally transferring knowledge, the pruned model possesses stronger representational capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flowchart of the compression method based on the original model provided in Example 1 of the present invention.

[0027] Figure 2 This is a flowchart of the compression method based on the BERT model provided in Example 1 of the present invention.

[0028] Figure 3 Schematic diagram of the knowledge distillation architecture provided in Example 1 of the present invention.

[0029] Figure 4 Schematic diagram of a BiLSTM network for a single-sentence text classification task provided in Example 1 of the present invention.

[0030] Figure 5 Schematic diagram of a BiLSTM network for the sentence pair matching task provided in Example 1 of the present invention.

[0031] Figure 6 This is a flowchart for fine-tuning BERT-base for the sentence pair matching task provided in Example 1 of the present invention.

[0032] Figure 7 This is a schematic diagram comparing the power consumption and inference time of the compressed model provided in Example 1 of the present invention and the original model in a single sentence classification task. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solution of the present invention, the long short-term memory compression method based on knowledge distillation recovery strategy pruning provided by the present invention is described in detail below with reference to the accompanying drawings.

[0034] Example 1

[0035] This embodiment aims to improve the accuracy of pruned Long Short-Term Memory (LSTM) models by introducing a knowledge distillation method. Typically, an over-parameterized original model has strong learning and representation capabilities. However, after pruning, the network becomes smaller or subject to certain constraints, and its inherent learning capabilities may be unable to obtain complex representations. Therefore, even fine-tuning can hardly fully restore the accuracy of the original model. Knowledge distillation is an effective method to address this problem. It extracts supervisory information from a large model (such as the over-parameterized original LSTM network or the better-generalized BERT network) and uses this knowledge to guide the training of the pruned model. This improves the representational capabilities of the pruned model and enables better accuracy restoration. Specifically, the fine-tuning process in the LSTM pruning process is replaced with knowledge distillation, which transfers knowledge from the over-parameterized network to the pruned network. The teacher network can be the over-parameterized original LSTM network before pruning or a BERT network with better generalization capabilities. Through knowledge distillation, the pruned model can obtain richer information from the teacher network, thereby improving its accuracy and achieving better compression. The innovation of this embodiment lies in applying knowledge distillation to the LSTM pruning process. By properly transferring knowledge, the pruned model has stronger representational capabilities. This provides a new solution for deploying high-performance LSTM models in resource-constrained environments.

[0036] This embodiment provides a pruned model recovery strategy based on knowledge distillation, designed to better restore model accuracy during the pruning process and achieve better compression. This strategy aims to address the problem of poor generalization caused by pruning a large number of parameters. Through knowledge distillation, the knowledge of the teacher model is introduced into the pruned student model, thereby improving the accuracy of the pruned model.

[0037] Traditional fine-tuning methods typically use the one-hot labels of the original training dataset when training on a pruned model. This approach may not fully utilize the knowledge of the teacher model on the pruned small model, resulting in low recovery accuracy. However, the pruned model recovery strategy based on knowledge distillation proposed in this embodiment uses the "dark knowledge" output by the teacher model as a supervisory signal, avoiding one-hot label training. This allows the pruned model to better learn from the teacher model with stronger generalization capabilities, thereby significantly improving the accuracy of the pruned model.

[0038] This embodiment introduces a knowledge distillation step during the pruning process, enabling the pruned model to obtain important information from the teacher model, effectively compensating for any information loss that may occur during the pruning process. By meticulously transferring the knowledge of the teacher model, the representational capabilities of the pruned model are enhanced, thereby achieving better compression results under resource-constrained conditions. By selecting teacher models from different sources, the strategies are divided into two types: one based on the original model's accuracy recovery strategy and one based on the BERT model's accuracy recovery strategy. Figure 1 This paper demonstrates a compression method that combines an accuracy recovery strategy based on the original model with an iterative pruning method. The overall compression method is similar to iterative pruning, with the main difference being the use of knowledge distillation to restore the accuracy of the pruned model during the retraining step, rather than the traditional fine-tuning method. The specific steps are as follows:

[0039] 1. Train the original LSTM model: Train the LSTM on the given dataset to obtain an original model with strong generalization ability and save it.

[0040] 2. Set pruning parameters: including weight pruning method, initial value of sparsity (pruning rate), and desired final sparsity.

[0041] 3. Pruning: Evaluate the importance of connections or weight blocks according to the pruning method, determine the pruning ratio according to sparsity after sorting, set unimportant parameters to zero and prohibit their update to obtain a pruned model.

[0042] 4. Knowledge Distillation: Use the knowledge distillation method to retrain the pruned model. The original model from the first step serves as the teacher, and the pruned model as the student. By adding a distillation loss to the loss function, the student model is able to fit the logits output of the teacher model. Repeat the training several times to obtain a model with restored accuracy.

[0043] 5. Determine whether to continue pruning: Evaluate model accuracy and adjust sparsity. Based on the expected accuracy loss, increase or decrease sparsity and return to step 3 to continue pruning until the desired final sparsity is reached or the termination condition is met.

[0044] In the original model-based compression method, both the teacher model and the pruned model are generated by the same original model, so they have high similarity and compatibility, making it easier for the pruned model to imitate the logits output of the teacher model.

[0045] Figure 2This paper demonstrates a compression method that combines an accuracy recovery strategy based on the BERT model with an iterative pruning method. Similar to the compression method based on the original model, the difference is that the teacher is no longer the original model, but a BERT model fine-tuned for a specific task. The fine-tuned BERT model has superior generalization and learning capabilities, allowing the student model to learn richer knowledge from it, thereby better restoring the accuracy of the pruned model. To promote effective knowledge transfer from the BERT network to the LSTM network, this patent uses data augmentation to generate unlabeled datasets and uses the teacher model to provide predicted labels for these samples. During training, the student model learns not only from the original dataset but also from the output of the augmented samples processed by the teacher model. This patent covers common data augmentation methods used in computer vision and natural language processing tasks, ensuring effective knowledge transfer.

[0046] The knowledge distillation architecture used in this embodiment is as follows Figure 3 As shown in Figure 1, it includes a student model and a teacher model. The student model is a smaller, simpler LSTM sparse model after pruning, while the teacher model can be the original LSTM model before pruning or a large, complex BERT model obtained through fine-tuning. Figure 3 The knowledge distillation architecture used in this embodiment is shown. The distillation loss between the student model and the teacher model is a key part of achieving the accuracy recovery of the pruned model. By adding distillation loss to the loss function, the student model is forced to fit the output of the teacher model, so that the accuracy can be better restored. The design and calculation of distillation loss can help the student model learn from the "dark knowledge" of the teacher model, so that the pruned model can approach the performance of the teacher model, thereby improving the accuracy of the pruned model. In the implementation details of knowledge distillation, this embodiment modifies the loss function of the pruned LSTM model and adds a distillation loss term to penalize the difference between the output of the student model and the soft label generated by the teacher model. Specifically, this embodiment uses mean square error loss to directly compare the difference in logits results to realize the calculation of distillation loss. The distillation loss is shown as follows:

[0047]

[0048] Among them, z T is the logits output of the teacher model, z S is the logits output of the student model, and n is the number of predicted categories.

[0049] In addition to fitting the student model to the teacher model's logits, this embodiment also needs to ensure that the output of the student model matches the true label corresponding to the sample. This part of the loss is the same as the loss function used in traditional model training. Specifically, this embodiment uses the cross entropy loss between the output probability distribution and the true label as part of the objective function. The final loss function is as follows:

[0050]

[0051] Among them, y S is the output probability distribution of the student's prediction, which is obtained by passing the logits through the softmax function. When the sample is distilled from the original labeled dataset, t is the true value label of the label. When the sample comes from the dataset generated by data augmentation, the result predicted by the BERT model can be regarded as the true value label. α is a weight hyperparameter. In summary, in the implementation of knowledge distillation, this embodiment not only introduces the distillation loss to allow the student model to fit the logits of the teacher model, but also adopts the cross entropy loss of the output probability distribution and the true label to ensure that the output of the student model matches the true label of the sample. These two parts of the loss together constitute the objective function, which helps the student model learn from the "dark knowledge" of the teacher model and optimizes the output probability distribution of the model, thereby improving the accuracy of the pruned model.

[0052] Current mainstream pruning methods typically use fine-tuning to restore the accuracy of the pruned model. However, for coarse-grained pruning, which is more suitable for inference acceleration, fine-tuning often fails to effectively restore model accuracy. On the other hand, knowledge distillation can be used not only as a compression method but also as a small network training method. Currently, research on combining LSTM weight pruning with knowledge distillation training is relatively scarce. To achieve compression of the LSTM model, this embodiment introduces a pruned model accuracy recovery strategy based on knowledge distillation, aiming to better restore model accuracy.

[0053] Based on this, this embodiment proposes two compression methods: one based on the original model and one based on the BERT model. By combining a pruning model accuracy recovery strategy based on knowledge distillation, coarse-grained pruning, and quantization, the compressed model achieves higher inference speed and energy efficiency than the original dense model without sacrificing model accuracy.

[0054] During the experiment, this embodiment combined a pruning model accuracy recovery strategy based on knowledge distillation, coarse-grained pruning, and quantization. By combining these techniques, the compressed model maintained high accuracy while significantly improving inference speed and energy efficiency. The combined application of these innovative technologies provides an effective solution for model compression and acceleration. Therefore, this embodiment has significant application value in the field of LSTM model compression and is eligible for patent application.

[0055] This embodiment chooses to evaluate on the single-sentence text classification and sentence-pair semantic matching tasks. In the single-sentence text classification task, the model needs to classify the single-sentence input text into different categories. This task is often used in scenarios such as spam identification and news topic classification. In the sentence-pair semantic matching task, the model needs to determine whether the semantics between two texts are the same. This task is mostly used in information retrieval, question-answering systems, dialogue systems and other fields. Through evaluation on these two tasks, this embodiment can comprehensively verify the effectiveness and applicability of the knowledge distillation accuracy recovery strategy for the pruned model. At the same time, these tasks have important application value in the field of natural language processing, and therefore have certain guiding significance for improving the performance of the pruned model in practical applications.

[0056] First, let's consider the accuracy of the pruned model. In classification tasks, labels can be categorized as positive or negative. After inputting a sample, the classification algorithm can produce only two possible outcomes: a positive prediction or a negative prediction. Combining the true label corresponding to the sample with the predicted result yields four possible outcomes, as shown in Table 1 below:

[0057] Table 1 Four scenarios predicted by the model

[0058]

[0059] Among them, TP and TN are cases where the judgment is correct, and FP and FN are cases where the judgment is incorrect. By counting the number of TP, TN, FP, and FN, we can obtain the following three evaluation indicators of model accuracy:

[0060] Accuracy (Acc) refers to the ratio of correctly classified samples to the total number of samples. Accuracy is a statistic for all samples and measures the overall performance of the classifier. Acc is calculated as follows:

[0061]

[0062] Precision refers to the ratio of correctly classified positive examples to the number of positive examples determined by the classifier. Precision is a statistic for a subset of samples, focusing on the data determined by the classifier to be positive examples. Precision is calculated as follows:

[0063]

[0064] Recall refers to the ratio of correctly classified positive examples to the number of true positive examples. Recall is also a statistic for some samples, focusing on the statistics of true positive examples. The calculation of recall is as follows:

[0065]

[0066] In addition to accuracy, you can also use the F1 score to measure model accuracy. F1 is the harmonic mean of precision and recall, taking both into account. A higher F1 score indicates a good balance between precision and recall. The F1 score is calculated as follows:

[0067]

[0068] Sparsity is the proportion of zero values ​​in the pruned weight matrix. Under the same sparsity, the higher the accuracy of the pruned model, the better the compression effect of the compression method.

[0069] To evaluate the performance and energy efficiency of the model before and after compression, the following metrics need to be measured:

[0070] Speed: Performance can be compared by comparing speed. When executing the same prediction task, you can measure the model's inference time to compare speed. The shorter the inference time, the faster the speed, and the better the model performance.

[0071] Power consumption: The power consumption when executing the prediction task, measured in watts (W). The hardware platform used in this chapter's experiments is an Nvidia GPU. This example uses nvidia-smi to record the GPU power consumption during model inference.

[0072] Energy efficiency ratio: Energy efficiency refers to the energy required to complete a prediction task. It is calculated by multiplying the hardware power consumption by the inference time. When executing the same prediction task, the energy efficiency ratio can be calculated by calculating the ratio of speed to power consumption. A higher energy efficiency ratio indicates a more efficient model.

[0073] For the single-sentence text classification task, the SST-2 (Stanford Sentiment Treebank) dataset from GLUE (The General Language Understanding Evaluation) was used. For the sentence pair matching task, this example used the QQP (Quora Question Pairs) dataset from GLUE. GLUE is a multi-task natural language understanding benchmark and analysis platform developed by institutions such as New York University and the University of Washington.

[0074] The SST-2 corpus is a collection of film reviews compiled by Stanford University. It consists of 8,551 sentences in the training set, 1,043 in the development set, and 1,063 in the test set. The samples consist of sentences from film reviews and their sentiment annotations. Sample reviews are labeled as either positive (with a sample label of 1) or negative (with a sample label of 0). SST-2 is a single-sentence binary classification task, requiring the task to determine whether a single sentence is positive or negative in sentiment.

[0075] QQP is a similarity and paraphrase task, and the corpus is drawn from question pairs on the community question-and-answer website Quora. The training set consists of 363,870 sentences, the development set of 40,431, and the test set of 390,965. Each example consists of two sentences separated by a "[tab]." Semantically equivalent sentence pairs are positive samples (labeled 1); semantically unequal sentence pairs are negative samples (labeled 0). QQP has an imbalanced positive and negative sample population, with negative samples accounting for 63% and positive samples accounting for 37%. The task behind QQP is sentence-to-text matching, which involves inputting a pair of texts and determining whether they are semantically equivalent.

[0076] The SST-2 task uses Acc as the evaluation metric, and QQP uses Acc and F1 as evaluation metrics.

[0077] BiLSTM is a classic LSTM network that can better extract contextual features of text in long text processing. This embodiment uses BiLSTM and trains models for single sentence text classification and sentence pair matching. Figure 4 As shown in Figure 2. The network consists of an embedding layer, a BiLSTM layer, a maximum pooling layer, a fully connected layer, and a softmax function. The specific reasoning process is as follows: for a sentence of T words {W t} t＝1,2...,T , first the embedding layer will process the word vector sequence {x t} t＝1,2,...,T The BiLSTM layer processes the word vector sequence to obtain a set of feature vectors {h t} t＝1,2,...,T .h t is the hidden state passed in the forward direction and reverse propagation The concatenated vector is calculated as follows:

[0078]

[0079]

[0080] In order to reduce the dimension of the feature vector, this embodiment uses the maximum pooling layer to extract the maximum value of each row of the feature vector and combine them into the sentence representation vector V. The representation vector V is fed into the fully connected layer to obtain logits, which are then calculated using the softmax function to obtain the probability distribution of the predicted category. The category corresponding to the largest probability value in the probability distribution is the model's predicted category for the sentence text. The BiLSTM network structure for the sentence pair matching task is as follows: Figure 5 The specific calculation process is to first input sentence A in the sentence pair into network A and obtain the representation vector V corresponding to sentence A. a , then input sentence B in the sentence pair into network B to obtain the representation vector V corresponding to sentence B b , and then apply the following formula between the representation vectors of the two sentences to obtain the matching vector M:

[0081] M=f(V a , V b )=[V a ,V b , V a e V b ,|V a -V b |]

[0082] The matching vector M is fed into the fully connected layer to obtain logits, which are then calculated using the softmax function to obtain the probability distribution of the predicted category. Here, there are only two categories: semantic match of the sentence pairs and semantic mismatch of the sentences. It should be noted that Network A and Network B in the figure are actually the same network, meaning that the weight parameters of the embedding layer and BiLSTM layer are the same. In terms of specific implementation details, this embodiment sequentially inputs Sentence A and Sentence B to obtain their respective representation vectors, which are then input into the fully connected layer for further processing to obtain the final prediction result. This design allows for effective representation and matching of Sentence A and Sentence B while maintaining network parameter sharing, thereby completing the sentence pair semantic matching task. This shared parameter design helps reduce the model's storage space and computational complexity, and enables efficient deployment by better utilizing limited resources. To improve model accuracy, the embedding layer generally uses a pre-trained Word2Vec or Glove model. This embodiment uses a 300-dimensional Word2Vec model trained on Google News. Secondly, in general, the sequence needs to be padded to a fixed length before being input into the model. For the SST-2 task, this embodiment sets the fixed length value to 200, and for the QQP task, the value is set to 100.

[0083] BERT has proven highly effective in sequence processing tasks, achieving state-of-the-art results on many benchmarks. Its generalization and learning capabilities are significantly superior to those of LSTM. BiLSTM is a bidirectional model that processes sequences in both forward and backward directions to generate contextual representations. BERT also considers the context of a word when generating representations, so it can also be considered a bidirectional model. This similarity between the two facilitates knowledge transfer from BERT to LSTM in this example.

[0084] Unlike LSTM, which is trained end-to-end from scratch on a specific task, BERT is typically fine-tuned on a specific task. This example selects BERT-base as the teacher model and fine-tunes it to obtain the teacher model. BERT has several variants with different scales and capabilities, and this example selects the BERT-base version. BERT-base consists of 12 Transformer layers, each with 768 hidden units and 12 self-attention heads, for a total of 110 million parameters. Because BERT-base has been trained on large-scale pre-training tasks, it has strong generalization and learning capabilities, and can serve as a teacher model to provide useful guidance and knowledge transfer to the pruned LSTM model.

[0085] The pre-trained BERT-base model used in this embodiment comes from TensorFlow Hub. The pre-trained BERT-base model does not include a classification layer for text classification tasks. Therefore, when using the BERT-base model, it is necessary to add a classification layer for a specific task on top of it. Since simple binary classification is required, it is chosen to directly create a fully connected layer with 2 output units and add it to the network. When using the BERT-base model for text classification, the input sequence needs to be processed according to a specific format, in which a special "[CLS]" tag needs to be added at the beginning of the sequence. Then, the representation of the position corresponding to "[CLS]" is fed to the fully connected layer to obtain logits. Finally, logits is calculated through the softmax function to obtain the probability distribution of the predicted category. This processing method makes the BERT-base model applicable to specific text classification tasks and obtains the corresponding classification results through a simple classification layer.

[0086] The BERT-base fine-tuning process for the sentence pair matching task is as follows Figure 6As shown in the figure, it can be observed that the BERT for the sentence pair matching task and the BERT for the single-sentence text classification task are structurally identical. The difference lies in the stage of creating the input sequence. Since the input is a sentence pair, an additional "[SEP]" tag needs to be added between the two sentences to separate the sentences. This processing method enables the BERT-base model to be applicable to the sentence pair matching task while maintaining the same network structure as the single-sentence text classification task. During the fine-tuning process, by inputting sentence pairs and training the corresponding tasks, the BERT-base model can learn semantic representations suitable for the sentence pair matching task according to different task requirements. In this way, the BERT-base model can play a role in the sentence pair matching task while maintaining the same network structure as the single-sentence text classification task, thereby improving the generalization ability and applicability of the model.

[0087] In order to verify the performance of the compression method of this embodiment under weight pruning of different granularities, we conducted experiments on two pruning granularities: fine-grained pruning and coarse-grained pruning. At the same time, in order to better transfer the knowledge within the BERT model to the pruned LSTM model, we used data augmentation methods to generate unlabeled data. In terms of specific implementation details, this embodiment adopts a random masking method, that is, randomly selecting words in the sentence with a probability Pmask. The selected words will be masked and replaced with the "[MASK]" mark. We set Pmask to 0.15. Finally, in the process of coarse-grained pruning of the BiLSTM model with a hidden layer size of 1600, setting the block size to 8*8 can achieve a balance between performance and accuracy.

[0088] Table 2 Experimental environment

[0089]

[0090] Table 2 provides some experimental environment information. The experiments were implemented using TensorFlow. TensorFlow is an open-source software library developed by Google for numerical computing and machine learning. Released in 2015, it has since become one of the most widely used and popular machine learning frameworks. In addition to TensorFlow, the experiments also utilized tools such as NumPy, GenSim, Matplotlib, and Anaconda. This example applies the compression method of this example to a BiLSTM model and compares iterative pruning and fine-tuning methods proposed in some studies. Two compression methods are used in this example: a compression method based on the original model, using the original LSTM model as the teacher network for knowledge distillation; and a compression method based on BERT, using the BERT-base model as the teacher network for knowledge distillation. To verify the performance of the compression method at different granularities and sparsity levels, the experiments used fine-grained and coarse-grained pruning, and three sparsity levels of 30%, 60%, and 90%.

[0091] Table 3 Experimental results of BiLSTM compression on SST-2 and QQP with fine pruning granularity

[0092]

[0093] aThe baseline model refers to the uncompressed BiLSTM model

[0094] b KD-LSTM refers to the compression method based on the original model (excluding quantization)

[0095] c KD-BERT refers to the compression method based on the BERT model (excluding quantization)

[0096] d↓ refers to the change in accuracy of the compressed model relative to the baseline model

[0097] Table 3 shows the experimental results of a compressed BiLSTM model on SST-2 and QQP tasks with fine pruning granularity. On the SST-2 task, the compressed model of this embodiment achieved 0.4 to 1.36 percentage points higher accuracy than the pruned and fine-tuned model. On the QQP task, the compressed model of this embodiment achieved 0.69 to 1.05 percentage points higher accuracy and 0.49 to 0.82 higher F1 scores than the pruned and fine-tuned model.

[0098] Table 4 Experimental results of BiLSTM compression on SST-2 and QQP with coarse pruning granularity

[0099]

[0100] Table 4 shows the experimental results of the compressed BiLSTM model on SST-2 and QQP tasks, with the pruning granularity set to coarse. On the SST-2 task, the compressed model of this embodiment achieved an accuracy increase of 0.32 to 0.92 percentage points compared to the pruned and fine-tuned model. On the QQP task, the compressed model of this embodiment achieved an accuracy increase of 0.32 to 0.81 percentage points compared to the pruned and fine-tuned model, and an F1 score increase of 0.4 to 0.97.

[0101] Experimental results show that at low sparsity, the BERT-based compression method achieves better model accuracy recovery than the original model-based compression method. At high sparsity, the accuracy recovery is comparable, with the original model-based compression method even achieving slightly better results. At low sparsity, the pruned model retains sufficient parameters to fit the BERT network's output and learn its internal knowledge. At high sparsity, however, fewer parameters remain, making it difficult for the pruned model to easily learn the BERT network's internal knowledge. In this case, knowledge from the original LSTM model, which has a more similar structure, is more easily transferred to the pruned model, improving accuracy recovery. Overall, the knowledge distillation-based compression strategy can better help the pruned model learn and fit the teacher model's knowledge, especially at low sparsity. In practical applications, we can choose the appropriate pruning granularity and compression method based on the required compression rate and model accuracy to achieve optimal compression and performance recovery.

[0102] Table 5. Impact of quantization operation on the accuracy of the compression model (set to coarse-grained pruning)

[0103]

[0104] In order to further compress and accelerate the pruned model, this embodiment applied a 16-bit quantization operation to a coarse-grained pruned model with a sparsity of 60% in the experiment. According to the results in Table 5, it can be seen that the quantization operation has little impact on the accuracy of the model. In this case, the compressed model can still maintain a high accuracy compared to the baseline model without obvious accuracy loss. This shows that the compression method proposed in this embodiment not only restores the accuracy of the model during the pruning process, but also maintains the stability of the model's accuracy in subsequent quantization operations. The quantization operation plays an important role in reducing model storage and accelerating the inference process, while maintaining a high level of model performance.

[0105] This embodiment also evaluates the speed and energy efficiency of the model before and after compression. Figure 7As shown, the inference time and power consumption of the compressed model (the compression process includes coarse-grained pruning, a precision recovery strategy based on knowledge distillation, and quantization) and the original model are compared when performing single-sentence classification on the same text. Sparsity 0 in the figure represents the original, uncompressed BiLSTM model. Experimental results show that as sparsity increases, the inference time required for single-sentence classification with the compressed model is significantly reduced, while power consumption during inference also decreases. Specifically, when the coarse-grained pruning sparsity is set to 60% and 16-bit quantization is performed, the accuracy of the compressed BiLSTM model is not significantly affected. This shows that the compression method proposed in this embodiment significantly improves the model's inference speed and energy efficiency without sacrificing model accuracy. Compared to the original model, the compressed model achieves an approximately 2.3x speedup and a 2.8x improvement in energy efficiency. This means that in resource-constrained scenarios, the compressed model of this embodiment can maintain high accuracy while providing more efficient inference capabilities, making it suitable for deployment in scenarios such as mobile devices, embedded systems, and edge devices.

[0106] In this embodiment, in addition to introducing a distillation loss to help the student model fit the teacher model's logits output, the cross-entropy loss between the output probability distribution and the true label is used to ensure that the student model's output matches the true label of the sample. These two losses together constitute the objective function, helping the student model learn from the teacher model's "hidden knowledge" and optimize the model's output probability distribution, thereby improving the accuracy of the pruned model. Furthermore, this embodiment applies knowledge distillation to the pruning process of the LSTM model. By rationally transferring knowledge, the pruned model possesses stronger representational capabilities.

[0107] It will be understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present invention, and the present invention is not limited thereto. Those skilled in the art will appreciate that various modifications and improvements can be made without departing from the spirit and substance of the present invention, and such modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A long short-term memory compression method based on knowledge distillation recovery strategy pruning, characterized in that: include: Step S1: train a long short-term memory model based on the obtained dataset to obtain an original model with a preset generalization capability, and save the original model; for the single sentence text classification task, select the GLUE SST-2 dataset; for the sentence pair matching task, select the GLUE QQP dataset; Step S2: setting pruning parameters, including weight pruning method, initial value of sparsity, and expected value of sparsity; Step S3: Evaluate the importance of connections or weight blocks according to the weight pruning method, determine the pruning ratio according to the sparsity after sorting, set the corresponding parameters to zero according to the pruning ratio, and prohibit updating the parameters that have been set to zero, so as to obtain a pruned model; Step S4: Use the knowledge distillation method to train the pruned model, using the original model as the teacher and the pruned model as the student. By adding distillation loss to the loss function, the student model fits the logits output of the teacher model. After iterative training for a preset number of times, a model with restored accuracy is obtained. Step S5: Evaluate the accuracy of the precision restored model, adjust the sparsity, increase or decrease the sparsity according to the preset precision loss range, and return to step S3 to continue pruning until the expected value of the sparsity is reached or the preset termination condition is met.

2. The long short-term memory compression method based on knowledge distillation recovery strategy pruning according to claim 1 is characterized in that: Also includes: Get the BERT model that has been fine-tuned on the preset task. The BERT model is used as a teacher and the pruned model is used as a student, and the pruned model is trained using a knowledge distillation method.

3. The long short-term memory compression method based on knowledge distillation recovery strategy pruning according to claim 2 is characterized in that: Also includes: The difference in the results of logits output is directly compared using mean squared error loss to calculate the distillation loss, which is expressed as follows: Among them, z T is the logits output of the teacher model, z S is the logits output of the student model, and n is the number of predicted categories.

4. The long short-term memory compression method based on knowledge distillation recovery strategy pruning according to claim 3 is characterized in that: Also includes: Using the cross entropy loss between the output probability distribution and the true label as part of the objective function, the final loss function is expressed as follows: Among them, y S is the output probability distribution predicted by the student model, which is obtained by passing logits through the softmax function; when the sample comes from the original labeled dataset, t is the labeled true value label; when the sample comes from the dataset generated by data enhancement, the prediction result of the BERT model is used as the true value label; α is the weight hyperparameter.

Images