A data-efficient knowledge distillation method and system based on dynamic target alignment of teaching assistants

Abstract

The present application relates to a kind of data efficient knowledge distillation method and system based on dynamic target alignment of teaching assistant.The method comprises: in data layer, using entropy-based strategy to select information-rich samples, which are input into student model for learning;In model layer, introduce teaching assistant model, and dynamically query teacher model or teaching assistant model according to the evolution of the ability of student model during training;In target layer, select information-rich layers from teacher model according to entropy value, so that student model and selected information-rich layers in teacher model are dynamically aligned;Through student model, teacher model and teaching assistant model, knowledge is transferred from teacher model to student model, and knowledge distillation performance is improved with the evolution of the ability of student model.The present application can comprehensively improve knowledge distillation effect from data, model and target level.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and deep learning technology, and specifically relates to a data-efficient knowledge distillation method and system based on dynamic goal alignment with teaching assistants. Background Technology

[0002] Large-scale pre-trained language models (PLMs), such as BERT, RoBERTa, T5, and GPT-4, have achieved excellent performance in natural language processing tasks and only require minor adjustments for downstream tasks to meet practical needs. However, due to their large number of model parameters, existing PLMs have high computational resource requirements during training, often leading to training failures due to insufficient resources. To reduce computational costs, knowledge distillation has become an effective method to address these issues. Its main idea is to compress large models into smaller, more concise models through knowledge transfer. In recent years, knowledge distillation has become one of the most popular model compression methods among researchers. Existing methods for distilling large language models are mainly divided into two types: general distillation and task-specific distillation.

[0003] General distillation primarily performs knowledge distillation on a general corpus. This distillation method is carried out during the pre-training of a large model and requires sufficient computational resources. In other words, general distillation is also a form of online distillation, where the teacher model is not fixed, and the student model learns distillation during the pre-training of the teacher model. For example, “Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.” provides a method for pre-training a smaller general language representation model, which can then be fine-tuned and performs well on various tasks. Pre-training remains crucial in the setup of small model architectures, and compact models with fine-tuned pre-trained models can generally rival more complex strategies employed in corresponding works. Furthermore, a direct approach called deep self-attention distillation can be used to compress large pre-trained Transformer-based models. Self-attention relation distillation can then be used to simplify deep self-attention distillation.

[0004] Task-related distillation trains a student model on a specific downstream task, while the teacher model is pre-trained, enabling effective distillation learning without requiring powerful computational resources. In fact, task-related distillation is also a form of offline distillation, where the teacher model is fixed, and the student model learns from the pre-trained teacher model. For example, “Sun, S.; Cheng, Y.; Gan, Z.; and Liu, J. 2019. Patient Knowledge Distillation for BERT Model Compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 4322–4331.” encourages student models to extract knowledge from the middle layers of the teacher model, rather than simply learning parameters from the last layer. Recently, some studies have explored the idea of ​​combining the knowledge of models with different capabilities. Furthermore, the concept of knowledge distillation has been extended from imitating the teacher model to integrating teacher knowledge, leading to the proposal of knowledge integration for PLMs. This knowledge integration attempts to train a flexible student model that can make predictions on a joint set of teacher model labels given multiple fine-tuned teacher models, each capable of classifying a unique set of labels.

[0005] Existing knowledge distillation methods based on large-scale pre-trained language models, whether general distillation or task-specific distillation, each have their limitations. General distillation has significant drawbacks: performing knowledge distillation during pre-training is extremely computationally expensive, requiring additional training costs and making it difficult to apply in resource-constrained environments. While task-specific distillation is more suitable for resource-constrained production environments, it still suffers from two problems: the phenomenon of student models learning repeatedly and the capability gap between large-scale teacher models and small-scale student models. For example, in Table 1, for a 3-layer student model, 50% of the data can approximate the accuracy on all data; furthermore, although BERT-large is stronger than BERT-base, the 3-layer student model distilled from BERT-large is weaker.

[0006] Table 1. Task-related distillations based on BERT

[0007]

[0008] To address these issues, it is necessary to design an efficient distillation method that can comprehensively improve distillation performance from the perspectives of data, models, and objectives. Summary of the Invention

[0009] To address the aforementioned problems, this invention provides a data-efficient knowledge distillation method and system based on dynamic goal alignment using teaching assistants.

[0010] The technical solution adopted in this invention is as follows:

[0011] An efficient knowledge distillation method based on dynamic goal alignment using teaching assistants includes the following steps:

[0012] At the data layer, an entropy-based strategy is used to select information-rich samples and input them into the student model for learning.

[0013] At the model layer, a teaching assistant model is introduced, which dynamically queries the teacher model or teaching assistant model based on the evolution of the student model's capabilities during training.

[0014] In the target layer, information-rich layers are selected from the teacher model based on the entropy value, so that the student model is dynamically aligned with the information-rich layers in the selected teacher model;

[0015] By using student models, teacher models, and teaching assistant models, knowledge is transferred from the teacher model to the student model, and the knowledge distillation performance is improved as the capabilities of the student model evolve.

[0016] Furthermore, the method of selecting information-rich samples using an entropy-based strategy includes:

[0017] Given a training batch of n instances, for each instance x, let P(y|x)=φ(S(x)) represent the output class probability distribution of the student model on the class label y, where φ represents the softmax function and S(x) represents the soft label logits vector output by the student model S;

[0018] The scaling entropy of the probability distribution is used to calculate the uncertainty score U of instance x. x U x The calculation method is as follows:

[0019]

[0020]

[0021] Where Y is the number of category labels, and U is the uncertainty score of the instance. xThe instances are sorted, and only the top N×r instances are selected to query the teacher model, where r∈(0,1], which refers to the selection ratio of the number of instances to be queried; the selected instances have a high uncertainty score, indicating that they are information-rich instances that the student model should learn from.

[0022] Furthermore, the step of dynamically querying the teacher model or teaching assistant model based on the evolution of the student model's capabilities during training includes: in the initial training phase, the student model learns from the teaching assistant model; as the student model becomes more powerful, it seeks more accurate supervision signals from the teacher model.

[0023] Furthermore, the step of dynamically querying the teacher model or teaching assistant model based on the evolution of the student model's capabilities during training includes:

[0024] A batch of instances is ranked based on the prediction confidence of the student model, where the confidence is measured by entropy and calculated as follows:

[0025] C x =Entropy(φ(S(x)))

[0026] Among them, C x C represents confidence level, Entropy represents entropy, and C represents confidence level. x The higher the value, the more uncertain the student model;

[0027] The instances are divided into deterministic and indeterminate instances. For deterministic instances, the student model learns supervision signals from the teacher model, while the teaching assistant model provides soft labels for the indeterminate instances in the student model. The loss function is as follows:

[0028]

[0029] in, This represents the loss function based on teaching assistants. The KL divergence distance between the student model and the teacher model. This represents the KL divergence distance between the student model and the teaching assistant model.

[0030] Furthermore, the step of selecting information-rich layers from the teacher model based on entropy values, and dynamically aligning the student model with the selected information-rich layers in the teacher model, includes:

[0031] Calculate the entropy of the hidden representation H of the teacher model:

[0032]

[0033] Among them, R x The larger the value, the more hidden information the corresponding layer contains. This means that the R value for each layer... x Sort from largest to smallest, select Rx The M layer has higher values; the loss function for dynamic target alignment is:

[0034]

[0035] Where M represents the number of layers in the student model, I(j) represents the alignment of the i-th layer of the student model with the j-th layer of the teacher model, and h s and h t Let represent the hidden representations of the student model and the teacher model, respectively. This represents the hidden representation of the i-th layer of the student model. This represents the hidden representation of the teacher model at layer j, which is aligned with the i-th layer of the student model.

[0036] Furthermore, the student model is trained by summing the three losses, with the total loss being:

[0037]

[0038] in, λ1, λ2, and λ3 are the cross-entropy loss function, and λ1, λ2, and λ3 are hyperparameters used to adjust the loss weights.

[0039] The entire training process includes: fine-tuning the BERT model on the corresponding downstream tasks to obtain the teacher model; distilling the teacher model to obtain the teaching assistant model; and training the final student model based on the total loss.

[0040] Furthermore, the training process of the student model includes:

[0041] Input training data D, fine-tune the number of rounds E, and train the parameter set Ω of the student model;

[0042] Begin fine-tuning the student model. In each round of fine-tuning, for each batch D... b ∈D, by the uncertainty fraction U x Select samples with rich information;

[0043] Based on confidence level C x The input samples are divided into two parts: certain and uncertain. The loss based on the teaching assistant is then...

[0044] Calculate the entropy R of the hidden representation H of the teacher model. x Student model aligned with teacher model in R x For layers M with larger values, the loss for dynamic target alignment is L. DA ;

[0045] Calculate total loss Update parameter set Ω;

[0046] By performing the above steps for round E fine-tuning, and repeating the cycle until the end of the round, the optimized student model is obtained.

[0047] A data-efficient knowledge distillation system based on dynamic target alignment using teaching assistants includes a student model, a teacher model, and a teaching assistant model. It employs an entropy-based strategy to select information-rich samples and input them into the student model for learning. The system dynamically queries the teacher model or teaching assistant model based on the evolution of the student model's capabilities during training. Information-rich layers are selected from the teacher model based on entropy values, enabling dynamic alignment between the student model and the selected information-rich layers in the teacher model. Knowledge is transferred from the teacher model to the student model through the student model, teacher model, and teaching assistant model, and the knowledge distillation performance improves as the student model's capabilities evolve.

[0048] The beneficial effects of this invention are as follows:

[0049] This invention addresses the problem of compressing pre-trained language models using knowledge distillation. Existing general-purpose distillation and task-specific distillation each have their drawbacks. General-purpose distillation requires additional training costs and substantial computational resources, making it unsuitable for resource-constrained scenarios. To address the shortcomings of general-purpose distillation, this invention employs offline distillation, achieving efficient distillation without consuming massive computational resources. To address the deficiencies of task-specific distillation, this invention, based on offline distillation, designs an entropy-based data selection strategy, introduces a teaching assistant model, and dynamic target alignment, mitigating the following issues: repeated knowledge learning by the student model and performance degradation due to the ability gap between the teacher and student models. This invention comprehensively improves distillation performance regardless of whether the student model has 3 or 6 layers. Attached Figure Description

[0050] Figure 1 This is a framework diagram of the present invention.

[0051] Figure 2 It compares four selection strategies under different selection ratios.

[0052] Figure 3 This represents the performance of a 3-layer student model with different levels of teaching assistants.

[0053] Figure 4 This represents the performance of a 6-layer student model with different levels of teaching assistants. Detailed Implementation

[0054] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0055] This invention proposes a data-efficient knowledge distillation method based on dynamic goal alignment using teaching assistants, addressing the problems of existing task-related distillation methods. Specifically, firstly, an entropy-based strategy is used at the data layer to select representative samples for learning, maximizing data efficiency and preventing repetitive learning by the student model at the data level. Then, a teaching assistant model is introduced at the model level, allowing the student model to decide whether to query the teacher model or the teaching assistant model to improve knowledge distillation performance. Furthermore, at the goal level, a dynamic goal alignment mechanism is designed to dynamically align the information-rich teacher layer, alleviating the goal supervision problem between the teacher and student models. Simultaneously considering the data, model, and goal levels, this approach facilitates knowledge transfer from the teacher model to the student model and improves distillation performance as the student model's capabilities evolve.

[0056] The efficient knowledge distillation based on dynamic goal alignment using teaching assistants proposed in this invention mainly targets three levels of knowledge distillation: the data layer, the model layer, and the goal layer. For example... Figure 1 The overall design of the invention is described below. First, at the data layer, an entropy-based method is used to select information-rich data to prevent repetitive learning by the student model. Then, at the model layer, a teaching assistant model is introduced to match the capabilities of the student model and the teacher model, mitigating the capability gap between them and thus improving the performance of the student model. Furthermore, at the target layer, aligning the information-rich layers in the teacher model also brings additional benefits. The specific implementation strategy is to select information-rich layers from the teacher model based on entropy values, and then align the student model with the hidden representations therein. This will be elaborated in detail from four aspects below.

[0057] (1) The first part is data selection. This part explores which data is more beneficial to the performance of the student model. As distillation proceeds, the student model becomes stronger, which can easily lead to the student model repeatedly learning from samples it has already mastered. Therefore, selecting information-rich samples is important to avoid the student model learning repeatedly. Formally, given n instances in a training batch, for each instance x, let P(y|x) = φ(S(x)) represent the output class probability distribution of the student model on the class label y, where φ represents the softmax function and S(x) represents the soft label (logits) vector output by the student model S. The scaling entropy of the probability distribution is used to calculate the uncertainty score U of x. x U x The calculation method is as follows:

[0058]

[0059]

[0060] Where Y is the number of category labels. Instances x are sorted according to their uncertainty scores, and only the top N×r instances are selected for querying the teacher model. Here, r∈(0,1] refers to the selection ratio of the number of instances to be queried. The selected instances have higher uncertainty scores, indicating that they are information-rich instances that the student model should learn from.

[0061] (2) The second part is the teaching assistant model. The capability gap between the teacher model and the student model is an inherent problem. In this part, we approach the issue from the model level by introducing a teaching assistant model. The teacher model or the teaching assistant model is dynamically queried based on the evolution of the student model's capabilities during training. The core purpose behind this is to enable the student model to adjust its learning process according to its current state. In the initial training phase, the student model can learn from the teaching assistant model. As the student model becomes more powerful, it can seek more accurate supervision signals from the teacher model. More specifically, a batch of instances is ranked based on the prediction confidence of the student model. The confidence level is measured by entropy, calculated as follows:

[0062] C x =Entropy(φ(S(x)))

[0063] Among them, C x C represents confidence level, Entropy represents entropy, and C represents confidence level. x The higher the value, the more uncertain the student model becomes. Therefore, for the student model, instances can be divided into deterministic and uncertain instances. For deterministic instances, the student model learns supervision signals from the teacher model, while the teaching assistant model provides soft labels for the uncertain instances in the student model. The loss function is then as follows:

[0064]

[0065] in, This represents the loss function based on teaching assistants. The KL (Kullback-Leibler, KL) divergence distance between the student model and the teacher model. This represents the KL divergence distance between the student model and the teaching assistant model.

[0066] (3) The third part is dynamic target alignment. At the target level, the performance of the student model is improved by learning the supervision signal of the teacher model. Inspired by existing work on the alignment of intermediate layers between the teacher and student models, this invention further studies dynamic target alignment to improve distillation performance. According to previous research, the corresponding alignment target weights are determined by hyperparameter search and remain unchanged during training. To address the above problem, an information-rich layer is selected based on the entropy of the corresponding layer to dynamically align the hidden representation of the teacher model, thereby preventing unnecessary alignment. First, the entropy of the hidden representation H of the teacher model is calculated:

[0067]

[0068] Among them, R x The larger the value, the more hidden representation information the layer contains. The R value for each layer... x Sort from largest to smallest, select R x For layers M with higher values, the loss function for dynamic target alignment becomes:

[0069]

[0070] Where M represents the number of layers in the student model. I(j) indicates the alignment of the i-th layer of the student model with the j-th layer of the teacher model. s and h t Let represent the hidden representations of the student model and the teacher model, respectively. This represents the hidden representation of the i-th layer of the student model. This represents the hidden representation of the teacher model at layer j, which is aligned with the i-th layer of the student model.

[0071] (4) The fourth part is the total loss. This invention trains the student model by summing the three losses. The total loss takes the following form:

[0072]

[0073] in, λ1, λ2, and λ3 are the cross-entropy loss function, and λ1, λ2, and λ3 are hyperparameters used to adjust the loss weights. The entire training process consists of three steps: First, fine-tuning the BERT model on the corresponding downstream task to obtain the teacher model; second, distilling the teacher model to obtain the teaching assistant model; and third, obtaining the final student model through Algorithm 1.

[0074] Downstream tasks can be a range of text classification tasks, such as semantic similarity matching, sentiment classification, and natural language inference. For example, for semantic similarity matching, sentiment classification, and natural language inference tasks, the method of this invention achieves the best distillation results compared to existing task-related distillation methods, achieving optimal accuracy regardless of whether the student model configuration is 3 or 6 layers. Specifically, with a 6-layer student model configuration, this invention achieves accuracy close to that of the original 12-layer BERT model on the corresponding tasks.

[0075] The following section provides a detailed description of the specific implementation of the invention. First, it introduces the dataset, evaluation metrics, comparison algorithms, and implementation details used in the experiments. Then, experiments are conducted on a recognized benchmark dataset, showcasing the results. Next, the contribution of each component of the proposed method is analyzed. Finally, model analysis is performed, including analysis of variance, data selection strategy analysis, and teaching assistant layer analysis, to derive the optimal parameter configuration.

[0076] The training process for the student model includes the following steps:

[0077] First, input the training data D, fine-tune the number of rounds E, and train the parameter set Ω of the student model;

[0078] Then, the student model is fine-tuned. In each round of fine-tuning, for each batch D... b ∈D, by uncertainty fraction Select samples with rich information;

[0079] Then, based on the confidence level C x =Entropy(φ(S(x))) divides the input sample into two parts: certain and uncertain. The loss based on the teaching assistant then becomes...

[0080] Then, the entropy of the teacher model's hidden representation H is calculated. Student model aligned with teacher model in R x For M layers with larger values, the loss for dynamic target alignment becomes...

[0081] Then, calculate the total loss. Update parameter set Ω;

[0082] Perform E-round fine-tuning using the above steps, repeating until the number of rounds ends;

[0083] Finally, the optimized student model is obtained.

[0084] 1. Introduction to the dataset

[0085] This invention primarily selected eight representative benchmark datasets for experiments, including five GLUE tasks and three other text classification tasks. The GLUE dataset is an important benchmark list in the field of natural language processing. The selected task types include paraphrasing similarity matching (PSM), sentiment classification (SC), and natural language inference (NLI). In addition, the three additional text classification tasks include AG News, Emotion, and IMDb. Table 2 provides detailed information about the datasets. Except for the MRPC task, which uses F1 score, the other tasks use accuracy as the evaluation metric.

[0086] Table 2 Introduction to the Benchmark Dataset

[0087] MRPC 3,668 408 1,725 SST-2 67,349 872 1,821 QNLI 104,743 5,463 5,463 MNL 392,702 9,832 9,847 AG News 120,000 - 7,600 QPP 363,849 40,430 390,965 Emotion 16,000 2,000 2,000 IMDb 20,000 5,000 25,000

[0088] For the PSM task, this invention uses the Microsoft Research Paragraph Corpus (MRPC) and Quora Question Pairs (QQP). Both MRPC and QQP datasets aim to determine whether two sentences in a question are semantically identical.

[0089] For the SC task, this invention selects the Stanford Sentiment Treebank (SST-2) dataset. The goal of SST-2 is to have the model predict the sentiment expressed in a given sentence.

[0090] For the NLI task, this invention employs Question Answering Natural Language Inference (QNLI) and Multi-Type Natural Language Inference (MNLI). QNLI is a dataset derived from the question answering dataset SQuAD, and its purpose is to predict whether the context sentence contains the answer to the question. MNLI is a corpus of sentence pairs extracted from multiple domains, used to evaluate the generality of the model.

[0091] For the three additional text classification tasks, AG News is a commonly used news text classification dataset. Its goal is to correctly classify news articles into their appropriate categories, thus testing the model's ability to understand and classify text across different categories. The Emotion dataset contains samples from various sources such as social media, news, and books, covering a wide range of emotion categories, including joy, sadness, anger, and fear. Each sample is typically a sentence or paragraph, which needs to be assigned to the correct emotion category based on the text's content and context. The Emotion dataset aims to help researchers develop and improve emotion recognition algorithms, enhancing the model's accuracy and generalization ability across different emotion categories. The IMDb dataset is a widely used movie review dataset. Its reviews come from user submissions on the IMDb (Internet Movie Database) website, covering various movie genres and styles. Each review is typically a paragraph consisting of one or more sentences, which needs to be labeled with positive or negative emotion based on the text's content and context. The IMDb dataset aims to provide a benchmark for emotion analysis tasks.

[0092] 2. Comparison Algorithm

[0093] This invention selects six representative comparison algorithms, including Vanilla KD (Hinton et al., 2015), BERT-PKD (Sun et al., 2019), DFA (Shen et al., 2019), CFL (Luo et al., 2019), UHC (Vongkulbhisal et al., 2019) and MUKI (Li et al., 2022).

[0094] Specifically, Vanilla KD primarily minimizes the original KL divergence loss, allowing the student model to mimic the soft-object logits output by the teacher model during training. To fully exploit the rich knowledge contained in the deep structure of the teacher models, the BERT-PKD method enables the student model to learn from the teacher models through a multi-layer distillation process. Among multiple trained teacher models, each specializing in a different classification problem, DFA attempts to learn a compact student model capable of handling complex classification problems. In other words, the classification problem solved by the student model is a superset of the problems solved by all teacher models. CFL maps the hidden representations of the teacher models to a common space, trains the student model by matching the mapped features with the features of the teacher models, and provides auxiliary supervised learning through soft-label logits combination. UHC partitions the soft-label logits of the student model into subsets based on the class set of the teacher models; each subset is trained to mimic the output of its corresponding teacher model. Based on model uncertainty, MUKI designed a model uncertainty-aware knowledge integration framework. The optimal supervision signal is approximated using two methods: one is to obtain the output of the most confident teacher model, and the other is to integrate the predicted outputs of different teachers based on the relative importance of the teacher models. Furthermore, for cases where teacher models obtain the same uncertainty score, the framework provides a reweighting strategy based on the uncertainty score to reduce interference from potentially contradictory supervision signals.

[0095] 3. Implementation details

[0096] This invention is based on the HuggingFace transformers library (Wolf et al., 2020). All experimental results were obtained from a single NVIDIA V100 GPU. First, a fine-tuned 12-layer BERT was used as the teacher model, and then distilled to obtain 6-layer and 9-layer teaching assistant models. The 6-layer teaching assistant model was configured for the 3-layer student model, and the 9-layer teaching assistant model was configured for the 6-layer student model. Then, the 3-layer and 6-layer student models were optimized using the method proposed in this invention to obtain the final student models. In the settings of this invention, the number of fine-tuning rounds was set to 3, the batch size to 32, the learning rate to 2e-5, and the distillation temperature to 5. Simultaneously, the loss balance coefficients λ1, λ2, and λ3 were set to 0.5, 0.3, and 0.2, respectively. The data selection rate r was set to 0.5. For the target alignment layer number M, M was set to 3 for the 3-layer student model and 6 for the 6-layer student model. All experiments were repeated five times, and the average results of the five experiments with different seeds were reported.

[0097] 4. Experimental Results

[0098] Table 3 shows the performance comparison between the present invention and the baseline model on various text classification tasks.

[0099] Table 3 Experimental results of the student model on the benchmark dataset.

[0100] BERT-base - 88.48 92.55 91.32 83.87 94.71 90.96 93.55 89.24 90.59 Vanilla KD 3-layer 75.90 79.66 78.15 71.22 60.11 81.32 53.91 79.25 72.44 BERT-PKD 3-layer 76.55 83.53 80.71 72.31 62.17 83.81 56.56 80.13 74.47 DFA 3-layer 75.47 82.61 79.33 71.01 63.23 80.57 54.63 79.36 73.28 CFL 3-layer 76.57 83.78 81.88 73.13 60.21 84.26 59.38 80.87 75.01 UHC 3-layer 78.57 84.86 82.35 73.76 75.67 84.68 64.72 81.35 78.25 MUKI 3-layer 80.99 85.67 83.05 74.37 88.19 85.81 72.91 82.93 81.74 DeKD(Ours) 3-layer 83.39 86.01 84.11 74.82 93.68 86.57 92.35 84.24 85.65 Vanilla KD 6-layer 80.21 81.37 79.86 72.33 62.58 82.37 55.71 80.33 74.35 BERT-PKD 6-layer 81.42 84.16 81.57 73.67 63.57 84.28 59.23 81.27 76.15 DFA 6-layer 81.12 83.36 80.76 72.81 65.46 82.24 58.62 80.22 75.57 CFL 6-layer 82.53 84.67 82.39 74.63 64.74 85.92 62.17 82.34 77.42 UHC 6-layer 83.72 85.74 83.62 75.67 78.19 86.71 68.16 83.66 80.68 MUKI 6-layer 84.97 86.29 85.16 77.54 90.63 87.35 79.64 84.51 84.51 DeKD(Ours) 6-layer 87.48 89.91 87.00 80.17 94.28 89.17 93.15 86.12 88.41

[0101] The following observations can be drawn from Table 3:

[0102] (1) It can be seen that the DeKD model of this invention performs best on all text classification tasks. This is because DeKD not only considers the effective learning of the student model at the data level, but also further improves distillation performance at both the model and target levels. Generally, DeKD's classification accuracy is about 0.34% to 19.44% higher than the best competitors. Apart from DeKD, none of these baseline models can stand out in all tasks simultaneously.

[0103] (2) The results of traditional knowledge distillation (Vanilla KD), intermediate representation-based knowledge distillation (BERT-PKD), and distillation methods considering multiple teacher models (DFA, CFL, UHC, and MUKI) are all consistent with expectations. Traditional distillation methods perform poorly on most tasks because they do not use additional supervision signals or intermediate representations. However, methods incorporating intermediate layer information outperform the original distillation methods. This is because intermediate representation-based methods encourage student models to extract knowledge from the earlier layers of the teacher model, rather than learning parameters from the last layer. In fact, this demonstrates that student models can increase their knowledge by learning multiple intermediate layers.

[0104] (3) Compared to MUKI, DFA, CFL, and UHC perform less well. Although DFA uses additional features to align the target, it fails to achieve satisfactory results because the supervision based on feature alignment is unstable, and the teacher features are fine-tuned for different semantic categories. CFL can learn a high-performance and lightweight student model, capable of mastering the ensemble knowledge of heterogeneous teacher models, but it faces the same problem as DFA: the supervision based on feature alignment is unstable. UHC outperforms DFA and CFL but not MUKI, indicating potential supervision conflicts, as the output of the student model in UHC is independent of the output of the teacher model, thus limiting its generalization across datasets. In summary, the above results indicate that simply combining different monitoring signals from multiple teacher models is ineffective. However, the present invention, DeKD, designs a more efficient framework that significantly improves the distillation performance of the student model by selecting information-rich data, using teaching assistant models to assist the evolution of the student model, and dynamically selecting alignment signals.

[0105] 5. Analysis of the contribution of each part

[0106] To evaluate the contributions of different parts of the DeKD method in this invention, corresponding ablation experiments were designed. The experimental results are shown in Table 4. Due to space limitations, only the results from the MRPC, SST-2, and IMDb datasets are presented. Other results are similar and therefore omitted. Three main variants were designed: with / o DS, with / o TA, and with / o DOA.

[0107] Table 4. Contribution results of each part of DeKD in this invention.

[0108] DeKD 87.48 89.91 86.12 w / o DS 86.84 87.84 85.52 w / o TA 86.74 87.61 85.50 w / o PRAYER 86.33 87.38 85.18

[0109] w / o DS: This variant removes the entropy-based selection strategy DS. Compared to DeKD, overall performance decreases without DS, meaning that selecting information-rich instances reduces redundant learning caused by data redundancy. This entropy-based selection strategy makes better use of limited queries.

[0110] w / o TA: Removing the teaching assistant (TA) model degrades the model's performance across all three tasks. This is because the auxiliary model, acting as an intermediate model, becomes a driving force in the evolution of the student model. Therefore, this variant performs worse than DeKD, indicating that adding the TA model can bridge the capability gap between the student and teacher models.

[0111] w / o DOA: This variant performs worse than DeKD when dynamic target alignment (DOA) is not considered. This means that learning through dynamic alignment of intermediate representations can help student models quickly understand the task, thereby improving the reliability of predictions.

[0112] Overall, DeKD outperforms the first three variants, demonstrating that all components together can improve the performance of the student model.

[0113] 6. Model Analysis

[0114] (1) Analysis of variance:

[0115] Table 5. Mean, Standard Deviation, and Significance Test ( 1 and 2 (This indicates a significant advantage over UHC and MUKI)

[0116] UHC 85.74,83.61,84.22,82.37,85.69 84.33 1.281 MUKI 86.29,82.16,85.34,84.68,86.12 84.92 1.495 <![CDATA[Ours 1,2 ]]> 89.91,89.88,89.65,89.17,88.96 89.51 0.383 IMDb Mean Stdev UHC 83.66,81.21,83.38,82.93,82.07 82.65 0.899 MUKI 84.51,83.23,84.47,83.13,82.98 83.66 0.679 <![CDATA[Ours 1,2 ]]> 86.12,86.08,86.02,85.85,85.93 86.00 0.099

[0117] Taking a 6-layer student model as an example, five experiments were conducted using different seeds, and their means and standard deviations (Stdev) were calculated. Furthermore, a two-sided statistical significance t-test (p-value) with a threshold of 0.05 was performed, and the two best baseline methods, UHC and MUKI, were compared with the present invention, DeKD. The experimental results are shown in Table 5. As shown in Table 5, the present invention has statistical significance compared to the baseline.

[0118] (2) Data selection strategy analysis:

[0119] In addition to the scaling-based entropy (SE) selection strategy, three other common strategies are implemented to compute the uncertainty score U for each instance x. x :

[0120] Random selects N×r instances randomly. The Random method is used as a baseline to evaluate the effectiveness of the selection strategy.

[0121] Least-Confdence (LC) indicates the model's confidence level in predicting the class. Uncertainty, U x Calculate as follows:

[0122]

[0123] Margin primarily calculates the first and second most likely categories. and The difference between them, U x The calculation is as follows:

[0124]

[0125] like Figure 2 As shown, the effectiveness of different strategies on the Emotion dataset was verified by varying the selection ratio r. Since the results on all datasets showed a consistent trend, one was chosen for analysis.

[0126] from Figure 2 We can observe the following: First, the selection strategy based on the student model's predictive entropy makes better use of limited queries than other strategies. Second, approximately 50% of the training data can be used to achieve full performance, indicating that about 50% of the data can adequately cover the training data, thus learning from these instances can sufficiently train the student model. Selecting information-rich instances helps reduce repetitive learning caused by data redundancy. Finally, there is a trade-off between performance and training cost: increasing the selection ratio generally improves the performance of the student model but leads to higher training costs.

[0127] (3) Analysis of Teaching Assistant Levels

[0128] Based on previous work (Yuan, F.; Shou, L.; Pei, J.; Lin, W.; Gong, M.; Fu, Y.; and Jiang, D. 2021. Reinforced Multi-Teacher Selection for Knowledge Distillation. In Third-Fifth AAAI Conference on Artificial Intelligence, 14284–14291.), this invention employs two configurations for the student model: 3 layers and 6 layers. The impact of different numbers of teaching assistant layers on the student model under these two configurations was explored. In the MRPC task, the number of teaching assistant layers in the 3-layer student model ranged from 4 to 11, while the number of teaching assistant layers in the 6-layer student model ranged from 7 to 11. Figure 3 In the study, the student model performed best when the number of teaching assistant layers was 6. Figure 4 In the study, the student model performed best when the number of teaching assistant layers was 9. This indicates that teaching assistant models, positioned between the teacher and student models, can address the issue of small student models failing to match the capabilities of large teacher models.

[0129] Key aspects of this invention include:

[0130] 1. This invention first designs an entropy-based selection strategy at the data level, which filters samples by scaling the entropy value to alleviate the problem of repetitive learning in student models, thereby achieving efficient knowledge distillation.

[0131] 2. At the model level, a teaching assistant model is introduced. During the distillation process, the student model can adaptively learn from either the teacher or the teaching assistant based on its own situation. For samples with a defined student model (i.e., low entropy), the student model learns from the teacher; for samples with an uncertain student model (i.e., high entropy), the student model learns from the teaching assistant, thereby mitigating the problem of performance degradation during distillation.

[0132] 3. At the goal level, a dynamic goal alignment strategy was designed. By dynamically selecting information-rich layers in the teacher model, the student model learns the supervision signals of these layers, thereby improving the effect of knowledge distillation at the goal level.

[0133] Another embodiment of the present invention provides a data-efficient knowledge distillation system based on dynamic target alignment using teaching assistants. The system comprises a student model, a teacher model, and a teaching assistant model. It employs an entropy-based strategy to select information-rich samples and inputs them into the student model for learning. It dynamically queries the teacher model or the teaching assistant model based on the evolution of the student model's capabilities during training. It selects information-rich layers from the teacher model based on entropy values, dynamically aligning the student model with the selected information-rich layers in the teacher model. Through the student model, teacher model, and teaching assistant model, knowledge is transferred from the teacher model to the student model, and the knowledge distillation performance improves as the student model's capabilities evolve.

[0134] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0135] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0136] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A data-efficient knowledge distillation method based on dynamic target alignment using teaching assistants, used for compressing large-scale pre-trained language models in natural language processing tasks, characterized in that... Includes the following steps: At the data layer, based on the benchmark dataset for natural language processing, an entropy-based strategy is used to select information-rich samples and input them into the student model for learning. At the model layer, a teaching assistant model is introduced, which dynamically queries the teacher model or teaching assistant model based on the evolution of the student model's capabilities during training. In the target layer, information-rich layers are selected from the teacher model based on the entropy value, so that the student model is dynamically aligned with the information-rich layers in the selected teacher model; Through student models, teacher models, and teaching assistant models, knowledge is transferred from the teacher model to the student model, and the knowledge distillation performance is improved as the capabilities of the student model evolve. The entire training process of the method includes: fine-tuning a large-scale pre-trained language model on a corresponding downstream task to obtain a teacher model, wherein the downstream task is a text classification task; distilling the teacher model to obtain a teaching assistant model; and training the final student model based on the total loss.

2. The method according to claim 1, characterized in that, The method of selecting information-rich samples based on entropy includes: Given a training batch One instance, for each instance ,use To represent the student model in category labels The output class probability distribution on, where This represents the softmax function. Student model The output soft-label logits vector; Use the scaling entropy of the probability distribution to calculate instances. Uncertainty score , The calculation method is as follows: in It is the number of category labels, based on the uncertainty score of the instance. Sort the instances and select only the top ones. Use an instance to query the teacher model, where The selection ratio refers to the number of instances to be queried; the selected instances have a high uncertainty score, indicating that they are information-rich instances that the student model should learn from.

3. The method according to claim 1, characterized in that, The method of dynamically querying the teacher model or teaching assistant model based on the evolution of the student model's capabilities during training includes: in the initial training phase, the student model learns from the teaching assistant model; as the student model becomes more powerful, it seeks more accurate supervision signals from the teacher model.

4. The method according to claim 3, characterized in that, The dynamic querying of the teacher model or teaching assistant model based on the evolution of the student model's capabilities during training includes: A batch of instances is ranked based on the prediction confidence of the student model, where the confidence is measured by entropy and calculated as follows: in, Indicates the confidence level. Represents entropy, The higher the value, the more uncertain the student model; The instances are divided into deterministic and indeterminate instances. For deterministic instances, the student model learns supervision signals from the teacher model, while the teaching assistant model provides soft labels for the indeterminate instances in the student model. The loss function is as follows: in, This represents the loss function based on teaching assistants. The KL divergence distance between the student model and the teacher model. This represents the KL divergence distance between the student model and the teaching assistant model.

5. The method according to claim 4, characterized in that, The step of selecting information-rich layers from the teacher model based on entropy values, and dynamically aligning the student model with the selected information-rich layers in the teacher model, includes: Computing the hidden representation of the teacher model Entropy: in, The larger the value, the more hidden information the corresponding layer contains, thus representing each layer... Sort by largest to smallest, select Higher values Layer; the loss function for dynamic target alignment is: in, This represents the number of layers in the student model. The student model represents the first The first layer and teacher model Align the layers. and Let represent the hidden representations of the student model and the teacher model, respectively. The student model represents the first Hidden representation of layers, Represents the first student model The first layer of the aligned teacher model Hidden representation of layers.

6. The method according to claim 5, characterized in that, The student model is trained by summing the three losses; the total loss is: in, It is the cross-entropy loss function. , and It is a hyperparameter used to adjust the loss weights; The entire training process includes: fine-tuning the BERT model on the corresponding downstream tasks to obtain the teacher model; distilling the teacher model to obtain the teaching assistant model; and training the final student model based on the total loss.

7. The method according to claim 6, characterized in that, The training process of the student model includes: Input training data Fine-tune the number of rounds The parameter set for training the student model ; Begin fine-tuning the student model, in each round of fine-tuning, for each batch , by uncertainty fraction Select samples with rich information; Based on confidence level The input samples are divided into two parts: certain and uncertain. The loss based on the teaching assistant is then... ; Computing the hidden representation of the teacher model entropy Student model aligned with teacher model Larger values At this layer, the loss for dynamic target alignment is ; Calculate total loss Update parameter set ; Follow the steps above. Fine-tune the model in rounds, repeating until the end of the rounds to obtain the optimized student model.

8. A data-efficient knowledge distillation system based on dynamic goal alignment using teaching assistants, characterized in that, The system, which implements the method of any one of claims 1 to 7, comprises a student model, a teacher model, and a teaching assistant model; it employs an entropy-based strategy to select information-rich samples and inputs them into the student model for learning; The system dynamically queries the teacher or teaching assistant model based on the evolution of the student model's capabilities during training; it selects information-rich layers from the teacher model based on entropy values, dynamically aligning the student model with the selected information-rich layers in the teacher model; and it transfers knowledge from the teacher model to the student model through the student model, teacher model, and teaching assistant model, improving knowledge distillation performance as the student model's capabilities evolve.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.

Images