Automatic teacher-student model matching method based on iterative pruning

Abstract

The invention relates to the technical field of knowledge distillation, and discloses a teacher-student model automatic matching method based on iterative pruning, comprising the following steps: S1, obtaining a pre-trained teacher model; s2, executing an iterative pruning process on the teacher model, wherein the iterative pruning process specifically comprises the steps of performing pruning operation on the teacher model; performing fine adjustment on the teacher model after pruning; storing the fine-tuned teacher model as a potential student model; judging whether a preset iteration termination condition is met or not, and if not, returning to the pruning operation; and S3, based on a preset scoring standard, selecting an optimal student model from the plurality of stored potential student models. By introducing a strategy based on iterative pruning, student models of different scales and performances can be automatically generated from a teacher model, and a student model most matched with the teacher model is selected from the student models by using a preset scoring standard; by means of the automatic generation and selection mode, the uncertainty of manual design is reduced, and high calculation overhead of a traditional NAS method is avoided.

Description

Technical Field

[0001] The present invention relates to the field of knowledge distillation technology, and in particular to an automatic teacher-student model matching method based on iterative pruning. Background Art

[0002] In recent years, deep learning technology has achieved remarkable success in image recognition, natural language processing and other fields. However, high-performance deep learning models usually have a large number of parameters and computational complexity, which severely limits their deployment and application on resource-constrained devices. In order to solve this contradiction, model compression technology has emerged, among which knowledge distillation has received widespread attention as an effective model compression method.

[0003] The core idea of ​​knowledge distillation is to transfer the knowledge of a large model with excellent performance, such as a teacher model, to a small model, such as a student model, so that the student model can be as close to the performance of the teacher model as possible while maintaining a small model size. Traditional knowledge distillation methods, such as FitNets and Distilling the Knowledge in a Neural Network, usually require manual pre-design of the network structure of the student model. This manual design method has obvious shortcomings:

[0004] Manual design is costly and inefficient: Engineers need to rely on a lot of experience and domain knowledge to repeatedly try different network structures, number of layers, parameter configurations, etc. to find a relatively suitable student model. This process is not only time-consuming and labor-intensive, but the results often rely on the subjective experience of the designer, and it is difficult to guarantee that the optimal structure is found. The fundamental reason is the lack of a systematic automated method to explore the architectural space of the student model.

[0005] There is a lot of uncertainty in the design, and it is difficult to find the architecture that best matches the teacher model: different teacher models may be suitable for different student model architectures, and it is difficult to exhaust all possible student model structures through manual design. Even if the designer is experienced, it is difficult to accurately predict which student model architecture is most suitable for the current specific teacher model, resulting in poor distillation results. For example, an overly simple student model may not be able to fully learn the complex knowledge of the teacher model, while an overly complex student model may introduce unnecessary parameters, increase the computational burden, and may overfit the training data. The root cause of this uncertainty lies in the lack of a quantitative evaluation standard and an automated search mechanism to determine the degree of match between the student model and the teacher model.

[0006] Automated methods based on neural architecture search (NAS) have the problem of high computational cost: In order to improve the efficiency and performance of knowledge distillation, researchers have also tried some automated methods, such as methods based on neural architecture search (NAS) to search the structure of the student model. However, these methods usually need to search the network structure from scratch, which is computationally expensive and requires a lot of computing resources and time for model search. In addition, traditional NAS methods usually focus on searching for a completely new network architecture, while ignoring the use of the structural information of the teacher model itself for pruning optimization, which to some extent limits its application in knowledge distillation scenarios. Its drawback is that the search space is huge and the prior knowledge of the teacher model is not fully utilized.

[0007] Therefore, how to efficiently and effectively generate a student model that matches the teacher model has become a key issue that needs to be solved in the field of knowledge distillation. When solving similar problems, existing technologies either rely on inefficient and uncertain manual design or face high computational costs and fail to fully utilize the structural information of the teacher model. Summary of the invention

[0008] The present invention provides an automatic teacher-student model matching method based on iterative pruning. By introducing a strategy based on iterative pruning, it can automatically generate a series of student models of different scales and performances from the teacher model, and select the student model that best matches the teacher model using a preset scoring standard. The automated generation and selection method significantly reduces the cost and uncertainty of manual design, avoids the high computational overhead of traditional NAS methods, and provides a more efficient and effective method for knowledge distillation.

[0009] The present invention provides a teacher-student model automatic matching method based on iterative pruning, comprising the following steps:

[0010] S1: Get the pre-trained teacher model;

[0011] S2: Perform an iterative pruning process on the teacher model, specifically including the following sub-steps:

[0012] Prune the teacher model;

[0013] Fine-tune the pruned teacher model;

[0014] Store the fine-tuned teacher model as a potential student model;

[0015] Determine whether the preset iteration termination condition is met. If not, return to the pruning operation;

[0016] S3: Based on a preset scoring criterion, the best student model is selected from several stored potential student models.

[0017] Preferably, in step S1, the teacher model is a deep learning model.

[0018] Preferably, in step S2, the pruning operation on the teacher model includes: using L1-norm as the filter importance evaluation criterion, pruning each layer of the teacher model according to a preset pruning rate, and obtaining a pruned teacher model.

[0019] Preferably, the pruned teacher model is specifically:

[0020] Given a CNN with L convolutional layers, and A = (C1, C2, ..., C L ) is the original network, where C1 is the number of channels in the first layer; before iterative pruning, manually determine the pruning rate of each layer and the maximum pruning rate of the model; during global iterative pruning, determine whether the current pruning rate is less than the maximum pruning rate of the model. If so, use L1-norm to calculate the importance of each channel layer by layer, and perform global pruning according to the pruning rate of each layer; otherwise, end pruning.

[0021] Preferably, in step S2, fine-tuning the pruned teacher model includes: using the original model as the teacher model and the pruned model as the student model, using knowledge distillation to perform fine-tuning to restore the accuracy loss caused by pruning, and obtaining the fine-tuned teacher model.

[0022] Preferably, the knowledge distillation includes:

[0023] Calculate the distillation loss of intermediate features, the distillation loss of soft labels, and the distillation loss of hard labels;

[0024] The intermediate feature loss, soft label loss and hard label loss are weightedly combined to obtain the total loss, and fine-tuning is performed based on the total loss.

[0025] Preferably, the intermediate feature loss is specifically:

[0026] An adaptive layer consisting of a point convolution and a batch normalization layer is introduced to map the channels of the student model to the corresponding channels of the teacher model;

[0027] Calculate the distance between the corresponding feature maps of the student model and the teacher model as the intermediate feature loss;

[0028] The soft label loss is specifically:

[0029] Use the divergence loss between the student model and the teacher model output as the distillation loss for output simulation;

[0030] Use the temperature parameter T to soften the output between the student model and the teacher model;

[0031] The distillation loss of the hard tag is specifically:

[0032] The cross entropy function is used to calculate the difference between the output of the student model and the hard label.

[0033] Preferably, the preset iteration termination condition is specifically: the current pruning rate reaches a preset maximum pruning rate.

[0034] Preferably, the selecting the best student model from a plurality of stored potential student models based on a preset scoring standard comprises:

[0035] Normalize the accuracy and memory usage of each potential student model;

[0036] Use weighted scores to evaluate how well each student model matches the teacher model; the weights are the accuracy score and the memory usage score;

[0037] The best student model is selected based on the highest match.

[0038] Preferably, the accuracy score includes:

[0039] Map the accuracy of the student model on the test dataset to the interval [0,1] as the accuracy score.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention discloses an automatic teacher-student model matching method based on iterative pruning. By introducing a strategy based on iterative pruning, a series of student models of different scales and performances can be automatically generated from the teacher model, and the student model that best matches the teacher model can be selected from them using a preset scoring standard. The automated generation and selection method significantly reduces the cost and uncertainty of manual design, avoids the high computational overhead of traditional NAS methods, and provides a more efficient and effective method for knowledge distillation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of an automatic teacher-student model matching method based on iterative pruning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] like Figure 1 As shown, the present application provides a teacher-student model automatic matching method based on iterative pruning, comprising the following steps:

[0045] S1: Get the pre-trained teacher model;

[0046] S2: Perform an iterative pruning process on the teacher model, specifically including the following sub-steps:

[0047] Prune the teacher model;

[0048] Fine-tune the pruned teacher model;

[0049] Store the fine-tuned teacher model as a potential student model;

[0050] Determine whether the preset iteration termination condition is met. If not, return to the pruning operation;

[0051] S3: Based on a preset scoring criterion, the best student model is selected from several stored potential student models.

[0052] In the above scheme, the present application first obtains a pre-trained teacher model. Subsequently, an iterative pruning process is performed on the teacher model. The iterative pruning process includes multiple loops. In each loop, the current model is first pruned to reduce the model size, and then the pruned model is fine-tuned to restore the performance and transfer the knowledge of the teacher model. The fine-tuned model is regarded as a potential student model and stored. Afterwards, it is determined whether the preset iteration termination condition is met, such as reaching the preset pruning rate or number of iterations. If the termination condition is not met, the pruning operation is returned to continue the next round of iteration. When the iteration termination condition is met, the iterative pruning process ends, and multiple potential student models with different model sizes and performances have been stored. Finally, based on the preset scoring criteria, such as comprehensive consideration of the model accuracy and model size, a student model that best matches the teacher model is selected from the stored multiple potential student models.

[0053] Preferably, in step S1, the teacher model is a deep learning model.

[0054] In the above scheme, a high-performance knowledge source is provided for the subsequent pruning and knowledge distillation process. The teacher model is usually a deep learning model with a large number of parameters and excellent performance. Its pre-training means that the model has learned rich knowledge and feature representation. This step provides an operation object for the subsequent iterative pruning process.

[0055] Preferably, in step S2, the pruning operation on the teacher model includes: using L1-norm as the filter importance evaluation criterion, pruning each layer of the teacher model according to a preset pruning rate, and obtaining a pruned teacher model.

[0056] Preferably, the pruned teacher model is specifically:

[0057] Given a CNN with L convolutional layers, and A = (C1, C2, ..., C L ) is the original network, where C1 is the number of channels in the first layer; before iterative pruning, manually determine the pruning rate of each layer and the maximum pruning rate of the model; during global iterative pruning, determine whether the current pruning rate is less than the maximum pruning rate of the model. If so, use L1-norm to calculate the importance of each channel layer by layer, and perform global pruning according to the pruning rate of each layer; otherwise, end pruning.

[0058] For the kth filter in the convolutional layer, its importance is calculated by the L1 norm:

[0059]

[0060] Among them, C in is the number of input channels, K h ×K w is the convolution kernel size, W k,i,j Represents the weight value of the k-th filter at the i-th input channel and the j-th spatial position.

[0061] In the above scheme, the input is the current model (the teacher model initially, and the model fine-tuned in the previous iteration in subsequent iterations). The pruning operation aims to reduce the size of the model, reduce its computational complexity and the number of parameters. There are many specific methods for pruning, such as pruning based on weight amplitude, gradient information or activation value. The pruning operation directly affects the difficulty of subsequent fine-tuning and the performance of the potential student model. The pruned model structure becomes sparse and the number of parameters is reduced.

[0062] Preferably, in step S2, fine-tuning the pruned teacher model includes: using the original model as the teacher model and the pruned model as the student model, using knowledge distillation to perform fine-tuning to restore the accuracy loss caused by pruning, and obtaining the fine-tuned teacher model.

[0063] The model has L layers in total, and each layer has an independent preset pruning rate:

[0064]

[0065] in, is the initial pruning rate of the lth layer, μ( l) is the attenuation coefficient of the lth layer, which can control the attenuation speed of the pruning rate of each layer.

[0066] The cumulative pruning rate of the lth layer after t rounds of iterations:

[0067]

[0068] Global model cumulative pruning rate:

[0069]

[0070] Among them, Params (l) is the parameter quantity of the lth layer.

[0071] The iteration termination condition is as follows: reaching the target pruning rate.

[0072] GlobalPruneRate t ≥MaxGlobalPruneRate

[0073] Preferably, the knowledge distillation includes:

[0074] Calculate the distillation loss of intermediate features, the distillation loss of soft labels, and the distillation loss of hard labels;

[0075] The intermediate feature loss, soft label loss and hard label loss are weightedly combined to obtain the total loss, and fine-tuning is performed based on the total loss.

[0076] In the above scheme, the original teacher model is used as the teacher and the pruned model is used as the student. The knowledge distillation method is used for fine-tuning to recover the accuracy loss caused by pruning. The fine-tuning process adopts a multi-level knowledge distillation strategy.

[0077] After the model is pruned, in order to restore its accuracy, the fine-tuning step is particularly critical. The fine-tuning process enables the pruned model to update its parameters to adapt to changes in the model structure and provide guidance during the pruning process. Traditional fine-tuning methods mainly rely on the training strategy of the original model, that is, the use of hard labels. However, compared to hard labels, soft labels contain richer information and can more effectively guide the fine-tuning process. Therefore, this application uses the unpruned model as the teacher model, and the pruned unfine-tuned model as the student model, forming a natural teacher-student structure. The advantage of this structure is that there is no need to find a teacher model with good performance. Instead, the soft labels of the teacher model are directly used to guide the fine-tuning of the student model in order to restore the model accuracy.

[0078] As pruning progresses, the soft labels of the teacher model may not be sufficient to effectively guide the shallow network. To overcome this challenge, this application introduces a knowledge distillation method for intermediate features. Specifically, the unpruned part of the teacher model is used as the teacher, and the pruned part is used as the student, so that the student block can absorb more intermediate feature knowledge. By combining the guidance of soft labels, this method helps to achieve better performance recovery.

[0079] Preferably, the intermediate feature loss is specifically:

[0080] An adaptive layer consisting of a point convolution and a batch normalization layer is introduced to map the channels of the student model to the corresponding channels of the teacher model;

[0081] Calculate the distance between the corresponding feature maps of the student model and the teacher model as the intermediate feature loss;

[0082] In the above scheme, during the pruning process of the student model, the structured pruning strategy is used to remove channels with smaller contributions, resulting in inconsistency in the number of channels between the student model and the teacher model. To solve this problem, this application introduces an adaptive layer consisting of a point convolution (1×1 convolution kernel) and a batch normalization (BN) layer. The function of this adaptive layer is to map the channels of the student model to the corresponding channels of the teacher model, thereby transferring knowledge more efficiently and reducing the difference in feature mapping between the pruned model and the original model. Through this mapping strategy, it can be ensured that the structural changes in the student model will not have a negative impact on the effect of knowledge distillation. The distillation loss of a single block is expressed as follows:

[0083]

[0084] in, Represented as the feature map of the student model, is the feature map of its corresponding teacher model A; r(·) is a regression variable composed of a 1×1 convolutional layer and a BN layer; D p It is a measure of the L2 distance between the student and teacher feature maps; the model intermediate feature distillation loss is expressed as:

[0085]

[0086] where B is the number of pruned blocks; this loss makes it possible for the student model to effectively learn the features of the teacher model during the structure pruning process.

[0087] The soft label loss is specifically:

[0088] Use the divergence loss between the student model and the teacher model output as the distillation loss for output simulation;

[0089] Use the temperature parameter T to soften the output between the student model and the teacher model;

[0090] In the above scheme, the present application also includes simulating the output soft labels of distillation learning. In order to learn more from the teacher model, it is also necessary to simulate the softened teacher output. The present application uses the KL divergence loss between the student and teacher outputs as the distillation loss for output simulation. The temperature T softens the output between the student and the teacher. This method enables the student model to more effectively learn the prediction results of the high-performance teacher model, thereby significantly reducing the classification error rate. The softened output and multi-teacher soft label distillation loss formula are as follows:

[0091]

[0092] where x ij represents the student model logical output of the jth class of the i-th batch of samples; X ij and They respectively represent the soft outputs of the student model and teacher model A of the jth class of the i-th batch of samples; the temperature hyperparameter T determines the degree of softening of the output. It can also be calculated.

[0093] The distillation loss of the hard tag is specifically:

[0094] The cross entropy function is used to calculate the difference between the output of the student model and the hard label.

[0095] In the above scheme, each student model uses a classic cross entropy function with hard labels and student output logarithms, which helps the model better learn the given dataset, as shown in the following formula,

[0096]

[0097] in, Represents the logical output of the jth class of the i-th batch of samples; Y ij represents the hard label of the jth category of the i-th batch of samples.

[0098] In the above scheme, the multi-teacher multi-level knowledge distillation proposed in this application is a weighted combination of the above three losses. The parameters of the student network are updated only in the training phase to better restore the accuracy of the pruned model. The mathematical expression is as follows:

[0099] L=δl inter +εl output +θl CE (δ+ε+θ=1)

[0100] Among them, δ, ε, and θ are the weight values ​​representing the intermediate feature loss, soft label loss, and hard label loss, respectively.

[0101] We dynamically update the values ​​of δ, ε, and θ, and determine their values ​​based on how fast the corresponding losses change.

[0102] Assume that the intermediate feature loss, soft label loss and hard label loss of the tth round are respectively: And the update is completed, and the losses of each part in the t+1 round are: Then the change in each loss in round t+1 is calculated as:

[0103]

[0104] Then, the formula for updating the weight is as follows:

[0105]

[0106] Finally, the latest weight values ​​are used to calculate the total loss of the t+1th round of training.

[0107] Preferably, the preset iteration termination condition is specifically: the current pruning rate reaches a preset maximum pruning rate.

[0108] Preferably, the selecting the best student model from a plurality of stored potential student models based on a preset scoring standard comprises:

[0109] Normalize the accuracy and memory usage of each potential student model;

[0110] Use weighted scores to evaluate how well each student model matches the teacher model; the weights are the accuracy score and the memory usage score;

[0111] The best student model is selected based on the highest match.

[0112] In the above scheme, when the iterative pruning process is completed, the student model list will contain a series of models with their own characteristics. These models achieve a diverse balance between model size and performance. Since different tasks have different requirements for models, the original large model may not be suitable for certain specific scenarios. Therefore, it is crucial to select a student model that can replace the original model to adapt to specific tasks. This application constructs a customized model scoring system to accurately screen out the most suitable student model to replace the original model. The scoring criteria comprehensively consider the accuracy of the model and the size of the model to ensure that the selected model can meet the performance requirements and achieve the optimal allocation of resources in practical applications. By conducting a comprehensive and detailed evaluation of each model, the most suitable model can be deployed for each task, so as to ensure efficient operation while meeting specific performance indicators.

[0113] In addition, the scoring system of this application is flexible and scalable, and can be adjusted according to different application scenarios and requirements. This means that as new tasks emerge or the requirements of existing tasks change, the scoring criteria of this application can be updated accordingly to ensure that the best matching model can always be selected. In this way, not only the efficiency and effectiveness of model deployment are improved, but also a solid foundation is provided for the continuous optimization and iteration of the model.

[0114] Preferably, the accuracy score includes:

[0115] Map the accuracy of the student model on the test dataset to the interval [0,1] as the accuracy score.

[0116] In the above scheme, it is assumed that there are n student models, denoted as M1, M2, ..., M n , use M teacher Represents the teacher model. For each model, there are corresponding indicators to describe its characteristics.

[0117] Model Accuracy Macc. represents the student model Mj (j=1,2,...,n), Represents the accuracy of the teacher model on the corresponding test data set, usually expressed as a percentage and then converted to a decimal for easy calculation.

[0118] Model memory usage Mmem. represents the student model Mj (j=1,2,...,n), Represents the memory size of the teacher model at runtime, in bytes, which depends on factors such as the number of model parameters and the amount of intermediate data generated at runtime.

[0119] In order to select the student model that best matches the teacher model, it is necessary to consider the above indicators comprehensively and construct a scoring formula by assigning weights to each indicator to measure the degree of match between the student model and the teacher model.

[0120] Since the standards of various indicators are different, in order to perform comprehensive calculations, it is necessary to first standardize each indicator and map them all to the [0,1] interval.

[0121] The model accuracy is normalized as follows:

[0122]

[0123] The model memory usage is normalized as follows:

[0124]

[0125] The specific scoring formula for judging the matching degree between the teacher model and the student model is as follows:

[0126]

[0127] Among them, score j Model M for students j Rating, Score the model's memory usage. is the model accuracy score. In addition, α and β represent the weight values ​​of the model memory usage score and the model accuracy score, respectively, and satisfy α+β=1.

[0128] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A teacher-student model automatic matching method based on iterative pruning, characterized in that: The following steps are involved: S1: Get the pre-trained teacher model; S2: Perform an iterative pruning process on the teacher model, specifically including the following sub-steps: Prune the teacher model; Fine-tune the pruned teacher model; Store the fine-tuned teacher model as a potential student model; Determine whether the preset iteration termination condition is met. If not, return to the pruning operation; S3: Based on a preset scoring criterion, the best student model is selected from several stored potential student models.

2. According to claim 1, a teacher-student model automatic matching method based on iterative pruning is characterized in that: In step S1, the teacher model is a deep learning model.

3. The automatic teacher-student model matching method based on iterative pruning according to claim 2 is characterized in that: In step S2, the pruning operation on the teacher model includes: using L1-norm as the filter importance evaluation criterion, pruning each layer of the teacher model according to a preset pruning rate, and obtaining a pruned teacher model.

4. The automatic teacher-student model matching method based on iterative pruning according to claim 3 is characterized in that: The pruned teacher model is specifically: Given a CNN with L convolutional layers, and A = (C1, C2, ..., C L ) is the original network, where C1 is the number of channels in the first layer; before iterative pruning, manually determine the pruning rate of each layer and the maximum pruning rate of the model; during global iterative pruning, determine whether the current pruning rate is less than the maximum pruning rate of the model. If so, use L1-norm to calculate the importance of each channel layer by layer, and perform global pruning according to the pruning rate of each layer; otherwise, end pruning.

5. The automatic teacher-student model matching method based on iterative pruning according to claim 4 is characterized in that: In step S2, fine-tuning the pruned teacher model includes: using the original model as the teacher model and the pruned model as the student model, using knowledge distillation to perform fine-tuning to restore the accuracy loss caused by pruning, and obtaining a fine-tuned teacher model.

6. The automatic teacher-student model matching method based on iterative pruning according to claim 5, characterized in that: The knowledge distillation includes: Calculate the distillation loss of intermediate features, the distillation loss of soft labels, and the distillation loss of hard labels; The intermediate feature loss, soft label loss and hard label loss are weightedly combined to obtain the total loss, and fine-tuning is performed based on the total loss.

7. The automatic teacher-student model matching method based on iterative pruning according to claim 6, characterized in that: The intermediate feature loss is specifically: An adaptive layer consisting of a point convolution and a batch normalization layer is introduced to map the channels of the student model to the corresponding channels of the teacher model; Calculate the distance between the corresponding feature maps of the student model and the teacher model as the intermediate feature loss; The soft label loss is specifically: Use the divergence loss between the student model and the teacher model output as the distillation loss for output simulation; Use the temperature parameter T to soften the output between the student model and the teacher model; The distillation loss of the hard tag is specifically: The cross entropy function is used to calculate the difference between the output of the student model and the hard label.

8. The automatic teacher-student model matching method based on iterative pruning according to claim 7, characterized in that: The preset iteration termination condition is specifically: the current pruning rate reaches a preset maximum pruning rate.

9. The automatic teacher-student model matching method based on iterative pruning according to claim 8, characterized in that: The step of selecting the best student model from a plurality of stored potential student models based on a preset scoring standard includes: Normalize the accuracy and memory usage of each potential student model; Use weighted scores to evaluate how well each student model matches the teacher model; the weights are the accuracy score and the memory usage score; The best student model is selected based on the highest match.

10. The automatic teacher-student model matching method based on iterative pruning according to claim 9, characterized in that: The accuracy score includes: Map the accuracy of the student model on the test dataset to the interval [0,1] as the accuracy score.

Images