A knowledge distillation method based on a dynamic multi-teacher model and a structured relationship

Abstract

A knowledge distillation method based on a dynamic multi-teacher model and structured relationships is presented. The main content is a novel knowledge distillation method that trains a student model using multiple teacher models and dynamically assigns weights to the teacher models based on their performance to guide the student model's training. Simultaneously, structured relationships are introduced to assist training, allowing the student model to learn the spatial relationships of the teacher models' sample outputs, thereby improving training effectiveness. Compared to traditional multi-teacher models, this invention addresses the limitations of single-teacher learning, blind learning, and average learning. By dynamically adjusting learning weights based on the quality of the teacher models, it achieves better training results. Furthermore, by introducing spatial relationships of samples as knowledge for learning, it overcomes the monotony of traditional knowledge distillation methods that rely solely on response learning, enabling the student model to learn richer knowledge and improving its performance.

Description

Technical Field

[0001] This invention belongs to the field of neural networks in machine learning, and particularly relates to a knowledge distillation method based on a dynamic multi-teacher model and structured relationships. Background Technology

[0002] In machine learning methods, model compression plays a crucial role, enabling a model to perform close to the original model with fewer parameters. Generally, without overfitting, model performance shows a certain degree of positive correlation with the number of parameters. However, this doesn't mean that more parameters always equate to better performance. This is because model performance gradually reaches a bottleneck as the number of parameters increases. Figure 1 As shown, although the final model performance will tend to reach a bottleneck, compared with other model compression methods (curve a), knowledge distillation can achieve more significant training effects by acquiring knowledge (curve b), and shows better results with fewer model parameters.

[0003] Knowledge distillation is a model compression method based on a teacher-student architecture. It trains a relatively simple student model with small parameters using a pre-trained, relatively complex teacher model. This allows the student model to learn the implicit knowledge of the complex teacher model and transfer this knowledge to its own model. Consequently, the student model with smaller parameters can exhibit powerful reasoning capabilities close to those of the teacher model on a restricted machine, achieving the goal of model compression.

[0004] Knowledge distillation can be optimized in many ways, resulting in numerous variations. These variations further improve upon traditional knowledge distillation, leading to improvements in accuracy, convergence speed, and parameter count. Currently, mainstream optimization schemes include multi-teacher distillation, self-distillation, and semi-supervised distillation. This paper focuses primarily on multi-teacher distillation and its derivative, multi-teacher weighted adaptive distillation.

[0005] Multi-teacher distillation, compared to traditional knowledge distillation, uses multiple pre-trained teacher models to guide student models. Compared to traditional single-teacher models, multi-teacher models offer better robustness and accuracy, avoiding the occasional biases of a single teacher. This improves the accuracy and convergence speed of the student model, resulting in better model compression. Multi-teacher models themselves have many classifications, such as the common average teacher model, and the focus of this paper, multi-teacher adaptive weight distillation.

[0006] The key breakthrough of multi-teacher adaptive weight distillation lies in its dynamic setting of teacher weights, compared to ordinary multi-teacher average distillation. This allows teachers with higher output accuracy to have higher weights, enabling the student model to learn more from these teachers in the current iteration. Because it learns more accurate knowledge, the student model naturally converges faster and improves its accuracy. Summary of the Invention

[0007] The goal of this invention is to achieve model compression using knowledge distillation, and to alleviate two types of distortion problems that occur during the compression process by introducing dynamic weights and structured relationships: first, different teacher models have different training effects and varying training levels; second, considering only the label output without considering the sample structure relationships leads to training distortion. Through this approach, we aim to achieve a student model with better performance.

[0008] The idea behind this invention is as follows: Teacher model weights are dynamically calculated using cosine similarity, allowing higher-performing teacher models to receive more training weights. Based on this, a structured relationship is introduced to calculate the structural differences between the teacher model and the student model across multiple samples, using these structural differences to guide the student model. Finally, gradient descent is performed using the teacher model loss, the structured loss, and the true labels to train the student model.

[0009] To address the aforementioned technical problems, the present invention provides a specific technical solution for a knowledge distillation method based on a dynamic multi-teacher model and structured relationships, as follows:

[0010] Step 1: Obtain the image dataset, which contains N samples. Each sample is represented as follows: , where x i A sample is represented as a multidimensional tensor. in This represents a multidimensional tensor, where h and w represent the dimensions, and c represents the number of image channels; y i The sample labels are used to annotate the results of the samples; then the dataset is randomly divided into training and test sets according to the proportion.

[0011] Step 2: Data preprocessing, preprocessing the samples in the dataset without changing the sample labels;

[0012] Step 3: Use the dataset to train and test K different teacher models to obtain the teacher model that meets the performance requirements of the current task, and perform performance testing on the teacher model to obtain the teacher model performance parameters;

[0013] Step 4: Design the student network and train the student model;

[0014] Step 5: Multi-Teacher Weight Calculation Stage; First, each iteration requires inputting the training set into multiple teacher networks to obtain the results of the teacher networks; second, the output of the teacher models needs to be evaluated by calculating the similarity between the teacher outputs and the sample label results, using cosine similarity:

[0015]

[0016] in, This represents the output of the k-th teacher model. and sample label results y t The similarity between them is calculated as follows: k represents the k-th teacher model, i represents the comparison dimension between the teacher model output and the sample label (totaling n dimensions), and j represents the j-th teacher among all teacher models. Weights are assigned to the k-th teacher model based on similarity.

[0017] Step 6: Based on the calculated weights, guide the multi-teacher model to perform knowledge distillation on the student model. Use KL divergence to generate the loss between the student and teacher models, then weight them together to obtain the final response-based dynamic multi-teacher loss.

[0018]

[0019] in, Represents the teacher model weights, o S This represents the output of the student model.

[0020] Step 7: Structured Relationship Calculation Stage; The structured relationships are calculated based on positional distance structured relationships and angular structured relationships; First, the Manhattan distance is used to calculate the distance structured relationships. Within a sample set, the Manhattan distance of the last layer features output by the model is calculated pairwise, and then summed.

[0021]

[0022] Where d(x,y) represents the Manhattan distance between samples. This represents the i-th dimension vector value of sample n. This represents the i-th dimension vector value of sample m;

[0023] The following activation function is used to prevent the Manhattan distance from becoming too large:

[0024]

[0025] Among them, L SRD Denotes distance structural loss, d t d represents the Manhattan distance generated by the teacher model. sThe Manhattan distance generated for the student model, where γ is a hyperparameter that controls values ​​exceeding a threshold.

[0026] Then, the angle-based structured relationships are calculated using the output of the last layer of the model. Within a sample set, the inner product is calculated pairwise using normalized vector differences, as follows:

[0027] a(t i ,t j ,t k )= <e ij ,e kj >

[0028]

[0029] Where a(t) i ,t j ,t k ) represents the angular feature value, t represents the sample, and three samples are used for calculation each time, e ij Indicate l i vector and l j The difference between vectors, e kj Similarly, vectors are in 2-norm form.

[0030] Similarly, an activation function is used to prevent excessive loss:

[0031]

[0032] Among them, L SRA Indicates angular structural loss, a t and a s These represent the angular feature values ​​of the teacher model and the student model, respectively.

[0033] Final structured loss The sum of distance structure loss and angle structure loss:

[0034]

[0035] Where α1 and β1 are hyperparameters;

[0036] Step 8: Training Optimization; Optimization is performed using multi-teacher dynamic weight loss, structured loss, and hard label loss of the model output. The total loss is calculated as follows:

[0037]

[0038] in Let represent the total loss, and α² and β² be hyperparameters. The loss between the model output and the true label is calculated as follows:

[0039]

[0040] Step 9: Backward inference of the model; calculate the backward gradient based on the loss;

[0041] Step 10: Repeat the above process until the final number of iterations is reached;

[0042] Step 11: Generating model classification results; After completing the above training using the training set, use the validation set to output the student model, and compare the output results with the true labels.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. This invention effectively solves the problem of single-teacher learning's limited knowledge by using multi-teacher dynamic weights, while also avoiding the mediocrity problem caused by conventional multi-teacher averaging. This allows each teacher to provide dynamic guidance to students based on their output, thus enabling the student model to better absorb the learning outcomes from multiple teachers.

[0045] 2. This invention uses structured relationships to assist the student model in learning. The student model no longer simply learns sample responses, that is, the content output by the sample model, but learns more about the spatial relationships between samples, such as distance and angle, enabling the student model to learn richer content and thus produce better learning results. Attached Figure Description

[0046] Figure 1 This diagram illustrates the relationship between model parameters and performance.

[0047] Figure 2 Graph of weight allocation strategy;

[0048] Figure 3 This is a schematic diagram of the structure of the present invention. Detailed Implementation

[0049] To better understand the purpose, structure, and function of this invention, the following detailed description of a knowledge distillation method based on a dynamic multi-teacher model and structured relationships is provided in conjunction with the accompanying drawings.

[0050] Based on the above-mentioned inventive concept, this invention proposes a knowledge distillation model based on structured relationships and dynamic teachers. First, the specific requirements of the knowledge distillation task are defined as follows:

[0051] This invention employs an image recognition model using the CIFAR100 training set. This study will also extensively utilize normalization functions, primarily used for classification tasks, to normalize the model output to a range of 0-1. In knowledge distillation, temperature τ will also be used for processing, as follows:

[0052]

[0053] Among them, σ(o c ) represents the output of the normalized function. Indicates the model's target classification output, o j The output of other classifications in the model is represented by τ, which represents the distillation temperature. τ is used for smoothing to highlight the importance of hidden knowledge, i.e., the knowledge of other classification outputs.

[0054] The main method employed in this invention is knowledge distillation, a model compression method that uses a mature, large model to train and guide a smaller model, enabling the smaller model to exhibit better performance in the given task. The main methods are as follows:

[0055]

[0056] in The soft target loss represents the student model output q. j and teacher model output The losses between The hard loss is represented by the student model output q. j and real label c j The losses between This represents the final model loss. The balance between the two types of losses is achieved by adjusting the hyperparameters α and β.

[0057] This study introduces a multi-teacher knowledge distillation method based on traditional knowledge distillation. Multi-teacher knowledge distillation allows the student model to learn from multiple models simultaneously, thereby improving learning efficiency and increasing robustness and regularity. Furthermore, we dynamically assign different weights to teacher models, giving those with better learning performance greater dynamic teaching weights to better guide the student model. In addition, we introduce a structured approach to enhance the distillation effect, allowing the student model to achieve better distillation results. Learning is performed by calculating the structured relationships between multiple samples, and an optimized activation function is proposed to achieve better results.

[0058] The specific methods of the method model will now be described in detail, such as... Figure 3 As shown, it includes the following steps:

[0059] S1, Collect the dataset:

[0060] This invention uses the CIFAR-100 dataset. CIFAR-100 (Canadian Institute for Advanced Research-100) is a commonly used computer vision dataset for image classification tasks. It is an extended version of the CIFAR-10 dataset, containing more categories and a wider variety of images. The CIFAR-100 dataset has 100 classes. Each class has 600 color images of size 32×32 pixels, randomly divided into training and test sets at a 5:1 ratio; each sample is represented as: x i Represented as a sample image, it is composed of multidimensional tensors, and is represented as... in This represents a multidimensional tensor, where h and w represent the dimension, c represents the number of image channels, and y represents the number of channels. i This represents the sample label, indicating the results of labeling the samples. N is the number of samples. For the sample.

[0061] S2, Data Preprocessing:

[0062] The main operations performed on the images included random cropping and horizontal flipping to increase data diversity without changing the image labels. Simultaneously, image standardization was applied, normalizing the pixel values ​​by channel. The mean and standard deviation are provided here, corresponding to the mean and standard deviation of the three RGB channels, respectively. Standardization makes the pixel value distribution of each channel closer to a standard normal distribution, which is helpful for model training and optimization.

[0063] S3 involves training and testing multiple different teacher models using a dataset to obtain a teacher model that meets the performance requirements for the current task, and then performing performance tests on the teacher model to obtain its performance parameters. The specific steps are as follows:

[0064] S31, Design a teacher network, input the training set into multiple different teacher networks, and obtain the corresponding outputs;

[0065] S32, adopt a gradient descent strategy for the difference loss function, update the model parameters, repeat the above several iterations (e poch), and complete the training of the teacher model;

[0066] S33, switch to test mode, input the test set into the teacher model, and record relevant parameters of the teacher model, including accuracy, convergence speed, etc.

[0067] S34, Repeat the above steps to obtain multiple teacher models;

[0068] S4. Design a student network and train a student model. The student network has fewer network layers and requires fewer parameters.

[0069] S5, Multi-Teacher Weight Calculation Stage; First, each iteration requires inputting the training set into multiple teacher networks to obtain the results of the teacher networks. Second, the output of the teacher models needs to be evaluated by calculating the similarity between the teacher output and the labeled results (ground truth), using cosine similarity:

[0070]

[0071]

[0072] in, This represents the output of the k-th teacher model. and sample label results y t The similarity between them is defined as follows: k represents the k-th teacher model, i represents the comparison dimension between the teacher model output and the sample label (totaling n dimensions), and j represents the j-th teacher among all teacher models, requiring normalization of all teacher models. Weights are assigned to the k-th teacher model based on similarity. The evaluation result is the weight. For example... Figure 2 As shown, the sample is output through the teacher model, and the output of the teacher model is then compared with the real label to calculate the similarity. After the similarity is normalized, it becomes the weight of the teacher model.

[0073] S6, based on the calculated weights, guide the multi-teacher model to perform knowledge distillation on the student model. Teacher models that are closer to the true labels receive greater weights, allowing the student model to learn more knowledge. The loss between the student and teacher models is generated using KL (Kullback-Leibler divergence), and then weighted and summed to obtain the final response-based dynamic multi-teacher loss.

[0074]

[0075] in, Indicates the teacher model weights. O represents the output of the k-th teacher model. S This represents the output of the student model.

[0076] S7, Structured Relationship Calculation Stage; the calculation of structured relationships is mainly based on two points: positional distance structured relationships and angular structured relationships. First, distance structured relationships are calculated. This paper primarily uses Manhattan distance. The dataset contains several sample sets. Within each sample set (batch size), the Manhattan distance of the last layer features output by the model is calculated pairwise and then summed.

[0077]

[0078] Where d(x,y) represents the Manhattan distance between sample images. This represents the i-th dimension vector value of sample n. This represents the i-th dimension vector value of sample m.

[0079] Meanwhile, to avoid excessive loss from structured relationships that could affect convergence, this invention designs a novel activation function that can limit the excessive Manhattan distance loss to a smaller range, thereby avoiding a significant impact on convergence.

[0080]

[0081] Where, d t d represents the Manhattan distance generated by the teacher model. s The Manhattan distance generated for the student model, threshold is the threshold value, and γ is a hyperparameter that controls values ​​exceeding the threshold.

[0082] S8. Next, we calculate the angle-based structured relationships. Similarly, we use the output of the last layer of the model, and within a batch size, we calculate the inner product between each pair of normalized vector differences (bmm), i.e., the angle between each pair of target feature vectors and source feature vectors. The calculation method is as follows:

[0083] a(t i ,t j ,t k )= <e ij ,e kj >

[0084]

[0085] Where a(t) i ,t j ,t k ) represents the vector difference inner product, i.e., the calculated angular eigenvalue, t represents the sample, and three samples are used for calculation each time, e ij Indicate l i vector and l j The difference between vectors, e kj Similarly, vectors are in 2-norm form.

[0086] Similarly, we use an activation function to prevent excessive loss:

[0087]

[0088] Among them, a t and a s These represent the angular feature values ​​of the teacher model and the student model, respectively.

[0089] The final structured relation loss is the sum of the distance structure loss and the angle structure loss:

[0090]

[0091] α1 and β1 are hyperparameters used to control the ratio between distance structure loss and angle structure loss.

[0092] S9, Training Optimization; For the obtained model output distribution, three main losses are used for optimization: multi-teacher dynamic weight loss, structured loss, and hard label loss of the model output. The total loss is calculated as follows:

[0093]

[0094] in This represents the total loss. α² and β² are predefined hyperparameters used to control the weights of the two loss functions. This refers to the structured loss mentioned above. The multi-teacher dynamic weight loss calculated above, The loss between the student model output and the true label is calculated as follows, where y is the label:

[0095]

[0096] S10, Backward inference of the model, calculates the backward gradient based on the loss;

[0097] S11, Repeat the above process repeatedly until the final number of iterations is reached;

[0098] S12, Generation of model classification results: After completing the above training using the training set, the student model is output using the validation set, and the output results are compared with the true labels.

[0099] Using the aforementioned method, this invention was trained and validated using the CIFAR100 dataset. After 240 iterations of training, with a learning rate of 0.1, the results of different knowledge distillation methods were compared and are shown in Table 1.

[0100] Table 1: Results of knowledge distillation on the dataset

[0101]

[0102] The remaining methods in the table are described below:

[0103] AVER is a multi-teacher knowledge distillation method that learns from multiple teachers in an average manner, enabling student models to acquire knowledge from mature models.

[0104] FitNets are a specific architecture for knowledge distillation in deep neural networks. Originally proposed in a paper titled "FitNets: Hints for Thin Deep Nets," FitNets aimed to address problems encountered in the knowledge distillation process of deep neural networks. FitNets employ a specific deep neural network architecture designed to enable student models (shallower models) to better learn the knowledge of teacher models (deeper models). This architecture typically includes a series of "hints" to guide the student model in learning the knowledge from the teacher model.

[0105] EBKD proposes an entropy-based knowledge distillation training method that makes greater use of teacher model labels with lower entropy across multiple teacher models.

[0106] AEKD investigates the diversity of teacher models in gradient space and treats ensemble knowledge distillation as a multi-objective optimization problem to determine better optimization directions for training student networks. Furthermore, we introduce a tolerance parameter to accommodate discrepancies among teachers. Thus, our method can be viewed as a dynamically weighted approach for each teacher in the ensemble.

[0107] All three deep learning methods are based on the knowledge distillation architecture.

[0108] As can be seen from the experimental results in Table 1, the present invention has better performance in using knowledge distillation to achieve model compression.

[0109] Those skilled in the art should recognize that the examples described are intended to help readers understand the principles of the invention, and the scope of protection of the invention should not be limited to these specific statements and examples. Based on the technical teachings disclosed in this invention, those skilled in the art can make various other specific modifications and combinations, and as long as these modifications and combinations do not depart from the essence of the invention, they still fall within the scope of protection of this invention.

Claims

1. A knowledge distillation method based on a dynamic multi-teacher model and structured relationships, characterized in that, Includes the following steps: Step 1: Obtain the image dataset, which contains N samples. Each sample is represented as follows: ,in A sample is represented as a multidimensional tensor. ,in Let h and w represent the dimensions, and c represent the number of image channels; The sample labels are used to annotate the results of the samples; then the dataset is randomly divided into training and test sets according to a certain ratio. Step 2: Data preprocessing, preprocessing the samples in the dataset without changing the sample labels; Step 3: Use the dataset to train and test K different teacher models to obtain the teacher model that meets the performance requirements of the current task, and perform performance testing on the teacher model to obtain the teacher model performance parameters; Step 4: Design the student network and train the student model; Step 5: Multi-teacher weight calculation stage; First, each iteration requires inputting the training set into multiple teacher networks to obtain the results of the teacher networks; Second, the output of the teacher models needs to be evaluated. The evaluation method is to calculate the similarity between the teacher output and the sample label results, using cosine similarity to calculate and obtain the teacher model weights. Step 6: Based on the calculated weights, guide the multi-teacher model to perform knowledge distillation on the student model. Use KL divergence to generate the loss between the student and teacher models, then weight them together to obtain the final response-based dynamic multi-teacher loss. : ； in, Indicates the teacher model weights. This represents the output of the student model. This represents the output of the teacher model; Step 7: Structured Relationship Calculation Stage; The structured relationships are calculated based on positional distance structured relationships and angular structured relationships; First, the Manhattan distance is used to calculate the distance structured relationships. Within a sample set, the Manhattan distance of the last layer features output by the model is calculated pairwise, and then summed. ； in, Indicates the Manhattan distance between samples. This represents the i-th dimension vector value of sample n. This represents the i-th dimension vector value of sample m; Then, the angle-based structured relationships are calculated using the output of the last layer of the model. Within a sample set, the inner product is calculated pairwise using normalized vector differences, as follows: ； ； in, Let t represent the angular feature value, and t represent the sample. Three samples are used for each calculation. express Vector sum The difference between vectors Similarly, vectors are in 2-norm form. Step 8: Training Optimization; Optimization is performed using multi-teacher dynamic weight loss, structured loss, and hard label loss of the model output. The total loss is calculated as follows: ； in Indicates the total loss. This represents the structured loss calculated from structured relationships. , It's a hyperparameter. The loss between the model output and the true label is calculated as follows: ； Step 9: Backward inference of the model; calculate the backward gradient based on the loss; Step 10: Repeat the above process until the final number of iterations is reached; Step 11: Generating model classification results; After completing the above training using the training set, use the validation set to output the student model, and compare the output results with the true labels.

2. The knowledge distillation method based on a dynamic multi-teacher model and structured relationships according to claim 1, characterized in that, Step 3 specifically includes the following steps: S31, Design the teacher network, input the training set into the teacher network, and obtain the corresponding output; S32, adopt a gradient descent strategy to address the difference loss, update the model parameters, repeat the above iterations several times, and complete the training of the teacher model; S33, switch to test mode, input the test set into the teacher model, and record relevant parameters of the teacher model, including accuracy and convergence speed; S34, Repeat the above steps to obtain multiple teacher models.

3. The knowledge distillation method based on a dynamic multi-teacher model and structured relationships according to claim 1, characterized in that, The data preprocessing includes randomly cropping and horizontally flipping the image, while standardizing the image by normalizing the pixel values ​​by channel.

4. The knowledge distillation method based on a dynamic multi-teacher model and structured relationships according to claim 1, characterized in that, The calculation of the cosine similarity is as follows: ； ； in, This represents the output of the k-th teacher model. and sample label results The similarity between them is calculated as follows: k represents the k-th teacher model, i represents the comparison dimension between the teacher model output and the sample label (totaling n dimensions), and j represents the j-th teacher among all teacher models. Weights are assigned to the k-th teacher model based on similarity. .

5. The knowledge distillation method based on a dynamic multi-teacher model and structured relationships according to claim 4, characterized in that, The structured relation calculation stage employs the following activation function to prevent the Manhattan distance from becoming too large: ； in, Indicates distance structural loss, This represents the Manhattan distance generated by the teacher model. The Manhattan distance generated for the student model. To control hyperparameters that exceed threshold values; ； in, Indicates angular structural loss, and These represent the angular feature values ​​of the teacher model and the student model, respectively. Final structured loss The sum of distance structure loss and angle structure loss: ； in , This is a hyperparameter.

Images