Knowledge distillation method based on dynamic multi-teacher model and structured relationship

Through the knowledge distillation method of dynamic multi-teacher model and structured relationship, the problem of different teacher models is solved, and the accuracy and convergence speed of the student model is improved, and the better model compression effect is achieved.

CN119067194AActive Publication Date: 2024-12-03UNIV OF ELECTRONICS SCI & TECH OF CHINA

Patent Information

Application Number
CN202411094316.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

The existing knowledge distillation method has the problem of uneven training effects of different teachers in the model compression process and only considering label output while ignoring the sample structure relationship, resulting in training distortion.

Method used

The knowledge distillation method of dynamic multi-teacher model and structured relationship is adopted to calculate the teacher model weight through cosine similarity, and the Manhattan distance and angle structured relationship loss is introduced to guide students' model learning, and gradient descent training is carried out in combination with teacher model loss.

Benefits of technology

It effectively solves the problem of single teacher learning knowledge and the mediocrity of multiple teachers' average processing, improves the accuracy and convergence speed of the student model, enables the student model to learn richer content, and achieves better model compression effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067194A_ABST
    Figure CN119067194A_ABST
Patent Text Reader

Abstract

The invention discloses a knowledge distillation method based on a dynamic multi-teacher model and a structured relationship, and the main content is a novel knowledge distillation method which comprises the steps: training a student model through a plurality of teacher models, and distributing a dynamic weight for the teacher model according to the performance effect of the teacher model to guide the training of the student model; and meanwhile, a structured relationship is introduced to assist training, and the training effect is improved by enabling the student model to learn the spatial relationship output by the teacher model sample. Compared with a traditional multi-teacher model, the method has the advantages that the problems of singleness, blindness and averaging of training and learning are solved, the learning weight is dynamically adjusted according to the advantages and disadvantages of the teacher model, and a better training effect is achieved; and meanwhile, the sample space relationship is introduced as knowledge for learning, so that the monotonicity that a traditional knowledge distillation method only depends on response knowledge learning is solved, the student model can learn richer knowledge, and the performance of the student model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of neural networks in machine learning, and in particular to a knowledge distillation method based on a dynamic multi-teacher model and structured relationships. Background Art

[0002] In machine learning methods, model compression plays an important role. It can make the model perform close to the original model with fewer parameters. Generally speaking, in the absence of overfitting, the performance of the model is positively correlated with the number of parameters to a certain extent. However, this does not mean that the more parameters, the better the performance. This is because the performance of the model will gradually reach a bottleneck as the parameters grow. Figure 1 As shown in the figure, although the final model performance will tend to a bottleneck, compared with other model compression methods (curve a), knowledge distillation can achieve more significant training effects (curve b) by acquiring knowledge, and show better results with fewer model parameters.

[0003] Knowledge distillation is a model compression method based on the teacher-student architecture. Through the pre-trained, relatively complex teacher model, the student model with small model parameters and relatively simple model parameters is trained, so that the student model can learn the hidden knowledge of the complex teacher model and transfer this knowledge to its own student model, so that the student model with relatively small model parameters can show strong reasoning ability close to the teacher model on a restricted machine, achieving the purpose of model compression.

[0004] There are many optimization methods for knowledge distillation in specific operations, resulting in many variants. Each variant model can continue to improve traditional knowledge distillation, thereby further improving accuracy, convergence speed, number of parameters, etc. The current mainstream optimization solutions include multi-teacher distillation, self-distillation, semi-supervised distillation, etc. This article mainly focuses on multi-teacher distillation and its derivative method multi-teacher weight adaptive distillation.

[0005] Compared with traditional knowledge distillation, multi-teacher distillation uses multiple pre-trained teacher models to guide the student model. Compared with the traditional single-teacher model, multi-teacher has better robustness and accuracy, avoids the occasional deviation of a single teacher, and improves the accuracy and convergence speed of the student model, thereby achieving better model compression effect. There are many classifications of multi-teacher models themselves, such as the common average teacher model, and the focus of this article, multi-teacher adaptive weight distillation.

[0006] The main breakthrough of multi-teacher adaptive weight distillation is that, compared with ordinary multi-teacher average distillation, the weight of the teacher is dynamically set, allowing teachers with higher output accuracy to have higher weights, so that the student model can learn more knowledge from this teacher in the current number of iterations. Since more correct knowledge is learned, the student model can naturally converge faster and improve its own accuracy. Summary of the invention

[0007] The goal of this invention is to achieve model compression using knowledge distillation, and introduce dynamic weights and structured relationships to alleviate two types of distortion problems that occur in the model during compression: one is that different teacher models have different training effects and uneven training levels; the other is that only label output is considered without considering the sample structure relationship, resulting in training distortion. In this way, we aim to achieve a student model with better model performance.

[0008] The idea of ​​the present invention is to dynamically calculate the teacher model weight through cosine similarity, so that the teacher model with better performance can obtain more training weights. On this basis, the structured relationship is introduced to calculate the difference between the multi-sample structure of the teacher model and the structure of the student model, and guide the student model through the structural difference. Finally, the gradient descent is performed through the teacher model loss, structured loss and true label to train the student model.

[0009] In order to solve the above technical problems, a specific technical solution of a knowledge distillation method based on a dynamic multi-teacher model and structured relationships of the present invention is as follows:

[0010] Step 1: Get an image dataset. There are N samples in the dataset, and each sample is represented as: , where x i Represented as a sample, it consists of a multi-dimensional tensor, represented as in Represents a multidimensional tensor, h, w represent the dimension, c represents the number of image channels; y i Represents the sample label and the result of labeling the sample; then the dataset is randomly divided into training set and test set according to the proportion.

[0011] Step 2: Data preprocessing: preprocess the samples in the data set without changing the labels of the samples;

[0012] Step 3: Use the data set to train and test K different teacher models to obtain a teacher model that meets the performance requirements for the current task, and perform performance tests on the teacher model to obtain the teacher model performance parameters;

[0013] Step 4: Design the student network and train the student model;

[0014] Step 5: Multi-teacher weight calculation stage; First, each iteration needs to input the training set into multiple teacher networks to obtain the results of the teacher network; secondly, the output of the teacher model needs to be evaluated by calculating the similarity between the teacher output and the sample label result, using cosine similarity to calculate:

[0015]

[0016] in, Represents the k-th teacher model output And the sample label result y t The similarity between them, k represents the k-th teacher model, i represents the comparison dimension between the teacher model output and the sample label, with a total of n dimensions, j represents the j-th teacher among all teacher models, and the k-th teacher model is weighted by similarity

[0017] Step 6: According to the calculated weights, guide the multi-teacher model to perform knowledge distillation on the student model, generate the loss between the student model and the teacher model through KL divergence, and then use the weights to weighted add to get the final response-based dynamic multi-teacher loss

[0018]

[0019] in, represents the teacher model weight, o S Represents the student model output.

[0020] Step 7: Structural relationship calculation stage; Calculation of structured relationships is based on position distance structure relationships and angle structure relationships; First, the distance structure relationship is calculated using Manhattan distance. In a sample set, the Manhattan distance of the last layer of features output by the model is calculated pairwise and summed up:

[0021]

[0022] Among them, d(x,y) represents the Manhattan distance between samples. Represents the i-th dimension vector value of sample n, Represents the i-th dimension vector value of sample m;

[0023] The following activation function is used to prevent the Manhattan distance from being too large:

[0024]

[0025] Among them, L SRD represents the distance structure loss, d t represents the Manhattan distance produced by the teacher model, d sis the Manhattan distance generated by the student model, and γ is a hyperparameter that controls the value exceeding the threshold;

[0026] Then, the angle-based structured relationship is calculated by taking the output of the last layer of the model and using the inner product between the normalized vector differences in a sample set. The calculation method is as follows:

[0027] a(t i ,t j ,t k )= <e ij ,e kj >

[0028]

[0029] Among them, a(t i ,t j ,t k ) represents the angle eigenvalue, t represents the sample, three samples are used for calculation each time, e ij Indicates l i Vector and l j Vector difference of vectors, e kj Similarly, the vector uses the 2-normalized vector;

[0030] Similarly, an activation function is used to prevent excessive loss:

[0031]

[0032] Among them, L SRA represents the angle structural loss, a t and a s denote the angle eigenvalues ​​of the teacher model and the student model respectively;

[0033] The final structured loss It is the sum of distance structure loss and angle structure loss:

[0034]

[0035] where α 1 , β 1 is a hyperparameter;

[0036] Step 8: Training optimization; use multi-teacher dynamic weight loss, structured loss and hard label loss of model output for optimization. The total loss is calculated as follows:

[0037]

[0038] in represents the total loss, α 2 , β 2 is a hyperparameter, represents the loss between the model output and the true label, which is calculated as follows:

[0039]

[0040] Step 9: Reverse inference of the model; perform reverse gradient calculation based on the loss;

[0041] Step 10: Repeat the above process until the final number of iterations is reached;

[0042] Step 11: Generate model classification results; After completing the above training using the training set, use the validation set to output the student model, and compare the output results with the true labels.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. The present invention effectively solves the problem of single-teacher knowledge uniformity through dynamic weights of multiple teachers, while also avoiding the mediocrity problem caused by conventional multi-teacher averaging processing, so that each teacher can dynamically guide students based on their output results, thereby enabling the student model to better absorb the learning results of multiple teachers.

[0045] 2. The present invention uses structured relationships to assist the student model in learning. The student model no longer simply learns sample responses, that is, the content output by the sample model, but learns more about the spatial relationship between samples, that is, distance, angle and other information, so that the student model can learn richer content, thereby producing better learning effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the relationship between model parameters and performance;

[0047] Figure 2 Assign a strategy graph to the weights;

[0048] Figure 3 It is a schematic diagram of the structure of the present invention. DETAILED DESCRIPTION

[0049] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of the knowledge distillation method based on a dynamic multi-teacher model and structured relationships of the present invention in conjunction with the accompanying drawings.

[0050] Based on the above invention ideas, the present invention proposes a knowledge distillation model based on structured relationships and dynamic teachers. First, the specific requirements of the knowledge distillation task are defined as follows:

[0051] The embodiment of the present invention adopts an image recognition model and a training set CIFAR100. This study will also use a large number of normalization functions. The normalization function is mainly used for classification tasks to normalize the model output to between 0 and 1. In knowledge distillation, temperature τ will also be used for processing, and the method is as follows:

[0052]

[0053] Among them, σ(o c ) represents the normalized function output, Represents the model target classification output, o j Represents other classification outputs of the model, τ represents the distillation temperature, and τ is used for smoothing to highlight the importance of dark knowledge, that is, the importance of other classification output knowledge.

[0054] The main method used in this paper is knowledge distillation, which is a model compression method that uses a more mature large model to train and guide a small model, so that the small model can show better performance under scene tasks. The main methods are as follows:

[0055]

[0056] in is the soft target loss, which means the student model output q j And the teacher model output The loss between is a hard loss, expressed as the student model output q j and the true label c j The loss between Represents the final model loss. The balance of the two types of losses is achieved by adjusting the hyperparameters α and β.

[0057] Based on traditional knowledge distillation, this study introduced a multi-teacher knowledge distillation method. Multi-teacher knowledge distillation allows the student model to learn from multiple models at the same time, thereby improving learning efficiency, increasing robustness and regularity. At the same time, we will dynamically assign different weights to the teacher model so that the teacher model with better learning effect can obtain a larger dynamic teaching weight, so as to better guide the student model. In addition, we also introduced a structured method to enhance the distillation effect, so that the student model can obtain a better distillation effect. Learning is carried out by calculating the structured relationship between multiple samples, and a set of optimized activation functions are proposed to achieve better results.

[0058] Now we will introduce the specific methods of the method model in detail, such as Figure 3 As shown, the following steps are included:

[0059] S1, collect data sets:

[0060] The present invention uses the CIFAR-100 dataset. CIFAR-100 (Canadian Institute for Advanced Research-100) is a commonly used computer vision dataset for image classification tasks. It is an extended version of the CIFAR-10 dataset, which contains more categories and more diverse images. The CIFAR100 dataset has 100 classes. Each class has 600 color images of size 32×32 32\times 3232×32, which are randomly divided into training sets and test sets in a ratio of 5:1; each sample is represented as: x i Represented as a sample image, it is composed of a multi-dimensional tensor, represented as in Represents a multidimensional tensor, h, w represent the dimension, c represents the number of image channels, y i represents the sample label, the result of labeling the sample. N is the number of samples, For sample.

[0061] S2, data preprocessing:

[0062] The image is mainly randomly cropped and horizontally flipped to increase the diversity of the data without changing the image label. At the same time, the image is standardized to normalize the values ​​of the image pixels by channel. The mean and standard deviation are given here, corresponding to the mean and standard deviation of the three RGB channels respectively. The standardization operation can make the pixel value distribution of each channel closer to the standard normal distribution, which is helpful for model training and optimization.

[0063] S3, use the data set to train and test multiple different teacher models, obtain the teacher model that meets the performance requirements of the current task, and perform performance testing on the teacher model to obtain the teacher model performance parameters; the specific steps are as follows:

[0064] S31, design a teacher network, input the training set into multiple different teacher networks, and obtain the corresponding output;

[0065] S32, adopting a gradient descent strategy for the difference loss function, updating the model parameters, repeating the above several iterations (epoch) to complete the training of the teacher model;

[0066] S33, adjust to the test mode, input the test set to the teacher model, and record the relevant parameters of the teacher model, including the accuracy, convergence speed, etc.;

[0067] S34, repeat the above steps to obtain multiple teacher models;

[0068] S4, design the student network and train the student model. The student network has a smaller network layer and requires fewer parameters;

[0069] S5, multi-teacher weight calculation stage; First, each iteration needs to first input the training set into multiple teacher networks to obtain the results of the teacher network. Secondly, the output of the teacher model needs to be evaluated. The evaluation method is to calculate the similarity between the teacher output and the label result (ground truth), using cosine similarity to calculate:

[0070]

[0071]

[0072] in, Represents the k-th teacher model output And the sample label result y t The similarity between them, k represents the k-th teacher model, i represents the comparison dimension between the teacher model output and the sample label, with a total of n dimensions, and j represents the j-th teacher among all teacher models. All teacher models need to be normalized. Weights are assigned to the k-th teacher model by similarity. The evaluation result is the weight. Figure 2 As shown in the figure, the sample is output by the teacher model, and the output of the teacher model is compared with the true label to calculate the similarity. After the similarity is normalized, it becomes the weight of the teacher model.

[0073] S6, according to the calculated weights, guide the multi-teacher model to perform knowledge distillation on the student model. The teacher model that is closer to the true label gets a larger weight, so that the student model can learn more knowledge from it. The loss between the student model and the teacher model is generated through the KL (Kullback-Leibl er divergence) divergence, and then the weights are used to weightedly add to obtain the final response-based dynamic multi-teacher loss

[0074]

[0075] in, represents the teacher model weight, represents the output of the k-th teacher model, o S Represents the student model output.

[0076] S7, structured relationship calculation stage; the calculation of structured relationships is mainly based on two points: position distance structure relationship and angle structure relationship. First, the distance structure relationship is calculated. This paper mainly uses Manhattan distance. The data set contains several sample sets. In a sample set (batch size), the Manhattan distance of the last layer of features output by the model is calculated two by two, and the sum is accumulated:

[0077]

[0078] Among them, d(x,y) represents the Manhattan distance between sample images. Represents the i-th dimension vector value of sample n, Represents the i-th dimension vector value of sample number m.

[0079] At the same time, in order to avoid excessive loss caused by structured relationships and affect convergence, the present invention designs a new activation function that can limit excessive Manhattan distance loss to a smaller range, thereby avoiding a significant impact on convergence:

[0080]

[0081] Among them, d t represents the Manhattan distance produced by the teacher model, d s is the Manhattan distance generated by the student model, threshold is the threshold, and γ is the hyperparameter that controls the value exceeding the threshold.

[0082] S8, next we calculate the angle-based structural relationship. Similarly, we use the output of the last layer of the model and use the inner product between the normalized vector differences (bmm) in a batch size, that is, calculate the angle between each pair of target feature vectors and source feature vectors. The calculation method is as follows:

[0083] a(t i ,t j ,t k )= <e ij ,e kj >

[0084]

[0085] Among them, a(t i ,t j ,t k ) represents the inner product of vector difference, that is, the calculated angle eigenvalue, t represents the sample, and three samples are used for calculation each time, e ij Indicates l i Vector and l j Vector difference of vectors, e kj Similarly, the vector uses the 2-normal form vector.

[0086] Similarly, we use an activation function to prevent the loss from being too large:

[0087]

[0088] Among them, a t and a s Represent the angle eigenvalues ​​of the teacher model and the student model respectively.

[0089] The final structured relationship loss is the sum of the distance structure loss and the angle structure loss:

[0090]

[0091] where α 1 , β 1 is a hyperparameter to control the ratio between distance structure loss and angle structure loss.

[0092] S9, training optimization; for the obtained model output distribution, three main losses are used for optimization, namely multi-teacher dynamic weight loss, structured loss and hard label loss of model output. The total loss is calculated as follows:

[0093]

[0094] in represents the total loss, α 2 , β 2 is a predefined hyperparameter used to control the weights of the two types of loss functions. represents the structured loss mentioned above, is the multi-teacher dynamic weight loss calculated above, represents the loss between the student model output and the true label, calculated as follows, where y is the label:

[0095]

[0096] S10, reverse reasoning of the model, performing reverse gradient calculation based on the loss;

[0097] S11, repeatedly iterate the above process until the final number of iterations is reached;

[0098] S12, generation of model classification results; after completing the above training using the training set, the student model is output using the validation set, and the output results are compared with the true labels.

[0099] Using the described method, the present invention uses the CIFAR100 data set for training and verification. After 240 training iterations, the learning rate is 0.1, and different knowledge distillation methods are compared. The results are shown in Table 1.

[0100] Table 1: Results of knowledge distillation on the dataset

[0101]

[0102] The remaining methods in the table are described as follows:

[0103] AVER is a multi-teacher knowledge distillation method that enables the student model to learn the knowledge of the mature model by averaging learning from multiple teachers.

[0104] FitNets is a specific architecture for knowledge distillation in deep neural networks. FitNets was originally proposed in a paper called "FitNets: Hints for Thin Deep Nets" to solve the problems encountered by deep neural networks in the process of knowledge distillation. FitNets uses a specific deep neural network architecture to enable the student model (shallower model) to better learn the knowledge of the teacher model (deeper model). The architecture usually includes a series of "hints" (hints) to guide the student model to learn the knowledge of the teacher model.

[0105] EBKD proposes an entropy-based knowledge distillation training method that makes more use of teacher model labels with lower entropy among multiple teacher models.

[0106] AEKD studies the diversity of teacher models in the gradient space and regards ensemble knowledge distillation as a multi-objective optimization problem in order to determine better optimization directions to train the student network. In addition, we introduce a tolerance parameter to accommodate the differences between teachers. In this way, our method can be viewed as a dynamic weighting method for each teacher in the ensemble.

[0107] All three deep learning methods are based on knowledge distillation.

[0108] It can be seen from the experimental results in Table 1 that the present invention has better performance in realizing model compression by utilizing knowledge distillation.

[0109] Those skilled in the art should be aware that the application examples are intended to help readers understand the principles of the present invention, and the protection scope of the present invention should not be limited to these specific statements and application examples. According to the technical enlightenment disclosed by the present invention, those skilled in the art can make various other specific modifications and combinations, as long as these modifications and combinations do not deviate from the essence of the present invention, they still fall within the protection scope of the present invention.

Claims

1. A knowledge distillation method based on a dynamic multi-teacher model and structured relations, characterized in that: The following steps are involved: Step 1: Get an image dataset. There are N samples in the dataset, and each sample is represented as: where x i Represented as a sample, it consists of a multi-dimensional tensor, represented as in Represents a multidimensional tensor, h, w represent the dimension, c represents the number of image channels; y i Represents the sample label and the result of labeling the sample; then the dataset is randomly divided into training set and test set according to the proportion. Step 2: Data preprocessing: preprocess the samples in the data set without changing the labels of the samples; Step 3: Use the data set to train and test K different teacher models to obtain a teacher model that meets the performance requirements for the current task, and perform performance tests on the teacher model to obtain the teacher model performance parameters; Step 4: Design the student network and train the student model; Step 5: Multi-teacher weight calculation stage; First, each iteration needs to input the training set into multiple teacher networks to obtain the results of the teacher network; secondly, the output of the teacher model needs to be evaluated by calculating the similarity between the teacher output and the sample label result, using cosine similarity to calculate and obtain the teacher model weight; Step 6: According to the calculated weights, guide the multi-teacher model to perform knowledge distillation on the student model, generate the loss between the student model and the teacher model through KL divergence, and then use the weights to weighted add to get the final response-based dynamic multi-teacher loss in, represents the teacher model weight, o S represents the student model output, represents the teacher model output; Step 7: Structural relationship calculation stage; Calculation of structured relationships is based on position distance structure relationships and angle structure relationships; First, the distance structure relationship is calculated using Manhattan distance. In a sample set, the Manhattan distance of the last layer of features output by the model is calculated pairwise and summed up: Among them, d(x,y) represents the Manhattan distance between samples. Represents the i-th dimension vector value of sample n, Represents the i-th dimension vector value of sample m; Then, the angle-based structured relationship is calculated by taking the output of the last layer of the model and using the inner product between the normalized vector differences in a sample set. The calculation method is as follows: a(t i ,t j ,t k )=<e ij ,e kj > Among them, a(t i ,t j ,t k ) represents the angle eigenvalue, t represents the sample, three samples are used for calculation each time, e ij Indicates l i Vector and l j Vector difference of vectors, e kj Similarly, the vector uses the 2-normalized vector; Step 8: Training optimization; use multi-teacher dynamic weight loss, structured loss and hard label loss of model output for optimization. The total loss is calculated as follows: in represents the total loss, The structured loss calculated for the structured relationship, α2, β2 are hyperparameters, represents the loss between the model output and the true label, which is calculated as follows: Step 9: Reverse inference of the model; perform reverse gradient calculation based on the loss; Step 10: Repeat the above process until the final number of iterations is reached; Step 11: Generate model classification results; After completing the above training using the training set, use the validation set to output the student model, and compare the output results with the true labels.

2. A knowledge distillation method based on a dynamic multi-teacher model and structured relations according to claim 1, characterized in that: The step 3 specifically comprises the following steps: S31, design the teacher network, input the training set into the teacher network, and obtain the corresponding output; S32, adopting a gradient descent strategy for the difference loss, updating the model parameters, repeating the above several iterations, and completing the training of the teacher model; S33, adjust to the test mode, input the test set to the teacher model, and record the relevant parameters of the teacher model, including the accuracy, convergence speed, etc.; S34, repeat the above steps to obtain multiple teacher models.

3. The knowledge distillation method based on dynamic multi-teacher model and structured relationship according to claim 1, characterized in that: The data preprocessing includes randomly cropping and horizontally flipping the image, and standardizing the image to normalize the values ​​of the image pixels by channel.

4. The knowledge distillation method based on dynamic multi-teacher model and structured relationship according to claim 1, characterized in that: The calculation of the cosine similarity is as follows: in, Represents the k-th teacher model output And the sample label result y t The similarity between them, k represents the k-th teacher model, i represents the comparison dimension between the teacher model output and the sample label, with a total of n dimensions, j represents the j-th teacher among all teacher models, and the k-th teacher model is weighted by similarity 5. A knowledge distillation method based on a dynamic multi-teacher model and structured relations according to claim 4, characterized in that: The structured relationship calculation stage uses the following activation function to prevent the Manhattan distance from being too large: Among them, L SRD represents the distance structure loss, d t represents the Manhattan distance produced by the teacher model, d s is the Manhattan distance generated by the student model, and γ is a hyperparameter that controls the value exceeding the threshold; Among them, L SRA represents the angle structural loss, a t and a s denote the angle eigenvalues ​​of the teacher model and the student model respectively; The final structured loss It is the sum of distance structure loss and angle structure loss: Among them, α1 and β1 are hyper parameters.

Citation Information

Patent Citations

  • Multi-teacher self-adaptive joint knowledge distillation

    CN112418343A

  • Image classification method and system based on isomorphic multi-teacher guidance knowledge distillation

    CN113627545A

  • Multi-teacher knowledge distillation method and device based on adversarial samples

    CN114219043A

  • Model distillation method, device and equipment based on multi-teacher model and storage medium

    CN114386604A

  • Knowledge distillation method, device and equipment based on multiple teachers and storage medium

    CN116644798A

Cited By

  • Incremental target detection method based on multi-network mean distillation loss function

    CN116310480A

  • Anti-forgetting multi-field Chinese spelling error correction knowledge migration method

    CN119808765A

  • Image depth estimation method, system and equipment based on knowledge mutual distillation and medium

    CN120259397A