Staged adaptive knowledge distillation algorithm

Through the stage-adaptive knowledge distillation algorithm, the gradient information of the loss function and the chain derivative rule are used to dynamically adjust the distillation loss weight, which solves the problem of ignoring the training performance of student models in existing technologies and achieves efficient training and performance improvement in resource-constrained environments.

CN120671766AInactive Publication Date: 2025-09-19YANGEN UNIV
2 Cites 0 Cited by

Patent Information

Application Number
CN202510694872.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-28
Publication Date
2025-09-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing knowledge distillation methods ignore the discussion of student model training performance and perform poorly in resource-constrained environments.

Method used

A stage-adaptive knowledge distillation algorithm is adopted to perform backpropagation through the gradient information of the loss function and the chain rule, dynamically adjust the distillation loss weight, and decide whether to perform distillation based on the comprehensive scoring indicators of the student and teacher models. It is integrated into existing methods to adapt to the characteristics of the training data.

Benefits of technology

It improves the training performance of the student model, improves the performance in resource-constrained environments, and enhances the model's generalization ability and classification accuracy.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a stage-adaptive knowledge distillation algorithm. The algorithm comprises the following specific steps: inputting an initial data set as a main supervision task; a group of convolutional neural networks (a teacher model and a student model) is initialized, a decoder is initialized and trained, and a decision layer is initialized; inputting the data set into a convolutional neural network, obtaining feature output of the penultimate layer of the teacher-student model, inputting the feature output into a decoding module to obtain a reconstructed image, and inputting the reconstructed image of the teacher-student model and the original image into a decision layer to obtain a decision result of whether distillation is performed; the gradient information of the loss function is utilized, back propagation is carried out through a chain derivation rule, and the gradient obtained through calculation is used for updating the weight of the network; and repeating the step 3 to the step 4 until the loss function value converges or reaches a preset training period. The method not only can be integrated into an existing distillation method, but also can avoid limitation of a distillation algorithm in a resource-limited environment.
Need to check novelty before this filing date? Find Prior Art