A knowledge distillation method and system based on adaptive local-global collaboration
Patent Information
- Application Number
- CN202611037067.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-13
- Publication Date
- 2026-10-30
AI Technical Summary
[0009]本发明旨在解决现有知识蒸馏技术中权重分配僵化、局部与全局知识衔接生硬的技术问题，提供一种基于自适应局部-整体协同的知识蒸馏方法
[0039]1、获取教师模型和学生模型的原始Logit向量；通过Z-score标准化消除量纲差异，计算全局蒸馏损失以约束整体分布对齐；基于学生模型置信度选取Top-k类别构建局部Logit子空间，通过遍历子空间内类别对的散度均值计算局部关系损失，强化关键特征区分；设定总训练轮次E，采用分段自适应机制动态调整局部损失权重系数α(epoch)，使训练前期侧重局部拟合、后期侧重全局收敛；融合交叉熵损失、全局损失及加权局部损失更新学生模型。
Smart Images
Figure CN123014072A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of deep learning and model compression technology, specifically relating to a knowledge distillation method, and more particularly to a distillation method and system that can adaptively adjust local and global knowledge weights according to the training process. Background Technology
[0002] With the development of deep learning, neural network models have achieved remarkable results in tasks such as image classification and object detection. However, high-performance models are usually accompanied by a huge number of parameters and computational overhead, making them difficult to deploy directly on mobile or embedded devices. Knowledge distillation (KD), a classic model compression technique, effectively solves this problem by transferring the "soft-label" knowledge output by a large teacher model to a lightweight student model.
[0003] In existing distillation algorithms, Logit Standardization is often used to eliminate the problem of inconsistent output layer scales between the teacher and student models. By using Z-score standardization, Logit is transformed into a distribution with a mean of 0 and a variance of 1, thus providing a stable distillation baseline. In addition, Local Relation Distillation has also been proposed, aiming to improve the relevance of knowledge transfer by focusing on the categories with high confidence in the student model.
[0004] However, existing technologies have the following drawbacks:
[0005] Static weight conflict: Existing local and global distillation methods typically use fixed weight ratios. In the early stages of training, when the student model's capabilities are insufficient, excessive global loss can lead to optimization difficulties; while in the later stages of training, excessive local loss can limit the model's ability to generalize to overall class relationships.
[0006] Abrupt transition: The lack of a smooth transition mechanism from "local fine-grained learning" to "global knowledge integration" leads to inconsistent learning objectives of the model at different training stages, affecting the final convergence effect.
[0007] Poor adaptability: For emerging algorithms such as Decoupled Knowledge Distillation (DKD), due to the lack of targeted scheduling strategies, traditional distillation components cannot be directly reused.
[0008] Therefore, there is an urgent need for an adaptive method that can dynamically coordinate local and global knowledge and adapt to various distillation frameworks. Summary of the Invention
[0009] This invention aims to solve the technical problems of rigid weight allocation and awkward connection between local and global knowledge in existing knowledge distillation techniques, and provides a knowledge distillation method based on adaptive local-global synergy.
[0010] To achieve the above objectives, the present invention provides the following technical solution: a knowledge distillation method based on adaptive local-global synergy, comprising the following steps:
[0011] (1) Obtain the teacher model and student model: the original Logit vector output for the same input sample. and ;
[0012] (2) Global semantic alignment: for and Distribution standardization is performed to eliminate dimensional differences. The global distillation loss L between the two is calculated based on the temperature coefficient, and the overall output distribution of the student model is constrained to approximate that of the teacher model.
[0013] (3) Local relation modeling: Select the Top-k category index based on the confidence score output by the student model, construct the local Logit subspace of the corresponding index of the teacher and student model, and obtain the local relation loss L by calculating the divergence of the relative relationship between categories in the subspace to strengthen the distinguishing features between key categories;
[0014] (4) Dynamic weight scheduling: Set the total number of training rounds E, and adaptively adjust the local loss weight coefficients according to the current training progress. This allows the model to focus on fitting local details in the early stages of training and on global distribution convergence in the later stages of training, achieving a smooth transition.
[0015] (5) Loss Fusion and Training: Fusion of cross-entropy loss, global distillation loss, and loss processing. The weighted local relation loss yields the total loss, which is used to update the student model parameters until convergence.
[0016] Furthermore, the global semantic alignment described in step (2) is implemented as follows:
[0017] right and Perform Z-score standardization on a sample-by-sample basis to obtain the standardized Logit vector. and The KL divergence between the two is calculated based on the temperature coefficient T, and the global distillation loss L is obtained.
[0018] Furthermore, the Z-score normalization formula is as follows:
[0019]
[0020] where z is the original Logit vector, μ is the mean, σ is the standard deviation, and ε is a very small constant.
[0021] Further, the calculation method of the local relation loss described in step (3) is: among the selected top k category indices, traverse all unique category pairs (i,j) that satisfy i<j, and calculate the divergence of the corresponding Logit of each pair; construct a rank correlation weight according to the ranking difference and ranking position of the category pair in the local subspace, and construct a hard example relation weight according to the preference distribution difference of the teacher model and the student model on the category pair; multiply the rank correlation weight and the hard example relation weight to obtain an adaptive category pair weight, so as to obtain the local relation loss.
[0022] Further, the dynamic weight coefficient described in step (4) adopts a piecewise adaptive mechanism, which attenuates non-linearly along with the training process, and the piecewise mechanism at least includes three stages: early training stage, middle training stage and late training stage.
[0023] Further, the dynamic weight coefficient is specifically configured as follows:
[0024] In the early training stage, a relatively high value is maintained to promote local feature learning;
[0025] In the middle training stage, it decreases smoothly from a high value to a low value;
[0026] In the late training stage, it converges to a relatively low stable value to maintain global stability.
[0027] Further, the threshold and convergence value of the dynamic weight coefficient in different stages can be adaptively adjusted according to the difficulty of the specific task and the architecture difference between the teacher model and the student model.
[0028] Further, the method can be migrated and adapted to the decoupled knowledge distillation DKD framework, and during adaptation, the stage division ratio and weight convergence value in the piecewise mechanism are dynamically adjusted according to the characteristics of the target framework.
[0029] Further, the teacher model and the student model support ResNet, ShuffleNet, WideResNet or Vision Transformer architectures.
[0030] A system for implementing the above method is also provided, and the system includes:
[0031] a data preprocessing module, configured to perform enhancement and normalization processing on training data;
[0032] The model management module is used to load pre-trained teacher models and initialize student models;
[0033] The forward inference module is used to obtain the raw Logit output of the teacher-student model;
[0034] The global distillation module is used to perform Logit normalization and global KL divergence calculation;
[0035] The local distillation module is used to construct a local Logit subspace and calculate the local relation loss;
[0036] The dynamic scheduling module is used to generate adaptive weight coefficients based on the training rounds;
[0037] The optimization module is used to fuse multiple loss functions and update the parameters of the student model.
[0038] Compared with the prior art, the beneficial effects of the present invention are:
[0039] 1. Obtain the original Logit vectors of the teacher and student models; eliminate dimensional differences through Z-score standardization, calculate the global distillation loss to constrain the overall distribution alignment; select the Top-k categories based on the student model confidence to construct a local Logit subspace, calculate the local relation loss by traversing the mean divergence of class pairs within the subspace to strengthen the differentiation of key features; set the total number of training epochs E, and use a segmented adaptive mechanism to dynamically adjust the local loss weight coefficient α (epoch), so that the early stage of training focuses on local fitting and the later stage focuses on global convergence; integrate cross-entropy loss, global loss and weighted local loss to update the student model.
[0040] 2. This invention smoothly transitions the learning objective based on the training process and adapts the decoupled knowledge distillation DKD framework by adjusting the segment ratio and convergence value. This method improves the accuracy of the ResNet-8x4 student model to 77.37% on the CIFAR-100 dataset, significantly enhancing model convergence stability and distillation performance. Attached Figure Description
[0041] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.
[0042] Figure 2 This is a system flowchart of the present invention. Detailed Implementation
[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0044] Example 1: Standard distillation experiment based on the CIFAR-100 dataset
[0045] This example demonstrates the implementation process using a conventional convolutional neural network (CNN) architecture. Dataset and preprocessing:
[0046] The CIFAR-100 dataset, containing 100 categories, was used. Input images were normalized with a mean of [0.5071, 0.4867, 0.4408] and a standard deviation of [0.2675, 0.2565, 0.2761]. During training, random horizontal flipping, 32×32 random cropping, and 16×16 CutOut occlusion enhancement were employed.
[0047] Model configuration: The teacher model uses ResNet-32x4, and the student model uses ResNet-8x4. The teacher model is pre-trained until convergence and its parameters are fixed, while the student model is randomly initialized.
[0048] Training parameters: The optimizer used is SGD with a momentum of 0.9 and a weight decay of 5e-4. The initial learning rate is set to 0.05, and cosine annealing is used for decay. The batch size is 64, and the total number of training epochs E is 240. The distillation temperature coefficient T is set to 4.0, and the number of locally focused classes k is set to 7.
[0049] Implementation steps:
[0050] Step S1: Input the batch data into the teacher-student model respectively to obtain the raw Logit. and .
[0051] Step S2 (Global Distillation): For and Perform sample-by-sample Z-score standardization (following the formula) ), calculate the standardized KL divergence as the global loss. .
[0052] Step S3 (local distillation): Extract the indices of the top-7 categories in the student Logits, intercept the corresponding Logit subspaces of the teacher and the student according to the indices, traverse all unique category pairs satisfying i<j, calculate the divergence of the corresponding Logits for each pair, and take the arithmetic mean of the divergence results of all pairs to obtain the local relation loss .
[0053] Step S4 (dynamic scheduling): Set dynamic weights , and adopt a piecewise adaptive mechanism. When epoch < 72 (0.3E, early training stage), α=1; when 72 ≤ epoch < 180 (0.75E, middle training stage), α smoothly decreases from a high value to a low value; when epoch ≥ 180 (late training stage), α converges to a stable value of 0.4.
[0054] Step S5 (loss fusion): Total loss . Perform back propagation to update the parameters of the student model.
[0055] Experimental results: As shown in Table 1, the accuracy of the student model is significantly improved after applying the coordination strategy.
[0056]
[0057] Embodiment 2: Transfer adaptation to Decoupled Knowledge Distillation (DKD)
[0058] This embodiment verifies the universality of the framework, and demonstrates how to adjust the dynamic weight strategy for the DKD architecture.
[0059] Adaptation adjustment: Aiming at the characteristic of DKD that decouples the loss into target class and non-target class, dynamically adjust the stage division ratio and the weight convergence value in the piecewise mechanism according to the characteristics of the target framework. Adjust the threshold of the local dominant period from 0.3E to 0.2E (i.e., 48 epochs), and adjust the lower limit of the weight in the later stage from 0.4 to 0.3.
[0060] Results: On the same ResNet-32x4 → ResNet-8x4 model pair, the accuracy of the adapted method reaches 77.63%. This result is better than the original DKD algorithm (76.32%) and the unadapted basic version of the present method (77.37%), which proves that adjusting the dynamic weight parameters can effectively adapt to different distillation frameworks.
[0061] Embodiment 3: Ablation experiment
[0062] In order to verify the necessity of each technical module, a control experiment is carried out on the ResNet-32x4 → ResNet-8x4 model pair, as shown in Table 2.
[0063] Table 2
[0064]
[0065] Experimental conclusions: The accuracy of group B dropped significantly, proving that Logit standardization is the key to building a stable baseline; both groups C and D showed a decrease compared to group A, proving that the local relation distillation and dynamic weight reduction strategies make substantial contributions to improving model performance.
[0066] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0067] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A knowledge distillation method based on adaptive local-global synergy, characterized in that: comprising the following steps: Step 1: Obtain the teacher model and student model: the original Logit vector z output for the same input sample. t and z s ; Step 2: Global semantic alignment: Align z t and z s Distribution standardization is performed to eliminate dimensional differences. The global distillation loss L between the two is calculated based on the temperature coefficient, and the overall output distribution of the student model is constrained to approximate that of the teacher model. Step 3: Local relationship modeling: Select Top-k category indexes according to the confidence output by the student model, construct a local Logit subspace corresponding to the indexes of the teacher-student model, obtain a local relationship loss L by calculating the divergence of the relative relationships between categories in the subspace, and enhance the distinguishing features between key categories; Step 4: Dynamic weight scheduling: Set the total number of training epochs E, adaptively adjust the local loss weight coefficient α(epoch) according to the current training progress, so that the model focuses on local detail fitting in the early training stage and focuses on global distribution convergence in the late training stage, thereby achieving a smooth transition; Step 5: Loss fusion and training: Fuse the cross-entropy loss, the global distillation loss and the local relationship loss weighted by α(epoch) to obtain a total loss, and update the parameters of the student model until convergence.
2. The method according to claim 1, characterized in that, In Step 2, the implementation manner of the global semantic alignment is as follows: For z t and z s Perform Z-score standardization on a sample-by-sample basis to obtain the standardized Logit vector ẑ t and ẑ s The KL divergence between the two is calculated based on the temperature coefficient T, and the global distillation loss L is obtained.
3. The method according to claim 2, characterized in that: The Z-score normalization processing formula is as follows: ẑ=(z−μ) / (σ+ε) where z is the original Logit vector, μ is the mean, σ is the standard deviation, and ε is a very small constant.
4. The method according to claim 1, characterized in that, In Step 3, the calculation manner of the local relationship loss is as follows: Among the selected top k category indexes, traverse all unique category pairs (i,j) that satisfy i<j, construct adaptive weights for the category pairs in the Top-k local category subspace, and calculate to obtain the local relationship loss.
5. The method according to claim 1, characterized in that, In Step 4, the dynamic weight coefficient α(epoch) adopts a piecewise adaptive mechanism, which decays nonlinearly with the training process, and the piecewise mechanism at least includes three stages: an early training stage, a middle training stage and a late training stage.
6. The method according to claim 5, characterized in that, The specific configuration of the dynamic weight coefficient α(epoch) is as follows: In the early training stage, α(epoch) maintains a relatively high value to promote local feature learning; In the middle training stage, α(epoch) decreases smoothly from high to low; In the late training stage, α(epoch) converges to a relatively low stable value to maintain global stability.
7. The method according to claim 6, characterized in that, The thresholds and convergence values of the dynamic weight coefficient at different stages can be adaptively adjusted according to the difficulty of a specific task and the architecture difference between the teacher model and the student model.
8. The method according to claim 1, characterized in that, The method can be migrated and adapted to the Decoupled Knowledge Distillation DKD framework, and during adaptation, the stage division ratio and weight convergence value in the piecewise mechanism are dynamically adjusted according to the characteristics of the target framework.
9. The method according to claim 1, characterized in that, The teacher model and the student model support ResNet, ShuffleNet, WideResNet or Vision Transformer architectures.
10. A system for implementing the knowledge distillation method based on adaptive local-global synergy as claimed in claims 1-9, characterized in that, The system comprises: a data preprocessing module, configured to perform enhancement and normalization processing on training data; a model management module, configured to load a pre-trained teacher model and initialize a student model; a forward inference module, configured to obtain original Logit outputs of the teacher model and the student model; a global distillation module, configured to perform Logit normalization and global KL divergence calculation; a local distillation module, configured to construct a local Logit subspace and calculate a local relationship loss; a dynamic scheduling module, configured to generate an adaptive weight coefficient according to the training epoch; an optimization module, configured to fuse multiple loss functions and update parameters of the student model.