Knowledge distillation training method and system of cross-architecture target detection model
By introducing feature relationship preservation distillation, prediction space alignment, and cross-detection head prediction mechanisms into cross-architecture object detection models, the stability and performance improvement issues in cross-architecture knowledge transfer are solved, achieving efficient knowledge transfer and detection performance improvement between heterogeneous models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-02-02
- Publication Date
- 2026-05-01
AI Technical Summary
Existing object detection knowledge distillation methods are difficult to perform stably in cross-architecture scenarios, leading to training oscillations or performance degradation. Moreover, the distillation effect is limited, especially when there are large differences in feature dimensions and distributions between heterogeneous models, making it difficult to effectively transfer the intermediate representation capabilities of the teacher model.
By introducing feature relationship preservation distillation, intermediate prediction branches, and cross-detector prediction mechanisms, we utilize Pearson correlation coefficients to align feature relationships, introduce removable exit branches for prediction space alignment, and combine real annotation supervision and distillation supervision to construct a cross-detector prediction mechanism to achieve cross-architecture knowledge transfer.
Without increasing computational and storage overhead during the inference phase, it improves the target detection performance of the student model, enhances the stability and detection accuracy of cross-architecture distillation, is applicable to a variety of heterogeneous model combinations, and has good versatility and practical application value.
Smart Images
Figure CN121962815A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a knowledge distillation training method and system for cross-architecture target detection models. Background Technology
[0002] Object detection is a crucial research area in computer vision, with wide applications in autonomous driving, intelligent monitoring, and industrial inspection. In recent years, with the development of deep learning technology, object detection models have evolved from traditional convolutional neural networks to the Transformer architecture, which incorporates a self-attention mechanism. Transformer-based object detection models can model the relationships between image features globally, resulting in a significant improvement in detection accuracy.
[0003] However, the Transformer architecture typically comes with a large parameter scale and computational complexity, which is often limited by computing power and power consumption in practical deployments. In contrast, object detection models based on convolutional neural networks have advantages such as high inference efficiency and low deployment cost, but their performance ceiling is relatively limited.
[0004] Knowledge distillation, as a model compression and performance transfer technique, typically introduces supervised information from the teacher model during the training phase to guide the student model to learn the knowledge from the teacher model, thereby improving the performance of the student model without increasing inference overhead.
[0005] Most existing object detection knowledge distillation methods are based on the premise that the teacher model and the student model have similar structures, and mainly include the following categories:
[0006] 1) The output layer-based distillation method achieves distillation by aligning the prediction results of the teacher model and the student model in classification or regression output;
[0007] 2) Feature-based distillation methods convey the representational power of the teacher model by aligning intermediate feature maps or feature vectors;
[0008] 3) Relationship-based distillation methods achieve knowledge transfer by constraining the relative relationships between samples or features.
[0009] The above methods are effective when the teacher and student models have similar structures, but they are often difficult to apply directly in cross-architecture scenarios and have the following drawbacks:
[0010] 1) When performing feature alignment directly between heterogeneous models, the distillation process is unstable due to the large differences in feature dimensions and distributions, which can easily lead to training oscillations or performance degradation.
[0011] 2) Distillation only at the output layer is insufficient to fully transfer the intermediate representation capabilities of the teacher model, resulting in limited distillation effectiveness;
[0012] 3) In object detection tasks, distillation supervision and supervision based on real annotations may act on the same prediction path, which can conflict with the model convergence. Summary of the Invention
[0013] This invention addresses the problem of effective knowledge distillation between existing heterogeneous network architecture target detection models by proposing a knowledge distillation training method and system for cross-architecture target detection models. By introducing feature relationship-preserving distillation, intermediate prediction branches, and cross-detector head prediction mechanisms, stable and efficient cross-architecture knowledge transfer is achieved without increasing computational and storage overhead in the inference stage, thereby improving the target detection performance of student models.
[0014] To achieve the above objectives, the present invention provides the following technical solution:
[0015] In a first aspect, the present invention provides a knowledge distillation training method for a cross-architecture target detection model, comprising the following steps:
[0016] Obtain a teacher object detection model and a student object detection model, wherein the teacher model and the student model are heterogeneous models using different network architectures;
[0017] During the training phase, at least one layer of intermediate features is extracted from the teacher model and the student model;
[0018] The first distillation step based on feature relation alignment: For the extracted intermediate features, calculate the Pearson correlation coefficient between the corresponding student features and teacher features at each layer, and construct the first distillation loss based on the Pearson correlation coefficient to constrain the student model to learn the feature relation structure of the teacher model;
[0019] The second distillation step based on prediction space alignment: a removable exit branch is introduced in at least one intermediate layer of the student model, the exit branch including a feature projector and a classifier, for mapping the intermediate features of the student model to a space with the same prediction space dimension as the teacher model, and generating a first prediction distribution; the difference between the first prediction distribution and the final prediction distribution of the teacher model is calculated to construct the second distillation loss;
[0020] The third distillation step based on cross-detector prediction alignment is as follows: at least one layer of intermediate features from the student model is input into the parameter-frozen detector head in the teacher model to generate a second prediction distribution; the difference between the second prediction distribution and the final prediction distribution of the teacher model is calculated to construct the third distillation loss.
[0021] Joint training steps: The detection loss based on real annotations and the third distillation loss are weighted and summed to obtain the total training loss, and the parameters of the student model are updated based on the total training loss;
[0022] Inference model generation steps: After training is complete, all exit branches in the student model are removed to obtain a final student model for inference that does not contain any distillation auxiliary structures.
[0023] Furthermore, the first distillation step based on feature relation alignment specifically includes:
[0024] The l-th layer feature maps extracted from the teacher model and the student model are respectively represented as follows: and ;
[0025] Each feature map is standardized by channel to obtain the standardized features:
[0026]
[0027] Where μ c and σ c Let represent the mean and standard deviation of the c-th channel, respectively;
[0028] Flatten the standardized features into vectors and calculate the Pearson correlation coefficient between the two vectors:
[0029]
[0030] in and This represents the mean of the corresponding vector;
[0031] First distillation loss The calculation formula is: .
[0032] Furthermore, in the second distillation step based on predicted spatial alignment, the second distillation loss... The calculation method is as follows:
[0033]
[0034] in, This represents the first predicted distribution of the output of the k-th exiting branch. This represents the final predicted distribution of the teacher model. Indicates the true label, Represents cross-entropy loss, Denotes KL divergence, K is the balance coefficient, and K is the total number of branches that exit.
[0035] Furthermore, the third distillation step based on cross-detector prediction alignment specifically includes:
[0036] (1) Selecting intermediate layer features: Selecting the i-th layer features from the feature pyramid network of the student model. As input;
[0037] (2) Input Teacher Detection Header: The input to the teacher model's detection head, which has its parameters frozen, is processed by its classification and regression subnetworks to generate cross-head prediction results. ;
[0038] (3) Calculate the distillation loss: compare it with the original prediction of the teacher model. Perform matching to construct the third distillation loss; the third distillation loss Including losses from classified distillation and regressive distillation loss ,in The cross-head prediction result, This represents the final predicted distribution of the teacher model.
[0039] Furthermore, in the joint training step, the total training loss The calculation formula is:
[0040]
[0041] in, For detection loss based on real annotations, For the third distillation loss, These are the weighting coefficients.
[0042] Furthermore, the network architecture differences between the teacher model and the student model are selected from one of the following combinations: convolutional neural networks with visual transformers, convolutional neural networks of different depths or widths, and target detectors with different feature pyramid designs.
[0043] Secondly, the present invention provides a knowledge distillation training system for cross-architecture target detection models, characterized in that it includes the following modules to implement the method described in any of the above-mentioned embodiments:
[0044] The model acquisition module is used to acquire teacher target detection models and student target detection models; the feature extraction module is used to extract intermediate features of the teacher model and the student model during the training phase.
[0045] The first distillation module is used to calculate the first distillation loss based on the extracted intermediate features using the Pearson correlation coefficient.
[0046] The second distillation module is used to calculate the second distillation loss through a removable exit branch set in the student model;
[0047] The third distillation module is used to calculate the third distillation loss by inputting student features into the teacher detection head;
[0048] The training module is used to jointly combine the detection loss and the third distillation loss to update the student model parameters;
[0049] The model optimization module is used to remove the exit branches in the student model after training is complete, generating the final inference model.
[0050] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described in any of the preceding claims.
[0051] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the preceding claims.
[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0053] This invention proposes a knowledge distillation training method for cross-architecture object detection models. This method introduces removable intermediate prediction branches into the student model and combines this with prediction supervision of intermediate features from the teacher model's detection head, constructing cross-detector head distillation supervision to achieve cross-architecture alignment at the prediction space level. Simultaneously, by constructing distillation constraints based on feature relationships, it avoids forced matching of feature values, enhancing the stability of cross-architecture distillation and facilitating integration with existing object detection training processes. The method is applicable to various combinations of teacher and student models, improving the detection performance of the student model without increasing the number of parameters and computational load in the inference stage. It possesses good versatility, a clear engineering implementation path, and practical application value. Attached Figure Description
[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.
[0055] Figure 1 The system architecture diagram of the knowledge distillation training method for cross-architecture target detection models provided in this embodiment of the invention is shown. Detailed Implementation
[0056] To better understand this technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.
[0057] A knowledge distillation training method for cross-architecture object detection models, with architectures such as... Figure 1 As shown, it includes the following steps:
[0058] Obtain a teacher object detection model and a student object detection model, wherein the teacher model and the student model are heterogeneous models using different network architectures;
[0059] During the training phase, at least one layer of intermediate features is extracted from the teacher model and the student model;
[0060] The first distillation step based on feature relation alignment: For the extracted intermediate features, calculate the Pearson correlation coefficient between the corresponding student features and teacher features at each layer, and construct the first distillation loss based on the Pearson correlation coefficient to constrain the student model to learn the feature relation structure of the teacher model;
[0061] The second distillation step based on prediction space alignment: a removable exit branch is introduced in at least one intermediate layer of the student model, the exit branch including a feature projector and a classifier, for mapping the intermediate features of the student model to a space with the same prediction space dimension as the teacher model, and generating a first prediction distribution; the difference between the first prediction distribution and the final prediction distribution of the teacher model is calculated to construct the second distillation loss;
[0062] The third distillation step based on cross-detector prediction alignment is as follows: at least one layer of intermediate features from the student model is input into the parameter-frozen detector head in the teacher model to generate a second prediction distribution; the difference between the second prediction distribution and the final prediction distribution of the teacher model is calculated to construct the third distillation loss.
[0063] Joint training steps: The detection loss based on real annotations and the third distillation loss are weighted and summed to obtain the total training loss, and the parameters of the student model are updated based on the total training loss;
[0064] Inference model generation steps: After training is complete, all exit branches in the student model are removed to obtain a final student model for inference that does not contain any distillation auxiliary structures.
[0065] The following is a detailed explanation of each step.
[0066] 1. First distillation based on feature relation alignment
[0067] In cross-architecture knowledge distillation scenarios, heterogeneous models exhibit significant differences in feature amplitude distribution, channel semantics, and receptive field coverage. Especially in object detection frameworks, the feature response scales of different FPN levels and channels vary considerably. If numerical matching losses such as mean squared error (MSE) are directly applied, the distillation process is easily dominated by a few high-amplitude channels, thereby weakening the effective supervisory signal.
[0068] Therefore, instead of forcing students to precisely reproduce the teacher's feature values, the focus is on the consistency of the internal relationship structure between the two features. Specifically, the Pearson Correlation Coefficient (PCC) is used as a feature distillation measure. By maximizing the linear correlation between teacher and student features, students are guided to learn the teacher's relative activation patterns, rather than absolute amplitudes.
[0069] The first distillation step based on feature relation alignment as described in this invention specifically includes:
[0070] The l-th layer feature maps extracted from the teacher model and the student model are respectively represented as follows: and ;
[0071] First, each feature map is standardized by channel to eliminate scale differences between different channels and levels, resulting in standardized features:
[0072]
[0073] Where μ c and σ c Let represent the mean and standard deviation of the c-th channel, respectively;
[0074] Then, the standardized features are flattened into vectors, and the Pearson correlation coefficient between the two vectors is calculated:
[0075]
[0076] in and This represents the mean of the corresponding vector;
[0077] Ultimately, the first distillation loss The calculation formula is: .
[0078] This loss is numerically equivalent to the MSE for zero-mean, unit-variance features, but semantically it emphasizes the consistency of feature relationship structure. Experimental results show that this strategy significantly outperforms direct feature matching methods among heterogeneous detector pairs such as Transformer–CNN.
[0079] 2. Second distillation based on prediction spatial alignment
[0080] While relation-preserving feature distillation can alleviate the scale mismatch problem between heterogeneous features, intermediate features inevitably still contain a large amount of inductive bias information related to the specific network structure, which to some extent limits the upper limit of cross-architecture knowledge transfer. In contrast, the logits space, located at the end of the model decision, carries information that is highly relevant to the specific task and relatively weakens architectural differences, making it more suitable for cross-architecture distillation.
[0081] Based on this observation, several removable exit branches are introduced into the student model to project intermediate layer features into the logits space and align them with the final predictions of the teacher model. The logits space contains the model's final prediction information, and the representations of models with different architectures in the logits space are relatively consistent. By performing knowledge transfer in the logits space, the challenge of feature space alignment can be bypassed, and the prediction information can be used directly.
[0082] Each exit branch consists of a lightweight feature projector and a classifier, which are jointly optimized with the student backbone network during the training phase and completely removed during the inference phase, thus introducing no additional inference overhead.
[0083] The specific implementation involves introducing an exit branch into the student model. A feature projector projects the intermediate features of the student model onto the Logits space, while a classifier generates the predicted probabilities. The Logits output of the exit branch is matched with the Logits output of the teacher model, using KL divergence as the loss function to measure the difference in their distributions.
[0084] In the second distillation step based on prediction spatial alignment of the present invention, let the output logits of the k-th exit branch be... The final logits of the teacher model are The corresponding predicted distributions are as follows: and The second distillation loss The calculation method is as follows:
[0085]
[0086] in, This represents the first predicted distribution of the output of the k-th exiting branch. This represents the final predicted distribution of the teacher model. Indicates the true label, Represents cross-entropy loss, Denotes KL divergence, K is the balancing coefficient used to balance real-label supervision and distillation supervision, and K is the total number of branches that exit.
[0087] For CNN detectors with a pyramid structure, exit branches are inserted at the end of each stage; for non-hierarchical models such as Transformers, the network depth is evenly divided into several stages and corresponding branches are set. This design allows intermediate representations of different depths to directly benefit from teacher predictions, effectively alleviating the problem of feature space alignment difficulties in heterogeneous architectures.
[0088] In the logits space, redundant architecture-specific information is removed, thus enabling feature alignment across any architecture.
[0089] 3. Third distillation based on cross-detector prediction alignment
[0090] In object detection tasks, student models are simultaneously constrained by both ground truth label supervision and teacher prediction distillation supervision during training. However, the teacher model's predictions are not entirely accurate, especially when the teacher and student models employ different label assignment strategies, often resulting in significant discrepancies. This discrepancy leads to conflicting supervision signals received by the student model. Specifically, the teacher model's predicted probability distribution deviates from the ground truth label in certain regions, particularly for targets with blurred boundaries or severe occlusion. Furthermore, when attempting to match both teacher predictions and ground truth targets simultaneously, the student model's gradient update directions may cancel each other out, leading to training instability and limited performance improvement.
[0091] To alleviate the aforementioned problems, this invention employs a cross-head prediction mechanism, decoupling detection supervision and distillation supervision into different computational paths. Specifically, the detection head parameters of the teacher model are fixed, and the intermediate features of the student model are input into this detection head to generate a cross-head prediction result. Since this prediction depends on both student features and the teacher head structure, its distribution is semantically closer to the teacher prediction while maintaining consistency with the student representation.
[0092] The third distillation step based on cross-detector prediction alignment described in this invention specifically includes:
[0093] (1) Selecting intermediate layer features: Selecting the i-th layer features from the feature pyramid network of the student model. As input;
[0094] (2) Input Teacher Detection Header: The input to the teacher model's detection head, which has its parameters frozen, is processed by its classification and regression subnetworks to generate cross-head prediction results. ;
[0095] (3) Calculate the distillation loss: compare it with the original prediction of the teacher model. Perform matching to construct the crosshead distillation loss (third distillation loss):
[0096]
[0097] The third distillation loss Including losses from classified distillation and regressive distillation loss ,in The cross-head prediction result, This represents the final predicted distribution of the teacher model.
[0098] Meanwhile, the student model's original detection head still only accepts supervision based on standard detection loss according to real-world annotations.
[0099] Therefore, in the joint training step, the total training loss The calculation formula is:
[0100]
[0101] in, For detection loss based on real annotations, For the third distillation loss, These are the weighting coefficients.
[0102] The detection loss (based on the true target) and distillation loss (based on the teacher prediction) are applied to different branches to avoid direct conflict between them. During backpropagation, the gradient of the detection loss propagates through the entire detection head of the student model, while the gradient of the distillation loss propagates through the frozen teacher layer to the latent features of the student model, thereby improving the consistency between the two.
[0103] This design introduces a structured and consistent distillation signal while maintaining the stability of the original detection training. Experimental results show that cross-detector prediction can significantly improve the performance of the student model in heterogeneous distillation scenarios.
[0104] The differences in network architecture between the teacher model and the student model described in this invention are selected from one of the following combinations: convolutional neural networks and visual transformers, convolutional neural networks of different depths or widths, and target detectors with different feature pyramid designs.
[0105] Secondly, the present invention provides a knowledge distillation training system for cross-architecture target detection models, characterized in that it includes the following modules to implement the method described in any of the above-mentioned embodiments:
[0106] The model acquisition module is used to acquire teacher target detection models and student target detection models; the feature extraction module is used to extract intermediate features of the teacher model and the student model during the training phase.
[0107] The first distillation module is used to calculate the first distillation loss based on the extracted intermediate features using the Pearson correlation coefficient.
[0108] The second distillation module is used to calculate the second distillation loss through a removable exit branch set in the student model;
[0109] The third distillation module is used to calculate the third distillation loss by inputting student features into the teacher detection head;
[0110] The training module is used to jointly apply the detection loss, first distillation loss, second distillation loss, and third distillation loss to update the student model parameters.
[0111] The model optimization module is used to remove the exit branches in the student model after training is complete, generating the final inference model.
[0112] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described in any of the preceding claims.
[0113] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the preceding claims.
[0114] Experimental Example
[0115] 1. Experimental setup
[0116] 1.1 Dataset and Evaluation Metrics
[0117] The effectiveness of the proposed method was validated on the MS COCO 2017 dataset, where the training set contains 118k images and the validation set contains 5k images. All experiments adopted the official COCO evaluation protocol, and the standard mean accuracy (AP) was reported as the primary evaluation metric.
[0118] The COCO dataset is a large-scale benchmark dataset widely used in the field of computer vision, containing over 330,000 images, 200,000 of which are detailedly annotated. Each image contains an average of 7.7 instance annotations, covering object bounding boxes, segmentation masks, keypoint information, and five sentences of image description text. The dataset is divided into 80 object categories, including people, bicycles, cars, animals (such as cats, dogs, and elephants), everyday items (such as backpacks and umbrellas), and sports equipment (such as skateboards and tennis rackets); and 91 material categories, such as grass, walls, and sky, providing contextual information.
[0119] 1.2 Model Selection
[0120] Experiments were conducted using several representative heterogeneous teacher-student combinations. Teacher models included powerful Transformer-based detectors such as SwinTransformer-Tiny and Co-DINO; student models employed efficient and widely used CNN detectors, such as RetinaNet-ResNet50.
[0121] 1.3 Implementation Details
[0122] The experiments were implemented using MMDetection and PyTorch. All models were initialized with ImageNet pre-trained weights and trained end-to-end on COCO. Except for distillation-related modules, all other training configurations (such as learning rate, number of training epochs, and data augmentation strategies) were consistent with the corresponding official implementations to ensure fair comparison.
[0123] Within the framework, incorporating additional exit branches into the student model requires determining appropriate branch insertion points. For models with a pyramid structure, the end of each stage is considered a potential insertion point, resulting in a total of four points. For other models such as ViT, it is divided into four equal parts, and the end of each part is designated as a feasible insertion point for the branch. The exit branches of CNNs are constructed using depth-width convolutional layers, while the exit branches of ViT and MLP, etc., use ViT blocks. In the implementation, different optimizers are used to train the student model according to its architecture.
[0124] 2. Comparison with existing methods
[0125] First, the proposed cross-architecture distillation method is compared with several mainstream distillation strategies, including:
[0126] Logits-based KD: KL distillation is performed only at the output layer;
[0127] Feature-based KD (MSE): Directly aligns intermediate features;
[0128] PKD: A general detection distillation method based on Pearson correlation coefficient.
[0129] Specifically, logic-based original KD methods and feature-based methods include FitNet and popular PKD methods.
[0130] In the original KD method, after the teacher model training converges, the unnormalized scores (Logits) of the output layer contain not only the correct category, but also information on the similarity and correlation between categories. This is the "hidden knowledge" that improves generalization ability, which cannot be reflected by standard hard labels and is used to guide student learning.
[0131] step:
[0132] Soft label generation: Introduce a temperature coefficient T and perform Softmax(z / T) processing on the teacher's Logits. When T>1, the distribution is smoother, amplifying the probability of incorrect categories and making tacit knowledge easier for students to learn, thus obtaining soft labels.
[0133] Dual-loss training: The student model learns two supervision signals simultaneously and optimizes its own parameters using the total loss.
[0134] Soft loss: Using KL divergence, the softened distribution of students is fitted to the soft labels of teachers to learn hidden knowledge.
[0135] Hard loss: Using cross-entropy, students fit the true labels of the dataset, ensuring basic classification accuracy.
[0136] After training, the student model removes the temperature T during inference and uses standard Softmax to output results, which are consistent with the inference of the ordinary model.
[0137] FitNet's teacher model's intermediate feature layers (Hint layers) contain hierarchical abstract representations of the data, which is key knowledge not utilized by the original KD model. The student model learns feature mapping capabilities by aligning the output of its corresponding intermediate layer (Guided layer) with the teacher's Hint layer through the Adaptation Layer.
[0138] step:
[0139] First, select a certain intermediate layer for the teacher as the Hint layer and the corresponding layer for the student as the Guided layer. Unify the feature dimension through the adaptation layer and use MSE loss for feature pre-training so that students can learn high-quality feature representation first.
[0140] Then, the output is fine-tuned using the soft label loss from the original knowledge distillation plus the hard label loss, ultimately resulting in a more powerful lightweight model.
[0141] The PKD method, the process is as follows:
[0142] Pre-trained teacher model, with fixed parameters, outputs intermediate / output layer features.
[0143] The student model undergoes forward propagation to obtain the corresponding features / output.
[0144] Calculate the Pearson correlation coefficient of the corresponding features of teachers and students, construct the distillation loss, and optimize the student using the joint task loss (such as cross-entropy).
[0145] Key metric: Pearson coefficient ρ(X,Y), with a value of [-1,1], measures the strength of the linear correlation between variables. It is invariant to the feature scale / bias and avoids the strict numerical constraints of MSE / KL.
[0146] Loss Construction: Commonly Used As distillation loss (the closer ρ is to 1, the closer the loss is to 0).
[0147] Table 1. Teacher Swin-T(T) → Student Retinanet(R50)
[0148]
[0149] The results show that, under the same teacher model, the proposed method outperforms existing distillation strategies in overall detection performance and enables the student model to achieve a performance level comparable to that of the teacher.
[0150] Cross-architecture distillation under 3 strong faculty
[0151] Most existing methods are designed for homogeneous detection pairs. Our proposed method is general enough to extract knowledge between homogeneous and heterogeneous detector pairs. To further validate the effectiveness of our method in high-performance teacher scenarios, we use Co-DINO as the teacher model to distill RetinaNet-R50. A stronger heterogeneous teacher detector is used, as shown in Table 2. Compared to Table 1, the student detector clearly performs better under the stronger teacher detector; for example, the Retina-Res50 model achieves higher performance under Co-DINO than classic logit distillation and PKD distillation methods. The results demonstrate that mimicking the feature maps of a stronger heterogeneous teacher detector can further improve student performance when applying this method.
[0152] Table 2 Teacher Co-DINO → Student Retinanet-R50
[0153]
[0154] It can be observed that the performance of the student model continues to improve with the gradual introduction of different distillation modules, verifying the complementarity among the three distillation mechanisms.
[0155] 4 Ablation Experiment
[0156] Further ablation analysis was conducted on each core component to quantify its independent contribution.
[0157] Table 3 Comparison of different characteristic distillation metrics (Swin-T → RetinaNet-R50)
[0158]
[0159] The results show that the relationship based on the Pearson correlation coefficient is more advantageous in heterogeneous scenarios.
[0160] For the teacher-student Co-DINO → student Retinanet (R50) pair, the individual distillation strategies were analyzed separately, and the results are summarized in Table 4.
[0161] Table 4. Individual comparison of different distillation modules (Co-DINO → Retinanet-R50)
[0162]
[0163] The results show that all distillation strategies can continuously improve the performance of the RetinaNet-R50 student model, which verifies the effectiveness of knowledge transfer in the Co-DINO teacher model.
[0164] In a single-module distillation setup, a cross-architecture aligned distillation strategy based on logits space ( The method exhibits the most significant performance improvement, with an average accuracy (AP) of 44.2, which significantly outperforms feature-level distillation and logit-based distillation methods, indicating that this strategy is more effective than forcing intermediate feature consistency, especially in heterogeneous architectures.
[0165] For detection tasks, logits themselves lack sufficient information and cannot fully convey localization, scale, and contextual information, thus limiting their knowledge transfer capabilities. Intermediate layer distillation is more effective in comparison, but the improvement is limited by architectural differences (Co-DINO vs RetinaNet), feature semantic misalignment, and other inherent flaws, resulting in certain performance bottlenecks.
[0166] Furthermore, although distillation can be predicted using cross-detection heads alone ( While the effect was relatively poor at the time, Table 2 shows that it can still provide significant gains, indicating that it has a complementary role with other modules in adjusting the consistency of predictions across heads.
[0167] 5. Discussion
[0168] Experimental results show that:
[0169] 1. Single distillation strategies offer limited benefits among heterogeneous detectors, while multi-level, decoupled distillation can significantly improve performance;
[0170] 2. Logits spatial distillation plays a key bridging role in cross-architecture knowledge transfer;
[0171] 3. Cross-detection head prediction effectively alleviates the conflict between distillation supervision and real labeling in detection tasks.
[0172] The above results validate the effectiveness and stability of the proposed method in cross-architecture target detection distillation.
[0173] 6. Analysis of computational overhead and inference efficiency
[0174] The proposed cross-architecture distillation framework introduces additional distillation modules only during the training phase, including FPN feature relation constraints, logits space exit branches, and cross-detector prediction paths.
[0175] During the inference phase, all auxiliary branches and teacher-related modules were completely removed, and the network structure of the student model remained consistent with the baseline detector.
[0176] The modules in the method are analyzed one by one as follows:
[0177] (1) Feature distillation based on Pearson coefficients: calculated only in training loss; no new convolutional layers are introduced; and it does not participate in the inference graph.
[0178] (2) Exiting the branch: As an auxiliary branch, it is completely removed during reasoning → Params / FLOPs for reasoning = 0
[0179] (3) Cross-head prediction: The frozen teacher detection head is used only for calculating distillation loss; the teacher is not considered during inference.
[0180] The number of parameters, computational cost, and inference overhead of the student model before and after distillation (taking RetinaNet-ResNet50 as an example) are compared in Table 5.
[0181] Table 5 Comparison of computational resources before and after distillation
[0182]
[0183] As can be seen, the proposed method significantly improves detection performance without introducing additional inference costs and with only a limited increase in computation, making it highly feasible for practical deployment scenarios. The FLOPs statistics mentioned above are based on the COCO standard input resolution (800×1333), consistent with mainstream object detection work. It should be noted that the Transformer teacher model used in the experiment has a significantly larger parameter scale and computational cost than the student detector, which further highlights the practical value of transferring its knowledge to an efficient CNN model.
[0184] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A knowledge distillation training method for a cross-architecture object detection model, characterized in that, Includes the following steps: Obtain a teacher object detection model and a student object detection model, wherein the teacher model and the student model are heterogeneous models using different network architectures; During the training phase, at least one layer of intermediate features is extracted from the teacher model and the student model; The first distillation step based on feature relation alignment: For the extracted intermediate features, calculate the Pearson correlation coefficient between the corresponding student features and teacher features at each layer, and construct the first distillation loss based on the Pearson correlation coefficient to constrain the student model to learn the feature relation structure of the teacher model; The second distillation step based on prediction space alignment: a removable exit branch is introduced in at least one intermediate layer of the student model, the exit branch including a feature projector and a classifier, for mapping the intermediate features of the student model to a space with the same prediction space dimension as the teacher model, and generating a first prediction distribution; the difference between the first prediction distribution and the final prediction distribution of the teacher model is calculated to construct the second distillation loss; The third distillation step based on cross-detector prediction alignment is as follows: at least one layer of intermediate features from the student model is input into the parameter-frozen detector head in the teacher model to generate a second prediction distribution; the difference between the second prediction distribution and the final prediction distribution of the teacher model is calculated to construct the third distillation loss. Joint training steps: The detection loss based on real annotations and the third distillation loss are weighted and summed to obtain the total training loss, and the parameters of the student model are updated based on the total training loss; Inference model generation steps: After training is complete, all exit branches in the student model are removed to obtain a final student model for inference that does not contain any distillation auxiliary structures.
2. The knowledge distillation training method for cross-architecture target detection models according to claim 1, characterized in that, The first distillation step based on feature relation alignment specifically includes: The l-th layer feature maps extracted from the teacher model and the student model are respectively represented as follows: and ; Each feature map is standardized by channel to obtain the standardized features: ， Where μ c and σ c Let represent the mean and standard deviation of the c-th channel, respectively; Flatten the standardized features into vectors and calculate the Pearson correlation coefficient between the two vectors: ， in and This represents the mean of the corresponding vector; First distillation loss The calculation formula is: .
3. The knowledge distillation training method for cross-architecture target detection models according to claim 1, characterized in that, In the second distillation step based on predicted spatial alignment, the second distillation loss The calculation method is as follows: ， in, This represents the first predicted distribution of the output of the k-th exiting branch. This represents the final predicted distribution of the teacher model. Indicates the true label, Represents cross-entropy loss, Denotes KL divergence, K is the balance coefficient, and K is the total number of branches that exit.
4. The knowledge distillation training method for cross-architecture target detection models according to claim 1, characterized in that, The third distillation step based on cross-detector prediction alignment specifically includes: (1) Selecting intermediate layer features: Selecting the i-th layer features from the feature pyramid network of the student model. As input; (2) Input Teacher Detection Header: The input to the teacher model's detection head, which has its parameters frozen, is processed by its classification and regression subnetworks to generate cross-head prediction results. ; (3) Calculate the distillation loss: compare it with the original prediction of the teacher model. Perform matching to construct the third distillation loss; the third distillation loss Including losses from classified distillation and regressive distillation loss ,in The cross-head prediction result, This represents the final predicted distribution of the teacher model.
5. The knowledge distillation training method for cross-architecture target detection models according to claim 1, characterized in that, In the joint training step, the total training loss The calculation formula is: ， in, For detection loss based on real annotations, For the third distillation loss, These are the weighting coefficients.
6. The knowledge distillation training method for cross-architecture target detection models according to claim 1, characterized in that, The differences in network architecture between the teacher model and the student model are selected from one of the following combinations: convolutional neural networks with visual transformers, convolutional neural networks of different depths or widths, and target detectors with different feature pyramid designs.
7. A knowledge distillation training system for cross-architecture object detection models, characterized in that, The following modules are included to implement the method of any one of claims 1-6: The model acquisition module is used to acquire teacher target detection models and student target detection models; the feature extraction module is used to extract intermediate features of the teacher model and the student model during the training phase. The first distillation module is used to calculate the first distillation loss based on the extracted intermediate features using the Pearson correlation coefficient. The second distillation module is used to calculate the second distillation loss through a removable exit branch set in the student model; The third distillation module is used to calculate the third distillation loss by inputting student features into the teacher detection head; The training module is used to jointly apply the detection loss, first distillation loss, second distillation loss, and third distillation loss to update the student model parameters. The model optimization module is used to remove the exit branches in the student model after training is complete, generating the final inference model.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.