A knowledge distillation method common to DETR type detectors

Abstract

The application discloses a knowledge distillation method for DETR type detectors, which comprises the following steps: delivering a pretreated image to a student model which has been trained to output a prediction result; the training process of the student model is as follows: S1: constructing a training set; S2: selecting a teacher model and a student model, and fully training the teacher model to convergence; S3: delivering the images in the training set to the teacher model and the student model simultaneously; S4: performing logits-level distillation on the student model by the teacher model; S5: performing feature-level distillation on the student model by the teacher model; and S6: performing assignment-level distillation on the student model by the teacher model. The knowledge distillation method can accurately transfer the knowledge contained in the teacher model to the student model, and improves the detection performance of the DETR detector.

Description

Technical Field

[0001] This invention relates to the field of target detection technology in machine vision, and more particularly to a knowledge distillation method applicable to DETR-type detectors. Background Technology

[0002] Knowledge distillation is a technique used to reduce the number of parameters in a large deep neural network (DNN) model and improve its inference speed. It aims to transfer knowledge from a large model to a smaller model, allowing the smaller model to perform similar tasks with fewer parameters and higher efficiency.

[0003] In the process of knowledge distillation, knowledge from a large model (teacher model) is typically transferred to a smaller model (student model). The teacher model is usually a pre-trained deep neural network that has been trained on a large amount of data and learned the complex patterns needed to solve a specific task. The student model is a smaller network whose goal is to learn the knowledge from the teacher model in order to perform efficient reasoning with limited computing resources and memory.

[0004] The core idea of ​​knowledge distillation is to transfer knowledge by comparing the output probability distributions of the teacher model and the student model. Besides comparing output probability distributions, other methods can be used to transfer knowledge. For example, using the feature space representation of the teacher model to guide the learning process of the student model can help the student model better capture complex patterns in the data, thereby improving its performance.

[0005] Knowledge distillation is a highly effective model compression and acceleration technique that can reduce model size and increase inference speed without sacrificing performance. This technique is particularly useful for deploying deep learning models on resource-constrained platforms such as mobile devices, embedded systems, and IoT devices.

[0006] However, knowledge distillation also presents some challenges and limitations. For example, differences between the teacher and student models can lead to incomplete or inaccurate knowledge transfer. Furthermore, since knowledge distillation requires additional training steps, it can increase the training time and computational cost of the model. Summary of the Invention

[0007] Based on the technical problems existing in the background technology, this invention proposes a knowledge distillation method applicable to DETR-type detectors, which reduces the difference between the teacher model and the student model, can transfer the knowledge of the teacher model to the student model more accurately, and improves the detection performance of DETR detectors with a small number of parameters.

[0008] This invention proposes a knowledge distillation method applicable to DETR-type detectors, which feeds a preprocessed image into a trained student model to output prediction results;

[0009] The training process of the student model is as follows:

[0010] S1: Construct a training set, which includes multiple preprocessed images;

[0011] S2: Select the teacher model and student model, train the teacher model until it converges, and freeze the parameters of the teacher model. The teacher model usually includes a feature extraction network, a feature fusion pyramid network, an encoder, and a decoder.

[0012] S3: Feed the images from the training set to the teacher model and the student model;

[0013] S4: Transfer the knowledge output from the decoder of the teacher model to the decoder output of the student model using an incremental distillation method, that is, perform logits-level distillation on the student model through the teacher model;

[0014] S5: Temporarily store the feature map output by the encoder of the teacher model and the teacher query predicted by the decoder, and calculate the foreground and background regions of each feature map to realize the feature-level distillation of the student model by the teacher model.

[0015] S6: Save the pre-trained teacher query in the teacher model and collect the sample matching results between the teacher query and the ground truth in each batch online. Input the teacher query as a set of additional training samples into the decoder part of the student model and provide the assignment-level distillation loss function as supervision to realize the assignment-level distillation of the student model by the teacher model.

[0016] S7: Train the student model based on logits-level distillation, feature-level distillation, and assignment-level distillation until the student model converges.

[0017] Furthermore, step S4 specifically includes:

[0018] The prediction results output by the teacher model decoder are divided into positive and negative samples according to minimizing the loss: The student model's prediction is expressed as: in and M represents the prediction result set of the teacher model and the student model. pos and M negThese represent the number of positive and negative samples, respectively. N represents the number of queries for the student model, and T... pos T represents the positive sample partition. neg Indicates negative sample partitioning. This represents the positive sample prediction results of the teacher model. This indicates the prediction result of the teacher model for negative samples. This represents the prediction result for each query in the student model;

[0019] Using the training loss function built into the DETR detector. match Perform Hungarian matching on the positive and negative sample sets respectively to obtain and Matching relationships are used to derive the initial loss function for logits-level distillation. in express Loss function under matching relationship express Loss function under matching relationship;

[0020] Both the teacher model and the student model decoders contain multiple cascaded stages. A progressive distillation method is used to transfer the decoder outputs of different stages of the teacher model to the decoder outputs of different stages of the student model.

[0021] The formula for the incremental distillation method is as follows:

[0022]

[0023] Where K is the total number of decoder layers, k represents the current layer of the decoder, and L[k] represents the distillation loss at the corresponding layer.

[0024] Furthermore, step S5 specifically includes:

[0025] The feature map output by the encoder of the teacher model and the teacher query predicted by the decoder are temporarily stored.

[0026] After aligning the feature maps and teacher query dimensions, perform an inner product operation to generate M similarity heatmaps ψ related to each query. i ;

[0027] The output of the teacher model is then analyzed using a weighted similarity heatmap ψ based on the quality coefficient q. i And M similarity heatmaps ψ i The fusion yields a weighted mask image;

[0028] Based on the weighted mask image as the loss weight of the feature-level distillation loss function, the loss function L of feature-level distillation is calculated. featKD ;

[0029] Loss function L featKD The calculation formula is as follows:

[0030]

[0031] Where M = M pos +M neg M represents the total number of teacher queries, F T ∈R H×W×d and These represent the feature maps generated by the teacher model and the student model, respectively. H and W are the height and width of the feature map, respectively, and d represents the number of channels in the teacher feature. S Represents the feature dimensions of the student model. Let R represent a learnable adaptive feature dimension mapping layer to align the feature maps of the student model, ψ∈R. H×W This represents a weighted mask image that divides the foreground and background regions, q i Let q represent the quality coefficient corresponding to the i-th query in the teacher model, and ⊙ represent the product.

[0032] Furthermore, in step S6, the loss function for assignment-level distillation is specifically as follows:

[0033]

[0034] Among them, L assignKD This represents the assignment-level distillation loss function. This represents the matching results between teacher queries and ground truth samples, where M represents the total number of teacher queries, and M = M pos +M neg y represents the ground truth sample, y i This represents the i-th ground truth sample. express Student model prediction under matching.

[0035] Furthermore, when performing assignment-level distillation, the motivation and problem to be solved by distillation are visualized. Specifically, the instability IS of positive and negative sample matching in the teacher model, the original student model, and the student model after distillation during the training process is defined and statistically analyzed. It is found that the teacher query and GT matching in the teacher model are relatively stable in different decoding layers. The student model has chaotic matching and slow convergence at the beginning of training. The training stability of the student model is improved after assignment-level distillation.

[0036] Furthermore, the preprocessed image is an image that has undergone data augmentation and cropping, and the data augmentation method is random horizontal flipping / random rotation.

[0037] Furthermore, in the actual use of the student model after training, the network structures corresponding to the teacher model, logits-level distillation, feature-level distillation, and assignment-level distillation will be deleted, and the optimized student model with a small number of parameters will be retained to perform inference on the preprocessed image to output prediction results.

[0038] The advantages of the knowledge distillation method for DETR detectors provided by this invention are as follows: This knowledge distillation method reduces the difference between the teacher model and the student model, and can transfer the knowledge contained in the teacher model to the student model more accurately, thereby improving the detection performance of the DETR detector; Specifically: (1) Logits-level distillation solves the uncertainty problem of matching between the teacher model and the student model, and adopts a progressive strategy to adapt to the cascaded detection head in DETR; (2) Feature-level distillation proposes to use a soft label mask map based on similarity as a foreground and background partitioning strategy in the feature map, which helps the student model to more accurately locate the knowledge that needs to be learned; (3) Assignment-level distillation proposes to use the stable prediction and GT correspondence of the pre-trained teacher model, and at the same time help the student model to converge better by adding extra samples. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the framework of the present invention;

[0040] Figure 2 A visual illustration illustrating the different activation regions of CNNs and attention mechanisms;

[0041] Figure 3 This is a visual diagram illustrating the relationship between teacher query and feature similarity. Figure 3In the image, (a) is the input image, (b), (c), and (d) are similarity heatmaps corresponding to teacher query queries for foreground targets, and (e) and (f) are heatmaps corresponding to background noise.

[0042] Figure 4 This is a flowchart illustrating the feature-level distillation process of the teacher model on the student model.

[0043] Figure 5 This is a statistical graph of sample matching instability (IS). Detailed Implementation

[0044] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0045] Currently, knowledge distillation methods in the field of detection models are almost all based on detectors with convolutional neural network (CNN) structures. Recently, DETR series detectors based on the Transformer structure have become a research hotspot. However, previous distillation work may not be suitable for this new structure. This invention designs a general distillation framework for DETR series detectors to fully leverage the characteristics of the model itself for knowledge transfer. Based on this, this embodiment proposes a knowledge distillation method applicable to DETR-type detectors. Compared with the current mainstream CNN-based distillation methods, this knowledge distillation method can achieve a more powerful and better generalization model for DETR detectors, as detailed below.

[0046] like Figures 1 to 5 As shown, this invention proposes a knowledge distillation method applicable to DETR-type detectors, which feeds a pre-processed image into a trained student model to output prediction results. This embodiment introduces multiple outputs from a pre-trained teacher model as additional supervision signals during the student model's training process to help improve the student model's performance and achieve model compression.

[0047] This paper theoretically summarizes the structural similarities and differences between CNN detectors and DETR detectors, including aspects such as feature extraction, positive and negative sample matching during training, and detector head design. It also analyzes and summarizes potential new difficulties encountered by DETR detectors during the distillation process. Furthermore, it incorporates prior experiments and visualization techniques. Figure 2 The method shown revealed the failure of the distillation strategy of the CNN detector on the DETR detector. By further analyzing the reasons, an improvement strategy was proposed.

[0048] Based on the findings and experience summarized above, a general distillation framework for DETR detectors is proposed. This framework can be inserted into small-parameter detectors with any Transformer architecture during training to achieve enhanced performance. The distillation process only occurs during model training. During the inference phase after training, the distilled network and teacher model are removed, retaining only the optimized small-parameter student model. Therefore, no additional custom structures or computational overhead are introduced during deployment. In summary, this embodiment summarizes the differences between CNN detectors and DETR detector architectures and the resulting difficulties in model distillation. Furthermore, experiments demonstrate the incompatibility of previous methods with the DETR architecture.

[0049] In the logits-level distillation task, the DETR detector lacks a direct and explicit correspondence between the predictions of the teacher and student models. CNN detectors do not have this problem because they use a fully convolutional structure, where the convolutional operations have the characteristic of local sliding window movement, thus naturally establishing a one-to-one spatial correspondence between the teacher and student model predictions. However, since the DETR detector uses a query-based decoder layer as its detection head, and the query is an unordered set, the relationship between the teacher and student model predictions needs to be established before applying the distillation loss function. Secondly, in feature-level distillation, foreground and background regions are distinguished, and different weights are assigned to them in the distillation loss to prevent noise from the background regions from overwhelming the foreground regions. Currently, different distillation methods have different strategies for foreground and background region segmentation. Prior experiments have shown that the foreground regions activated by the convolutional detection head in CNN detectors are significantly different from those generated by the DETR detection head. Therefore, the DETR detector needs to adapt its attention mechanism's foreground and background segmentation strategy. Finally, the query and decoder parameters in the DETR detector are randomly initialized before training begins. Therefore, unstable sample matching in the early stages leads to a somewhat ambiguous distinction between positive and negative samples used in training, resulting in slow model convergence. However, empirically, trained queries in the teacher model typically produce consistent matching results with the ground truth (GT). Introducing this into the student model within the distillation framework can better aid model convergence.

[0050] Therefore, in order to enable the student model to acquire more knowledge from the teacher model during training, the training process of the student model is as follows:

[0051] S1: Construct a training set, which includes multiple preprocessed images;

[0052] S2: Select a teacher model and a student model, train the teacher model until it converges, and freeze the parameters of the teacher model. The teacher model typically includes a feature extraction network, a feature fusion pyramid network, an encoder, and a decoder connected in sequence.

[0053] The feature extraction network, feature fusion pyramid network, encoder, and decoder can adopt existing network structures. The feature extraction network and feature fusion pyramid network are mainly used to fuse the features extracted from the image, while the encoder and decoder are mainly used to encode and decode the extracted and fused data.

[0054] Select the required large-parameter DETR teacher model and the small-parameter DETR student model to be distilled, and pre-train the teacher model to convergence. Construct the overall distillation framework: during the training of the student model from scratch, use the pre-trained teacher model as an additional branch to infer the input data. Here, the parameters of the teacher model need to be frozen. The overall distillation framework is as follows: Figure 1 As shown.

[0055] S3: Feed the images from the training set to the teacher model and the student model;

[0056] The images in the training set are preprocessed images, which are images that have undergone data augmentation and cropping. The data augmentation methods include random horizontal flipping / random rotation, etc.

[0057] S4: Transfer the knowledge output of the decoder of the teacher model to the decoder output of the student model based on the progressive distillation method, that is, perform logits-level distillation on the student model through the teacher model, specifically including steps S41 to S43.

[0058] S41: DETR-type detectors struggle to establish an ordered one-to-one correspondence between teacher and student model predictions. To address this issue, the Hungarian matching algorithm is reused to perform a one-to-one matching between teacher and student model predictions based on minimizing loss. Once the matching relationship is obtained, the student model can apply knowledge from the teacher's predictions. In form:

[0059] The prediction results output by the teacher model decoder are divided into positive and negative samples according to minimizing the loss: The student model's prediction is expressed as: in and M represents the prediction result set of the teacher model and the student model. pos and M neg These represent the number of positive and negative samples, respectively. N represents the number of queries for the student model, and T... pos T represents the positive sample partition.neg Indicates negative sample partitioning. This represents the positive sample prediction results of the teacher model. This represents the prediction results of the teacher model for negative samples. This represents the prediction result for each query in the student model;

[0060] S42: Use the training loss function L built into the DETR detector. match Perform Hungarian matching on the positive and negative sample sets respectively to obtain and Matching relationships are used to derive the initial loss function for logits-level distillation. in express Loss function under matching relationship express The loss function under the matching relationship is the same as the initial distillation loss function and the matching part loss function, which ensures that the cost of matching the sample with the GT is minimized in each batch.

[0061] S43: Considering that DETR decoders typically consist of multiple stages in a cascaded structure, knowledge can be transferred from the teacher model to the decoder output of the student model stage by stage. The parameters of the teacher model at different stages contain different knowledge; a progressive extraction method is beneficial for fully learning the knowledge distribution at different stages. The progressive distillation expression is as follows:

[0062]

[0063] Where K is the total number of decoder layers, k represents the current layer of the decoder, and L[k] represents the distillation loss at the corresponding layer.

[0064] By using the progressive distillation strategy on the student model through steps S41 to S43, we achieve greater improvement compared to simply using the final stage output of the teacher model.

[0065] S5: Temporarily store the feature map output by the encoder of the teacher model and the teacher query predicted by the decoder, and calculate the foreground and background regions of each feature map to realize the feature-level distillation of the teacher model to adapt to the student model. Specifically, this includes steps S51 to S53.

[0066] S51: Obtain a batch of images that have undergone data augmentation, including random horizontal flipping and random rotation, and simultaneously feed them into the teacher model and student model. Here, it is necessary to temporarily store the feature map of the teacher model after the encoder and the teacher query predicted by the decoder.

[0067] S52: Typically, the feature-level distillation loss function L featKD It can be summarized into the following form:

[0068]

[0069] Where F T ∈R H×W×d and These represent the feature maps generated by the teacher model and the student model, respectively. H and W are the height and width of the feature map, respectively, and d represents the number of channels in the teacher feature. S Represents the feature dimensions of the student model. Let R represent a learnable adaptive feature dimension mapping layer to align the feature maps of the student model, ψ∈R. H×W This represents a weighted mask that divides the foreground and background regions, and ⊙ represents the product.

[0070] In this embodiment, the optimized teacher query from the teacher model is used in interaction with the feature map to obtain ψ, which is adapted to the DETR structure. Specifically, as shown... Figure 4 As shown, after mapping the feature map to the teacher query, an inner product operation is performed to generate M similarity heatmaps ψ related to the query. i The output of the teacher model is then processed using a weighted similarity heatmap ψ based on the quality coefficient q. i And M similarity heatmaps ψ i The resulting weighted mask image is the soft mask matrix of the loss weights.

[0071] The specific formula for calculating the weighted mask image is as follows:

[0072] Given the optimized query set Q of the entire teacher model T ∈R M×d The original weighted mask image can be obtained by the following formula:

[0073]

[0074] in Let represent the query of the i-th teacher model, and M represent the number of such queries. Each query is inner-productted with a feature map to generate M similarity heatmaps ψ related to the query. i The similarity heatmap ψ i This indicates the area related to the query.

[0075] S53: The query represents information about different locations and targets, including both foreground and background areas. For example... Figure 3As shown, (a) is the input image, (b, c, d) are similarity heatmaps corresponding to teacher queries for the foreground target, and (e, f) are heatmaps corresponding to background noise. To prevent background noise from overwhelming the foreground region, the proposed quality score q is used to suppress the region corresponding to the background query:

[0076]

[0077] Where c i and These represent the classification score and the predicted bounding box output by the i-th teacher query, respectively. This represents the ground truth box (GT box) matched by the predicted bounding box. γ = 0.5 is the balance coefficient, and IoU is the intersection-over-union ratio of the boxes. The q for the foreground portion approaches 1, while the q for the background portion approaches 0. The final loss function L for feature-level distillation is... featKD (corresponding to the appendix) Figure 4 The L2 loss in the equation is expressed as:

[0078]

[0079] Where, q i Let q represent the quality coefficient corresponding to the i-th query in the teacher model, and ⊙ represent the product.

[0080] Through steps S51 to S53, feature-level distillation of the student model based on the teacher model is performed, and the feature distribution of the intermediate layer of the student model will be similar to that of the teacher model in the vicinity of the target region.

[0081] S6: Save the pre-trained teacher query in the teacher model and collect the matching results of the teacher query and GT samples in each batch online. Input the teacher query as a set of additional training samples into the decoder part of the student model to obtain the assignment-level distillation loss function, so as to realize the assignment-level distillation of the student model by the teacher model.

[0082] When performing assignment-level distillation, the motivation for distillation and the problem to be solved are visualized, specifically including steps S61 to S62.

[0083] S61: First, the instability (IS) of positive and negative sample matching in the teacher model, the original student model, and the distilled student model during training is defined and statistically analyzed. For example... Figure 5As shown, the matching between the teacher query and the ground truth (GT) in the pre-trained teacher model is relatively stable across different decoding layers, while the student model exhibits chaotic matching at the beginning of training, leading to slow convergence. Therefore, it is necessary to save the pre-trained teacher queries in the teacher model and collect the sample matching results between the teacher query and the GT (ground truth / label) in each batch online. The training stability of student models is improved after assignment-level distillation.

[0084] S62: The teacher's query is input as an additional set of training samples into the decoder part of the student model, and the corresponding prediction directly uses the above... Dividing the training samples into positive and negative samples alleviates the instability problem; therefore, the distillation loss function for assignment-level distillation is as follows:

[0085]

[0086] Among them, L assignKD This represents the assignment-level distillation loss function. This represents the matching results between teacher queries and ground truth samples, where M represents the total number of teacher queries, and M = M pos +M neg y represents the ground truth sample, y i This represents the i-th ground truth sample. express Student model prediction under matching. S7: Train the student model based on logits-level distillation, feature-level distillation, and assignment-level distillation until the student model converges.

[0087] During training, three distillation losses are applied simultaneously to the student model. The total number of training epochs follows the model's own settings (typically 50 epochs) for different types of DETR detectors. The Adamw optimizer is used to optimize the overall network, and cosine annealing is employed as the learning strategy. Distillation can be performed in various ways to construct a smaller parameter model from a larger one, such as replacing the backbone feature extraction network with a lightweight one, reducing the number of encoder and decoder layers, and decreasing the number of model queries.

[0088] To quantitatively evaluate the effectiveness of this embodiment, this invention was quantitatively evaluated on the COCO and VOC datasets along with methods such as FGD, MGD, and FitNet. The mAP (mean accuracy per class) performance metric was chosen as the numerical evaluation metric. In the quantitative analysis, even when other methods failed, this invention achieved state-of-the-art average performance on four benchmark models and two test datasets, and its robustness was demonstrated in more than five backbone feature extraction networks.

[0089] According to steps S1 to S7, compared with the prior art, the beneficial effects of the present invention are as follows:

[0090] The distillation framework is a general framework that is adaptable to different DETR models. It mainly includes the learning of three parts of distillation loss: (1) logits-level distillation solves the uncertainty problem of matching between teacher and student models and adopts a progressive strategy to adapt to the cascaded detection head in DETR; (2) feature-level distillation proposes to use a similarity-based soft label mask map as a foreground and background partitioning strategy in the feature map to help the student model more accurately locate the knowledge to be learned; (3) assignment-level distillation proposes to use the stable prediction and GT correspondence of the pre-trained teacher model and help the student model converge better by adding extra samples.

[0091] Experimental results show that this distillation framework can consistently improve performance on various benchmark DETR models. Furthermore, the distillation framework only introduces a small amount of additional custom models and training loss during training, making it plug-and-play. During testing, the distillation component is removed, retaining only the student model, and deployment does not incur additional resource consumption.

[0092] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A knowledge distillation method applicable to DETR-type detectors, characterized in that a preprocessed image is fed into a trained student model to output prediction results; The training process of the student model is as follows: S1: Construct a training set, which includes multiple preprocessed images; S2: Select a teacher model and a student model, train the teacher model until it converges, and freeze the parameters of the teacher model. The teacher model includes an encoder and a decoder connected in sequence. S3: Simultaneously feed the images from the training set to both the teacher model and the student model; S4: Transfer the knowledge output from the decoder of the teacher model to the decoder output of the student model using an incremental distillation method, thereby realizing logits-level distillation of the student model through the teacher model; S5: Temporarily store the feature map output by the encoder of the teacher model and the teacher query predicted by the decoder, and calculate the foreground and background regions of each feature map to realize feature-level distillation of the teacher model to adapt to the student model. S6: Save the pre-trained teacher query in the teacher model and collect the sample matching results between the teacher query and the ground truth in each batch online. Input the teacher query as a set of additional training samples into the decoder part of the student model and provide the assignment-level distillation loss function as supervision to realize the assignment-level distillation of the student model by the teacher model. S7: Train the student model based on logits-level distillation, feature-level distillation, and assignment-level distillation until the student model converges; Step S4 specifically includes: The prediction results output by the teacher model decoder are divided into positive and negative samples according to minimizing the loss: The student model's prediction is expressed as: ,in and This represents the prediction result set of the teacher model and the student model. and These represent the number of positive samples and the number of negative samples, respectively. N This indicates the number of queries for the student model. Indicates positive sample partitioning. Indicates negative sample partitioning. This represents the positive sample prediction results of the teacher model. This represents the prediction results of the teacher model for negative samples. This represents the prediction result for each query in the student model; Using the training loss function built into the DETR detector Perform Hungarian matching on the positive and negative sample sets respectively to obtain and Matching relationships are used to derive the initial loss function for logits-level distillation. ,in express Loss function under matching relationship express Loss function under matching relationship; Both the teacher model and the student model decoders contain multiple cascaded stages. A progressive distillation method is used to transfer the knowledge contained in the decoder outputs of different stages of the teacher model to the decoder parts of different stages of the student model. The formula for the incremental distillation method is as follows: in, It is the total number of decoder layers. Indicates the current layer number of the decoder. This indicates the distillation loss at the corresponding layer number.

2. The knowledge distillation method for DETR-type detectors according to claim 1, wherein step S5 specifically includes: The feature map output by the encoder of the teacher model and the teacher query predicted by the decoder are temporarily stored. After aligning the feature map and the teacher query dimensions, perform an inner product operation to generate... Heatmap of similarity related to each query ; The output of the teacher model is expressed through a quality coefficient. Weighted similarity heatmap and will Similarity heatmap The fusion yields a weighted mask image; The loss function for feature-level distillation is calculated based on the weighted mask image as the loss weights in the feature-level distillation loss function. ; loss function The calculation formula is as follows: in, , This represents the total number of teacher queries. and These represent the feature maps generated by the teacher model and the student model, respectively. and These represent the height and width of the feature map, respectively. The number of channels representing teacher characteristics. Represents the feature dimensions of the student model. This represents a learnable, adaptive feature dimension mapping layer to align the feature maps of the student model. This represents a weighted mask that divides the foreground and background regions. The teacher model represents the first Quality coefficient for each query , ⊙ represents the product.

3. The knowledge distillation method for DETR-type detectors according to claim 1, in step S6, the assignment-level distillation loss function is specifically: in, This represents the assignment-level distillation loss function. This indicates the matching results between the teacher's query and the GT sample. This represents the total number of teacher queries. , Represents GT samples, Indicates the first GT samples, express Student model prediction under matching.

4. The knowledge distillation method for DETR-type detectors as described in claim 3, wherein during assignment-level distillation, the motivation for distillation and the problem to be solved are visualized, specifically: First, we define and statistically analyze the instability coefficients (IS) of positive and negative sample matching in the teacher model, the original student model, and the distilled student model during training. We find that the teacher query and ground truth matching in the teacher model are relatively stable in different decoding layers. The student model has chaotic matching and slow convergence at the beginning of training. The training stability of the student model is improved after assignment-level distillation.

5. The knowledge distillation method for DETR-type detectors according to claim 1, wherein the preprocessed image is an image after data augmentation and cropping, and the data augmentation method is random horizontal flipping / random rotation, etc.

6. In the knowledge distillation method applicable to DETR-type detectors according to claim 1, during the actual use of the student model after training, the network structures corresponding to the teacher model, logits-level distillation, feature-level distillation, and assignment-level distillation are directly deleted, and the optimized student model with a small number of parameters is retained to perform inference on the preprocessed image to output prediction results.

Images