A knowledge distillation method of an isomorphic 2D detector in a natural scene
By employing the consistency distillation and dynamic region feature distillation modules in the knowledge distillation method, the problem of inconsistency between classification and localization in target detection algorithms is solved, achieving a highly efficient improvement in accuracy, which is superior to existing methods.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2023-12-05
- Publication Date
- 2026-07-10
AI Technical Summary
Existing object detection algorithms exhibit inconsistencies between classification confidence and localization quality, leading to prediction error suppression issues in the post-processing stage. Furthermore, existing methods typically rely on additional networks or prior assumptions, increasing model overhead and limiting accuracy improvement.
The knowledge distillation method is adopted, which uses consistency distillation and dynamic region feature distillation modules to guide student model training with consistency scoring, and balances classification and localization features through dynamic region fusion feature distillation to achieve efficient knowledge transfer and accuracy improvement.
Without increasing additional overhead, it effectively alleviates the problem of inconsistent classification and positioning, improves detection accuracy, and is superior to existing distillation methods.
Smart Images
Figure CN117649517B_ABST
Abstract
Description
Technical Field
[0001] This invention pertains to 2D object detection methods under natural conditions, specifically involving a knowledge distillation technique for isomorphic 2D detection models. Background Technology
[0002] Object detection tasks typically consist of two subtasks: classification and localization, used to locate and identify the attributes and spatial position of objects. Existing object detection algorithms utilize two parallel detection heads to achieve classification confidence prediction and spatial location regression based on deep features, respectively. However, this classic design paradigm inevitably leads to an inconsistency in the spatial distribution between classification confidence and localization quality (IOU). This inconsistency causes, in the post-processing stage (NMS), predictions with high classification confidence but low localization quality to suppress predictions with high localization accuracy but low classification confidence, resulting in locally optimal predictions.
[0003] Existing work has focused on mitigating this problem in three ways: adding an additional detector head to predict regression scores and thus redefining the NMS score, as seen in IOUNet and FCOS; restricting output predictions to a consistent prediction region; and enhancing the dependency between classification and regression tasks to output more consistent predictions. While these studies have made significant progress in mitigating the effects of spatial distribution inconsistency, they often rely on additional networks or prior assumptions, increasing model overhead and limiting improvements in model accuracy.
[0004] In conclusion, it is crucial to propose an efficient and robust detection method to alleviate the problem of inconsistent classification and localization distributions in existing detection architectures. Summary of the Invention
[0005] To address the problems existing in the background technology, a knowledge distillation method for isomorphic detectors is proposed, which improves the detection accuracy of isomorphic small models through knowledge distillation. It mainly consists of two modules: consistency distillation and dynamic region feature distillation. First, a novel consistency score is defined to estimate the matching degree of classification and regression quality in dense detection, and the proposed consistency score is used as prior knowledge to provide additional supervision to the student model through consistency distillation. When performing feature distillation, inconsistent spatial distribution can lead to inconsistent core region feature selection. To solve this problem, a dynamic region fusion feature distillation method is proposed, which dynamically fuses classification-related and localization-related core regions based on the current model's prediction state and the teacher model's prediction labels.
[0006] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:
[0007] I. A knowledge distillation method for isomorphic 2D detectors in natural scenes, comprising the following steps:
[0008] S1: Obtain the RGB image dataset;
[0009] S2: Train the 2D detection algorithm for RGB images using the RGB image dataset from step S1, and optimize the training process, specifically as follows:
[0010] 2.1) Design consistency scores for teacher and student models to quantitatively quantify the inconsistency between classification confidence and localization quality in 2D detection algorithms;
[0011] 2.2) In the training of the output dimension of the 2D detection algorithm, based on the consistency metric score in step 2.1), a consistency distillation knowledge transfer method is designed, and efficient consistency transfer is achieved by weighted consistency loss of key regions;
[0012] 2.3) In the training of the feature dimensions of the 2D detection algorithm, a feature distillation method with dynamic region fusion is designed: by dynamically fusing the classification probability and localization probability of the current student model and teacher model, the classification region and localization region are fused to realize the 2D detection feature imitation of the student model.
[0013] S3: The trained 2D detection algorithm is used to detect the RGB image under test, and the category and location coordinates of the detected objects in the image are output.
[0014] To mitigate the problem of errors in the post-processing stage caused by the inconsistency between classification confidence and localization quality, step 2.1) proposes the calculation of consistency scores for the teacher model and the student model.
[0015] The specific process of step 2.1) is as follows:
[0016] 2.1.1) Convert the prediction results of the 2D detection model into classification probability and localization probability;
[0017] Given a predicted bounding box, use P cls and P reg Let P represent the prediction results for classification and localization, respectively; where P cls Let P be a C-dimensional vector, where C represents the number of categories in the detection dataset, and P is a vector of dimension C. reg It is a 4-dimensional vector representing the position coordinates of the rectangle;
[0018] For a classification task, the classification probability p c The calculation is as follows:
[0019]
[0020] Where k is the index variable, and softmax is the spatial softmax function, which is used to normalize the maximum output of the retained classification branch to [0,1].
[0021] For the localization branch, the values predicted by the 2D detection model's localization network are first mapped to the real image space using a decode function. Then, the localization quality IOU between each bounding box and the ground truth ground truth (GT) is evaluated. For each predicted bounding box, the maximum IOU score is defined as the regression probability p. r :
[0022]
[0023] Where g is the index variable, and G represents the actual number of rectangles;
[0024] 2.1.2) Derive the consistency score based on the deviation between the classification probability and the localization probability:
[0025] Given a pair (p) c ,p r The consistency score HS is calculated as follows:
[0026] Δp=|p c -p r |
[0027]
[0028] In step 2.1.1):
[0029] For a single-stage detector, P cls and P reg This indicates the output of the detection head;
[0030] For a two-stage detector, P cls and P reg This indicates the output of the RPN network.
[0031] The consistency distillation method used in step 2.2) for the isomorphism detection model is specifically as follows:
[0032] Based on the consistency score HS in step 2.1), the teacher model's HS is used as new knowledge to guide the training of the student model, and the superscripts t and s are used to represent the teacher's and student's HS respectively.
[0033] Considering the number of HS boxes in dense detection scenarios, to highlight the contribution of foreground prediction boxes, the teacher model is used to locate the p generated by the branch. r As a spatial mask ψ, the calculation of consistency loss is weighted:
[0034]
[0035]
[0036] Where i,j, m,n are the indices of the image size; l is the index of the FPN level, 1-5; r represents localization; c represents classification; W,H are the image sizes; and L is the number of FPN levels.
[0037] The feature distillation method for dynamic region fusion in step 2.3) is specifically as follows:
[0038] The existing feature distillation representation based on FPN is as follows:
[0039]
[0040] Here, φ is an adaptive layer used to align the feature dimensions of teachers and students. Since foreground pixels are far fewer than background pixels in object detection tasks, the aforementioned feature distillation will be dominated by learning background features. To address this, a dynamic region fusion feature distillation method is proposed. Unlike existing methods, the dynamic core region distillation proposed in this invention comprehensively considers the inconsistency between classification and localization.
[0041] 2.3.1) Comprehensive utilization of the teacher model's prediction pairs (p c ,p r The original feature distillation is weighted using a spatial mask to obtain the weighted classification region feature distillation L. FPN-cls and positioning area feature distillation L FPN-reg The weighted classification region feature distillation and location region feature distillation are calculated as follows:
[0042]
[0043]
[0044] Where F represents a feature.
[0045] 2.3.2) The weights of the balanced feature distillation amplitudes are obtained through a dynamic core region generation method to balance the feature distillation amplitudes of the classification region and the localization region. The specific process is as follows:
[0046] Prediction masks for teachers and students Connect along the channel dimension:
[0047]
[0048] The concatenated prediction P is compressed by average pooling and then fed into two lightweight fully connected layers to generate task-related weights.
[0049] Finally, the output is processed using the softmax function to obtain normalized weights, ensuring that the sum of the weights is 1.
[0050] T 0 ,T 1 =softmax(FC(FC(AvgPool(P))))
[0051] Given task-related weights T 0 ,T 1 The characteristic distillation calculation for dynamic region fusion is as follows:
[0052] Loss TFD =T 0 ×L FPN-cls +T 1 ×L FPN-reg .
[0053] This invention designs a prediction-based consistency distillation module and a dynamic region fusion feature distillation module. Based on the original classification and regression branch outputs, a novel consistency score is defined to evaluate the accuracy of the model's consistency predictions. Using this consistency score representation, a novel knowledge transfer method is proposed, whereby the consistency score of the teacher model is used as prior knowledge to guide the model output of the student model. To achieve more efficient knowledge transfer, unlike existing feature distillation methods, this invention designs a dynamic region fusion feature distillation method, dedicated to transferring features from the teacher model in key regions to the student model.
[0054] The beneficial effects of this invention are:
[0055] This invention leverages the advantages of knowledge distillation to achieve consistent knowledge and key feature transfer without incurring additional overhead, effectively avoiding the overhead disadvantages of existing detection methods. Furthermore, by selecting the teacher model, the accuracy of the student model can be maximized. Attached Figure Description
[0056] Figure 1 This diagram illustrates the inconsistency between classification and localization in existing object detection algorithms.
[0057] Figure 2 This is a schematic diagram of the knowledge distillation process under a single-stage detection architecture.
[0058] Figure 3 This is a schematic diagram of the knowledge distillation process under a two-stage detection architecture. Detailed Implementation
[0059] To better understand the present invention, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.
[0060] Figures 1-3 This paper demonstrates the inconsistency between classification and localization, and illustrates the implementation of this invention in both single-stage and two-stage algorithms. The following section uses the single-stage detection algorithm as an example to describe in detail the implementation steps of the method of this invention:
[0061] 1. Algorithm validation was performed on the large-scale public dataset COCO. The training data consisted of 118k natural dataset images across 80 classes, and a 5k subset was used for evaluation on the test set.
[0062] 2. Calculation of Consistency Score. To mitigate the problem of post-processing errors caused by inconsistency between classification confidence and localization quality, this inconsistency is first quantitatively quantified. A concept of consistency score is proposed, which includes two steps: converting the model's prediction results into classification and localization probabilities; and deriving the consistency score based on the deviation between these two probabilities.
[0063] Given a predicted bounding box, use P cls and P reg These represent the prediction results for classification and regression, respectively. Where P... cls Let P be a C-dimensional vector, where C represents the number of categories in the detection dataset, and P is a vector of dimension C. reg This is a 4-dimensional vector representing the position coordinates of the bounding box. For a classification task, its classification probability p... c It can be calculated as follows:
[0064]
[0065] Here, k is the index variable, and softmax is the spatial softmax function, which is used to normalize the maximum output of the retained classification branch to [0,1].
[0066] Similarly, for the regression branch, the network-predicted values are first mapped to the ground truth image space using a decode function. Then, the quality of localization (IOU) between each bounding box and the ground truth (GT) is evaluated. For each predicted box, the maximum IOU score is defined as the regression probability p. r :
[0067]
[0068] Where g is the index variable. Given a tuple (p c ,p r The consistency score HS can be calculated as follows:
[0069] Δp=|p c -p r |
[0070]
[0071] 2. Consistency Distillation Method for Detection Models. Based on the consistency score HS mentioned above, the teacher's HS is used as new knowledge to guide student training. Superscripts t and s are used to represent the teacher's and student's HS, respectively. Considering the number of HSs in dense detection scenarios, to highlight the contribution of the foreground prediction box, a regression branch of the teacher model is used to generate p. r ψ is used as a spatial mask to weight the calculation of consistency loss:
[0072]
[0073]
[0074] 3. Feature distillation method for dynamic region fusion. Existing FPN-based feature distillation can be represented as follows:
[0075]
[0076] Here, φ is an adaptive layer used to align the feature dimensions of the teacher and student. Since the foreground pixels are far fewer than the background pixels in object detection tasks, the above feature distillation will be dominated by the learning of background features. A dynamic region fusion feature distillation method is proposed. Unlike existing methods, the dynamic core region distillation proposed in this invention comprehensively considers the problem of inconsistency between classification and localization. Specifically, the proposed method comprehensively utilizes the prediction tuples (p...) of the teacher model... c ,p r The original feature distillation is weighted using a spatial mask. The feature distillation for classification and regression regions is calculated as follows:
[0077]
[0078]
[0079] To balance the feature distillation amplitudes of classification and localization regions, a dynamic core region generation method is proposed, using a lightweight neural network to predict these weights. In implementing TWG, the prediction masks for both teachers and students are used. First, connect along the channel dimension:
[0080]
[0081] The concatenated prediction P is compressed using average pooling and then fed into two lightweight fully connected (FC) layers to generate task-relevant weights. Finally, the output is subjected to a softmax function to obtain normalized weights, ensuring that the sum of these weights is 1.
[0082] T 0 ,T 1=softmax(FC(FC(AvgPool(P))))
[0083] Given task-related weights, dynamic core region feature distillation can be calculated as follows:
[0084] Loss TFD =T 0 ×L FPN-cls +T 1 ×L FPN-reg
[0085] For single-stage detectors, such as Figure 2 As shown, the distillation loss function directly affects the final detection head; however, for a two-stage detector, since the RPN network can be considered a special single-stage detector, the distillation loss only affects the RPN network, as shown below. Figure 3 As shown.
[0086] 4. Network Training. The network training follows the backpropagation algorithm, with the SGD algorithm using momentum as the optimizer. To ensure a fair comparison with existing distillation methods, the total number of iterations is set to 24 (2x). The initial learning rates are set to 0.01 and 0.02 for single-stage and two-stage detectors, respectively. Training is performed using eight 1080Ti GPUs. The training loss consists of three parts: the loss of the original detector, the loss of consistency distillation, and the loss of dynamic region fusion feature distillation, calculated as follows:
[0087] Loss = loss detector +α1×loss HD +α2×loss TFD
[0088] Where {α1, α2} are weighting parameters for each distillation loss.
[0089] 5. Test Set Evaluation. Save the trained model weight file, input test set images from publicly available data, and load the test model and the trained weight file. Load the test images onto the GPU and normalize them to the (800, 1300) scale. Select the mean Average Precision (mAP) in COCO format as the final detection accuracy evaluation. Comparisons with existing state-of-the-art (SOTA) distillation schemes on several well-known detectors (e.g., RetianNet and Faster RCNN). To verify the superiority of the proposed method, we used seven recent SOTA models for comparison, as shown in Table 1. Furthermore, to compare with the recent detection distillation method LD, this invention also completed relevant experiments on the GFL benchmark, and the experimental results are shown in Table 2. It can be seen that the distillation scheme of this invention significantly outperforms existing SOTA distillation methods.
[0090] Table 1 Comparison of detection distillation based on RetinaNet and Faster RCNN
[0091]
[0092] Table 2 Comparison of detection and distillation based on GFL
[0093]
Claims
1. A knowledge distillation method for isomorphic 2D detectors in natural scenes, characterized in that, Includes the following steps: S1: Obtain the RGB image dataset; S2: Train the 2D detection algorithm for RGB images using the RGB image dataset from step S1, and optimize the training process, specifically as follows: 2.1) Design consistency scores for teacher and student models to quantitatively quantify the inconsistency between classification confidence and localization quality in 2D detection algorithms; The specific process of step 2.1) is as follows: 2.1.1) Convert the prediction results of the 2D detection model into classification probabilities. and location probability ; 2.1.2) Derive the consistency score based on the deviation between the classification probability and the localization probability: Given a pair The consistency score HS is calculated as follows: 2.2) In the training of the output dimension of the 2D detection algorithm, based on the consistency metric score in step 2.1), a consistency distillation knowledge transfer method is designed, and efficient consistency transfer is achieved by weighted consistency loss of key regions; The uniform distillation method in step 2.2) is specifically as follows: Based on the consistency score HS in step 2.1), the teacher model's HS is used as new knowledge to guide the training of the student model, and the superscripts t and s are used to represent the teacher's and student's HS, respectively. Using the teacher model to locate branches As a spatial mask The consistency loss is calculated using a weighted average. 2.3) In the training of the feature dimensions of the 2D detection algorithm, a feature distillation method with dynamic region fusion is designed: by dynamically fusing the classification probability and localization probability of the current student model and teacher model, the classification region and localization region are fused to realize the 2D detection feature imitation of the student model. The feature distillation method for dynamic region fusion in step 2.3) is specifically as follows: 2.3.1) Comprehensive utilization of the teacher model's prediction pairs The original feature distillation is weighted using a spatial mask to obtain the weighted classification region feature distillation. and location area feature distillation ; 2.3.2) The weights for balancing feature distillation amplitudes are obtained through a dynamic core region generation method to balance the feature distillation amplitudes of the classification region and the localization region. The specific process is as follows: Prediction masks for teachers and students Connect along the channel dimension: The concatenated prediction P is compressed by average pooling and then fed into two lightweight fully connected layers to generate task-related weights. Finally, the output is processed using the softmax function to obtain normalized weights, ensuring that the sum of the weights is 1. Given task-related weights The characteristic distillation calculation for dynamic region fusion is as follows: S3: The trained 2D detection algorithm is used to detect the RGB image under test, and the category and location coordinates of the detected objects in the image are output.
2. The knowledge distillation method for isomorphic 2D detectors in natural scenes according to claim 1, characterized in that: The specific process of step 2.1) is as follows: 2.1.1) Convert the prediction results of the 2D detection model into classification probabilities. and location probability ; Given a prediction bounding box, use and These represent the prediction results for classification and localization, respectively; where, for 3D vector Indicates the number of categories in the detection dataset. It is a 4-dimensional vector representing the position coordinates of the rectangle; For classification tasks, the classification probability The calculation is as follows: Where k is the index variable, and softmax is the spatial softmax function; For the localization branch, the values predicted by the 2D detection model's localization network are first mapped to the real image space using a decode function. Then, the localization quality IOU between each bounding box and the ground truth (GT) is evaluated. For each predicted box, the maximum IOU score is defined as the regression probability. : Where g is the index variable, and G represents the actual number of rectangles; 2.1.2) Derive the consistency score based on the deviation between the classification probability and the localization probability: Given a pair The consistency score HS is calculated as follows: 。 3. The knowledge distillation method for isomorphic 2D detectors in natural scenes according to claim 2, characterized in that: In step 2.1.1): For single-stage detectors and This indicates the output of the detection head; For two-stage detectors and This indicates the output of the RPN network.
4. The knowledge distillation method for isomorphic 2D detectors in natural scenes according to claim 1, characterized in that: The uniform distillation method in step 2.2) is specifically as follows: Based on the consistency score HS in step 2.1), the teacher model's HS is used as new knowledge to guide the training of the student model, and the superscripts t and s are used to represent the teacher's and student's HS, respectively. Using the teacher model to locate branches generated As a spatial mask The consistency loss is calculated using weighted averages: in, , All are indexes of the image size; For FPN level indexes, 1-5; Indicates location; Indicates classification; Image size, This represents the number of FPN layers.
5. The knowledge distillation method for isomorphic 2D detectors in natural scenes according to claim 4, characterized in that: The weighted classification region feature distillation and location region feature distillation are calculated as follows: in, Indicates characteristics.