Object detection method based on task-adaptive knowledge distillation

Abstract

This invention discloses a method for object detection based on task-adaptive knowledge distillation. The method comprises the following steps: obtaining a dataset with dataset labels for training an object detector; using the dataset to train a teacher detector with large model capacity and good performance; and, when training the object detector, adding two distillation loss functions in addition to its own target loss function: a classification distillation loss function based on binary cross entropy for the classification head and a localization distillation loss function based on intersection-over-union for the localization head; and finally, obtaining a trained object detector. This method fully considers the characteristics of the object detector task and adapts to these characteristics, proposing a more appropriate distillation loss function and achieving better knowledge distillation results.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular, to a target detection method based on task-adaptive knowledge distillation. Background Art

[0002] Object detection, a technique for locating all objects in a given image and assigning their categories, is one of the most fundamental and important tasks in computer vision. Object detection technology has a wide range of real-world applications, including intelligent security, autonomous driving, medical diagnosis, and industrial inspection.

[0003] Knowledge distillation, a model compression technique, aims to transfer knowledge from a large model to a smaller one, improving the performance of the smaller model. Existing object detection knowledge distillation methods primarily focus on feature distillation. This approach fails to account for the inherent differences in features between detectors with different architectures, and therefore is ineffective in practical scenarios, particularly for knowledge distillation of heterogeneous detectors.

[0004] Response-based object detection knowledge distillation schemes transfer knowledge at the result level, unconstrained by detector architecture. However, mainstream approaches fail to consider the inherent characteristics of object detection tasks when applying response-based object detection knowledge distillation schemes, resulting in poor performance and limited adoption. Summary of the Invention

[0005] To address the above issues, the present invention provides a target detection method based on task-adaptive knowledge distillation. The specific technical solutions adopted by the present invention are as follows:

[0006] A target detection method based on task-adaptive knowledge distillation includes the following steps:

[0007] S1. Obtain a training data set for training an object detector, wherein image samples in the training data set are pre-labeled with the location and classification of target objects in the image;

[0008] S2. Using the training dataset, a teacher object detector is trained to generate position and classification soft labels for an auxiliary student object detector;

[0009] S3. Train the student target detector using the training data set, and input the position and classification soft labels provided by the teacher target detector during the training process. Calculate the classification distillation loss function based on binary cross entropy on the classification head of the student target detector, and calculate the positioning distillation loss function based on intersection-over-union ratio on the positioning head of the student target detector. Add the classification distillation loss function and the positioning distillation loss function to the target loss function of the student target detector itself as a total loss function for optimization, thereby enhancing the performance of the student target detector in target detection through the soft label information provided by the teacher target detector.

[0010] S4. Use the trained student target detector to perform target detection on the image to be detected.

[0011] Preferably, the training data set in S1 includes multiple image samples, and each image sample is pre-labeled with the location and classification label of the target object. i ,c i}.

[0012] Preferably, in S2, the training of the teacher target detector includes the following sub-steps:

[0013] S21. Input the image sample I into the teacher object detector. The teacher object detector consists of a feature extractor f(·) and a detection head h(·). The detector first extracts features F = f(i) from the image sample I, and then inputs the features into the detection head to generate a final prediction P = h(F). The final prediction P contains the location of the object in the image and the classification prediction value;

[0014] S22, based on the final prediction P and the label {o i ,c i}, calculate the loss function value and use the back-propagation algorithm to update the feature extractor f(·) and detection head h(·) of the teacher object detector;

[0015] S23. Repeat S21 and S22 continuously until the upper limit of the training round is reached or the loss function of the detector has converged, and then stop training to obtain the trained teacher target detector.

[0016] Preferably, in S3, the classification distillation loss function based on binary cross entropy is calculated as follows:

[0017] S311: For each image sample x during the training process, input it into the student target detector and the trained teacher target detector respectively, and each generates a classification spectrum through the feature extractor f(·) and the detection head h(·) The classification spectrum l contains K-dimensional classification logical values ​​at n feature map positions, where n = H × W, H, W and K represent the feature map height, feature map width and number of classification categories extracted by the feature extractor respectively;

[0018] S312. Perform normalization operations on the classification spectra obtained by the teacher target detector and the student target detector, respectively, and convert the K-dimensional classification logical value of each position in the classification spectrum into a k-dimensional classification score, thereby converting the two classification spectra into a classification score matrix:

[0019] p t ′=Rrot sig (l t )

[0020] p s ′=Prot Sig (l s )

[0021] Where: Prot Sig () represents the Sigmoid normalization operation, l t and l s are the classification spectra l obtained by the teacher target detector and the student target detector respectively;

[0022] S313. Based on the classification score obtained in S312, the classification distillation loss at each position (i, j) in the classification spectrum is calculated as follows:

[0023]

[0024] Where: Represents the classification score matrix p t ′ and p s The value at position (i, j) in ′;

[0025] S314, according to the classification score difference matrix w of the teacher target detector and the student target detector, the value of the classification distillation of each point is weighted to obtain the classification distillation loss based on binary cross entropy, and the classification distillation loss function is The form is:

[0026] w=|p t ′-p s ′|,

[0027]

[0028] Where: w i,j represents the value at position (i, j) in the classification score difference matrix w, and || represents the absolute value operation. Preferably, in S3, the calculation method of the positioning distillation loss function based on the intersection-over-union ratio is as follows:

[0029] S321: For each image sample x during the training process, input it into the student target detector and the trained teacher target detector respectively, and each generates a positioning spectrum through the feature extractor f(·) and the detection head h(·) The positioning spectrum o contains the positioning prediction values ​​at n feature map positions, and its dimension is 4;

[0030] S322. Decode the positioning prediction value of the teacher target detector into a regression frame through its own decoder:

[0031]

[0032] Among them A i represents the i-th anchor point, represents the i-th positioning prediction value of the teacher target detector, represents the i-th regression box of the teacher target detector;

[0033] S323. Decode the positioning prediction value of the student target detector into a regression box:

[0034]

[0035] Among them A i represents the i-th anchor point, represents the i-th positioning prediction value of the student target detector, represents the i-th regression box of the teacher target detector;

[0036] S324, calculation and The intersection-over-union ratio between

[0037]

[0038] S325, based on the intersection-over-union ratio of the two detectors' respective positioning regression frames calculated in S324, calculate the positioning distillation loss based on the intersection-over-union ratio and the classification distillation loss function The form is:

[0039]

[0040] Where max(w .,j ) is the jth column w of the classification score difference matrix w .,j The maximum value in .

[0041] Preferably, in S3, the total loss function of the student target detector during training is as follows:

[0042]

[0043] in: and They represent the classification loss function and localization loss function of the student target detector itself, respectively. α1 and α2 are two hyperparameters, representing the loss weights of the classification distillation loss and the localization distillation loss, respectively.

[0044] Preferably, the weight value α1 = 1.0, and the weight value α2 = 4.0.

[0045] This paper proposes a knowledge distillation scheme that is more suitable for object detection tasks. It designs targeted distillation loss functions for the classification head and the localization head, which can bring better knowledge distillation performance gains to the object detector. Compared with traditional object detection knowledge distillation schemes, this paper has the following beneficial effects:

[0046] First, this paper proposes a feasible solution for target detection based on task-adaptive knowledge distillation.

[0047] Secondly, the present invention fully considers the characteristics of the target detection task and specifically designs the knowledge distillation loss function for the adaptation task on the classification head and regression head.

[0048] Finally, the target detection knowledge distillation scheme proposed in this invention has wide versatility and can be easily extended to heterogeneous target detectors based on the knowledge distillation scheme, and can achieve good results. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the process of target detection method based on task-adaptive knowledge distillation of the present invention.

[0050] Figure 2 Schematic diagram of the framework for knowledge distillation of the target detector of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] On the contrary, the present invention covers any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention as defined by the claims. Furthermore, to facilitate a better understanding of the present invention, certain specific details are described in detail below in the detailed description of the present invention. Those skilled in the art will be able to fully understand the present invention without these details.

[0053] This invention provides a target detection method based on task-adaptive knowledge distillation, which is used to assist in training a student target detector using a teacher target detector. It should be noted that the specific detector types of the teacher target detector and the student target detector in the present invention are not limited. Generally speaking, the teacher target detector has a larger number of parameters and better performance than the student target detector, and improves the detection performance of the small model by transferring knowledge from the large model to the small model. The network structures of the teacher target detector and the student target detector in the present invention are allowed to be different.

[0054] refer to Figure 1 In a preferred embodiment of the present invention, the target detection method based on task-adaptive knowledge distillation specifically includes the following steps:

[0055] S1. Obtain a training data set for training a target detector, wherein image samples in the training data set are pre-labeled with the location and classification of target objects in the image.

[0056] In the above step S1 of this embodiment, the training data set includes a series of image samples, and each image sample is pre-labeled with the location and classification label of the target object. i ,c i}.

[0057] In addition, in the embodiment of the present invention, in order to test the specific technical effects of the present invention, in addition to the above-mentioned training data set, a corresponding test data set can also be set. In the present invention, the ultimate algorithm goal is to give a test image I test Or, given the actual image to be detected, the detector predicts the location and category of all objects in the image. If a test dataset exists, the performance metrics can be calculated using the labels in the test dataset.

[0058] S2. Use the training dataset to train a teacher target detector with a large model capacity and good performance to generate position and classification soft labels for the auxiliary student target detector.

[0059] In the above step S2 of this embodiment, the training of the teacher object detector includes the following sub-steps:

[0060] S21. Input the image sample I into the teacher object detector. The teacher object detector consists of a feature extractor f(·) and a detection head h(·). The detector first extracts features F = f(I) from the image sample I, and then inputs the features into the detection head to generate a final prediction P = h(F). The final prediction P contains the location of the object in the image and the classification prediction value;

[0061] S22, based on the final prediction P and the label {o i ,ci}, calculate the loss function value and use the back-propagation algorithm to update the feature extractor f(·) and detection head h(·) of the teacher object detector;

[0062] S23. Repeat S21 and S22 continuously until the upper limit of the training round is reached or the loss function of the detector has converged, and then stop training to obtain the trained teacher target detector.

[0063] It should be noted that in this embodiment, the teacher object detector in S2 has a larger capacity and better performance than the student object detector model, thereby providing higher-quality soft labels. For example, if the student detector to be trained is an object detector based on the ResNet50 backbone network, the teacher detector can be an object detector based on the ResNet101 backbone network with larger capacity and better performance.

[0064] It should be noted that, based on the basic structure of the target detector, the detection head h(·) actually includes a classification head and a localization head. The classification head is used for target classification, while the localization head is used for target localization.

[0065] S3. Use the training data set to train the student target detector, and input the position and classification soft labels provided by the teacher target detector during the training process. Calculate the classification distillation loss function based on binary cross entropy on the classification head of the student target detector, and calculate the positioning distillation loss function based on intersection-over-union ratio on the positioning head of the student target detector. Add the classification distillation loss function and the positioning distillation loss function to the target loss function of the student target detector itself as the total loss function for optimization, thereby enhancing the performance of the student target detector in target detection through the soft label information provided by the teacher target detector.

[0066] It should be noted that the target loss function of the student target detector itself can be determined according to the corresponding student target detector type, which generally includes two parts: classification loss and positioning loss. While training the student target detector through its own target loss function, in addition to its own classification loss function, it is also necessary to use a classification distillation loss function based on binary cross entropy on the classification head to enhance the performance of the student detector in target object classification through the information provided by the teacher detector. Similarly, while training the student target detector, in addition to its own positioning loss function, it is also necessary to use a positioning distillation loss function based on intersection-over-union on the positioning head to enhance the performance of the student detector in target object positioning through the information provided by the teacher detector;

[0067] In the present invention, both the teacher and student object detectors can be considered as a combination of a feature extractor f(·) and a detection head h(·). Given an input image I, the detector first extracts features F = f(I) and then generates a final prediction P = h(F). The object detector generates classification spectra of different sizes through the feature extractor f(·) and the detection head h(·). The size of the classification spectrum is H×W×K, where H, W, and K represent the height, width, and number of classification categories of the feature, respectively. x is an image sample, Represents its classification logic value. Use Sigmoid normalization to obtain the classification score of each position point on the classification spectrum. The dimension is still K. The normalization process is expressed by the formula p = Prot Sig (l).

[0068] In the above step S3 of this embodiment, the classification distillation loss function based on binary cross entropy is calculated as follows:

[0069] S311. For each image sample x during the training process, input it into the student target detector and the trained teacher target detector respectively, and each of them passes through the feature extractor f(·) and the classification head in the detection head h(·) to generate a classification spectrum. The classification spectrum l contains K-dimensional classification logical values ​​at n feature map positions, where n = H × W, H, W and K represent the feature map height, feature map width and number of classification categories extracted by the feature extractor respectively;

[0070] S312. Perform normalization operations on the classification spectra obtained by the teacher target detector and the student target detector, respectively, and convert the K-dimensional classification logical value of each position in the classification spectrum into a K-dimensional classification score, thereby converting the two classification spectra into a classification score matrix:

[0071] p t ′=Prot Sig (l t )

[0072] p s ′=Prot Sig (l s )

[0073] Where: Prot Sig () represents the Sigmoid normalization operation, l t and l s are the classification spectra l obtained by the teacher target detector and the student target detector respectively;

[0074] S313. Based on the classification score obtained in S312, the classification distillation loss at each position (i, j) in the classification spectrum is calculated as follows:

[0075]

[0076] Where: Represents the classification score matrix p t ′ and p s The value at position (i, j) in ′;

[0077] S314, according to the classification score difference matrix w of the teacher target detector and the student target detector, the value of the classification distillation of each point is weighted to obtain the classification distillation loss based on binary cross entropy, and the classification distillation loss function is The form is:

[0078] w=|p t ′-p s ′|,

[0079]

[0080] Where: w i,j Represents the value at position (i, j) in the classification score difference matrix w, and || represents the absolute value operation.

[0081] In addition, the calculation method of the above-mentioned positioning distillation loss function based on intersection-over-union ratio is as follows:

[0082] S321, for each image sample x (i.e., the aforementioned I) during the training process, input it into the student target detector and the trained teacher target detector, respectively, and each generates a positioning spectrum through the feature extractor f(·) and the positioning head in the detection head h(·) The positioning spectrum o contains the positioning prediction values ​​at n feature map positions, and its dimension is 4;

[0083] S322. Decode the positioning prediction value of the teacher target detector into a regression frame through its own decoder:

[0084]

[0085] Among them A i represents the i-th anchor point, represents the i-th positioning prediction value of the teacher target detector, represents the o-th regression box of the teacher target detector;

[0086] S323. Decode the positioning prediction value of the student target detector into a regression box:

[0087]

[0088] Among them A i represents the oth anchor point, represents the i-th positioning prediction value of the student target detector, represents the i-th regression box of the teacher target detector;

[0089] S324, calculation and The intersection-over-union ratio between

[0090]

[0091] S325, based on the intersection-over-union ratio of the two detectors' respective positioning regression frames calculated in S324, calculate the positioning distillation loss based on the intersection-over-union ratio and the classification distillation loss function The form is:

[0092]

[0093] Where max(w .,j ) is the jth column w of the classification score difference matrix w .,j The maximum value in .

[0094] Therefore, through the joint guidance of the target loss function of the student target detector itself and the two types of distillation loss functions mentioned above, the student target detector can be trained on the training dataset through knowledge distillation to obtain the trained student target detector. The framework of the knowledge distillation process is as follows Figure 2 As shown in Figure 2, the total loss function of the student target detector during training is as follows:

[0095]

[0096] in: and Respectively represent the classification loss function and localization loss function of the student object detector itself. α1 and α2 are two hyperparameters, representing the loss weights of the classification distillation loss and localization distillation loss, respectively, which can be optimized based on actual conditions. In this embodiment of the present invention, the weight value α1 = 1.0 and the weight value α2 = 4.0.

[0097] The student detector is trained to minimize the total loss function As the goal, the student target detector is updated through the back-propagation algorithm and gradient descent. After several rounds of iterative training, the student target detector trained on the training dataset is obtained after reaching the maximum number of iterations or convergence.

[0098] S4. Use the trained student target detector to perform target detection on the image to be detected. The image to be detected is input into the student target detector, and its detection head outputs the classification and positioning results.

[0099] The target detection method based on task-adaptive knowledge distillation shown in S1 to S4 above is applied to a specific example to demonstrate its specific technical effect.

[0100] Example

[0101] The implementation method of this embodiment is as described in S1 to S4 above. The specific steps will not be elaborated in detail. The following only demonstrates its effect based on case data. The present invention is implemented on a dataset with true value annotations:

[0102] COCO dataset: The training set of this dataset contains 118,287 images and the test set contains 5,000 images. The COCO dataset has 80 object labels, and each image is annotated with the location and category of all objects in the image.

[0103] This example is trained on the COCO training set and tested on the COCO test set. The validation metric used is mAP. mAP (mean Average Precision) is a commonly used evaluation metric in object detection, used to measure model accuracy and performance. mAP is calculated by taking the arithmetic average of the AP (Average Precision) values ​​for all categories at different thresholds. AP represents the precision of a single category. AP50 and AP75 are also calculated by taking the average of the AP values ​​for all categories at thresholds of 0.5 and 0.75, respectively. APS and APL require constraints on the size of the object box when calculating AP. Specifically, for small objects, only predicted boxes with an area smaller than a certain threshold are considered in the AP calculation; for large objects, only predicted boxes with an area larger than a certain threshold are considered in the AP calculation. Finally, the AP values ​​for small and large objects are averaged to obtain the APS and APL values. APS and APL can evaluate small and large objects separately, providing a more comprehensive and accurate assessment of model performance.

[0104] In this example, a fully trained GFL target detector based on the ResNet101 backbone network is used as the teacher detector to guide the GFL target detector based on the ResNet50 / ResNet34 / ResNet18 backbone network to perform knowledge distillation. The detection accuracy of the detection results is shown in the following table.

[0105] Table 1

[0106]

[0107]

[0108] The task-adaptive knowledge distillation-based object detection method proposed in this paper is independent of the network backbone structure and can still achieve good results on heterogeneous networks. The detection accuracy of several sets of detection results on heterogeneous detector pairs is shown in the table below.

[0109] Table 2

[0110]

[0111] The detection accuracy of the above-mentioned groups of detection results and the target detection method based on task-adaptive knowledge distillation proposed in this invention have significantly improved the performance of the student detector.

[0112] The embodiment of the present invention proposes a knowledge distillation scheme that is more suitable for target detection tasks based on this method, and designs targeted distillation loss functions for the classification head and the localization head respectively, which can bring better knowledge distillation performance gain to the target detector.

[0113] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A target detection method based on task-adaptive knowledge distillation, characterized by The following steps are involved: S1. Obtain a training data set for training an object detector, wherein image samples in the training data set are pre-labeled with the location and classification of target objects in the image; S2. Using the training dataset, a teacher object detector is trained to generate position and classification soft labels for an auxiliary student object detector; S3. Train the student target detector using the training data set, and input the position and classification soft labels provided by the teacher target detector during the training process. Calculate the classification distillation loss function based on binary cross entropy on the classification head of the student target detector, and calculate the positioning distillation loss function based on intersection-over-union ratio on the positioning head of the student target detector. Add the classification distillation loss function and the positioning distillation loss function to the target loss function of the student target detector itself as a total loss function for optimization, thereby enhancing the performance of the student target detector in target detection through the soft label information provided by the teacher target detector. S4, using the trained student target detector to perform target detection on the image to be detected; In S3, the calculation method of the positioning distillation loss function based on the intersection-over-union ratio is as follows: S321: For each image sample x during the training process, input it into the student target detector and the trained teacher target detector respectively, and each generates a positioning spectrum through the feature extractor f(·) and the detection head h(·) The positioning spectrum o contains the positioning prediction values ​​at n feature map positions, and its dimension is 4; S322. Decode the positioning prediction value of the teacher target detector into a regression frame through its own decoder: Among them A i represents the i-th anchor point, represents the i-th positioning prediction value of the teacher target detector, represents the i-th regression box of the teacher target detector; S323. Decode the positioning prediction value of the student target detector into a regression box: Among them A i represents the i-th anchor point, represents the i-th positioning prediction value of the student target detector, represents the i-th regression box of the teacher target detector; S324, calculation and The intersection-over-union ratio between S325, based on the intersection-over-union ratio of the two detectors' respective positioning regression frames calculated in S324, calculate the positioning distillation loss based on the intersection-over-union ratio, the positioning distillation loss function The form is: Where max(w .,j ) is the jth column w of the classification score difference matrix w .,j The maximum value in .

2. The object detection method based on task-adaptive knowledge distillation according to claim 1 is characterized in that The training data set in S1 includes multiple image samples, and each image sample is pre-labeled with the location and classification label of the target object {o i ,c i }.

3. The object detection method based on task-adaptive knowledge distillation according to claim 1 is characterized in that In S2, the training of the teacher target detector includes the following sub-steps: S21. Input the image sample I into the teacher object detector. The teacher object detector consists of a feature extractor f(·) and a detection head h(·). The detector first extracts features F = f(I) from the image sample I, and then inputs the features into the detection head to generate a final prediction P = h(F). The final prediction P contains the location of the object in the image and the classification prediction value; S22, based on the final prediction P and the label {o i ,c i }, calculate the loss function value and use the back-propagation algorithm to update the feature extractor f(·) and detection head h(·) of the teacher object detector; S23. Repeat S21 and S22 continuously until the upper limit of the training round is reached or the loss function of the detector has converged, and then stop training to obtain the trained teacher target detector.

4. The object detection method based on task-adaptive knowledge distillation according to claim 1 is characterized in that In S3, the classification distillation loss function based on binary cross entropy is calculated as follows: S311: For each image sample x during the training process, input it into the student target detector and the trained teacher target detector respectively, and each generates a classification spectrum through the feature extractor f(·) and the detection head h(·) The classification spectrum l contains K-dimensional classification logical values ​​at n feature map positions, where n = H × W, H, W and K represent the feature map height, feature map width and number of classification categories extracted by the feature extractor respectively; S312. Perform normalization operations on the classification spectra obtained by the teacher target detector and the student target detector, respectively, and convert the K-dimensional classification logical value of each position in the classification spectrum into a K-dimensional classification score, thereby converting the two classification spectra into a classification score matrix: p t′ =Prot Sig (l t ) p s′ =Prot Sig (l s ) Where: Prot Sig () represents the Sigmoid normalization operation, l t and l s are the classification spectra l obtained by the teacher target detector and the student target detector respectively; S313. Based on the classification score obtained in S312, the classification distillation loss at each position (i, j) in the classification spectrum is calculated as follows: Where: Represents the classification score matrix p t ′ and p s The value at position (i, j) in ′; S314, according to the classification score difference matrix w of the teacher target detector and the student target detector, the value of the classification distillation of each point is weighted to obtain the classification distillation loss based on binary cross entropy, and the classification distillation loss function is The form is: w=|p t ′-p s ′|, Where: w i,j Represents the value at position (i, j) in the classification score difference matrix w, and || represents the absolute value operation.

5. The object detection method based on task-adaptive knowledge distillation according to claim 1 is characterized in that In S3, the total loss function of the student target detector during training is as follows: in: and They represent the classification loss function and localization loss function of the student target detector itself, respectively. α1 and α2 are two hyperparameters, representing the loss weights of the classification distillation loss and the localization distillation loss, respectively.

6. The object detection method based on task-adaptive knowledge distillation according to claim 5, characterized in that: The weight value α1 = 1.0, and the weight value α2 = 4.0.

Images