Entropy difference guided unsupervised domain adaptation learning image object detection method

Abstract

The application discloses an entropy difference guided source-free domain migration learning image target detection method and belongs to the technical field of image processing and computer vision. The application comprises the following steps: initialization of a teacher model and a student model, enhancement of a target domain data set, generation of pseudo labels by screening teacher model output using a fixed confidence threshold, calculation of image entropy difference and intersection over union of detection boxes of the same category, pseudo label screening based on entropy difference guidance, model training and updating of the teacher model. The application trains the student model by using the finally screened pseudo labels as supervision information, updates the teacher model by exponential moving average, completes model migration, effectively solves the problem of insufficient reliability of pseudo labels in a cross-domain scene, and realizes high-performance target detection in the absence of source domain data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and computer vision, in particular to an entropy difference guided source-free domain transfer learning image target detection method. BACKGROUND

[0002] In the target detection task, traditional deep learning methods usually rely on a large amount of labeled data for supervised training. However, in actual industrial application scenarios, the data of the target domain (such as foggy days or different city street scenes) often lacks labels, and manual labeling requires a large amount of manpower, financial resources and time cost. More seriously, source domain data may not be directly accessible and used due to privacy protection requirements (such as medical images), data storage limitations (such as satellite remote sensing data), or commercial secrets. This double dilemma of "source domain invisible but target domain unlabeled" has given birth to the emerging research direction of source-free domain adaptive object detection. This method only uses pre-trained source models and target domain unlabeled data to achieve cross-domain knowledge transfer through pseudo-label self-training technology.

[0003] In recent years, the average teacher framework has become the mainstream framework for source-free domain adaptive object detection tasks, and the quality of pseudo-labels plays a decisive role in knowledge transfer. In the source-free domain adaptive object detection method, Non-Maximum Suppression (NMS) is a common post-processing technique for obtaining reliable pseudo-labels. NMS mainly filters overlapping detection results by calculating the Intersection over Union (IoU) and class confidence between detection boxes: first, set the IoU threshold (usually 0.5), when the IoU of two detection boxes exceeds the threshold, keep the detection box with higher confidence and suppress the detection box with lower confidence. However, NMS has two main problems when removing overlapping pseudo-labels:

[0004] First, when the detection boxes of two different target instances overlap, NMS will keep the detection box with higher confidence and discard the detection box with lower confidence, which leads to a decrease in the utilization rate of pseudo-labels and affects the performance of the model;

[0005] Second, when multiple detection boxes of the same target instance overlap, NMS only keeps the detection box with the highest confidence, but in fact, the detection box with lower confidence may be closer to the real label;

[0006] These problems result in low-quality pseudo-labels, which not only weaken the performance of the student model but also hinder the forward update of the teacher model in the average teacher framework. Therefore, an entropy difference guided source-free domain transfer learning image target detection method is proposed to solve the above problems. SUMMARY

[0007] The application aims to provide an entropy difference guided source-free domain transfer learning image target detection method, which initializes teacher and student model parameters and enhances target domain data, screens pseudo labels through a confidence threshold, then innovatively introduces an image entropy difference index for pseudo label optimization screening, solving the problem of pseudo label missed detection and false detection caused by simply relying on intersection over union and detection box confidence in traditional non-maximum suppression method in cross-domain target detection.

[0008] The application is implemented by the following technical solutions:

[0009] The application is an entropy difference guided source-free domain transfer learning image target detection method, comprising the following steps:

[0010] S1: initialization of teacher and student models:

[0011] The teacher model and the student model are parameterized using the same pre-trained source domain detection model.

[0012] S2: enhance the target domain dataset:

[0013] First, the target domain dataset is respectively strongly and weakly enhanced, and then the strongly enhanced dataset is mask represented.

[0014] S3: use a fixed confidence threshold to screen the teacher model output to generate pseudo labels:

[0015] The weakly enhanced image is input into the teacher model, and the detection results output by the teacher model are screened using a fixed threshold, and the screening process retains all detection boxes with a confidence exceeding the threshold.

[0016] S4: calculation of intersection over union and entropy difference between two detection boxes of the same category:

[0017] The image entropy value of each detection box is calculated, and the intersection over union and entropy difference between two detection boxes of the same category are calculated.

[0018] S5: entropy difference guided pseudo label screening:

[0019] After the above steps, entropy difference guided pseudo label screening is performed within each category.

[0020] S6: model training:

[0021] The student model is trained using the finally screened pseudo labels as supervision information.

[0022] S7: update the teacher model:

[0023] The teacher model is updated by Exponential Moving Average (EMA).

[0024] Further, the detection model in S1 adopts a DETR model based on a Transformer architecture (N. Carion et al. "End-to-end object detection with transformers." European conference on computer vision. 2020: 213-229.), specifically:

[0025] ,

[0026] wherein, is a pre-trained source domain detection model, and are the teacher and student models.

[0027] Further, the detection model uses a target detection network DETR (N. Carion et al. "End-to-end object detection with transformers." European conference on computer vision. 2020: 213-229.).

[0028] Further, the operation steps in S2 are:

[0029] S2.1: First, the weak augmentation only includes geometric transformation operations, including random horizontal flip and center crop, wherein the probability of random horizontal flip is 0.5.

[0030] S2.2: Second, the strong augmentation includes color jittering, grayscale and Gaussian blur.

[0031] wherein color jittering is achieved by adjusting brightness, contrast and saturation, and the probability of color jittering is 0.2.

[0032] wherein the probability of grayscale is 0.8 and the probability of Gaussian blur is 0.5.

[0033] S2.3: Finally, on the basis of the strong augmented data, the random patch mask M is used to mask the local area of the image to save it as a mask image, and the patch size is set to 64x64 pixels.

[0034] wherein the probability of random patch mask M is 0.5.

[0035] Further, the cross-domain scenarios used in step S3 are natural weather scene 1 (Cityscapes)—foggy scene (Foggy Cityscapes), natural weather scene 1 (Cityscapes)—natural scene 2 (BDD100K), and generated image scene (Sim10K)—natural scene 1 (Cityscapes).

[0036] Further, the confidence is set to 0.3 in the natural weather scene 1 (Cityscapes)—foggy scene (Foggy Cityscapes) cross-domain migration, and the confidence is set to 0.4 in the natural weather scene 1 (Cityscapes)—natural scene 2 (BDD100K) and generated image scene (Sim10K)—natural scene 1 (Cityscapes) cross-domain migration.

[0037] Further, the operation steps in S4 are as follows:

[0038] S4.1: The image in the detection box region is converted from RGB form to grayscale image, and the image entropy is obtained by calculating the information entropy of 256 gray levels in the detection box region.

[0039] S4.2: The intersection over union IoU of the detection box is calculated.

[0040] S4.3: The entropy difference ED is defined as the absolute difference of the entropy values of the two detection box regions.

[0041] Further, the operation steps in S5 are as follows:

[0042] S5.1: When the intersection over union is less than or equal to a given threshold, it is determined that different target instances exist, and both detection boxes are retained, and when the intersection over union is greater than the given threshold and the entropy difference is less than or equal to a given threshold, it is determined that the same target instance exists, and the detection box with lower entropy value is retained.

[0043] S5.2: When the intersection over union is greater than a given threshold and the entropy difference is greater than a given threshold, it is determined that different target instances exist, and both detection boxes are retained, wherein the intersection over union threshold is 0.5, and the entropy difference threshold is 0.1, which can effectively distinguish detection targets with different complexity features, and the final retained detection box is the final pseudo-label after screening.

[0044] The present application has the following beneficial effects:

[0045] In the present application, for detection boxes of the same category, if the intersection over union thereof exceeds a preset threshold, then the image entropy difference of the detection box area is further calculated: if the entropy difference is less than the threshold, it is determined that the same target instance is retained, and the detection box with smaller entropy value is retained; if the entropy difference is greater than the threshold, it is determined that different target instances are retained, and both are retained. If the intersection over union does not exceed the preset threshold, it is determined that different target instances are retained, and both detection boxes are retained. Finally, the pseudo-labels screened through the final screening are used as supervision information to train the student model, and the teacher model is updated through Exponential Moving Average (EMA) to complete model migration, thereby effectively solving the problem of insufficient reliability of pseudo-labels in the cross-domain scene, and realizing high-performance target detection in the case of no source domain data.

[0046] Of course, implementing any product of the present application does not necessarily require all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The logical block diagram of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0049] Please refer to Figure 1 The present application provides a technical solution: an entropy difference guided source-free domain transfer learning image target detection method, comprising the following steps:

[0050] S1: initialization of teacher and student models:

[0051] The teacher model and the student model are parameterized using the same pre-trained source domain detection model.

[0052] Among them, the detection model uses a DETR model (N. Carion et al. "End-to-end object detection with transformers." European conference on computer vision. 2020: 213-229.) based on the Transformer architecture, specifically:

[0053] ,

[0054] Among them, For the pre-trained source domain detection model, and For the teacher and student models.

[0055] In this embodiment, the detection model uses the target detection network DETR (N. Carion et al. "End-to-end object detection with transformers." European conference on computer vision. 2020: 213-229.).

[0056] S2: Augment the target domain dataset:

[0057] S2.1: First, weak augmentation only contains geometric transformation operations, including random horizontal flip and center crop.

[0058] where the probability of random horizontal flip is 0.5.

[0059] S2.2: Second, strong augmentation includes color jittering, grayscale and Gaussian blur.

[0060] where color jittering is achieved by adjusting brightness, contrast and saturation, and the probability of color jittering is 0.2.

[0061] where the probability of grayscale is 0.8 and the probability of Gaussian blur is 0.5.

[0062] S2.3: Finally, on the basis of strong augmented data, random patch mask M is used to mask the local area of the image and save it as a mask image, and the patch size is set to 64x64 pixels.

[0063] where the probability of random patch mask M is 0.5.

[0064] Correspondingly, the calculation method is:

[0065] ,

[0066] ,

[0067] ,

[0068] where, is the target domain dataset, represents the nth image of the target domain, is the weak augmented image, is the strong augmented image, is the mask image.

[0069] For weak data augmentation, specifically including random horizontal flip and center crop, where the probability of random horizontal flip is 0.5.

[0070] For strong data augmentation, including color jittering, grayscale and Gaussian blur, where color jittering is achieved by adjusting brightness, contrast and saturation, the probability of color jittering is 0.2, the probability of grayscale is 0.8, and the probability of Gaussian blur is 0.5.

[0071] Correspondingly, the datasets used are: Cityscapes (M. Cordts, et al. "The cityscapes dataset for semantic urban scene understanding." Computer Vision and Pattern Recognition. 2016: 3213-3223), Foggy Cityscapes (C. Sakaridis, et al. "Semantic foggy scene understanding with synthetic data." International Journal of Computer Vision, vol. 126, pp. 973-992, 2018.), BDD100K (F. Yu, et al. "Bdd100k: A diverse driving dataset for heterogeneous multitask learning." Computer Vision and Pattern Recognition. 2020: 2636-2645) and Sim10K (M. Johnson-Roberson, et al. "Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks." arXiv preprint arXiv:1610.01983, 2016).

[0072] S3: Use a fixed confidence threshold to filter the teacher model output to generate pseudo labels:

[0073] The weak augmented images The input is input to the teacher model, and the detection results output by the teacher model are screened using a fixed threshold. The screening process retains all detection boxes with a confidence level exceeding the threshold.

[0074] In this step, the cross-domain scenarios used are natural weather scenario 1 (Cityscapes) - foggy scenario (Foggy Cityscapes), natural weather scenario 1 (Cityscapes) - natural scenario 2 (BDD100K), and generated image scenario (Sim10K) - natural scenario 1 (Cityscapes). In the natural weather scenario 1 (Cityscapes) - foggy scenario (Foggy Cityscapes) cross-domain migration, the confidence level is set to 0.3. In the natural weather scenario 1 (Cityscapes) - natural scenario 2 (BDD100K) and generated image scenario (Sim10K) - natural scenario 1 (Cityscapes) cross-domain migration, the confidence level is set to 0.4.

[0075] The screening results are as follows:

[0076] ,

[0077] ,

[0078] wherein, represents the n-th prediction detection box output by the teacher model, which specifically includes the coordinate information of the detection box, represents the class probability vector of the n-th prediction box; is a confidence threshold, , respectively represent the detection box coordinates and the corresponding class probability vectors that satisfy In the natural weather scenario 1 (Cityscapes) - foggy scenario (Foggy Cityscapes) cross-domain migration, it is set to 0.3. In the natural weather scenario 1 (Cityscapes) - natural scenario 2 (BDD100K) and generated image scenario (Sim10K) - natural scenario 1 (Cityscapes) cross-domain migration, it is set to 0.4, is the screened detection box.

[0079] S4: Calculation of intersection over union and entropy difference between two detection boxes of the same class:

[0080] S4.1: Convert the image in the detection box region from RGB form to grayscale image. The image entropy is obtained by calculating the information entropy of the 256-level grayscale value in the detection box region.

[0081] S4.2: Calculate the intersection over union IoU of the detection box.

[0082] S4.3: Entropy difference ED is defined as the absolute difference of the two detection frame region entropy values.

[0083] Accordingly, the specific steps are as follows:

[0084]

[0085]

[0086]

[0087] wherein, is the proportion of pixels with image gray value i in the detection frame region, represents the confidence threshold, after screening, two different high-confidence detection frames belonging to the same target category;

[0088] represents the intersection over union of the detection frames of the same category, is the entropy difference of the detection frames of the same category, represents the detection frames of different categories.

[0089] S5: Pseudo-label screening based on entropy difference guidance:

[0090] S5.1: When the intersection over union is less than or equal to a given threshold, it is determined that it is a different target instance, and both detection frames are retained. When the intersection over union is greater than the given threshold and the entropy difference is less than or equal to the given threshold, it is determined that it is the same target instance, and the detection frame with lower entropy value is retained.

[0091] S5.2: When the intersection over union is greater than the given threshold and the entropy difference is greater than the given threshold, it is determined that it is a different target instance, and both detection frames are retained. The intersection over union threshold is 0.5, and the entropy difference threshold is 0.1. This value can effectively distinguish detection targets with different complexities. The final retained detection frame is the final pseudo-label after screening.

[0092] The specific steps are as follows:

[0093]

[0094] wherein, is the final pseudo-label, is the IoU threshold, set to 0.5, is the entropy difference threshold, set to 0.1. For detection frames of the same category, when the intersection over union is less than or equal to the given threshold, it is determined that it is a different target instance, and both detection frames are retained. When the intersection over union is greater than the given threshold and the entropy difference is less than or equal to the given threshold, it is determined that it is the same target instance, and the detection frame with lower entropy value is retained. ​​​​​​Second, when the intersection over union is greater than a given threshold and the entropy difference is greater than a given threshold, it is determined that it is a different target instance, while retaining two detection boxes.

[0095] S6: The model is trained:

[0096] The final screened pseudo-labels are used as supervision information to train the student model.

[0097] The specific way is: after completing the above steps, the strongly enhanced image and the mask image are input into the student model to obtain the detection result, and the pseudo-labels of step five are used as supervision information to train the student model, specifically as follows:

[0098] ,

[0099] ,

[0100] ,

[0101] ,

[0102] ,

[0103] ,

[0104] wherein, and are the detection results of the student model output for the strongly enhanced data and the mask data respectively, specifically, is the detection box and confidence corresponding to the strongly enhanced data, is the detection box and confidence corresponding to the mask image data. is the final pseudo-label screened in step five. is the total loss function, is the Focal Loss loss, wherein is 0.25, is 2.0, and y is the total number of categories of the detection box. is the bounding box regression loss, is the GIoU loss, wherein C is and the minimum closed box.

[0105] S7: Update the teacher model:

[0106] The teacher model is updated by Exponential Moving Average (EMA).

[0107] Specifically, after the above steps are completed, the teacher model is updated by exponential moving average (EMA), and the specific method is as follows:

[0108] ,

[0109] Wherein, t represents the current training iteration number, t-1 represents the previous iteration, is an EMA decay coefficient, which is set to 0.999 in the cross-domain scene of natural weather scene 1 (Cityscapes) - foggy scene (Foggy Cityscapes) and generated image scene (Sim10K) - natural scene 1 (Cityscapes) cross-scene migration, and is set to 0.9998 in the natural weather scene 1 (Cityscapes) - natural scene 2 (BDD100K) cross-scene migration. And Respectively represent the updated teacher model parameters of the current iteration (step t) and the teacher model parameters of the previous iteration (step t-1), is the trainable parameter of the current student model.

[0110] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details, nor limit the application to the specific embodiments described. Obviously, according to the content of the specification, many modifications and changes can be made. The specification selects and describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and utilize the application. The application is limited only by the claims and their entire scope and equivalents.

Claims

1. An entropy difference guided free-source domain transfer learning image object detection method, characterized in that, Comprising the following steps: S1: initialization of teacher and student models: The teacher model and the student model are parameterized using the same pre-trained source domain detection model; S2: enhance the target domain dataset: First, the target domain dataset is respectively enhanced by strong and weak enhancement, and then the strong enhanced dataset is masked; S3: using a fixed confidence threshold to filter the teacher model output to generate pseudo labels: the weakly augmented images are input to the teacher model and the detection results output by the teacher model are filtered using a fixed threshold, the filtering process retaining all bounding boxes whose confidence exceeds the threshold; S4: calculate the intersection over union and entropy difference between each pair of detection boxes of the same category: After the operation of S3 is completed, the image entropy value of each detection box is calculated, and the intersection over union and entropy difference between each pair of detection boxes of the same category are calculated, and the specific operation is as follows: S4.1: convert the image in the detection box region from RGB form to grayscale image, and the image entropy is obtained by calculating the information entropy of the 256-level grayscale value in the detection box region; S4.2: calculate the intersection over union IoU of the detection box; S4.3: the entropy difference ED is defined as the absolute difference between the entropy values of the two detection box regions; S5: entropy difference guided pseudo label screening: After the above steps are completed, entropy difference guided pseudo label screening is performed within each category, and the specific operation is as follows: S5.1: when the intersection over union is less than or equal to the given threshold, it is determined that the two detection boxes are different target instances, and both detection boxes are retained; when the intersection over union is greater than the given threshold and the entropy difference is less than or equal to the given threshold, it is determined that the two detection boxes are the same target instance, and the detection box with lower entropy value is retained; S5.2: when the intersection over union is greater than the given threshold and the entropy difference is greater than the given threshold, it is determined that the two detection boxes are different target instances, and both detection boxes are retained, wherein the intersection over union threshold is 0.5 and the entropy difference threshold is 0.1, which can effectively distinguish detection targets with different complexity features, and the final retained detection box is the final pseudo label after screening; S6: model training: The student model is trained using the final screened pseudo label as the supervision information; S7: update the teacher model: The teacher model is updated by exponential moving average.

2. The entropy difference guided source-free domain adaptation learning image object detection method according to claim 1, characterized in that, The detection model in S1 uses DETR model based on Transformer architecture, specifically: ， wherein, is a pre-trained source domain detection model, and is a teacher and student model.

3. The entropy difference guided source-free domain adaptation learning image object detection method according to claim 1, characterized in that, The operation steps in S2 are as follows: S2.1: first, weak enhancement only includes geometric transformation operations, including random horizontal flip and center crop, wherein the probability of random horizontal flip is 0.5; S2.2: second, strong enhancement includes color jitter, grayscale and Gaussian blur; Wherein, color jitter is achieved by adjusting brightness, contrast and saturation, and the probability of color jitter is 0.2; Wherein, the probability of grayscale is 0.8 and the probability of Gaussian blur is 0.5; S2.3: finally, on the basis of strong enhancement data, the local area of the image is randomly patched and masked M to save it as a mask image, and the patch size is set to 64*64 pixels, wherein the probability of random patch mask M is 0.

5.

4. The entropy difference guided source-free domain adaptation learning image object detection method according to claim 1, characterized in that, The cross-domain scenarios used in step S3 include natural weather scene 1-fog scene, natural weather scene 1-natural scene 2, and generated image scene-natural scene 1.

5. The entropy difference guided free-source domain transfer learning image object detection method according to claim 4, characterized in that, In the natural weather scene 1-fog scene cross-domain migration, the confidence is set to 0.3, and in the natural weather scene 1-natural scene 2 and generated image scene-natural scene 1 cross-domain migration, the confidence is set to 0.

4. In the natural weather scene 1-fog scene cross-domain migration, the confidence is set to 0.3, and in the natural weather scene 1-natural scene 2 and generated image scene-natural scene 1 cross-domain migration, the confidence is set to 0.4.

Images