Entropy-difference-guided source-domain-free transfer learning image target detection method

Abstract

The invention discloses an entropy-difference-guided source-domain-free transfer learning image target detection method, and belongs to the technical field of image processing and computer vision. The method comprises the following steps: initializing a teacher model and a student model, enhancing a target domain data set, screening teacher model output by using a fixed confidence threshold to generate a pseudo tag, calculating an image entropy difference and an intersection-to-union ratio of the same category of detection frames, screening the pseudo tag based on entropy difference guidance, training the models and updating the teacher model. According to the method, the finally screened pseudo tag is used as supervision information to train the student model, and the teacher model is updated through index moving average to complete model migration, so that the problem of insufficient reliability of the pseudo tag in a cross-domain scene is effectively solved, and high-performance target detection under the condition of passive domain data is realized.

Description

Technical Field

[0001] The present invention relates to the technical fields of image processing and computer vision, and in particular to an entropy difference-guided source domain transfer learning-free image target detection method. Background Art

[0002] In object detection tasks, traditional deep learning methods typically rely on large amounts of labeled data for supervised training. However, in real-world industrial applications, data in the target domain (such as foggy days or street scenes from different cities) is often unlabeled, and manual labeling requires significant manpower, financial resources, and time. Furthermore, source domain data may be inaccessible and unusable due to privacy protection requirements (such as medical imaging), data storage limitations (such as satellite remote sensing data), or commercial confidentiality. This dual dilemma of "unseen source domain and unlabeled target domain" has given rise to the emerging research direction of source-free domain adaptive object detection. This method utilizes only a pre-trained source model and unlabeled data from the target domain, achieving cross-domain knowledge transfer through pseudo-label self-training techniques.

[0003] In recent years, the average teacher framework has become the mainstream framework for source-free domain transfer target detection tasks, where the quality of pseudo-labels plays a decisive role in the effectiveness of knowledge transfer. Among source-free domain transfer target detection methods, non-maximum suppression (NMS) is a common post-processing technique for obtaining reliable pseudo-labels. NMS mainly filters overlapping detection results by calculating the intersection over union (IoU) and category confidence between detection boxes: first, an IoU threshold is set (usually 0.5). When the IoU of two detection boxes exceeds this threshold, the detection box with higher confidence is retained and the detection box with lower confidence is suppressed. However, NMS has two main problems when removing overlapping pseudo-labels: First, when the detection boxes of two different target instances overlap, NMS will retain the detection box with higher confidence and discard the detection box with lower confidence, which reduces the utilization of pseudo labels and affects the model performance. Second, when multiple detection boxes of the same target instance overlap, NMS only retains the detection box with the highest confidence, while the detection box with lower confidence may actually be closer to the true annotation; These problems lead to the low quality of the generated pseudo labels, which not only weakens the performance of the student model, but also hinders the positive update of the teacher model in the average teacher framework. Therefore, to address the above problems, an entropy difference guided source-free transfer learning image object detection method is proposed. Summary of the Invention

[0004] The purpose of the present invention is to provide an entropy difference-guided source domain transfer learning-free image target detection method. By initializing the teacher and student model parameters and enhancing the target domain data, pseudo labels are screened by confidence thresholds. Then, the image entropy difference indicator is innovatively introduced to optimize and screen pseudo labels, thereby solving the problems of pseudo label omission and false detection caused by traditional non-maximum suppression methods in cross-domain target detection due to their simple reliance on intersection-over-union and detection box confidence.

[0005] The present invention is achieved through the following technical solutions: The present invention provides an entropy difference-guided source domain transfer learning-free image target detection method, comprising the following steps: S1: Initialization of teacher and student models: The teacher model and the student model use exactly the same pre-trained source domain detection model for parameter initialization.

[0006] S2: Enhanced target domain dataset: First, the target domain dataset is strongly enhanced and weakly enhanced respectively, and then the strongly enhanced dataset is masked.

[0007] S3: Generate pseudo labels by filtering the teacher model output using a fixed confidence threshold: The weakly enhanced image The input is sent to the teacher model, and the detection results output by the teacher model are filtered using a fixed threshold. The screening process retains all detection boxes whose confidence exceeds the threshold.

[0008] S4: Calculation of the intersection-over-union ratio and entropy difference between two detection boxes of the same category: Calculate the image entropy value of each detection box, and calculate the intersection-over-union ratio and entropy difference of two detection boxes of the same category.

[0009] S5: Pseudo-label screening based on entropy difference guidance: After completing the above steps, entropy difference-guided pseudo-label screening is performed within each category.

[0010] S6: Model training: The student model is trained using the final filtered pseudo labels as supervision information.

[0011] S7: Update the teacher model: The teacher model is updated using the Exponential Moving Average (EMA).

[0012] Further, the detection model in S1 adopts a DETR model (N. Carion et al. "End-to-end object detection with transformers." European conference on computer vision. 2020: 213-229.) based on a Transformer architecture, specifically: , wherein, is a pre-trained source domain detection model, and are teacher and student models.

[0013] Further, the detection model uses a target detection network DETR (N. Carion et al. "End-to-end object detection with transformers." European conference on computer vision. 2020: 213-229.).

[0014] Further, the operation steps in S2 are: S2.1: First, the weak augmentation only includes geometric transformation operations, including random horizontal flip and center crop, wherein the probability of random horizontal flip is 0.5.

[0015] S2.2: Second, the strong augmentation includes color jittering, grayscale, and Gaussian blur.

[0016] wherein color jittering is achieved by adjusting brightness, contrast, and saturation, and the probability of color jittering is 0.2.

[0017] wherein the probability of grayscale is 0.8 and the probability of Gaussian blur is 0.5.

[0018] S2.3: Finally, based on the strong augmented data, a random patch mask M is used to mask the local area of the image to save it as a mask image, and the patch size is set to 64x64 pixels.

[0019] wherein the probability of random patch mask M is 0.5.

[0020] Furthermore, the cross-domain scenes used in step S3 include natural weather scene 1 (Cityscapes) - foggy scene (Foggy Cityscapes), natural weather scene 1 (Cityscapes) - natural scene 2 (BDD100K), and generated image scene (Sim10K) - natural scene 1 (Cityscapes).

[0021] Furthermore, the confidence level is set to 0.3 in the cross-domain migration of natural weather scene 1 (Cityscapes) to foggy scene (Foggy Cityscapes), and the confidence level is set to 0.4 in the cross-domain migration of natural weather scene 1 (Cityscapes) to natural scene 2 (BDD100K) and generated image scene (Sim10K) to natural scene 1 (Cityscapes).

[0022] Furthermore, the operation steps in S4 are: S4.1: Convert the image within the detection frame area from RGB format to grayscale. The image entropy is obtained by calculating the information entropy of the 256 grayscale values ​​within the detection frame area.

[0023] S4.2: Calculate the intersection over union (IoU) of the detection boxes.

[0024] S4.3: The entropy difference ED is defined as the absolute difference between the entropy values ​​of the two detection box regions.

[0025] Furthermore, the operation steps in S5 are: S5.1: When the IoU ratio is less than or equal to a given threshold, they are judged to be different target instances and the two detection boxes are retained. When the IoU ratio is greater than a given threshold and the entropy difference is less than or equal to a given threshold, they are judged to be the same target instance and the detection box with the lower entropy value is retained.

[0026] S5.2: When the intersection-over-union ratio is greater than a given threshold and the entropy difference is greater than a given threshold, they are determined to be different target instances, and the two detection boxes are retained. The intersection-over-union ratio threshold is 0.5, and the entropy difference threshold is 0.1. This value can effectively distinguish detection targets with different complexity features. The detection box that is finally retained is the final pseudo-label after screening.

[0027] The present invention has the following beneficial effects: In the present application, for detection boxes of the same category, if the intersection over union thereof exceeds a preset threshold, then the image entropy difference of the detection box area is further calculated: if the entropy difference is less than a threshold, it is determined that the same target instance is detected, and the detection box with smaller entropy value is retained; if the entropy difference is greater than the threshold, it is determined that different target instances are detected, and both of them are retained. If the intersection over union does not exceed the preset threshold, it is determined that different target instances are detected, and both of the detection boxes are retained. Finally, the pseudo-labels screened through the final screening are used as supervision information to train the student model, and the teacher model is updated through Exponential Moving Average (EMA) to complete model migration, thereby effectively solving the problem of insufficient reliability of pseudo-labels in a cross-domain scene, and realizing high-performance target detection in the case of no source domain data.

[0028] Of course, implementing any product of the present application does not necessarily require all the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The logical block diagram of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0031] Please refer to Figure 1 The present application provides a technical solution: an entropy difference guided source-free domain transfer learning image target detection method, comprising the following steps: S1: initialization of teacher and student models: The teacher model and the student model are parameterized by using the same pre-trained source domain detection model.

[0032] Among them, the detection model uses a DETR model (N. Carion et al. "End-to-end object detection with transformers." European conference on computer vision. 2020: 213-229.) based on the Transformer architecture, specifically: , Among them, is a pre-trained source domain detection model, and Teacher and student models.

[0033] In this embodiment, the detection model uses a target detection network DETR (N. Carion et al. "End-to-end object detection with transformers." European conference on computer vision. 2020: 213-229.).

[0034] S2: Augment the target domain dataset: S2.1: First, weak augmentation only contains geometric transformation operations, including random horizontal flip and center crop.

[0035] wherein the probability of random horizontal flip is 0.5.

[0036] S2.2: Second, strong augmentation includes color jittering, graying and Gaussian blur.

[0037] wherein color jittering is achieved by adjusting the brightness, contrast and saturation, and the probability of color jittering is 0.2.

[0038] wherein the probability of graying is 0.8 and the probability of Gaussian blur is 0.5.

[0039] S2.3: Finally, on the basis of strong augmented data, the random patch mask M is used to mask the local area of the image to save it as a mask image, and the patch size is set to 64x64 pixels.

[0040] wherein the probability of random patch mask M is 0.5.

[0041] Correspondingly, the calculation method is: , , , wherein, is the target domain dataset, represents the nth image in the target domain, is the weak augmented image, is the strong augmented image, is the mask image.

[0042] is weak data augmentation, specifically including random horizontal flip and center crop, wherein the probability of random horizontal flip is 0.5.

[0043] For strong data augmentation, including color jittering, grayscaling and Gaussian blur, wherein color jittering is achieved by adjusting the brightness, contrast and saturation, the probability of color jittering is 0.2, the probability of grayscaling is 0.8, and the probability of Gaussian blur is 0.5.

[0044] Correspondingly, the datasets used are: Cityscapes (M. Cordts, et al. "The cityscapes dataset for semantic urban scene understanding." Computer Vision and Pattern Recognition. 2016: 3213-3223), Foggy Cityscapes (C. Sakaridis, et al. "Semantic foggy scene understanding with synthetic data." International Journal of Computer Vision, vol. 126, pp. 973-992, 2018.), BDD100K (F. Yu, et al. "Bdd100k: A diverse driving dataset for heterogeneous multitask learning." Computer Vision and Pattern Recognition. 2020: 2636-2645) and Sim10K (M. Johnson-Roberson, et al. "Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks." arXiv preprint arXiv:1610.01983, 2016).

[0045] S3: Generate pseudo labels using a fixed confidence threshold to filter the teacher model output: The weakly augmented images are input into the teacher model, and the detection results output by the teacher model are filtered using a fixed threshold, and the filtering process retains all detection boxes with a confidence score above the threshold.

[0046] In this step, the cross-domain scenes used are natural weather scene 1 (Cityscapes) - foggy scene (FoggyCityscapes), natural weather scene 1 (Cityscapes) - natural scene 2 (BDD100K), and generated image scene (Sim10K) - natural scene 1 (Cityscapes). The confidence is set to 0.3 in the cross-domain migration of natural weather scene 1 (Cityscapes) - foggy scene (FoggyCityscapes), and the confidence is set to 0.4 in the cross-domain migration of natural weather scene 1 (Cityscapes) - natural scene 2 (BDD100K) and generated image scene (Sim10K) - natural scene 1 (Cityscapes).

[0047] The screening results are as follows: , , in, Represents the nth predicted detection box output by the teacher model, which specifically contains the coordinate information of the detection box. Represents the category probability vector of the nth prediction box; is the confidence threshold, , Respectively express satisfaction The detection box coordinates and their corresponding category probability vectors are set to 0.3 in the cross-domain migration of natural weather scene 1 (Cityscapes) to foggy scene (Foggy Cityscapes), and to 0.4 in the cross-domain migration of natural weather scene 1 (Cityscapes) to natural scene 2 (BDD100K) and generated image scene (Sim10K) to natural scene 1 (Cityscapes). is the detection frame after filtering.

[0048] S4: Calculation of the intersection-over-union ratio and entropy difference between two detection boxes of the same category: S4.1: Convert the image within the detection frame area from RGB format to grayscale. The image entropy is obtained by calculating the information entropy of the 256 grayscale values ​​within the detection frame area.

[0049] S4.2: Calculate the intersection over union (IoU) of the detection boxes.

[0050] S4.3: The entropy difference ED is defined as the absolute difference between the entropy values ​​of the two detection box regions.

[0051] Accordingly, the steps to obtain are as follows: , , , in, Detection frame area The proportion of pixels with grayscale value i in the domain, , Indicates that the confidence threshold has been passed After screening, two different high-confidence detection boxes belong to the same target category; Represents the intersection-over-union ratio of detection boxes of the same category, is the entropy difference of the detection boxes of the same category, Represents detection boxes of different categories.

[0052] S5: Pseudo-label screening based on entropy difference guidance: S5.1: When the IoU ratio is less than or equal to a given threshold, they are judged to be different target instances and the two detection boxes are retained. When the IoU ratio is greater than a given threshold and the entropy difference is less than or equal to a given threshold, they are judged to be the same target instance and the detection box with the lower entropy value is retained.

[0053] S5.2: When the intersection-over-union ratio is greater than a given threshold and the entropy difference is greater than a given threshold, they are determined to be different target instances, and the two detection boxes are retained. The intersection-over-union ratio threshold is 0.5, and the entropy difference threshold is 0.1. This value can effectively distinguish detection targets with different complexity features. The detection box that is finally retained is the final pseudo-label after screening.

[0054] The details are as follows: , in, is the final pseudo label, is the IoU threshold, set to 0.5, is the entropy difference threshold, set to 0.1. For detection frames of the same category, when the intersection-over-union ratio is less than or equal to the given threshold, they are judged as different target instances and the two detection frames are retained. When the intersection-over-union ratio is greater than the given threshold and the entropy difference is less than or equal to the given threshold, they are judged as the same target instance and the detection frame with the lower entropy value is retained. ,Secondly, when the intersection-over-union ratio is greater than a given threshold and the ,entropy difference is greater than a given threshold, they are determined to be different ,target instances and both detection boxes are retained.

[0055] S6: Model training: The student model is trained using the final filtered pseudo labels as supervision information.

[0056] The specific method is: After completing the above steps, the enhanced image and mask image Input the data into the student model and obtain the test results. Use the pseudo labels from step 5 as supervision information to train the student model. The details are as follows: , , , , , , in, and They are the detection results output by the student model corresponding to the strongly enhanced data and mask data, specifically, To strongly enhance the detection box and confidence corresponding to the data, is the detection box and confidence corresponding to the mask image data. The final pseudo label selected in step 5. is the total loss function, is the Focal Loss loss, where is 0.25, is 2.0, the total number of categories of the y detection box. is the bounding box regression loss, is the GIoU loss, where C is and The minimum closure box of .

[0057] S7: Update the teacher model: The teacher model is updated using the Exponential Moving Average (EMA).

[0058] The specific method is: After completing the above steps, the teacher model is updated by exponential moving average (EMA), as follows: , Among them, t represents the current training iteration number, t-1 represents the previous iteration, Expressed as the EMA attenuation coefficient, it is set to 0.999 in the cross-domain scenarios of natural weather scene 1 (Cityscapes) - foggy scene (Foggy Cityscapes) and generated image scene (Sim10K) - natural scene 1 (Cityscapes), and is set to 0.9998 in the cross-scene migration of natural weather scene 1 (Cityscapes) - natural scene 2 (BDD100K). and represent the updated teacher model parameters of the current iteration (step t) and the teacher model parameters of the previous iteration (step t-1), respectively. are the trainable parameters of the current student model.

[0059] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An entropy difference-guided source-domain transfer learning-free image object detection method, characterized in that: The following steps are involved: S1: Initialization of teacher and student models: Initialize the parameters of the teacher model and the student model using the same pre-trained source domain detection model; S2: Enhanced target domain dataset: First, the target domain dataset is strongly enhanced and weakly enhanced respectively, and then the strongly enhanced dataset is masked. S3: Generate pseudo labels by filtering the teacher model output using a fixed confidence threshold: The weakly enhanced image Input to the teacher model, and use a fixed threshold to filter the detection results output by the teacher model. The screening process retains all detection boxes whose confidence exceeds the threshold; S4: Calculation of the intersection-over-union ratio and entropy difference between two detection boxes of the same category: After completing the operation of S3, the image entropy value of each detection frame is calculated, and the intersection-over-union ratio and entropy difference of two detection frames of the same category are calculated; S5: Pseudo-label screening based on entropy difference guidance: After completing the above steps, perform entropy difference guided pseudo label screening within each category; S6: Model training: Use the final filtered pseudo labels as supervision information to train the student model; S7: Update the teacher model: The teacher model is updated using exponential moving average.

2. The entropy difference-guided source-domain transfer learning image target detection method according to claim 1, characterized in that: The detection model in S1 adopts the DETR model based on the Transformer architecture, specifically: ， in, is the pre-trained source domain detection model, and Model for teachers and students.

3. The entropy difference-guided source-domain transfer learning image target detection method according to claim 1, characterized in that: The operation steps in S2 are: S2.1: First, weak enhancement only includes geometric transformation operations, including random horizontal flipping and center cropping, where the probability of random horizontal flipping is 0.5; S2.2: Secondly, strong enhancement includes color dithering, grayscale, and Gaussian blur; Among them, color dithering is achieved by adjusting brightness, contrast and saturation, and the probability of color dithering is 0.2; Among them, the probability of grayscale is 0.8, and the probability of Gaussian blur is 0.5; S2.3: Finally, based on the strong enhancement data, a random patch mask M is used to block the local area of ​​the image and save it as a mask image. The patch size is set to 64×64 pixels, where the probability of the random patch mask M is 0.

5.

4. The entropy difference-guided source-domain transfer learning image target detection method according to claim 1, characterized in that: The cross-domain scenes used in step S3 include natural weather scene 1—fog scene, natural weather scene 1—natural scene 2, and generated image scene—natural scene 1.

5. The entropy difference-guided source-domain transfer learning image target detection method according to claim 4, characterized in that: The confidence level is set to 0.3 in the cross-domain migration from natural weather scene 1 to foggy scene, and is set to 0.4 in the cross-domain migration from natural weather scene 1 to natural scene 2 and from generated image scene to natural scene 1.

6. The entropy difference guided source domain transfer learning image target detection method according to claim 1, characterized in that: The operation steps in S4 are: S4.1: Convert the image within the detection frame from RGB to grayscale. The image entropy is obtained by calculating the information entropy of the 256 grayscale values ​​within the detection frame. S4.2: Calculate the intersection over union (IoU) of the detection box. S4.3: The entropy difference ED is defined as the absolute difference between the entropy values ​​of the two detection box regions.

7. The entropy difference-guided source-domain transfer learning image target detection method according to claim 6, characterized in that: The operation steps in S5 are: S5.1: When the IoU ratio is less than or equal to a given threshold, the two detection boxes are determined to be different target instances and the two detection boxes are retained. When the IoU ratio is greater than a given threshold and the entropy difference is less than or equal to a given threshold, the two detection boxes are determined to be the same target instance and the detection box with the lower entropy value is retained. S5.2: When the intersection-over-union ratio is greater than a given threshold and the entropy difference is greater than a given threshold, they are determined to be different target instances, and the two detection boxes are retained. The intersection-over-union ratio threshold is 0.5, and the entropy difference threshold is 0.

1. This value can effectively distinguish detection targets with different complexity features. The detection box that is finally retained is the final pseudo-label after screening.

Images