Source-free domain adaptation learning method for target detection based on dual adapter pseudo-label generation

Abstract

The application discloses a source-free domain migration learning target detection method based on double adapter pseudo-label generation, first, a double teacher model and a student model are initialized, and data enhancement is performed on target domain images; a confidence adapter is introduced, according to the confidence distribution characteristics of each category, the confidence screening range is dynamically adjusted through statistical analysis, and candidate pseudo-labels are generated; a category imbalance adapter is introduced, according to the frequency of each category, the final confidence threshold value is obtained by fusing the confidence historical threshold value and the current threshold value; finally, the student model is trained by using the screened pseudo-labels, and the double teacher model parameters are updated through the exponential moving average, so that the source-free domain migration of the model is realized. The source-free domain target detection method based on the double adapter pseudo-label generation effectively alleviates the problems of inconsistent pseudo-label confidence distribution and category sample imbalance, and significantly improves the pseudo-label quality and cross-domain detection performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image processing and computer vision, and relates to a target detection method, in particular to a source-free domain transfer learning target detection method based on double adapter pseudo-label generation. BACKGROUND

[0002] As one of the core tasks in the field of computer vision, target detection has a wide range of applications in many practical scenarios such as automatic driving and intelligent security. With the continuous development of deep neural network technology, the target detection model trained by relying on large-scale labeled data has achieved remarkable performance. However, when these models are applied to the target domain for testing, the performance of the models often decreases significantly. In order to solve this problem, an unsupervised domain adaptive target detection method is proposed. This method uses feature alignment technology to effectively enhance the generalization ability of the model in the target domain. However, this method requires source domain data during training, and once the source domain data cannot be obtained due to privacy protection or security factors, this method is difficult to play a role. In this context, a source-free target detection (SFOD) method is proposed. This method only needs to use the trained source domain model and the unlabeled data of the target domain to realize the adaptive adjustment of the model in the target domain.

[0003] In recent years, pseudo-label paradigm has become a core technology in SFOD tasks. Under this paradigm, a teacher-student framework is constructed, the teacher model is used to infer and predict the weakly augmented target domain images to generate pseudo-labels, these pseudo-labels are used as supervision information to train the student model, and finally the student model is adapted to the target domain. In particular, the quality of the pseudo-labels plays a decisive role in the performance of the model. When generating pseudo-labels, a confidence threshold is set for screening, and when the confidence is higher than the threshold, the predicted result is regarded as a pseudo-label. Therefore, the setting of the confidence threshold is one of the key factors that affect the quality of the pseudo-labels and in turn determine the domain adaptation effect of the model. Due to the differences in confidence distribution among different classes, and the unbalanced distribution of class samples in the target domain, how to accurately obtain the confidence threshold becomes a big challenge. The traditional fixed confidence threshold method sets a pre-defined threshold for all classes, but this strategy cannot take into account the characteristics of all classes. Specifically, a higher threshold may result in that some classes with low confidence cannot generate effective pseudo-labels, while a lower threshold may introduce a large number of noisy pseudo-labels, which seriously interfere with the training process of the model. In order to overcome the limitations of the fixed threshold method, researchers have proposed a dynamic threshold strategy in recent years. This method adaptively adjusts the confidence threshold on a class-by-class basis, and usually calculates it by fusing the historical threshold and the mean confidence of the predictions falling within a pre-set confidence interval. However, the confidence statistical interval relied on by this method is usually fixed. Since this fixed interval is likely to contain a large number of low-confidence noisy predictions, the calculated mean confidence is pulled down, which makes the subsequent updated threshold continue to decrease, affecting the quality of the pseudo-labels. Therefore, this method cannot fundamentally solve the problem of large differences in confidence distribution among different classes. SUMMARY

[0004] To solve the problem of low quality of pseudo-labels caused by significant differences in confidence distribution among different classes and unbalanced class samples in the target domain, the application proposes a source-free transfer learning target detection method based on double adapter pseudo-label generation, which comprises:

[0005] Initialize the double teacher model and the student model, obtain the target domain image dataset, and perform strong data augmentation 1 processing, strong data augmentation 2 processing and weak data augmentation processing on the target domain image dataset, and perform mask representation on the image dataset after strong data augmentation 1 processing and strong data augmentation 2 processing.

[0006] Introduce a confidence adapter, input the image after weak data augmentation processing into the double teacher model, use the confidence adapter to dynamically adjust the lower bound of the confidence of the detection results output by the double teacher model, select the candidate pseudo-labels, and calculate the current confidence threshold of each class.

[0007] An imbalanced class adapter is introduced to fuse historical confidence thresholds and current confidence thresholds based on the candidate pseudo-labels and the frequency of each class, thereby obtaining the current final confidence threshold for each class. Pseudo-labels are then selected based on the final confidence threshold.

[0008] To train the student model, the images processed by strong data augmentation 1 and strong data augmentation 2, along with their corresponding mask images, are alternately input into the student model in each training cycle to obtain detection results. Then, the selected pseudo-labels are used as supervision signals to continuously train the student model.

[0009] The parameters of the dual-teacher model are updated using the exponential moving average method to achieve passive domain transfer of the model.

[0010] Preferably, the image after weak data augmentation is input into the dual-teacher model, which uses an alternating activation method, in which one of the teacher models is activated in turn for inference in each training cycle.

[0011] Preferably, the step of dynamically adjusting the lower bound of the confidence level of the detection results output by the dual-teacher model using a confidence adapter includes:

[0012] First, calculate the initial confidence lower bounds for each category. ,

[0013] ,

[0014] in, This represents the increment step size in each iteration during the refinement of the lower confidence bound. It is the index of the current iteration. This represents the total number of valid iterations. It is the maximum value of the lower bound of the confidence level. and Calculate the mean and median of a given set, respectively. and These represent the mean function and the median function, respectively. This indicates the relationship between the predictions from the dual-teacher model and the category. The corresponding set of confidence scores, This indicates that it falls within the adjustment range. A subset of;

[0015] Then, obtain the initial lower bound of confidence for each category. Then, a median boundary calibration mechanism is used to further optimize the lower bound of the confidence level for each category, and the final lower bound of the confidence level is dynamically determined. ,

[0016] ,

[0017] wherein, is the current candidate interval is the median of the confidence scores.

[0018] Preferably, the current final confidence threshold of each class is obtained by fusing the confidence history threshold and the current confidence threshold according to the candidate pseudo-label and the frequency of each class, and the specific calculation is as follows:

[0019] ,

[0020] wherein, represents the number of candidate pseudo-labels of class , is the total number of classes existing in the current iteration, and represents the frequency of class , represents the fusion weight, is the base weight coefficient, is a hyperparameter, is a balance factor, defined as the average of the frequencies of all classes in the current iteration, represents the confidence threshold of class in the current batch, is the confidence threshold of class in the history batch, is the final confidence threshold of class .

[0021] Preferably, the pseudo-labels generated by the dual-teacher model are screened according to the final confidence threshold, so as to ensure the diversity and robustness of the pseudo-labels.

[0022] Further, the weak data enhancement processing includes random horizontal flip and center crop; the strong data enhancement 1 processing includes color jittering and grayscale; and the strong data enhancement 2 processing includes Gaussian blur.

[0023] Further, the mask representation of the image dataset after the strong data enhancement 1 processing and the strong data enhancement 2 processing specifically includes: on the basis of the strong data enhancement 1 processing and the strong data enhancement 2 processing, the local area of the image is randomly masked by a patch mask M, and the size of the patch is set to 64x64 pixels.

[0024] Further, the dual-teacher model and the student model both use a target detection network DETR.

[0025] The target detection method for source-free domain migration learning based on double adapter pseudo-label generation provided in the application first initializes a double teacher model and a student model of a pre-trained source domain detection model, and performs weak enhancement, strong enhancement and random mask on target domain data; then, the weak enhancement data is alternately input into the double teacher model, and the two kinds of strong enhancement target data and the mask data are alternately input into the student model; secondly, a confidence adapter is introduced, the confidence range is dynamically adjusted according to the confidence distribution characteristics of each class output by the teacher model, candidate pseudo-labels are generated, and the current threshold is obtained; then, a class imbalance adapter is introduced, the current final confidence threshold is obtained by fusing the confidence historical threshold and the current confidence threshold according to the frequency of each class; finally, the student model is supervised and trained by using the screened pseudo-labels, and the double temporary teacher model parameters are updated through Exponential Moving Average (EMA). The method provided in the application solves the problem of low pseudo-label quality caused by significant differences in confidence distribution of different classes and class sample imbalance in the target domain, effectively alleviates the inconsistency of pseudo-label confidence distribution and class sample imbalance, significantly improves the pseudo-label quality and cross-domain detection performance, and enhances the self-adaptive ability of the detection model under the condition of no source domain data. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The figure is a basic flowchart of the method of the embodiment of the application.

[0027] Figure 2 The figure is a basic flowchart of the method of the embodiment of the application. DETAILED DESCRIPTION

[0028] In order to better understand the above technical solutions, the exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0029] As an implementation scheme, Figures 1-2 The figure is a basic flowchart of the method of the embodiment of the application.

[0030] As Figures 1-2 shown, the method in this embodiment includes the following steps:

[0031] Step one, initialize the double teacher model and the student model.

[0032] Initialize the pre-trained source domain detection model, the dual-teacher model , and the student model as follows:

[0033] ,

[0034] wherein, is the pre-trained source domain detection model, , , teacher model I, teacher model II and student model, respectively.

[0035] wherein the detection model used in the embodiment is a target detection network DETR (DEtection TRansformer).

[0036] Step two, enhance the target domain data set.

[0037] Obtain the target domain image data set, first perform strong data enhancement 1 processing, strong data enhancement 2 processing and weak data enhancement on the target domain image data set, and then perform mask representation on the image data set after strong data enhancement 1 processing and strong data enhancement 2 processing. Specifically, as shown below:

[0038] ,

[0039] wherein, is the target domain data set, represents the nth image of the target domain. is the weak data enhanced image, is the strong data enhanced 1 image, is the mask image corresponding to the strong data enhanced 1 image. is the strong data enhanced 2 image, is the mask image corresponding to the strong data enhanced 2 image. is weak data enhancement, specifically including random horizontal flip (probability 0.5) and center crop. is strong data enhancement 1 processing, including color jitter (brightness / contrast / saturation adjustment, probability 0.2), and grayscale (probability 0.8). is strong data enhancement 2 processing, including Gaussian blur (probability 0.5). Finally, on the basis of strong enhancement data, random patch mask M (probability 0.5) is used to mask the local area of the image, and the patch size is set to 64x64 pixels.

[0040] The image datasets used in this embodiment are: Cityscapes (M. Cordts, et al. "The cityscapes dataset for semantic urban scene understanding." Computer Vision and Pattern Recognition. 2016: 3213-3223), Foggy Cityscapes (C. Sakaridis, et al. "Semantic foggy scene understanding with synthetic data." International Journal of Computer Vision, vol. 126, pp. 973-992, 2018.), BDD100K (F. Yu, et al. "Bdd100k: A diverse driving dataset for heterogeneous multitask learning." Computer Vision and Pattern Recognition. 2020: 2636-2645) and Sim10K (M. Johnson-Roberson, et al. "Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks. arXiv preprint arXiv:1610.01983 , 2016). Cross-domain scenes are natural weather scene 1 (Cityscapes) - foggy scene (Foggy Cityscapes); natural weather scene 1 (Cityscapes) - natural scene 2 (BDD100K); generated image scene (Sim10K) - natural scene 1 (Cityscapes).

[0041] Step three, using a confidence adapter to screen candidate pseudo-labels output by the dual teacher model.

[0042] After completing step one and step two, the weak data augmented images are alternately input into the dual teacher model in each training round, and the confidence adapter is used to dynamically adjust the lower bound of the confidence of the detection results output by the teacher model, screen out candidate pseudo-labels, and calculate the current confidence threshold, which is calculated as follows:

[0043] ,

[0044] where, denotes the incremental step size of each iteration in the lower bound of confidence refinement process, is the index of the current iteration, denotes the total number of valid iterations, is the maximum value of the lower bound of confidence, and are the mean and median of a given set respectively, and denote the mean function and median function respectively, denotes the set of confidence scores corresponding to class predicted from the dual teacher model, denotes the subset of that falls within the adjustment range, is the lower bound of confidence.

[0045] After obtaining the initial lower bound of confidence for each class , a median boundary calibration mechanism is used to further optimize the lower bound of confidence for each class. First, the median m of the confidence scores within the current candidate interval is calculated, and then m is compared with the preset maximum allowed lower bound to dynamically determine the final lower bound .

[0046] ,

[0047] Finally, based on the determined lower bound of confidence, the confidence threshold used for class in the current training round t is calculated:

[0048] ,

[0049] Teacher model I and teacher model II participate in candidate pseudo-label generation alternately in different rounds, i.e., only one of the teacher models is activated for inference in each training round.

[0050] Step four, calculate the final confidence threshold using the unbalanced adapter.

[0051] On the basis of step three, the current final confidence threshold is obtained by fusing the historical threshold and the current confidence threshold according to the candidate pseudo-labels and the frequency of each class:

[0052] ,

[0053] where, denotes the number of candidate pseudo-labels for class , is the total number of classes present in the current iteration, and represents the frequency of class , represents the fusion weight, is the base weight coefficient, is a hyperparameter, is a balancing factor defined as the average of all class frequencies in the current iteration, represents the confidence threshold for class in the current batch, is the confidence threshold for class in the historical batch, is the final confidence threshold for class . The pseudo-labels are filtered according to the final confidence threshold. Taking the teacher model I as an example, the specific process is as follows:

[0054] ,

[0055] wherein, represents the class confidence of the nth predicted bounding box output by the teacher model I, which specifically contains the coordinate information of the bounding box, represents the class probability vector of the nth predicted bounding box, and respectively represent the corresponding bounding box coordinate information and class probability vector after filtering according to the confidence threshold, is the confidence threshold, is the final pseudo-label.

[0056] Step five, train the student model.

[0057] After completing the above steps, the strong augmented image , and the mask image , are alternately input into the student model to obtain the detection result in each training period, and the pseudo-labels filtered in step four are combined as a supervision signal to continuously train the student model. The pseudo-labels are alternately generated by the double-teacher model to ensure the diversity and robustness of the pseudo-labels. Taking the strong image and the mask image as an example, the prediction result is:

[0058] ,

[0059] The total loss function is:

[0060] ,

[0061] wherein, and respectively are the detection results output by the student model corresponding to the strong augmented data 1 and its corresponding mask data 1, specifically, , are the bounding boxes and confidence of the strong augmented data 1, , are the bounding boxes and confidence of the mask data 1. is the total loss function, is the Focal Loss loss, where is 0.25, is 2.0, and y is the total number of categories of the bounding box. is the bounding box regression loss, is the GIoU loss, where is and is the minimum closed box.

[0062] Step six, update the teacher model.

[0063] The double teacher model is updated by Exponential Moving Average (EMA) to realize the source-free domain migration, and the specific calculation is as follows:

[0064] ,

[0065] where t represents the current training iteration number, t-1 represents the previous iteration, is the EMA decay coefficient. In the cross-domain scene of natural weather scene 1 (Cityscapes) - foggy scene (Foggy Cityscapes) and generated image scene (Sim10K) - natural scene 1 (Cityscapes) cross-scene migration, it is set to 0.999, and in the natural weather scene 1 (Cityscapes) - natural scene 2 (BDD100K) cross-scene migration, it is set to 0.9998. and respectively represent the updated teacher model 1 parameters of the current iteration (step t) and the teacher model 1 parameters of the previous iteration (step t-1). is the trainable parameter of the current student model.

[0066] The present example has carried out experiments in the cross-domain scene of natural weather scene 1 (Cityscapes) - foggy scene (Foggy Cityscapes), and the experimental results are shown in Table 1. The present example significantly improves the pseudo-label quality and cross-domain detection performance, and enhances the adaptive ability of the detection model under the condition of source-free domain data.

[0067] Table 1. Comparison of target detection performance (AP50%) under natural weather scene to foggy scene cross-domain experiment, as follows:

[0068] ,

[0069] Reference sources for comparison methods in Table 1:

[0070] [1] Q. Chu, S. Li, G. Chen, K. Li, and X. Li, “Adversarial alignment for source free object detection,” in Proc. AAAI, 2023, pp. 452–460.

[0071] [2] S. Li, M. Ye, X. Zhu, L. Zhou, and L. Xiong, “Source-free object detection by learning to overlook domain style,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 8014–8023.

[0072] [3] Q. Liu, L. Lin, Z. Shen, and Z. Yang, “Periodically exchange teacher student for source-free object detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2023, pp. 6414–6424.

[0073] [4] V. VS, P. Oza, and V. M. Patel, “Instance relation graph guided sourcefree domain adaptive object detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2023, pp. 3520–3530.

[0074] [5] S. Zhang, L. Zhang, G. Li, P. Li, and Z. Liu, “Multi-prototype guided source-free domain adaptive object detection for autonomous driving,”IEEE Trans. Intell. Veh., vol. 9, no. 1, pp. 1589–1601, 2024.

[0075] [6] J. Deng, W. Li, and L. Duan, “Balanced teacher for source-free object detection,” IEEE Trans. Circuit Syst. Video Technol., vol. 34, no. 8,pp.7231–7243, 2024.

[0076] [7] T. L. B. Khanh, H. Nguyen, L. H. Pham, D. N. Tran, and J. W.Jeon,“Dynamic retraining-updating mean teacher for source-free object detection,”in Proc. Eur. Conf. Comput. Vis., 2024, pp. 328–344.

[0077] The embodiment solves the problem of low pseudo-label quality caused by significant differences in confidence distribution of different categories and unbalanced category samples in the target domain through the above steps, effectively alleviates the inconsistency of pseudo-label confidence distribution and the unbalanced category samples, significantly improves the pseudo-label quality and cross-domain detection performance, and enhances the adaptive ability of the detection model under the condition of source-free domain data.

Claims

1. A source-domain-free target detection method based on dual-adaptor pseudo-label generation, characterized in that, The method includes: Initialize the dual-teacher model and student model, obtain the target domain image dataset, perform strong data augmentation 1, strong data augmentation 2 and weak data augmentation on the target domain image dataset respectively, and perform mask representation on the image datasets after strong data augmentation 1 and strong data augmentation 2 respectively. A confidence adapter is introduced to input the image after weak data augmentation into the dual-teacher model. The confidence adapter is used to dynamically adjust the lower bound of the confidence of the detection results output by the dual-teacher model, filter out candidate pseudo-labels, and calculate the current confidence threshold for each category. A class imbalance adapter is introduced to fuse the historical confidence threshold and the current confidence threshold based on the candidate pseudo-labels and the frequency of each class to obtain the current final confidence threshold for each class. Pseudo-labels are then selected based on the final confidence threshold. To train the student model, the images processed by strong data augmentation 1 and strong data augmentation 2, along with their corresponding mask images, are alternately input into the student model in each training cycle to obtain detection results. Then, the selected pseudo-labels are used as supervision signals to continuously train the student model. The parameters of the dual-teacher model are updated using the exponential moving average method to achieve passive domain transfer of the model. The method of dynamically adjusting the lower bound of the confidence level of the detection results output by the dual-teacher model using a confidence adapter includes: First, calculate the initial confidence lower bounds for each category. , ， in, This represents the increment step size in each iteration during the refinement of the lower confidence bound. It is the index of the current iteration. This represents the total number of valid iterations. It is the maximum value of the lower bound of the confidence level. and Calculate the mean and median of a given set, respectively. and These represent the mean function and the median function, respectively. This indicates the relationship between the predictions from the dual-teacher model and the category. The corresponding set of confidence scores, This indicates that it falls within the adjustment range. A subset of; Then, obtain the initial lower bound of confidence for each category. Then, a median boundary calibration mechanism is used to further optimize the lower bound of the confidence level for each category, and the final lower bound of the confidence level is dynamically determined. , ， in, For the current candidate interval The median of the built-in reliability score.

2. The source domain-free transfer learning target detection method according to claim 1, characterized in that, The image after weak data augmentation is input into the dual-teacher model, which uses an alternating activation method, activating one of the teacher models in turn for inference in each training cycle.

3. The source-domain-free transfer learning target detection method according to claim 1, characterized in that, The calculation of the current confidence threshold for each category specifically involves taking the mean of the candidate pseudo-labels as the current confidence threshold for each category. , ， in, This indicates the relationship between the predictions from the dual-teacher model and the category. The corresponding set of confidence scores, Indicates category Final confidence lower bound.

4. The source domain-free transfer learning target detection method according to claim 1, characterized in that, Based on the candidate pseudo-labels and the frequency of each category, the historical confidence threshold and the current confidence threshold are fused to obtain the current final confidence threshold for each category, specifically calculated as follows: ， in, Indicates category The number of candidate pseudo-tags, It is the total number of categories existing in the current iteration, while Indicates category frequency, Indicates the fusion weight. These are the basic weighting coefficients. For hyperparameters, The balancing factor is defined as the average frequency of all classes in the current iteration. Indicates the category in the current batch The confidence threshold Category in historical batches The confidence threshold Then it is a category The final confidence threshold.

5. The source-domain-free transfer learning target detection method according to claim 1, characterized in that, The pseudo-labels are selected based on the final confidence threshold. The pseudo-labels are generated alternately by a dual-teacher model to ensure the diversity and robustness of the pseudo-labels.

6. The source domain-free transfer learning target detection method according to any one of claims 1-5, characterized in that, The weak data augmentation process includes random horizontal flipping and center cropping; the strong data augmentation 1 process includes color dithering and grayscale conversion; the strong data augmentation 2 process includes Gaussian blurring.

7. The source-domain-free transfer learning target detection method according to any one of claims 1-5, characterized in that, The process of masking the image datasets after strong data augmentation 1 and strong data augmentation 2 specifically includes: randomly patching a mask M to occlude local areas of the image based on strong data augmentation 1 and strong data augmentation 2, wherein the patch size is set to 64×64 pixels.

8. The source domain-free transfer learning target detection method according to any one of claims 1-5, characterized in that, Both the dual-teacher model and the student model use the DETR object detection network.

Images