An unsupervised domain adaptation target detection method based on prior knowledge diffusion generation

Abstract

The application discloses an unsupervised domain adaptive target detection method based on prior knowledge diffusion generation, and belongs to the technical field of computer vision. Firstly, aiming at the modal and style difference between the source domain and the target domain, picture data is normalized to the same space through a space mapping function, and field prior knowledge text prompt words in multiple style scenes are constructed to guide a text-image generation model to realize image generation from the normalized space to the field style space. Then, based on the space normalization strategy and the prior knowledge driven generation model, the source domain and the target domain images are subjected to style transformation to generate prior scene style data. Finally, a self-training framework is constructed by combining the source domain, the target domain and the synthesized data to realize accurate prediction of unknown samples in the target domain and obtain more reliable pseudo-label samples. Through the prior knowledge guided diffusion generation and the field style conversion self-training, the application effectively improves the domain adaptive ability and the target detection robustness of the model in the cross-style and cross-modal scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an unsupervised adaptive target detection method based on prior knowledge diffusion generation. Background Technology

[0002] Object detection is a core technology in computer vision, designed to automatically identify and locate one or more target objects of a specific category from images or video frames. It generates bounding boxes for each detected object to accurately pinpoint its location, typically accompanied by corresponding category labels. This technology has been widely applied in various practical scenarios, including security monitoring, autonomous driving, medical image analysis, drone navigation, and sports event analysis.

[0003] In recent years, deep learning-based object detection methods (such as fast region convolutional neural networks) have achieved significant performance improvements on source domain data, greatly promoting the development of object detection technology. However, these models often rely on large amounts of labeled data for training, and their detection performance often drops significantly when faced with object domains with different distributions than the training data. Real-world image data exhibits great diversity, with different scenes, devices, and acquisition conditions introducing significant modal differences. In specific fields such as medical imaging and autonomous driving, the cost of acquiring labeled data is high, further limiting the generalization ability of object detection models in these areas.

[0004] To address the performance degradation of object detection models in the target domain, unsupervised domain adaptation (UDA) has become a key research focus in recent years. Its core objective is to improve model detection performance in the target domain by utilizing existing annotation information from the source domain, even in the absence of labeled target domain data, thereby reducing reliance on manual annotation. Although various UDA methods based on feature alignment, image style transfer, and adversarial learning have made some progress in object detection tasks, many challenges remain in practical applications.

[0005] Significant feature distribution shift occurs. There are often significant differences in feature distribution between the source and target domains, making it difficult for the model to learn discriminative features with good generalization ability, thus affecting the detection performance in the target domain.

[0006] Style and modality variations are complex. Images from different target domains may exhibit a variety of style differences (such as hue and lighting conditions) and modal variations (such as image sharpness and sensor type), making existing methods insufficiently stable when adapting across domains.

[0007] Lack of robustness to complex scene factors. In practical applications, factors such as changes in lighting, background complexity, and differences in target appearance often interfere with the detection results, and existing methods still struggle to guarantee accuracy and reliability when dealing with such complex scenes.

[0008] In summary, existing unsupervised adaptive target detection techniques still have significant limitations in practical applications, and there is an urgent need to propose solutions with greater robustness and generalization ability to effectively improve the detection performance of models in complex, dynamic, and data-scarce target domain scenarios. Summary of the Invention

[0009] This invention aims to provide an unsupervised domain-adaptive object detection method based on prior knowledge diffusion. It utilizes a domain-prior knowledge-driven image generation strategy and designs a style transfer self-training framework applicable to different style and modality target domains to improve the domain adaptability of the object detection model under unsupervised conditions. By fusing text-image prior knowledge, this invention guides the style transfer process, enabling the effective generation of images from different styles and modalities into images of a specific target domain. This results in more high-quality pseudo-label samples and enhances the model's prediction accuracy for unknown samples in the target domain. This method effectively improves the performance of the detection model in a specific target domain, significantly addresses the insufficient generalization ability of existing technologies when handling multi-style and multi-modal target domains, and ensures the stability and reliability of the detection results.

[0010] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0011] An unsupervised adaptive target detection method based on prior knowledge diffusion includes the following steps:

[0012] Step 1: Image generation driven by domain prior knowledge: Given a set of domain prior knowledge and the original input image, extract text prompts and image features respectively. After fusing the text prompts and image features, input them into a stable diffusion generation model for training to obtain a trained text-image generation model.

[0013] Step 2: Self-training mechanism for domain style transfer: In the first stage, the teacher network is trained on samples from the source domain and prior knowledge domain. Then, in the second stage, the student network is initialized with the parameters of the trained teacher network. The text-image generation model constructs synthetic data based on the target domain image and prior knowledge domain. The student network uses the synthetic data to improve its performance through a self-training mechanism.

[0014] Step 1 is described in detail as follows:

[0015] First, a set of prior knowledge in the domain is given, including semantic element pairs, where each pair of elements represents semantic information in the fields of weather scene and target scale; then, text prompt word feature representations are extracted from the set of prior knowledge in the domain.

[0016] Given an input image, normalize the input image and then extract image features;

[0017] Then, the text prompt word feature representation and image features are fused to obtain modal features;

[0018] Modal features are input into a stable diffusion generation model for training. First, pre-training is performed on a multimodal dataset, and then fine-tuning is performed on source and target domain data. After training, a trained text-image generation model is obtained.

[0019] The self-training mechanism for the domain style transfer is as follows:

[0020] Phase 1: Train the teacher network on samples from the source domain and a specific domain; the specific domain refers to an intermediate domain with specific style attributes defined based on domain prior knowledge. The style features of this domain are specifically determined by semantic element pairs (such as weather scene and target scale) in the domain prior knowledge, and are different from the style features of the source domain and the target domain.

[0021] The second stage: First, initialize the student network using the trained teacher network parameters;

[0022] A self-trained dataset is constructed, including pseudo-source domain images, specific domain images, and pseudo-target domain images. The specific domain refers to an intermediate domain with specific style attributes defined based on domain prior knowledge. The style features of this domain are specifically determined by semantic element pairs (such as weather scene and target scale) in the domain prior knowledge, and differ from the style features of the source and target domains. Specifically, source domain prior knowledge and specific domain prior knowledge are introduced to construct source domain text prompts and specific domain text prompts, respectively. Based on a trained text-image generation model, the target domain images are converted into pseudo-source domain images and specific domain images. Both the pseudo-source domain images and specific domain images obtain synthetic data pseudo-labels through a trained teacher network. A pseudo-target domain image is generated using a text-image generation model based on the source domain image and prior knowledge of the target domain. The label of the pseudo-target domain image is consistent with the label of the source domain image used, denoted as ;

[0023] Finally, the following joint loss function is used. Optimize student networks:

[0024]

[0025] in, Let the target detection loss function be the one used by the student network itself. These are the loss weighting coefficients; For the target domain image.

[0026] Furthermore, before constructing the self-training dataset, the target domain images are first... Perform spatial normalization mapping to a unified feature space.

[0027] Furthermore, the modal features are calculated using the following formula. :

[0028]

[0029] in, Represents modal features, For image features, Text prompt word feature representation; and All are modules consisting of two linear layers and a normalization layer. ( )and ( ) represent operations to calculate the mean and standard deviation, respectively.

[0030] This invention addresses the challenges of large differences in sample style and modality, and the difficulty in accurately predicting unknown samples, encountered in unsupervised domain adaptation for object detection tasks. By combining domain prior knowledge with a diffusion-assisted strategy based on text-image generation mechanisms, it achieves the following significant technical effects:

[0031] Enhance the model's adaptability to multi-style target domains: By constructing a spatial normalization mapping and text-image generation model guided by domain prior knowledge, it is possible to convert target domain images of different modalities and styles into specific domain styles, significantly reducing the modal differences between the source and target domains and improving the model's generalization performance.

[0032] Improve the accuracy of unknown sample prediction: By introducing a self-supervised prediction mechanism and a self-training framework for domain style transfer, it is possible to automatically mine potential high-quality pseudo-label samples in the target domain, thereby optimizing the training of the detection model and improving the detection accuracy of unknown samples in the target domain.

[0033] Improving label quality and robustness: By combining a priori-guided synthetic sample training mechanism, highly diverse and semantically consistent pseudo-source and pseudo-target domain samples can be generated, providing more representative training data for the self-training stage and enhancing the reliability and generalization ability of the labels.

[0034] Applicable to a variety of practical applications: The unsupervised domain adaptive method proposed in this invention can be widely applied to fields lacking labeled data, such as autonomous driving, security monitoring, remote sensing images, and medical images, and has good practicality and prospects for promotion. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the framework of the method of the present invention.

[0036] Figure 2 This is a flowchart illustrating the training process of the method of the present invention. Detailed Implementation

[0037] This invention proposes an unsupervised adaptive target detection method based on prior knowledge diffusion, which mainly includes two key technical modules: image generation guided by domain prior knowledge and a self-training mechanism for domain style transfer. The specific steps are as follows:

[0038] Step 1: Image Generation Based on Domain Prior Knowledge: This invention constructs an image generation framework based on domain prior knowledge to assist the detection model in completing cross-modal and cross-style data generation and style transfer. Given a set of domain prior knowledge and the original input image, text prompts and image features are extracted respectively. The text prompts and image features are then fused and input into a stable diffusion generation model for training, resulting in a trained text-image generation model; specifically as follows:

[0039] First, given a set of prior knowledge in the domain. This includes semantic element pairs, where each pair represents semantic information in areas such as weather scene and target scale, specifically including category semantic embedding, spatial structure priors, and scene context information, for example... For the input image , If its corresponding prior is This indicates that the image comes from a large target instance within a "snowy" style scene. Then, text cue word feature representations are extracted using a Contrastive Language-Image Pre-training (CLIP) CLIP text encoder. .

[0040] To unify the feature distribution of input images from different modalities, the input images are first processed... Normalization is performed to map it to a unified space. :

[0041]

[0042] in, and Original image The mean and standard deviation, and Let the mean and standard deviation be the values ​​in the target space. This is the normalized control constant. Subsequently, a ControlNet model is used to extract image features. ;

[0043] Then, the fused modal features are calculated using the following formula. :

[0044]

[0045] in and Each module consists of two linear layers and a normalization layer, used to fuse and represent visual and semantic features. ( )and ( The ) symbols represent the operations for calculating the mean and standard deviation, respectively. The fused modal features are then input into the Stable Diffusion generative model for training, with the following training objective:

[0046]

[0047] in It is input random noise. It is a noise network that is predicted to increase in the stable diffusion generation model. Indicates the time step of uniform sampling. It is the total time step. These are network parameters. Furthermore... This represents the noise in the model's predictions. Represents real noise. This is the data after noise has been added. Represents the variable The variable takes the expected value. · This is the formula for mean squared error.

[0048] The training process of this stable diffusion generation model is divided into two stages:

[0049] Phase 1: Pre-training on Wild-TI2 and ImageNet-R-TI2, two multimodal datasets containing text-image pairs;

[0050] The second stage involves fine-tuning the source and target domain data to improve the generation quality and style transfer capabilities in the target domain.

[0051] After training, a well-trained text-image generation model is obtained.

[0052] Step 2: Self-training mechanism for domain style transfer;

[0053] This invention designs a self-training process under domain style transfer to improve the generalization ability of detection models to the target domain under unsupervised conditions.

[0054] The entire training process is divided into two stages:

[0055] The first stage involves training a teacher network on samples from the source and specific domains. Typically, the source and specific domain images are trained by the teacher network and will each obtain pseudo-labels. Based on the NMS (Non-Maximum Suppression) method, all boxes are first sorted by confidence level, and then judged from high to low. If the current box overlaps with a selected box by more than a threshold, it is discarded. Finally, there is only one pseudo-label.

[0056] The second stage involves initializing the student network with the parameters of the trained teacher network. The text-image generation model constructs synthetic data based on the target domain image and domain prior knowledge. The student network then uses the synthetic data to improve its performance through a self-training mechanism.

[0057] Specifically, such as Figure 2 As shown, in the second stage, the target domain training image data is first... Perform spatial normalization mapping to a unified feature space This is to mitigate the differences between different modal data.

[0058] Subsequently, a self-trained dataset is constructed, including pseudo-source domain images, specific domain images, and pseudo-target domain images. The specific domain refers to an intermediate domain with specific style attributes defined based on domain prior knowledge. The style features of this domain are specifically determined by semantic element pairs (such as weather scene and target scale) in the domain prior knowledge, and differ from the style features of the source and target domains. Specifically, source domain text prompts and specific domain text prompts are constructed by introducing source domain prior knowledge and specific domain prior knowledge, respectively. Using the previously trained text-image generation model, the conversion from target domain data to source and specific domain data is further realized, transforming target domain images into pseudo-source domain images. With specific domain images Pseudo-source domain image With specific domain images High-quality synthetic data pseudo-labels for images were obtained through a well-trained teacher network. .

[0059] To enhance the generalization ability of student networks in the target domain, pseudo-target domain samples are also generated. The pseudo-target domain samples are a collection of images generated by a text-image generation model using source domain images and prior knowledge of the target domain. Therefore, the pseudo-target domain images have the same labels as their corresponding source domain images, denoted as... .

[0060] Finally, the following joint loss function is used. Optimize student networks:

[0061]

[0062] in, Let the target detection loss function be the one used by the student network itself. This is the loss weighting coefficient. Indicates samples based on pseudo-target domain Labels corresponding to the source domain image The calculated loss term, Indicates samples based on the target domain and synthetic data pseudo-labels The calculated loss term.

[0063] This self-training mechanism effectively enhances the adaptability of the detection model to complex target domains by integrating domain style transfer and high-quality pseudo-label guidance, and significantly alleviates the performance degradation caused by style transfer and modality differences.

[0064] Experimental Setup: This method is based on Faster R-CNN, using a Residual Network-50 (ResNet-50) pre-trained on the ImageNet dataset as the base detector. All student and teacher detectors have the same structure as the base detector. In the domain generator, an Aesthetic model is used as the text-image generation model, and the Exact Feature Distribution Matching (EFDM) style transfer method is employed.

[0065] Algorithm parameters: The optimizer uses stochastic gradient descent, with a momentum parameter of 0.9, a weight decay coefficient of 1e-4, and an initial learning rate of 3e-4. The number of clusters K is set to 2.

[0066] Hardware parameters: This experiment is built using the PyTorch framework and includes 4 GPUs and 64GB of memory.

[0067] Dataset setup: The outdoor street view dataset (Cityscapes) was used as the source domain, including 2975 training images and 500 validation images. The foggy street view dataset (FoggyCityscapes) was used as the target domain. The validation set of the foggy street view dataset was evaluated, and the standard mean precision (mAP) with an intersection-over-union (IoU) threshold of 0.5 was used as the evaluation metric for all experiments.

[0068] Table 1: Experimental Comparison Results

[0069] Source domain training only 36.2 41.1 43.6 7.0 20.5 1.5 25.7 37.6 26.7 This method 41.7 51.7 60.3 27.4 45.5 17.8 40.3 45.3 41.2

[0070] As shown in Table 1, when trained in the source domain and tested in the target domain using this method, the method achieved a 14.5% improvement compared to the results of the basic detector trained in the source domain and tested in the target domain, demonstrating the effectiveness of the method.

[0071] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. An unsupervised adaptive target detection method based on prior knowledge diffusion, characterized in that, Includes the following steps: Step 1: Image generation driven by domain prior knowledge: Given a set of domain prior knowledge and the original input image, extract text prompts and image features respectively. After fusing the text prompts and image features, input them into a stable diffusion generation model for training to obtain a trained text-image generation model. Step 2: Self-training mechanism for domain style transfer: In the first stage, the teacher network is trained on samples from the source domain and prior knowledge domain. Then, in the second stage, the student network is initialized with the parameters of the trained teacher network. The text-image generation model constructs synthetic data based on the target domain image and prior knowledge domain. The student network uses the synthetic data to improve its performance through a self-training mechanism. The self-training mechanism for the domain style transfer is as follows: Phase 1: Training the teacher network on samples from the source domain and a specific domain; the specific domain refers to an intermediate domain with specific style attributes defined based on domain prior knowledge, the style features of which are specifically determined by semantic elements in the domain prior knowledge. The second stage: First, initialize the student network using the trained teacher network parameters; A self-trained dataset is constructed, including pseudo-source domain images, specific domain images, and pseudo-target domain images. Specifically, source domain prior knowledge and specific domain prior knowledge are introduced to construct source domain text prompts and specific domain text prompts, respectively. Based on a trained text-image generation model, target domain images are converted into pseudo-source domain images and specific domain images. Both pseudo-source domain images and specific domain images are used to obtain synthetic data pseudo-labels through a trained teacher network. A pseudo-target domain image is generated using a text-image generation model based on the source domain image and prior knowledge of the target domain. The label of the pseudo-target domain image is consistent with the label of the source domain image used, denoted as ; Finally, the following joint loss function is used. Optimize student networks: in, Let the target detection loss function be the student network itself. These are the loss weighting coefficients; For the target domain image.

2. The unsupervised adaptive target detection method based on prior knowledge diffusion generation according to claim 1, characterized in that, Step 1 is described in detail as follows: First, a set of prior knowledge in the domain is given, including semantic element pairs, where each pair of elements represents semantic information of the weather scene and the target scale domain; then, text prompt word feature representations are extracted from the set of prior knowledge in the domain. Given an input image, normalize the input image and then extract image features; Then, the text prompt word feature representation and image features are fused to obtain modal features; Modal features are input into a stable diffusion generation model for training. First, pre-training is performed on a multimodal dataset, and then fine-tuning is performed on source and target domain data. After training, a trained text-image generation model is obtained.

3. The unsupervised adaptive target detection method based on prior knowledge diffusion generation according to claim 2, characterized in that, Before constructing the self-training dataset, the target domain images are first... Perform spatial normalization mapping to a unified feature space.

4. The unsupervised adaptive target detection method based on prior knowledge diffusion generation according to claim 3, characterized in that, Modal features are calculated using the following formula. : in, Representing modal features, For image features, Text prompt word feature representation; and Both are modules consisting of two linear layers and a normalization layer. ( )and ( The symbols ) represent operations for calculating the mean and standard deviation, respectively.

Images