A cross-domain target detection method based on auxiliary domain guidance

Abstract

The present invention relates to the field of computer vision technology, and provides a cross-domain target detection method based on auxiliary domain guidance. The method comprises: generating an annotated auxiliary domain image; inputting an annotated source domain image, an unannotated target domain image, and an annotated auxiliary domain image into the backbone network of the target detector to obtain source domain, target domain, and auxiliary domain image-level feature sets; inputting the source domain and target domain image-level feature sets into the image-level domain adaptation module, and inputting the source domain and auxiliary domain image-level feature sets into the instance-level domain adaptation module; calculating the target detection loss, image-level domain adaptation loss, and instance-level domain adaptation loss, and performing end-to-end training; removing the image-level domain adaptation module and the instance-level domain adaptation module to obtain a trained target detector. The present invention can improve the detection accuracy of the target detection model when the target domain annotation information is missing, and is particularly suitable for scenarios where image datasets are difficult to annotate under severe weather conditions.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-domain target detection method based on auxiliary domain guidance. Background Art

[0002] Object detection is a computer vision technology whose primary task is to identify specific objects in images or videos and determine their locations. Object detection not only identifies the objects in an image but also accurately locates each object. This technology has high practical value and broad application prospects.

[0003] Existing deep learning-based object detection methods can be divided into two categories: two-stage object detection methods and single-stage object detection methods. In two-stage object detection, a set of candidate regions are first generated, and then these regions are classified and regressed into bounding boxes. Typical examples are the Convolutional Neural Network (CNN) family of detectors (including R-CNN, Fast R-CNN, and Faster R-CNN). However, this method is slow because the generation of candidate regions and the classification are performed separately, resulting in a relatively slow performance. Single-stage methods simultaneously predict object locations and categories directly from the original image without the region proposal generation stage. Representative methods include YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector). Single-stage methods are fast and suitable for real-time detection, especially for scenarios requiring low latency, such as video stream analysis and autonomous driving.

[0004] Although many target detection methods have been proposed, the existing methods all have certain shortcomings. The detection accuracy on images under bad weather conditions is low. At the same time, they all rely on large-scale annotated datasets for supervised learning under undesirable environments such as bad weather conditions.

[0005] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a cross-domain target detection method based on auxiliary domain guidance.

[0007] The present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a cross-domain target detection method based on auxiliary domain guidance, comprising:

[0009] Using annotated source domain images I S and the unlabeled target domain image I T Training a style transfer model;

[0010] There is a labeled source domain image I S Input into the trained style transfer model to obtain the annotated auxiliary domain image I A ; There will be a labeled source domain image I S , unlabeled target domain image I T and annotated auxiliary domain image I A Input into the backbone network of the target detector to calculate the source domain image level feature set Target domain image-level feature set and auxiliary domain image-level feature set Where i=1, 2, 3 represents image-level features at different scales;

[0011] The source domain image-level feature set and the target domain image-level feature set Perform image-level domain adaptation in the input image-level domain adaptation module;

[0012] The source domain image-level feature set and auxiliary domain image-level feature set Input into the instance-level domain adaptation module for instance-level domain adaptation;

[0013] For the labeled source domain image I input to the target detector S and annotated auxiliary domain image I A Calculate the object detection loss and combine the image-level domain adaptation loss and instance-level domain adaptation loss for end-to-end training;

[0014] After completing the domain adaptation training, the image-level domain adaptation module and the instance-level domain adaptation module are removed to obtain a trained object detector adapted to the target domain.

[0015] Preferably, the labeled source domain image I S and the unlabeled target domain image I T Training the style transfer model, specifically including:

[0016] There is a labeled source domain image I S and the unlabeled target domain image I T Input to the style transfer model, the style transfer model uses the generator G to transform the labeled source domain image I S Converted into an annotated auxiliary domain image I A ;

[0017] There will be an annotated auxiliary domain image I A Input into the target detection framework pre-trained in the source domain to obtain the target detection result;

[0018] A first loss function is calculated according to the target detection result, and the style transfer model is trained using the first loss function until the style transfer model converges.

[0019] Preferably, the first loss function is L FCUT =L CUT +λ D L DET (I A ,B S ,C S ); where L CUT is the loss function of the style transfer model, L DET For the annotated auxiliary domain image I A The target detection loss calculated above, λ D is the weight coefficient, B S is a labeled source domain image I S The location information of the labeled object on C S is a labeled source domain image I S Category information of the labeled object.

[0020] Preferably, the source domain image level feature set and auxiliary domain image-level feature set The input is sent to the instance-level domain adaptation module for instance-level domain adaptation, which includes:

[0021] From the source domain image-level features Extract source domain instance-level features

[0022] Image-level features from auxiliary domain Extract auxiliary domain instance level features

[0023] Among them, ROIAlign() represents the region of interest alignment operation, B S Represents the labeled source domain image I S The location information of the annotation object on the . is the source domain image-level feature set Source domain image-level features in , is the auxiliary domain image-level feature set Auxiliary domain image-level features in

[15] .

[0024] Preferably, the labeled source domain image I input to the target detector S and annotated auxiliary domain image IA Calculate the target detection loss and combine the image-level domain adaptation loss and instance-level domain adaptation loss for end-to-end training, including:

[0025] Calculate the second loss function and use the second loss function for end-to-end training; wherein the second loss function is L TOTAL =L DET +λ1L DC +λ2L INST ; Among them, λ1 is the weight coefficient corresponding to the image-level domain adaptation loss, λ2 is the weight coefficient corresponding to the multi-scale instance-level domain adaptation loss, L DET is the target detection loss, L DC is the image-level domain adaptation loss, L INST is a multi-scale instance-level domain adaptation loss.

[0026] Preferably, image-level domain adaptation loss Among them, t i is the domain label of the i-th image, the label of the source domain is t i =0, the label of the target domain t i =1, is the domain classification probability of the i-th image at the (x, y) position of the feature map.

[0027] Preferably, multi-scale instance-level domain adaptation loss Among them, l i is the instance contrast loss at a single scale.

[0028] Preferably, instance contrast loss at a single scale

[0029] Where τ is the temperature parameter, τ>0, Represents the source domain image-level feature set from the i-th scale The j-th instance feature obtained, P(j) is The positive sample index set in the auxiliary domain, N J is the number of objects in the image, for The positive sample instance feature in the auxiliary domain, Neg(j) is The set of negative sample indexes in the auxiliary domain, for Negative instance features in the auxiliary domain.

[0030] Preferably, the style transfer model is a CUT model.

[0031] Preferably, the target detector is a YOLO single-stage target detector.

[0032] The present invention can improve the detection accuracy of the target detection model under the premise of lacking target domain annotation information, and is particularly suitable for scenes in the real world where image datasets are difficult to annotate under severe weather conditions. The present invention uses a single-stage target detector to achieve cross-domain target detection, while having fewer model parameters and computational complexity. It has more advantages in terms of lightweight, accuracy, speed and training efficiency, and is more suitable for most application scenarios in reality (such as autonomous driving); and the auxiliary domain image synthesized by the present invention has the target domain style while maintaining the foreground object information of the source domain image, which is more suitable for cross-domain target detection. The reason is that the real label is used in the instance-level module, which ensures that the instance features obtained from the source domain and the auxiliary domain are noise-free; the present invention uses contrast loss to reduce the difference between domains while emphasizing the feature differences of different types of targets, accurately aligning the instance features in the source domain and target domain images, which helps to improve the detection accuracy of the target detector in the target domain image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0034] Figure 1 This is a flow chart of a cross-domain target detection method based on auxiliary domain guidance provided by an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of a source domain image in a cross-domain object detection method based on auxiliary domain guidance provided by an embodiment of the present invention;

[0036] Figure 3 is a schematic diagram of a target domain image in a cross-domain target detection method based on auxiliary domain guidance provided by an embodiment of the present invention;

[0037] Figure 4 is a schematic diagram of an auxiliary domain image in a cross-domain target detection method based on auxiliary domain guidance provided by an embodiment of the present invention;

[0038] Figure 5 is a schematic diagram of a cross-domain target detection method based on auxiliary domain guidance provided by an embodiment of the present invention;

[0039] Figure 6 is a schematic diagram of an image-level domain adaptation module in a cross-domain object detection method based on auxiliary domain guidance provided by an embodiment of the present invention;

[0040] Figure 7is a schematic diagram of an instance-level domain adaptation module in a cross-domain object detection method based on auxiliary domain guidance provided by an embodiment of the present invention;

[0041] Figure 8 1 is a flow chart of another cross-domain target detection method based on auxiliary domain guidance provided by an embodiment of the present invention;

[0042] Figure 9 1 is a schematic diagram comparing the application effects of a cross-domain target detection method based on auxiliary domain guidance provided by an embodiment of the present invention and various methods in the prior art;

[0043] Figure 10 This is a schematic diagram comparing the application effects of a cross-domain target detection method based on auxiliary domain guidance provided by an embodiment of the present invention and various methods in the prior art. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as meaning open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" and the like are intended to indicate that the specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0046] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "multiple" means two or more. In addition, for example, the description may also use the method of adding "A" and "B" at the end to describe the same type of nouns as two independent individuals. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the description purposes of the same type of individuals, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0047] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0048] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0049] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0050] Embodiment 1:

[0051] The cross-domain object detection method uses a labeled dataset under normal weather conditions (source domain dataset) and an unlabeled dataset under severe weather conditions (target domain dataset) to train a cross-domain object detector, thereby improving the detection accuracy on images under severe weather conditions.

[0052] Existing cross-domain object detection methods can be divided into three categories. The first category is domain transfer-based methods. These methods use a style transfer model and a source domain dataset to synthesize a simulated dataset with target domain style and annotations. Object detectors are then trained on this simulated dataset, thereby improving the performance of the object detector on the target domain dataset. However, this method relies heavily on the accuracy of the synthesized dataset and suffers from the problem of missing foreground annotated objects in the synthesized dataset. Another category is cross-domain object detection methods based on adversarial feature learning. These methods leverage the concept of Generative Adversarial Networks (GANs) to extract domain-invariant features. However, during instance-level domain adaptation, feature differences unrelated to domain differences may be aligned, such as alignment of objects with different appearance differences, which is detrimental to the detection task. Finally, domain adaptation methods based on average teacher learning use pseudo-labels generated by a teacher model to supervise the training of a student model in the target domain. However, pseudo-labels introduce noise, and training the average teacher model requires two models, which increases complexity and training cost. Moreover, most of the cross-domain target detection methods in the prior art are based on two-stage target detectors, which have the disadvantage of slow inference time. In addition, the existing cross-domain target detection methods inevitably reduce the differences in other features while reducing the feature differences between domains, which is not conducive to target detection. In order to improve the detection accuracy of target detection, embodiment 1 of the present invention provides a cross-domain target detection method based on auxiliary domain guidance, such as Figure 1 As shown, including:

[0053] In step 201, a labeled source domain image I is used S and the unlabeled target domain image I T Train the style transfer model; wherein the source domain image can be understood as an image with a detection target under normal weather conditions, such as Figure 2 As shown in , the target domain image can be understood as an image under bad weather (it does not need to have a detection target, but is used to represent environmental factors under bad weather), such as Figure 3 shown.

[0054] In step 202, the labeled source domain image I S Input into the trained style transfer model to obtain the annotated auxiliary domain image I A ; There will be a labeled source domain image I S , unlabeled target domain image I T and annotated auxiliary domain image I A Input into the backbone network of the target detector to calculate the source domain image level feature set Target domain image-level feature set and auxiliary domain image-level feature set Wherein, i=1, 2, 3 represent image-level features at different scales; in an optional embodiment, the style transfer model is a CUT (contrastive-learning-based unpaired image-to-image translation) model, and the target detector is a YOLO single-stage target detector. The auxiliary domain image can be understood as an image obtained by adding a bad weather filter of the target domain image to the source domain image, that is, it is equivalent to extracting the effect of bad weather on the target domain image as a filter, and using the image formed by the source domain image under the filter as the auxiliary domain image, such as Figure 4 for the reason Figure 2 The source domain images shown are attached Figure 3 The auxiliary domain image obtained by the bad weather of the target domain image shown. S , target domain image I T and auxiliary domain image I A In the auxiliary domain image I A and source domain image I S They have a paired relationship and share consistent annotation information. In the backbone network of the target detector, the corresponding image feature set is obtained. and is the source domain image-level feature set Source domain image-level features in , is the target domain image-level feature set The target domain image-level features in is the auxiliary domain image-level feature set The auxiliary domain image level features in, where i = 1, 2, 3 represent image level features at different scales; the backbone network can be a YOLO backbone network, for the output of the YOLO backbone network, the source domain image level features and They are 8-fold down-sampled image features, 16-fold down-sampled image features, and 24-fold down-sampled image features relative to the input image, respectively. That is, i=1 corresponds to an 8-fold down-sampling scale, i=2 corresponds to a 16-fold down-sampling scale, and i=3 corresponds to a 24-fold down-sampling scale. The image-level features of different scales in the target domain and the auxiliary domain also have a scale relationship corresponding to the image-level features in the source domain.

[0055] In step 203, the source domain image level feature set and the target domain image-level feature set Image-level domain adaptation is performed in the input image-level domain adaptation module.

[0056] In step 204, the source domain image level feature set and auxiliary domain image-level feature set Input into the instance-level domain adaptation module for instance-level domain adaptation, that is, firstly from the source domain image level features and auxiliary domain image-level features Extract source domain instance-level features and auxiliary domain instance-level characteristics Then, the contrast loss at a single scale is calculated, and finally the contrast losses at multiple scales of i = 1, 2, 3 are accumulated to perform multi-scale instance-level domain adaptation.

[0057] In step 205, the labeled source domain image I input to the target detector is S and annotated auxiliary domain image I A Calculate the target detection loss and perform end-to-end training by combining image-level domain adaptation loss and instance-level domain adaptation loss. The image-level domain adaptation loss is the loss corresponding to step 203, and the instance-level domain adaptation loss is the loss corresponding to step 204. This loss is based on the source domain instance-level features. and auxiliary domain instance-level characteristics Calculated.

[0058] In step 206 , after the domain adaptation training is completed, the image-level domain adaptation module and the instance-level domain adaptation module are removed to obtain a trained object detector adapted to the target domain.

[0059] It is understandable that during the training phase, the target detector includes the YOLO backbone network, the image level domain adaptation module and the instance level domain adaptation module, such as Figure 5 As shown in , after the training to the target detector converges, the image-level domain adaptation module and the instance-level domain adaptation module are removed from the target detector to obtain the trained target detector. The network structure of the image-level domain adaptation module is shown in Figure 6 As shown, specifically: the source domain image level feature set and the target domain image-level feature set After the gradient reversal layer is input, an image-level domain discriminator is connected, which outputs the domain discrimination result on the feature map. The image-level domain adaptation module is used to use the domain classifier to extract domain-invariant features of the source and target domains.

[0060] This embodiment first trains a style transfer model using a set of labeled source domain images and a set of unlabeled target domain images; the labeled source domain image set is input into the trained style transfer model to obtain auxiliary domain images with the target domain style that can share annotations with the source domain image set; next, the labeled source domain image set, the unlabeled target domain image set, and the auxiliary domain image set that shares annotations with the source domain image set are simultaneously input into a cross-domain target detection model, and image-level domain adaptation is performed on the source domain features and target domain features in the feature space, and instance-level domain adaptation is performed on the source domain features and auxiliary domain features using contrastive learning. The present invention can improve the detection accuracy of the target detection model under the premise of lacking target domain annotation information, and is particularly suitable for scenarios in the real world where image datasets are difficult to label under severe weather conditions. This embodiment uses a single-stage target detector to achieve cross-domain target detection, and has fewer model parameters and computational complexity. It has more advantages in terms of lightweight, accuracy, speed and training efficiency, and is more suitable for most application scenarios in reality (such as autonomous driving); moreover, the auxiliary domain image synthesized in this embodiment has the style of the target domain while maintaining the foreground object information of the source domain image, which is more suitable for cross-domain target detection. The reason is that the real label is used in the instance-level module, which ensures that the instance features obtained from the source domain and the auxiliary domain are noise-free; this embodiment uses contrast loss to reduce the difference between domains while emphasizing the feature differences of different types of targets, accurately aligning the instance features in the source domain and target domain images, which helps to improve the detection accuracy of the target detector in the target domain image.

[0061] Wherein, the use of labeled source domain image I S and the unlabeled target domain image I T Training the style transfer model, specifically including:

[0062] There is a labeled source domain image I S and the unlabeled target domain image I T Input to the style transfer model, the style transfer model uses the generator G to transform the labeled source domain image I S Converted into an annotated auxiliary domain image I A ; That is, use the generator G in the style transfer model to transform the source domain image into the auxiliary domain image, that is, I A =G(I S ). There will be annotated auxiliary domain image I A Input into the target detection framework pre-trained in the source domain to obtain the target detection result; calculate the first loss function according to the target detection result, and use the first loss function to train the style transfer model until the style transfer model converges. The calculation of the first loss function specifically includes: using the source domain image I SThe target detection loss is calculated based on the target annotation information; the detection loss is weighted with the style transfer loss to obtain the first loss function of the style transfer model for generating the auxiliary domain image. The first loss function is L FCUT =L CUT +λ D L DET (I A ,B S ,C S ); where L CUT is the loss function of the style transfer model CUT, L DET For the auxiliary domain image I A The target detection loss calculated above, λ D is the weight coefficient, which is obtained by those skilled in the art based on demand analysis, B S is a labeled source domain image I S The location information of the labeled object on C S is a labeled source domain image I S The category information of the labeled objects on the auxiliary domain image is shared with the source domain image.

[0063] Among them, L DET The calculation is based on existing technology, L CUT The calculation method is: L CUT =L GAN (G,D dis ,I S ,I T )+λ1L PatchNCE (G,H,I S )+λ2L PatchNCE (G,H,I T ); in the face of loss L GAN middle, I S and I T Represent the source domain image and the target domain image respectively, G is the generator that synthesizes the source domain data set into a target domain style data set, and the discriminator D dis It is used to judge whether the image belongs to the target domain image, and the contrast loss L PatchNCE In , H constructs contrast loss positive and negative feature samples from the input and output images of G. S Input, loss L PatchNCE (G,H,I S ) can make the image synthesized by the generator G have the style of the target domain while retaining the original image D s The structural semantic information of the target domain image I T Input, loss L PatchNCE (G,H,I T ) The input and output of the constraint generator G should be consistent.

[0064] In an optional embodiment, the network structure of the instance-level domain adaptation module is as follows: Figure 7 As shown, the source domain image level feature set and auxiliary domain image-level feature set The input is sent to the instance-level domain adaptation module for instance-level domain adaptation, which includes:

[0065] From the source domain image-level features Extract source domain instance-level features Image-level features from auxiliary domain Extract auxiliary domain instance level features Among them, ROIAlign() represents the region of interest alignment operation, B S Represents the labeled source domain image I S The position information of the labeled object on the image includes the target location and the size of the bounding box, that is, the position information. is the source domain image-level feature set Source domain image-level features in , is the auxiliary domain image-level feature set Auxiliary domain image-level features in

[15] .

[0066] In practical application scenarios, the labeled source domain image I S and annotated auxiliary domain image I A Calculate the target detection loss and combine the image-level domain adaptation loss and instance-level domain adaptation loss for end-to-end training, including:

[0067] Calculate the second loss function and use the second loss function for end-to-end training; wherein the second loss function is combined with the target detection loss L DET , image-level domain adaptation loss L DC , multi-scale instance-level domain adaptation loss L INST Calculated, that is, the second loss function is L TOTAL =L DET +λ1L DC +λ2L INST ; Wherein, λ1 is the weight coefficient corresponding to the image-level domain adaptation loss, λ2 is the weight coefficient corresponding to the multi-scale instance-level domain adaptation loss, λ1 and λ2 are obtained by those skilled in the art based on empirical analysis, L DET is the target detection loss, L DC is the image-level domain adaptation loss, L INST is a multi-scale instance-level domain adaptation loss.

[0068] Image-level domain adaptation loss Among them, ti is the domain label of the i-th image, the label of the source domain is t i =0, the label of the target domain t i =1, is the domain classification probability of the i-th image at the (x, y) position of the feature map.

[0069] Multi-scale instance-level domain adaptation loss Among them, l i is the instance contrast loss at a single scale. The contrast loss is used to constrain the extracted instance features during training to extract domain-invariant features, where instance features at the same position are considered positive samples and instance features at different positions are considered negative samples.

[0070] Wherein, τ is a temperature parameter, τ>0, τ is obtained by those skilled in the art based on empirical analysis, Represents the source domain image level features from the i-th scale The j-th instance feature obtained, P(j) is In the auxiliary domain, the positive sample index set is Auxiliary domain instance-level features that share the same location annotation, N J is the number of objects in the image, for The positive sample instance feature in the auxiliary domain, Neg(j) is The set of negative sample indexes in the auxiliary domain, for Negative instance features in the auxiliary domain.

[0071] Example 2:

[0072] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0073] The present invention provides a cross-domain target detection method based on auxiliary domain guidance, such as Figure 8 As shown, the following steps are included:

[0074] In step 301, we first use the labeled source domain image I S and the unlabeled target domain image I T Train a style transfer model; the source domain image I with labeled information S Input into the trained style transfer model to obtain the auxiliary domain image I with labeled information A .

[0075] In step 302, the labeled source domain image IS , unlabeled target domain image I T , with annotated auxiliary domain image I A Input the backbone network of the target detector and calculate the corresponding image feature set Where i=1, 2, 3 represents image-level features at different scales, is the source domain image-level feature set Source domain image-level features in , is the target domain image-level feature set The target domain image-level features in is the auxiliary domain image-level feature set Auxiliary domain image-level features in

[15] .

[0076] In step 303, the image feature set and Image-level domain adaptation is performed in the input image-level domain adaptation module.

[0077] In step 304, the image level feature set as well as Input to the instance-level domain adaptation module, respectively from the single-scale image features and Extract instance-level features and Perform contrast loss constraints, and then accumulate contrast losses at multiple scales to perform instance-level domain adaptation at multiple scales.

[0078] In step 305, the source domain image I input to the target detector is S and auxiliary domain image I A Calculate the object detection loss and combine the image-level domain adaptation loss and instance-level domain adaptation loss for end-to-end training.

[0079] In step 306 , after the domain adaptation training is completed, the domain adaptation related modules are removed to obtain an object detector adapted to the target domain.

[0080] The step 301 specifically includes: training the style transfer model requires inputting the source domain image I S and target domain image I T , using the generator G in the style transfer model to transform the source domain image into the auxiliary domain image, i.e. I A =G(I S ); Next, the auxiliary domain image I A It is fed into the target detection framework pre-trained in the source domain, using the source domain image I SThe target detection loss is calculated based on the target annotation information of the target; the detection loss is weighted with the style transfer loss to obtain the loss function of the style transfer model for generating auxiliary domain images, namely L FCUT =L CUT +λ D L DET (I A ,B S ,C S ), where L CUT is the loss function of the style transfer model CUT, L DET For the auxiliary domain image I A The target detection loss calculated above, λ D is the weight coefficient. A represents the auxiliary domain image, which shares annotation information with the source domain image. S Indicates the location information of the labeled object on the corresponding image, C S Indicates the category information of the corresponding labeled object.

[0081] In step 302: the input source domain image I S , target domain image I T and auxiliary domain image I A In the auxiliary domain image I A and source domain image I S They have a paired relationship and share consistent annotation information. In the backbone network of the target detector, the corresponding image feature set is obtained. Where i = 1, 2, 3 represents image-level features at different scales; for the output of the YOLO backbone network, the source domain image features and Relative to the input image, they are 8 times downsampled image features, 16 times downsampled image features, and 24 times downsampled image features, respectively. The image-level features of different scales in the target domain and the auxiliary domain also have a scale relationship consistent with the scale of the source domain.

[0082] The step 304 specifically includes: in the instance-level domain adaptation module, by using the real annotation information of the source domain and the region of interest alignment operation, respectively, from the image-level features and Extract instance-level features and Right now Where ROIAlign() represents the region of interest alignment operation, B S Represents the true annotation information of the source domain, including the location of the target and the size of the bounding box. is the source domain image-level feature set Source domain image-level features in , is the auxiliary domain image-level feature set Auxiliary domain image-level features in

[15] .

[0083] During the training process, the instance features extracted are constrained using contrast loss to extract domain-invariant features, where instance features at the same position are considered positive samples and instance features at different positions are considered negative samples. i as follows: in represents the jth instance feature obtained from image-level FS i. P(j) is the instance feature The positive sample index set, where the positive sample is the instance feature Auxiliary domain instance features that share the same position annotation. N J is the number of objects in the image, for A positive sample instance feature, Neg(j) is the instance feature The negative sample index set, τ is a hyperparameter greater than 0. The instance contrast loss L corresponding to the joint multi-scale image features INST The formula is as follows:

[0084] In step 305, combined with the target detection loss L DET , image-level domain adaptation loss L DC , multi-scale instance-level domain adaptation loss L INST The total loss function is as follows: TOTAL =L DET +λ1L DC +λ2L INST , where λ1 and λ2 represent hyperparameters. Specifically, λ1 is the weight coefficient corresponding to the image-level domain adaptation loss, and λ2 is the weight coefficient corresponding to the multi-scale instance-level domain adaptation loss. Image-level domain adaptation loss where t i is the domain label of the i-th image, where t i =0, the label of the target domain t i =1, is the domain classification probability of the i-th image at the (x, y) position of the feature map.

[0085] After the above six steps in sequence, the final cross-domain target detector has high detection accuracy on the severe weather dataset (target domain dataset).

[0086] In a specific application scenario, steps 301 to 306 specifically include:

[0087] like Figure 5 As shown, in Figure 5 In the first auxiliary domain generation stage, we first use the annotated source domain image I S and the unlabeled target domain image I T Train a style transfer model. S and target domain image I T Input style transfer model CUT, use the generator G in the style transfer model to transform the source domain image into the auxiliary domain image, that is, I A =G(I S ); Next, the auxiliary domain image I A It is fed into the target detection framework pre-trained in the source domain, using the source domain image I S The target detection loss is calculated based on the target annotation information of the target domain; the detection loss is weighted with the style transfer loss to obtain the loss function (i.e., the first loss function) of the style transfer model used to generate the auxiliary domain image as follows:

[0088] L FCUT =L CUT +λ D L DET (I A ,B S ,C S ) (1)

[0089] Among them L FCUT is the overall loss of training the style transfer model, L CUT is the loss function of the style transfer model CUT, L DET For the auxiliary domain image I A The target detection loss calculated above, the auxiliary domain image I A and source domain image I S Shared annotation information, B S Indicates the location information of the labeled object on the corresponding image, C S Indicates the category information of the corresponding labeled object, weight coefficient λ D Set to 10.

[0090] From formula (1), we can see that the detection loss L DET The style transfer model can be constrained so that the auxiliary domain image style is close to the target image domain while retaining the detection features that are beneficial to the high-level object detection task. Finally, the auxiliary domain image is obtained by using the style transfer model trained with formula (1), that is, I A =G(I S ), where G is the generator in the trained style transfer model. The auxiliary domain image has the following properties: 1. The style of the auxiliary domain image is close to that of the target domain image; 2. The content of the auxiliary domain image is consistent with the source domain, well preserving the target features while sharing annotation information with the source domain image.

[0091] There is a labeled source domain image I S , unlabeled target domain image I T , with annotated auxiliary domain image I A Input the backbone network of the target detector to obtain a multi-scale image feature set in different domains Where i=1, 2, 3 represents image-level features at different scales.

[0092] Its network structure is as follows Figure 5 As shown in the cross-domain target detection training stage, the input source domain image I S , target domain image I T and auxiliary domain image I A In the auxiliary domain image I A and source domain image I S They have a paired relationship and share consistent annotation information. In the backbone network of the target detector, the corresponding image feature set is obtained. Where i=1, 2, 3 represent image-level features at different scales. The object detection framework here takes the single-stage object detector YOLO as an example. Note that the present invention is not limited to a specific object detector and can also be used on other object detectors. The backbone network of the YOLO object detector is the feature extraction module, and the source domain image features are obtained. and Relative to the input image, they are 8 times downsampled image features, 16 times downsampled image features, and 24 times downsampled image features.

[0093] The image feature set and image-level feature sets Image-level domain adaptation is performed in the input image-level domain adaptation module.

[0094] The network structure of the image-level domain adaptation module is as follows Figure 6 As shown, specifically, the image feature set and image-level feature sets After the gradient reversal layer is input, an image-level domain discriminator is connected, which outputs the domain discrimination result on the feature map. The image-level domain adaptation module is used to extract domain-invariant features of the source and target domains using the domain classifier. During the training process, the constraints of image-level domain adaptation are expressed as formula (2):

[0095]

[0096] where t i is the domain label of the i-th image, where t i=0, the label of the target domain t i =1, is the domain classification probability of the i-th image at the (x, y) position of the feature map.

[0097] The image-level feature set as well as Input to the instance-level domain adaptation module, at a single scale from and Extract instance-level features and Perform contrast loss constraint, and then accumulate contrast loss at each scale to perform instance-level domain adaptation at multiple scales; its network structure is as follows Figure 7 In the instance-level domain adaptation module, by using the true annotation information of the source domain and the alignment operation of the region of interest, we can and Extract instance-level features and Right now Where ROIAlign() represents the region of interest alignment operation, B S Represents the true annotation information of the source domain (i.e., the labeled source domain image I S The location information of the labeled object on the image is included, including the location of the target and the size of the bounding box. During the training process, the extracted instance features are constrained using contrast loss to extract domain-invariant features, where instance features at the same location are considered positive samples and instance features at different locations are considered negative samples. The instance contrast loss at a single scale is l i as follows:

[0098]

[0099] The hyperparameter τ is set to 0.07, where Represents image level The jth instance feature is obtained. P(j) is the instance feature The positive sample index set, where the positive sample is the instance feature Auxiliary domain instance features that share the same position annotation. N J is the number of objects in the image, for A positive sample instance feature, Neg(j) is the instance feature The negative sample index set. The instance contrast loss L corresponding to the image features at multiple scales INST The formula is as follows:

[0100]

[0101] Formula (4) can make the instance alignment process pay more attention to and reduce the inter-domain differences, while the feature differences that are irrelevant to the domain differences are preserved.

[0102] The source domain image I input to the target detection framework S and auxiliary domain image I A Perform target detection loss constraints and combine them with image-level domain adaptation loss L DC , multi-scale instance-level domain adaptation loss L INST For end-to-end training, the overall loss (i.e., the second loss function) is as follows:

[0103] L TOTAL =L DET +λ1L DC +λ2L INST (5)

[0104] In actual use, the weight coefficients λ1 and λ2 are set to 0.5 and 0.01, where L DET is the detection loss of the target detector in the source domain and auxiliary domain images, L DC is the image-level domain adaptation loss, L INST It is a multi-scale instance-level domain adaptation loss. After the end-to-end training is completed, remove Figure 6 The image-level domain adaptation and Figure 7 The relevant modules of instance-level domain adaptation are shown in Figure 1, thereby obtaining the final cross-domain object detector. As can be seen, the cross-domain object detector still has the same detection speed as the original YOLO object detector and does not increase the computational complexity during inference.

[0105] The training process uses images of 640x640 pixels in size, and Stochastic Gradient Descent (SGD) is used for gradient updates. The learning rate, weight decay, and momentum in the SGD algorithm are set to 0.01, 0.005, and 0.937, respectively. The batch size is 12, and each batch inputs 4 source domain images, 4 auxiliary domain images, and 4 target domain images. The number of iterations (epochs) during training is set to 300, and early stopping is used to prevent overfitting. After completing domain adaptation training, the domain adaptation-related modules are removed to obtain an object detector adapted to the target domain.

[0106] The cross-domain target detection method based on auxiliary domain guidance described in this embodiment is compared with other existing detection methods. The experiment is carried out in the pytorch framework using python, and the target detector trained by this embodiment is compared with the existing DAF, IDF, CMT+AT, CMT+PT, YOLOv5s(S), S-DAYOLO, MS-DAYOLO, and IMS-DAYOLO methods. Among them, YOLOv5s(S) refers to training the YOLOv5s version detector in the source domain and testing it directly in the target domain. Therefore, it can be considered as the lower limit of the performance of the cross-domain target detector; other cross-domain target detectors are trained with labeled source domain datasets and unlabeled target domain datasets. Among them, S-DAYOLO, MS-DAYOLO, IMS-DAYOLO and the methods described in the embodiment all use YOLOv5s as the target detector for training.

[0107] The performance of the method described in this embodiment is evaluated using the dataset BDD100K, which contains real night scenes, and the dataset RTTS, which contains real foggy scenes, as examples. Specifically, when evaluating the domain adaptation detector for daytime to nighttime scenes, the sub-datasets BDD_day_s, BDD_night_t, and BDD_night_test in BDD100K are used. BDD_day_s represents the labeled source domain dataset used during training, consisting of 6,647 images; BDD_night_t is the unlabeled target domain dataset used during training, consisting of 15,090 images; and BDD_night_test is the target domain dataset used during testing, consisting of 2,133 images. The datasets used for training the cross-domain object detector for scenes from normal weather to foggy weather include VOC_clear and URHI. VOC_clear is a labeled source domain dataset, consisting of 8,111 images; URHI is an unlabeled target domain dataset, consisting of 4,807 images. When testing foggy scenes, RTTS, which contains 4,322 images, is used as the target domain dataset.

[0108] The quantitative comparison results on the night dataset BDD_night_test and the foggy dataset RTTS are shown in the following table:

[0109]

[0110]

[0111] The above table shows the mAp50 (%) quantitative comparison results of different cross-domain target detection algorithms on the night dataset BDD_night_test (abbreviated as BDD_N_T in the table) and the foggy dataset RTTS, where the bold numbers represent the best results and the underlined numbers represent the suboptimal results. The comparison indicators are: mean Average Precision (mAP) with an intersection over Union (IoU) threshold of 0.5, parameters, and runtime. Parameters and Runtime are the results measured on images with a resolution of 640×640 after removing the domain adaptation-related modules. All experiments were run on an NVIDIA GTX 3090Ti GPU. From the results in the above table, it can be seen that the method described in this embodiment is better than the existing cross-domain target detection method. Compared with the suboptimal CMT+AT algorithm, the method of this embodiment requires fewer parameters, has a faster runtime, and is more suitable for on-site scenarios.

[0112] This embodiment also compares the detection results based on the RTTS data set, such as Figure 9 The upper left corner shows the manual annotation results of a foggy image in the RTTS dataset. The results of target detection on this foggy image using the existing technologies CMT+AT, YOLOv5s(S), MS-DAYOLO, IMS-DAYOLO and the method described in this embodiment are shown as follows: Figure 9 As shown. Figure 10 The upper left corner shows the manual annotation results of a dark light image in the RTTS dataset. The results of target detection on this dark light image using the existing technologies CMT+AT, YOLOv5s(S), MS-DAYOLO, IMS-DAYOLO and the method described in this embodiment are shown as follows: Figure 10 As shown in the figure, the detection results of foggy images and dark-light images show that, compared with other methods, the detection results of the method described in this embodiment are closer to the results of manual annotation and have the best cross-domain object detection effect.

[0113] In summary, the method described in this embodiment has the following three advantages. First, based on a single-stage target detector, it has fewer model parameters and computational complexity, and has more advantages in terms of lightweight, accuracy, speed, and training efficiency, and is more suitable for most application scenarios in reality (such as autonomous driving); second, the auxiliary domain image synthesized in this embodiment has the style of the target domain while maintaining the foreground object information of the source domain image, and is more suitable for cross-domain target detection. The reason is that the real label is used in the instance-level module, which ensures that the instance features obtained from the source domain and the auxiliary domain are noise-free; third, this embodiment uses contrast loss to reduce the difference between domains while emphasizing the feature differences of different types of targets, accurately aligning the instance features in the source domain and target domain images, which helps to improve the detection accuracy of the target detector in the target domain image.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-domain target detection method based on auxiliary domain guidance, characterized in that: include: In step 201, a labeled source domain image I is used S and the unlabeled target domain image I T Training a style transfer model; In step 202, the labeled source domain image I S Input into the trained style transfer model to obtain the annotated auxiliary domain image I A ; There will be a labeled source domain image I S , unlabeled target domain image I T and annotated auxiliary domain image I A Input into the backbone network of the target detector to calculate the source domain image level feature set Target domain image-level feature set and auxiliary domain image-level feature set Where i=1, 2, 3 represents image-level features at different scales; In step 203, the source domain image level feature set and the target domain image-level feature set Perform image-level domain adaptation in the input image-level domain adaptation module; In step 204, the source domain image level feature set and auxiliary domain image-level feature set Input into the instance-level domain adaptation module for instance-level domain adaptation; In step 205, the labeled source domain image I input to the target detector is S and annotated auxiliary domain image I A Calculate the object detection loss and combine the image-level domain adaptation loss and instance-level domain adaptation loss for end-to-end training; In step 206 , after the domain adaptation training is completed, the image-level domain adaptation module and the instance-level domain adaptation module are removed to obtain a trained object detector adapted to the target domain.

2. The cross-domain target detection method based on auxiliary domain guidance according to claim 1 is characterized in that The use of labeled source domain image I S and the unlabeled target domain image I T Training the style transfer model, specifically including: There is a labeled source domain image I S and the unlabeled target domain image I T Input to the style transfer model, the style transfer model uses the generator G to transform the labeled source domain image I S Converted into an annotated auxiliary domain image I A ; There will be an annotated auxiliary domain image I A Input into the target detection framework pre-trained in the source domain to obtain the target detection result; A first loss function is calculated according to the target detection result, and the style transfer model is trained using the first loss function until the style transfer model converges.

3. The cross-domain target detection method based on auxiliary domain guidance according to claim 2 is characterized in that The first loss function is L FCUT =L CUT +λ D L DET (I A ,B S ,C S ); where L CUT is the loss function of the style transfer model, L DET For the annotated auxiliary domain image I A The target detection loss calculated above, λ D is the weight coefficient, B S is a labeled source domain image I S The location information of the labeled object on C S is a labeled source domain image I S Category information of the labeled object.

4. The cross-domain target detection method based on auxiliary domain guidance according to claim 1 is characterized in that The source domain image level feature set and auxiliary domain image-level feature set The input is sent to the instance-level domain adaptation module for instance-level domain adaptation, which includes: From the source domain image-level features Extract source domain instance-level features Image-level features from auxiliary domain Extract auxiliary domain instance level features Among them, ROIAlign() represents the region of interest alignment operation, B S Represents the labeled source domain image I S The location information of the marked object on the is the source domain image-level feature set Source domain image-level features in , is the auxiliary domain image-level feature set Auxiliary domain image-level features in [15].

5. The cross-domain target detection method based on auxiliary domain guidance according to claim 1 is characterized in that The labeled source domain image I is input to the target detector S and annotated auxiliary domain image I A Calculate the target detection loss and combine the image-level domain adaptation loss and instance-level domain adaptation loss for end-to-end training, including: Calculate the second loss function and use the second loss function for end-to-end training; wherein the second loss function is L TOTAL =L DET +λ1L DC +λ2L INST ; Among them, λ1 is the weight coefficient corresponding to the image-level domain adaptation loss, λ2 is the weight coefficient corresponding to the multi-scale instance-level domain adaptation loss, L DET is the target detection loss, L DC is the image-level domain adaptation loss, L INST is a multi-scale instance-level domain adaptation loss.

6. The cross-domain target detection method based on auxiliary domain guidance according to claim 5 is characterized in that Image-level domain adaptation loss Among them, t i is the domain label of the i-th image, the label of the source domain is t i =0, the label of the target domain t i =1, is the domain classification probability of the i-th image at the (x, y) position of the feature map.

7. The cross-domain target detection method based on auxiliary domain guidance according to claim 5 is characterized in that: Multi-scale instance-level domain adaptation loss Among them, l i is the instance contrast loss at a single scale.

8. The cross-domain target detection method based on auxiliary domain guidance according to claim 7 is characterized in that: Instance contrast loss at a single scale Where τ is the temperature parameter, τ>0, Represents the source domain image level features from the i-th scale The j-th instance feature obtained, P(j) is The positive sample index set in the auxiliary domain, N J is the number of objects in the image, for The positive sample instance feature in the auxiliary domain, Neg(j) is The set of negative sample indexes in the auxiliary domain, for Negative instance features in the auxiliary domain.

9. The cross-domain target detection method based on auxiliary domain guidance according to any one of claims 1 to 8, characterized in that: The style transfer model is a CUT model.

10. The cross-domain target detection method based on auxiliary domain guidance according to any one of claims 1 to 8, characterized in that: The target detector is a YOLO single-stage target detector.

Images