A target detection domain adaptation method based on background and foreground cropping and interchanging

Abstract

The application discloses a target detection domain adaptation method based on background and foreground cropping and interchanging. The application adopts a teacher-student model framework of semi-supervised learning, slowly updates through an exponential moving average of a teacher model, enables the model to obtain information from main knowledge learned by a student model, and thus realizes a self-supervision effect. A domain discriminator is introduced to perform adversarial training, enhances the ability of the model to extract domain-invariant representation, and thus improves robustness. In addition, a foreground-background mixed instance strategy further enhances the domain adaptation capability of the model, effectively creates target domain instance images with source domain backgrounds and source domain instance images with target domain backgrounds through cropping, padding and pasting operations. Finally, an iterative optimization strategy is adopted to continuously improve the model performance, a student model is updated through gradient back propagation by calculating multiple loss functions, and a teacher model is updated using an exponential moving average.

Description

Technical Field

[0001] The present invention belongs to the field of target detection domain adaptation and image processing, and particularly relates to a target detection domain adaptation method based on background and foreground cropping interchange. Background Art

[0002] Object detection, a fundamental task in computer vision, encompasses two main aspects: object classification and localization. Its core goal is to accurately locate and identify objects of interest belonging to predefined target categories using bounding boxes. Object detection has attracted considerable attention due to its wide range of applications in areas such as autonomous driving, video surveillance, and robotic perception.

[0003] However, deep learning-based detectors often suffer from severe performance degradation when the models are applied to new domains, that is, when the visual appearance of the new samples is significantly different from the samples for which the model was trained. This phenomenon is known as domain shift.

[0004] To overcome this challenge, recent research has focused on the development of unsupervised domain adaptation techniques. These techniques aim to build deep network models that can learn from labeled source domain datasets and transfer this learning to target domain datasets without relying on tedious manual labeling. In the field of object detection, most unsupervised domain adaptation methods rely on adversarial training and utilize gradient reversal layers for image- and instance-level adaptation. Other methods are based on pseudo-label self-training methods, utilizing a teacher model trained in the source domain to generate high-confidence pseudo-labels in the target domain. These pseudo-labels are then used to supervise the student model's predictions for the target domain. However, the pseudo-labels generated directly by the teacher model in the target domain are highly noisy, so most methods use confidence thresholds to filter out low-confidence pseudo-labels. However, due to the significant difference in visual appearance between the target and source domains, the teacher-student model struggles to adapt to the target domain. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method for target detection domain adaptation based on background and foreground cropping interchange. The method performs target detection domain adaptation through a teacher-student model with background and foreground cropping interchange.

[0006] The present invention proposes to use the teacher-student model framework commonly used in the field of semi-supervised learning to perform cross-domain adaptation tasks. The teacher model is slowly updated by the exponential moving average of the student model. The update coefficient α determines the knowledge update speed of the teacher model for the student model. The teacher model is regarded as a robust set of the student model, which can obtain the main knowledge learned by the student model. Therefore, the prediction output by the teacher model can be regarded as the pseudo label of the student model, so as to achieve the effect of self-supervision. During the training phase, the source domain and target domain data are trained simultaneously, with half of the source domain data and half of the target domain data in a batch. The source data uses its own labels for supervised training, while the target domain data uses the output of the teacher model to obtain pseudo labels after confidence screening. This is used for unsupervised training. The algorithm will be based on the source domain label y s and target domain pseudo labels The bounding box detected in the source domain is cropped to obtain the instance image and instance images of the target domain The original position is filled with average pixels, and the background image containing only background information is obtained after filling. and At the same time, the source domain instance is pasted onto the target domain background image, and the target domain instance is pasted onto the source domain background image, thereby forming a target domain instance image with a source domain background. and source domain instance images with target domain background We first use a domain discriminator D to distinguish the background images of the two domains. and From which domain respectively, through such an adversarial training, the backbone network of the model can extract the domain invariant information in the two domain backgrounds as much as possible, so that the model is less sensitive to background changes. Then we use the source domain label y s Supervised target domain background mixed with source domain instances Use target domain pseudo labels Supervised source domain background mixed with target domain instance images This strengthens the model's cross-domain robustness, allowing the student model to achieve similar detection results for instances in different domains. After training through the above steps, the teacher and student models are ultimately able to enhance their robustness across domains, increase their focus on the instances themselves, and thus improve detection performance.

[0007] A method for object detection domain adaptation based on background and foreground cropping interchange, the specific steps are as follows:

[0008] Step 1: Use ResNet50 pre-trained on ImageNet as the backbone network of the student model and the teacher model to extract image features. The detector uses Faster-RCNN. At the beginning of training, the student model is copied to the teacher model.

[0009] Step 2: The labeled source domain data (I s ,y s ) Input the student model and output the test result y' s with label y s Calculate the supervised loss L sup , conduct supervised training; at the same time, the unlabeled target domain data (I t ,none) inputs the teacher model and selects high-confidence prediction targets as pseudo labels based on the output classification confidence δ.

[0010] Step 3: When the teacher model output meets the requirements for generating pseudo labels, the foreground-background mixed instance strategy is started:

[0011] First, according to the source domain label y s and target domain pseudo labels Obtain the bounding box location information of each instance for the source domain image I s and target domain image I t Perform cropping to obtain the cropped source domain instance image and target domain instance images At the same time, the original position is filled with the average pixel to obtain the background image of the two domains and Copy a background image of the two domains and paste it crosswise with the two domain instance targets. Paste to target domain background On the target domain instance Paste to source domain background The foreground-background mixed image of the two domains is obtained and The two domain background images obtained Load the domain discriminator D and calculate the adversarial loss L adv , so as to conduct adversarial training with the backbone network; at the same time, and Perform data enhancement and then input the student model to calculate the mixed loss L with the obtained detection results, labels and pseudo labels mix .

[0012] Step 4: After steps 1 to 3, one iteration is completed. By calculating L sup , L adv , L mixPerform gradient backpropagation to update the student model, and then update the teacher model through exponential moving average. sup Enhanced the baseline capability of the student model in the source domain; L adv Effectively reduces the model's attention to the background; L mix By cross-exchanging the foreground and background images of the two domains, the model's attention to foreground instances is increased, thereby enhancing the model's cross-domain adaptability.

[0013] The beneficial effects of the present invention are as follows:

[0014] This paper innovatively proposes a teacher-student model with foreground and background cropping interchangeability to solve the domain adaptation problem of target detection. This method adopts a teacher-student model framework for semi-supervised learning and slowly updates the teacher model through the exponential moving average, allowing the model to obtain information from the main knowledge learned by the student model, thereby achieving a self-supervisory effect. By introducing a domain discriminator for adversarial training, the model's ability to extract domain-invariant representations is enhanced, thereby improving robustness. In addition, the foreground-background mixed instance strategy further enhances the model's domain adaptation ability. Through cropping, padding, and pasting operations, it effectively creates target domain instance images with source domain backgrounds and source domain instance images with target domain backgrounds. Finally, an iterative optimization strategy is adopted to continuously improve model performance. The student model is updated by calculating multiple loss functions for gradient backpropagation, and the teacher model is updated using exponential moving average. This comprehensive approach enables the target detection model to have cross-domain detection capabilities, providing another solution to the cross-domain adaptation task of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of foreground-background mixed instance strategy;

[0016] Figure 2 This is an overall framework diagram of an embodiment of the present invention;

[0017] Figure 3 It is a comparative test of the performance convergence curves between different methods;

[0018] Figure 4 This is a visualization of the detection results of the embodiment of the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0020] First, the specific details of the implementation of the embodiment are described:

[0021] In this embodiment of the present invention, we employ an unsupervised domain adaptation criterion. The training data is divided into two parts: source domain training data, which contains images and their labels (bounding boxes and instance categories), and target domain training data, which contains only unlabeled images. The fine-tuned model is tested on a test set with the same data distribution as the target domain training data. In all experiments, we use mean average precision (mAP) as the primary evaluation metric, with a threshold of 0.5.

[0022] The teacher-student model of the present invention is isomorphic. The present invention uses ResNet-50 as the backbone network of the teacher-student model and adopts FPN to divide the features into different scales. All training and test images are resized so that the length of the shorter side is 800 pixels to fit into the GPU memory. The teacher-student model is initialized using weights pre-trained on ImageNet. Each batch contains 1 source domain image and 1 target domain image. The model is trained for 60,000 iterations with an initial learning rate of 0.0025 and a weight decay of 0.0001. The exponential moving average EMA weight α = 0.9, and color enhancement (brightness, contrast, saturation, hue and blur) is performed. The experiment sets the threshold δ = 0.8 for the pseudo-label box, and the experiment is performed on an RTX 2080Ti or Titan RTX GPU, depending on the required GPU memory.

[0023] Then introduce the required data set:

[0024] Our method is evaluated by studying domain differences between weather conditions. Weather is a significant source of domain differences, as scenes visually differ as weather conditions change. Reliable operation of detection systems in diverse weather conditions is crucial for safe autonomous driving. Our experiments use Cityscapes as the source domain, where images are primarily captured on sunny days. In this experiment, we obtain the ground-truth bounding box by fitting the tightest rectangle around the corresponding instance mask.

[0025] For the target domain, we used the Foggy Cityscapes dataset. Foggy Cityscapes is a synthetic foggy dataset that simulates fog in real scenes. Foggy images are created using original clear-sky images from Cityscapes as well as depth maps. The semantic annotations and data splitting of Foggy Cityscapes are inherited from Cityscapes, which makes it ideal for studying domain differences caused by visibility changes. Foggy Cityscapes contains multiple versions, each with a fixed fog density level, but in this experiment, we choose the version with the most challenging fog density level corresponding to a visibility distance of 150 meters. Therefore, we ultimately use the Cityscapes training set as the (labeled) source domain training set, and the Foggy Cityscapes training set as the (unlabeled) target domain training set.

[0026] Here are the specific implementation steps:

[0027] Step 1: Perform supervised training: The labeled source domain image I s Input the student model and the detection result y′ of the student model s with label y s Calculate the loss, the supervised loss is:

[0028] L sup =L cla (C′ S , C s )+L reg (B′ S , B s )

[0029] The test result y′ s Contains the predicted bounding box position B′ s and target box type C′ s , B s 、C s is the label y s The bounding box locations and categories in .

[0030] At the same time, the unlabeled target domain image I t The teacher model is input, and the detection boxes with a classification confidence higher than 0.8 output by the teacher model are selected to serve as pseudo labels.

[0031]

[0032] Step 2: When there is a target with a confidence score greater than 0.8, get the target domain pseudo label Then, the foreground-background mixed instance strategy is started:

[0033] First, according to the source domain label ys and the target domain pseudo label The bounding box position information of each instance in is cropped to obtain the instance targets cropped in the two domains, where the position information is expressed as follows:

[0034] for bbox inB s : [x1, y1, x2, y2]

[0035] At the same time, the original positions of these cropped targets are filled with the average pixel value of the RGB channel on ImageNet [123.68, 116.78, 103.94] to generate the source domain background image containing background information and the target domain background image Then copy the background images of the two domains and perform a cross-paste operation on these copied background images and instance targets of the two domains: the source domain instance is pasted on the background of the target domain to obtain the source domain instance image of the target domain background The target domain instance is pasted onto the source domain background to obtain the target domain instance image on the source domain background.

[0036] Next, the background images of the two domains are obtained The input is fed into the domain discriminator, and the adversarial loss is calculated based on its output and the domain label.

[0037] L adv =D(I bg ,y domain )

[0038] Where D represents the domain discriminator, I bg Represents the background image y domain is the domain label, where the source domain is 0 and the target domain is 1. This adversarial loss plays an important role in training the model. It helps to reduce the model's sensitivity to background changes, thereby improving the model's adaptability and domain robustness in different domains. In addition, a series of data augmentation operations are performed on the foreground-background mixed images of the two domains: the specific data augmentation operations include: Gaussian noise, random horizontal inversion, random angle rotation, random scaling, center cropping, and random padding, where each type of data augmentation operation is enabled with a probability of 50% to further enrich the diversity of the training data. Subsequently, the enhanced mixed image is input into the student model for processing, and its detection results are used together with the actual labels and pseudo labels to calculate the mixed loss. The purpose of this step is to enhance the generalization ability of the model when processing cross-domain data, thereby improving its detection performance of target instances in different domains.

[0039] An image where the source domain background is mixed with the target domain instance Its prediction results should be consistent with the pseudo labels Consistent, while the target domain background is mixed with the source domain instance should be the same as the source domain label y s Consistently, the hybrid loss is expressed as:

[0040]

[0041] in Represents the student model for The predicted output of Represents the student model for The predicted output of It is a pseudo label The bounding box locations and categories in .

[0042] This meticulous process aims to effectively handle the differences between the source and target domains through domain adaptation techniques, thereby improving the performance and robustness of the object detection model. The key to this method lies in the use of adversarial training of domain discriminators and mixed images, as well as the calculation of mixed losses, which enables the model to better adapt to data in different domains and achieve cross-domain object detection capabilities.

[0043] Step 3: After a series of operations from step 1 to step 2, a round of iteration is completed, which is the key step to achieve cross-domain target detection. In this step, we first use the calculated supervised loss L sup , adversarial loss L adv and mixing loss L mix , performing gradient backpropagation to update the parameters of the student model. This process is a crucial part of deep learning training. Through gradient backpropagation, the model is continuously optimized in each round of iteration, gradually improving its detection performance.

[0044] Next, the exponential moving average is introduced to update the teacher model.

[0045] The teacher model does not use gradient backpropagation for parameter updates, but instead uses a smoothing approach. By taking an exponential moving average of the student model's parameters, the teacher model gradually accumulates knowledge from multiple iterations. This accumulation of knowledge makes the teacher model a robust ensemble of the student model, integrating key knowledge learned by the student model and further improving the model's stability and robustness. The update formula is as follows:

[0046]

[0047] in represents the parameters of the teacher model, Represents the parameters of the student model, α is the weighted weight value, which represents the update rate of the model, usually between 0.9-0.9999, and the value here is 0.9.

[0048] The above steps are iterated 60,000 times to complete the entire training task. The foreground-background mixed instance strategy trains the model to detect targets under different background conditions, making the model more robust and able to operate reliably in various complex and changing road environments. This is crucial for coping with changes in bad weather, lighting conditions, or road conditions. Simultaneously swapping the foreground and background helps the model learn the essential relationship between the target and the background, rather than just memorizing the training data. This reduces the risk of overfitting and makes the model more robust in new scenarios. adv It effectively reduces the model's attention to the background; by cross-exchanging the foreground and background images of the two domains, it increases the model's attention to foreground instances, thereby enhancing the model's cross-domain adaptability.

[0049]

[0050] Table 1 Cross-domain adaptation experiment of Cityscapes → Foggy Cityscapes (visible at 150 meters)

[0051] Table 1 shows cross-domain quantitative experiments conducted on the Cityscapes and Foggy Cityscapes datasets using an embodiment of the present invention. Cityscapes is the source domain, and Foggy Cityscapes is the target domain. This embodiment uses mean average precision (mAP) as the primary evaluation metric, with a threshold of 0.5. It can be seen that the proposed method outperforms other state-of-the-art methods, demonstrating the effectiveness of the foreground-background instance mixing strategy.

[0052] Figure 3 This is a comparative test of the performance convergence curves of different methods. It can be found that the embodiments of the present invention can converge quickly and the performance is continuously improved. Figure 4 This is a visualization of the detection results of the embodiment. It can be found that the present invention can accurately identify easily confused categories, and has accurate positioning and good recognition effect.

[0053] Object detection is an essential component of autonomous driving technology. In autonomous driving systems, driverless vehicles must use visual sensors (such as cameras and lidar) to obtain real-time information about the road and traffic environment in order to make accurate driving decisions. Object detection technology primarily identifies and tracks various traffic elements, such as traffic signs, other vehicles, pedestrians, and bicycles, during driving, assisting autonomous vehicles in making accurate driving decisions.

[0054] However, in real-world road environments, many factors (such as weather and road conditions) can affect image quality and feature distribution, which can lead to differences between the captured images and the datasets used for model training. This discrepancy causes the performance of object detection models in real-world scenarios to be inferior to that on standard datasets, a problem known as domain adaptation.

[0055] Our proposed method aims to enhance the robustness and generalization capabilities of autonomous driving systems. This method utilizes unlabeled target domain data for domain adaptation and improves the robustness of cross-domain detection by training a deep learning model, enabling the model to effectively adapt to real-world scenarios even with limited training data.

[0056] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0057] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A method for object detection domain adaptation based on background and foreground cropping interchange, characterized in that: The specific steps are as follows: Step 1: Use ResNet50 pre-trained on ImageNet as the backbone network of the student model and the teacher model to extract image features. The detector uses Faster-RCNN. At the beginning of training, copy the student model to the teacher model. Step 2: The labeled source domain data (I s ,y s ) Input the student model and output the test result y' s with label y s Calculate the supervised loss L sup , conduct supervised training; at the same time, the unlabeled target domain data (I t ,none) input the teacher model, and select the high-confidence prediction targets as pseudo labels according to the output classification confidence δ; Step 3: When the teacher model output meets the requirements for generating pseudo labels, the foreground-background mixed instance strategy is started: First, according to the source domain label y s and target domain pseudo labels Obtain the bounding box location information of each instance for the source domain image I s and target domain image I t Perform cropping to obtain the cropped source domain instance image and target domain instance images At the same time, the original position is filled with the average pixel to obtain the background image of the two domains and Copy a background image of the two domains and paste it crosswise with the two domain instance targets. Paste to target domain background On the target domain instance Paste to source domain background The foreground-background mixed image of the two domains is obtained and The two domain background images obtained Load the domain discriminator D and calculate the adversarial loss L adv , so as to conduct adversarial training with the backbone network; at the same time, and Perform data enhancement and then input the student model to calculate the mixed loss L with the obtained detection results, labels and pseudo labels mix ; Step 4: After steps 1 to 3, one iteration is completed. By calculating L sup 、L adv 、L mix Perform gradient backpropagation to update the student model, and then update the teacher model through exponential moving average.

2. The object detection domain adaptation method based on background and foreground cropping interchange according to claim 1, characterized in that: The step 2 is specifically implemented as follows: The labeled source domain image I s Input the student model and the detection result y' of the student model s with label y s Calculate the loss, the supervised loss is: L sup =L cla (C' S ,C s )+L reg (B' S ,B s ) The test result y' s Contains the predicted bounding box location B' s and target box type C' s , B s 、C s is the label y s The bounding box location and category in ; At the same time, the unlabeled target domain image I t Input the teacher model and select the detection boxes with a classification confidence higher than 0.8 as pseudo labels based on the classification confidence output by the teacher model; if δ>0.8:

3. The object detection domain adaptation method based on background and foreground cropping interchange according to claim 1 or 2, characterized in that: Step 3: When there is a target with a confidence greater than 0.8, the target domain pseudo label is obtained Then, the foreground-background mixed instance strategy is started: First, according to the source domain label y s and target domain pseudo labels The bounding box position information of each instance in is cropped to obtain the instance targets cropped in the two domains, where the position information is expressed as follows: for bbox in B s :[x1,y1,x2,y2] At the same time, the original positions of these cropped targets are filled with the average pixel value of the RGB channel on ImageNet [123.68, 116.78, 103.94] to generate the source domain background image containing background information and the target domain background image Then copy the background images of the two domains and perform a cross-paste operation on these copied background images and instance targets of the two domains: the source domain instance is pasted on the background of the target domain to obtain the source domain instance image of the target domain background The target domain instance is pasted onto the source domain background to obtain the target domain instance image on the source domain background. Next, the background images of the two domains are obtained Input into the domain discriminator and calculate the adversarial loss based on its output and domain label; L adv =D(I bg ,y domain ) Where D represents the domain discriminator, I bg Represents the background image y domain is the domain label, where the source domain is 0 and the target domain is 1; In addition, data augmentation operations are performed on the foreground-background mixed images of the two domains; Subsequently, the enhanced mixed image is fed into the student model for processing, and its detection results are used together with the actual labels and pseudo labels to calculate the mixed loss; An image where the source domain background is mixed with the target domain instance Its prediction results should be consistent with the pseudo labels Consistent, while the target domain background is mixed with the source domain instance should be the same as the source domain label y s Consistently, the hybrid loss is expressed as: in Represents the student model for The predicted output of Represents the student model for The predicted output of It is a pseudo label The bounding box locations and categories in .

4. The object detection domain adaptation method based on background and foreground cropping interchange according to claim 3, characterized in that: Step 4: First, we use the calculated supervised loss L sup , adversarial loss L adv and mixing loss L mix , perform gradient backpropagation operations to update the parameters of the student model; Next, the exponential moving average is introduced to update the teacher model; The teacher model updates its parameters in a smooth manner. By taking the exponential moving average of the student model parameters, the teacher model gradually accumulates knowledge from multiple rounds of iterations. The update formula is as follows: in represents the parameters of the teacher model, represents the parameters of the student model, α is the weighted weight value; The above steps are iterated 60,000 times to complete the entire training task; foreground-background mixed example.

Images