A domain adaptation target detection method based on target domain generalization estimation

Abstract

The application discloses a domain self-adaption target detection method based on target domain generalization estimation, relates to the field of computer vision, and particularly relates to a target detection method for unsupervised domain self-adaption in deep learning.The application uses an auxiliary module to perform inconsistency prediction, thereby avoiding huge computing cost caused by multiple training of the whole detection model.The generalization error on the target domain is estimated according to the inconsistency predicted by the auxiliary module, and then the target detection performance on the target domain is directly optimized, so that the method is more effective than an indirect optimization method through adversarial training;the classification and regression subtasks of the target detection are respectively optimized, so that the classification accuracy is better and the frame positioning is more accurate.The high-confidence main classifier prediction is screened as a pseudo label through a confidence module, and source domain knowledge is fully explored to help improve the cross-domain detection performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a target detection method for unsupervised domain adaptation in deep learning. BACKGROUND

[0002] The target detection task is an important research direction in computer vision, which needs to classify and identify the target objects contained in a picture and locate them. The task has a wide range of applications in finance, military, autonomous driving and other fields. In recent years, with the help of deep learning algorithms, the performance of target detection models has been significantly improved. However, when dealing with test images with different sample distributions from the training set, the target detection model often faces a serious performance loss problem, and retraining with supervision requires a lot of manpower and material resources for data labeling, which greatly limits the application scenarios of the target detection model.

[0003] At present, the common solution is to use unsupervised domain adaptation algorithms to adjust the target detection model. These methods either align the feature distributions of the source domain and the target domain from the feature level through adversarial training, indirectly optimizing the performance indicators on the target domain, or need additional model to generate pseudo labels when directly optimizing the performance indicators on the target domain, or do not consider the difference in cross-domain performance between the classification task and the positioning task in target detection, or need to train an additional style conversion network to assist the domain adaptation of the target detection model. Therefore, there is an urgent need for an unsupervised domain adaptation target detection algorithm with small training overhead and more detailed optimization of the performance indicators on the target domain. SUMMARY

[0004] In order to solve the problems in the prior art, the present application provides an unsupervised domain adaptation target detection method based on target domain generalization estimation, which solves the problems mentioned in the background.

[0005] To achieve the above purpose, the present application provides the following technical scheme: on the basis of the existing adversarial training model, an additional auxiliary classification module and an auxiliary regression module are introduced, and the inconsistency between the different outputs of these modules is used to estimate the generalization error on the target domain, and then the detection performance indicators on the target domain are directly optimized.

[0006] Specifically, the technical scheme of the present application is an unsupervised domain adaptation target detection method based on target domain generalization estimation, which comprises the following steps:

[0007] S1, data preprocessing, scaling and data augmentation according to the size of the source domain samples and the target domain samples;

[0008] S2, a target detection base network is constructed, the target detection base network comprising: a feature extraction network, a region proposal network RPN, a fully connected layer FC, a main target classifier, and a main regressor; an output of the feature extraction network is subjected to the region proposal network RPN to obtain a region of interest; a feature of the region of interest is obtained according to the region of interest in combination with the output of the feature extraction network, and the feature of the region of interest is input to the main classifier and the main regressor after being subjected to the fully connected layer FC; outputs of the main classifier and the main regressor are used to calculate a main classifier loss and a main regressor loss respectively when the domain adaptive target detection network is input with a source domain sample; an output of the main classifier is a pseudo label when the domain adaptive target detection network is input with a target domain sample, and the pseudo label is used in combination with an output of an auxiliary classification module to calculate a pseudo label loss;

[0009] S3, a domain classification module and a confidence prediction module are constructed, the domain classification module being used for adversarial training, and the confidence prediction module being used for pseudo label training;

[0010] The domain classification module comprises: a gradient reversal layer and four domain classifiers, an input of the gradient reversal layer being outputs of last three blocks of the feature extraction network and the feature of the region of interest obtained in step S2; each input is output to a different domain classifier after passing through the gradient reversal layer; and outputs of the domain classifiers are used to calculate a domain classification loss.

[0011] An input of the confidence prediction module is an output of the fully connected layer FC, an output of the confidence prediction module being used to calculate a confidence loss when the domain adaptive target detection network is input with the source domain sample, and the output of the confidence prediction module being used for pseudo label screening when the domain adaptive target detection network is input with the target domain sample.

[0012] S4, an auxiliary classification module and an auxiliary regression module are constructed.

[0013] The auxiliary classification module comprises M auxiliary classifiers, and an input of each auxiliary classifier is the output of the fully connected layer FC; the auxiliary regression module comprises N auxiliary regressors, and an input of each auxiliary regressor is the output of the fully connected layer FC.

[0014] Outputs of the auxiliary classification module are used to calculate an auxiliary classifier loss, and outputs of the auxiliary regressors are used to calculate an auxiliary regressor loss when the domain adaptive target detection network is input with the source domain sample.

[0015] Outputs of the auxiliary classification module are used to calculate a classification inconsistency loss, and outputs of the auxiliary regressors are used to calculate a regression inconsistency loss when the domain adaptive target detection network is input with the target domain sample.

[0016] S5, construct a total loss function Ltotal, which mainly includes target detection loss Ldet, domain classification loss Lda, prediction inconsistency loss Ldis, pseudo label loss Lpse and confidence loss Lconf;

[0017] Ltotal = Ldet + λ1·Lda + λ2·Ldis + λ3·(Lpse + Lconf)

[0018] The target detection loss Ldet includes: main classifier loss, main regressor loss, auxiliary classifier loss and auxiliary regressor loss; the prediction inconsistency loss Ldis includes: classification inconsistency loss and regression inconsistency loss; λ1, λ2, λ3 are weights for balancing different losses;

[0019] Ldet = Lrpn + Lroi + Laux

[0020] Wherein, Lrpn and Lroi are the loss functions of the target detection basic network, and Laux is the sum of the auxiliary classifier loss and the auxiliary regressor loss;

[0021] Lda = Ld1 + Ld2 + Ld3 + Ld4

[0022] Wherein Ld1, Ld2, Ld3, Ld4 represent four domain classification losses respectively, Ld1 uses MSE loss, Ld2 and Ld4 use cross entropy loss, and Ld3 uses Focal Loss loss;

[0023]

[0024] Wherein, is the classification inconsistency loss; is the regression inconsistency loss, and α is a weight coefficient for balancing the two losses;

[0025] Lconf uses cross entropy loss, and the confidence label of the source domain training sample is 1 during training;

[0026] S6, train the domain adaptive target detection network constructed in steps S2 to S4, and the loss function is the loss function constructed in step S5;

[0027] S7, after the training is completed, the target detection basic network in step 2 is used for target detection test on the target domain.

[0028] The beneficial effects of the above technical solutions are as follows:

[0029] 1. Use the auxiliary module for inconsistency prediction to avoid huge computing cost caused by multiple training of the entire detection model.

[0030] 2、Through the inconsistency predicted by the auxiliary module to estimate the generalization error on the target domain, and then directly optimize the target detection performance on the target domain, which is more effective than the indirect optimization method through adversarial training

[0031] 3、The classification and regression sub-tasks of target detection are optimized respectively, so that the classification accuracy is better and the frame positioning is more accurate.

[0032] 4、Through the confidence module to filter high-confidence main classifier predictions as pseudo-labels, the source domain knowledge is fully explored to help improve cross-domain detection performance. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The network model structure diagram implemented by the present application is shown in the figure;

[0034] Figure 2 The network structure diagram of the domain classifier D1 of the present application is shown in the figure;

[0035] Figure 3 The network structure diagram of the domain classifiers D2 and D4 of the present application is shown in the figure;

[0036] Figure 4 The network structure diagram of the domain classifier D3 of the present application is shown in the figure;

[0037] Figure 5 The network structure diagram of the confidence module used in the present application is shown in the figure. DETAILED DESCRIPTION

[0038] Please refer to Figure 1 The present application provides a technical solution: an unsupervised domain adaptive target detection method based on target domain generalization estimation, which inputs the labeled samples of the source domain and the unlabeled samples of the target domain into the target detection model for domain adaptive training, and obtains a detection model with better performance on the target domain. The target detection network during training includes a basic Faster R-CNN detection network, an additional domain classifier module, an additional confidence module, an additional auxiliary classification module and an additional auxiliary regression module. The configuration environment is: Python3.8, Pytorch1.9.0, CUDA11.1.

[0039] S1, data preprocessing, according to the image size of the source domain and the target domain to scale appropriately; in this embodiment, the training set of Pascal VOC 2007 and Pascal VOC 2012 is used as the source domain, and all the pictures of Clipart1K are used as the target domain images. In order to ensure the consistency of the image data size during training, they are all scaled to pictures with a short side length of 600 pixels when input.

[0040] In addition, in order to improve the robustness of the detection model, the regions of interest of these images are enhanced by horizontal flipping at the same time.

[0041] During training, one source domain image and one target domain image are input at the same time each time, and when all the samples of Clipart are traversed, the samples of the data set will be traversed again until 120,000 iterations are reached.

[0042] S2, build the target detection base network Faster R-CNN. Specifically, use the Faster R-CNN network with RoI Align, and the backbone network is ResNet101.

[0043] S3, build the domain classification module and the confidence prediction module, the former is used for adversarial training, and the latter is used for subsequent pseudo-label training. The domain classification module uses the features output by the last three residual networks of ResNet101 and the instance features for domain classification, and the specific structure is as shown in Figures 2-4 The first classifier D1 uses MSE loss, the second and fourth classifiers D2 and D4 use cross-entropy loss, and the third classifier D3 uses Focal Loss loss. The confidence prediction module takes the output of the full connection layer FC as input, and finally gives the predicted confidence. The specific structure is as shown in Figure 5 .

[0044] S4, build the auxiliary classification module and the auxiliary regression module, both of which include multiple auxiliary classifiers and auxiliary regressors. The auxiliary classifiers are completely consistent with the main classifier structure of the original Faster R-CNN, and the difference is that the auxiliary classifiers are initialized using different random seeds between each other to achieve the purpose of predicting generalization error. The auxiliary regressor is the same. Specifically, this embodiment uses 2 additional auxiliary classifiers as the auxiliary classification module and 8 additional auxiliary regressors as the additional auxiliary regression module.

[0045] S5, build the loss function Ltotal, which includes the target detection loss Ldet, the domain classification loss Lda, the prediction inconsistency loss Ldis, and the pseudo-label loss Lpse. The loss function expression is as follows:

[0046] Ltotal= Ldet +λ1·Lossda +λ2·Lossdis +λ3·Losspse (1)

[0047] λ1, λ2, λ3 are weights used to balance different losses;

[0048] Ldet = Lrpn + Lroi + Laux (2)

[0049] wherein Lrpn and Lroi are the loss function of the original Faster R-CNN, and Laux is the supervised loss function of the source domain picture in the auxiliary module, consistent with Faster R-CNN, cross-entropy loss is used in the auxiliary classification module, and smooth L1 loss is used in the auxiliary regression module;

[0050] Lda= Ld1+Ld2+Ld3+Ld4 (3)

[0051] wherein Ld1 uses the MSE loss, Ld2 and Ld4 use the cross-entropy loss, and Ld3 uses the Focal Loss loss;

[0052]

[0053] wherein is the classification inconsistency loss; is the regression inconsistency loss, and alpha is the weight coefficient for balancing the two losses;

[0054] S6, the domain adaptation target detection network constructed by steps S2-4 is trained, the loss function is the loss function constructed in step S5, and the network is optimized using the SGD optimizer, wherein the initial learning rate is set to 0.001, and becomes 0.1 times of the original after every 50,000 iterations; the values of the hyperparameters lambda are all 1, and the value of alpha is 0.01;

[0055] S7, after the training is completed, the Faster R-CNN network in step 2 is used for target detection test on the target domain. When the test is performed, no additional modules (the domain classification module, the confidence module, the auxiliary classification module and the auxiliary regression module) proposed in the method are needed, and only the target detection network Faster R-CNN is used;

[0056] The method proposed in the application has great innovation compared with the original unsupervised domain adaptation method, the auxiliary module is introduced, the generalization error on the target domain is estimated by the inconsistency predicted by the auxiliary module under the condition of saving the calculation overhead, and then the target detection performance on the target domain is directly optimized, which is more effective than the indirect optimization mode by simply using the adversarial training. In addition, the target detection classification and the regression subtask are optimized respectively and the pseudo label strategy further improves the detection performance on the target domain.

[0057] Table 1: Comparison of cross-domain detection performance from the Pascal VOC dataset to the Clipart dataset

[0058]

[0059] Table 2: Comparison of cross-domain detection performance from the Cityscapes dataset to the Foggy Cityscapes dataset

[0060]

Claims

1. A domain adaptation target detection method based on target domain generalization estimation, the method comprising the following steps: S1, data preprocessing, scaling and data augmentation according to the size of the source domain sample and the target domain sample; using the training set of PascalVOC 2007 and PascalVOC 2012 as the source domain, and using all the pictures of Clipart1K as the target domain image; S2, constructing a target detection basic network, the target detection basic network comprising: a feature extraction network, a region proposal network RPN, a fully connected layer FC, a main target classifier, and a main regressor; the output of the feature extraction network is obtained after passing through the region proposal network RPN to obtain a region of interest; the feature of the region of interest is obtained according to the region of interest combined with the output of the feature extraction network, and the feature of the region of interest is input to the main classifier and the main regressor after passing through the fully connected layer FC; when the input of the domain adaptation target detection network is the source domain sample, the outputs of the main classifier and the main regressor are used to calculate the main classifier loss and the main regressor loss respectively; when the input of the domain adaptation target detection network is the target domain sample, the output of the main classifier is a pseudo label, and the pseudo label combined with the output of the auxiliary classification module is used to calculate the pseudo label loss; S3, constructing a domain classification module and a confidence prediction module, the domain classification module is used for adversarial training, and the confidence prediction module is used for pseudo label training; the domain classification module comprises: a gradient reversal layer and four domain classifiers, the input of the gradient reversal layer is the output of the last three blocks of the feature extraction network and the feature of the region of interest obtained in step S2; each input passes through the gradient reversal layer and is output to different domain classifiers; the output of the domain classifier is used to calculate the domain classification loss; the input of the confidence prediction module is the output of the fully connected layer FC, when the input of the domain adaptation target detection network is the source domain sample, the output of the confidence prediction module is used to calculate the confidence loss, and when the input of the domain adaptation target detection network is the target domain sample, the output of the confidence prediction module is used for pseudo label screening; S4, constructing an auxiliary classification module and an auxiliary regression module; the auxiliary classification module comprises M auxiliary classifiers, and the input of each auxiliary classifier is the output of the fully connected layer FC; the auxiliary regression module comprises N auxiliary regressors, and the input of each auxiliary regressor is the output of the fully connected layer FC; when the input of the domain adaptation target detection network is the source domain sample, the output of the auxiliary classification module is used to calculate the auxiliary classifier loss, and the output of the auxiliary regressor is used to calculate the auxiliary regressor loss; when the input of the domain adaptation target detection network is the target domain sample, the output of the auxiliary classification module is used to calculate the classification inconsistency loss, and the output of the auxiliary regressor is used to calculate the regression inconsistency loss; S5, constructing a total loss function Ltotal, the loss function comprising a target detection loss Ldet, a domain classification loss Lda, a prediction inconsistency loss Ldis, a pseudo label loss Lpse and a confidence loss Lconf; Ltotal=Ldet+λ1·Lda +λ2·Ldis+λ3·(Lpse+Lconf); The target detection loss Ldet includes a main classifier loss, a main regressor loss, an auxiliary classifier loss, and an auxiliary regressor loss; the prediction inconsistency loss Ldis includes a classification inconsistency loss and a regression inconsistency loss; λ1, λ2, and λ3 are weights for balancing different losses; Ldet = Lrpn + Lroi + Laux; Wherein, Lrpn and Lroi are loss functions of a target detection basic network, and Laux is a sum of an auxiliary classifier loss and an auxiliary regressor loss; Lda = Ld1 + Ld2 + Ld3 + Ld4; Wherein, Ld1, Ld2, Ld3, and Ld4 respectively represent four domain classification losses, Ld1 uses an MSE loss, Ld2 and Ld4 use cross-entropy losses, and Ld3 uses a Focal Loss loss; Ldis= Ld - Ls + α ; wherein, is a classification inconsistency loss; is a regression inconsistency loss, and a is a weight coefficient for balancing the two losses. Lconf uses a cross-entropy loss, and the confidence label of the source domain training sample is 1 during training; S6, the domain self-adaptive target detection network constructed by steps S2 to S4, and the loss function is the loss function constructed in step S5; S7, after the training is completed, the target detection basic network in step 2 is used for target detection testing on the target domain.

Images