A domain adaptation object detection method based on triple feature decoupling

Abstract

The application discloses a domain self-adaptive target detection method based on triple feature decoupling, which is characterized by utilizing a multistage decoupling module, a cycle decoupling module and a category decoupling module to fully extract domain-invariant features of a source domain and a target domain, and achieving good detection effect on the target domain without labels, and the method specifically comprises the following steps: data set preprocessing, network overall framework construction, multistage feature decoupling, cycle feature decoupling, category feature decoupling and network training and testing. Compared with the prior art, the application has the advantages that fast detection can be realized on the target domain without labels, the domain-invariant features extracted by the triple decoupling contain as little domain private information as possible, and based on the adversarial learning idea, the gradient reversal layer and the domain discriminator are utilized to assist decoupling, and the attention mechanism is utilized to make the network pay more attention to the domain-invariant features, so that the performance of the model is further improved, the method is simple, the detection speed is fast, and the method has higher use value and application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of domain adaptive object detection, and in particular to a domain adaptive object detection method based on triple feature decoupling. Background Art

[0002] Object detection is a key research topic in computer vision and has found widespread applications in fields such as face recognition, autonomous driving, and industrial inspection. In recent years, with the advancement of deep learning, deep learning-based object detection has made significant progress. Current high-performing detectors are typically trained using fully supervised training, which requires that all images in the dataset be labeled. Labeling is labor-intensive, time-consuming, and labor-intensive. Without labeling, directly using a detector trained on another dataset for detection on a given dataset can yield suboptimal results due to domain shift. To address these issues, developing algorithms that can transfer knowledge learned from a labeled source domain to an unlabeled target domain is crucial for object detection. Therefore, unsupervised domain adaptation has been applied to object detection, and a new task, domain adaptive object detection, has been proposed to reduce the domain shift between two domains. Existing methods use gradient reversal layers and domain discriminators to achieve this goal. This allows detectors to achieve acceptable detection performance on unlabeled target domains. However, these methods may overlook the impact of domain-specific information.

[0003] For domain-adaptive object detection tasks, obtaining domain-invariant features is crucial to mitigate the impact of domain shift. Therefore, traditional methods utilize the concept of disentangled learning, aiming to separate domain-specific features from domain-invariant features. The final classification and regression predictions rely on domain-invariant features, which should contain as little domain-private information as possible. However, researchers have shown that these existing methods, using simple disentanglement operations, fail to effectively separate the two. Domain-invariant features extracted through simple operations may contain domain-private information, ultimately leading to detection errors due to the inability to accurately locate the region of interest.

[0004] In their paper "Vector-decomposed disentanglement for domain-invariant object detection," Aming Wu et al. promote vector decomposition to minimize domain-private information in domain-invariant features. In their paper "Decompose to adapt: ​​Cross-domain object detection via feature disentanglement," Dongnan Liu et al. propose feature decoupling at both global and local levels, approaching complete decoupling as closely as possible.

[0005] Existing disentangled learning ignores the impact of domain-private information. Despite its potential benefits, disentangled learning is still not fully utilized in the field of domain-adaptive object detection. Most existing methods rely on simple disentanglement. The domain-invariant features extracted by simple operations may contain domain-private information, which ultimately leads to detection errors due to the inability to accurately locate the region of interest. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the existing technology and propose a domain-adaptive target detection method based on triple feature decoupling. This method uses three different decoupling methods to assist model learning. Through three different mechanisms, a novel decoupling strategy is implemented to maximize the distribution difference between domain-specific features and domain-invariant features, and minimize the domain-private information contained in domain-invariant features. This method first uses a single decoupling operation to simply separate domain-specific features from domain-invariant features. These are then input into a re-decoupling module for secondary decoupling, which can improve the distribution difference between the features. The domain-specific features obtained from the secondary decoupling are input into a cyclic decoupling module, and the domain-specific features output by the cycle are continuously expanded from the domain-invariant features. Finally, category decoupling is achieved at multiple scales, which can improve the recognition accuracy of targets of different scales and has broad and promising application prospects.

[0007] The specific technical solution for achieving the purpose of the present invention is: a domain adaptive target detection method based on triple feature decoupling, which is characterized by using a multi-level decoupling module, a cyclic decoupling module and a category decoupling module to perform triple feature decoupling on the input image, and using auxiliary model learning to expand the distribution difference between domain-specific features and domain-invariant features. The trained model is used to perform target detection on an unlabeled target domain. The multi-level decoupling module is used to make the domain-invariant features contain as little domain-private information as possible. The cyclic decoupling module is used to cyclically extract domain-invariant features and expand the distance between domain-invariant features and domain-specific features during the cyclic process. The category decoupling module is used to improve the detection capability of categories. The domain adaptive detection specifically includes the following steps:

[0008] Step S1: Preprocessing of the dataset

[0009] A source domain dataset and a target domain dataset are selected, where the source domain has labels and the target domain does not have labels, and image enhancement is performed on the training data divided in the source domain and the target domain. The image enhancement specifically includes: random adjustment of image brightness, saturation, and hue, random cropping, random horizontal flipping, random vertical flipping, and random scaling.

[0010] Step S2: Construction of the overall network framework

[0011] The network model is designed based on a commonly used one-stage target detector SSD, uses a VGG16 network as a backbone feature extraction network, and divides the VGG16 network into two parts to serve as domain-invariant feature extractors of different levels for extracting six feature maps of different levels, and uses the feature maps of different levels to detect targets of different sizes, wherein the network model comprises two domain-specific feature extractors and two discriminators, the domain-specific feature extractors each comprise six convolutional layers and three convolutional layers, and the two domain discriminators are global domain discriminators and local domain discriminators, each comprising three convolutional layers.

[0012] Step S3: multi-level feature decoupling

[0013] The domain-invariant features and the domain-specific features are extracted by using the domain-invariant feature extractor and the domain-specific feature extractor respectively, and then input into a secondary decoupling module for secondary decoupling, and the local domain discriminator and the global domain discriminator are used to promote decoupling.

[0014] Step S4: cyclic feature decoupling

[0015] The domain-specific features obtained by the secondary decoupling module are input into a cyclic decoupling module for cyclic decoupling, and a pyramid pooling attention mechanism is used to make the extracted features contain as little domain public information as possible, and in the process, the distance between the domain-specific features and the domain-invariant features obtained by the cycle is expanded.

[0016] Step S5: class feature decoupling

[0017] Class features are extracted from the domain-invariant features and the domain-specific features respectively, the feature distribution of the same class of features obtained by different domain-invariant features is narrowed, and the feature distribution of the same class of features obtained by the same domain-invariant feature is expanded; for the same class of features extracted from the domain-invariant features and the domain-specific features obtained by the same domain, the feature distribution thereof is expanded.

[0018] Step S6: training and use of the network model

[0019] The network model is trained using the training data set of step S1, the model parameters with the best effect on the target domain test set are saved, and the saved model is used for target detection, wherein the saved model is used for target detection by sending a target domain image to be detected into the saved model, and through the processing of the neural network, good detection results can be obtained on the target domain without labels.

[0020] Compared with the prior art, the present application has the following beneficial technical effects and significant technical progress:

[0021] 1) A multi-level decoupling module is proposed, which further separates the domain-specific features and the domain-invariant features after the first decoupling, so that the domain-invariant features contain less domain private information and the detection accuracy is improved.

[0022] 2) Based on multi-level decoupling, we propose cyclic decoupling, continuously extracting domain-invariant features and widening the gap between them and domain-private features. Furthermore, we incorporate a feature pyramid pooling attention mechanism to help filter public information with domain-private features at multiple scales.

[0023] 3) Category decoupling is achieved at both the inter-domain and intra-domain levels, significantly improving the classification capability of the detector. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the network model constructed for the present invention;

[0025] Figure 2 Schematic diagram of the re-decoupling module;

[0026] Figure 3 Schematic diagram of domain discriminator application;

[0027] Figure 4 Schematic diagram of the feature pyramid pooling attention module;

[0028] Figure 5 A schematic diagram showing the comparison of detection effects. DETAILED DESCRIPTION

[0029] See Figure 1 The present invention performs domain adaptive object detection on an unlabeled target domain according to the following steps:

[0030] Step S1: Preprocessing of the dataset

[0031] A source domain dataset and a target domain dataset are selected, where the source domain has labels and the target domain does not have labels, and image enhancement is performed on the training data divided in the source domain and the target domain. The image enhancement specifically includes: random adjustment of image brightness, saturation, and hue, random cropping, random horizontal flipping, random vertical flipping, and random scaling.

[0032] Step S2: Construction of the overall network framework

[0033] The network model is based on the commonly used one-stage target detector SSD design, uses the VGG-16 network as the backbone feature extraction network, and divides it into two, which serve as domain-invariant feature extractors at different levels, respectively, for extracting feature maps of six different levels and using feature maps of different levels to detect targets of different sizes. The network model contains two domain-specific feature extractors and two domain discriminators. The two domain-specific feature extractors are a domain-invariant feature extractor and a domain-specific feature extractor; the two domain discriminators are a global domain discriminator and a local domain discriminator.

[0034] Step S3: Multi-level feature decoupling

[0035] Domain-invariant feature extractor and domain-specific feature extractor are used to extract domain-invariant features and domain-specific features respectively, and then input into the re-decoupling module for secondary decoupling, and local domain discriminator and global domain discriminator are used to promote decoupling respectively.

[0036] Step S4: Loop feature decoupling

[0037] The domain-specific features obtained by the re-decoupling module are input into the recurrent decoupling module for recurrent decoupling. The pyramid pooling attention mechanism is used to ensure that the extracted features contain as little domain-public information as possible, and in the process, the distance between the domain-specific features obtained in the recycle and the domain-invariant features is expanded.

[0038] Step S5: Category feature decoupling

[0039] Category features are extracted from domain-invariant features and domain-specific features respectively, the distribution of features of the same category obtained from domain-invariant features of different domains is narrowed, and the distribution of features of the same category obtained from the same domain-invariant features is expanded; for features of the same category extracted from domain-invariant features and domain-specific features obtained from the same domain, their feature distribution is expanded.

[0040] Step S6: Training and use of network model

[0041] Use the training data set in step S1 to train the constructed network structure, save the model parameters that perform best on the target domain test set, and use the saved model for target detection. Through the processing of the neural network, good detection results can be achieved on the target domain without labels.

[0042] The network model uses the VGG-16 backbone feature extraction network to extract domain-invariant features, and the domain-specific feature extractor to extract domain-invariant features. In addition, a domain discriminator will be used to assist in extracting domain-invariant features. The extracted domain-invariant features will be input into the detection head for target positioning and classification.

[0043] Use the VGG16 network as the backbone feature extraction network,

[0044] The network model is based on a single-stage object detector (SSD). All auxiliary modules, such as the domain-invariant feature extractor, domain discriminator, and feature pyramid pooling attention, are not required during the inference and detection phase, so their parameters do not need to be saved when saving the model. Therefore, inference speed and model size are not affected.

[0045] The present invention will be further described below with reference to specific examples:

[0046] Example 1

[0047] Step S1: Preprocessing of the dataset

[0048] This example uses the publicly available object detection dataset Pascal VOC as the source domain and the Clipart1K dataset as the target domain. The Pascal VOC dataset contains 16,551 real-world images from 20 categories: airplane, bicycle, bird, boat, bottle, bus, car, cat, chair, cow, dining table, dog, horse, motorbike, person, potted plant, sheep, sofa, train, and TV monitor. Clipart1K contains 1,000 clipart-style images from the same 20 categories as Pascal VOC. Clipart1K uses 500 of these images as a training set and 500 as a test set.

[0049] Data preprocessing is performed on the images in the source domain and the training set of the target domain, and the images are resized, randomly cropped, randomly flipped horizontally, randomly flipped vertically, and randomly scaled to enhance the diversity of the training sample dataset.

[0050] Step S2: Construction of the overall network framework

[0051] See attached Figures 1 and 2 The overall structural design of the network model is based on a one-stage object detector (SSD). The VGG-16 feature extractor, pre-trained on the ImageNet dataset, is used as the feature extractor. It is divided into two parts: two domain-invariant feature extractors, E and E, which extract domain-invariant features F and F. Furthermore, E and E are domain-specific feature extractors that extract domain-specific features P and P.

[0052] Step S3: Multi-level feature decoupling

[0053] See Figures 1 and 2 In this paper, we use a domain-invariant feature extractor E and a domain-specific feature extractor E to extract features F and P, respectively. We also use orthogonal loss to increase the domain-invariant features F and domain-specific features P extracted from the source and target domains, respectively. We then use Euclidean distance to narrow the gap between the domain-invariant features F and F extracted from the two domains, completing the first step of decoupling. Based on this, we propose a re-decoupling module that feeds the features F and P obtained in the previous step into E and E for a second decoupling.

[0054] See Figure 3The features obtained by applying secondary decoupling to the domain discriminators D and D are applied. For domain-invariant features, a discriminator and gradient reversal layer are used. This creates an adversarial relationship between the discriminator and the extractor, which helps reduce domain-private information in domain-invariant features. For domain-specific features, a domain discriminator is used. If it can accurately identify the source of the feature, it indicates that it contains a large amount of domain-private information. The domain-invariant features F from the source domain and the domain-specific features P from the target domain are respectively input into the two feature extractors in the re-decoupling module. New domain-invariant features F and F and domain-specific features P and P are re-extracted, and prediction results are obtained based on F.

[0055] Step S4: Loop feature decoupling

[0056] See Figure 1 、 Figure 2 and Figure 4 In the present invention, cyclic decoupling is proposed on the basis of multi-level decoupling. In order to make the domain-invariant features contain as little domain-private information as possible, the domain-specific feature P0 is cyclically input into E1, and through iterative extraction (the number of iterations is set to 2 in this embodiment), it is made to contain almost no domain-public information. In addition, the present invention also develops a pyramid pooling attention mechanism, which can effectively improve the performance and make the extraction of domain-invariant features pay more attention to domain-private information. For each domain-invariant feature output by the cycle, the cosine similarity is used to reduce the similarity between it and F1, thereby assisting F1 in reducing domain-private information.

[0057] Step S5: Category feature decoupling

[0058] See Figure 1 , based on the previous module, the category decoupling is developed to make the decoupling more detailed. The present invention respectively performs the domain-specific features obtained by F1 and loop Extract category features and use cosine similarity to expand the features of the same category obtained by the two features. For the category features obtained by F1 of the two domains, the present invention uses JS divergence to narrow the gap of the same category and uses cosine similarity to expand the gap of different categories.

[0059] Step S6: Training and use of network model

[0060] The network was trained using the SGD optimizer with weight decay set to 0.0005, momentum set to 0.9, and an initial learning rate of l = 0.001. After 50 epochs, the learning rate was decayed to one-tenth of its original value. The batch size was set to 16. The initial parameters of the backbone network were those pre-trained on ImageNet. This example used an NVIDIA GeForce RTX 3090 graphics card for training, with 70 training epochs and a warmup strategy.

[0061] The loss function included in this embodiment includes four parts, namely detection loss, multi-level decoupling loss, cycle decoupling loss and category decoupling loss.

[0062] The detection loss function It is expressed using the following formulas (a) and (b):

[0063]

[0064]

[0065] Where N is the number of positive samples is an indicator parameter. When , it means that the i-th prior box matches the j-th ground truth, and the category of the ground truth is k. l is the prediction result, and g is the ground truth.

[0066] The multi-level decoupling loss function L mdm =L ort +L ed +L adv +L dis It consists of four parts, including L ort It is expressed by the following formula (c):

[0067]

[0068] Where A = [P0, P1, P2, P2], B = [F0, F1, F2, F1], N is the number of samples, c is the number of channels, is L2 regularization, |·| represents the absolute value, Represents matrix dot product.

[0069] The distance loss L ed It is represented by the following formula (d):

[0070]

[0071] Where Φ(a, b) represents the Euclidean distance.

[0072] The adversarial loss L adv =L loc +L glb , where L loc and L glb It is represented by the following formula (e) and formula (f):

[0073]

[0074]

[0075] Among them, D l and D g represents the local and global domain discriminators, W and H represent the width and height of F, L dis and L adv The formula is the same, except that F is replaced by P and the gradient reversal layer is not used.

[0076] The cycle decoupling loss L cdm As shown in the following formula (g):

[0077]

[0078] Where T represents the number of cycles, represents the domain-specific features obtained by the loop, and sim(a, b) represents the cosine similarity.

[0079] The class decoupling loss consists of three parts, expressed as: L cfdm =L js +γL cd +γL cos , where γ is a hyperparameter and is set to 0.1 in the experiment. js It can be expressed by the following formula (h) and formula (j):

[0080]

[0081]

[0082] Among them, L represents the number of features, N c Indicates the number of categories, C represents the category features, N e Indicates the number of elements in a and b. cd It can be expressed by the following formula (j):

[0083]

[0084] Among them, C p represents the categorical features obtained from domain-specific features. cfdm The last part of L cos It can be expressed by the following formula (k):

[0085]

[0086] So the final overall loss function L total It can be expressed by the following formula (1):

[0087] L total =L det +λ1L mdm +λ2L cdm +λ3L cfdm (1).

[0088] Among them, λ1=λ3=1, λ2=0.5.

[0089] This example uses the mAP evaluation metric with an Intersection over Union (IoU) threshold of 0.5 to quantitatively analyze the performance of the object detection model. AP represents the maximum precision corresponding to each recall rate for positive samples in a category. mAP is the average AP across all categories.

[0090] The intersection over union (IoU) is calculated by the following formula (m):

[0091]

[0092] Where S1 represents the area of ​​the intersection of the predicted box and the true value, S2 represents the area of ​​the union of the predicted box and the true value, and the mAP is calculated by the following (n) formula:

[0093]

[0094] The precision is calculated by the following formula (o):

[0095]

[0096] The recall rate (Recall) is calculated by the following formula (p):

[0097]

[0098] Among them, TP is the positive sample predicted by the model as the positive class; FP is the negative sample predicted by the model as the positive class; FN is the positive sample predicted by the model as the negative class.

[0099] The training is performed according to the above process, and the model with the best performance in the validation set is saved as the best model. The target domain image to be detected is sent to the best network model obtained by the above training. After being processed by the neural network, the detection result map can be obtained.

[0100] See Figure 5 , a qualitative comparison of the neural network constructed by the present invention with other detection networks shows that the present invention has the best detection effect on the target. The quantitative detection results are shown in Table 1 below:

[0101] Table 1 Comparison of test results

[0102]

[0103] As can be seen from the above table, the network constructed by the present invention is superior to other methods in multiple categories of AP and final mAP.

[0104] The above embodiments provide a detailed introduction to the domain-adaptive target detection method based on triple feature decoupling proposed by the present invention. Based on the main architecture of the present invention, researchers in this field may make changes in implementation strategies and application data, such as replacing different backbone networks to extract features. In summary, this specification is not intended to limit the present invention. Any modifications and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A domain-adaptive object detection method based on triple feature decoupling, characterized in that: This paper proposes a method for triple feature decoupling of the input image using a multi-level decoupling module, a cyclic decoupling module, and a category decoupling module. Furthermore, it uses auxiliary model learning to expand the distribution difference between domain-specific features and domain-invariant features. The trained model is then used to perform object detection in an unlabeled target domain. Specifically, the method includes the following steps: Step S1: Preprocessing of the dataset Select a labeled source domain dataset and an unlabeled target domain dataset, and perform image enhancement on the training data divided in the source domain and the target domain. The image enhancement specifically includes: random adjustment of image brightness, saturation and hue, random cropping, random horizontal flipping, random vertical flipping and random scaling; Step S2: Construction of the overall network framework A network model is built based on the one-stage object detector SSD, and the VGG16 network is used as the backbone feature extraction network, which serves as a domain-invariant feature extractor for extracting feature maps at six different levels and detecting objects of different sizes using feature maps at different levels. The network model contains two domain-specific feature extractors and two domain discriminators. The domain-specific feature extractor is divided into a domain-invariant feature extractor containing six convolutional layers and a domain-specific feature extractor containing three convolutional layers. The domain discriminator is divided into a global domain discriminator containing three convolutional layers and a local domain discriminator. Step S3: Multi-level feature decoupling Domain-invariant feature extractors and domain-specific feature extractors are used to extract domain-invariant features and domain-specific features, respectively. These features are then input into the re-decoupling module for secondary decoupling, and local domain discriminators and global domain discriminators are used to promote decoupling. Step S4: Loop feature decoupling The domain-specific features obtained by the re-decoupling module are input into the recurrent decoupling module for recurrent decoupling. The pyramid pooling attention mechanism is used to ensure that the extracted features contain as little domain-public information as possible, and in this process, the distance between the recurrently obtained domain-specific features and the domain-invariant features is increased. Step S5: Category feature decoupling Category features are extracted from domain-invariant features and domain-specific features respectively, and category feature decoupling is completed in two steps. The first step of category feature decoupling is to extract category features from domain-invariant features and domain-specific features respectively, narrowing the distribution of features of the same category obtained from different domain-invariant features and expanding the distribution of features of the same category obtained from the same domain-invariant features; the second step of category feature decoupling is to extract features of the same category from domain-invariant features and domain-specific features obtained from the same domain; Step S6: Training and use of network model Use the training dataset in step S1 to train the constructed network model, save the model parameters that work best on the target domain test set, and use the model to perform target detection on the target domain without labels.

2. The domain adaptive object detection method based on triple feature decoupling according to claim 1, characterized in that: The re-decoupling module in step S3 includes: a domain-invariant feature extractor including six convolutional layers and a domain-specific feature extractor including three convolutional layers. The specific re-decoupling process is: the domain-invariant feature F0 extracted from the source domain and the domain-specific feature P0 from the target domain are respectively input into the two feature extractors in the re-decoupling module, and new domain-invariant features F1 and F2 and domain-specific features P1 and P2 are re-extracted, and the prediction result is obtained based on F1.

3. The domain adaptive object detection method based on triple feature decoupling according to claim 1, characterized in that: The cyclic feature decoupling module in step S4 consists of a spatial pooling attention module, two convolutions and an upsampling layer. The input of the spatial pooling attention module is obtained by upsampling, convolution and adding the domain-specific features of the current level to the domain-specific features of the previous level and then convolution. Its output will be used as input again and input into the domain-invariant feature extractor again. Through a specified number of feedback loops, the distribution difference between the spatial pooling attention module and the domain-specific features will be gradually expanded in each loop.

4. The domain adaptive object detection method based on triple feature decoupling according to claim 1, characterized in that: The category feature decoupling module in step S5 includes: category feature decoupling within the domain and category feature alignment outside the domain. The category feature decoupling within the domain decouples the category features extracted from the domain-specific features and the domain-invariant features in each domain respectively; the category feature alignment outside the domain is used to align the source domain and the target domain invariant features to obtain category features of the same category, and expand the feature differences between different categories, so that the model can better perform classification.

5. The domain adaptive object detection method based on triple feature decoupling according to claim 1, characterized in that: The target detection using the saved model is to send the target domain image to be detected into the saved model, and obtain the detection result image through the processing of the neural network.

Images