Domain Adaptive DINO Model Object Detection Method Based on Feature Fusion and Alignment

Abstract

The present application discloses a domain adaptive DINO model object detection method based on feature fusion and alignment, which includes: sequentially inputting the features output by the backbone network into the first feature fusion module and the first domain discriminator to obtain the backbone network image-level adversarial training loss; inputting the features output by the encoder into the second domain discriminator to obtain the encoder pixel-level adversarial training loss; sequentially inputting the features output by the decoder into the second feature fusion module and the third domain discriminator to obtain the decoder object-level adversarial training loss; according to the backbone network image-level adversarial training loss, the encoder pixel-level adversarial training loss, the decoder object-level adversarial training loss, and the source domain detection loss, enabling the DINO model to learn cross-domain invariant features and complete cross-domain object detection of the target domain image data. The present application can reduce the data distribution shift and learn transferable feature representations, enabling the model to also have good detection performance in the target domain.

Description

Technical Field

[0001] This application relates to the technical field of cross-domain object detection, and particularly to an object detection method for a domain adaptive DINO model based on feature fusion and alignment. Background Art

[0002] Object detection is one of the fundamental tasks in the field of computer vision, showing great application potential in fields such as autonomous driving, video surveillance, and industrial quality inspection. Early detection methods based on Convolutional Neural Networks (CNNs) achieved efficient object localization and classification through the anchor box mechanism and region proposal networks. However, these methods rely on complex post-processing procedures, such as non-maximum suppression, and the model design is highly coupled, restricting flexibility and scalability in different scenarios. Detection Transformer (DETR) was proposed in 2020, which first introduced the Transformer architecture into the object detection task, modeling the object detection task as a set prediction problem and eliminating the dependence on anchor boxes and post-processing in previous methods in an end-to-end manner. DETR made bold innovations in architecture and process, but problems such as slow convergence speed and insufficient detection performance restricted its practical applications. In response to the defects of DETR, researchers have carried out a series of improvements, such as Conditional DETR and Deformable DETR. Among them, DETR with Improved deNOising anchor boxes (DINO model) significantly improved the convergence speed and detection accuracy through improvements such as contrastive denoising training, hybrid query selection, and gradient transfer optimization, which also made the DINO model the detector architecture with the best performance in the field of object detection.

[0003] The training of object detection models depends on large-scale high-quality labeled data, and the acquisition of such data is often time-consuming and laborious. Compared with detectors based on the CNN architecture, detectors based on the Transformer architecture, such as the DINO model, are more dependent on data. In addition, the generalization performance of detectors obtained through supervised training is also limited. When facing object data with domain shifts (such as different object shapes, complex backgrounds, and lighting changes), the detection performance of the model will drop sharply. To alleviate the limitation of domain shifts on the model generalization performance and avoid cumbersome data annotation work, more and more cross-domain learning methods have been proposed. These methods are based on domain adaptation techniques, using the knowledge of existing data and the correlation between existing data and target data to improve the generalization ability of the model to target data and reduce the dependence of model training on labeled samples and the demand for computing resources.

[0004] Object detection is more complex than image segmentation or classification tasks because a single input sample may correspond to multiple objects to be detected, and object localization and classification need to be completed simultaneously in a single model, which poses a huge challenge to cross-domain detection research but also inspires more ideas. Domain Adaptive Faster-RCNN is the first cross-domain detection method, which improves the model's cross-domain detection ability through image-level and instance-level adaptation. Based on this, cross-domain object detection research has become one of the hot issues in the field of object detection. However, existing cross-domain detection methods are based on early detector architectures such as Faster RCNN and YOLO, and the performance of their native detectors limits the cross-domain performance of the model. Moreover, cross-domain training methods based on the CNN architecture are difficult to be directly applied to the Transformer architecture. Summary of the Invention

[0005] Aiming at the problems of the limited performance of the native detector of the existing cross-domain detection model and the incompatibility between the existing cross-domain adaptation strategy and the DINO model detector, this application provides a domain adaptive DINO model object detection method based on feature fusion and alignment, which improves the feature representation ability through the method of differential feature fusion, and performs image-level, pixel-level and object-level feature alignment at different stages of the network to fully reduce the difference in feature distributions in different domains and improve the performance of the DINO model in cross-domain detection tasks.

[0006] This application discloses a domain adaptive DINO model object detection method based on feature fusion and alignment, which includes:

[0007] Step 1: Input the source domain data and the target domain data into the backbone network of the DINO model for feature extraction; the network model includes a DINO model, a first feature fusion module, a first domain discriminator, a second domain discriminator, a second feature fusion module and a third domain discriminator; the DINO model includes a backbone network, an encoder, a decoder and a feed-forward network connected in sequence; the source domain data includes source domain images and their corresponding target labels, and the target labels include the bounding box coordinates and category information of the objects; the target domain data only includes target domain images;

[0008] Step 2: Input the features output by the backbone network into the first feature fusion module and the first domain discriminator in sequence to obtain the backbone network image-level adversarial training loss;

[0009] Step 3: Input the features output by the encoder into the second domain discriminator to obtain the encoder pixel-level adversarial training loss;

[0010] Step 4: Input the features output by the decoder into the second feature fusion module and the third domain discriminator in sequence to obtain the decoder object-level adversarial training loss;

[0011] Step 5: Optimize the network model according to the backbone network image-level adversarial training loss, the encoder pixel-level adversarial training loss, the decoder target-level adversarial training loss, and the source domain detection loss, so that the network model learns cross-domain invariant features and enhances the cross-domain generalization ability of the network model;

[0012] Step 6: After the network model is trained, only load the DINO model weights to complete the cross-domain object detection of the target domain image data.

[0013] Further, the said Step 2 includes:

[0014] Step 21: The first feature fusion module obtains the image-level fusion feature according to the multi-scale features extracted by the backbone network;

[0015] Step 22: The first domain discriminator obtains the image-level domain prediction result according to the image-level fusion feature;

[0016] Step 23: Obtain the backbone network image-level adversarial training loss according to the image-level domain prediction result.

[0017] Further, the said Step 21 includes:

[0018] The different scale features in the multi-scale features extracted by the backbone network are learned through the convolutional layer in the first feature fusion module, then the size of the feature map is unified through the global average pooling layer, and finally the image-level fusion feature is obtained through the Concat operation layer; the first feature fusion module consists of a convolutional layer, a global average pooling layer and a fully connected layer;

[0019] The said Step 22 includes:

[0020] The first feature fusion module sequentially inputs the image-level fusion feature into the gradient reversal layer and the fully connected layer in the first domain discriminator, and outputs the image-level domain prediction result; the first domain discriminator consists of a gradient reversal layer and a fully connected layer.

[0021] Further, the said Step 23 includes:

[0022] Obtain the backbone network image-level adversarial training loss through the following formula:

[0023] (1)

[0024] where is the backbone network image-level adversarial training loss, is the image-level domain prediction result of the sample and is the domain label of the sample The domain label includes the source domain label and the target domain label, and their values are 0 and 1 respectively.

[0025] Furthermore, step 3 includes:

[0026] Step 31: The encoder models the global context relationship through the self-attention mechanism. Its input is obtained by adding the positional embedding to the one-dimensional sequence feature transformed from the multi-scale features output by the encoder. The input of each subsequent layer of the encoder is the output of the previous layer;

[0027] Step 32: The output of the encoder is sequentially input into the gradient reversal layer and the fully connected layer in the second domain discriminator to obtain the pixel-level domain prediction result , is the output of the encoder, is the second domain discriminator; the structure of the second domain discriminator is the same as that of the first domain discriminator, and it consists of a gradient reversal layer and a fully connected layer;

[0028] Step 33: According to the pixel-level domain prediction result, obtain the pixel-level adversarial training loss of the encoder.

[0029] Furthermore, step 33 includes:

[0030] The pixel-level adversarial training loss of the encoder is obtained through the following formula:

[0031] (2)

[0032] where, is the pixel-level adversarial training loss of the encoder, represents the domain prediction result corresponding to the j-th pixel in the multi-scale feature map of sample , is the sample 's domain label. The domain label includes the source domain label and the target domain label, and their values are 0 and 1 respectively.

[0033] Furthermore, step 4 includes:

[0034] Step 41: The output of the encoder is input into the decoder to optimize the query anchor box and the classification result. The input of each subsequent layer of the decoder is the output of the previous layer, and the output of each layer is denoted as where , and , is the number of target queries, is the number of layers of the decoder, is the total number of layers of the decoder, C is the number of channels of the feature map, or the number of channels of the hidden layer, that is, the feature dimension of the sequence feature;

[0035] Step 42: The feature After learning through the fully connected layer and the activation function, the target-level fusion feature is obtained through the Concat operation layer. ; The second feature fusion module consists of a fully connected layer and a Concat operation layer;

[0036] Step 43: Input the target-level fusion feature sequentially into the gradient reversal layer and the fully connected layer in the third domain discriminator to obtain the domain prediction result , which is the third domain discriminator; The structure of the third domain discriminator is the same as that of the first domain discriminator and consists of a gradient reversal layer and a fully connected layer;

[0037] Step 44: Obtain the decoder target-level adversarial training loss according to the domain prediction result.

[0038] Furthermore, the step 44 includes:

[0039] The decoder target-level adversarial training loss is obtained through the following formula:

[0040] (3)

[0041] where, is the decoder target-level adversarial training loss, represents the domain prediction result corresponding to the j-th target query of the sample , is the domain label of the sample , and the domain label includes the source domain label and the target domain label, and their values are 0 and 1 respectively, or their values are 1 and 0 respectively.

[0042] Furthermore, the step 5 includes:

[0043] Step 51: Obtain the adversarial training loss of the network model according to the backbone network image-level adversarial training loss, the encoder pixel-level adversarial training loss, and the decoder target-level adversarial training loss;

[0044] Step 52: Obtain the optimal network model according to the adversarial training loss of the network model and the source domain detection loss.

[0045] Furthermore, the step 51 includes:

[0046] The adversarial training loss of the network model is obtained through the following formula:

[0047] (4)

[0048] where, is the adversarial training loss of the network model, is the backbone network image-level adversarial training loss, is the encoder pixel-level adversarial training loss, is the decoder target-level adversarial training loss;

[0049] The step 52 includes:

[0050] Complete the learning by jointly optimizing the adversarial training loss and the source domain detection loss through the following formula:

[0051] (5)

[0052] where, represents the set of the first domain discriminator , the second domain discriminator and the third domain discriminator ; represents the remaining part of the network model except ; is the source domain detection loss, is the trade-off parameter for balancing the adversarial training loss and the source domain detection loss; the remaining part of the network includes the backbone network, the encoder, the decoder, the feed-forward network, the first feature fusion module, and the second feature fusion module.

[0053] Due to the above technical solutions, the present application has the following advantages:

[0054] (1) Design a feature fusion module and a global feature alignment strategy in the backbone network part of the DINO model, fuse the multi-scale features of the backbone network, and perform global coarse-grained alignment on the fused features to reduce the influence of factors such as illumination conditions and complex backgrounds, and improve the overall adaptability of the model to the target scene.

[0055] (2) Propose a local feature alignment strategy in the encoder part of the DINO model, perform local fine-grained alignment on the encoder features, alleviate the influence of factors such as scene layout on the cross-domain performance of the model, and reduce the pixel-level feature distribution difference.

[0056] (3) Propose a feature fusion module and a target-level alignment strategy in the decoder part of the DINO model, fuse the hierarchical features of the decoder, and perform target-level feature alignment on the fused features to avoid the negative transfer phenomenon and ensure the cross-domain detection generalization and discrimination ability of the model.

[0057] (4) After the DINO model is trained, when performing the object detection task, only the weights of the DINO model need to be loaded, which can improve the performance of the model in the cross-domain object detection task without increasing the computational amount. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments described in the embodiments of the present application. For those of ordinary skill in the art, other accompanying drawings can also be obtained based on these drawings.

[0059] Figure 1 It is a schematic structural diagram corresponding to a domain adaptive DINO model object detection method based on feature fusion and alignment in the embodiments of the present application;

[0060] Figure 2 It is a schematic structural diagram of the first feature fusion module in the embodiments of the present application;

[0061] Figure 3 It is a schematic structural diagram of the first domain discriminator, or the second domain discriminator, or the third domain discriminator in the embodiments of the present application;

[0062] Figure 4 It is a schematic structural diagram of the second feature fusion module in the embodiments of the present application. Detailed implementation manners

[0063] The present application will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art shall fall within the scope of protection of the embodiments of the present application.

[0064] See Figure 1 , the present application provides an embodiment of a domain adaptive DINO model object detection method based on feature fusion and alignment. Based on the currently best-performing DINO model, it improves the feature representation ability through the method of differential feature fusion, and performs image-level, pixel-level, and target-level feature alignment at different stages of the network. The feature fusion and alignment strategies only play a role in the model training process. The trained model only needs to load the DINO model weights to perform cross-domain inference, which can improve the cross-domain detection performance of the model without increasing additional inference burden.

[0065] For the convenience of description, relevant knowledge of cross-domain object detection is first introduced: the source domain with labeled information , where represents the number of source domain samples, represents the source domain images, is the corresponding target label, including the bounding box coordinates and class information of the target. The target domain without labeled information , where represents the number of target domain samples, Denote the target domain image. The label spaces of the source domain and the target domain are consistent, but their data distributions are different. The purpose of this application is to train a network model to reduce the data distribution shift and learn transferable feature representations so that the model also has good detection performance in the target domain.

[0066] Secondly, briefly introduce the DINO model detection process: The DINO model is an end-to-end object detection model based on the DETR architecture. As Figure 1 shown, specifically, it includes the following parts: (1) Backbone network: Generally, CNN or vision Transformer is used to extract multi-scale feature representations of the input image , , where is the number of feature scales, and there is . After that, through operations such as flattening, embedding, and splicing, the multi-scale features are transformed into one-dimensional sequence features , where H and W represent the height and width of the image respectively; (2) Transformer encoder: Model the global context relationship through the self-attention mechanism. Its input is obtained by adding positional embedding and other information to the above one-dimensional sequence feature , denoted as . The input of each subsequent layer of the encoder is the output of the previous layer. Denote the final output of the encoder as ; (3) Transformer decoder: Continuously optimize the query anchor boxes and classification results. The input of each subsequent layer of the decoder is also the output of the previous layer. Denote the output of each layer as , where , is the total number of layers of the encoder, and there is , is the number of target queries; (4) The feed-forward network predicts the target class probabilities and bounding box positions through the outputs of all decoder layers; (5) Model optimization: The DINO model uses the cross-entropy loss to predict the target class and uses the L1 loss and GIOU loss for bounding box regression. Combined, it is the source domain detection loss during the training of the DINO model, denoted as .

[0067] This application takes the DINO model as the main body, and adds a backbone network feature fusion module (the first feature fusion module) and a global feature alignment strategy (corresponding to the first domain discriminator). The feature fusion module is designed to ensure the model's detection ability for various targets in the target domain data, and further design a global feature alignment strategy to reduce the difference in image-level feature distributions, suppress domain shifts caused by factors such as image style, lighting conditions, and complex backgrounds, and improve the overall adaptability of the model to the target scene; an encoder local feature alignment strategy is added. The local feature alignment strategy (corresponding to the second domain discriminator) is designed to perform local fine-grained alignment on the encoder features, alleviate the impact of factors such as scene layout on the model's cross-domain performance, and reduce the difference in pixel-level feature distributions; a decoder feature fusion module (the second feature fusion module) and a target-level feature alignment strategy (corresponding to the third domain discriminator) are added. The decoder feature fusion module is designed to learn the rich target semantic information of sequence features, refine features to avoid negative transfer, and further design a target-level alignment strategy to ensure the cross-domain detection generalization and discrimination ability of the model.

[0068] The embodiments of this application include the following technical solutions:

[0069] Step 1: Input the source domain data and the target domain data into the backbone network of the DINO model for feature extraction; the network model includes a DINO model, a first feature fusion module, a first domain discriminator, a second domain discriminator, a second feature fusion module, and a third domain discriminator; the DINO model includes a backbone network, an encoder, a decoder, and a feed-forward network connected in sequence; the source domain data includes source domain images and their corresponding target labels, and the target labels include the bounding box coordinates and category information of the targets; the target domain data only includes target domain images;

[0070] Step 2: Input the features output by the backbone network into the first feature fusion module and the first domain discriminator in sequence to obtain the backbone network image-level adversarial training loss;

[0071] Step 3: Input the features output by the encoder into the second domain discriminator to obtain the encoder pixel-level adversarial training loss;

[0072] Step 4: Input the features output by the decoder into the second feature fusion module and the third domain discriminator in sequence to obtain the decoder target-level adversarial training loss;

[0073] Step 5: Optimize the network model according to the backbone network image-level adversarial training loss, the encoder pixel-level adversarial training loss, the decoder target-level adversarial training loss, and the source domain detection loss, so that the network model learns cross-domain invariant features and enhances the cross-domain generalization ability of the network model;

[0074] Step 6: After the network model is trained, only the weights of the DINO model need to be loaded to complete the cross-domain object detection of the target domain image data.

[0075] This application is applicable to the situation where the target domain data lacks annotation information or there is a distribution shift between the target domain data and the training data. It can utilize the existing data knowledge and the correlation between the existing data and the target data to improve the discriminative and generalization abilities of the model for the target data, and reduce the dependence on labeled samples and the demand for computing resources in model training.

[0076] Optionally, Step 2 includes:

[0077] Step 21: The first feature fusion module obtains image-level fusion features based on the multi-scale features extracted by the backbone network;

[0078] Step 22: The first domain discriminator obtains an image-level domain prediction result based on the image-level fusion features;

[0079] Step 23: Based on the image-level domain prediction result, the image-level adversarial training loss of the backbone network is obtained.

[0080] Optionally, Step 21 includes:

[0081] The different-scale features in the multi-scale features extracted by the backbone network are learned through the convolutional layer in the first feature fusion module, then the size of the feature map is unified through the global average pooling layer, and finally the image-level fusion features are obtained through the Concat operation layer; the first feature fusion module consists of a convolutional layer, a global average pooling layer, and a fully connected layer;

[0082] Specifically, the feature quality of the backbone network is the cornerstone of the entire model's detection performance. It can effectively guide the learning process of the Transformer encoder and decoder. Learning domain-invariant backbone network features is a basic element of the model's generalization ability. Considering that the DINO model adopts a multi-scale feature output design in the backbone network, this application designs a feature fusion module and a global feature alignment strategy to ensure the model's detection ability for various targets in the target domain data. The structure of the feature fusion module of the backbone network is as Figure 2 shown. Fusing the multi-scale features of the backbone network can increase the representational ability of the features and extract image-level fusion features to adapt to the subsequent global feature alignment strategy. The feature fusion module of the backbone network is overall composed of a convolutional layer, global average pooling, and Concat operation. Different-scale features are learned through the shared convolutional layer and then the size of the feature map is unified through the global average pooling operation , and finally the image-level fusion features are obtained through the Concat operation Due to the reduction in the size and quantity of the feature maps, this fusion module also effectively reduces the computational load in the feature alignment process.

[0083] Step 22 includes:

[0084] The first feature fusion module sequentially inputs the image-level fusion features into the gradient reversal layer and the fully connected layer in the first domain discriminator, and outputs the image-level domain prediction result; the first domain discriminator consists of a gradient reversal layer and a fully connected layer.

[0085] Specifically, the feature alignment strategy refers to reducing the difference in feature distributions through adversarial training to improve the generalization of the model. In cross-domain adversarial training, the network usually includes a feature extractor (referring to the DINO model) and a domain discriminator. The domain discriminator is used to predict whether the input features belong to the source domain or the target domain, and the feature extractor attempts to confuse the judgment of the domain discriminator and perform supervised learning on the source domain data. The model implicitly learns domain-invariant features during the adversarial process between the two. The structure of the domain discriminator is as Figure 3 shown, consisting of a gradient reversal layer and a fully connected layer. The domain discriminator of the backbone network is denoted as , with the input being the backbone network fusion feature map , and the output being the image-level domain prediction result .

[0086] Optionally, step 23 includes:

[0087] Obtain the image-level adversarial training loss of the backbone network through the following formula:

[0088] (1)

[0089] where is the image-level adversarial training loss of the backbone network, is the image-level domain prediction result of sample , is the domain label of sample . The domain label includes the source domain label and the target domain label, and their values are 0 and 1 respectively. Optionally, let the source domain label be 0 and the target domain label be 1, or let the source domain label be 1 and the target domain label be 0.

[0090] The above optimizations can effectively reduce the differences in the feature distributions of the backbone network, but still cannot guarantee the alignment of local distributions and the transferability of Transformer serialized features. Therefore, aligning Transformer serialized features is necessary for improving the cross-domain performance of the model. The Transformer encoder is a bridge connecting the backbone network and the decoder. To make up for the deficiencies of the global alignment strategy of the backbone network, this application further designs an encoder local feature alignment strategy to mitigate the impact of factors such as scene layout on the cross-domain performance of the model and reduce the differences in pixel-level feature distributions. Optionally, step 3 includes:

[0091] Step 31: The encoder models the global context relationship through the self-attention mechanism. Its input is obtained by adding the positional embedding to the one-dimensional sequence features transformed from the multi-scale features output by the encoder. The input of each subsequent layer of the encoder is the output of the previous layer;

[0092] Step 32: The structure of the second domain discriminator used for encoder local feature alignment is also as Figure 3 shown. The output of the encoder is sequentially input into the gradient reversal layer and the fully connected layer in the second domain discriminator to obtain the pixel-level domain prediction result , is the output of the encoder, is the second domain discriminator; the structure of the second domain discriminator is the same as that of the first domain discriminator, and it consists of a gradient reversal layer and a fully connected layer;

[0093] Step 33: According to the pixel-level domain prediction result, the encoder pixel-level adversarial training loss is obtained.

[0094] Optionally, step 33 includes:

[0095] The encoder pixel-level adversarial training loss is obtained through the following formula:

[0096] (2)

[0097] where, is the encoder pixel-level adversarial training loss, represents the domain prediction result corresponding to the j-th pixel in the multi-scale feature map of sample , is the domain label of sample . The domain label includes the source domain label and the target domain label, and their values are 0 and 1 respectively, or their values are 1 and 0 respectively.

[0098] The prediction result of the DINO model depends on the output of each decoder layer of the Transformer. Due to the lack of supervision information in the target domain, the decoder features will inevitably be biased towards the source domain data, thereby damaging the discrimination and generalization performance of the model on the target data. Therefore, it is necessary to perform cross-domain modeling on the decoder feature distribution. To avoid the negative transfer phenomenon and ensure the domain invariance of the model output information, this application designs a decoder feature fusion module and a target-level feature alignment strategy.

[0099] Optionally, step 4 includes:

[0100] Step 41: Input the output of the encoder into the decoder to optimize the query anchor box and classification result. The input of each subsequent layer of the decoder is the output of the previous layer, and the output of each layer is denoted as , where , and , is the number of target queries, is the number of decoder layers, is the total number of decoder layers, C is the number of feature map channels, or the number of hidden layer channels, that is, the feature dimension of the sequence feature;

[0101] Step 42: The feature output by the decoder is learned through a fully connected layer and an activation function, and then the target-level fusion feature is obtained through the Concat operation layer; The second feature fusion module consists of a fully connected layer and a Concat operation layer; The feature fusion module (second feature fusion module) of the Transformer decoder is as Figure 4 shown. By fusing the sequence features of the decoder stack, it learns rich target semantic information, while avoiding complex feature alignment and alleviating the negative transfer phenomenon;

[0102] Step 43: Perform target-level feature alignment in the decoder part to reduce the instance-level distribution difference and suppress the influence of factors such as target shape, texture, and color. Compared with the previous two alignment strategies, the target-level alignment strategy not only improves the generalization of the model but also ensures its discriminability. The structure of the domain discriminator (third domain discriminator) in the decoder part is also as Figure 3 shown; The target-level fusion feature is sequentially input into the gradient reversal layer and the fully connected layer in the third domain discriminator to obtain the domain prediction result , is the third domain discriminator; The structure of the third domain discriminator is the same as that of the first domain discriminator, consisting of a gradient reversal layer and a fully connected layer;

[0103] Step 44: Obtain the decoder target-level adversarial training loss according to the domain prediction result.

[0104] Optionally, step 44 includes:

[0105] Obtain the decoder target-level adversarial training loss through the following formula:

[0106] (3)

[0107] where is the decoder target-level adversarial training loss, represents the domain prediction result corresponding to the j-th target query of sample , is the domain label of sample . The domain label includes the source domain label and the target domain label, and their values are 0 and 1 respectively, or their values are 1 and 0 respectively.

[0108] Optionally, step 5 includes:

[0109] Step 51: Obtain the adversarial training loss of the network model according to the backbone network image-level adversarial training loss, the encoder pixel-level adversarial training loss, and the decoder target-level adversarial training loss;

[0110] Step 52: Obtain the optimal network model according to the adversarial training loss of the network model and the source domain detection loss.

[0111] Optionally, step 51 includes:

[0112] Obtain the adversarial training loss of the network model through the following formula, which is composed of the backbone network image-level adversarial training loss, the encoder pixel-level adversarial training loss, and the decoder target-level adversarial training loss, and can be expressed as:

[0113] (4)

[0114] where is the adversarial training loss of the network model, is the backbone network image-level adversarial training loss, is the encoder pixel-level adversarial training loss, is the decoder target-level adversarial training loss;

[0115] Step 52 includes:

[0116] The specific training process of the network is as follows: the source domain and target domain data are input into the DINO model network together, and feature extraction is performed through the backbone network, encoder and decoder. In the above three feature extraction stages, the backbone network image-level adversarial training loss, encoder pixel-level adversarial training loss and decoder target-level adversarial training loss can be obtained according to formulas (1), (2) and (3) respectively, and the overall adversarial training loss can be obtained according to formula (4). The source domain samples have labeled information, and the source domain detection loss can be obtained. .

[0117] Finally, learning is completed by jointly optimizing the adversarial training loss and the source domain detection loss, as shown in the following formula:

[0118] (5)

[0119] in, Represents the first domain discriminator , Second Domain Discriminator and the third domain discriminator A collection of Indicates that the network model The rest of the network, is the source domain detection loss, is a trade-off parameter used to balance adversarial training loss and source domain detection loss. The remaining network components include the backbone network, encoder, decoder, feedforward network, first feature fusion module, and second feature fusion module. Through this learning process, the network model can fully reduce the differences in data distribution across different domains, improving the model's generalization and discriminability in cross-domain detection tasks.

[0120] For ease of understanding, this application provides a more specific embodiment:

[0121] refer to Figure 1 , this application proposes an embodiment of a domain-adaptive DINO model target detection method based on feature fusion and alignment, including:

[0122] (1) Obtaining source domain and target domain data: Source domain data needs to contain annotation information, while target domain data does not need annotation information.

[0123] (2) Model and environment: Figure 1 As shown, this application is based on the DINO model, which includes a backbone network, encoder, decoder, and feature fusion and alignment strategies. In this example, the backbone network uses ResNet-50, and both the encoder and decoder use a six-layer stack. In addition, this example is implemented using the Docker + VScode + PyTorch coding and runtime environment, and the model is trained on a single NVIDIA A800 GPU.

[0124] (3) Network training process: The data of the source domain and the target domain are input into the model designed in this application together. Feature extraction is carried out through the backbone network, the encoder and the decoder, and the loss is calculated according to the formulas (1), (2), (3) and (4) in the above embodiments. By optimizing the formula (5), the method proposed in this application can fully reduce the difference in data distribution between different domains and improve the generalization and discriminability of the model in cross-domain detection tasks.

[0125] (4) Inference and detection stage: The inference process of the network model is exactly the same as that of the DINO model, that is, the target domain image data is input into the network model for detection. After training, the network model only needs to load the DINO model weights. Therefore, the method proposed in this application will not bring additional computational overhead and memory occupancy when used.

[0126] (5) Detection task: In this example, a target detection task under changing weather conditions is constructed based on the publicly available street scene datasets Cityscapes (clear) and FoggyCityscapes (foggy). The quantitative results of cross-domain target detection for the method proposed in this application are analyzed. As shown in Table 1, there are Domain Adaptive Faster RCNN (DAF), Strong-Weak Distribution Alignment (SWDA), Graph-induced Prototype Alignment (GPA), Spatial Alignment YOLO (SA-YOLO), Sequence Feature Alignment (SFA), Domain Adaptive DETR (DA-DETR) and the method proposed in this application. It can be seen that this application has achieved the best performance, with a large improvement in the detection accuracy of almost all categories. The overall mAP is 57.8, and there is also a good performance improvement in difficult example categories such as trucks, buses, trains, motorcycles, and bicycles, fully demonstrating that the proposed method has excellent cross-domain detection capabilities.

[0127] Table 1 Cityscapes Quantitative results of cross-domain detection for Foggy Cityscapes

[0128]

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application rather than limit it. Although the present application has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications or equivalent substitutions can still be made to the specific implementation manners of the present application, and any modification or equivalent substitution that does not depart from the spirit and scope of the present application should be covered within the scope of protection of the claims of the present application.

Claims

1. A target detection method for a domain adaptive DINO model based on feature fusion and alignment, characterized in that, Including: Step 1: Input the source domain data and the target domain data into the backbone network of the DINO model for feature extraction; the network model includes a DINO model, a first feature fusion module, a first domain discriminator, a second domain discriminator, a second feature fusion module, and a third domain discriminator; the DINO model includes a backbone network, an encoder, a decoder, and a feed-forward network connected in sequence; the source domain data includes source domain images and their corresponding target labels, and the target labels include the bounding box coordinates and class information of the target; the target domain data only includes target domain images. Step 2: Input the features output by the backbone network into the first feature fusion module and the first domain discriminator in sequence to obtain the backbone network image-level adversarial training loss. Step 3: Input the features output by the encoder into the second domain discriminator to obtain the encoder pixel-level adversarial training loss. Step 4: Input the features output by the decoder into the second feature fusion module and the third domain discriminator in sequence to obtain the decoder target-level adversarial training loss. Step 5: Optimize the network model according to the backbone network image-level adversarial training loss, the encoder pixel-level adversarial training loss, the decoder target-level adversarial training loss, and the source domain detection loss, so that the network model learns cross-domain invariant features and enhances the cross-domain generalization ability of the network model. Step 6: After the network model is trained, only load the DINO model weights to complete the cross-domain object detection of the target domain image data.

2. The method according to claim 1, wherein The said Step 2 includes: Step 21: The first feature fusion module obtains image-level fusion features according to the multi-scale features extracted by the backbone network. Step 22: The first domain discriminator obtains an image-level domain prediction result according to the image-level fusion features. Step 23: Obtain the backbone network image-level adversarial training loss according to the image-level domain prediction result.

3. The method according to claim 2, wherein The said Step 21 includes: The different scale features in the multi-scale features extracted by the backbone network are learned through the convolutional layers in the first feature fusion module, then the size of the feature map is unified through the global average pooling layer, and finally the image-level fusion features are obtained through the Concat operation layer; the first feature fusion module consists of convolutional layers, a global average pooling layer, and a fully connected layer. The said Step 22 includes: The first feature fusion module inputs the image-level fusion features into the gradient reversal layer and the fully connected layer in the first domain discriminator in sequence, and outputs the image-level domain prediction result; the first domain discriminator consists of a gradient reversal layer and a fully connected layer.

4. The method according to claim 2, wherein The said Step 23 includes: Obtain the backbone network image-level adversarial training loss through the following formula: (1) Among them, is the backbone network image-level adversarial training loss, is the image-level domain prediction result of the sample , is the domain label of the sample The domain label includes the source domain label and the target domain label, and their values are 0 and 1 respectively.

5. The method according to claim 1, wherein The said Step 3 includes: Step 31: The encoder models the global context relationship through the self-attention mechanism, and its input is obtained by adding the one-dimensional sequence features converted from the multi-scale features output by the encoder plus the position embedding, and the input of each subsequent layer of the encoder is the output of the previous layer. Step 32: Input the output of the encoder into the gradient reversal layer and the fully connected layer in the second domain discriminator in sequence to obtain the pixel-level domain prediction result , is the output of the encoder, is the second domain discriminator; the structure of the second domain discriminator is the same as that of the first domain discriminator and consists of a gradient reversal layer and a fully connected layer; Step 33: Obtain the encoder pixel-level adversarial training loss according to the pixel-level domain prediction result.

6. The method according to claim 5, characterized in that The said Step 33 includes: Obtain the encoder pixel-level adversarial training loss through the following formula: (2) Among them, is the encoder pixel-level adversarial training loss, represents the prediction result corresponding to the j-th pixel in the multi-scale feature map of the sample is the domain label of the sample The domain label includes the source domain label and the target domain label, and their values are 0 and 1 respectively.

7. The method according to claim 1, characterized in that, The said Step 4 includes: Step 41: Input the output of the encoder into the decoder to optimize the query anchor boxes and classification results. The input of each subsequent layer of the decoder is the output of the previous layer. Denote the output of each layer as , where , and , is the number of target queries, is the number of layers of the decoder, is the total number of layers of the decoder. C is the number of feature map channels or the number of hidden layer channels, that is, the feature dimension of the sequence features; Step 42: Features output by the decoder Undergo learning through a fully connected layer and an activation function, and then obtain the target-level fusion features through a Concat operation layer ; The second feature fusion module consists of a fully connected layer and a Concat operation layer; Step 43: Input the target-level fusion features into the gradient reversal layer and the fully connected layer in the third domain discriminator in sequence to obtain the domain prediction result , where the third domain discriminator is; the structure of the third domain discriminator is the same as that of the first domain discriminator and consists of a gradient reversal layer and a fully connected layer; Step 44: Obtain the decoder target-level adversarial training loss according to the domain prediction result.

8. The method according to claim 7, wherein The said Step 44 includes: The decoder target-level adversarial training loss is obtained through the following formula: (3) Among them, is the decoder target-level adversarial training loss, represents the domain prediction result corresponding to the j-th target query of the sample , is the domain label of the sample The domain label includes the source domain label and the target domain label, and their values are 0 and 1 respectively, or their values are 1 and 0 respectively.

9. The method according to claim 1, wherein The said step 5 includes: Step 51: Obtain the adversarial training loss of the network model according to the backbone network image-level adversarial training loss, the encoder pixel-level adversarial training loss, and the decoder target-level adversarial training loss; Step 52: Obtain the optimal network model according to the adversarial training loss of the network model and the source domain detection loss.

10. The method according to claim 9, characterized in that, The said step 51 includes: Obtain the adversarial training loss of the network model through the following formula: (4) Among them, is the adversarial training loss of the network model, is the image-level adversarial training loss of the backbone network, is the pixel-level adversarial training loss of the encoder, is the target-level adversarial training loss of the decoder; The said step 52 includes: Complete the learning by jointly optimizing the adversarial training loss and the source domain detection loss through the following formula (5) Among them, represents the set of the first domain discriminator , the second domain discriminator and the third domain discriminator ; represents the remaining part of the network model except ; is the source domain detection loss is the trade-off parameter used to balance the adversarial training loss and the source domain detection loss; the remaining part of the network includes a backbone network, an encoder, a decoder, a feed-forward network, a first feature fusion module, and a second feature fusion module.

Images