Domain adaptive target detection method based on multi-scale feature fusion
Patent Information
- Application Number
- CN202410911678.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-08
AI Technical Summary
There are problems in the existing domain adaptive object detection algorithms with inconsistent inter-domain feature scales and forced alignment background noise, resulting in degradation of detection accuracy and generalization performance.
A multi-scale feature fusion module is used to fuse the feature information of deep and shallow networks, design the background suppression module to reduce the impact of background noise, and introduce domain classification and consistency regularizer to generate domain invariant features.
It effectively solves the problem of inconsistent feature scales between domains, reduces background noise interference, and improves the detection accuracy and generalization performance of the target detection model in complex scenarios.
Smart Images

Figure CN118918310A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a domain adaptive target detection method based on multi-scale feature fusion. Background Art
[0002] Object detection plays a very important role in the field of computer vision. It uses trained detection models to accurately predict and identify the category and location of objects in a given complex scene image. With the development of life, the demand for object detection in real life is increasing, and it has important application value in many areas of life. With the help of object detection algorithms, computers can recognize and perceive the world, thereby realizing more complex functions. With the continuous leap of data computing power, the learning ability of neural networks is constantly enhanced, and the detection performance of object detection algorithms based on various neural networks is getting better and better, but there are still some problems. When facing a new detection task, tens of thousands of annotated data sets are often required for model training. However, due to various factors, a large number of data sets are difficult to obtain. Even if a large amount of training data can be obtained, the training scenarios and application scenarios of object detection often have significant differences and do not belong to the same distribution. Under such conditions, the detection accuracy and generalization performance of the object detection model will drop sharply. To solve the above problems, researchers have proposed a domain adaptation method. This allows the object detection model to maintain good detection performance even when no annotated data is required in the target domain. However, in the current domain adaptive object detection algorithm, there are still problems such as inconsistent feature scales between domains and forced alignment of foreground target features and background noise. Summary of the invention
[0003] The purpose of the present invention is to use the method of reducing the difference between the source domain and the target domain, that is, by realizing the alignment of features between the source domain and the target domain, to propose a domain adaptive target detection algorithm MFDA based on multi-scale feature fusion. The multi-scale feature fusion method is adopted to fuse the feature information of the deep network and the shallow network to solve the problem of inconsistent feature scales between domains. The background suppression method is adopted to focus the network's attention on the foreground target features, thereby reducing the influence of background noise, and a domain classification and consistency regularizer is designed to help the network generate domain invariant feature information, thereby improving the detection rate of the target detection model in complex scenes.
[0004] The technical solution adopted by the present invention is:
[0005] Step 1: Design a multi-scale feature fusion module based on the benchmark model to align and fuse the generated feature maps with different scales;
[0006] Step 2: In order to reduce the impact of background feature noise on the accuracy of feature alignment, a background suppression module is added to process the feature map fused in step 1;
[0007] Step 3: Input the updated feature map into two different domain classifiers for processing. Considering that the same feature area may be classified separately in different domain classifiers, the algorithm uses the domain classification consistency regularizer;
[0008] Step 4: In order to verify the effectiveness of the proposed method, two groups of experiments are set up in this section, namely, two groups of experiments with different backgrounds (the source domain is the CityScape dataset under sunny scenes, and the target domain is the Foggy CityScape dataset under foggy scenes) and different image styles and image qualities (the source domains are the Simil10k dataset of game scenes and the KITTI dataset with lower image quality; the target domains are all CityScape datasets of real scenes).
[0009] Compared with the prior art, the present invention has the following beneficial effects:
[0010] (1) Improve the existing problems of DA algorithm, including the single alignment scale and the forced alignment of background features while aligning foreground features;
[0011] (2) A multi-scale feature fusion module is proposed to effectively solve the problem of inconsistent feature scales between domains and a single alignment scale. To address the problem of forced alignment of background features, a background suppression module is proposed to reduce the interference of background features on the alignment between the source domain and the target domain, allowing the algorithm to focus more on the alignment of foreground target features. At the same time, a domain classification consistency regularizer is proposed to help the network generate domain-invariant features.
[0012] (3) The proposed method is compared with mainstream algorithms and trained and tested on commonly used public datasets. The results show that it has significant advantages in domain adaptive object detection, which verifies the effectiveness of the MFDA algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 For: Domain adaptive target detection algorithm structure diagram based on multi-scale feature fusion
[0014] Figure 2 For: Domain classifier D1 network structure diagram
[0015] Figure 3 Comparison table of MFDA and mainstream algorithm mAP indicators under different background environments
[0016] Figure 4Figure 2: Detection effect of each algorithm (a) Faster; (b) DA; (c) MFDA (VGG16)
[0017] Figure 5 For: Car AP in different image styles and quality scenarios when the backbone network uses VGG16
[0018] Figure 6 For: Comparison of prediction boxes of different algorithms DETAILED DESCRIPTION
[0019] In order to solve the problems of inconsistent feature scales between domains and forced alignment of background features and foreground features in domain adaptive object detection algorithms, this paper selects FasterR-CNN as the original model and proposes a domain adaptive algorithm MFDA based on multi-scale feature fusion. The algorithm framework diagram is shown in the attached figure. Figure 1 shown.
[0020] The present invention is further described below in conjunction with the accompanying drawings. The specific method for implementing the present invention comprises the following steps:
[0021] Step 1: Multi-scale feature cross-layer fusion module
[0022] In the VGG16 network model, the second to fifth feature maps of the output are m2, m3, m4 and m5 respectively. The size of the feature maps output by adjacent convolution blocks will become half of the original size. In order to achieve effective fusion of feature maps at different levels, it is necessary to first upsample the fused feature maps using bilinear interpolation. Take the fused feature map m2 and feature map m4 as an example, where the scale ratio of feature map m2 and feature map m4 is set to 4:1. Therefore, the feature map is first upsampled four times, and then the number of feature channels is adjusted by a 1×1 convolution kernel to match the number of channels of feature map m4. Next, the two feature maps are added pixel by pixel and fused. After the fusion is completed, a 3×3 convolution kernel is used to further convolve each fusion result to eliminate the aliasing effect caused by feature fusion and upsampling. At the same time, the information near each feature point is fused to finally generate a new feature map M. 2,4 (M 2,4 ∈R M×N×C ). Similarly, the same operation is performed on feature map m3 and feature map m5 to generate a new feature map. Finally, the generated feature map M 2,4 and M 3,5 By integrating the location information of the shallow network and the semantic information of the deep network, and carefully fusing the feature maps of different levels, the representation ability of the network has been comprehensively improved.
[0023] The idea of feature pyramid (FPN) is borrowed, but unlike FPN which adopts layer-by-layer fusion method, the MFDA algorithm in this paper adopts a cross-layer fusion strategy. This cross-layer feature map fusion strategy effectively integrates the high-level and low-level features generated by the network. Selecting cross-layer fusion can improve the applicability of the MFDA algorithm in various backbone networks. In the basic network architecture of VGG16, the number of convolutional layers is relatively small. Compared with networks with deep convolutional structures such as ResNet50 or ResNet101, if the feature pyramid network method is still used to fuse convolutional layers layer by layer, the high-level semantic information will be passed to the surface convolutional layer after being fused with the continuous and intermediate feature layers, and its original information may be lost. Therefore, it is more reasonable to adopt a cross-layer fusion strategy to retain the semantic content between layers to the greatest extent.
[0024] Step 2: Background Suppression Module
[0025] In order to make the feature map pay more attention to the foreground target in the image after processing, a background suppression method is proposed in this chapter. It mainly includes the following steps: first, the new feature map M generated after feature fusion is 2,4 and M 3,5 Process, identify and filter out feature graph M 2,4 and M 3,5 The feature points whose amplitude is lower than the overall average level can effectively suppress the background information that may interfere with the alignment quality. 2,4 The processing process of M 2,4 The dimension of the feature space is C. The global average pooling (GAP) method is used to process the feature vectors of each dimension in the feature map and calculate the global average eigenvalue vector A1 (A1∈R 1×1×C ), the calculation of this vector is based on the average of the global information of each dimension of the feature map, reflecting the overall feature strength. Then, all feature points below the global average are reset to zero, and these feature points identified as background noise are removed to remove low-value feature points that are not conducive to model learning. Finally, the new feature map M obtained after background suppression processing is 2,4 Marked as G 2,4 . Feature map M 3,5 The same processing flow is also adopted to generate the corresponding new feature map.
[0026] Step 3: Domain Classification and Consistency Regularizer
[0027] In the design of the MFDA algorithm, D1 and D2 are set as feature maps G 2,4 and G 3,5The design idea of the domain classifier in the DA algorithm is borrowed from the design of the domain classifier. The domain classifiers D1 and D2 are responsible for the feature map G 2,4 and G 3,5 Each feature point in the D1 domain is classified into domains, and the original image area corresponding to each feature point is judged to be from the source domain or the target domain. The specific structural details of the D1 domain classifier are as follows: Figure 2 Assume that the feature map G 2,4 The dimension of is M×N×C, and the Conv1 convolution layer is used to perform 1×1 convolution to reduce the number of feature channels to 128. The Conv2 convolution layer also uses the same operation to further compress the data dimension to 1, and comprehensively strengthen the feature information from different dimensions. The size of the feature map is M×N×1. The sigmoid activation function is applied to the feature map to predict the probability of each feature point belonging to the source domain or the target domain, which clearly defines the domain attributes of each region in the original image. The final domain probability map P1 (P1∈R M×N×1 ). It should be noted that although the output dimension is adjusted to 256 in the Conv1 convolutional layer to accommodate different feature dimension requirements, the D2 and D1 domain classifiers remain highly consistent in structure, ensuring that the algorithms are unified when performing domain classification.
[0028] Based on the previous analysis, the loss function caused by the domain classification process is defined as follows:
[0029]
[0030] In the formula, i refers to the image sample in the dataset. i is the domain label to which the sample belongs. i =1 or 0, it means that the sample belongs to the target domain or the source domain; i (u,v) It represents the predicted probability that the feature point with coordinates (u, v) on the feature map corresponding to the i-th image sample is classified as the source domain or the target domain. If the probability value is close to 0, it indicates that the feature point corresponding to the position in the original image is likely to belong to the source domain. Conversely, if the probability value is close to 1, it indicates that the feature point is more likely to be part of the target domain.
[0031] In order to solve the problem that the domain classification may be different in the same region, the outputs of the two domain classifiers D1 and D2 are regularized. Specifically, as shown in the formula.
[0032]
[0033] In the formula, I1 and I2 refer to the feature map G 2,4 and G 3,5 The total number of feature points in and It refers to the feature map G of the i-th image sample 2,4 and G 3,5 In , the probability value of whether the feature point mapping area at the position (u, v) is from the source domain or the target domain. ‖·‖2 represents the norm, which is used to measure the difference in domain classification probability of two domain classifiers at the same feature point. Based on this, the overall loss function definition formula of the MGFA algorithm is as follows.
[0034] L MFDA =L glo +L cst
[0035] The goal of regularization is to reduce the differences in domain attribution predictions between different domain classifiers at the same feature point, thereby enhancing the consistency and accuracy of the model when performing domain classification.
[0036] Step 4: Train and test the proposed method and mainstream algorithms on commonly used public datasets
[0037] In order to verify the effectiveness of the algorithm proposed in this paper, two groups of experiments are set up in this part, namely, two groups of experiments with different backgrounds (the source domain is the CityScape dataset under sunny scenes, and the target domain is the Foggy CityScape dataset under foggy scenes, denoted as CF) and different image styles and image qualities (the source domain is the Simil10k dataset of game scenes and the KITTI dataset with lower image quality; the target domain is the CityScape dataset of real scenes, denoted as SC and KC). The algorithm in this paper is tested on standard datasets, and quantitative and qualitative analysis is performed with mainstream algorithms from an experimental perspective.
[0038] (1) Comparative experiments on AP and mAP of MFDA algorithm and mainstream algorithms for different target categories
[0039] In order to verify the effectiveness of the MFDA algorithm, it is compared with the mainstream algorithms in the field of domain adaptation. The evaluation criteria used are AP and mAP values. The AP and mAP values of different target categories are compared under the conditions of using different backbone networks. The effect of using VGG16 as the backbone network is compared. Figure 3 As shown in the figure, the detection effect examples of each algorithm are shown as follows Figure 4 As shown in the figure. In the experiment, the source domain uses the CityScape dataset and the target domain uses the Foggy CityScape dataset. The comparison is carried out under the following 5 experimental conditions. Each experimental condition is briefly introduced below:
[0040] 1. Faster R-CNN is the baseline model,
[0041] 2.DA is the indicator of the improved baseline model in the target domain;
[0042] 3.MAF is the indicator of the improved baseline model in the target domain;
[0043] 4.MFDA is the indicator of the algorithm in this chapter on the target domain;
[0044] 5. Oracle is an indicator on the target domain after training with both source domain data and target domain data.
[0045] By comparison, we can conclude that the generalization performance of the baseline model is poor, and there is a big difference compared with Oracle. The MFDA algorithm has improved AP and mAP indicators for different categories of targets compared with mainstream algorithms, which are 6.9% and 1.6% higher respectively. The algorithm in this paper can more accurately detect targets of different scales, such as bicycles and cars. Compared with the DA algorithm, the AP and mAP indicators for different categories of targets have improved to varying degrees, especially for smaller targets. This shows that the multi-scale feature fusion of the model proposed in this paper is better for detecting targets of different scales.
[0046] (2) Different image styles and qualities
[0047] In order to further verify the effectiveness of the algorithm in this chapter, the test model set up two groups of experiments on image style (the source domain uses the Sim10k dataset, and the target domain uses the CityScape dataset, (SC)) and image quality (the source domain uses the KITTI dataset, and the target domain uses the CityScape dataset, (KC)). For a more intuitive representation, the car AP indicator is selected as the category for effect comparison. Its indicators are as follows: Figure 5 As shown in the figure, it can be seen that when VGG16 is used as the backbone network of the MFDA algorithm, the car AP index of the improved algorithm is improved to varying degrees in different image style and quality scenarios.
[0048] The quality of the prediction box determines whether the model can accurately locate and identify the target in the image, because it is analyzed from the perspective of the prediction box. The IoU threshold of the MFDA algorithm is set to 0.5. Based on this, the prediction boxes are divided into the following three types: the first type, IoU greater than 0.5, is a correct prediction box; the second type, IoU greater than 0.3 and less than 0.5, is a partially correct prediction box; the third type, IoU less than 0.3, is an incorrect prediction box. In scenarios with different image quality, the distribution of the prediction boxes of the top 100 confidence levels of the target domain samples by the algorithm proposed in this article and the mainstream algorithm is shown below. Figure 6As shown in the figure, the proportion of the third prediction box generated by the Faster R-CNN algorithm is greater than that of the first prediction box. Although the proportion of the first prediction box generated by the DA algorithm has increased, it is still lower than that of the MFDA algorithm.
[0049] In summary, this paper conducts research on the optimization of domain adaptive target detection methods, and makes corresponding improvements to the problems existing in the DA algorithm, mainly including the single scale of alignment and the forced alignment of background features while aligning foreground features. To address the problem of single scale alignment, this paper proposes a multi-scale feature fusion module to fuse shallow feature information with deep feature information. To address the problem of forced alignment of background features, a background suppression module is proposed to suppress the background of the fused feature information, thereby aligning foreground features. At the same time, a domain classification consistency regularizer is proposed to ensure the consistency of domain classification results. Finally, the algorithm in this paper is trained and tested with the mainstream algorithm on commonly used public datasets. The test results show that the algorithm proposed in this paper has significantly improved the effect of the target detection model.
Claims
1. A domain adaptive object detection method based on multi-scale feature fusion, characterized in that: The following steps are involved: Step 1: Design a multi-scale feature fusion module based on the baseline model to align and fuse the generated feature maps with different scales; Step 2: In order to reduce the impact of background feature noise on the accuracy of feature alignment, a background suppression module is added to process the feature map fused in step 1; Step 3: Input the updated feature map into two different domain classifiers for processing. Considering that the same feature area may be classified separately in different domain classifiers, the algorithm uses the domain classification consistency regularizer; Step 4: To verify the effectiveness of the proposed method, two sets of experiments were set up in this section, one with different backgrounds (the source domain is the CityScape dataset under sunny scenes, and the target domain is the Foggy CityScape dataset under foggy scenes) and different image styles and image quality (the source domains are the Simil10k dataset of game scenes and the KITTI dataset with lower image quality; the target domains are all the CityScape datasets of real scenes).
2. The method according to claim 1, wherein: The cross-layer feature map fusion strategy in step 1 is different from the layer-by-layer fusion method of the feature pyramid (FPN). Cross-layer fusion effectively integrates the high-level and low-level features generated by the network. Compared with layer-by-layer fusion, the cross-layer fusion strategy better preserves the semantic content between layers and avoids the loss of high-level semantic information when it is passed to the surface convolution layer after fusion of continuous intermediate feature layers.
3. The method according to claim 1, wherein: The new feature map M generated after feature fusion in step 2 2,4 and M 3,5 Processing is performed to identify and filter out feature points that are below the overall average level. The global average pooling (GAP) method is used to process the feature vectors of each dimension in the feature map, and the global average eigenvalue vector is calculated. All feature points below the global average are set to zero, and background noise feature points are removed to generate a new feature map G after background suppression processing. 2,4 and G 3,5 .
4. The method according to claim 1, wherein: In step 3, domain classifiers D1 and D2 are designed, where D1 is used for feature map G 2,4 , D2 is used for feature map G 3,5 , using a similar structure to maintain consistency, the feature map G 2,4 Processing: The Conv1 and Conv2 convolution layers are used to adjust the dimension and enhance the information of the feature map to generate the domain probability map P1. The sigmoid activation function is used to classify each feature point as the source domain or the target domain, and the domain classification loss function L is defined. glo , to ensure that the domain attributes of each feature point are correctly identified, the regularization loss function L is introduced cst , used to reduce the difference in domain classification between D1 and D2 at the same feature point and enhance the domain classification consistency of the model. The overall loss function L MFDA =L glo +L cst ,used to optimize the training and performance of domain-adaptive object detection algorithms.
Citation Information
Patent Citations
Cross-domain target detection method based on regional full convolutional network and self-adaptation
CN111553397A
Global feature alignment target detection method based on multi-scale feature fusion
CN113052187A
Domain adaptive target detection method and system based on foreground-category perception alignment
CN114821152A
Domain alignment for object detection domain adaptation tasks
US20210312232A1
Cited By
Feature pyramid network detection method based on weighted cross-level channel attention mechanism
CN120071066A
