A domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment

Abstract

This paper discloses a domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment. The proposed multi-view adaptive feature alignment module first utilizes a cross-attention mechanism to adaptively capture domain-specific features in the image. Subsequently, the method employs a contrastive learning training approach to enhance the diversity of features aggregated by the feature alignment module, thereby achieving feature aggregation from different viewpoints. Compared with existing methods, this method can more effectively capture domain-specific features in optical remote sensing images and achieve more accurate feature alignment.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical remote sensing image processing, and in particular relates to a domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment. Background Art

[0002] Object detection in optical remote sensing images can locate and classify land features, providing crucial support for fields such as environmental monitoring, urban planning, and resource management. In the early days, this task relied primarily on manual interpretation. While this method offered excellent accuracy, it was labor-intensive and slow, making it difficult to meet the demands of modern society for efficient processing. With the advent of deep learning technology, deep learning-based object detection algorithms have been widely applied in remote sensing scenarios, significantly improving the efficiency of remote sensing image processing.

[0003] However, deep learning-based object detection methods are data-driven and require consistent distributions between training and test data. However, in real-world remote sensing applications, captured images are often affected by factors such as scene, weather, or season, leading to domain differences between their representations and the training data, significantly degrading the performance of detection algorithms.

[0004] To solve this problem, the present invention proposes a domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment, which has strong generalization ability and high detection accuracy and can effectively support a variety of practical application scenarios. Summary of the Invention

[0005] The technical problem solved by this invention is to propose a domain adaptation method for target detection in optical remote sensing images. This method effectively captures domain-specific features in optical remote sensing images using a designed multi-view adaptive feature alignment module to achieve more accurate feature alignment.

[0006] The technical solution of the present invention is:

[0007] A domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment, the method comprising the following steps:

[0008] Step 1: using two different types of satellites to observe the area of ​​interest where the target exists, and collect images to obtain a set of optical remote sensing image pairs;

[0009] Step 2: Build a detection network, which includes a detector architecture DINO and a multi-view adaptive feature alignment module; the detector architecture DINO includes a backbone network resnet-50 and a result interpretation network; the multi-view adaptive feature alignment module includes two learnable vectors QA , Q B and a cross-attention computation module;

[0010] Step 3: Extract the features of the image pair formed in step 1 through the backbone network in the detection network constructed in step 2 to form an image pair feature map F;

[0011] Step 4: Use step 2 to build a cross attention calculation module to perform correlation calculations on the two learnable vectors and the feature map F obtained in step 3, and obtain the learnable vectors of the specific features of the aggregation domain. and

[0012] Step 5: Use the contrastive learning loss function to optimize the two sets of learnable vectors obtained in step 4. and At the same time, the optimized image pair feature map F is obtained;

[0013] Step 6: Use the result interpretation network in the detection network constructed in step 2 to decode the image feature map F optimized in step 5 to obtain the target prediction result;

[0014] Step 7: Calculate the error loss using the target prediction result obtained in step 6 and the manually labeled true value, and train the detection network constructed in step 2 based on the calculated error loss to obtain a trained detection network;

[0015] In step eight, the newly acquired optical remote sensing image is input into the detection network trained in step seven to obtain the prediction results of the target location and category of interest in the corresponding optical remote sensing image.

[0016] Furthermore, in step 3, the specific method of using the backbone network in the constructed detection network to extract the feature map F of the remote sensing image obtained in step 1 is:

[0017] F1=Backbone(X1), F2=Backbone(X2)

[0018] Among them, Backbone represents the backbone network in constructing the detection network; X1 represents the first image in the image pair, and X2 represents the first image in the image pair.

[0019] Furthermore, in step 4, the cross attention calculation module is used to perform correlation calculations on the two learnable vectors with the feature map F obtained in step 3 to obtain two sets of learnable vectors of aggregated domain-specific features. and The specific method is:

[0020]

[0021] Among them, CrossAtten(·) represents the standard cross attention calculation method.

[0022] Furthermore, in step 5, the two sets of learnable vectors obtained in step 4 are optimized using the contrastive learning loss function. and The specific formula is:

[0023]

[0024] L con (·), L adv (·) represents the loss size, exp(·) represents the exponential function with the natural constant e as the base, and log(·) represents the natural logarithm.

[0025] Furthermore, in step 6, the optimized feature map F is decoded using the prediction network in the detection network constructed in step 2 to obtain the final target prediction result, which can be expressed as:

[0026] P=Prediction(F)

[0027] Among them, Prediction represents the result interpretation network constructed in the detector, and P represents the prediction result for the input image pair X.

[0028] Furthermore, in step seven, the target prediction result obtained in step six and the manually labeled true value are used to calculate the error loss to implement network training. The specific error loss calculation formula is as follows:

[0029]

[0030] Among them, P is the prediction result of image pair X obtained in step 6, Y det Represents the manually labeled true value of the target position.

[0031] Beneficial effects

[0032] (1) The multi-view adaptive feature alignment module proposed in this paper uses multi-view queries to capture domain-specific features extracted by CNNs. It adaptively aggregates domain-specific features through a cross-attention mechanism, determines the source of features, and then performs adversarial training. Compared with existing methods, this method ensures that the detection network can effectively extract domain-invariant features in optical remote sensing images, thereby maintaining high detection performance for different data sources.

[0033] (2) Contrastive learning is exploited to enhance the uniqueness of domain-specific features aggregated from each single-view query; compared with existing methods, this method achieves more accurate multi-view feature alignment.

[0034] (3) The framework is integrated into a DINO-based detector; compared with existing methods, this method can further utilize the unlabeled data in the target domain to significantly reduce the inter-domain differences and more effectively solve the performance degradation problem caused by domain differences between training and test data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 、 one Flowchart of a domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment;

[0036] Figure 2 ,Multi-view adaptive alignment module structure diagram;

[0037] Figure 3 ,Flowchart for extracting features of input image pairs based on ResNet-50 backbone network;

[0038] Figure 4 ,Flowchart of using the result interpretation network to predict target positioning results;

[0039] Figure 5 ,Finally, compare the experimental results on different data sets. DETAILED DESCRIPTION

[0040] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0041] like Figure 1 As shown, the present invention provides a domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment, the method comprising the following steps:

[0042] Step 1: Use two different types of satellites to observe the area of ​​interest where the target exists, and collect images to obtain a set of optical remote sensing image pairs; the size of each collected image is 800×800 pixels.

[0043] Step 2: Build a detection network, which includes a detector architecture DINO and a multi-view adaptive feature alignment module; the detector architecture DINO includes a backbone network resnet-50 and a result interpretation network, such as Figure 1 As shown; the multi-view adaptive feature alignment module includes two learnable vectors Q A , Q B and a cross-attention computation module, such as Figure 2 shown.

[0044] Step 3: Through the backbone network in the detection network constructed in step 2, the features of the image pair formed in step 1 are extracted to form an image pair feature map F, as shown in Figure 3As shown; the specific method of using the backbone network in the constructed detection network to extract the optical remote sensing image feature map F obtained in step 1 is:

[0045] F1=Backbone(X1), F2=Backbone(X2)

[0046] Among them, Backbone represents the backbone network in the detection network; X1 represents the first image in the image pair, and X2 represents the first image in the image pair. Since the size of each image is 800×800 pixels, X1 and X2 are both image tensors. 3 indicates that the image is a three-channel RGB image, and 800×800 indicates that the image size is 800 pixels in width and height. The feature maps F1 and F2 are obtained by performing feature extraction on the input images X1 and X2 using the backbone network resnet-50. Among them, 256 represents the channel dimension of the feature map, and 100×100 represents the width and height of the feature map after 8 times downsampling.

[0047] Step 4: Use step 2 to build a cross attention calculation module to perform correlation calculations on the two learnable vectors and the feature map F obtained in step 3, and obtain the learnable vectors of the specific features of the aggregation domain. and like Figure 2 As shown; using the cross attention calculation module to make two learnable vectors Q A ,Q B Perform correlation calculations with the feature map F obtained in step 3 to obtain two sets of learnable vectors of aggregated domain-specific features and The specific method is:

[0048]

[0049] Among them, CrossAtten(·) represents the standard cross attention calculation method. Dimensional consistency Both are one-dimensional vectors with 256 channels.

[0050] Step 5: Use the contrastive learning loss function to optimize the two sets of learnable vectors obtained in step 4. and At the same time, the optimized image pair feature map F is obtained; the two sets of learnable vectors obtained in step 4 are optimized using the contrastive learning loss function. and The specific formula is:

[0051]

[0052] Among them, L con (·), L adv (·) represents the loss size, exp(·) represents the exponential function with the natural constant e as the base, log(·) represents the natural logarithm, D represents the domain discriminator, and Marked as d=0, and the target domain is marked as d = 1. and All dimension values ​​are averaged after loss calculation using the formula, so L con , L adv It is a specific value between 0 and 1.

[0053] Step 6: Use the result interpretation network in the detection network constructed in step 2 to decode the image feature map F optimized in step 5 to obtain the target prediction result, such as Figure 4 As shown in Figure 2, the prediction network in the detection network constructed in step 2 is used to decode the optimized feature map F to obtain the final target prediction result, which can be expressed as:

[0054] P=Prediction(F)

[0055] Among them, Prediction represents the result interpretation network in the detector, and P represents the prediction result for the input image pair X. P is the predicted positioning result for the target of interest, and its dimension is N×4, where N represents the number of targets in the prediction result. For each specific prediction result, its positioning coordinate is expressed as (x c ,x y ,x w ,x h ) represent the horizontal coordinate, vertical coordinate, width and height of the center point of the target in the figure respectively.

[0056] Step 7: Use the target prediction results obtained in step 6 and the manually labeled true value to calculate the error loss, and train the detection network constructed in step 2 according to the calculated error loss to obtain the trained detection network; use the target prediction results obtained in step 6 and the manually labeled true value to calculate the error loss to achieve network training. The specific error loss calculation formula is as follows:

[0057]

[0058] Among them, P is the prediction result of image pair X obtained in step 6, Y det Represents the manually labeled true value of the target position.

[0059] In step eight, the newly acquired optical remote sensing image is input into the detection network trained in step seven to obtain the prediction results of the target location and category of interest in the corresponding optical remote sensing image.

[0060] In this example, a comparative experiment was conducted between the proposed method and existing methods to verify the robustness of the proposed method by comparing the running results of the models. This experiment used four public datasets, xView, dota, UCAS-AOD, and CARPK, to evaluate the performance of our proposed method in two domain adaptation scenarios and compare it with the most advanced methods. Specifically, the following results were obtained:

[0061] (1) Adaptation between different optical satellites

[0062] We use the xView to dota dataset to demonstrate the effectiveness of our method for cross-domain detection between different optical satellites. As shown in Table 1, our method achieves the highest mAP of 63.7%, which is 0.4% higher than the state-of-the-art method.

[0063] Table 1 Classification accuracy of the method of the present invention and the existing model for different data sets

[0064]

[0065] (2) Adaptation between satellites and drones

[0066] We demonstrate our cross-domain detection performance on the UCAS-AOD and CARPK datasets, spanning satellite and drone scenarios. As shown in Table 2, our method achieves a mAP of 77.7%, demonstrating the power of our approach.

[0067] Table 2 Classification accuracy of the proposed method and existing models in other data sets

[0068]

[0069] The visualization results of DINO in all experimental domain adaptation scenarios are shown in Figure 2. Figure 5 As shown, the left figure is the experimental result of DINO detection before adding the multi-view adaptive feature alignment module, and the right figure is the experimental result of DINO detection after adding the multi-view adaptive feature alignment module.

[0070] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment, characterized in that: The method comprises the following steps: Step 1: using two different types of satellites to observe the area of ​​interest where the target exists, and collect images to obtain a set of optical remote sensing image pairs; Step 2: Build a detection network, which includes a detector architecture DINO and a multi-view adaptive feature alignment module; The detector architecture DINO includes a backbone network resnet-50 and a result interpretation network; the multi-view adaptive feature alignment module includes two learnable vectors 、 and a cross-attention computation module; Step 3: Through the backbone network in the detection network constructed in step 2, the features of the image pair formed in step 1 are extracted to form an image pair feature map. ; Use the backbone network in the constructed detection network to extract the feature map of the optical remote sensing image obtained in step 1 The specific method is: ， ； in, It represents the backbone network in building the detection network; represents the first image in the pair, Represents the second image in an image pair; Step 4: Use step 2 to build a cross attention calculation module to make the two learnable vectors and the feature map obtained in step 3 Perform relevant calculations to obtain the learnable vectors of the specific features of the aggregation domain and ; Use the cross attention calculation module to make the two learnable vectors respectively correlated with the feature map obtained in step 3 Perform relevant calculations to obtain two sets of learnable vectors that aggregate domain-specific features and ; The specific method is: = ， = ； = ， = ； in, Expressed as a standard crisscross attention calculation method; Step 5: Use the contrastive learning loss function to optimize the two sets of learnable vectors obtained in step 4. and , and the optimized image pair feature map is obtained ; Use the contrastive learning loss function to optimize the two sets of learnable vectors obtained in step 4 and The specific formula is: ； ； Indicates the size of the loss, exp represents the exponential function with the natural constant e as the base, Natural logarithm, D represents the domain discriminator, the source domain Marked as d=0, and the target domain is marked as d=1; the method is and All dimension values ​​are averaged after loss calculation using the formula, so , It is a specific value between 0 and 1; Step 6: Use the result interpretation network in the detection network built in step 2 to analyze the feature map of the image optimized in step 5. Decode and obtain the target prediction result; Step 7: Calculate the error loss using the target prediction result obtained in step 6 and the manually labeled true value, and train the detection network constructed in step 2 based on the calculated error loss to obtain a trained detection network; In step eight, the newly acquired optical remote sensing image is input into the detection network trained in step seven to obtain the prediction results of the target location and category of interest in the corresponding optical remote sensing image.

2. The method for domain adaptation of target detection in optical remote sensing images based on multi-view adaptive feature alignment according to claim 1, characterized in that: In step 6, the prediction network in the detection network constructed in step 2 is used to optimize the feature map Decode to get the final target prediction result, which can be expressed as: ； in, Indicates the result interpretation network in the construction detector, It is represented as the prediction result for the input image pair X.

3. The domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment according to claim 1, characterized in that: In step 7, the target prediction result obtained in step 6 and the manually labeled true value are used to calculate the error loss to achieve network training. The specific error loss calculation formula is as follows: ； in, For the image pair obtained in step 6 The prediction results, Represents the manually labeled true value of the target position.

Images