Optical remote sensing image target detection domain adaptation method based on multi-view adaptive feature alignment

Abstract

The invention discloses an optical remote sensing image target detection domain adaptation method based on multi-view adaptive feature alignment. A multi-view adaptive feature alignment module provided by the invention firstly adaptively captures domain specific features in an image by using a cross attention mechanism; subsequently, the method adopts a training mode of comparative learning to enhance the difference of aggregation features of the feature alignment module so as to realize feature aggregation of different view angles. Compared with the prior art, the method has the advantages that the domain specific features in the optical remote sensing image can be captured more effectively, and more accurate feature alignment is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical remote sensing image processing, and in particular relates to an optical remote sensing image target detection domain adaptation method based on multi-view adaptive feature alignment. Background Art

[0002] Target detection in optical remote sensing images can locate and classify land objects, providing important support for environmental monitoring, urban planning, resource management and other fields. In the early days, this task was mainly completed by manual interpretation. Although this method has excellent accuracy, it consumes a lot of manpower and has a slow processing speed, which is difficult to meet the requirements of modern society for efficient processing. With the introduction of deep learning technology, target detection algorithms based on deep learning have been widely used in remote sensing scenarios to significantly improve the processing efficiency of remote sensing images.

[0003] However, the deep learning-based object detection method belongs to a data-driven architecture, which requires that the distribution of training data and test data is consistent. However, in actual remote sensing applications, the collected images are often affected by factors such as scene, weather or season, resulting in inter-domain differences between their representation and training data, which significantly reduces the performance of the detection algorithm.

[0004] To solve this problem, the present invention proposes a domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment, which has strong generalization ability and high detection accuracy and can effectively support a variety of practical application scenarios. Summary of the invention

[0005] The technical problem solved by the present invention is: for optical remote sensing images, a domain adaptation method for target detection in optical remote sensing images is proposed. The method effectively captures domain-specific features in optical remote sensing images by using a designed multi-view adaptive feature alignment module to achieve more accurate feature alignment.

[0006] The technical solution of the present invention is:

[0007] A method for domain adaptation of target detection in optical remote sensing images based on multi-view adaptive feature alignment, the method comprising the following steps:

[0008] Step 1: using two different types of satellites to observe the area of ​​interest where the target exists, and collect images to obtain a set of optical remote sensing image pairs;

[0009] Step 2: Build a detection network, which includes a detector architecture DINO and a multi-view adaptive feature alignment module; the detector architecture DINO includes a backbone network resnet-50 and a result interpretation network; the multi-view adaptive feature alignment module includes two learnable vectors QA , Q B and a cross-attention computation module;

[0010] Step 3: extract the features of the image pair formed in step 1 through the backbone network in the detection network constructed in step 2 to form an image pair feature map F;

[0011] Step 4: Use step 2 to construct a cross-attention calculation module to perform correlation calculations on the two learnable vectors and the feature map F obtained in step 3, and obtain the learnable vectors of the specific features of the aggregation domain. and

[0012] Step 5: Use the contrastive learning loss function to optimize the two sets of learnable vectors obtained in step 4 and At the same time, the optimized image pair feature map F is obtained;

[0013] Step 6, using the result interpretation network in the detection network constructed in step 2 to decode the image feature map F optimized in step 5, to obtain the target prediction result;

[0014] Step 7: Calculate the error loss using the target prediction result obtained in step 6 and the manually labeled true value, and train the detection network constructed in step 2 according to the calculated error loss to obtain a trained detection network;

[0015] Step eight, input the newly acquired optical remote sensing image into the detection network trained in step seven, and obtain the prediction results of the target position and category of interest in the corresponding optical remote sensing image.

[0016] Furthermore, in the step 3, the specific method of using the backbone network in the constructed detection network to extract the feature map F of the remote sensing image obtained in step 1 is:

[0017] F 1 =Backbone(X 1 ), F 2 =Backbone(X 2 )

[0018] Among them, Backbone represents the backbone network in building the detection network; X 1 represents the first image in the image pair, X 2 Represents the first image in the pair.

[0019] Furthermore, in step 4, the cross attention calculation module is used to make the two learnable vectors correlated with the feature map F obtained in step 3 to obtain two sets of learnable vectors of aggregated domain-specific features. and The specific method is:

[0020]

[0021] Among them, CrossAtten(·) represents the standard cross attention calculation method.

[0022] Furthermore, in step 5, the two sets of learnable vectors obtained in step 4 are optimized using the contrastive learning loss function. and The specific formula is:

[0023]

[0024] L con (·), L adv (·) represents the loss size, exp(·) represents the exponential function with the natural constant e as the base, and log(·) represents the natural logarithm.

[0025] Furthermore, in step 6, the optimized feature map F is decoded using the prediction network in the detection network constructed in step 2 to obtain the final target prediction result, which can be expressed as:

[0026] P=Prediction(F)

[0027] Among them, Prediction represents the result interpretation network in the detector, and P represents the prediction result for the input image pair X.

[0028] Furthermore, in step 7, the target prediction result obtained in step 6 and the manually labeled true value are used to calculate the error loss to achieve network training. The specific error loss calculation formula is as follows:

[0029]

[0030] Among them, P is the prediction result of image pair X obtained in step 6, and Y det Represents the manually labeled true value of the target position.

[0031] Beneficial Effects

[0032] (1) The multi-view adaptive feature alignment module proposed in this paper uses multi-view query to capture the domain-specific features extracted by CNN, adaptively aggregates the domain-specific features through the cross-attention mechanism, determines the source of the features and then performs adversarial training. Compared with the existing methods, this method ensures that the detection network can effectively extract domain-invariant features in optical remote sensing images, so that the detection method can maintain efficient detection performance for different data sources.

[0033] (2) Contrastive learning is used to enhance the uniqueness of domain-specific features aggregated for each single-view query; compared with existing methods, this method achieves more accurate multi-view feature alignment.

[0034] (3) The framework is integrated into a DINO-based detector. Compared with existing methods, this method can further utilize the unlabeled data in the target domain to significantly reduce the inter-domain differences and more effectively solve the performance degradation problem caused by domain differences between training and test data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 , one A flow chart of a domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment;

[0036] Figure 2 ,Multi-view adaptive alignment module structure diagram;

[0037] Figure 3 , Flowchart for extracting features of input image pairs based on the ResNet-50 backbone network;

[0038] Figure 4 , Flowchart of using the result interpretation network to predict target positioning results;

[0039] Figure 5 ,Finally, compare the experimental results on different data sets. DETAILED DESCRIPTION

[0040] The present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0041] like Figure 1 As shown, the present invention provides a method for domain adaptation of optical remote sensing image target detection based on multi-view adaptive feature alignment, the steps of the method include:

[0042] Step 1: Use two different types of satellites to observe the area of ​​interest where the target exists, and collect images to obtain a set of optical remote sensing image pairs; the size of each collected image is 800×800 pixels.

[0043] Step 2: Build a detection network, which includes a detector architecture DINO and a multi-view adaptive feature alignment module; the detector architecture DINO includes a backbone network resnet-50 and a result interpretation network, such as Figure 1 As shown; the multi-view adaptive feature alignment module includes two learnable vectors Q A , Q B and a cross-attention computation module, such as Figure 2 shown.

[0044] Step 3: Through the backbone network in the detection network constructed in step 2, the features of the image pair formed in step 1 are extracted to form an image pair feature map F, such as Figure 3 As shown; the specific method of using the backbone network in the constructed detection network to extract the optical remote sensing image feature map F obtained in step 1 is:

[0045] F 1 =Backbone(X 1 ), F 2 =Backbone(X 2 )

[0046] Among them, Backbone represents the backbone network in building the detection network; X 1 represents the first image in the image pair, X 2 Represents the first image in the image pair. Since each image is 800×800 pixels in size, X 1 and X 2 Both are image tensors 3 indicates that the image is a three-channel RGB image, and 800×800 indicates that the image size is 800 pixels wide and 800 pixels high. 1 and X 2 After performing feature extraction operations respectively, the feature map F is obtained 1 and F 2 , feature map F 1 and Among them, 256 represents the channel dimension of the feature map, and 100×100 represents the width and height of the feature map after 8 times downsampling.

[0047] Step 4: Use step 2 to construct a cross-attention calculation module to perform correlation calculations on the two learnable vectors and the feature map F obtained in step 3, and obtain the learnable vectors of the specific features of the aggregation domain. and like Figure 2 As shown; using the cross attention calculation module to make two learnable vectors Q A ,Q B Correlate with the feature map F obtained in step 3 to obtain two sets of learnable vectors of aggregated domain-specific features and The specific method is:

[0048]

[0049] Among them, CrossAtten(·) represents the standard cross attention calculation method. Dimensional consistency Both are one-dimensional vectors with 256 channels.

[0050] Step 5: Use the contrastive learning loss function to optimize the two sets of learnable vectors obtained in step 4 and At the same time, the optimized image pair feature map F is obtained; the two sets of learnable vectors obtained in step 4 are optimized using the contrastive learning loss function and The specific formula is:

[0051]

[0052] Among them, L con (·), L adv (·) represents the loss size, exp(·) represents the exponential function with the natural constant e as the base, log(·) represents the natural logarithm, D represents the domain discriminator, and is marked as d=0, and the target domain is marked as d = 1. and All dimension values ​​are averaged after loss calculation through the formula, so L con , L adv It is a specific value between 0 and 1.

[0053] Step 6: Use the result interpretation network in the detection network constructed in step 2 to decode the image feature map F optimized in step 5 to obtain the target prediction result, such as Figure 4 As shown; the prediction network in the detection network constructed in step 2 is used to decode the optimized feature map F to obtain the final target prediction result, which can be expressed as:

[0054] P=Prediction(F)

[0055] Prediction represents the result interpretation network in the detector, and P represents the prediction result for the input image pair X. P is the predicted positioning result for the target of interest, and its dimension is N×4, where N represents the number of targets in the prediction result. For each specific prediction result, its positioning coordinates are expressed as (x c ,x y ,x w ,x h ) represent the horizontal coordinate, vertical coordinate, width and height of the center point of the target in the figure respectively.

[0056] Step 7: Use the target prediction result obtained in step 6 and the manually labeled true value to calculate the error loss, and train the detection network constructed in step 2 according to the calculated error loss to obtain the trained detection network; use the target prediction result obtained in step 6 and the manually labeled true value to calculate the error loss to realize network training. The specific error loss calculation formula is as follows:

[0057]

[0058] Among them, P is the prediction result of image pair X obtained in step 6, and Y det Represents the manually labeled true value of the target position.

[0059] Step eight, input the newly acquired optical remote sensing image into the detection network trained in step seven, and obtain the prediction results of the target position and category of interest in the corresponding optical remote sensing image.

[0060] In this example, a comparative experiment is conducted between the method of the present invention and the existing method, and the robustness of the method of the present invention is verified by comparing the running results of the model. The experiment uses four public datasets, xView, dota, UCAS-AOD and CARPK, to evaluate the performance of our proposed method in two domain adaptation scenarios and compare it with the most advanced methods, including:

[0061] (1) Adaptation between different optical satellites

[0062] The present invention uses the xView to dota dataset to demonstrate the effectiveness of the method used in cross-domain detection between different optical satellites. As shown in Table 1, the method of the present invention achieves the highest mAP of 63.7%, which is 0.4% higher than the state-of-the-art method.

[0063] Table 1 Classification accuracy of the proposed method and existing models for different data sets

[0064]

[0065] (2) Adaptation between satellites and drones

[0066] We use the UCAS-AOD and CARPK datasets to demonstrate our cross-domain detection performance from satellite to drone scenarios. As shown in Table 2, our method achieves a mAP of 77.7%. This shows the power of our method.

[0067] Table 2 Classification accuracy of the proposed method and existing models in other data sets

[0068]

[0069] The visualization results of DINO in all experimental domain adaptation scenarios are shown in the figure. Figure 5 As shown, the left figure is the experimental result of DINO detection before adding the multi-view adaptive feature alignment module, and the right figure is the experimental result of DINO detection after adding the multi-view adaptive feature alignment module.

[0070] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A domain adaptation method for target detection in optical remote sensing images based on multi-view adaptive feature alignment, characterized in that: The method comprises the following steps: Step 1: using two different types of satellites to observe the area of ​​interest where the target exists, and collect images to obtain a set of optical remote sensing image pairs; Step 2: construct a detection network, which includes a detector architecture DINO and a multi-view adaptive feature alignment module; The detector architecture DINO includes a backbone network resnet-50 and a result interpretation network; the multi-view adaptive feature alignment module includes two learnable vectors Q A , Q B and a cross-attention computation module; Step 3: extract the features of the image pair formed in step 1 through the backbone network in the detection network constructed in step 2 to form an image pair feature map F; Step 4: Use step 2 to construct a cross-attention calculation module to perform correlation calculations on the two learnable vectors and the feature map F obtained in step 3, and obtain the learnable vectors of the specific features of the aggregation domain. and Step 5: Use the contrastive learning loss function to optimize the two sets of learnable vectors obtained in step 4 and At the same time, the optimized image pair feature map F is obtained; Step 6, using the result interpretation network in the detection network constructed in step 2 to decode the image feature map F optimized in step 5, to obtain the target prediction result; Step 7: Calculate the error loss using the target prediction result obtained in step 6 and the manually labeled true value, and train the detection network constructed in step 2 according to the calculated error loss to obtain a trained detection network; Step eight, input the newly acquired optical remote sensing image into the detection network trained in step seven, and obtain the prediction results of the target position and category of interest in the corresponding optical remote sensing image.

2. The method for domain adaptation of target detection in optical remote sensing images based on multi-view adaptive feature alignment as claimed in claim 1, characterized in that: In the step 3, the specific method of using the backbone network in the constructed detection network to extract the optical remote sensing image pair feature map F obtained in step 1 is: F1=Backbone(X1), F2=Backbone(X2) Among them, Backbone represents the backbone network in building the detection network; X1 represents the first image in the image pair, and X2 represents the second image in the image pair.

3. The method for domain adaptation of target detection in optical remote sensing images based on multi-view adaptive feature alignment as claimed in claim 1, characterized in that: In step 4, the cross attention calculation module is used to make the two learnable vectors correlated with the feature map F obtained in step 3 to obtain two sets of learnable vectors with specific features of the aggregation domain. and The specific method is: Among them, CrossAtten(·) represents the standard cross attention calculation method.

4. The method for domain adaptation of target detection in optical remote sensing images based on multi-view adaptive feature alignment according to claim 1, characterized in that: In step 5, the two sets of learnable vectors obtained in step 4 are optimized using the contrastive learning loss function. and The specific formula is: L con (·), L adv (·) represents the loss size, exp(·) represents the exponential function with the natural constant e as the base, log(·) represents the natural logarithm, D represents the domain discriminator, and is marked as d=0, and the target domain is marked as d=1; the method and All dimension values ​​are averaged after loss calculation through the formula, so L con , L adv It is a specific value between 0 and 1.

5. The method for domain adaptation of target detection in optical remote sensing images based on multi-view adaptive feature alignment according to claim 1, characterized in that: In step 6, the optimized feature map F is decoded using the prediction network in the detection network constructed in step 2 to obtain the final target prediction result, which can be expressed as: P=Prediction(F) Among them, Prediction represents the result interpretation network in the detector, and P represents the prediction result for the input image pair X.

6. The method for domain adaptation of target detection in optical remote sensing images based on multi-view adaptive feature alignment as claimed in claim 1, characterized in that: In step 7, the target prediction result obtained in step 6 and the manually labeled true value are used to calculate the error loss to implement network training. The specific error loss calculation formula is as follows: Among them, P is the prediction result of image pair X obtained in step 6, and Y det Represents the manually labeled true value of the target position.

Images