Change detection method for optical-SAR (Synthetic Aperture Radar) heterogeneous remote sensing image

Abstract

The invention relates to a change detection method for optical-SAR (Synthetic Aperture Radar) heterogeneous remote sensing images, develops an improved GAN (Generalized Area Network) structure to eliminate the difference of the heterogeneous images, and proposes an integrated system architecture in which domain self-adaptive conversion and change detection are coordinated to realize high-precision robust change detection among the optical-SAR heterogeneous images. According to the method, the dual-time heterogeneous image features are adaptively converted into the same subspace, meanwhile, change detection is guided by using information of a discriminator in a domain adaptive GAN through a difference feature guide module, a feedback guide mechanism is further designed, and domain adaptive conversion is more accurate through a change detection result.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing image processing, and in particular relates to a change detection method for optical-SAR heterogeneous remote sensing images. Background Art

[0002] Change detection technology is widely used in various scenarios, such as agricultural monitoring, disaster detection, and land cover change detection. Due to the limitations of the imaging mechanisms of different remote sensing images, different remote sensing images have different advantages and disadvantages. For example, optical remote sensing images have higher resolutions, but can only image objects in cloudless areas, which to some extent hinders the application potential of remote sensing images. Heterogeneous change detection aims to utilize the complementary advantages of different remote sensing images to perform heterogeneous remote sensing image change detection. Heterogeneous remote sensing image change detection is not constrained by a single image type, and at the same time utilizes the complementary advantages of optical image high resolution and SAR all-weather imaging characteristics to solve the impact of cloud cover or other weather conditions on change detection tasks.

[0003] Existing heterogeneous change detection methods can be roughly divided into four categories: methods based on image fusion, methods based on domain conversion, methods based on image local feature extraction and analysis, and methods based on classification. Among them, the image fusion-based method analyzes the fused image to avoid direct comparison between heterogeneous images. The domain conversion-based method converts the heterogeneous image features into an intermediate common domain to make the heterogeneous image features comparable. The image local feature extraction and analysis method assumes that the unchanged areas of heterogeneous images have similar local features, and performs change analysis through the extracted local features. In the classification-based method, in order to avoid the differences in heterogeneous images, the change detection results are obtained by comparing the classification result maps of the dual-time heterogeneous images. Image domain conversion methods based on deep learning have become a hot topic and difficulty in the research of heterogeneous change detection. In general, the existing methods have the following two points that need to be improved: (1) The huge image domain differences between heterogeneous images cannot be ignored, which makes it difficult to obtain accurate conversion images; (2) Existing heterogeneous change detection methods focus on directly using the converted images in the same image domain. In the overall network framework, the image domain conversion and change detection parts are only a simple sequence relationship, lacking the exploration of the deep correlation between image domain conversion and change detection tasks, which is of great significance for the integrated research of heterogeneous image change detection. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method for detecting changes in optical-SAR heterogeneous remote sensing images.

[0005] The change detection method of optical-SAR heterogeneous remote sensing images includes the following steps:

[0006] Step 1: Extract features of different scales from optical remote sensing images and SAR images, and use the discriminator to make the multi-scale features of optical remote sensing images and SAR images close to each other, so as to achieve domain adaptive conversion between heterogeneous image features;

[0007] Step 2: Use the difference feature guidance module to explore the relationship between image domain conversion GAN and heterogeneous change detection tasks;

[0008] Step 3: Fusion of salient features and original dual-temporal image features through attention mechanism;

[0009] Step 4: Decode the change information layer by layer to obtain the final change detection result.

[0010] Preferably, in step one, multi-scale features of optical remote sensing images and SAR images are generated through a domain adaptive feature generator module; the domain adaptive feature generator module consists of four convolution modules, and batch normalization and nonlinear activation are performed after each convolution; the step size of the last convolution is set to 2, and the output feature scale is 1 / 2 of the scale of the input feature.

[0011] Preferably, in step one, features of different scales extracted from the optical remote sensing image and the SAR image are input into a discriminator to obtain corresponding identification results; the discriminator performs identification on features at four levels, and each layer of the discriminator includes six convolutions.

[0012] Preferably, in step three, by performing pixel-by-pixel differential calculation on the dual-time image features, salient features for highlighting the changed areas are obtained, and by performing pixel-by-pixel addition on the features, salient features for highlighting the unchanged areas are obtained. The salient feature fusion module fuses the obtained salient features with the original dual-time image features through an attention mechanism.

[0013] Preferably, in step 4, features at different levels are first passed through a multi-scale information extraction module to extract multi-scale change features on features at different levels, and then decoded using multi-scale change features at adjacent levels; by decoding the change information layer by layer, the final change detection result is decoded using the final output of the decoder.

[0014] Preferably, step five is implemented after step four. In step five, when the heterogeneous features are converted to a suitable common domain, a deep association is formed between the feature conversion and the change detection task through a feedback guidance mechanism and a loss function; the generation loss and the discrimination loss in the feature domain adaptive GAN are corrected through the change detection results, forming a feedback guidance mechanism for the change information to guide the fitting of the GAN parameters.

[0015] The beneficial effects of the present invention are:

[0016] 1) This paper constructs a domain-adaptive optical-SAR image heterogeneous change detection framework, develops an improved GAN structure to eliminate the differences in heterogeneous images, and proposes an integrated system architecture that combines domain-adaptive transformation and change detection to achieve high-precision and robust change detection between optical-SAR heterogeneous images.

[0017] 2) The present invention establishes a domain-adaptive optical-SAR image heterogeneous change detection model, which adaptively transforms the dual-time heterogeneous image features into the same subspace. At the same time, through the difference feature guidance module, the information of the discriminator in the domain adaptive GAN is used to guide the change detection, and a feedback guidance mechanism is further designed to make the domain adaptive conversion more accurate through the change detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the multi-source remote sensing change detection method in heterogeneous dual-time image scenes;

[0019] Figure 2 Schematic diagram of the convolution module of the domain adaptive feature generator;

[0020] Figure 3 is the discriminator structure diagram;

[0021] Figure 4 It is a structural schematic diagram of the difference feature guidance module;

[0022] Figure 5 It is a structural diagram of the salient feature fusion module;

[0023] Figure 6 It is a schematic diagram of the structure of the change detection module;

[0024] Figure 7 This is a comparison chart of visualization results on the ShuGuang dataset;

[0025] Figure 8 This is a comparison chart of visualization results on the GloucesterⅠ dataset;

[0026] Fig. 9 This is a comparison chart of visualization results on the GloucesterⅡ dataset;

[0027] Fig.10 This is a comparison chart of the visualization results on the California dataset. DETAILED DESCRIPTION

[0028] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0029] Embodiment 1

[0030] As an embodiment, this optical-SAR heterogeneous remote sensing image change detection method is a multi-source remote sensing change detection method in a heterogeneous dual-time image scene. By mining the deep relationship between image conversion and change detection, the performance of the image conversion module and the change detection module can be mutually enhanced, thereby improving the overall heterogeneous change detection performance.

[0031] This multi-source remote sensing change detection system in heterogeneous dual-time image scenes includes:

[0032] Domain adaptive GAN, including domain adaptive feature generator and discriminator, is used to extract features of different scales from optical remote sensing images and SAR images and realize domain adaptive conversion between heterogeneous image features;

[0033] The difference feature guidance module is used to explore the deeper connection between the image domain conversion GAN network and the heterogeneous change detection task;

[0034] The salient feature fusion module is used to inject different saliencies into the original dual-temporal image features through the attention mechanism;

[0035] The change detection module is used to decode the change information layer by layer and obtain the final change detection result.

[0036] This method is Figure 1 As shown, including:

[0037] Step 1: Establish a domain adaptive feature generator and discriminator module to extract features of different scales for optical remote sensing images and SAR images and realize domain adaptive conversion between heterogeneous image features.

[0038] Step 1.1, DAFGM consists of four convolution modules, in which the convolution module performs four convolutions with a kernel size of 3×3 on the input features to fully extract feature information, and performs batch normalization (BN) and nonlinear activation (ReLU) after each convolution. In order to extract features of different scales, the step size of the last convolution is set to 2, so that the output feature scale becomes 1 / 2 of the scale of the input feature.

[0039] In order to make the model fit better and prevent gradient explosion, a residual structure is added after the second convolution, and the residual feature is made consistent with the output feature size by using maximum pooling.

[0040] Step 1.2: The discriminant obtains a feature with the same size as the discriminator input. In other words, the discriminator evaluates each point on the feature. In order to correspond to the four-layer feature generated by the difference feature guidance module, the discriminator performs discrimination on four levels of features, and each layer of the discriminator consists of six convolutions. The structure of the discriminator is shown in Figure 1. Figure 3 shown.

[0041] Step 2: Explore the deeper connection between the image domain conversion GAN network and the heterogeneous change detection task.

[0042] Considering that in the existing heterogeneous remote sensing image change detection, the use of the discriminator is only reflected in the constraint generator, and the information in the discriminator has not been fully utilized, the present invention constructs a difference feature guidance module to enable the image domain conversion GAN network to better play a role in heterogeneous image change detection. The features are calculated through channel attention to obtain the channel position where the change information is located, and then the plane position where the change information is located is obtained through spatial attention calculation. Through the joint use of channel attention and spatial attention, the change information in the discrimination result is injected into the feature generated by the difference feature guidance module, thereby enhancing the significance of the change information in the feature. Figure 4 shown.

[0043] Step 3: Inject different saliencies into the original dual-temporal image features through the attention mechanism to highlight the salient features of the changed areas.

[0044] like Figure 5 As shown in the figure, the salient features of the highlighted changed areas are obtained by performing pixel-by-pixel difference calculation on the dual-time image features, and the salient features of the highlighted unchanged areas are obtained by performing pixel-by-pixel addition on the features. The salient feature fusion module injects the above different saliencies into the original dual-time image features through the attention mechanism, which also avoids the information redundancy caused by the simultaneous analysis of different salient features.

[0045] Step 4: Decode the change information layer by layer to obtain the final change detection result.

[0046] In step 4, the scales of the changed areas in the remote sensing image vary greatly, and even the same type of changed areas may have huge scale differences. The present invention constructs a change detection module, which first passes the features of different levels through a multi-scale information extraction module to fully mine the multi-scale change information, and then obtains the final change detection result through layer-by-layer decoding of the change information and a change detection module. The specific structure is as follows: Figure 6 shown.

[0047] Embodiment 2

[0048] As another embodiment, this embodiment 2 proposes a more specific change detection method for optical-SAR heterogeneous remote sensing images based on the embodiment 1:

[0049] The expression of step 1.1 is:

[0050]

[0051] Among them, Conv(·) represents the convolution operation with a kernel size of 3×3, a stride and padding of 1, BN(·) represents the batch normalization operation, and ReLU(·) represents the introduction of nonlinear mapping through the ReLU activation function. dwon (·) represents a convolution operation with a kernel size of 3×3, a stride of 2, and a padding of 1. is the feature of the input domain adaptive feature generator module, where t = S, O represents the features from SAR or optical images, and i = 1, 2, 3, 4 represents the different levels of features. base (·) and Conv basedwon (·) are two types of convolutions that make up the convolutional module of the generator, where Conv basedwon (·) The step size is set to 2, so that the output feature scale becomes 1 / 2 of the input feature scale. S (·) and DAFGM O (·) are the difference feature guidance for SAR and optical image features, respectively. With the continuous increase of the level, the extracted feature channels are C = 3, 32, 64, 128, 256, and the feature size is continuously reduced to H = W = 256, 128, 64, 32, 16.

[0052] The expression of step 1.2 is:

[0053]

[0054] Among them, Conv(·) represents the convolution operation with a kernel size of 3×3, a stride and padding of 1, BN(·) represents the batch normalization operation, and ReLU(@) represents the ReLU activation function. Representative features Dis(·) consists of four levels of discriminators, which can generate four different levels of identification results.

[0055] It should be noted that the parts in this embodiment that are the same or similar to those in the first embodiment can be referenced to each other and will not be described in detail in this application.

[0056] Embodiment 3

[0057] Based on the first and second embodiments, the third embodiment proposes a more specific change detection method for optical-SAR heterogeneous remote sensing images:

[0058] The expression of step 2 is:

[0059]

[0060]

[0061] in, and Represent the features extracted by DAFGM and the discrimination results of the discriminator respectively, Sigmoid(·) represents the Sigmoid activation function, SA(·) represents the spatial attention operation, AvePool2d(·) and MaxPool2d(·) represent the spatial average pooling and the spatial maximum pooling respectively, Conv1(·) represents the convolution operation with a kernel size of 3×3, a stride and padding of 1, CA(·) represents the channel attention operation, AvePool1d(·) and MaxPool1d(·) represent the channel average pooling and the channel maximum pooling respectively, and Conv2(·) represents the convolution operation with a kernel and a stride of 1.

[0062] The expression of step three is:

[0063]

[0064] in, and represent dual-temporal features, salient features of unchanged regions, and salient features of changed regions, respectively; SA(·) represents spatial attention operation; CA(·) represents channel attention operation; Conv(·) represents convolution operation with a kernel size of 3×3, a stride and padding of 1; Sigmoid(·) represents nonlinear mapping using the Sigmoid function; SFFM(·) represents the fusion of salient features with dual-temporal image features through the attention mechanism, thereby improving the efficiency of the subsequent change detection module.

[0065] The expression of step 4 is:

[0066] MSCDM(H i )=Re LU(BN(Cat(Conv1(H i ),Conv3(Conv1(H i )),

[0067] Conv5(Conv1(H i )),Conv1(Max2d(H i )))))

[0068] DI i =MSCDM(H i )

[0069] Decoder(DI i ,DI i-1 )=ReLU(BN(Conv(Cat(UP(DI i-1 ),DI i )))

[0070]

[0071] Among them, MSCDM(·) represents the extraction of multi-scale features at different levels of features, H i Represents the i-th level feature output by the SFFM module, DI i represents the multi-scale variation feature of the i-th level, Decoder(·) represents decoding using two-level variation features, Cat(·) represents feature superposition on the channel, UP(·) represents interpolation bilinear upsampling, Conv(·) represents a convolution operation with a kernel size of 3×3, a step size and padding of 1, BN(·) represents batch normalization of features, ReLU(·) represents nonlinear mapping of features through the ReLU activation function, and in the process of decoding variation features by Decoder, when i=2,3, DI i Output the result of the previous decoder, and finally use the final output of the decoder Decode the final change detection result DI.

[0072] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiments 1 to 3 can be referenced to each other and will not be described in detail in this application.

[0073] Embodiment 4

[0074] This fourth embodiment proposes a more specific change detection method for optical-SAR heterogeneous remote sensing images on the basis of the first to third embodiments: there is a fifth step after the fourth step.

[0075] Step 5: When converting heterogeneous features to a suitable common domain, finding a suitable common domain is beneficial to the heterogeneous change detection task. By building a feedback guidance mechanism and guiding the fitting of GAN parameters through change information, a mutually beneficial relationship is established between finding the common domain of features and the heterogeneous change detection task.

[0076] The designed method relies on the ability of GAN style transfer to achieve the search for common domains of heterogeneous features, where the process of finding the common domain is jointly affected by the generator and the discriminator. However, the existence of the change area will cause errors in the search for the intermediate domain. In order to eliminate the influence of the change area, the change results are used to guide the optimization of the generator and the discriminator, so that the common domain search of heterogeneous features is more accurate, thereby improving the accuracy of subsequent change detection. The more accurate the change detection, the more accurately the influence of the change area can be eliminated, which promotes more accurate common domain search, thereby achieving a mutual benefit and a virtuous cycle between the two parts.

[0077] In step 5, the feedback guidance mechanism and loss function setting details are as follows:

[0078]

[0079] in, and Respectively represent the output of the SAR or optical feature of the i-th layer after the discriminator, and They are "true" and "false" to each other, so as to realize the training of GAN without a fixed true value. Di, Zero, One and MSE(·) respectively represent the change detection result map, all-0 matrix, all-1 matrix and mean square error calculation. Through these two formulas, a deep connection is formed between feature conversion and change detection tasks. Di is used to correct the generation loss and identification loss in the proposed feature domain adaptive GAN, avoiding the influence of the change area on the search for the common domain, and realizing the feedback guidance of the change information.

[0080] In step 5, the expression of the change detection loss function is:

[0081] Loss_Change=3·(Cross entropyloss(GT,Di)+Dice loss(GT,Di))

[0082] Among them, Cross entropyloss() represents cross entropy loss, Dice loss() represents segmentation loss, and GT represents label.

[0083] The whole method first optimizes the discriminator parameters under the constraint of Loss_Dis, and the rest of the network parameters are frozen and fixed. Secondly, the parameters of the generator in GAN are updated, that is, the difference feature guidance module parameters are updated under the constraint of Loss_Gan, and the rest of the network parameters are also frozen and fixed. Since there are three parameter update processes in each training of the designed network parameters, the third parameter update uses Loss_Change as a constraint. In order to make the network parameters mainly serve the change detection task, the original change loss is weighted three times.

[0084] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiments 1 to 3 can be referenced to each other and will not be described in detail in this application.

[0085] Embodiment 5

[0086] This fifth embodiment is proposed on the basis of the first to fourth embodiments, and an experimental verification is conducted on the change detection method of optical-SAR heterogeneous remote sensing images:

[0087] In this embodiment, two unsupervised traditional heterogeneous image change detection methods (PCC, IGR-McS), four unsupervised deep learning-based methods (CAN, SCCN, ACE-Net, X-Net) and two supervised deep learning-based methods (M-Unet, DTCDN) are compared with the ChangeGAN method proposed in the present invention.

[0088] Figure 7 , Figure 8 , Fig. 9 , Fig.10 The qualitative results are respectively obtained on the ShuGuang dataset, the GloucesterⅠ dataset, the GloucesterⅡ dataset and the California dataset. The change detection results of the four sets of images using different comparison methods show that the traditional method and the existing deep learning method have certain defects compared with the method of the present invention.

[0089] Figure 7 and Figure 8 It can be seen that PCC and IGR-McS have obvious defects in spatial detail fidelity. CAN, SCCN, ACE-Net, and X-Net methods have made some improvements compared with traditional methods, but there is still spatial distortion. M-Unet and DTCDN methods have made some improvements compared with unsupervised methods in spatial distortion, but there are still false detection and missed detection phenomena. On the contrary, the method proposed in the present invention can retain more spatial information.

[0090] Fig. 9 and Fig.10It can be clearly seen that both the traditional method and the deep learning-based method are difficult to accurately detect the edges and detailed structures of the changing regions in scenes with complex edge changing regions and large changing regions with complex texture structures, while the method proposed in the present invention has achieved good results in both missed detection in the changing regions and misjudgment in the unchanged regions. The comprehensive results show that the domain adaptive optical and SAR remote sensing image change detection network proposed in the present invention can perform well in heterogeneous change detection tasks and has strong practicality.

[0091] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiments 1 to 4 can be referenced to each other and will not be described in detail in this application.

[0092] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

Claims

1. A change detection method for optical-SAR heterogeneous remote sensing images, characterized in that: The following steps are involved: Step 1: Extract features of different scales from optical remote sensing images and SAR images, and use the discriminator to make the multi-scale features of optical remote sensing images and SAR images close to each other, so as to achieve domain adaptive conversion between heterogeneous image features; Step 2: Use the difference feature guidance module to explore the relationship between image domain conversion GAN and heterogeneous change detection tasks; Step 3: Fusion of salient features and original dual-temporal image features through attention mechanism; Step 4: Decode the change information layer by layer to obtain the final change detection result.

2. The change detection method of optical-SAR heterogeneous remote sensing images according to claim 1 is characterized in that: In step 1, multi-scale features of optical remote sensing images and SAR images are generated through the domain adaptive feature generator module; The domain adaptive feature generator module consists of four convolutional modules, and batch normalization and nonlinear activation are performed after each convolution; the stride of the last convolution is set to 2, and the output feature scale is 1 / 2 of the scale of the input feature.

3. The change detection method of optical-SAR heterogeneous remote sensing images according to claim 1 is characterized in that: In step 1, the features of different scales extracted from the optical remote sensing images and SAR images are input into the discriminator to obtain the corresponding identification results; the discriminator performs identification on features at four levels, and each layer of the discriminator includes six convolutions.

4. The change detection method of optical-SAR heterogeneous remote sensing images according to claim 1 is characterized in that: In step three, by performing pixel-by-pixel differential calculation on the dual-time image features, the salient features that highlight the changed area are obtained, and by adding the features pixel-by-pixel, the salient features that highlight the unchanged area are obtained. The salient feature fusion module fuses the obtained salient features with the original dual-time image features through the attention mechanism.

5. The change detection method of optical-SAR heterogeneous remote sensing images according to claim 1 is characterized in that: In step 4, the features of different levels are first passed through the multi-scale information extraction module to extract multi-scale change features on the features of different levels, and then decoded using the multi-scale change features of adjacent levels; by decoding the change information layer by layer, the final change detection result is decoded using the final output of the decoder.

6. The change detection method of optical-SAR heterogeneous remote sensing images according to claim 1, characterized in that: Step 5 is implemented after step 4. In step 5, when the heterogeneous features are converted to a suitable common domain, a deep connection is formed between the feature conversion and the change detection task through the feedback guidance mechanism and the loss function; the generation loss and the discrimination loss in the feature domain adaptive GAN are corrected through the change detection results, forming a feedback guidance mechanism for the change information to guide the fitting of the GAN parameters.

Images