An unsupervised change detection method for heterogeneous remote sensing images

Abstract

The invention discloses an unsupervised change detection method for heterogeneous remote sensing images. Based on the domain conversion mode of Cycle GAN, the heterogeneous images are converted into the same feature space through a style transfer network, and a lightweight attention mechanism is added to the generator and the discriminator of Cycle GAN to accelerate the convergence of the model and effectively complete the conversion of the heterogeneous space. The lightweight attention mechanism realizes the effective conversion of the image space domain from low dimension to high dimension. Compared with other existing methods, under the condition of only adding a small amount of model parameters, a large amount of high-dimensional feature information is provided for generating a difference map. The Euclidean distance between dual-phase images in the same feature space is calculated channel by channel, which is called a feature difference matrix, and then the maximum value of the channel dimension is taken as the difference map. Finally, a change detection map is generated through a series of threshold calculations and denoising operations, thereby enhancing the accuracy and completeness of the generated change map and realizing the effective improvement of the change detection accuracy. In addition, the heterogeneous change detection framework proposed by the invention is unsupervised, thereby avoiding a large amount of time and energy spent on collecting labeled data in supervised learning.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image change detection, and in particular relates to an unsupervised change detection method for heterogeneous remote sensing images. Background Art

[0002] As an important method in the field of remote sensing, change detection aims to study remote sensing images acquired at different times in a specific geographical location and identify the changed area by analyzing the ground object information (Gong et al., 2019). With the rapid development of remote sensing science and technology, real-time images of the earth's surface can be easily obtained, which makes CD technology more widely used, such as urban change monitoring (Wu et al., 2021), natural disaster assessment (Qiao et al., 2020) and geological resource survey.

[0003] According to the type of data source, change detection methods can be divided into homogeneous and heterogeneous change detection. Homogeneous change detection relies on homogeneous data, that is, a set of images acquired by the same or similar sensors. Heterogeneous data means that two images have different domains, different statistical distributions, and inconsistent category features, which makes it more difficult to use heterogeneous images for change detection tasks than homogeneous change detection. At present, driven by a large number of homogeneous remote sensing data samples, homogeneous change detection based on deep learning has achieved competitive results. However, in practical applications, the acquisition speed of multi-temporal homogeneous images is limited, and it is difficult to detect and effectively evaluate changes caused by emergencies and natural disasters in real time. The more convenient accessibility of heterogeneous data provides a solution to this problem. Therefore, change detection methods based on heterogeneous remote sensing images are particularly important in dealing with real-time tasks.

[0004] At present, domain conversion methods based on deep learning are commonly used to perform change detection in dual-temporal heterogeneous remote sensing images. The domain conversion method converts dual-temporal heterogeneous images into the same feature space, and then generates the final change map based on the homogeneous change detection method. Among the existing heterogeneous change detection methods based on domain conversion, the commonly used models are mainly autoencoders, generative adversarial networks and their variant networks.

[0005] One of the most important methods in the literature on domain adaptation and data conversion is the Generative Adversarial Network (GAN) (Goodfellow et al., 2014), which is an unsupervised deep learning model that achieves effective transformation of two feature spaces through the mutual game between the generator and the discriminator. In 2019, Gong et al. proposed a coupled transformation network (CPTN) based on GAN to transform heterogeneous images into homogeneous images and then detect changes in their observation space. In 2018, Niu et al. used conditional generative adversarial networks (cGAN) to transform heterogeneous synthetic aperture radar (SAR) and optical images into a space where their information has a more consistent representation for change detection. In 2021, Luppino et al. proposed a new heterogeneous change detection network architecture Adversarial Cyclic Encoder Network (ACE-Net) based on CycleGAN, which ensures the correlation between the latent space and the original space by adding a cycle consistency loss.

[0006] The existing methods have the following two main shortcomings: First, the domain conversion between heterogeneous images in the existing methods is only applicable to simple data, such as visible light, infrared and SAR. To perform domain conversion on multi-spectral heterogeneous images, a more complex network must be designed, otherwise it is difficult to fully obtain the potential change details. Second, almost all existing methods directly compare the converted heterogeneous images and generate the final change map through threshold segmentation. There is a lack of effective processing of the noise and errors caused by domain conversion, which makes the change detection area not accurate and compact enough. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide an unsupervised change detection method for heterogeneous remote sensing images, which provides a large amount of high-dimensional feature information for generating difference maps under the condition of only adding a small amount of model parameters.

[0008] An unsupervised change detection method for heterogeneous remote sensing images, comprising:

[0009] Cycle GAN is used to transform heterogeneous remote sensing images. Channel attention module and spatial attention module are added to the generator and discriminator sub-networks of Cycle GAN network. Specifically:

[0010] In the residual module of the generator, the input features are distributed to the first branch and the second branch. In the first branch, the input features are successively sent to the first normalization and activation function layer after passing through the first convolution layer and a channel attention module, and then output; in the second branch, the input features are successively sent to the second normalization and activation function layer after passing through the second convolution layer and a spatial attention module, and then output; the outputs of the first branch and the second branch are added, and then successively sent to the third convolution layer and the normalization layer for output;

[0011] In the discriminator, the image output by the generator is processed by the fourth and fifth convolution layers and then sent to a channel attention module. It is then processed by the third normalization and activation function layer and the sixth convolution layer and then sent to a spatial attention module. The output result is processed by the fourth normalization and activation function layer and the seventh convolution layer, and then sent to a channel attention module. The output is then processed by the fifth normalization and activation function layer and the eighth convolution layer, and finally sent to a spatial attention module to output the discrimination result.

[0012] The channel attention module first averages each channel in the input feature block and compresses it into a C×1×1 vector, then forms a channel weight coefficient after a one-dimensional convolution operation and an activation function, and finally multiplies the input feature by the weight coefficient in the channel dimension to obtain a globally enhanced channel attention feature map; wherein C represents the number of channels of the input heterogeneous remote sensing image;

[0013] The spatial attention module first performs a maximum pooling operation on each spatial element in the input feature block according to the channel dimension, compresses the feature block into a 1×W×H feature map, and then forms a spatial weight coefficient after a two-dimensional convolution operation and activation function processing; each position pixel in the input feature is multiplied by the spatial weight coefficient to obtain a locally enhanced spatial attention feature map; wherein W and H represent the length and width of the input heterogeneous remote sensing image, respectively.

[0014] Furthermore, suppose two heterogeneous remote sensing images are image A and image B. After the Cycle GAN network transforms image A into an image in the domain where image B is located, the Euclidean distance of the corresponding spatial pixels with image B is calculated according to the channel dimension. Then, the maximum Euclidean distance of each spatial pixel in each channel latitude is taken as the final Euclidean distance of the spatial pixel, thereby obtaining a single-channel difference map; finally, the single-channel difference map is converted into a binary image.

[0015] Preferably, a threshold segmentation method is used to convert the single-channel difference map into a binary map.

[0016] Preferably, the maximum inter-class variance method (OTSU) is used to determine the threshold, and then the single-channel difference map is converted into a binary map using a threshold segmentation method.

[0017] Preferably, filtering is performed on the binary image.

[0018] Preferably, a discrete cosine transform (DCT) filter is used to filter the binary image.

[0019] The present invention has the following beneficial effects:

[0020] Based on the domain conversion method of Cycle GAN, the present invention proposes a change detection method for dual-temporal heterogeneous remote sensing images. The heterogeneous images are converted to the same feature space through a style transfer network. A lightweight attention mechanism is added to the generator and discriminator of Cycle GAN to accelerate the convergence of the model and effectively complete the conversion of the heterogeneous space. The lightweight attention mechanism realizes the effective conversion of the image space domain from low dimension to high dimension. Compared with other existing methods, it provides a large amount of high-dimensional feature information for generating difference maps under the condition of only adding a small number of model parameters.

[0021] The Euclidean distance between the dual-phase images in the same feature space is calculated channel by channel, which is called the feature difference matrix. Then the maximum value of the channel dimension is taken as the difference map. Finally, a series of threshold calculations and denoising operations are performed to generate a change detection map, which enhances the accuracy and completeness of the generated change map and effectively improves the change detection accuracy.

[0022] In addition, the heterogeneous change detection framework proposed in the present invention is unsupervised, which avoids the large amount of time and effort spent on collecting labeled data in supervised learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a basic flow chart of the heterogeneous change detection method proposed by the present invention;

[0024] Figure 2 The attention module is used in the image conversion network, (a) is the channel attention module, (b) is the spatial attention module;

[0025] Figure 3 It is the residual module based on the attention mechanism in the generator;

[0026] Figure 4 It is the discriminator structure based on the attention mechanism;

[0027] Figure 5 The change detection results of different methods on the Shuguang dataset, (a) is the image at time T1, (b) is the image at time T2, (c) is the change reference map, (d) is the SCCN result, (e) is the NPSG result, (f) is the INLPG result, (g) is the CAA result, and (h) is the result of the method of the present invention;

[0028] Figure 6 The change detection results of different methods on the Texas dataset, (a) is the image at time T1, (b) is the image at time T2, (c) is the change reference map, (d) is the SCCN result, (e) is the NPSG result, (f) is the INLPG result, (g) is the CAA result, and (h) is the result of the present invention. DETAILED DESCRIPTION

[0029] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0030] The present invention provides an unsupervised change detection method for remote sensing heterogeneous images based on image conversion network and post-processing algorithm. Firstly, the heterogeneous images are converted to the same spatial domain through the improved Cycle GAN network, and then the errors are eliminated through post-processing steps to generate an accurate and complete change map. Figure 1 Shown is a basic flow chart of the present invention.

[0031] 1. Domain Conversion Steps

[0032] The present invention improves the domain conversion network based on the Cycle Generative Adversarial Network (Cycle GAN). Cycle GAN has two sub-networks: the generator and the discriminator. Since the proposed method is unsupervised and the data is small, it is difficult to ensure that the conversion of the entire feature domain is fully realized in CycleGAN. Therefore, the present invention adds a lightweight attention module to the two sub-networks to improve the efficiency of domain conversion and accelerate the convergence of the model.

[0033] The present invention designs an attention module from two dimensions, channel and space, and applies it to the generator and discriminator of the above-mentioned conversion network to achieve a more accurate domain conversion effect. Figure 2 Shown is the proposed attention module structure for the domain conversion network, (a) is the channel attention module, and (b) is the spatial attention module.

[0034] Channel attention aims to extract important information in each channel of the feature map. First, each channel in the feature block is averaged and pooled to compress the original feature into a C×1×1 vector. Then, after a one-dimensional convolution operation and an activation function, a channel weight coefficient is formed. The original feature is multiplied by the channel dimension and the weight coefficient to obtain a globally enhanced channel attention feature map (M C ), the expression is as follows:

[0035]

[0036] Among them, C represents the number of channels of the image, F is the original feature map, represents channel-by-channel multiplication, σ(·) is the sigmoid activation function, conv1d and Avgpool represent one-dimensional convolution and average pooling operations, respectively.

[0037] Spatial attention aims to extract important information of each pixel space in the feature map. First, the maximum pooling operation is performed on the spatial elements in each feature block according to the channel dimension, and the feature block is compressed into a 1×W×H feature map. Then, after a two-dimensional convolution operation and an activation function, a spatial weight coefficient is formed. The local enhanced spatial attention feature map (M) is obtained by multiplying each position pixel in the original feature with the corresponding spatial weight coefficient. S ), the expression is as follows:

[0038]

[0039] Where W and H represent the length and width of the image respectively; Represents element-wise multiplication, conv2d and Maxpool represent two-dimensional convolution and maximum pooling operations respectively. The channel and spatial attention designed by the present invention use average pooling and maximum pooling methods to reduce the dimensions of space and channels respectively. The two modules obtain the change information after global enhancement and local enhancement respectively, and perform cascade operations in different forms in the generator and discriminator to achieve better image conversion effect.

[0040] The present invention adds the above-mentioned spatial and channel attention to the generator and discriminator in Cycle GAN respectively, to improve the authenticity of the generated fake samples and the accuracy of the discrimination classification. The residual module in the generator is the key to achieve feature conversion. The residual block is the basic component unit of the ResNet skeleton (He et al., 2016), which is divided into two parts: direct mapping and residual. The present invention adds channel attention and spatial attention to these two places respectively, and then adds the two parts and sends them to the next residual block for feature conversion again, thereby forming a residual module based on the attention mechanism, whose structure is as follows Figure 3 By fusing the feature maps that have been channelized and spatially focused, we can take both global and local information into account, and accelerate the convergence of the generator model while accurately completing the image conversion. The specific solution is as follows:

[0041] The original residual module is divided into two branches. The main path is the feature obtained after the convolution layer, normalization and activation function, and the branch connects the original feature with the main path to finally obtain the residual feature. The present invention changes the structure of the residual module, adds the input features after passing through the two networks of "convolution (conv1) + channel attention (CAM)" and "convolution (conv1) + spatial attention (SAM)", and then normalizes after passing through the second convolution block (conv2) to obtain new output features.

[0042] The role of the discriminator is to take an image as input and predict whether it is the original image or the output image of the generator. Figure 4 The figure shows the proposed attention-based discriminator network, which extracts features from the input image through five convolutional layers, two channel attention modules and two spatial attention modules, as well as normalization and activation functions in series, thereby determining whether the image belongs to a specific category (real image or generated image).

[0043] Improvement of the discriminator based on the attention mechanism: In order to improve the efficiency of the discriminator, the present invention adds two channel attention modules (CAM) and two spatial attention modules (SAM) to the original discriminator. The specific structure is as follows:

[0044] 1) After the input image passes through two convolution blocks (conv1, conv2), channel attention (CAM1) is added, and then normalization and activation operations are performed;

[0045] 2) After the convolution operation (conv3), spatial attention (SAM1) is added, and then normalization and activation operations are performed;

[0046] 3) After the fourth convolution block (conv4), channel attention (CAM2) is added again for normalization and activation operations;

[0047] 4) After the fifth convolution block (conv5), spatial attention (SAM2) is added again, and the discrimination result is finally output.

[0048] 2. Post-processing steps

[0049] In the conversion network proposed in the present invention, the present invention converts two heterogeneous images from a feature domain with a lower dimension to a feature domain with a higher latitude, and then obtains the change area in the high-dimensional feature space. Since the high-dimensional feature domain contains more information and helps to detect more changing features, the high-dimensional feature domain in the heterogeneous image pair is selected as the common domain for subsequent change detection. Although the above-mentioned conversion network can convert heterogeneous images into similar feature domains, their pixel differences may still be large, and unnecessary noise will be introduced in the conversion process. Therefore, in order to obtain an accurate heterogeneous change map, the present invention proposes a post-processing algorithm, which is divided into three steps (such as Figure 1 shown).

[0050] 1) Calculate the difference map: Calculate the Euclidean distance of the corresponding spatial position between the image converted to domain B and the real image in domain B according to the channel dimension, and then take the maximum distance in each channel as the difference value of the spatial pixel point to generate a single-channel difference map (such as Figure 1 (as shown in (a)).

[0051] 2) Threshold segmentation: Determine the segmentation threshold of pixel intensity, retain only the pixels whose intensity value is greater than the threshold in the difference map, and convert the difference map into a binary change map (such as Figure 1 (b)). The threshold determination method of the present invention adopts the maximum between-class variance method (OTSU). Compared with other segmentation methods, OTSU is simple to calculate, is not affected by image brightness and contrast, does not require additional parameters, and has excellent segmentation effect.

[0052] 3) Reduce false detection noise: Since the change area of ​​interest in change detection is usually large, the present invention removes isolated small points in the binary change map by filtering out high-frequency noise to generate a more accurate final change map. Figure 1 (c)). Here, the present invention uses a filter based on discrete cosine transform (DCT) to remove the noise in the non-changing area of ​​the binary change image. The DCT-based filtering method has a good noise suppression effect in the homogeneous area of ​​the image, and can retain the edge and detail features, and the DCT transform has a fast calculation speed. Therefore, for image noise filtering in which the noise-related parameters are unknown and in a non-stationary state, the denoising technology using DCT transform can achieve better results.

[0053] The pseudo code of the above heterogeneous change detection post-processing algorithm is summarized in Algorithm 1, where A and B represent the spatial domains of the heterogeneous dual-phase images, W, H, and C are the length, width, and number of channels of the image, respectively, and patch is an image block determined according to the DCT coefficients.

[0054] To verify the effectiveness of the unsupervised heterogeneous change detection method proposed in this paper, experiments were conducted on two sets of heterogeneous change detection datasets and compared with other popular methods.

[0055] (1) Shuguang Dataset Experiment

[0056] Figure 5 The change detection results obtained by different methods on the Shuguang dataset, (a) and (b) are dual-phase heterogeneous images, (c) is the change reference map, (d)-(h) are the change maps obtained by SCCN, NPSG, INLPG, CAA and this method. By comparison, it can be seen that the results of this method ( Figure 5 (h) There are fewer missed detections and false detections for changed pixels, and the obtained changed region has complete boundaries and compact interiors. The consistency between the change map and the reference map is optimal among the above methods.

[0057] In order to quantitatively evaluate the performance of the detection results of the above methods, Table 1 shows the evaluation indicators of the experimental results of different methods on the Shuguang dataset. The proposed method achieved the best detection performance with the highest Pr (0.8115), F1-Score (0.8314) and Kappa (0.8241). Compared with the second-best method INLPG, the proposed method achieved an improvement of 9.02% and 9.35% in the comprehensive indicators F1-Score and Kappa, respectively.

[0058] Table 1 Quantitative results of different methods on the Shuguang dataset

[0059]

[0060]

[0061] (2) Texas Dataset Experiment

[0062] Figure 6 The change detection results obtained by different methods on the Texas dataset, (a) and (b) are dual-phase heterogeneous images, (c) is the change reference map, (d)-(h) are the change maps obtained by SCCN, NPSG, INLPG, CAA and this method. By comparison, we can see that Figure 6 As shown in (h), the change area detected by the method of the present invention is most similar to the reference image, and the change detection effect on the Texas multispectral heterogeneous data set is optimal.

[0063] In order to quantitatively evaluate the performance of the detection results of the above methods, Table 2 shows the evaluation indicators of the experimental results obtained by different methods on the Texas dataset. This method achieved the best detection performance with the highest Pr (0.9098), F1-Score (0.9075) and Kappa (0.8965). On the Texas dataset, the present invention found that the deep learning-based methods SCCN, CAA and the method of the present invention all showed good performance, while the graph methods NPSG and INLPG based on structural consistency did not perform well for the change detection of the multi-spectral heterogeneous dataset Texas. Compared with the suboptimal method SCCN, this method achieved an improvement of 3.96% and 4.39% in the comprehensive indicators F1-Score and Kappa, respectively.

[0064] Table 2 Quantitative results of different methods on the Texas dataset

[0065]

[0066] In order to further verify the contribution of the attention mechanism and post-processing algorithm proposed in the present invention to the experimental results, the present invention uses the model without attention module and post-processing algorithm, with attention module, with post-processing algorithm and with both attention module and post-processing algorithm to train and test on two data sets, and calculates the Kappa index of each model, as shown in Table 3, in which the Kappa index is used. It can be seen that after adding the attention module and post-processing algorithm, the change detection performance of the model has been significantly improved.

[0067] Table 3 Ablation experiment results of the attention module and post-processing algorithm of the present invention

[0068]

[0069]

[0070] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An unsupervised change detection method for heterogeneous remote sensing images, characterized in that: include: Cycle GAN is used to transform heterogeneous remote sensing images. Channel attention module and spatial attention module are added to the generator and discriminator sub-networks of Cycle GAN network. Specifically: In the residual module of the generator, the input features are distributed to the first branch and the second branch. In the first branch, the input features are successively sent to the first normalization and activation function layer after passing through the first convolution layer and a channel attention module, and then output; in the second branch, the input features are successively sent to the second normalization and activation function layer after passing through the second convolution layer and a spatial attention module, and then output; the outputs of the first branch and the second branch are added, and then successively sent to the third convolution layer and the normalization layer for output; In the discriminator, the image output by the generator is processed by the fourth and fifth convolution layers and then sent to a channel attention module. It is then processed by the third normalization and activation function layer and the sixth convolution layer and then sent to a spatial attention module. The output result is processed by the fourth normalization and activation function layer and the seventh convolution layer, and then sent to a channel attention module. The output is then processed by the fifth normalization and activation function layer and the eighth convolution layer, and finally sent to a spatial attention module to output the discrimination result. The channel attention module first averages each channel in the input feature block and compresses it into a C×1×1 vector, then forms a channel weight coefficient after a one-dimensional convolution operation and an activation function, and finally multiplies the input feature by the weight coefficient in the channel dimension to obtain a globally enhanced channel attention feature map; wherein C represents the number of channels of the input heterogeneous remote sensing image; The spatial attention module first performs a maximum pooling operation on each spatial element in the input feature block according to the channel dimension, compresses the feature block into a 1×W×H feature map, and then forms a spatial weight coefficient after a two-dimensional convolution operation and activation function processing; each position pixel in the input feature is multiplied by the spatial weight coefficient to obtain a locally enhanced spatial attention feature map; wherein W and H represent the length and width of the input heterogeneous remote sensing image, respectively.

2. The unsupervised change detection method for heterogeneous remote sensing images according to claim 1, characterized in that: Suppose there are two heterogeneous remote sensing images, image A and image B. After the Cycle GAN network transforms image A into an image in the domain where image B is located, the Euclidean distance of the corresponding spatial pixels with image B is calculated according to the channel dimension. Then, the maximum Euclidean distance of each spatial pixel in each channel latitude is taken as the final Euclidean distance of the spatial pixel, thereby obtaining a single-channel difference map; finally, the single-channel difference map is converted into a binary image.

3. The unsupervised change detection method for heterogeneous remote sensing images according to claim 2, characterized in that: The threshold segmentation method is used to convert the single-channel difference map into a binary image.

4. The unsupervised change detection method for heterogeneous remote sensing images according to claim 3, characterized in that: The maximum inter-class variance method (OTSU) is used to determine the threshold, and then the threshold segmentation method is used to convert the single-channel difference map into a binary map.

5. The unsupervised change detection method for heterogeneous remote sensing images according to claim 3 or 4, characterized in that: The binary image is subjected to filtering processing.

6. The unsupervised change detection method for heterogeneous remote sensing images according to claim 5, characterized in that: The binary image is filtered using a discrete cosine transform (DCT) filter.

Images