A remote sensing image change detection method based on spatial-spectral feature fusion network

Abstract

The present invention discloses a remote sensing image change detection method based on a spatial-spectral feature fusion network. First, the remote sensing image is preprocessed by geometric correction and image registration; then the training set is input into the DESSN network for training; finally, the test image is input into the DESSN network model to output the segmentation result of dual-phase remote sensing image change detection. The present invention uses an asymmetric dual convolution module combined with Ghost to replace the original dual convolution module in the U‑Net network to enhance the feature learning ability and reduce the number of parameters. A difference enhancement module for suppressing irrelevant changes caused by noise is added after the feature extraction layer to strengthen the focus on the change target; finally, a non-local spatial-spectral information fusion module is designed in the feature fusion stage to enhance the boundary integrity and internal tightness of the change object, and finally realizes high-precision change detection of remote sensing images, which can effectively improve the level of remote sensing image change detection and reduce memory consumption.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology and pattern recognition technology, and relates to theoretical knowledge in the field of deep learning and image segmentation. The present invention specifically relates to a remote sensing image change detection method based on a spatial-spectral feature fusion network. Background Art

[0002] Remote sensing image change detection is the process of identifying the differences in images of a certain area at different times. It is an important branch of remote sensing image analysis and is widely used in urban expansion, land exploration, disaster assessment, environmental monitoring and other fields. With the continuous development of optical sensor equipment, it has become more convenient to obtain remote sensing images. The resolution of the acquired remote sensing images has been continuously improved, providing more and more abundant surface information, breaking through the problem of shortage and low quality of remote sensing data. But on the other hand, with the increase of resolution, the variance between classes becomes larger, and the detection results are greatly affected by noise. Therefore, how to effectively realize the research of ultra-high resolution remote sensing image change detection has become a hot research in the field of remote sensing image analysis.

[0003] Before deep learning technology was used for change detection, methods based on transformation and image algebra, such as principal component analysis, independent component analysis, and change vector analysis, were usually used to implement change detection. The main idea of ​​these methods is to first obtain a difference image, then perform threshold-based or cluster-based pixel classification on the difference image, extract change features, or obtain a change image by maximizing the difference. Since these methods only extract the spectral information of the image and ignore the contextual relationship, they are only applicable to medium and low resolution images. In response to the problem that the texture of objects in high-resolution and ultra-high resolution images is richer and the heterogeneity within the class is enhanced, Tan, Lei, and others proposed a solution. They grouped pixels into objects, then regarded these objects as units, and determined whether they had changed based on their color, shape, and neighborhood information. Although these methods have improved the change detection accuracy of ultra-high resolution remote sensing images to a certain extent, they only use manual features and require complex classifiers, so they have the defects of being sensitive to noise, low detection accuracy, and high computational cost.

[0004] Compared with traditional change detection methods, deep learning methods can make full use of the data diversity of a large number of training samples, automatically learn abstract and multi-level features of complex ground objects, and show strong robustness to noise. Fully Convolutional Networks (FCN) is a classic end-to-end deep learning framework, which usually adopts a multi-level encoder-decoder structure, and the encoder and decoder are usually composed of a large number of standard convolutional and deconvolutional layers. In order to overcome the shortcomings of the original deep learning methods such as large propagation errors and high computational complexity, Peng et al. designed a change detection method based on FCN. Although FCN does not require a fully connected layer, since they mainly use a single-branch network, such methods are prone to loss of original information of dual-phase images. To this end, Daudt et al. introduced twin networks into remote sensing image change detection, making full use of the richer image features in dual-phase images and greatly improving the detection accuracy. Since then, twin networks have been widely used as part of feature extraction in the change detection process. In order to further improve the detection accuracy, some methods introduce long short-term memory networks and recursive neural networks to explore spatiotemporal relationships, and some methods introduce attention mechanisms to improve the performance of change detection by utilizing the importance of difference feature maps and spatial positions. In general, in order to improve the detection accuracy, these methods continue to expand and deepen the network model based on the dual branch, which greatly increases the computational cost and easily causes overfitting problems.

[0005] In the research of change detection of ultra-high resolution remote sensing images, although many effective detection models have emerged, there are still some problems. First, the existing methods cannot effectively construct the relationship between the two-phase images, resulting in the adverse effects of irrelevant changes on the detection results. Second, the boundary integrity and internal tightness of the changed objects are not fully considered, resulting in the loss of edge information in the predicted change map. Third, the existing dual-branch network expands the scale of the model, increases the computational cost, and easily causes overfitting problems.

[0006] In the research of change detection in ultra-high resolution remote sensing images, although many effective detection models have emerged, there are still some problems. First, the existing methods do not fully and effectively construct the relationship between the two-phase images, resulting in the adverse effects of irrelevant changes on the detection results. Second, the boundary integrity and internal tightness of the changed objects are not fully considered, resulting in the loss of edge information in the predicted change map. Third, the existing two-branch network expands the scale of the model, increases the computational cost, and is prone to overfitting problems. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes a remote sensing image change detection method based on a spatial-spectral feature fusion network, which enhances the lightweight difference of remote sensing image change detection and fuses non-local spatial-spectral information, aiming to suppress the influence of non-changing areas on the detection results by using a difference enhancement module; the spatial-spectral information is effectively fused by using a non-local spatial-spectral information fusion module, which not only suppresses the redundancy of features, but also strengthens the long-range relationship between features; the asymmetric dual convolution module combined with Ghost is used to strengthen the extraction of edge information, and greatly reduce the amount of parameters and calculation, finally realizing a lightweight and high-precision change detection method.

[0008] The specific technical solutions of the present invention are as follows:

[0009] A remote sensing image change detection method based on a spatial-spectral feature fusion network. The remote sensing image data set contains a training set, a validation set, and a test set. First, the remote sensing image is preprocessed by geometric correction and image registration; then the training set is input into the DESSN network model for training; finally, the test set image is input into the trained DESSN network model to output the segmentation results of dual-temporal remote sensing image change detection.

[0010] The DESSN network model is a dual-branch U-shaped network, including a difference enhancement module, a non-local spatial-spectral information fusion module, and an asymmetric dual convolution module combined with Ghost; the DESSN network model replaces the convolution in the original U-Net network with an asymmetric dual convolution module combined with Ghost; a difference enhancement module is added after each feature extraction layer of the encoding part of the U-shaped network, and the enhanced difference image is used for jump connection; after the encoder extracts the deep semantic features, the non-local spatial-spectral information fusion module is used to integrate the context information and suppress redundant features, and the feature decoder is used to restore the segmentation map of change detection.

[0011] The asymmetric double convolution module combined with Ghost uses half of the asymmetric double convolution module for feature extraction in each layer of convolution operation, and then uses the obtained feature map to perform linear transformation to obtain a new feature map

[0012] The difference enhancement module uses an attention mechanism on the difference map of the dual-phase feature map in each feature extraction layer, and then adds the obtained weighted difference map to the original feature map.

[0013] The non-local spatial-spectral information fusion module is a multi-scale parallel sampling module embedded in the non-local module.

[0014] The specific implementation steps are as follows:

[0015] Step 1: Dataset preprocessing: Apply geometric correction and image registration preprocessing operations to the remote sensing images to reduce the impact of sunlight angle and atmospheric thickness on the detection results, and then perform image cropping and data enhancement operations;

[0016] Step 2: Training the DESSN network model: Given the operating parameters of the DESSN network model, train the DESSN network model until convergence;

[0017] Step 3: Perform change detection: Input the test set into the DESSN network model to obtain the change detection segmentation results of the dual-temporal remote sensing image.

[0018] In step 1, the data enhancement operation is random flipping and rotation.

[0019] In step 1, the image is cropped to a size of 256×256.

[0020] In step 2, the operating parameters of the DESSN network model are: the learning rate is 1×10 -4 , combined with Ghost’s asymmetric dual convolution module, the compression factor in the Ghost module is set to 2, and the kernel size of the linear operation is set to 3.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] 1. In order to build the relationship between the two-phase images, the difference enhancement module is used to enhance the feature map of the feature extraction layer, so that the network pays more attention to the changed area, thereby suppressing the irrelevant changes caused by noise and better distinguishing the object from the background.

[0023] 2. In order to strengthen the boundary integrity and internal tightness of the changing objects, a non-local spatial-spectral information fusion module is designed and used in the feature fusion stage. On the one hand, the proposed module is different from the conventional spatial-spectral feature fusion because the non-local spatial-spectral information fusion module can provide long-range correlation; on the other hand, it is also different from the conventional non-local module because it also integrates multi-scale spatial information, thereby achieving better feature representation for classifying and locating the changing objects.

[0024] 3. In order to reduce the number of network parameters, an asymmetric dual convolution module combined with Ghost is designed to replace ordinary convolution. Although asymmetric convolution can reduce the number of parameters to a certain extent, it may cause a slight degradation of network performance. Adding asymmetric convolution to ordinary convolution can enhance feature learning ability, but the ability to reduce parameters is limited. Therefore, the proposed asymmetric dual convolution module combined with Ghost combines the advantages of asymmetric convolution and Ghost. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flow chart of the detection method of the present invention.

[0026] Figure 2 It is a schematic diagram of the DESSN network model of the present invention used for change detection of dual-temporal remote sensing images.

[0027] Figure 3 It is the model of the difference enhancement module in the DESSN network model.

[0028] Figure 4 It is the model of the non-local spatial-spectral information fusion module in the DESSN network model.

[0029] Figure 5 It is a model of the asymmetric dual convolution module combined with Ghost in the DESSN network model.

[0030] Figure 6 This is a visual effect diagram obtained by using 8 remote sensing image change detection methods to perform change detection on the public dataset LEVIR-CD.

[0031] Figure 7 This is the output visualization of the last difference enhancement module in the encoding part.

[0032] Figure 8 This is the output visualization effect diagram of the non-local spatial spectrum information fusion module. DETAILED DESCRIPTION

[0033] See also Figure 1 , the figure is a flow diagram of the detection method described in the present invention. In view of the problems of information loss in the process of acquiring difference features, poor integrity of the edges and internal compactness of the detection results, and large models in the existing methods, the present invention designs a lightweight difference enhancement and non-local spatial spectrum information fusion method.

[0034] The specific process of the method of the present invention is as follows:

[0035] (1) Dataset preprocessing: The remote sensing images were first subjected to atmospheric correction, image registration and other preprocessing operations. The obtained data resolution was 0.5 m and the size was 1024 × 1024 pixels. The images were then cropped to a size of 256 × 256. To prevent overfitting, data augmentation operations such as random flipping and rotation were performed.

[0036] (2) Training the DESSN network model: This network training phase uses high-resolution dual-temporal remote sensing images as input. The DESSN network model operating parameters are: the learning rate is 1×10 -4, combined with Ghost’s asymmetric double convolution module, the compression factor in the Ghost module is set to 2, and the kernel size of the linear operation is set to 3. Adam gradient descent is used to optimize the model.

[0037] (3) Change detection of dual-temporal remote sensing images: The test image is input into DESSN, and the change detection segmentation result is output with a size of 256×256.

[0038] The DESSN network model structure is as follows Figure 2 As shown in the figure, starting from the task requirements of change detection, the idea of ​​Siamese network is introduced into the U-Net network to obtain a dual-branch U-shaped network. In order to solve the defects of the current dual-branch network for change detection, the model mainly includes three modules, namely the difference enhancement module, the non-local spatial spectrum information fusion module, and the asymmetric dual convolution module combined with Ghost.

[0039] The specific design of the DESSN network model is as follows:

[0040] a) Difference enhancement module: Difference enhancement module such as Figure 3 Specifically, in order to further effectively construct the relationship between the two-phase images, an attention mechanism is used on the difference map of the two-phase feature map in each feature extraction layer, and then the obtained weighted difference map is added to the original feature map, thereby strengthening the relationship between the changed targets and reducing the impact of the unchanged area on the detection results. The enhanced difference image is used for jump connection to provide more accurate location information for upsampling.

[0041] b) Non-local spatial-spectral information fusion module: The non-local spatial-spectral information fusion module is as follows: Figure 4 As shown in the figure. Specifically, in order to strengthen the boundary integrity and internal tightness of the changing object and reduce the redundancy in feature fusion, a non-local module is proposed. However, the non-local module involves matrix multiplication operations, which requires a large amount of calculation. Therefore, a multi-scale parallel sampling module is designed and embedded in the non-local module, which can not only reduce the computational complexity, but also fuse the spatial information with the spectral information, thereby improving the detection accuracy.

[0042] c) Combined with Ghost's asymmetric double convolution module: Combined with Ghost's asymmetric double convolution module such as Figure 5 As shown in the figure. Specifically, in order to reduce the memory consumption of the model and improve the extraction of edge information, the double convolution in the original U-Net network is replaced by an asymmetric double convolution module. In order to further reduce the number of parameters of the model, the idea of ​​Ghost is introduced. Specifically, in each layer of convolution operation, half of the asymmetric double convolution modules are used for feature extraction, and then the obtained feature map is linearly transformed to obtain a new feature map.

[0043] The effects of the present invention can be further illustrated by the following examples.

[0044] In order to test the accuracy and superiority of the change detection of the present invention, the hardware experimental platform: CPU is Intel Corei9-9900X 10, 3.5GHz, GPU is GeForce RTX 3090Ti, and video memory is 24GB; software platform: DESSN network is implemented under the PyTorch open source framework. 70% of the data is used as a training set, 20% as a validation set, and 10% as a test set. They are cropped into image pairs of size 256×256 by random cropping. In order to enhance the diversity of the data and prevent overfitting, we perform necessary data enhancement operations.

[0045] The present invention evaluates the algorithm performance by calculating the following index parameters: precision (Pre), recall (Rec), and F1 score (F1).

[0046] We first verify the role of the two core modules designed in the DESSN proposed in this invention, the difference enhancement module and the non-local spatial spectrum information fusion module, through ablation experiments. The difference enhancement module is used to avoid the adverse effects of pseudo changes caused by noise, sunlight, etc. Some visualization results are shown in Figure 7 To further verify its effectiveness, we choose the last difference enhancement module for visualization because the last layer contains rich semantic information and shows better performance.

[0047] In the visualization results, you can use colors to distinguish the attention level. Red indicates the area with the highest attention level, and blue indicates the area with the lowest attention level. Figure 7 Each column in is: the previous phase image, the next phase image, the two inputs of the difference enhancement module, the three outputs of the difference enhancement module, and the annotation map.

[0048] exist Figure 7In the figure, (d) and (e) represent the post-temporal images of the input and output of the difference enhancement module, respectively. The enhanced image pays more attention to the building, and the attention area of ​​the difference map (g) is closer to the real change, which is conducive to providing better location information for the decoder and improving the contour accuracy of the changed object. Since the channel fusion of two feature maps easily leads to extremely redundant features, we use a non-local channel attention module to suppress redundancy. Channel attention can obtain the relationship between pixels. When the relationship between some pixels is similar, they can be regarded as one category, which makes it easier and clearer to classify. To further improve the detection accuracy, we use a method that combines spatial and channel information. In addition, in view of the high computational cost of the previous method combining spatial and channel information, we propose a multi-scale parallel sampling module, embed it into the non-local module, and introduce spatial information into the channel attention, which greatly reduces the computational cost and achieves higher accuracy.

[0049] To demonstrate the effectiveness of this module, the qualitative results are shown in Figure 8. Each column in the figure is: the previous phase image, the next phase image, the input of the non-local spatial spectrum information fusion module, the output of the non-local spatial spectrum information fusion module, and the annotation map. We can find that the effect of this module is very obvious. Although it also pays attention to the road, this problem can be solved by combining the difference enhancement module at the same time. Because the skip connection performed on the difference image can provide more detailed information to solve the wrong attention. In this way, the final combined model can get satisfactory results.

[0050] In order to verify the role of the DESSN network proposed in the present invention in high-resolution dual-temporal remote sensing image change detection, we compared the detection results of the DESSN network proposed in the present invention with those of 8 other neural networks used for remote sensing image change detection. Figure 6 The comparison of segmentation results of different detection networks for remote sensing image change detection is shown. The methods in each column of the figure are: pre-phase image, post-phase image, annotation image, U-Net, FC-EF, FC-Siam-conc, FC-Siam-diff, W-Net, DSIFN, FCN-PP, FDCNN, DESSN.

[0051] The quantitative analysis is shown in Table 1 , where the best values ​​are shown in bold.

[0052] Table 1 Change detection results of 9 change detection methods on test images

[0053] Methods OE(%) Pre(%) Rec(%) F1(%) Siam-UNet 0.9637 84.75 89.60 87.11 FC-EF 1.2248 82.27 86.18 84.18 FC-Siam-conc 1.1940 76.17 86.90 81.18 FC-Siam-diff 1.1680 84.44 86.38 85.40 W-Net 1.0233 81.01 87.02 83.91 FCN-PP 1.4631 82.09 84.48 83.27 DSIFN 0.9468 86.00 89.73 87.83 FDCNN 1.2230 83.87 87.56 85.68 ours 0.8899 90.99 91.73 91.36

[0054] In the case of sparse change detection, there will always be some false detection areas, such as Figure 6The first row in (d, e, f, h). Due to the influence of sunlight angle and tree coverage, the results of these methods are blurry in some areas, but in general our network has a considerable advantage in boundary accuracy. The detection of large buildings mainly depends on whether the boundary is smooth and whether there are missed detections and false detections. In the second row of Figure 6, (dk) shows these problems to varying degrees, and our network shows the best performance. Figure 6 In the third row, when the changed buildings are concentrated, (dk) will be stuck and (d, e, f, j, k) will be missing. Our network shows the best performance in terms of the completeness and compactness of the changed objects. To further illustrate the superiority of our method, the quantitative evaluation results are shown in Table 1. Our network obtains the best results, which is 3.53% higher than the highest result in the comparison method. This shows that our network can effectively improve the performance of change detection in high-resolution remote sensing images by refining the edges and strengthening the completeness and internal compactness of the changed objects. In summary, Figure 6 These experimental images show that the DESSN proposed in this paper can better realize change detection of high-resolution remote sensing images.

[0055] In addition, another starting point of the present invention is to reduce the memory consumption of the model. In order to overcome the problem of large amount of calculation when combining channel and spatial attention, a multi-scale parallel sampling module is proposed and embedded in the non-local module, which can not only better express the relationship between features, but also reduce the computational complexity. In order to further reduce the number of parameters of the model, the asymmetric dual convolution module combined with Ghost proposed in the present invention optimizes the extraction of edge information and further greatly reduces the amount of calculation. Table 2 is the result of calculating the following four indicator parameters after using 9 change detection methods to perform change detection on the test image: floating point operations (FLOPs), number of parameters (Params), storage usage of the model (Storage Usage) and F1 score (F1).

[0056] The comparison results of the performance of the present invention are shown in Table 2. Overall, our model compression is effective.

[0057] Table 2 Results of change detection index parameters of the 9 change detection methods on the test images

[0058] Methods FLOPs (G) Params (M) Storage usage (MB) F1 (%) Siam-UNet 80.65 39.70 151.51 87.11 FC-EF 2.63 0.85 3.34 84.18 FC-Siam-conc 4.07 1.07 4.08 81.18 FC-Siam-diff 3.47 0.85 3.33 85.40 W-Net 95.80 31.57 120.48 83.91 FCN-PP 34.65 28.13 107.39 83.27 DSIFN 112.15 43.50 116.92 87.83 FDCNN 32.40 1.86 7.09 85.68 ours 36.75 19.35 73.95 91.36

[0059] Through Figure 5 , Figure 6 , Figure 7 , Figure 8,From the comprehensive analysis of Table 1 and Table 2, the DESSN proposed in the ,presentation can achieve higher accuracy remote sensing image change ,detection, and the model requires less memory consumption, and has ,broad application prospects in remote sensing image analysis.

Claims

1. A remote sensing image change detection method based on a spatial-spectral feature fusion network, characterized in that: The remote sensing image dataset contains training set, validation set and test set. First, the remote sensing image is preprocessed by geometric correction and image registration. Then the training set is input into the DESSN network model for training. Finally, the test set image is input into the trained DESSN network model to output the segmentation results of dual-temporal remote sensing image change detection. The DESSN network model is a dual-branch U-shaped network, including a difference enhancement module, a non-local spatial spectrum information fusion module, and an asymmetric dual convolution module combined with Ghost; the DESSN network model replaces the convolution in the original U-Net network with an asymmetric dual convolution module combined with Ghost; a difference enhancement module is added after each feature extraction layer of the encoding part of the U-shaped network, and the enhanced difference image is used for jump connection; after the encoder extracts the deep semantic features, the non-local spatial spectrum information fusion module is used to integrate the context information and suppress the redundant features, and the segmentation map of the change detection is restored through the feature decoder; The asymmetric double convolution module combined with Ghost uses half of the asymmetric double convolution module to extract features in each layer of convolution operation, and then uses the obtained feature map to perform linear transformation to obtain a new feature map; The difference enhancement module uses an attention mechanism on the difference map of the bi-phase feature map in each feature extraction layer, and then adds the obtained weighted difference map to the original feature map; The non-local spatial-spectral information fusion module is a multi-scale parallel sampling module embedded in the non-local module.

2. The remote sensing image change detection method based on spatial-spectral feature fusion network according to claim 1 is characterized in that: The specific implementation steps are as follows: Step 1: Dataset preprocessing: Apply geometric correction and image registration preprocessing operations to the remote sensing images to reduce the impact of sunlight angle and atmospheric thickness on the detection results, and then perform image cropping and data enhancement operations; Step 2: Training the DESSN network model: Given the operating parameters of the DESSN network model, train the DESSN network model until convergence; Step 3: Perform change detection: Input the test set into the DESSN network model to obtain the change detection segmentation results of the dual-temporal remote sensing image.

3. The remote sensing image change detection method based on spatial-spectral feature fusion network according to claim 2 is characterized in that: In step 1, the data enhancement operation is random flipping and rotation.

4. The remote sensing image change detection method based on spatial-spectral feature fusion network according to claim 3 is characterized in that: In step 1, the image is cropped to a size of 256×256.

5. The remote sensing image change detection method based on spatial-spectral feature fusion network according to claim 4 is characterized in that: In step 2, the operating parameters of the DESSN network model are: the learning rate is 1×10 -4 , combined with Ghost’s asymmetric dual convolution module, the compression factor in the Ghost module is set to 2, and the kernel size of the linear operation is set to 3.

Images