A twin network and method for change detection of remote sensing images with ultra-high spatial resolution

Abstract

The present invention relates to the field of image processing technology, and in particular to a twin network and method for change detection of ultra-high spatial resolution remote sensing images. The present invention provides a change detection network MDSANet, which uses an asymmetric convolution residual block to extract deep features with more discriminative change information. The fused multi-scale difference features are enhanced in the spatiotemporal domain using a spatiotemporal attention mechanism to enhance the areas of real changes while suppressing areas of unreal changes. At the same time, the correlation between channels of feature maps of different scales is established to improve the characterization ability of features of different scales. The present invention performs change detection on high spatial resolution remote sensing images, can effectively extract real changes in land objects, and eliminates the influence of external factors such as light and season, reducing the false detection rate of pseudo changes.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a twin network and method for ultra-high spatial resolution remote sensing image change detection. Background Art

[0002] Change detection (CD) refers to the detection and analysis of changes that occur in the same geographic space at different times. Usually, the detected changes are real changes, not pseudo changes caused by sensor factors, seasonal light changes, etc. Some traditional change detection methods process and analyze pixels separately, ignoring the spatial and semantic information of the objects, and it is difficult to overcome the interference of noise, resulting in false detection and missed detection problems in the change detection results. In recent years, convolutional neural networks (CNNs) have gradually been widely used in the field of change detection due to their powerful nonlinear representation ability and generalization. In order to further improve the accuracy and robustness of change detection, a large number of change detection models have been built on the basis of CNN, and various mechanisms have been explored to optimize the performance of change detection networks. For example, dilated convolution is introduced in the change detection model to expand the receptive field and obtain a richer global feature representation; the attention mechanism is used to enhance the features of the real change area while suppressing the features of the non-real change area; or the self-attention mechanism is used to capture the global spatial context information and establish the spatial correlation between pixels.

[0003] At present, the change detection methods based on CNN can be roughly divided into architectures based on single-branch, dual-branch encoding and dual-branch encoder-decoder according to their network architectures. The change detection network based on the single-branch architecture uses early fusion to fuse the dual-phase images and input them into the single-branch encoder and decoder to detect the changed area. Peng et al. proposed a dense attention single-branch architecture change detection model to establish the correlation between low-level features and deep features, and used the difference enhancement module to further eliminate the influence of factors such as sensors and seasonal illumination. The change detection network based on the dual-branch encoding architecture uses a twin network structure to extract the deep features of the dual-phase original image, and uses a mid-term fusion method to fuse the deep image features to detect the changed area. Zhang et al. proposed an image fusion network based on deep supervision to fuse the difference features and the original image features. Lei et al. proposed a difference enhancement twin network architecture, and used a non-local pyramid attention model to enhance the fused deep features. Zhang et al. used VGG16 to extract multi-scale features of the dual-phase original image, and superimposed the multi-scale features along the channel direction to achieve the fusion between fine-grained and coarse-grained features. Zhang et al. designed a difference-enhanced twin change detection network, which performs difference enhancement on deep features and shallow features respectively. Based on the change detection network with a dual-branch encoder-decoder architecture, both encoding and decoding adopt a dual-branch architecture, and the decoder features are fused by late fusion to detect the changed area. Zheng et al. designed a multi-task difference-enhanced twin change detection network, and used a combination of feature difference and feature concatenation to enhance the difference features.

[0004] The existing Chinese patent document CN111539316A discloses a high-resolution remote sensing image change detection method based on a dual-attention twin network, which uses the twin spatial attention mechanism and channel attention mechanism to obtain highly discriminative feature expressions, and uses a dual threshold loss function to further screen effective features, effectively filtering out remote sensing image changes with pseudo-change information, thereby obtaining better high-resolution remote sensing image change detection results. Although the patent document CN111539316A uses an attention mechanism to obtain highly discriminative feature expressions, and uses a heavy threshold loss function to further screen effective features. However, it does not use multi-scale encoder features to reconstruct the changed area, and cannot solve the problems of small target loss and edge blur.

[0005] In order to make full use of the multi-scale difference features, the present invention stacks the difference features of different scales along the channel direction, and uses the spatiotemporal attention mechanism to enhance the multi-scale difference features in the spatiotemporal domain to enhance the features of the real change area and suppress the features of the irrelevant area. At the same time, the correlation between the channels of the different scale difference feature maps is established to improve the representation ability of the different scale difference features. Since the CNN change detection model will cause the loss of image detail information during the feature extraction and downsampling process, most change detection models use a jump connection to fuse the feature map after the decoder is upsampled with the feature map of the same scale encoder in the feature decoding and reconstruction change map to restore the detail information of the change area. However, there is a semantic gap between the feature map after the decoder is upsampled and the feature map of the same scale encoder, in order to further eliminate the semantic difference between the two. The present invention designs a feature fusion module to fuse the enhanced difference features with the dual-phase encoder features of the same scale. Summary of the invention

[0006] The purpose of the present invention is to provide a twin network and method for ultra-high spatial resolution remote sensing image change detection, which is suitable for remote sensing image analysis and processing for ultra-high spatial resolution remote sensing image change detection; to solve the problem of pseudo-changes caused by external factors such as lighting and seasons. The present invention uses asymmetric convolution to extract deep features and enhance edge areas. It also uses spatiotemporal attention to enhance multi-scale difference features, and uses layer-by-layer decoding to restore the changed areas. Compared with the prior art, the present invention effectively alleviates the problems of small target loss and edge blur.

[0007] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0008] A twin network for change detection of remote sensing images with ultra-high spatial resolution, wherein the twin network is a change detection network MDSANet, comprising an asymmetric convolutional residual encoder, a multi-scale difference feature spatiotemporal attention enhancement module and a feature fusion decoder.

[0009] Furthermore, the deep feature extraction is performed through the asymmetric convolution residual encoder, and then the difference features of different scales are stacked along the channel direction, and the spatiotemporal attention mechanism is used to enhance the multi-scale difference features in the spatiotemporal domain to improve the representation ability of features of different scales. The multi-scale difference features after spatiotemporal domain enhancement are split according to the number of channels of different scales, and the enhanced difference features are fused with the dual-phase encoder features of the same scale through the feature fusion decoder.

[0010] On the other hand, the present invention provides a method for detecting changes in remote sensing images with ultra-high spatial resolution, comprising the following steps:

[0011] S1: Assuming that the size and number of channels of the two-phase images I1 and I2 are H×W×C0, multi-scale features are extracted by inputting them into the asymmetric convolutional residual encoder with shared weights. for i=1,2. In calculating multi-scale difference features

[0012] S2: df 1 ,df 2 ,df 3 Downsample by 2×, 4×, and 8× to convert the spatial size to H / 8×W / 8, and then convert the downsampled df 1 ,df 2 ,df 3 and df 4 The multi-scale difference fusion feature ff is obtained by concatenation, and its spatial size and number of channels are H / 8×W / 8×C. The spatiotemporal attention mechanism (CBAM) is used to enhance ff in the spatiotemporal domain. After enhancement, the multi-scale difference fusion feature eff is split into channels to obtain the enhanced multi-scale difference features EDF at different scales = {edf 1 ,edf 2 ,edf 3 ,edf 4}.

[0013] S3: CBAM consists of spatial attention (SAM) and channel attention (CAM); its spatial attention uses the spatial relationship between features to generate a spatial attention map, and the channel attention uses the channel relationship between features to generate a channel attention map. The present invention first concatenates the acquired multi-scale difference maps to obtain a concatenated multi-scale difference map ff to establish the channel correlation of different scale difference features. Secondly, ff is enhanced in the channel domain and the spatial domain to obtain an enhanced multi-scale difference map eff, thereby improving the expression ability of different scale features.

[0014] S4: Then, the EDF is combined with the bi-temporal encoder features of the same scale. Perform feature fusion to obtain the fused decoder feature F d ={F1,F2,F3,F4}.

[0015] S5: Multi-scale features The channels are connected in series, and the enhanced difference features EDF are multiplied and added element by element to obtain the fused feature map F. d .

[0016] S6: Perform upsampling, convolution and concatenation operations to progressively fuse F1, F2, F3, and F4 layer by layer to obtain feature C3. And through upsampling and convolution operations, F1, F2, F3, and C3 are converted into feature maps of size H×W×2, and the network is optimized using deep supervision.

[0017] Furthermore, the present invention adopts a deep supervision strategy to train the intermediate layers F1, F2, F3 and the output layer C3 of the network, and uses a loss function that is a hybrid loss consisting of dice loss and weighted cross-entropy loss.

[0018] Furthermore, in order to quantify the effectiveness of the project proposal method, the present invention measures the accuracy of the present invention through precision, recall and F1-score. That is:

[0019]

[0020] Among them, TP represents the number of samples that are actually positive but predicted to be positive, FP represents the number of samples that are actually negative but predicted to be positive, FN represents the number of samples that are actually positive but predicted to be negative, and TN represents the number of samples that are actually negative but predicted to be negative.

[0021] Beneficial effects of the present invention:

[0022] The present invention can extract deep features with more discriminative change information through asymmetric convolution residual blocks and enhance the focus on edge features; the fused multi-scale difference features are enhanced in the spatiotemporal domain through the spatiotemporal attention mechanism, which can enhance the areas of real changes and suppress the areas of unreal changes; the layer-by-layer decoding method can effectively help reconstruct the changed areas; the specific beneficial effects include the following:

[0023] The present invention performs change detection on high spatial resolution remote sensing images, can effectively extract real changes in land objects, and eliminate the influence of external factors such as illumination and season, thereby reducing the false detection rate of pseudo changes. The present invention provides a change detection network MDSANet, which uses asymmetric convolution residual blocks to extract deep features with more discriminative change information. The spatiotemporal domain enhancement of the fused multi-scale difference features is performed using a spatiotemporal attention mechanism to enhance the areas of real changes while suppressing the areas of unreal changes. At the same time, the correlation between channels of feature maps of different scale differences is established to improve the representation ability of features of different scale differences.

[0024] The asymmetric convolution residual encoder is the core component of the present invention. It adopts a special convolution structure to better extract deep features. Since there is a lot of detail information in ultra-high spatial resolution remote sensing images, the quality of feature extraction is crucial to the accuracy of change detection. Through the asymmetric convolution residual encoder, the present invention can make full use of deep features and improve the representation ability and robustness of the change detection network.

[0025] In order to improve the ability of the change detection network to identify real change areas, the present invention introduces a spatiotemporal attention mechanism to enhance the multi-scale difference features. By weighting the difference features in the spatial and temporal dimensions, the features of the real change areas can be highlighted, while the features of the non-real change areas can be suppressed. This can improve the accuracy and robustness of change detection and reduce false detection and missed detection.

[0026] The feature fusion decoder is a key part of the present invention. By fusing the enhanced difference features with the dual-phase encoder features of the same scale, the detail information of the changed area can be better restored. The traditional jump connection method may have semantic difference problems, and the feature fusion decoder of the present invention can effectively eliminate the semantic difference between the feature map after the decoder upsamples and the feature map of the encoder of the same scale, fuse the enhanced difference features and the dual-phase encoder features of the same scale, and improve the performance of the change detection network in detail recovery.

[0027] In summary, the twin network provided by the present invention includes an asymmetric convolution residual encoder, a multi-scale difference feature spatiotemporal attention enhancement module and a feature fusion decoder, which can make full use of the features of high spatial resolution remote sensing images and improve the accuracy and robustness of change detection. The comprehensive application of this series of technologies makes the present invention have important professional value and application prospects in the field of ultra-high spatial resolution remote sensing image change detection.

[0028] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0030] Figure 1 It is a flow chart of the implementation of the present invention;

[0031] Figure 2 This is a schematic diagram of an asymmetric residual convolution block;

[0032] Figure 3 It is a schematic diagram of the CBAM network architecture;

[0033] Figure 4 is a schematic diagram of the feature fusion module;

[0034] Figure 5 It is a schematic diagram of the change detection results of LEVIR-CD and WHU-CD on different models. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Example 1

[0037] like Figure 1 Shown

[0038] The present embodiment describes a twin network for change detection of remote sensing images with ultra-high spatial resolution, wherein the twin network is a change detection network MDSANet, comprising an asymmetric convolutional residual encoder, a multi-scale difference feature spatiotemporal attention enhancement module, and a feature fusion decoder.

[0039] In this embodiment, an asymmetric convolution residual block is used for deep feature extraction, and then the difference features of different scales are stacked along the channel direction, and the spatiotemporal attention mechanism is used to enhance the multi-scale difference features in the spatiotemporal domain to improve the representation ability of features of different scales. The multi-scale difference features enhanced in the spatiotemporal domain are split according to the number of channels of different scales, and the enhanced difference features are fused with the dual-phase encoder features of the same scale through the feature fusion decoder.

[0040] On the other hand, the present invention provides a method for detecting changes in remote sensing images with ultra-high spatial resolution, comprising the following steps:

[0041] S1: Assuming that the size and number of channels of the two-phase images I1 and I2 are H×W×C0, multi-scale features are extracted by inputting them into the asymmetric convolutional residual encoder with shared weights. for i=1,2. In calculating multi-scale difference features The asymmetric convolution residual block proposed in the present invention is as follows: Figure 2 shown.

[0042] S2: df 1 ,df 2 ,df 3Downsample by 2×, 4×, and 8× to convert the spatial size to H / 8×W / 8, and then convert the downsampled df 1 ,df 2 ,df 3 and df 4 The multi-scale difference fusion feature ff is obtained by concatenation, and its spatial size and number of channels are H / 8×W / 8×C. The spatiotemporal attention mechanism (CBAM) is used to enhance ff in the spatiotemporal domain. After enhancement, the multi-scale difference fusion feature eff is split into channels to obtain the enhanced multi-scale difference features EDF at different scales = {edf 1 ,edf 2 ,edf 3 ,edf 4}.

[0043] S3: CBAM consists of spatial attention (SAM) and channel attention (CAM). The structure of CBAM is as follows Figure 3 As shown. Its spatial attention uses the spatial relationship between features to generate a spatial attention map, and channel attention uses the channel relationship between features to generate a channel attention map. The present invention first connects the obtained multi-scale difference maps in series to obtain a multi-scale difference map ff after connection, so as to establish the channel correlation of different scale difference features. Secondly, ff is enhanced in the channel domain and the spatial domain to obtain an enhanced multi-scale difference map eff, thereby improving the expression ability of different scale features.

[0044] S4: Then, the EDF is combined with the bi-temporal encoder features of the same scale. Perform feature fusion to obtain the fused decoder feature F d ={F1,F2,F3,F4}.

[0045] S5: Its feature fusion module is as follows Figure 4 As shown. Multi-scale features The channels are connected in series, and the enhanced difference features EDF are multiplied and added element by element to obtain the fused feature map F. d .

[0046] S6: Perform upsampling, convolution and concatenation operations to progressively fuse F1, F2, F3, and F4 layer by layer to obtain feature C3. And through upsampling and convolution operations, F1, F2, F3, and C3 are converted into feature maps of size H×W×2, and the network is optimized using deep supervision.

[0047] In this embodiment, a deep supervision strategy is adopted to train the intermediate layers F1, F2, F3 and the output layer C3 of the network, and the loss function used is a mixed loss consisting of dice loss and weighted cross-entropy loss.

[0048] In this embodiment, in order to quantify the effectiveness of the project proposal method, the present invention measures the accuracy of the present invention through precision, recall and F1-score. That is:

[0049]

[0050] Among them, TP represents the number of samples that are actually positive but predicted to be positive, FP represents the number of samples that are actually negative but predicted to be positive, FN represents the number of samples that are actually positive but predicted to be negative, and TN represents the number of samples that are actually negative but predicted to be negative.

[0051] Example 2

[0052] Experimental data

[0053] The data used in the present invention are LEVIR-CD and WHU-CD datasets. LEVIR-CD is a large-scale satellite remote sensing image CD dataset of urban buildings. The dataset includes a total of 637 pairs of VHR images with a spatial resolution of 0.5m, each image has a spatial size of 1024 pixels × 1024 pixels, and the number of spatial channels is 3. After cropping to 256 pixels × 256 pixels and removing the unchanged image pairs, the LEVIR-CD dataset is divided into a training set / 3107 pairs, a validation set / 433 pairs, and a test set / 923 pairs. WHU-CD is a CD dataset of aerial images of urban disaster buildings. The dataset contains a pair of VHR images with a spatial resolution of 0.075m, a spatial size of 32507 pixels × 15354 pixels, and the number of spatial channels is 3. After cropping to 256 pixels × 256 pixels and removing the unchanged image pairs, a total of 1403 pairs of training sets, 173 pairs of validation sets, and 173 pairs of test sets are obtained.

[0054] Comparative experiment and experimental evaluation

[0055] In order to verify the effectiveness of the proposed MDSANet model, the present invention selects the more popular change detection models in recent years for comparison. FC-Siam-conc is a fully convolutional twin change detection network. CDNet is a deconvolutional network for street scene change detection. L-UNet is a change detection model that combines semantic segmentation and a fully convolutional LSTM network. SNUNet is a densely connected Siamese network. DSIFN is a deep supervised image fusion network. BIT is a Transformer-based change detection network. Compared with other models, the quantitative and qualitative experimental results of the present invention are shown in Table 1 and Figure 5 shown.

[0056] Table 1: Quantitative experimental results of LEVIR-CD and WHU-CD on different models

[0057]

[0058] Example 3

[0059] The accuracy of the present invention and the prior art (patent document CN111539316A) is measured by precision, recall and F1-score.

[0060] Table 1: Comparative experiment of the present invention and the prior art solutions on the LEVIR-CD dataset

[0061] Precision Recall F1-score The present invention 90.59 90.08 90.34 Existing technical solutions 81.10 78.81 79.94

[0062] Table 2: Comparative experiment of the present invention and the prior art solutions on the WHU-CD dataset

[0063] Precision Recall F1-score The present invention 93.13 89.27 91.16 Existing technical solutions 88.23 84.62 86.39

[0064] It can be seen from Tables 1 and 2 that the present invention has a more outstanding detection effect, and its precision, recall and F1-score are all higher than those of the prior art.

[0065] In summary, the present invention discloses a twin network and method for change detection of remote sensing images with ultra-high spatial resolution, and the twin network is referred to as MDSANet. MDSANet uses asymmetric convolution to extract deep semantic features, uses the CBAM attention mechanism to focus on real change areas and suppress non-real change areas, and uses feature fusion strategy to reduce semantic differences in the fusion process. In terms of network architecture, MDSANet consists of an asymmetric convolution feature extraction encoder and a feature fusion decoder, and uses the CBAM attention mechanism between the encoder and the decoder to enhance multi-scale difference features. Comparative experimental results show that MDSANet has achieved excellent performance on both LEVIR-CD and WHU-CD datasets, with F1-Score reaching 90.34% and 91.16% respectively, and has lower missed detection rate and false detection rate than other models.

[0066] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for detecting changes in ultra-high spatial resolution remote sensing images based on a twin network, characterized in that: The twin network is a change detection network MDSANet, which includes an asymmetric convolutional residual encoder, a multi-scale difference feature spatiotemporal attention enhancement module and a feature fusion decoder; the asymmetric convolutional residual encoder is used to extract deep features, and then the difference features of different scales are stacked along the channel direction, and the spatiotemporal attention mechanism is used to enhance the multi-scale difference features in the spatiotemporal domain to improve the representation ability of features of different scales; The multi-scale difference features enhanced in the spatiotemporal domain are split according to the number of channels at different scales, and the enhanced difference features are fused with the dual-phase encoder features of the same scale through the feature fusion decoder; The following steps are involved: S1: Setting dual-phase image I 1 , I 2 The size and number of channels are H×W×C 0, and then input it into the weight-sharing asymmetric convolutional residual encoder to extract multi-scale features = { , , , }, i =1 or 2; then calculate the multi-scale difference features DF =| - |={ , , , }; S2: , , The sizes are H×W, H / 2×W / 2, and H / 4×W / 4 respectively. After 8×, 4×, and 2× downsampling, the spatial size is converted to H / 8×W / 8 , and then the downsampled , , and Multi-scale difference fusion features are obtained by concatenation ff , the space size and number of channels are H / 8×W / 8×C ; And use the spatiotemporal attention mechanism to ff Perform spatiotemporal enhancement to obtain multi-scale difference fusion features eff Split the number of channels to obtain enhanced multi-scale difference features at different scales EDF ={ , , , }; S3: First input , The features are first concatenated, then 1×1 convolution and batch normalization are performed; secondly EDF The feature branch one performs 1×1 convolution, batch normalization and activation function, and the other branch two performs 1×1 convolution and batch normalization; after batch normalization , The channel splicing features are first EDF The branch features of branch one are matrix multiplied and then summed EDF The branch features of branch 2 are processed by matrix addition to obtain ={ , , , }; S4: Upsampling, convolution and concatenation operations will be performed , , , Layer-by-layer progressive fusion to obtain features C 3; and through upsampling and convolution operations , , , C 3Convert to H×W×2 We use feature maps of different sizes and optimize the network using deep supervision.

2. The ultra-high spatial resolution remote sensing image change detection method according to claim 1, characterized in that: Use deep supervision strategy to train the middle layers of the network F 1, F 2, F 3 and output layer C 3. Use a mixed loss consisting of dice loss and weighted cross-entropy loss.

3. The ultra-high spatial resolution remote sensing image change detection method according to claim 2, characterized in that: It also includes measuring accuracy through precision, recall and F1-score, namely: ； ； ； Among them, TP represents the number of samples that are actually positive but predicted to be positive, FP represents the number of samples that are actually negative but predicted to be positive, FN represents the number of samples that are actually positive but predicted to be negative, and TN represents the number of samples that are actually negative but predicted to be negative.

Images