Change Detection Method Based on Spatial Adaptation and Feature Enhancement Siamese Network

Abstract

The present invention discloses a change detection method based on a spatially adaptive and feature-enhanced Siamese network. The method comprises: constructing a Siamese network based on ResNet-18 to extract bi-temporal features; using three convolution kernels after a convolutional block attention model to obtain three pairs of feature maps with different spatial information, and calculating the Euclidean distance between each pair of feature maps obtained by the same convolution kernel to obtain a difference map; defining a spatially adaptive portion to obtain adaptive spatial knowledge; defining a feature enhancement portion to strengthen or weaken changing and invariant features; introducing a weighted contrast loss as a metric between the difference map and the ground truth map, and training the network by minimizing the loss; and obtaining a more accurate change map from the difference map through threshold segmentation. The present invention can effectively solve the spatial ambiguity problem caused by objects of different sizes and shapes being processed by the same filter kernel during feature extraction.

Description

Technical Field

[0001] The present invention belongs to the field of change detection, and in particular relates to a change detection method based on a spatially adaptive and feature-enhanced Siamese network. Background Art

[0002] The task of change detection is to identify regions that have changed and those that have not changed by analyzing a set of images collected at the same location but at different times. The difficulty of change detection lies in eliminating the effects of "non-semantic changes" caused by changes in lighting and seasons, so that the result only contains "semantic changes."

[0003] Change detection plays an increasingly important role in Earth observation, revealing surface changes using multi-temporal imagery. Due to its importance in many applications, a large number of methods have been proposed in the field of image change detection. Many traditional change detection methods, such as those based on image transformation and image classification, can only detect changes in large areas but are generally ineffective in detecting changes in small areas.

[0004] With the development of deep learning, many deep learning-based change detection methods have been proposed. Compared with traditional change detection methods, although deep learning-based change detection methods have good modeling capabilities, objects of different sizes and shapes are usually processed by the same filter kernel during feature extraction, which leads to spatial blurring and reduces change detection performance. Summary of the Invention

[0005] The purpose of the present invention is to provide a change detection method based on spatial adaptation and feature enhancement Siamese network to solve the spatial ambiguity problem caused by the fact that objects of different sizes and shapes are usually processed by the same filter kernel during the feature extraction process.

[0006] The technical solution to achieve the purpose of the present invention is as follows: In a first aspect, the present invention provides a change detection method based on a spatially adaptive and feature-enhanced Siamese network, comprising the following steps:

[0007] The first step is to construct a Siamese network based on ResNet-18 to extract bitemporal features;

[0008] In the second step, 3×3, 5×5 and 7×7 convolution kernels are used after the convolution block attention model to obtain three pairs of feature maps with different spatial information. The Euclidean distance of a pair of feature maps obtained by the same convolution kernel is calculated to obtain the difference map.

[0009] The third step is to define the spatial adaptation part to obtain adaptive spatial knowledge;

[0010] The fourth step is to define the feature enhancement part to strengthen or weaken the changing and unchanged features;

[0011] In the fifth step, weighted contrast loss is introduced as a measure between the difference map and the ground truth map, and the network is trained by minimizing the loss;

[0012] In the sixth step, the change map is obtained from the difference map through threshold segmentation.

[0013] In a second aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.

[0014] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0016] Compared with the prior art, the present invention has the following significant features: (1) The spatial adaptation part is designed to acquire adaptive spatial knowledge so that objects of different sizes and shapes can be clearly displayed, which can solve the spatial blurring problem caused by the fact that objects of different sizes and shapes are usually processed by the same filter kernel during the feature extraction process; (2) The feature enhancement part is designed to strengthen or weaken the changing and unchanged features to highlight the changing features; (3) Since there are more unchanged pixels than changed pixels, weight pairs are introduced in the training process to balance the changing and unchanged objects, and weighted contrast loss is introduced as a measure between the difference map and the ground truth map, and the network is trained by minimizing the weighted contrast loss.

[0017] The present invention is further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the present invention.

[0019] Figure 2 This is the change detection network structure of the present invention.

[0020] Figure 3 It is the spatial adaptation and feature enhancement module of the present invention.

[0021] Figure 4: The figure is the result effect diagram of the method of the present invention on the change detection dataset, where (a) is the image of the change detection dataset at time T1, (b) is the image of the change detection dataset at time T2, (c) is the ground truth image of the change detection dataset, (d) is the change map of the change detection dataset using the FC-EF method, (e) is the change map of the change detection dataset using the FC-Siam-diff method, (f) is the change map of the change detection dataset using the FC-Siam-conc method, (g) is the change map of the change detection dataset using the SRCDNet method, and (h) is the change map of the change detection dataset using the change detection method based on spatial adaptation and feature enhancement Siamese network. DETAILED DESCRIPTION

[0022] This paper proposes a change detection method based on a spatially adaptive and feature-enhanced Siamese network. Specifically, pixels belonging to different objects possess unique spatial knowledge, which is captured through soft fusion of multi-scale difference images, known as the spatial adaptation component. The feature enhancement component combines object features with each difference map, thereby strengthening or weakening changing and unchanged regions. Furthermore, since unchanged pixels outnumber changing pixels, weight pairs are introduced during training to balance changing and unchanged objects.

[0023] This paper proposes a spatial adaptation and feature enhancement module to address the spatial blurring problem caused by objects of varying sizes and shapes being processed by the same filter kernel during feature extraction. The module consists of a spatial adaptation component that acquires adaptive spatial knowledge to clearly display objects of varying sizes and shapes, and a feature enhancement component that enhances or weakens varying and unchanging features.

[0024] Combine Figure 1 , the implementation process of the present invention is described in detail below, the steps are as follows:

[0025] The first step is to construct a Siamese network based on ResNet-18 to extract bitemporal features, such as Figure 2 As shown in Figure 1, ResNet-18 consists of four residual blocks. T1 and T2 are dual-temporal images acquired at the same geographical location but at different times. The input of the network is two registered dual-temporal remote sensing images. The Siamese network can obtain feature maps of different scales.

[0026] The second step is to calculate the Euclidean distance of a pair of feature maps obtained by the same convolution kernel to obtain a difference map. The specific process is as follows:

[0027] After each residual block, we obtain the bi-temporal feature map and use it as the input of the spatial adaptation and feature enhancement module. Considering the powerful ability of the Convolutional Block Attention Module (CBAM) in feature learning, the proposed spatial adaptation and feature enhancement module is performed on the convolutional block attention model features, such as Figure 3 Specifically, after the convolutional block attention model, 3×3, 5×5, and 7×7 convolution kernels are used to obtain three pairs of feature maps with different spatial information. The Euclidean distance of a pair of feature maps obtained by the same convolution kernel is calculated to obtain the difference map. The difference map can be obtained by:

[0028] DI(F)=E(σ(f(F1)),σ(f(F2)))

[0029] Here, F1 and F2 are feature maps obtained from each residual block at different times, f(·) represents a convolution operation with kernel sizes of 3×3, 5×5, and 7×7, σ(·) represents the sigmoid function, and E(·) represents the Euclidean distance. Different convolution kernels produce different difference maps, denoted as DI1, DI2, and DI3.

[0030] The third step is to define the spatial adaptation part. The specific process is as follows:

[0031] (1) We fuse the bi-temporal feature maps F1 and F2 obtained from each residual block in the channel dimension. The number of channels of the fused feature map is twice that of the feature map before fusion. Through the convolution operation with a convolution kernel size of 3×3, the number of channels of the feature map becomes half of the fused feature map. Finally, we sum and normalize in the channel dimension to obtain the adaptive-weight map, as shown in Figure 3 As shown. The adaptive weight map can be obtained by the following formula:

[0032]

[0033] Among them, f 3×3 (·) represents a convolution operation with a kernel size of 3×3, and sum(·) represents summation over the channel dimension.

[0034] (2) After obtaining the adaptive weight map, multiply the adaptive weight map with DI1, DI2, and DI3 to obtain DI'1, DI'2, and DI'3. DI'1, DI'2, and DI'3 are expressed as:

[0035]

[0036]

[0037]

[0038] The fourth step is to define the feature enhancement part. The specific process is as follows:

[0039] We multiply the feature maps obtained from the 3×3, 5×5 and 7×7 convolution kernels at time T2 with DI'1, DI'2 and DI'3 respectively to obtain DI"1, DI"2 and DI"3. DI"1, DI"2 and DI"3 can be expressed as:

[0040]

[0041]

[0042]

[0043] where f 3×3 (·), f 5×5 (·) and f 7×7 (·) represents the convolution operation with kernel size of 3×3, 5×5 and 7×7, respectively. Denotes a multiplication operation. DI'1, DI'2, and DI'3 are fused in the channel dimension to obtain the final difference map as the output of the spatial adaptation and feature enhancement module.

[0044] In the fifth step, we introduce weighted contrast loss as a measure between the difference map and the ground truth map, and train the network by minimizing the loss. The specific process is as follows:

[0045] (1) The four difference maps obtained by the four spatial adaptation and feature enhancement modules are convolved to obtain four 96-channel difference maps. The four 96-channel difference maps are uniformly resized to 128×128 by bilinear interpolation and then fused to obtain a 384-channel difference map. The 384-channel difference map is convolved to obtain a 1-channel difference map. The 1-channel difference map is uniformly resized to 256×256 by trilinear interpolation.

[0046] (2) Weighted contrast loss is introduced as a measure between the difference map and the ground truth map. The loss function is defined as:

[0047]

[0048] Among them, M represents the size of dt, w1 and w2 represent the weights of the unchanged and changed feature pairs, gt i,j and dt i,j They represent the values ​​of the ground truth image and the difference image at point (i, j), and m is the margin.

[0049]

[0050]

[0051] Among them, pos_num and neg_num represent the number of unchanged and changed pixel pairs respectively.

[0052] In the sixth step, a more accurate change map is obtained from the difference map through threshold segmentation.

[0053] This paper defines a spatial adaptation and feature enhancement module for change detection. The spatial adaptation component aims to acquire adaptive spatial knowledge to clearly display objects of varying sizes and shapes, while the feature enhancement component strengthens or weakens varying and invariant features. This effectively addresses the spatial blurring problem caused by objects of varying sizes and shapes being processed by the same filter kernel during feature extraction. Weight pairs are introduced during training to balance varying and invariant objects, thereby addressing the imbalance between invariant and varying pixels.

[0054] The effect of the present invention can be further illustrated by the following simulation experiments:

[0055] (1) Simulation conditions

[0056] The simulation experiments used a change detection dataset. The spatial resolution of the images in the change detection dataset ranged from 3 cm / px to 100 cm / px. The dataset contained 16,000 pairs of seasonal images with a size of 256×256. In our experiments, the 16,000 image pairs were divided into training, validation, and test sets using a 10:3:3 ratio. The proposed change detection method was implemented using the PyTorch framework. During training, the learning rate was set to 0.0001. The simulation experiments were conducted on an NVIDIA Titan V for 100 iterations.

[0057] For quantitative indicator evaluation, the present invention uses four indicators: Precision, Recall, F1-Score and Intersection-over-Union (IoU).

[0058] (2) Simulation content

[0059] This paper uses a seasonal change detection dataset to test the performance of its algorithms. Four representative change detection algorithms are used for comparison: FC-EF (Fully Convolutional Early Fusion), FC-Siam-diff (Fully Convolutional Siamese-Difference), FC-Siam-conc (Fully Convolutional Siamese-Concatenation), and SRCDNet (super-resolution-based change detection network).

[0060] (3) Analysis of simulation experiment results

[0061] Table 1 shows the comparison results of the change detection dataset under different change detection algorithms.

[0062] Table 1 Quantitative evaluation of different change detection algorithms on change detection datasets

[0063] Methods Precision(%) Recall (%) F1(％) IoU (%) FC-EF 79.61 49.10 60.74 43.62 FC-Siam-diff 73.41 73.71 73.56 58.18 FC-Siam-conc 80.81 65.66 72.45 56.80 SRCDNet 94.52 91.02 92.74 86.46 Ours 95.91 92.78 94.32 89.25

[0064] As shown in Table 1, compared with the four change detection algorithms, the change detection method based on spatial adaptation and feature enhancement Siamese network proposed in the present invention performs best on the change detection dataset and achieves the highest precision, recall, F1 and IoU, which are 95.91%, 92.78%, 94.32% and 89.25% respectively. The spatial adaptation and feature enhancement modules can capture adaptive spatial knowledge and enhance the features of the changes, thereby effectively solving the spatial ambiguity problem caused by objects of different sizes and shapes being processed by the same filter kernel during feature extraction. This is why the present invention achieves the best performance on the change detection dataset. The performance of SRCDNet on the change detection dataset is second only to the method of the present invention. The four indicators are 94.52%, 91.02%, 92.74% and 86.46%, respectively, which are 1.39%, 1.76%, 1.58% and 2.79% lower than the method of the present invention, respectively. The three FCN variants perform poorly on the change detection dataset. The result effect diagram of the method of the present invention on the change detection dataset is as shown in the figure. Figure 4 The simulation results of the above real change detection datasets demonstrate the effectiveness of the proposed method.

[0065] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A change detection method based on spatially adaptive and feature-enhanced Siamese networks, characterized in that: The following steps are involved: In the first step, a Siamese network based on ResNet-18 is constructed to extract bi-temporal features. The input of the network is two registered bi-temporal remote sensing images. In the second step, 3×3, 5×5 and 7×7 convolution kernels are used after the convolution block attention model to obtain three pairs of feature maps with different spatial information. The Euclidean distance of a pair of feature maps obtained by the same convolution kernel is calculated to obtain the difference map. The third step is to define the spatial adaptation part to obtain adaptive spatial knowledge; The bi-temporal feature maps obtained from each residual block are fused in the channel dimension and then subjected to a convolution operation with a kernel size of 3×3. Finally, they are summed and normalized in the channel dimension to obtain an adaptive weight map. After obtaining the adaptive weight map, multiply the adaptive weight map with the difference map to obtain DI'1, DI'2 and DI'3 The fourth step is to define the feature enhancement part to strengthen or weaken the changing and unchanged features; Multiply the feature maps obtained from the 3×3, 5×5, and 7×7 convolution kernels with DI'1, DI'2, and DI'3 respectively; In the fifth step, weighted contrast loss is introduced as a measure between the difference map and the ground truth map, and the network is trained by minimizing the loss; In the sixth step, the change map is obtained from the difference map through threshold segmentation.

2. The change detection method based on spatial adaptive and feature enhanced Siamese network according to claim 1, characterized in that: The first step is to construct a Siamese network based on ResNet-18 to extract bi-temporal features, that is, to expand ResNet-18 into a Siamese structure. The input of the network is two registered bi-temporal remote sensing images, and the Siamese network can obtain feature maps of different scales.

3. The change detection method based on spatial adaptive and feature enhanced Siamese network according to claim 1, characterized in that The second step is to use 3×3, 5×5 and 7×7 convolution kernels after the convolution block attention model to obtain three pairs of feature maps with different spatial information. The Euclidean distance of a pair of feature maps obtained by the same convolution kernel is calculated to obtain the difference map. The specific process is as follows: After each residual block, a bi-temporal feature map is obtained and used as the input of the spatial adaptation and feature enhancement module. The spatial adaptation and feature enhancement module is performed on the convolutional block attention model features. 3×3, 5×5 and 7×7 convolution kernels are used after the convolutional block attention model to obtain three pairs of feature maps with different spatial information. The Euclidean distance between a pair of feature maps obtained by the same convolution kernel is calculated to obtain a difference map. The difference map is obtained by the following method: DI(F)=E(σ(f(F1)),σ(f(F2))) Where F1 and F2 are feature maps obtained from each residual block at different times, f(·) represents the convolution operation with kernel sizes of 3×3, 5×5, and 7×7, σ(·) represents the sigmoid function, and E(·) represents the Euclidean distance. Different difference maps can be obtained under different convolution kernels, which are denoted as DI1, DI2, and DI3 respectively.

4. The change detection method based on spatial adaptive and feature enhanced Siamese network according to claim 3, characterized in that: The third step is to define the spatial adaptation part. The specific process is as follows: (1) The bi-temporal feature maps F1 and F2 obtained from each residual block are fused in the channel dimension and then subjected to a convolution operation with a convolution kernel size of 3×3. Finally, the summation and normalization are performed in the channel dimension to obtain an adaptive weight map. The adaptive weight map is obtained by the following formula: Among them, f 3×3 (·) represents a convolution operation with a kernel size of 3×3, and sum(·) represents summation in the channel dimension; (2) After obtaining the adaptive weight map, the adaptive weight map is multiplied with DI1, DI2 and DI3 to obtain DI'1, DI'2 and DI'3 respectively; DI'1, DI'2 and DI'3 are expressed as:

5. The change detection method based on spatial adaptive and feature enhanced Siamese network according to claim 4, characterized in that: The fourth step is to define the feature enhancement part. The specific process is as follows: The feature maps obtained from the 3×3, 5×5 and 7×7 convolution kernels are multiplied with DI'1, DI'2 and DI'3 respectively to obtain DI"1, DI"2 and DI"3; DI"1, DI"2 and DI"3 are expressed as: where f 3×3 (·), f 5×5 (·) and f 7×7 (·) represents the convolution operation with kernel size of 3×3, 5×5 and 7×7, respectively. Indicates a multiplication operation; DI"1, DI"2 and DI"3 are fused in the channel dimension to obtain the final difference map as the output of the spatial adaptation and feature enhancement module.

6. The change detection method based on spatial adaptive and feature enhanced Siamese network according to claim 5, characterized in that: The fifth step introduces weighted contrast loss as a measure between the difference map and the ground truth map, and trains the network by minimizing the loss. The specific process is: (1) The four difference maps obtained by the four spatial adaptation and feature enhancement modules are convolved to obtain four 96-channel difference maps; the four 96-channel difference maps are uniformly resized to 128×128 by bilinear interpolation and then fused to obtain a 384-channel difference map; the 384-channel difference map is convolved to obtain a 1-channel difference map; the 1-channel difference map is uniformly resized to 256×256 by trilinear interpolation; (2) Weighted contrast loss is introduced as a measure between the difference map and the ground truth map. The loss function is defined as: Among them, M represents the size of dt, w1 and w2 represent the weights of the unchanged and changed feature pairs, gt i,j and dt i,j They represent the values ​​of the ground truth image and the difference image at point (i, j), and m is the margin; Among them, pos_num and neg_num represent the number of unchanged and changed pixel pairs respectively.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images