A Remote Sensing Image Change Detection Method Based on Siamese Network and Swing Transformer

Abstract

This invention provides a remote sensing image change detection method based on Siamese networks and SwinTransformer. The downsampled and fused features are then upsampled through the upsampling subnets of two Siamese networks. This utilizes the spatial information lost during downsampling, improving the effectiveness of the Siamese network model. Furthermore, the first and second feature images are fused by summing, without concatenating the features. Concatenating features would double the number of image channels, requiring more parameters and significantly increasing computational complexity for subsequent upsampling. This invention directly sums the values ​​of corresponding pixels in the two feature images without increasing the number of channels. This ensures the Siamese network model's performance does not degrade while greatly reducing the number of model parameters and computational load for subsequent upsampling.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method for detecting changes in remote sensing images based on Siamese networks and Swing Transformers. Background Technology

[0002] With the continuous growth of the scale of satellite launches in China each year, satellite remote sensing data has entered a phase of rapid growth. Extracting information from these massive amounts of remote sensing images is a fundamental production method for remote sensing applications and the remote sensing industry. Therefore, how to detect change information from these remote sensing images has become an important research direction in the field of remote sensing applications, namely, remote sensing image change detection technology. This refers to the technology of quantitatively analyzing and determining the characteristics and processes of changes in ground features from remote sensing images from different periods. Existing remote sensing image change detection typically uses twin networks to extract features from two-time remote sensing images separately, then uses a stitching method to fuse the features of the two-time remote sensing images, and finally uses upsampling to restore the image resolution using the fused features.

[0003] For example, CN14842351A discloses a method for detecting semantic changes in remote sensing images based on twin Transformers, such as... Figure 1 As shown, two weight-shared Swin Transformers are used as the encoder of the model to directly model the global semantic relationships of the image. In the decoding stage, a 3D convolutional module and a deconvolutional module with axial attention are used to generate multi-class change maps from the cascaded bi-temporal features. Feng Weiming et al. proposed a method for detecting changes in high-resolution remote sensing images by fusing Transformer structures, such as... Figure 2 The TSU-Net model, as shown, integrates a Transformer structure with self-attention, a skip connection structure, and a Siamese structure. This allows the model to acquire global contextual information from remote sensing imagery and fuse the high-resolution feature maps obtained by the CNN with the global contextual information through the skip connection structure, thus obtaining a remote sensing image change detection map with boundary contours. However, CN14842351A does not fuse the features lost during downsampling during upsampling, and its feature fusion module uses Conv3D, resulting in a large number of model parameters. TSU-Net, on the other hand, does not introduce an attention mechanism during upsampling and uses a stitching method to fuse image features, increasing the number of model parameters during subsequent upsampling. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a remote sensing image change detection method based on Siamese networks and Swing Transformers. This method can utilize more spatial features lost during downsampling, and both the upsampling and downsampling processes use Swing-Transformer blocks, thereby introducing an attention mechanism to improve the effectiveness of the Siamese network model. Simultaneously, feature fusion uses an additive operation, which can reduce the number of parameters in the Siamese network model.

[0005] A method for detecting changes in remote sensing images based on Siamese networks and Swing Transformers, wherein the Siamese network includes a downsampling subnetwork and an upsampling subnetwork, and the method includes the following steps:

[0006] The remote sensing images acquired at two different times are input into two downsampling subnets for downsampling, resulting in the first downsampled image and the second downsampled image, respectively.

[0007] The first and second captured images are subjected to Swing Transformer block operations to obtain the first feature image and the second feature image, respectively.

[0008] The first feature image and the second feature image are fused by summation to obtain the first fused image;

[0009] The first fused image is input into two upsampling subnets for upsampling to obtain the first upsampled image and the second upsampled image, respectively.

[0010] The first and second images are multiplied together to fuse features, resulting in a change image that represents the relative changes between the remote sensing images acquired at the two time points.

[0011] Furthermore, the size of the remote sensing image is (W, H, 3), and the downsampling subnet performs downsampling on the remote sensing image as follows:

[0012] Perform a patch partition operation on the remote sensing image to obtain a remote sensing image with a size of (W / 4, H / 4, 48);

[0013] An Embedding operation is performed on a remote sensing image of size (W / 4, H / 4, 48) to obtain a remote sensing image of size (W / 4, H / 4, 96).

[0014] The first Swing Transformer block operation is performed on the remote sensing image of size (W / 4, H / 4, 96) to obtain a first-order downsampled feature image of size (W / 4, H / 4, 96) after initial feature extraction.

[0015] The first Patch Merging operation is performed on the first-order downsampled feature image of size (W / 4,H / 4,96) to obtain a first-order downsampled feature image of size (W / 8,H / 8,192).

[0016] Perform a second Swin Transformerblock operation on the first-order downsampled feature image of size (W / 8, H / 8, 192) to obtain a second-order downsampled feature image of size (W / 8, H / 8, 192) after feature extraction.

[0017] The second-order downsampled feature image of size (W / 8, H / 8, 192) is subjected to a second patch merging operation to obtain a second-order downsampled feature image of size (W / 16, H / 16, 394). The second-order downsampled feature image of size (W / 16, H / 16, 394) is used as the final downsampled image output by the downsampled subnet.

[0018] Furthermore, the method by which the upsampling subnet upsamples the first fused image is as follows:

[0019] The first fused image of size (W / 16, H / 16, 394) is subjected to the first Patch Expanding operation to obtain a first fused image of size (H / 8, W / 8, 192);

[0020] The second-order downsampled feature image of size (W / 8, H / 8, 192) is stitched together with the first fused image of size (H / 8, W / 8, 192) in a skip link manner to obtain the first stitched feature image of size (W / 8, H / 8, 192).

[0021] Perform the first Swin Transformerblock operation on the first stitched feature image of size (W / 8,H / 8,192) to obtain a first-order upsampled feature image of size (W / 8,H / 8,192) after initial feature extraction.

[0022] The first-order upsampled feature image of size (W / 8, H / 8, 192) is subjected to a second Patch Expanding operation to obtain a first-order upsampled feature image of size (H / 4, W / 4, 96);

[0023] The first-order downsampled feature image of size (W / 4, H / 4, 96) is concatenated with the first-order upsampled feature image of size (H / 4, W / 4, 96) in a skip link manner to obtain the second concatenated feature image of size (W / 4, H / 4, 96).

[0024] The second stitched feature image of size (W / 4, H / 4, 96) is subjected to a second Swin Transformerblock operation to obtain a second-order upsampled feature image with the same size (W / 4, H / 4, 96) after the second feature extraction.

[0025] The second-order upsampled feature image of size (W / 4, H / 4, 96) is subjected to a third Patch Expanding operation to obtain a second-order upsampled feature image of size (H, W, 48).

[0026] A second-order upsampled feature image of size (H,W,48) is subjected to a Linear Projection operation to obtain an upsampled image of size (H,W,N), where N is the number of possible changes in the remote sensing image.

[0027] Furthermore, when N=2, possible changes include: the remote sensing images acquired at two different times remain unchanged relative to each other, and the remote sensing images acquired at two different times change relative to each other.

[0028] When N ≥ pairs, possible changes include: the remote sensing images acquired at two time points remain unchanged relative to each other, and the remote sensing images acquired at two time points change relative to each other and change from object A to object Bi, where i = 1, 2, ..., N-1.

[0029] Furthermore, twin networks share weights.

[0030] Beneficial effects:

[0031] 1. This invention provides a remote sensing image change detection method based on Siamese networks and a Twin Transformer. The downsampled and fused features are then upsampled through two upsampling subnets of the Siamese network. This utilizes the spatial information lost during downsampling, improving the effectiveness of the Siamese network model. Furthermore, the first and second feature images are fused by summing, without concatenating the features. Concatenating features would double the number of image channels, requiring more parameters and significantly increasing computational complexity for subsequent upsampling. This invention directly sums the values ​​of corresponding pixels in the two feature images without increasing the number of channels. This ensures the Siamese network model's performance does not degrade while significantly reducing the number of model parameters and computational load for subsequent upsampling.

[0032] Meanwhile, experiments have shown that directly using addition and multiplication operations on the two feature fusion modules does not diminish their effectiveness, and using addition and convolution operations can reduce the model's parameters. Therefore, this invention can utilize more spatial features lost during the downsampling process to improve the effectiveness of the Siamese network model. At the same time, using non-learnable operations on the feature fusion module can reduce the model's parameters.

[0033] 2. This invention provides a remote sensing image change detection method based on Siamese network and Swing Transformer. The upsampling subnet also adopts the Swing-Transformer block operation that can perform attention calculation on the feature map, instead of CNN, which is equivalent to introducing an attention mechanism. The attention mechanism is a mechanism that enables the computer to focus on local information. Because the attention mechanism can locate the information of interest and suppress useless information, this invention also introduces an attention mechanism during upsampling, which helps to better find the changed areas in two remote sensing images and generate a change map.

[0034] 3. This invention provides a remote sensing image change detection method based on Siamese network and Swing Transformer. During upsampling, the skip link structure stitches the downsampled image with the upsampled image, thereby ensuring that more spatial information is lost when downsampling dual-time remote sensing images. Attached Figure Description

[0035] Figure 1 This is a flowchart of the remote sensing image semantic change detection method based on twin Transformers in CN14842351A;

[0036] Figure 2 Here is a diagram of the existing TSU-Net model structure;

[0037] Figure 3 The flowchart illustrates a remote sensing image change detection method based on twin networks provided by this invention. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0039] This invention provides a method for detecting changes in remote sensing images based on Siamese networks and the Twin Transformer. The weight-sharing Siamese network includes a downsampling subnetwork and an upsampling subnetwork, such as... Figure 3 As shown, the method includes the following steps:

[0040] S1: The remote sensing images acquired at two different time points are input into two downsampling subnets for downsampling, resulting in a first downsampled image and a second downsampled image, respectively. The size of the remote sensing image is (W, H, 3), and the downsampling method used by the downsampling subnets is as follows:

[0041] S11: Perform a patch partition operation on the remote sensing image to obtain a remote sensing image with a size of (W / 4, H / 4, 48);

[0042] S12: Perform an embedding operation on the remote sensing image of size (W / 4,H / 4,48), which is to map the number of model channels to a fixed dimension to obtain a remote sensing image of size (W / 4,H / 4,96);

[0043] S13: Perform the first Swing Transformer block operation on the remote sensing image of size (W / 4, H / 4, 96) to obtain a first-order downsampled feature image of size (W / 4, H / 4, 96) after initial feature extraction;

[0044] It should be noted that the Swing Transformer block operation can perform attention calculations on the feature map, directly model the global semantic relationships of the image, and the image size remains unchanged when passing through the Swing Transformer block. This structure is effective in extracting image features and thus obtaining transformation maps.

[0045] S14: Perform the first Patch Merging operation on the first-order downsampled feature image of size (W / 4,H / 4,96), multiply the number of channels by 2, and obtain a first-order downsampled feature image of size (W / 8,H / 8,192);

[0046] S15: Perform a second SwinTransformer block operation on the first-order downsampled feature image of size (W / 8,H / 8,192) to obtain a second-order downsampled feature image of size (W / 8,H / 8,192) after feature extraction.

[0047] S16: Perform a second patch merging operation on the second-order downsampled feature image of size (W / 8, H / 8, 192), multiplying the number of channels by 2, to obtain a second-order downsampled feature image of size (W / 16, H / 16, 394), and use the second-order downsampled feature image of size (W / 16, H / 16, 394) as the final downsampled image output by the downsampled subnet.

[0048] S2: The first and second captured images are subjected to Swing Transformer block operations to obtain the first feature image and the second feature image, respectively.

[0049] S3: The first feature image and the second feature image are fused by summation to obtain the first fused image;

[0050] S4: The first fused image is input into two upsampling sub-networks for upsampling, resulting in a first upsampled image and a second upsampled image, respectively; wherein, the upsampling sub-networks perform upsampling on the first fused image as follows:

[0051] S41: Perform the first patch expanding operation on the first fused image with size (W / 16, H / 16, 394), multiply the number of channels by 1 / 2, and obtain the first fused image with size (H / 8, W / 8, 192);

[0052] It should be noted that the specific operation of Patch Expanding is as follows: Taking the first Patch Expanding layer as an example, before upsampling, a linear layer is applied to the input features (W / 32×H / 32×8C) to increase the feature dimension to twice the original (W / 32×H / 32×16C). Then, a rearrangement operation is used to expand the resolution of the original input features by twice, while reducing the feature dimension to 1 / 4 of the input dimension (W / 32×H / 32×16C->W / 16×H / 16×4C).

[0053] S42: The second-order downsampled feature image of size (W / 8,H / 8,192) is spliced ​​with the first fused image of size (H / 8,W / 8,192) in a skip link manner to obtain the first spliced ​​feature image of size (W / 8,H / 8,192);

[0054] It should be noted that during upsampling, the skip link structure stitches together the downsampled image with the upsampled image, thus ensuring that more spatial information from the downsampling process is utilized.

[0055] S43: Perform the first Swin Transformerblock operation on the first stitched feature image of size (W / 8,H / 8,192) to obtain a first-order upsampled feature image of size (W / 8,H / 8,192) after initial feature extraction;

[0056] S44: Perform a second PatchExpanding operation on the first-order upsampled feature image of size (W / 8, H / 8, 192), multiplying the number of channels by 1 / 2, to obtain a first-order upsampled feature image of size (H / 4, W / 4, 96);

[0057] S45: The first-order downsampled feature image of size (W / 4,H / 4,96) is concatenated with the first-order upsampled feature image of size (H / 4,W / 4,96) in a skip link manner to obtain a second concatenated feature image of size (W / 4,H / 4,96);

[0058] S46: Perform a second Swing Transformerblock operation on the second stitched feature image of size (W / 4, H / 4, 96) to obtain a second-order upsampled feature image with the same size (W / 4, H / 4, 96) after the second feature extraction.

[0059] S47: Perform a third Patch Expanding operation on the second-order upsampled feature image of size (W / 4, H / 4, 96), multiplying the number of channels by 1 / 2, to obtain a second-order upsampled feature image of size (H, W, 48);

[0060] S48: Perform a Linear Projection operation on the second-order upsampled feature image of size (H,W,48) to obtain an upsampled image of size (H,W,N), where N is the number of possible changes in the remote sensing image.

[0061] When N=2, the possible changes include: the remote sensing images acquired at two different times remain unchanged relative to each other, and the remote sensing images acquired at two different times change relative to each other.

[0062] When N≥3, possible changes include: the remote sensing images acquired at two different times remain unchanged relative to each other; and the remote sensing images acquired at two different times have changed relative to each other, with object A changing to object Bi, where i = 1, 2, ..., N-1. For example, the remote sensing images acquired at two different times may have changed relative to each other, with vegetation changing to houses, or vegetation changing to rivers, or vegetation changing to mountains, etc. In other words, when N=2, it is a binary classification, i.e., change and no change; when N≥3, in addition to change and no change, it also includes the type of change.

[0063] S5: The first and second acquired images are multiplied together to fuse features, resulting in a change image representing the relative changes between the remote sensing images acquired at the two time points.

[0064] In summary, this invention upsamples the features fused after downsampling through two upsampling subnetworks of a Siamese network, thereby utilizing the spatial information lost during downsampling and improving model effectiveness. Furthermore, experiments show that directly using addition and multiplication operations on the two feature fusion modules does not diminish their effectiveness, and using addition and convolution operations can reduce model parameters. Therefore, this invention can utilize more spatial features lost during downsampling, improving model effectiveness. Additionally, using non-learnable operations for the feature fusion module can reduce model parameters.

[0065] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A remote sensing image change detection method based on a twin network and a Swin Transformer, characterized in that, The twin network comprises a down-sampling sub-network and an up-sampling sub-network, the down-sampling sub-network comprises a Patch Partition module, an Embedding module, two Swin Transformer blocks and two Patch Merging modules; the up-sampling sub-network comprises three Patch Expanding modules, two Swin Transformer blocks and a LinearProjection module; The method comprises the following steps: The remote sensing images acquired at two time points are respectively input into two down-sampling sub-networks for down-sampling, to obtain a first down-sampled image and a second down-sampled image; The first down-sampled image and the second down-sampled image are respectively subjected to Swin Transformer block operation, to obtain a first feature image and a second feature image; The first feature image and the second feature image are subjected to feature fusion in an additive manner, to obtain a first fused image; The first fused image is respectively input into two up-sampling sub-networks for up-sampling, to obtain a first up-sampled image and a second up-sampled image; The first up-sampled image and the second up-sampled image are subjected to feature fusion in a multiplicative manner, to obtain a change image representing the relative change between the remote sensing images acquired at the two time points.

2. The remote sensing image change detection method based on twin network and Swin Transformer according to claim 1, wherein, The size of the remote sensing image is (W, H, 3), and the down-sampling sub-network performs down-sampling on the remote sensing image in the following manner: The remote sensing image is subjected to patch partition operation, to obtain a remote sensing image with a size of (W / 4, H / 4, 48); The remote sensing image with a size of (W / 4, H / 4, 48) is subjected to Embeding operation, to obtain a remote sensing image with a size of (W / 4, H / 4, 96); The remote sensing image with a size of (W / 4, H / 4, 96) is subjected to first Swin Transformer block operation, to obtain a first-order down-sampled feature image after initial feature extraction, and the size of the first-order down-sampled feature image is still (W / 4, H / 4, 96); The first-order down-sampled feature image with a size of (W / 4, H / 4, 96) is subjected to first Patch Merging operation, to obtain a first-order down-sampled feature image with a size of (W / 8, H / 8, 192); The first-order down-sampled feature image with a size of (W / 8, H / 8, 192) is subjected to second Swin Transformer block operation, to obtain a second-order down-sampled feature image after twice feature extraction, and the size of the second-order down-sampled feature image is still (W / 8, H / 8, 192); The second-order down-sampled feature image with a size of (W / 8, H / 8, 192) is subjected to second Patch Merging operation, to obtain a second-order down-sampled feature image with a size of (W / 16, H / 16, 394), and the second-order down-sampled feature image with a size of (W / 16, H / 16, 394) is taken as the down-sampled image finally output by the down-sampling sub-network.

3. The remote sensing image change detection method based on twin network and Swin Transformer according to claim 2, wherein, The up-sampling sub-network performs up-sampling on the first fused image in the following manner: performing a first Patch Expanding operation on the first fused image with a size of (W / 16, H / 16, 394) to obtain a first fused image with a size of (H / 8, W / 8, 192); splicing the second-order down-sampling feature image with a size of (W / 8, H / 8, 192) and the first fused image with a size of (H / 8, W / 8, 192) in a skip link manner to obtain a first spliced feature image with a size of (W / 8, H / 8, 192); performing a first Swin Transformer block operation on the first spliced feature image with a size of (W / 8, H / 8, 192) to obtain a first-order up-sampling feature image with a size of (W / 8, H / 8, 192) after initial feature extraction; performing a second Patch Expanding operation on the first-order up-sampling feature image with a size of (W / 8, H / 8, 192) to obtain a first-order up-sampling feature image with a size of (H / 4, W / 4, 96); splicing the first-order down-sampling feature image with a size of (W / 4, H / 4, 96) and the first-order up-sampling feature image with a size of (H / 4, W / 4, 96) in a skip link manner to obtain a second spliced feature image with a size of (W / 4, H / 4, 96); performing a second Swin Transformer block operation on the second spliced feature image with a size of (W / 4, H / 4, 96) to obtain a second-order up-sampling feature image with a size of (W / 4, H / 4, 96) after secondary feature extraction; performing a third Patch Expanding operation on the second-order up-sampling feature image with a size of (W / 4, H / 4, 96) to obtain a second-order up-sampling feature image with a size of (H, W, 48); performing a Linear Projection operation on the second-order up-sampling feature image with a size of (H, W, 48) to obtain an up-sampling image with a size of (H, W, N), wherein N is a number of possible change cases of the remote sensing image.

4. The remote sensing image change detection method based on twin network and Swin Transformer according to claim 3, wherein, When N=2, the possible change cases include that the remote sensing images acquired at two time points do not change relative to each other, and the remote sensing images acquired at two time points change relative to each other. When N≥3, the possible change cases include that the remote sensing images acquired at two time points do not change relative to each other, and the remote sensing images acquired at two time points change relative to each other and become object Bi from object A, wherein i=1, 2, …, N-1.

5. The remote sensing image change detection method based on twin network and Swin Transformer according to any one of claims 1-4, characterized in that, The twin network shares weights.

Images