A remote sensing image change detection method and device based on a multi-scale CNN-Transformer

Abstract

The embodiment of the application provides a remote sensing image change detection method and device based on a multi-scale CNN-Transformer, the method comprising: inputting a double-time remote sensing image into a multi-scale CNN-Transformer encoder; extracting features of the double-time remote sensing image by using the multi-scale CNN-Transformer encoder to obtain multi-scale global-local features of the double-time remote sensing image; performing layer-by-layer fusion reconstruction on the multi-scale global-local features of the double-time remote sensing image by using a cascaded decoder to obtain a change feature map; and performing pixel-by-pixel classification on the change feature map by using a classifier to obtain a change detection result map. The embodiment of the application effectively combines CNN and Transformer at multiple scales, fully extracts and fuses the multi-scale global-local features in the remote sensing image, and the change feature map obtained by fusion contains the global-local features at multiple scales, thereby enhancing the robustness of detecting different size change regions.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image analysis technology, and in particular to a method and apparatus for detecting changes in remote sensing images based on multi-scale CNN-Transformer. Background Technology

[0002] Remote sensing change detection is an important task in the field of remote sensing. It involves comparing co-registered images of the same geographic area taken at two different times to identify changes in the Earth's surface and take appropriate countermeasures. Remote sensing change detection has wide applications in agriculture, military, urban planning, environmental monitoring, and other fields.

[0003] Existing remote sensing change detection methods can be divided into traditional methods and machine learning-based methods. Traditional remote sensing change detection methods typically detect changed regions based on algebraic calculations or manual feature transformations. Algebraic calculation-based methods heavily rely on the manual selection of the segmentation threshold for the final change probability map, and inappropriate thresholds can lead to a large number of false alarms. Manual feature transformation-based methods require pre-selection of a feature space to be transformed, but finding a suitable feature space is very time-consuming and lacks good generalization ability. Machine learning-based remote sensing change detection methods include traditional machine learning-based methods and deep learning-based methods. Traditional machine learning-based methods usually adopt a post-classification approach, and their accuracy heavily depends on the choice of classification algorithm, so their generalization ability is still not high. Deep learning-based methods are usually composed of convolutional neural network models (CNNs). Although this method can effectively extract local features of remote sensing images, due to the locality of the convolution operation itself, it is difficult to focus on the global features of remote sensing images. In recent years, some studies have used the Transformer structure to enhance the ability of change detection algorithms to extract global features. However, these studies only combine CNN and Transformer at a single scale and do not take into account global-local features at multiple scales. As a result, they may miss changes in different sizes, leading to poor robustness and low accuracy in detection.

[0004] Therefore, there is an urgent need for a remote sensing image change detection method with better robustness and higher detection accuracy. Summary of the Invention

[0005] In view of the above problems, embodiments of the present invention provide a method and apparatus for detecting changes in remote sensing images based on multi-scale CNN-Transformer, so as to overcome the above problems or at least partially solve the above problems.

[0006] A first aspect of this invention discloses a method for detecting changes in remote sensing images based on a multi-scale CNN-Transformer, the method comprising:

[0007] Input dual-temporal remote sensing images into a multi-scale CNN-Transformer encoder;

[0008] The multi-scale CNN-Transformer encoder is used to extract features from the dual-temporal remote sensing image to obtain multi-scale global-local features of the dual-temporal remote sensing image;

[0009] A cascaded decoder is used to fuse and reconstruct the multi-scale global-local features of the dual-temporal remote sensing image layer by layer to obtain a change feature map.

[0010] The change feature map is classified pixel by pixel by a classifier to obtain a change detection result map, which represents the change region in the first and second temporal remote sensing images of the dual-temporal remote sensing image.

[0011] Optionally, the multi-scale CNN-Transformer encoder includes two sub-networks with identical structures and shared parameters. Each sub-network processes one temporal image from the dual-temporal remote sensing image. Each sub-network includes one convolutional block and N CovTrans blocks, corresponding to N+1 processing stages of a sub-network, with each processing stage corresponding to one scale. The feature extraction of the dual-temporal remote sensing image using the multi-scale CNN-Transformer encoder to obtain multi-scale global-local features of the dual-temporal remote sensing image includes:

[0012] In the first stage, the convolutional blocks in the two sub-networks are used to process the dual-temporal remote sensing image to obtain the initial features of the dual-temporal remote sensing image;

[0013] In stages 2 to N+1, the output features of the previous stage are processed sequentially using N CovTrans blocks in the two sub-networks to obtain the global-local features of the dual-temporal remote sensing image at each stage.

[0014] Optionally, the CovTrans block includes local branches, global branches, and an adaptive fusion module; in stages 2 to N+1, the output features of the previous stage are processed sequentially using the first to Nth CovTrans blocks in the two sub-networks to obtain the global-local features of the dual-temporal remote sensing image at each stage, including:

[0015] By using two cascaded local feature blocks in the local branch, local feature extraction is performed on the output features of the previous stage to obtain the local features at the corresponding scale of this stage;

[0016] Using the semantic coding block and Transformer encoder in the global branch, global feature extraction is performed on the output features of the previous stage to obtain the global features at the corresponding scale of this stage.

[0017] The adaptive fusion module is used to adaptively fuse the local features and the global features to obtain the global-local features corresponding to the scale of this stage.

[0018] Optionally, the step of extracting local features from the output features of the previous stage using two cascaded local feature blocks in the local branch to obtain local features at the corresponding scale of this stage includes:

[0019] Using the first local feature block of the two local feature blocks, the output features of the previous stage are locally processed to obtain the first local feature block processed features.

[0020] The first local feature block is processed and input into the second local feature block of the two local feature blocks for local processing to obtain the local features at the corresponding scale of this stage.

[0021] The local processing includes: performing convolution, batch normalization, linear activation, convolution, and batch normalization on the input features in sequence, adding the result of convolution with the input features, and then performing linear activation on the added result.

[0022] Optionally, the step of using the semantic coding blocks and Transformer encoder in the global branch to perform global feature extraction on the output features of the previous stage to obtain the global features at the corresponding scale of this stage includes:

[0023] The semantic coding blocks are used to sequentially perform convolution and flattening processes on the output features of the previous stage to obtain semantic blocks.

[0024] The semantic blocks are processed by a multi-head attention mechanism using a Transformer encoder to enhance the global dependencies between the semantic blocks, resulting in enhanced semantic blocks. The enhanced semantic blocks are then processed by multi-layer perceptual processing to obtain global features at the corresponding scale for this stage.

[0025] Optionally, the step of using the adaptive fusion module to fuse the local features and the global features to obtain the global-local features corresponding to the scale of this stage includes:

[0026] The local features and the global features are superimposed by summing pixel by pixel, and the superimposed features are encoded by global average pooling to obtain the encoded features;

[0027] The encoded features are compressed, and the compressed features are input into two fully connected layers to obtain local feature attention vectors and global feature attention vectors.

[0028] The local feature attention vector and the global feature attention vector are processed using the softmax operator to generate local feature selection weights and global feature selection weights;

[0029] Global-local features are generated based on the local features, the global features, the local feature selection weights, and the global feature selection weights.

[0030] Optionally, the cascaded decoder includes N+1 decoding stages; the step of using the cascaded decoder to perform layer-by-layer fusion and reconstruction of the multi-scale global and local features of the dual-temporal remote sensing image to obtain a change feature map includes:

[0031] In the N+1 decoder stage, the global-local features of the dual-temporal remote sensing images output by the Nth CovTrans block in the two sub-networks are fused to obtain the fusion result of the N+1 stage.

[0032] In the Nth to the 2nd decoder stages, the global-local features of the dual-temporal remote sensing images output by the CovTrans blocks in the two sub-networks are fused with the fusion result of the previous stage to obtain the fusion result of each stage.

[0033] In the first decoder stage, the initial features of the dual-temporal remote sensing images output by the convolutional blocks in the two sub-networks are fused with the fusion result of the second stage to obtain the fusion result of the first stage. The fusion result of the first stage is used as the change feature map.

[0034] Optionally, in the N+1th decoder stage, the global-local features of the dual-temporal remote sensing images output by the Nth CovTrans block in the two sub-networks are fused to obtain the fusion result of the N+1th stage, including:

[0035] The global-local features of the dual-temporal remote sensing images output by the Nth CovTrans block in the two sub-networks are concatenated to obtain the concatenated features. The concatenated features are then convolved to obtain the fusion result of the N+1th stage.

[0036] Optionally, in the Nth to the 2nd decoder stages, the global-local features of the dual-temporal remote sensing images output by the CovTrans blocks in the two sub-networks are sequentially fused with the fusion result from the previous stage to obtain the fusion result for each stage, including:

[0037] The global-local features of the dual-temporal remote sensing images output by the corresponding CovTrans blocks in the two sub-networks are concatenated with the fusion result of the previous stage to obtain the concatenated features. The concatenated features are then convolved to obtain the fusion result of each stage.

[0038] Optionally, in the first decoder stage, the initial features of the dual-temporal remote sensing images output by the convolutional blocks in the two sub-networks are fused with the fusion result of the second stage to obtain the fusion result of the first stage, including:

[0039] The initial features of the dual-temporal images output by the convolutional blocks in the two sub-networks and the fusion result of the second stage are concatenated to obtain the concatenated features. The concatenated features are then convolved to obtain the fusion result of the first stage.

[0040] Optionally, the classifier includes two 1×1 convolutions, and the step of classifying the change feature map pixel by pixel through the classifier to obtain the change detection result map includes:

[0041] The change feature map is processed sequentially using the two 1×1 convolutions to obtain a change detection probability feature map. The change detection probability feature map is a two-channel feature map. Each pixel in the first channel feature map represents the probability that each pixel in the dual-temporal remote sensing image changes, and each pixel in the second channel feature map represents the probability that each pixel in the dual-temporal remote sensing image does not change.

[0042] The first channel feature map and the second channel feature map in the change detection probability feature map are compared pixel by pixel;

[0043] The category of the feature map corresponding to the larger pixel value is determined as the final category of the pixel, where the category includes: changed and unchanged. A second aspect of this invention discloses a remote sensing image change detection device based on a multi-scale CNN-Transformer, the device comprising:

[0044] The image input module is used to input dual-temporal remote sensing images into the multi-scale CNN-Transformer encoder;

[0045] The feature extraction module is used to extract features from the dual-temporal remote sensing image using the multi-scale CNN-Transformer encoder to obtain multi-scale global-local features of the dual-temporal remote sensing image;

[0046] The feature fusion module is used to use a cascaded decoder to fuse and reconstruct the multi-scale global-local features of the dual-temporal remote sensing image layer by layer to obtain a change feature map.

[0047] The feature classification module is used to classify the change feature map pixel by pixel using a classifier to obtain a change detection result map, which represents the change region in the first and second temporal remote sensing images of the dual-temporal remote sensing image.

[0048] The embodiments of the present invention have the following advantages:

[0049] In this embodiment of the invention, to improve the detection accuracy and robustness of remote sensing image change detection, a remote sensing image change detection method based on multi-scale CNN-Transformer is proposed. First, a multi-scale CNN-Transformer encoder is used to extract features from a dual-temporal remote sensing image, obtaining multi-scale global-local features. Then, a cascaded decoder is used to fuse and reconstruct the multi-scale global-local features of the dual-temporal remote sensing image layer by layer, obtaining a change feature map. Finally, a classifier performs pixel-by-pixel classification on the change feature map to obtain a change detection result map. This embodiment of the invention effectively combines CNN and Transformer at multiple scales, fully extracting and fusing multi-scale global-local features in remote sensing images. The fused change feature map contains global-local features at multiple scales, thereby enhancing the accuracy and robustness of detecting change regions of different sizes. Compared with existing remote sensing change detection methods, the method provided in this embodiment significantly improves the robustness of detecting change regions of different sizes in remote sensing images, especially small change regions. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of the steps of a remote sensing image change detection method based on multi-scale CNN-Transformer provided in an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of a remote sensing image change detection method based on multi-scale CNN-Transformer provided by an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of a ConvTrans block structure provided in an embodiment of the present invention;

[0054] Figure 4This is a schematic diagram of a remote sensing image change detection device based on multi-scale CNN-Transformer provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] This invention provides a method for detecting changes in remote sensing images based on multi-scale CNN-Transformer, such as... Figure 1 As shown, Figure 1 A flowchart of a remote sensing image change detection method based on multi-scale CNN-Transformer provided in this embodiment of the invention includes steps S101 to S104:

[0057] Step S101: Input the dual-temporal remote sensing images into the multi-scale CNN-Transformer encoder.

[0058] In this embodiment, a dual-temporal remote sensing image refers to a pair of images of the same geographical area taken at two different time points, including a first temporal remote sensing image and a second temporal remote sensing image. The first temporal remote sensing image is the image before the change, and the second temporal remote sensing image is the image after the change. The dual-temporal remote sensing image can be an RGB format image. Specifically, acquiring dual-temporal remote sensing images at two different time points t1 and t2 can be represented as follows: and (Where H and W represent the length and width of the dual-temporal remote sensing image, respectively, and 3 indicates the number of channels in the RGB format image.) A multi-scale CNN-Transformer encoder is used to extract features from the dual-temporal remote sensing image at multiple scales. The multi-scale CNN-Transformer encoder consists of two sub-networks, each processing one temporal image from the dual-temporal remote sensing image. Therefore, inputting the dual-temporal remote sensing image into the multi-scale CNN-Transformer encoder means inputting the dual-temporal remote sensing image into each of the two sub-networks. In subsequent steps, the multi-scale CNN-Transformer encoder is used to perform feature extraction and change detection on the input dual-temporal remote sensing image to detect regions where changes have occurred.

[0059] Step S102: Use the multi-scale CNN-Transformer encoder to extract features from the dual-temporal remote sensing image to obtain multi-scale global-local features of the dual-temporal remote sensing image.

[0060] In this embodiment, the multi-scale global-local features of the dual-temporal remote sensing image include: multi-scale global-local features of the first temporal remote sensing image and multi-scale global-local features of the second temporal remote sensing image. Here, multi-scale global-local features refer to global-local features at multiple scales. For example, for a dual-temporal remote sensing image with length and width of H×W, its multi-scale global-local features can be global-local features at multiple scales such as H / 2×W / 2, H / 4×W / 4, and H / 8×W / 8.

[0061] In this embodiment, a multi-scale CNN-Transformer encoder is used to extract features from the input dual-temporal remote sensing image (i.e., the first temporal image and the second temporal image) from multiple scales, which has rich local and global information at each scale.

[0062] In one alternative embodiment, such as Figure 2 As shown, the multi-scale CNN-Transformer encoder includes two sub-networks with identical structures and shared parameters. Each sub-network processes one temporal image from the dual-temporal remote sensing images. Each sub-network includes one convolutional block and N CovTrans blocks, which correspond to N+1 processing stages of a sub-network. Each processing stage corresponds to one scale.

[0063] The two sub-networks have the same structure, both starting with a single convolutional block and then sequentially connecting N CovTrans blocks. After the dual-temporal remote sensing images are input into each sub-network of the multi-scale CNN-Transformer encoder, each sub-network processes the input images separately.

[0064] Specifically, the multi-scale CNN-Transformer encoder is used to extract features from the dual-temporal remote sensing image to obtain multi-scale global-local features of the dual-temporal remote sensing image, including stages 1 to N+1:

[0065] In the first stage, the convolutional blocks in the two sub-networks are used to process the dual-temporal remote sensing image to obtain the initial features of the dual-temporal remote sensing image.

[0066] The initial features of the dual-temporal remote sensing images mainly refer to the edge and texture features of the dual-temporal remote sensing images, namely, the edge and texture features of the first temporal remote sensing image and the second temporal remote sensing image. In the first stage, max pooling operations are performed on the first temporal image and the second temporal image using convolutional blocks in the two sub-networks, respectively, to obtain the corresponding initial features.

[0067] In stages 2 to N+1, the output features of the previous stage are processed sequentially using the first to Nth CovTrans blocks in the two sub-networks to obtain the global-local features of the dual-temporal remote sensing image at each stage.

[0068] The process of sequentially processing the output features of the previous stage using the first to Nth CovTrans blocks in the two sub-networks means that in each of the 2nd to N+1th stages, a CovTrans block from one sub-network is used to process the output features of the previous stage. Specifically, the 2nd stage uses the first CovTrans block to process the initial features obtained in the 1st stage, resulting in the global-local features processed in the 2nd stage. Then, the 3rd stage uses the first CovTrans block to process the global-local features output in the 2nd stage, resulting in the global-local features processed in the 3rd stage. This process is repeated to obtain the global-local features of the dual-temporal remote sensing image for each of the 2nd to N+1th stages. Since each stage extracts features based on the output features of the previous stage, the output features of each stage are smaller in scale than the features of the previous stage, ultimately resulting in global-local features of the dual-temporal remote sensing image at different scales, each scale containing rich local and global information.

[0069] In one alternative embodiment, a ConvTrans block based on a hybrid CNN and Transformer architecture was designed to extract multi-scale global-local features from dual-temporal remote sensing images. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of the ConvTrans block. The ConvTrans block includes local branches, global branches, and an adaptive fusion module.

[0070] The local branches are designed based on CNNs, using the BasicBlock proposed in ResNet as the building block for local branches (i.e., convolutional branches). Two BasicBlocks (i.e., local feature blocks) are concatenated within each local branch to extract local features from the remote sensing image. The global branches are built based on transformers, including semantic block coding blocks and a transformer encoder, to extract global features from the remote sensing image. The adaptive fusion module includes two operators: feature statistics and feature selection, used to fuse information from the local and global branches.

[0071] Specifically, in stages 2 to N+1, the output features of the previous stage are processed sequentially using the first CovTrans block and the Nth CovTrans block in the two sub-networks to obtain the global-local features of the dual-temporal remote sensing image at each stage, including steps A1 to A3:

[0072] Step A1: Use two cascaded local feature blocks in the local branch to extract local features from the output features of the previous stage.

[0073] In this embodiment, to ensure more complete extraction of local detail information, two identical local feature blocks are used for feature extraction. The features obtained after processing the two local feature blocks are local features of the remote sensing image. Specifically, the step of using two cascaded local feature blocks in the local branch to extract local features from the output features of the previous stage to obtain local features at the corresponding scale of this stage includes steps A11 to A12:

[0074] Step A11: Using the first local feature block of the two local feature blocks, perform local processing on the output features of the previous stage to obtain the first local feature block processed features;

[0075] Step A12: Input the processed features of the first local feature block into the second local feature block of the two local feature blocks for local processing to obtain the local features at the corresponding scale of this stage;

[0076] The local processing includes: performing convolution, batch normalization, linear activation, convolution, and batch normalization on the input features in sequence, adding the result of convolution with the input features, and then performing linear activation on the added result.

[0077] In this embodiment, the specific structure of the local feature block is as follows: Figure 3 As shown, the first and second convolutional layers on the serial branch of the local feature block are both 3×3 convolutions, while the convolutional layers on the parallel branch are 1×1 convolutions. The process function H(·) of the local feature block can be expressed as:

[0078] H(x) = σ(BN(2(((W1x)))) + W i x)

[0079] Where W1 and W2 represent the weights of the two 3×3 convolution operations, W i Let F represent the weights of a 1×1 convolution operation, σ(·) be the ReLU activation function, and BN(·) be a standard batch normalized layer. Therefore, let F... i-1 If the input features are for stage i-1 (2≤i≤N+1), then the local features output after processing through two concatenated local feature blocks in stage i are... It can be represented as:

[0080]

[0081] Here, H1 and H2 represent two local feature blocks, respectively.

[0082] Furthermore, the stride of the first convolution operation in the local feature block depends on its downsampling rate in that stage of the encoder, while the stride of the other three convolution operations in the local feature block is always 1.

[0083] In this embodiment, based on the characteristic that CNN can effectively extract local features of remote sensing images, a local branch is designed to extract local features of remote sensing images, ensuring the accuracy of local feature extraction.

[0084] Step A2: Use the semantic coding block and Transformer encoder in the global branch to extract global features from the output features of the previous stage, and obtain the global features at the corresponding scale of this stage.

[0085] In this embodiment, the semantic coding block includes convolutional layers and flattening layers, and the Transformer encoder includes a multi-head attention mechanism and a multi-layer perceptron mechanism. The semantic coding block is used to process the output features of the previous stage to extract semantic blocks, and then the Transformer encoder is used to process the semantic blocks to obtain global features.

[0086] Specifically, the step of using the semantic coding block and Transformer encoder in the global branch to extract global features from the output features of the previous stage to obtain global features at the corresponding scale of this stage includes steps A21 and A22:

[0087] Step A21: Using semantic coding blocks, perform convolution and flattening processes on the output features of the previous stage sequentially to obtain semantic blocks;

[0088] The structure of the semantic coding block is as follows Figure 3 As shown, the output features of the previous stage i-1 are... As the input to the global branch in stage i, let function h(·) represent the semantic block coding module, which will input F i-1 Convert to channel size C i A semantic block is essentially a convolution operation with a kernel size of p×p and a stride of s. Then, the resulting semantic block... The shape is flattened into H i W i ×C i This allows for subsequent processing in the transformer encoder of stage i. As the stages progress, the network gradually increases the feature dimension of semantic blocks while reducing the number of semantic blocks. This enables the global information branch to extract increasingly complex semantic representations over increasingly larger receptive fields and generate global features with different scales at different stages. Therefore, the output of the semantic block encoding module in stage i... It can be represented as:

[0089] T i =flatten(h(F) i-1 ,,))

[0090] Step A22: Use the Transformer encoder to process the semantic block using a multi-head attention mechanism to enhance the global dependency between semantic blocks and obtain an enhanced semantic block. Then, process the enhanced semantic block using multi-layer perception to obtain the global features at the corresponding scale of this stage.

[0091] The structure of a Transformer encoder is as follows: Figure 3 As shown, a transformer encoder is used to model the global context between semantic blocks. To reduce the computational cost of global self-attention, the transformer encoder design from CvT is adopted. In the transformer encoder, a multi-head attention mechanism is applied to the semantic blocks to enhance the global dependencies between them, resulting in enhanced semantic blocks. These enhanced semantic blocks are feature maps with preliminary global features. Then, multilayer perceptron processing is applied to the enhanced semantic blocks to correct some redundant information, thereby obtaining the final scaled global features.

[0092] Specifically, multi-head attention mechanism processing includes:

[0093] (1) The input semantic block is normalized by layer normalization, and then Q, K, and V are generated using compressed convolution and linear projection layers. The specific calculation is as follows:

[0094]

[0095]

[0096]

[0097]

[0098] Among them, W q ,k,v represent the weights of the three linear projections, Norm(·) represents layer normalization, and SC(·) is a compressed convolution used to reduce the spatial dimension of the input labels (i.e., K and V). This can be expressed as:

[0099]

[0100] Wherein, Resh2(·) is a sequence of one-dimensional semantic blocks. Operations to reshape into a two-dimensional semantic block graph This represents a compressed convolutional projection with a kernel size of R. i And the step size is also R i R i This indicates the compression ratio of the attention layer in the current stage. Flatten(·) is the operation that flattens the projected markers into 1D in the spatial dimension.

[0101] Then, the multi-head self-attention mechanism (MHA) computes multiple independent self-attention heads in parallel, concatenates the outputs of all attention heads, and finally generates an enhanced semantic block through linear projection. This can be expressed as:

[0102]

[0103]

[0104]

[0105] in, These represent the weights of the three linear projection layers Q, K, and V in each attention head, respectively, where n is the number of self-attention heads in the MHA, and d... head This represents the channel size of each self-attention head, which is equal to C. i / , It is a linear projection matrix used to integrate the outputs of each self-attention head.

[0106] It is important to note that by using compressed convolution, the spatial dimension of the semantic block will be significantly reduced to H. i W i / i 2 Therefore, the computational cost of using compressed semantic blocks in MHA will be lower than the computational cost of using uncompressed semantic blocks directly in MHA. i2 times.

[0107] (2) The output of the MHA, after being normalized, is then input into the Multilayer Perceptron (MLP) module. The output channel dimension is C. i In MLP, the hidden layer has a channel dimension of 4. i The MLP module is expressed by the following formula:

[0108]

[0109]

[0110] in, and It is a linear projection matrix, W D It is a weight of a depthwise convolution operation, which is beneficial for encoding positional information of features in the hidden layers of an MLP. GELU(·) is an activation function. The output semantic block of the MLP will be reshaped into a 2D feature map. This feature map contains rich global information in stage i.

[0111] It should be noted that residual processing is introduced in both the multi-head attention mechanism (MHA) and the multilayer perceptron (MLP) to avoid learning redundant features during feature processing and to prevent overfitting. This ensures that the final global and local features are more accurate.

[0112] In this embodiment, based on the global feature extraction feature of the Transformer structure, a global branch is designed to fully extract the semantic features (semantic blocks) of the remote sensing image using semantic coding blocks. Then, the semantic features are processed in the Transformer encoder using multi-head attention mechanism and multi-layer perception mechanism to obtain rich global features of the remote sensing image.

[0113] Step A3: Use the adaptive fusion module to adaptively fuse the local features and the global features to obtain the global-local features at the corresponding scale of this stage.

[0114] In this embodiment, the scale of the local and global features obtained at each stage is the same. After the local and global features are extracted, the extracted features are integrated to obtain global-local features. Specifically, the adaptive fusion module is used to adaptively fuse the local and global features to obtain global-local features at the corresponding scale for that stage, including steps A31 to A34:

[0115] Step A31: The local features and the global features are superimposed by element-wise summation, and the superimposed features are encoded by global average pooling to obtain the encoded features.

[0116] In this embodiment, the scale phase diagrams of local and global features at the same stage are superimposed by element-wise summation, that is, the elements at the same position in the local and global features are added together to obtain a superimposed feature. Then, global average pooling is performed based on the superimposed feature to obtain the corresponding encoded feature.

[0117] Specifically, the processing can be represented as follows:

[0118]

[0119]

[0120] Step A32: Compress the encoded features and input the compressed features into two fully connected layers to obtain local feature attention vectors and global feature attention vectors.

[0121] In this embodiment, the encoded feature obtained after global average pooling is a feature with relatively large dimensionality. Therefore, this encoded feature is compressed by multiplying the encoding matrix by a linear compression matrix weight to obtain a compressed feature. Represented as:

[0122] z = sW0

[0123] in, These are the weights of the linear compression matrix.

[0124] After obtaining the compression matrix, the compressed features are linked to two fully connected layers respectively to generate two attention vectors for local and global features.

[0125] Step A33: Use the softmax operator to process the local feature attention vector and the global feature attention vector to generate local feature selection weights and global feature selection weights.

[0126] Specifically, by applying the softmax operator to the local feature attention vector and the global feature attention vector, the generated local feature selection weights and global feature selection weights can be represented as follows:

[0127]

[0128] in, These are the parameters of two fully connected layers, which generate selection weights for local detail features and global context features, respectively.

[0129] Step A34: Generate global-local features based on the local features, the global features, the local feature selection weights, and the global feature selection weights.

[0130] In this embodiment, the global-local features of the remote sensing image are related to both local and global features. To better reflect the overall features of the remote sensing image, the weights corresponding to the local and global features are considered to construct the final global-local features. Specifically, the final global-local features are obtained by multiplying the local features by their selection weights and adding the global features by their selection weights. This is specifically expressed as follows:

[0131]

[0132] Step S103: Use a cascaded decoder to fuse and reconstruct the multi-scale global-local features of the dual-temporal remote sensing image layer by layer to obtain a change feature map.

[0133] In this embodiment, the cascaded decoder is used to fuse multi-scale global-local features. Each different scale contains rich local and global information. Therefore, by fusing global-local features at different scales, a change feature map is obtained that contains global-local features at each scale. When subsequent steps perform change detection based on this change feature map, the detection results are more accurate.

[0134] In one alternative embodiment, such as Figure 2 As shown, the cascaded decoder includes N+1 decoder stages. Each decoder stage includes channel stitching and convolutional fusion processing. Each decoder stage processes the initial features of the dual-temporal remote sensing image output by the convolutional blocks in the sub-network or the global-local features of the dual-temporal image output by each CovTrans block.

[0135] Specifically, the method of using a cascaded decoder to fuse and reconstruct the multi-scale global and local features of the dual-temporal image layer by layer to obtain a change feature map includes decoder stages 1 to N+1:

[0136] In the N+1 decoder stage, the global-local features of the dual temporal images output by the Nth CovTrans block in the two sub-networks are fused to obtain the fusion result of the N+1 stage.

[0137] The cascaded decoder performs feature fusion processing starting from the N+1 decoder stage, and then proceeds sequentially from the Nth decoder stage to the 1st decoder stage. The N+1 decoder stage processes the global-local features output from the N+1th stage (i.e., the Nth CovTrans block) of the multi-scale CNN-Transformer encoder.

[0138] Specifically, in the N+1 decoder stage, the global-local features of the dual-temporal images output by the Nth CovTrans block in the two sub-networks are fused to obtain the N+1 stage fusion result. This includes: stitching together the global-local features of the dual-temporal remote sensing images output by the Nth CovTrans block in the two sub-networks to obtain stitched features, and performing convolution processing on the stitched features to obtain the N+1 stage fusion result.

[0139] The convolutional layer can be used to fuse global and local features across multiple dimensions in the spatiotemporal dimension and generate change information. The convolutional layer is a 3×3 convolution, which is then processed, batch normalized, and activated. Finally, the processed features are subjected to a linear upsampling layer to restore the resolution of the change features, so that the scale (resolution) of the fusion result in stage N+1 is the same as the scale of the global and local features fused in stage N.

[0140] The processing in the (N+1)th decoder stage can be represented as follows:

[0141]

[0142] in, This represents the global-local features of the bitemporal image output by the cascaded decoder in the N+1 stage, where W is the number of stages. k The weights represent the weights of a 3×3 convolution operation, U(·) represents the upsampling operation, σ(·) is the ReLU activation layer, and BN(·) is the batch normalization layer.

[0143] In the Nth to the 2nd decoder stages, the global-local features of the dual temporal images output by the CovTrans blocks in the two sub-networks are fused with the fusion result of the previous stage to obtain the fusion result of each stage.

[0144] The processing steps in the Nth to the 2nd decoder stages are the same. After the N+1th decoder stage is completed, the fusion result of the N+1th stage is fused with the global-local features of the dual-temporal remote sensing image output by the N-1th CovTrans block in the two sub-networks. Then, after the Nth stage is completed, the next stage of fusion processing begins.

[0145] Specifically, the step of sequentially fusing the global-local features of the dual-temporal images output by the CovTrans blocks in the two sub-networks with the fusion result of the previous stage to obtain the fusion result of each stage includes: stitching together the global-local features of the dual-temporal remote sensing images output by the corresponding CovTrans blocks in the two sub-networks with the fusion result of the previous stage to obtain stitched features, and performing convolution processing on the stitched features to obtain the fusion result of each stage.

[0146] Since the fusion features from the previous stage were processed by a linear upsampling layer, and their scale is the same as the global-local features of the dual-temporal image of the remote sensing image to be processed in the current stage, the fusion processing is combined with the fusion features from the previous stage. First, a stitching process is performed. Then, a convolutional layer is used to fuse these global-local features from multiple dimensions in the spatiotemporal dimension to generate change information. The convolutional layer is a 3×3 convolution. Then, processing, batch normalization, and activation processing are performed. Finally, the processed features are subjected to a linear upsampling layer to restore the resolution of the change features.

[0147] The processing steps in the Nth to 2nd decoder stages can be represented as follows:

[0148]

[0149] in, Represents the global-local features of the dual-temporal image of the cascaded decoder, the Wth... k The weights represent the weights of a 3×3 convolution operation, U(·) represents the upsampling operation, σ(·) is the ReLU activation layer, and BN(·) is the batch normalization layer.

[0150] In the first decoder stage, the initial features of the dual temporal images output by the convolutional blocks in the two sub-networks are fused with the fusion result of the second stage to obtain the fusion result of the first stage. The fusion result of the first stage is used as the change feature map.

[0151] Specifically, the initial features of the bitemporal images output from the convolutional blocks in the two sub-networks are concatenated with the fusion result of the second stage to obtain concatenated features. These concatenated features are then convolved to obtain the fusion result of the first stage. It should be noted that there is no upsampling layer in the first stage because its size is already the same as the input bitemporal image.

[0152] The processing in the first decoder stage can be represented as follows:

[0153]

[0154] in, Represents the global-local features of a dual-temporal remote sensing image from a cascaded decoder, the Wth...k σ represents the weights of a 3×3 convolution operation, σ(·) is the ReLU activation layer, and BN(·) is the batch normalization layer.

[0155] In this embodiment, global-local features at different scales from dual-temporal remote sensing images are input to a cascaded decoder via skip connections for efficient layer-by-layer fusion. During each fusion layer, the global-local feature maps from the multi-scale CNN-Transformer encoder are first concatenated along the channel dimension, and then convolutional layers are used to fuse the concatenated feature maps. In this embodiment, the saliency of changes in the feature maps is enhanced.

[0156] Step S104: The change feature map is classified pixel by pixel by a classifier to obtain a change detection result map, which represents the change region of the first temporal remote sensing image and the second temporal remote sensing image in the dual temporal remote sensing image.

[0157] In this embodiment, the classifier includes two 1×1 convolutions, and then uses the change feature maps input from the two 1×1 convolutions for dimensionality reduction to obtain the final change detection result map. The final change detection result map is a binary mask image, where areas that have not changed are represented in black, and areas that have changed are represented in white.

[0158] In one optional embodiment, the step of classifying the change feature map pixel by pixel using a classifier to obtain a change detection result map includes:

[0159] The change feature map is processed sequentially using the two 1×1 convolutions to obtain a change detection probability feature map. The change detection probability feature map is a two-channel feature map. Each pixel in the first channel feature map represents the probability that each pixel in the dual-temporal remote sensing image changes, and each pixel in the second channel feature map represents the probability that each pixel in the dual-temporal remote sensing image does not change.

[0160] The first channel feature map and the second channel feature map in the change detection probability feature map are compared pixel by pixel;

[0161] The category of the feature map corresponding to the larger pixel value is determined as the final category of the pixel, and the category includes: change and no change.

[0162] In this embodiment, each pixel in the first channel feature map and the second feature map of the change detection result feature map is compared. If the pixel value in the first channel feature map is greater than the pixel value in the second feature map, the pixel is determined to be unchanged and represented as 0 (black). If the pixel value in the first channel feature map is less than the pixel value in the second feature map, the pixel is determined to be changed and represented as 1 (white).

[0163] It should be noted that the remote sensing image change detection model in this embodiment was pre-trained. To ensure network accuracy, during the model training phase, a weighted cross-entropy loss (WCE) is used to optimize the network parameters to alleviate the class imbalance between changed and invariant pixels. The WCE is defined as follows:

[0164]

[0165] Where w0 and w1 represent the loss weights for invariant and variable pixels, respectively, and are generally set to 0.4 and 0.6 to mitigate the impact of sample imbalance between invariant and variable pixels, P∈R H×W×2 This is the network's predicted change probability graph, where its two channels represent the probabilities of a predicted change and a predicted no change, respectively, Y∈R. H×W×1 It is the truth label, the lower corner point hw represents the position coordinates (h,k) of the pixel in P and Y, and [·] represents the indexing on the channel in the probability map.

[0166] In this embodiment, to improve the detection accuracy and robustness of remote sensing image change detection, a remote sensing image change detection method based on multi-scale CNN-Transformer is proposed. First, a multi-scale CNN-Transformer encoder is used to extract features from the dual-temporal remote sensing image, obtaining multi-scale global-local features. Then, a cascaded decoder is used to fuse and reconstruct the multi-scale global-local features of the dual-temporal remote sensing image layer by layer, obtaining a change feature map. Finally, a classifier performs pixel-by-pixel classification on the change feature map to obtain the change detection result map. This embodiment effectively combines CNN and Transformer at multiple scales, fully extracting and fusing multi-scale global-local features in the remote sensing image. The fused change feature map contains global-local features at multiple scales, thereby enhancing the accuracy and robustness of detecting change regions of different sizes. Compared with existing remote sensing change detection methods, the method provided in this embodiment significantly improves the robustness of detecting change regions of different sizes in remote sensing images, especially small change regions.

[0167] This invention also provides a remote sensing image change detection device based on multi-scale CNN-Transformer, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of a remote sensing image change detection device based on multi-scale CNN-Transformer, provided in an embodiment of the present invention. The device includes:

[0168] Image input module 41 is used to input dual-temporal remote sensing images into a multi-scale CNN-Transformer encoder;

[0169] Feature extraction module 42 is used to extract features from the dual-temporal remote sensing image using the multi-scale CNN-Transformer encoder to obtain multi-scale global-local features of the dual-temporal remote sensing image;

[0170] Feature fusion module 43 is used to use a cascaded decoder to fuse and reconstruct the multi-scale global-local features of the dual-temporal remote sensing image layer by layer to obtain a change feature map;

[0171] The feature classification module 44 is used to classify the change feature map pixel by pixel through a classifier to obtain a change detection result map, which represents the change region of the first time-series remote sensing image and the second time-series remote sensing image in the dual-time-series remote sensing image.

[0172] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0173] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods and apparatus according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0177] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0178] The above provides a detailed description of the remote sensing image change detection method and apparatus based on multi-scale CNN-Transformer provided by the present invention. Specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting changes in remote sensing images based on multi-scale CNN-Transformer, characterized in that, The method includes: Input dual-temporal remote sensing images into a multi-scale CNN-Transformer encoder; The multi-scale CNN-Transformer encoder is used to extract features from the dual-temporal remote sensing image to obtain multi-scale global-local features of the dual-temporal remote sensing image; A cascaded decoder is used to fuse and reconstruct the multi-scale global-local features of the dual-temporal remote sensing image layer by layer to obtain a change feature map. The change feature map is classified pixel by pixel by a classifier to obtain a change detection result map, which represents the change area in the first and second temporal remote sensing images in the dual-temporal remote sensing image. The multi-scale CNN-Transformer encoder comprises two sub-networks with identical structures and shared parameters. Each sub-network processes one temporal image from the dual-temporal remote sensing image. Each sub-network includes one convolutional block and N CovTrans blocks, corresponding to N+1 processing stages of the sub-network, with each processing stage corresponding to one scale. The multi-scale CNN-Transformer encoder is used to extract features from the dual-temporal remote sensing image, obtaining multi-scale global-local features of the dual-temporal remote sensing image, including: In the first stage, the convolutional blocks in the two sub-networks are used to process the dual-temporal remote sensing image to obtain the initial features of the dual-temporal remote sensing image; In stages 2 to N+1, the output features of the previous stage are processed sequentially using the first to Nth CovTrans blocks in the two sub-networks to obtain the global-local features of the dual-temporal remote sensing image at each stage.

2. The method according to claim 1, characterized in that, The CovTrans block includes local branches, global branches, and an adaptive fusion module. In stages 2 to N+1, the output features of the previous stage are processed sequentially using the first to Nth CovTrans blocks in the two sub-networks to obtain the global-local features of the dual-temporal remote sensing image at each stage's corresponding scale, including: By using two cascaded local feature blocks in the local branch, local feature extraction is performed on the output features of the previous stage to obtain the local features at the corresponding scale of this stage; Using the semantic coding block and Transformer encoder in the global branch, global feature extraction is performed on the output features of the previous stage to obtain the global features at the corresponding scale of this stage; The adaptive fusion module is used to adaptively fuse the local features and the global features to obtain the global-local features at the corresponding scale of this stage.

3. The method according to claim 2, characterized in that, The step of extracting local features from the output features of the previous stage using two cascaded local feature blocks in the local branch to obtain local features at the corresponding scale of this stage includes: Using the first local feature block of the two local feature blocks, the output features of the previous stage are locally processed to obtain the first local feature block processed features; The first local feature block is processed and input into the second local feature block of the two local feature blocks for local processing to obtain the local features at the corresponding scale of this stage. The local processing includes: performing convolution, batch normalization, linear activation, convolution, and batch normalization on the input features in sequence, adding the result of convolution with the input features, and then performing linear activation on the added result.

4. The method according to claim 2, characterized in that, The step of using the semantic coding blocks and Transformer encoder in the global branch to extract global features from the output features of the previous stage, and obtaining global features at the corresponding scale of this stage, includes: The semantic coding blocks are used to sequentially perform convolution and flattening processes on the output features of the previous stage to obtain semantic blocks. The semantic blocks are processed by a multi-head attention mechanism using a Transformer encoder to enhance the global dependencies between semantic blocks, resulting in enhanced semantic blocks. The enhanced semantic blocks are then processed by multi-layer perceptual processing to obtain global features at the corresponding scale for this stage.

5. The method according to claim 2, characterized in that, The step of using the adaptive fusion module to adaptively fuse the local features and the global features to obtain the global-local features corresponding to the scale of this stage includes: The local features and the global features are superimposed by summing pixel by pixel, and the superimposed features are encoded by global average pooling to obtain the encoded features; The encoded features are compressed, and the compressed features are input into two fully connected layers to obtain local feature attention vectors and global feature attention vectors. The local feature attention vector and the global feature attention vector are processed using the softmax operator to generate local feature selection weights and global feature selection weights; Global-local features are generated based on the local features, the global features, the local feature selection weights, and the global feature selection weights.

6. The method according to claim 1, characterized in that, The cascaded decoder comprises N+1 decoder stages; the step of using the cascaded decoder to perform layer-by-layer fusion and reconstruction of the multi-scale global-local features of the dual-temporal remote sensing image to obtain a change feature map includes: In the N+1 decoder stage, the global-local features of the dual-temporal remote sensing images output by the Nth CovTrans block in the two sub-networks are fused to obtain the fusion result of the N+1 stage. In the Nth to the 2nd decoder stages, the global-local features of the dual-temporal remote sensing images output by the CovTrans blocks in the two sub-networks are fused with the fusion result of the previous stage to obtain the fusion result of each stage. In the first decoder stage, the initial features of the dual-temporal remote sensing images output by the convolutional blocks in the two sub-networks are fused with the fusion result of the second stage to obtain the fusion result of the first stage. The fusion result of the first stage is used as the change feature map.

7. The method according to claim 6, characterized in that, In the N+1 decoder stage, the global-local features of the dual-temporal remote sensing images output by the Nth CovTrans block in the two sub-networks are fused to obtain the fusion result of the N+1 stage, including: The global-local features of the dual-temporal remote sensing images output by the Nth CovTrans block in the two sub-networks are concatenated to obtain the concatenated features. The concatenated features are then convolved to obtain the fusion result of the N+1th stage.

8. The method according to claim 6, characterized in that, In the Nth to the 2nd decoder stages, the global-local features of the dual-temporal remote sensing images output by the CovTrans blocks in the two sub-networks are sequentially fused with the fusion result of the previous stage to obtain the fusion result of each stage, including: The global-local features of the dual-temporal remote sensing images output by the corresponding CovTrans blocks in the two sub-networks are concatenated with the fusion result of the previous stage to obtain the concatenated features. The concatenated features are then convolved to obtain the fusion result of each stage.

9. The method according to claim 6, characterized in that, In the first decoder stage, the initial features of the dual-temporal remote sensing images output from the convolutional blocks of the two sub-networks are fused with the fusion result from the second stage to obtain the fusion result of the first stage, including: The initial features of the dual-temporal images output by the convolutional blocks in the two sub-networks and the fusion result of the second stage are concatenated to obtain the concatenated features. The concatenated features are then convolved to obtain the fusion result of the first stage.

10. The method according to claim 1, characterized in that, The classifier comprises two 1×1 convolutions. The classifier performs pixel-by-pixel classification on the change feature map to obtain a change detection result map, including: The change feature map is processed sequentially using the two 1×1 convolutions to obtain a change detection probability feature map. The change detection probability feature map is a two-channel feature map. Each pixel in the first channel feature map represents the probability that each pixel in the dual-temporal remote sensing image changes, and each pixel in the second channel feature map represents the probability that each pixel in the dual-temporal remote sensing image does not change. The first channel feature map and the second channel feature map in the change detection probability feature map are compared pixel by pixel; The category of the feature map corresponding to the larger pixel value is determined as the final category of the pixel, and the category includes: change and no change.

11. A remote sensing image change detection device based on multi-scale CNN-Transformer, characterized in that, The device includes: The image input module is used to input dual-temporal remote sensing images into a multi-scale CNN-Transformer encoder; The feature extraction module is used to extract features from the dual-temporal remote sensing image using the multi-scale CNN-Transformer encoder to obtain multi-scale global-local features of the dual-temporal remote sensing image; The feature fusion module is used to use a cascaded decoder to fuse and reconstruct the multi-scale global-local features of the dual-temporal remote sensing image layer by layer to obtain a change feature map. The feature classification module is used to classify the change feature map pixel by pixel using a classifier to obtain a change detection result map, which represents the change region in the first temporal remote sensing image and the second temporal remote sensing image in the dual temporal remote sensing image; The multi-scale CNN-Transformer encoder comprises two sub-networks with identical structures and shared parameters. Each sub-network processes one temporal image from the dual-temporal remote sensing image. Each sub-network includes one convolutional block and N CovTrans blocks, corresponding to N+1 processing stages of the sub-network, with each processing stage corresponding to one scale. The multi-scale CNN-Transformer encoder is used to extract features from the dual-temporal remote sensing image, obtaining multi-scale global-local features, including: In the first stage, the convolutional blocks in the two sub-networks are used to process the dual-temporal remote sensing image to obtain the initial features of the dual-temporal remote sensing image; In stages 2 to N+1, the output features of the previous stage are processed sequentially using the first to Nth CovTrans blocks in the two sub-networks to obtain the global-local features of the dual-temporal remote sensing image at each stage.

Images