Remote sensing image change detection method based on adaptive Transform and deformable convolution

Abstract

The invention discloses a remote sensing image change detection method based on an adaptive Transform and a deformable convolution. The method comprises the following steps: firstly, establishing a twin feature extraction encoder fusing adaptive Transform and deformable convolution for extracting multi-scale local-global features of a dual-temporal remote sensing image; secondly, an adaptive Cross-Transform high-order feature interaction module is designed, and the adaptive Cross-Transform high-order feature interaction module is used for extracting high-level semantic change information and inhibiting irrelevant information interference; thirdly, providing a difference extraction module to generate different levels of difference feature maps so as to obtain a multi-scale change region; and finally, constructing a multi-scale fusion decoder, fusing the multi-level feature maps layer by layer, and further enhancing the feature expression of the change region. According to the method, the advantages of the Transform and the CNN are combined, the designed adaptive Transform and adaptive Cross-Transform can effectively extract the global features of the change region, and the adopted deformable convolution can effectively extract the local features of the change region, so that the overall performance of change detection is improved.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a remote sensing image change detection method based on an adaptive Transformer and deformable convolution. Background Art

[0002] Change detection in high-resolution remote sensing images can be regarded as a special semantic segmentation problem, that is, each pair of pixels in an image pair is divided into two labels: changed or unchanged. This technology is an important research direction in the field of image processing and can be applied to many fields, such as environmental monitoring, disaster assessment, land cover identification, etc. With the rapid development of remote sensing technology, a large amount of high-resolution remote sensing image data has emerged, injecting unprecedented vitality and opportunities into the field of change detection. In traditional remote sensing image change detection algorithms, complex preprocessing work is included, and features need to be extracted manually through experience, such as principal component analysis features (PCA), change vector analysis features (CVA), slow feature analysis features (SFA), etc. These methods rely heavily on professional skills and domain knowledge, with complex processes and slow development. They are time-consuming and labor-intensive to operate, and have low intelligence and high costs, which brings huge challenges to the development of change detection technology.

[0003] In recent years, with the development of artificial intelligence technology, remote sensing image change detection methods based on deep learning have gradually been applied to the field of change detection due to their ultra-high accuracy and degree of automation. According to the network structure and model architecture, remote sensing image change detection methods based on deep learning can be divided into: change detection methods based on convolutional neural networks (CNN), change detection methods based on attention mechanisms, and change detection methods based on Transformers.

[0004] Chen et al. published a paper titled "Remote Sensing Image Change Detection With Transformers" in the 2021 IEEE Transactions on Geoscience and Remote Sensing. The paper designed a dual-temporal image Transformer architecture to model context in the spatiotemporal domain. First, the dual-temporal image is represented as several semantic tokens, and the Transformer encoder is used to model the context in token-based space and time; then the learned tokens are fed back to the pixel space, and the original features are optimized through the Transformer decoder. However, this method only uses single-scale features to generate the final change map, lacks multi-scale feature representation, fails to effectively fuse global and local information, and has difficulty identifying subtle changes in details.

[0005] Publication No. CN114821350A is a method for detecting building changes in high-resolution remote sensing images with multi-stage information fusion. The dual-phase images are subjected to shallow feature extraction, early fusion, convolutional feature encoder, mid-term feature fusion layer, convolutional feature decoder, and late feature fusion layer to obtain the final change detection result. This method extracts features from information between remote sensing images of different phases separately and performs multiple fusion processes at different stages, which can better detect changes in remote sensing images. However, this method has limited extraction capabilities for global features and cannot identify high-level semantic information of dual-phase remote sensing images, resulting in the inability to eliminate interference from irrelevant changes.

[0006] In summary, the general remote sensing image change detection methods have many limitations, mainly manifested in:

[0007] (1) Single scale of feature representation: Existing remote sensing image change detection methods are often limited to a single scale in feature representation and fail to fully integrate multi-scale information to comprehensively capture the characteristics of surface changes at all levels, making it difficult to detect changes in objects at different scales.

[0008] (2) Insufficient extraction of differential features: The extraction of differential features is a key step in identifying surface changes. Existing methods often fail to extract differential features that express surface changes separately due to limitations in algorithm design or computing power, thus affecting the accuracy of detection results.

[0009] (3) Lack of interaction between dual-temporal features: In the process of remote sensing image change detection, the interaction between dual-temporal features is often ignored, and the information at two time points is not fully integrated to enhance the representation of change features, which makes it easy to be disturbed by irrelevant external changes in complex land changes.

[0010] (4) Insufficient boundary extraction accuracy: Ordinary convolutional layers use fixed rectangular grids for sampling, which may have a good detection effect on areas with regular contour changes. However, for changes in objects with irregular or complex shapes, rectangular convolution kernels have difficulty capturing the edges and detailed features of these changes. Summary of the invention

[0011] Purpose of the invention: In view of the problems existing in the prior art, the present invention proposes a remote sensing image change detection method based on adaptive Transformer and deformable convolution. This method combines the advantages of Transformer and CNN. The twin feature extraction encoder that integrates adaptive Transformer and deformable convolution can extract multi-scale local-global features of dual-phase remote sensing images; the high-order feature interaction module of the designed adaptive Cross-Transformer can extract high-level semantic change information while suppressing irrelevant information interference; the proposed difference extraction module can efficiently extract the difference features of ground object changes; the constructed multi-scale fusion decoder can further enhance the feature expression of the changed area.

[0012] Technical solution: To achieve the purpose of the present invention, the technical solution adopted by the present invention is: a remote sensing image change detection method based on adaptive Transformer and deformable convolution, the specific steps are as follows:

[0013] (1) Construct a remote sensing image change detection dataset, divide the dataset into training set, validation set and test set, and perform data enhancement on the training set;

[0014] (2) Construct a remote sensing image change detection network AT-DCNet based on adaptive Transformer and deformable convolution;

[0015] (3) Use the training set to train the AT-DCNet network, and use the validation set to evaluate the performance of the model during the training process to obtain the optimal model;

[0016] (4) Input the test set into the trained optimal model to perform change identification and obtain the final change detection result.

[0017] In step (1), the data set division and data enhancement method are as follows:

[0018] (1.1) After constructing the remote sensing image change detection dataset, each image is segmented into several non-overlapping image blocks of 256×256 size, and all the image blocks are divided into training set, validation set and test set according to a certain ratio;

[0019] (1.2) A series of data augmentation operations are performed on the training set, including random horizontal flipping, random vertical flipping, random image cropping, and Gaussian blurring.

[0020] Among them, in step (2), a remote sensing image change detection network AT-DCNet based on adaptive Transformer and deformable convolution is constructed, which mainly includes a twin feature extraction encoder integrating adaptive Transformer and deformable convolution, a high-order feature interaction module of adaptive Cross-Transformer, a difference extraction module, and a multi-scale fusion decoder. The specific steps are as follows:

[0021] (2.1) Input dual-phase remote sensing image I t1 and I t2 , construct a twin feature extraction encoder that integrates adaptive Transformer and deformable convolution to extract multi-scale local-global features of dual-phase remote sensing images. The encoder contains a pair of overlapping downsampling modules and three pairs of local-global fusion modules. In the overlapping downsampling module, a convolution layer with a convolution kernel size of 7×7, a step size of 2, a padding size of 3, and an output channel number of 32 is defined. After batch normalization and Relu activation function operation, an output feature map with a size of 1 / 2 of the input image is obtained. The process can be described as:

[0022]

[0023] Among them, f 7×7 ReLU represents a convolutional layer with a kernel size of 7×7, ReLU represents a Relu activation function, and BN represents a batch normalization layer;

[0024] (2.2) The local-global fusion module in step (2.1) consists of two branches: local and global. The local branch uses a deformable residual module to extract local features of the remote sensing image. The global branch uses the Patch embedding module and the adaptive Transformer module to extract global features of remote sensing images. Where j = 2, 3, 4. The Patch embedding module passes the input feature through a convolution layer with a convolution kernel size of 2×2 and a stride of 2. The network is divided into several small blocks, and each small block is flattened into a one-dimensional vector. At the same time, in order to retain the spatial position information, the flattened vector sequence is encoded with a position code. Then a linear transformation is performed to embed the vector sequence with the position code into a high-dimensional feature space, and the obtained token sequence P′ is input into the adaptive Transformer module to extract global features. The local features extracted by the local branches And the global features extracted by the global branch Add element by element to get Then input into a convolutional block attention module for further fusion to obtain local-global features The output feature sizes of the three pairs of local-global fusion modules are 1 / 4, 1 / 8, and 1 / 16 of the original input image, and their channel numbers are 64, 128, and 256, respectively;

[0025] (2.3) The dual-phase high-order local-global features and Input to the high-order feature interaction module of the adaptive Cross-Transformer, and use the adaptive Cross-Transformer to transform the bi-phase features and Perform information interaction to obtain high-level features with semantic change information and The high-order feature interaction module of the adaptive Cross-Transformer is obtained by stacking N1 adaptive Cross-transformer modules;

[0026] (2.4) Four difference extraction modules are used to extract four scale bi-phase features Difference feature extraction is performed. First, the dual-phase features and Subtract pixel by pixel, and then take the absolute value to get the preliminary difference features Then, Respectively and Perform pixel-by-pixel subtraction and then take the absolute value to obtain the improved difference feature and Finally, and The concatenation is performed and input into a 1×1 convolutional layer for fusion feature dimensionality reduction to obtain the final difference feature. The process can be described as:

[0027]

[0028] Among them, f 1×1 represents a convolutional layer with a convolution kernel size of 1×1, and cat represents a feature map concatenation operation;

[0029] (2.5) In the multi-scale fusion decoder, the extracted multi-scale difference features are processed using feature map concatenation and transposed convolution-based upsampling operations. The upsampling module based on transposed convolution here contains a 3×3 transposed convolution layer with a stride of 2, a batch normalization layer, a Relu activation function layer, and a 3×3 transposed convolution layer with a stride of 1. First, the difference feature map Use the upsampling module based on transposed convolution to perform channel compression and upsampling to obtain the feature map Then, and the difference features with the same scale After concatenation, the 1×1 convolutional layer is used to reduce the dimension and then input into the upsampling module to obtain the feature map. Next, use similar operations to and Process until the change feature map F with the same resolution as the original input is obtained final The process can be described as:

[0030]

[0031] Among them, Up i (i=1,2,3,4) represents four upsampling modules based on transposed convolution, F 1×1 represents a convolutional layer with a convolution kernel of 1×1, and cat represents a feature map concatenation operation;

[0032] (2.6) F final The input is sent to the remote sensing image change detection classifier, which uses two convolution layers with a convolution kernel size of 3×3, a step size of 1, and an output channel of 2 to detect the change feature map F. final Perform channel dimensionality reduction, and then use the Softmax function to generate the final change probability map.

[0033] Among them, in step (2.2), the deformable residual module method in the local branch is as follows:

[0034] For input features First, two residual units are constructed. Each residual unit contains two 3×3 convolutional layers, two batch normalization layers, and two Relu activation functions. The step size of the first convolutional layer of the first residual unit is 2, and the step sizes of the other convolutional layers are all 1. Based on these two residual units, local features can be preliminarily extracted.

[0035] Then, the deformable residual unit is input. The deformable residual unit is constructed by replacing the second 3×3 convolutional layer in the residual unit with a deformable convolutional layer. The local features are finally obtained through the deformable residual unit. The process can be described as:

[0036]

[0037] Among them, f 3×3 represents a convolution layer with a convolution kernel size of 3×3, BN represents batch normalization, ReLU represents the ReLU activation function, Def 3×3 Represents a deformable convolutional layer with a kernel size of 3×3.

[0038] Among them, the construction methods of the adaptive Transformer module and the adaptive Cross-Transformer module in steps (2.2) and (2.3) are as follows:

[0039] The adaptive Transformer module first undergoes layer normalization, then uses adaptive multi-head self-attention to capture global information, and introduces a residual branch to add and fuse with the main branch. Among them, the adaptive multi-head self-attention is based on the query matrix Q and the key matrix K a After the dot product between generates the attention weight, the value matrix V a Multiply it with the attention weight to get the output sequence P SAMSA The specific calculation process can be expressed by the following formula:

[0040] MHAtten(q,k a ,v a )=cat(h1,…,h s ,…,h n )W O

[0041]

[0042] Among them, q represents the query vector, k a and v a They represent the key vector and value vector after adaptive average pooling and 1×1 convolution layer processing, respectively, and h s is a single-head attention mechanism, W O is the output mapping matrix, and Represents q and k in each head respectively a and v a The mapping matrix corresponding to the three vectors, d k K a Based on adaptive multi-head self-attention, we get P SAMSA :

[0043] P SAMSA =SAMSA(LN(P ′ ))+P ′

[0044] Among them, SAMSA represents the adaptive multi-head self-attention module, LN represents layer normalization processing, P ′ Represents the token sequence in step (2.2);

[0045] Next, the obtained sequence P SAMSA After passing through two branches respectively, the main branch is normalized and multi-layer perceptron is added and fused with the residual branch to obtain the output sequence P new, the formula is as follows:

[0046] P new =MLP(LN(P SAMSA ))+P SAMSA

[0047] Among them, LN represents layer normalization processing, and MLP represents multi-layer perceptron operation. Finally, P new Reshape into a two-dimensional feature map, that is, get the global feature map

[0048] The constructed adaptive Cross-transformer module is obtained by changing the adaptive multi-head self-attention module in the adaptive Transformer to the adaptive multi-head cross attention module (SAMCA). The main difference between SAMCA and SAMSA is that Derived k2, v2 after adaptive processing and The derived q1 performs cross attention calculation, Derived k1, v1 after adaptive processing and The derived q2 is used for cross attention calculation.

[0049] Among them, in step (2.2), the convolution block attention module is constructed as follows:

[0050] This module contains a channel attention module and a spatial attention module. The channel attention module enhances feature representation by weighting feature maps of different channels. First, average pooling and maximum pooling are performed, and then two 1×1 convolutional layers with shared weights are used to learn the weight of each channel. The two results are then added together and the channel attention weight M is obtained through a sigmoid activation function. c , and compare it with the input feature map Multiply channel by channel to get the channel attention feature map The formula is as follows:

[0051]

[0052] Among them, MLP represents multi-layer perceptron, Avgpool represents average pooling, Maxpool represents maximum pooling, δ represents sigmoid activation function, and ⊙ represents channel-by-channel multiplication operation;

[0053] The spatial attention module enhances feature representation by weighting feature maps at different spatial locations. Average pooling and maximum pooling are performed on the channel dimension respectively, and then the two feature maps are spliced ​​on the channel dimension. A 7×7 convolution layer is used to reduce the number of channels to 1, and the spatial attention weight M is obtained through the sigmoid activation function. s , and compare it with the input feature map Multiply to get the spatial attention feature map The formula is as follows:

[0054]

[0055] Among them, f 7×7 is a 7×7 convolutional layer, Avgpool is average pooling, Maxpool is maximum pooling, cat represents the feature map channel concatenation operation, δ is the sigmoid activation function, represents an element-by-element multiplication operation. This is the local-global feature extracted at this stage

[0056] Beneficial effects: The present invention adopts the above technical solution and has the following beneficial effects:

[0057] (1) The multi-scale fusion decoder designed in the present invention can fuse the difference features of different scales, fuse the high-order semantic features and the low-order detail features layer by layer, make full use of the multi-scale information to comprehensively capture the characteristics of the changes of the ground objects, ensure the integrity of the final change features generated, and thus accurately identify the changes of the ground objects of various scales in the dual-time image.

[0058] (2) The difference extraction module designed in the present invention performs difference feature extraction on multi-scale dual-phase features and efficiently obtains the difference feature map in a learnable manner.

[0059] (3) The high-order feature interaction module of the adaptive Cross-Transformer designed in the present invention can effectively realize the effective interaction of high-order dual-phase features, enhance the representation of changing features, highlight the changes of interest and suppress the interference of irrelevant information.

[0060] (4) The deformable residual module adopted by the present invention can extract local features, reduce the limitations of the fixed-shape convolution kernel of the ordinary convolution layer, make the convolution operation more flexible to adapt to the shape changes of the ground objects, and enhance the ability of the convolution kernel to extract the geometric features of the changing area. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The overall framework diagram of the proposed method is shown in Figure 2.

[0062] Figure 2 It is the structure diagram of the local-global fusion module;

[0063] Figure 3 It is the structural diagram of the deformable residual module in the local branch;

[0064] Figure 4 It is the global branch structure diagram;

[0065] Figure 5 This is the structure diagram of the high-order feature interaction module of the adaptive Cross-Transformer;

[0066] Figure 6 It is the structural diagram of the difference extraction module;

[0067] Figure 7 This is the structure diagram of the multi-scale fusion decoder. DETAILED DESCRIPTION

[0068] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0069] like Figure 1 As shown, the technical solution of the present invention is further described in detail as follows:

[0070] (1) Construct a remote sensing image change detection dataset, divide the dataset into training set, validation set and test set, and perform data enhancement on the training set. This paper uses the public datasets LEVIR-CD and CDD for remote sensing image change detection to conduct experiments. The LEVIR-CD dataset contains 637 pairs of images with a size of 1024×1024. These images cover more than 20 regions and include 31,333 change instances, mainly concentrated on buildings; the CDD dataset contains 16,000 pairs of remote sensing images that change with the seasons, with a spatial size of 256×256. In CDD, the changing land cover is diverse, including buildings, cars and roads.

[0071] (1.1) After constructing the remote sensing image change detection dataset, each image is divided into several non-overlapping image blocks of 256×256 size, and all the image blocks are divided into training set, validation set and test set according to a certain ratio. After the LEVIR-CD dataset is divided, the training set, validation set and test set are obtained with 7120, 1024 and 2048 image pairs respectively; after the CDD dataset is divided, the training set, validation set and test set are obtained with 10000, 3000 and 3000 image pairs respectively;

[0072] (1.2) A series of data augmentation operations are performed on the training set, including random horizontal flipping, random vertical flipping, random image cropping, and Gaussian blurring.

[0073] (2) Figure 1As shown in the figure, a remote sensing image change detection network AT-DCNet based on adaptive Transformer and deformable convolution is constructed.

[0074] The specific parameters of each layer of AT-DCNet are as follows:

[0075] (2.1) Input dual-temporal remote sensing image I with a size of 256×256×3 t1 and I t2 , construct a twin feature extraction encoder that integrates adaptive Transformer and deformable convolution to extract multi-scale local-global features of dual-phase remote sensing images. The encoder contains a pair of overlapping downsampling modules and three pairs of local-global fusion modules. In the overlapping downsampling module, a convolution layer with a convolution kernel size of 7×7, a step size of 2, a padding size of 3, and an output channel number of 32 is defined. After batch normalization and Relu activation function operation, an output feature map with a size of 1 / 2 of the input image is obtained. The process can be described as:

[0076]

[0077] Among them, f 7×7 ReLU represents a convolutional layer with a kernel size of 7×7, ReLU represents a Relu activation function, and BN represents a batch normalization layer;

[0078] (2.2) Figure 2 As shown in Figure 2, the local-global fusion module in step (2.1) contains two branches, local and global. The local branch uses a deformable residual module to extract local features of the remote sensing image. The global branch uses the Patch embedding module and the adaptive Transformer module to extract global features of remote sensing images. Where j = 2, 3, 4. The output feature sizes of the three pairs of local-global fusion modules are 1 / 4, 1 / 8, and 1 / 16 of the original input image, and their channel numbers are 64, 128, and 256, respectively;

[0079] (a) If Figure 3 As shown, in the local branch, for the input feature First, two residual units are constructed. Each residual unit contains two 3×3 convolutional layers, two batch normalization layers, and two Relu activation functions. The step size of the first convolutional layer of the first residual unit is 2, and the step sizes of the other convolutional layers are all 1. Based on these two residual units, local features can be preliminarily extracted.

[0080] Then, the deformable residual unit is input. The deformable residual unit is constructed by replacing the second 3×3 convolutional layer in the residual unit with a deformable convolutional layer. The local features are finally obtained through the deformable residual unit. The process can be described as:

[0081]

[0082] Among them, f 3×3 represents a convolution layer with a convolution kernel size of 3×3, BN represents batch normalization, ReLU represents the ReLU activation function, Def 3×3 represents a deformable convolutional layer with a kernel size of 3×3;

[0083] (b) If Figure 4 As shown, in the global branch, for the input feature The patch embedding module first passes a convolutional layer with a kernel size of 2×2 and a stride of 2. The network is divided into several small blocks, and each small block is flattened into a one-dimensional vector. At the same time, in order to retain the spatial position information, the flattened vector sequence is encoded with a position code; then a linear transformation is performed to embed the vector sequence with the position code into the high-dimensional feature space, and then the obtained token sequence P′ is input into the adaptive Transformer module; then in the adaptive Transformer module, it is first normalized by the layer, and then the adaptive multi-head self-attention is used to capture the global information, and the residual branch is introduced to be added and fused with the main branch. Among them, the adaptive multi-head self-attention is based on the query matrix Q and the key matrix K a After the dot product between generates the attention weight, the value matrix V a Multiply it with the attention weight to get the output sequence P SAMSA The specific calculation process can be expressed by the following formula:

[0084] MHAtten(q,k a ,v a )=cat(h1,…,h s ,…,h n )W O

[0085]

[0086] Among them, q represents the query vector, k a and v a They represent the key vector and value vector after adaptive average pooling and 1×1 convolution layer processing, respectively, and h s is a single-head attention mechanism, W O is the output mapping matrix, and Respectively represent q and k in each head a and v a The mapping matrix corresponding to the three vectors, d k K a Based on adaptive multi-head self-attention, we get P SAMSA :

[0087] P SAMSA =SAMSA(LN(P ′ ))+P ′

[0088] Among them, SAMSA represents the adaptive multi-head self-attention module, and LN represents layer normalization processing;

[0089] Next, the obtained sequence P SAMSA After passing through two branches respectively, the main branch is normalized and multi-layer perceptron is added and fused with the residual branch to obtain the output sequence P new , the formula is as follows:

[0090] P new =MLP(LN(P SAMSA ))+P SAMSA

[0091] Among them, LN represents layer normalization processing, and MLP represents multi-layer perceptron operation. Finally, P new Reshape into a two-dimensional feature map, that is, get the global feature map

[0092] (c) Extract local features from local branches And the global features extracted by the global branch Add element by element to get The input is then further fused into a convolutional block attention module, which includes a channel attention module and a spatial attention module. The channel attention module enhances the feature representation by weighting the feature maps of different channels. First, average pooling and maximum pooling are performed, and then two 1×1 convolutional layers with shared weights are used to learn the weight of each channel. The two results are then added together and the channel attention weight M is obtained through a sigmoid activation function. c , and compare it with the input feature map Multiply channel by channel to get the channel attention feature map The formula is as follows:

[0093]

[0094] Among them, MLP represents multi-layer perceptron, Avgpool represents average pooling, Maxpool represents maximum pooling, δ represents sigmoid activation function, and ⊙ represents channel-by-channel multiplication operation;

[0095] The spatial attention module enhances feature representation by weighting feature maps at different spatial locations. Average pooling and maximum pooling are performed on the channel dimension respectively, and then the two feature maps are spliced ​​on the channel dimension. A 7×7 convolution layer is used to reduce the number of channels to 1, and the spatial attention weight M is obtained through the sigmoid activation function. s , and compare it with the input feature map Multiply to get the spatial attention feature map The formula is as follows:

[0096]

[0097] Among them, f 7×7 is a 7×7 convolutional layer, Avgpool is average pooling, Maxpool is maximum pooling, cat represents the feature map channel concatenation operation, δ is the sigmoid activation function, represents an element-by-element multiplication operation. This is the local-global feature extracted at this stage

[0098] (2.3) Figure 5 As shown, the dual-phase high-order local-global features and Input to the high-order feature interaction module of the adaptive Cross-Transformer, and use the adaptive Cross-Transformer to transform the bi-phase features and Perform information interaction to obtain high-level features with semantic change information and The constructed adaptive Cross-transformer module is obtained by changing the adaptive multi-head self-attention module in the adaptive Transformer to the adaptive multi-head cross attention module (SAMCA). The main difference between SAMCA and SAMSA is that Derived k2, v2 after adaptive processing and The derived q1 performs cross attention calculation, Derived k1, v1 after adaptive processing and The derived q2 is used for cross attention calculation. The high-order feature interaction module of the adaptive Cross-Transformer is obtained by stacking 8 adaptive Cross-transformer modules;

[0099] (2.4) Figure 6 As shown in the figure, the designed difference extraction module is used. Four difference extraction modules are used to extract four scale dual-phase features. Difference feature extraction is performed. First, the dual-phase features and Subtract pixel by pixel, and then take the absolute value to get the preliminary difference features Then, Respectively and Perform pixel-by-pixel subtraction and then take the absolute value to obtain the improved difference feature and Finally, and The concatenation is performed and input into a 1×1 convolutional layer for fusion feature dimensionality reduction to obtain the final difference feature. The process can be described as:

[0100]

[0101] Among them, f 1×1 represents a convolutional layer with a convolution kernel size of 1×1, and cat represents a feature map concatenation operation;

[0102] (2.5) Figure 7 As shown in the multi-scale fusion decoder, the extracted multi-scale difference features are processed using feature map concatenation and upsampling based on transposed convolution. The upsampling module based on transposed convolution here contains a 3×3 transposed convolution layer with a stride of 2, a batch normalization layer, a Relu activation function layer, and a 3×3 transposed convolution layer with a stride of 1. First, the difference feature map Use the upsampling module based on transposed convolution to perform channel compression and upsampling to obtain the feature map Then, and the difference features with the same scale After concatenation, the 1×1 convolutional layer is used to reduce the dimension and then input into the upsampling module to obtain the feature map. Next, use similar operations to and Process until the change feature map F with the same resolution as the original input is obtained final The process can be described as:

[0103]

[0104] Among them, Up i (i=1,2,3,4) represents four upsampling modules based on transposed convolution, F 1×1 represents a convolutional layer with a convolution kernel of 1×1, and cat represents a feature map concatenation operation;

[0105] (2.6) F final The input is sent to the remote sensing image change detection classifier, which uses two convolution layers with a convolution kernel size of 3×3, a step size of 1, and an output channel of 2 to detect the change feature map F. final Perform channel dimensionality reduction, and then use the Softmax function to generate the final change probability map.

[0106] (3) Use the training set to train the AT-DCNet network, and use the validation set to evaluate the performance of the model during the training process to obtain the optimal model.

[0107] (3.1) Use the binary cross entropy loss function to calculate the error between the predicted image and the true label image:

[0108]

[0109] Among them, y i represents the true label, represents the predicted value of the model, and N is the number of samples. The smaller the cross entropy loss, the better the change detection model.

[0110] (3.2) The present invention uses pytorch to train the model on a single NVIDIA RTX4060 GPU. During training, the batch size is set to 8, SGD is used as the optimizer, the momentum size is 0.9, and the weight decay is 0.0005. The initial learning rate is set to 0.005 and decays linearly according to the number of training cycles. The number of training steps is set to 200. Validation is performed after each training iteration, and the best model on the validation set is saved.

[0111] (4) Input the test set into the trained remote sensing image change detection model to perform change identification and obtain a change result graph.

[0112] The present invention uses two different remote sensing image change detection algorithms to compare with the proposed method. The comparison algorithms used are:

[0113] A dual-temporal image Transformer method proposed by Chen et al. in “Remote Sensing Image Change Detection With Transformers. IEEE Transactions on Geoscience and Remote Sensing, 2021.”, referred to as Method 1.

[0114] A transformer-based Siamese network architecture proposed by Bandara et al. in “A Transformer-Based Siamese Network for Change Detection. IEEE International Geoscience and Remote Sensing Symposium, 2022.”, referred to as Method 2.

[0115] Table 1 shows the performance comparison of the three methods on two public high-resolution remote sensing image change detection datasets, LEVIR-CD and CDD. The mIoU index and mF1 index are two commonly used remote sensing image change detection performance metrics. The larger their values, the better the performance. It can be seen that the method proposed in this paper has the best effect on remote sensing image change detection.

[0116] Table 1 Comparison of accuracy indicators on LEVIR-CD and CDD datasets (%)

[0117]

Claims

1. A remote sensing image change detection method based on adaptive Transformer and deformable convolution, characterized by: The following steps are involved: (1) Construct a remote sensing image change detection dataset, divide the dataset into training set, validation set and test set, and perform data enhancement on the training set; (2) Construct a remote sensing image change detection network AT-DCNet based on adaptive Transformer and deformable convolution; (3) Use the training set to train the AT-DCNet network, and use the validation set to evaluate the performance of the model during the training process to obtain the optimal model; (4) Input the test set into the trained optimal model to perform change identification and obtain the final change detection result.

2. According to the remote sensing image change detection method based on adaptive Transformer and deformable convolution according to claim 1, in step (1), the data set division and data enhancement method are as follows: (1.1) After constructing the remote sensing image change detection dataset, each image is segmented into several non-overlapping image blocks of 256×256 size, and all the image blocks are divided into training set, validation set and test set according to a certain ratio; (1.2) A series of data augmentation operations are performed on the training set, including random horizontal flipping, random vertical flipping, random image cropping, and Gaussian blurring.

3. According to the remote sensing image change detection method based on adaptive Transformer and deformable convolution according to claim 1, in step (2), the remote sensing image change detection network AT-DCNet based on adaptive Transformer and deformable convolution is constructed, which mainly includes a twin feature extraction encoder integrating adaptive Transformer and deformable convolution, a high-order feature interaction module of adaptive Cross-Transformer, a difference extraction module, and a multi-scale fusion decoder, and the specific steps are as follows: (2.1) Input dual-phase remote sensing image I t1 and I t2 , construct a twin feature extraction encoder that integrates adaptive Transformer and deformable convolution to extract multi-scale local-global features of dual-phase remote sensing images. The encoder contains a pair of overlapping downsampling modules and three pairs of local-global fusion modules. In the overlapping downsampling module, a convolution layer with a convolution kernel size of 7×7, a step size of 2, a padding size of 3, and an output channel number of 32 is defined. After batch normalization and Relu activation function operation, an output feature map with a size of 1 / 2 of the input image is obtained. The process can be described as: in, f 7×7 ReLU represents a convolutional layer with a kernel size of 7×7, ReLU represents a Relu activation function, and BN represents a batch normalization layer; (2.2) The local-global fusion module in step (2.1) consists of two branches: local and global. The local branch uses a deformable residual module to extract local features of the remote sensing image. The global branch uses the Patch embedding module and the adaptive Transformer module to extract global features of remote sensing images. Where j = 2, 3, 4. The Patch embedding module passes the input feature through a convolution layer with a convolution kernel size of 2×2 and a stride of 2. The network is divided into several small blocks, and each small block is flattened into a one-dimensional vector. At the same time, in order to retain the spatial position information, the flattened vector sequence is encoded with a position code. Then a linear transformation is performed to embed the vector sequence with the position code into a high-dimensional feature space, and the obtained token sequence P′ is input into the adaptive Transformer module to extract global features. The local features extracted by the local branches And the global features extracted by the global branch Add element by element to get Then input into a convolutional block attention module for further fusion to obtain local-global features (2.3) The dual-phase high-order local-global features and Input to the high-order feature interaction module of the adaptive Cross-Transformer, and use the adaptive Cross-Transformer to transform the bi-phase features and Perform information interaction to obtain high-level features with semantic change information and (2.4) Four difference extraction modules are used to extract four scale bi-phase features Difference feature extraction is performed. First, the dual-phase features and *∈{1,2,3,out}Subtract pixel by pixel, and then take the absolute value to get the preliminary difference feature Then, Respectively and Perform pixel-by-pixel subtraction and then take the absolute value to obtain the improved difference feature and Finally, and The concatenation is performed and input into a 1×1 convolutional layer for fusion feature dimensionality reduction to obtain the final difference feature. The process can be described as: Among them, f 1×1 represents a convolutional layer with a convolution kernel size of 1×1, and cat represents a feature map concatenation operation; (2.5) In the multi-scale fusion decoder, the extracted multi-scale difference features are processed using feature map concatenation and upsampling based on transposed convolution. The upsampling module based on transposed convolution here contains a 3×3 transposed convolution layer with a stride of 2, a batch normalization layer, a Relu activation function layer, and a 3×3 transposed convolution layer with a stride of 1. First, the difference feature map Use the upsampling module based on transposed convolution to perform channel compression and upsampling to obtain the feature map Then, and the difference features with the same scale After concatenation, the 1×1 convolutional layer is used to reduce the dimension and then input into the upsampling module to obtain the feature map. Next, use similar operations to and Process until the change feature map F with the same resolution as the original input is obtained final The process can be described as: Among them, Up i (i=1,2,3,4) represents four upsampling modules based on transposed convolution, F 1×1 represents a convolutional layer with a convolution kernel of 1×1, and cat represents a feature map concatenation operation; (2.6) final The input is sent to the remote sensing image change detection classifier, which uses two convolution layers with a convolution kernel size of 3×3, a step size of 1, and an output channel of 2 to detect the change feature map F. final Perform channel dimensionality reduction, and then use the Softmax function to generate the final change probability map.

4. According to the remote sensing image change detection method based on adaptive Transformer and deformable convolution according to claim 3, the deformable residual module design method in the local branch of step (2.2) is as follows: For input features First, two residual units are constructed. Each residual unit contains two 3×3 convolutional layers, two batch normalization layers, and two Relu activation functions. The step size of the first convolutional layer of the first residual unit is 2, and the step sizes of the other convolutional layers are all 1. Based on these two residual units, local features can be preliminarily extracted. Then, the deformable residual unit is input. The deformable residual unit is constructed by replacing the second 3×3 convolutional layer in the residual unit with a deformable convolutional layer. The local features are finally obtained through the deformable residual unit. The process can be described as: in, f 3×3 represents a convolution layer with a convolution kernel size of 3×3, BN represents batch normalization, ReLU represents the ReLU activation function, Def 3×3 Represents a deformable convolutional layer with a kernel size of 3×3.

5. According to the remote sensing image change detection method based on adaptive Transformer and deformable convolution according to claim 3, in steps (2.2) and (2.3), the adaptive Transformer module and the adaptive Cross-Transformer module are constructed as follows: The adaptive Transformer module first undergoes layer normalization, then uses adaptive multi-head self-attention to capture global information, and introduces a residual branch to add and fuse with the main branch. Adaptive multi-head self-attention through query matrix Q and key matrix K a After the dot product between generates the attention weight, the value matrix V a Multiply it with the attention weight to get the output sequence P SAMSA The specific calculation process can be expressed by the following formula: MHAtten(q,k a ,v a )=cat(h1,…,h s ,…,h n )W O Among them, q represents the query vector, k a and v a They represent the key vector and value vector after adaptive average pooling and 1×1 convolution layer processing, respectively, and h s is a single-head attention mechanism, W O is the output mapping matrix, and Respectively represent q and k in each head a and v a The mapping matrix corresponding to the three vectors, d k K a Based on adaptive multi-head self-attention, we get P SAMSA : Q SAMSA =SAMSA(LN(P ′ ))+P ′ Among them, SAMSA represents the adaptive multi-head self-attention module, LN represents layer normalization processing, P ′ Represents the token sequence in step (2.2); Next, the obtained sequence P SAMSA After passing through two branches respectively, the main branch is normalized and multi-layer perceptron is added and fused with the residual branch to obtain the output sequence P new , the formula is as follows: P new =MLP(LN(P SAMSA ))+P SAMSA Among them, LN represents layer normalization processing, and MLP represents multi-layer perceptron operation. Finally, P new Reshape into a two-dimensional feature map, that is, get the global feature map The constructed adaptive Cross-transformer module is obtained by changing the adaptive multi-head self-attention module in the adaptive Transformer to the adaptive multi-head cross attention module (SAMCA). The main difference between SAMCA and SAMSA is that Derived k2, v2 after adaptive processing and The derived q1 performs cross attention calculation, Derived k1, v1 after adaptive processing and The derived q2 is used for cross attention calculation.

6. According to the remote sensing image change detection method based on adaptive Transformer and deformable convolution according to claim 3, in step (2.2), the convolution block attention module is constructed as follows: This module contains a channel attention module and a spatial attention module. The channel attention module enhances feature representation by weighting feature maps of different channels. First, average pooling and maximum pooling are performed, and then two 1×1 convolutional layers with shared weights are used to learn the weight of each channel. The two results are then added together and the channel attention weight M is obtained through a sigmoid activation function. c , and compare it with the input feature map Multiply channel by channel to get the channel attention feature map The formula is as follows: in, MLP stands for multi-layer perceptron, Avgpool stands for average pooling, Maxpool stands for maximum pooling, δ stands for sigmoid activation function, and ⊙ stands for channel-by-channel multiplication operation; The spatial attention module enhances feature representation by weighting feature maps at different spatial locations. Average pooling and maximum pooling are performed on the channel dimension respectively, and then the two feature maps are spliced ​​on the channel dimension. A 7×7 convolution layer is used to reduce the number of channels to 1, and the spatial attention weight M is obtained through the sigmoid activation function. s , and compare it with the input feature map Multiply to get the spatial attention feature map The formula is as follows: Among them, f 7×7 is a 7×7 convolutional layer, Avgpool is average pooling, Maxpool is maximum pooling, cat represents the feature map channel concatenation operation, δ is the sigmoid activation function, represents an element-by-element multiplication operation. This is the local-global feature extracted at this stage

Images