A remote sensing image change detection method combining convolutional neural network and transformer

Abstract

This invention relates to the field of image processing technology, specifically a remote sensing image change detection method combining convolutional neural networks and Transformer. The invention extracts bi-temporal features from a ResNet18 residual network with max-pooling layers removed, improving feature resolution to preserve detailed information. By inputting PTokens into the Transformer encoder, global information is modeled in the feature space of the PTokens. Then, the global dependencies of the bi-temporal fusion features are calculated in the Transformer decoder using PTokens with global information, improving the problem of limited receptive field in convolutional operations. Resolution is restored through layer-by-layer upsampling, features are aggregated element-by-element, and a change probability map is obtained using a prediction head, ultimately making the change detection results more effective and reliable. This invention can more accurately detect irregular structures of buildings and avoid the adhesion of detection results for dense buildings, while effectively dealing with interference from spectral variations.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically a method for detecting changes in remote sensing images that combines convolutional neural networks and Transformers. Background Technology

[0002] With the development of Earth observation systems both domestically and internationally, remote sensing data acquisition has become increasingly convenient, real-time, and customizable. Massive amounts of remote sensing data with different spectra, modalities, and resolutions have accumulated, gradually evolving into remote sensing big data. As the resolution of remote sensing images improves, the details of ground features within the images become richer, which is crucial for refined remote sensing interpretation. Change detection, as an important research direction in remote sensing, can detect changes in surface objects, such as changes in location, extent, and state, based on the analysis of multi-temporal remote sensing data. Currently, traditional methods are no longer effective in processing remote sensing big data; therefore, researching the latest artificial intelligence technologies to improve the automation and intelligence of remote sensing image change detection is of significant practical importance.

[0003] In recent years, a large number of deep learning change detection methods have emerged, such as the fully convolutional change detection networks designed by Daudt et al., including FC-EF, FC-Siam-Conc, FC-Siam-Diff, and FC-EF-Res. However, convolution operations are limited by the receptive field and can only extract local information.

[0004] To improve the receptive field of convolutional networks, Zhang et al. proposed DSIFN, which models feature context information by introducing spatial attention and channel attention; however, spatial and channel attention units still cannot effectively model global information.

[0005] To address this, Chen et al. proposed the BIT_CD network, which for the first time introduced the Transformer structure into the change detection task. It represents the bi-temporal features extracted by the convolutional neural network as a series of semantic tokens, and then uses the Transformer encoder to model the global information of the semantic tokens. In the Transformer decoder, the semantic tokens with global information are used to establish the global dependencies of the original features. However, the Transformer structure of the BIT_CD network only processes a single feature layer, resulting in the decoder lacking multi-scale features and failing to generate accurate change maps. Summary of the Invention

[0006] The purpose of this invention is to provide a remote sensing image change detection method that combines convolutional neural networks and Transformers to solve the problems mentioned in the background art.

[0007] The technical solution of this invention is: a method for detecting changes in remote sensing images combining convolutional neural networks and Transformers, comprising the following steps:

[0008] Step 1: Construct a change detection dataset;

[0009] Step 2: Read the change detection dataset constructed in Step 1 and perform data augmentation;

[0010] Step 3: Construct a change detection network;

[0011] Step 4: Input the data read in Step 2 into the change detection network constructed in Step 3, and train and validate it.

[0012] Step 5: Read the test set constructed in Step 1, input the preprocessed data into the trained change detection model, and obtain the change detection results.

[0013] Preferably, step 2 includes: during the network training phase, reading image data from the training set and validation set, wherein only the training set data is subjected to data augmentation, while both the training set and validation set data are normalized and converted into tensor form.

[0014] Preferably, the change detection network constructed in step 3 includes: a feature extraction network, a pyramid semantic label generation module, a Transformer structure, an upsampling module, and a network output head.

[0015] Preferably, the feature extraction network is designed as a Siamese network structure, with the two branches of the Siamese network having identical structures and sharing weights. The Siamese network uses a ResNet18 residual network to extract multi-scale features from the T1 and T2 time phase images respectively. In the ResNet18 network, in order to preserve the detailed information in the high-resolution features, the max pooling layer is removed to improve the change detection effect.

[0016] Preferably, the pyramid semantic token generation module is used to represent features as some compact semantic tokens. Specifically, the fused features of the two-phase features are downsampled into four features of different scales (i.e., 1×1, 2×2, 4×4, 8×8) through spatial pyramid pooling. The four pooled features are flattened into a one-dimensional sequence matrix along the spatial dimension, and the four sequence matrices are concatenated along the channel dimension to form a pyramid semantic token (PToken).

[0017] Preferably, the Transformer structure is used to model global information of features. The Transformer structure includes an encoder and a decoder, and both the encoder and decoder are set as a single layer.

[0018] Specifically, the encoder is used to model global information in the feature space of PToken. The Transformer encoder consists of layer normalization, multi-head self-attention mechanism and feedforward neural network. Layer normalization is used to accelerate network convergence, multi-head self-attention mechanism is used to model the global dependency of features, and feedforward neural network is used to filter information in high-dimensional space. The decoder is used for information interaction between the output of the encoder and the input features of the decoder.

[0019] Preferably, the upsampling module is used to restore the resolution of deep features. The upsampling module is designed as a bottleneck structure, and the specific process is as follows: First, the feature dimension C×H×W is compressed to [value missing] through pointwise convolution. Then, a transposed convolution with a 3×3 kernel is used to restore the feature resolution. Finally, a pointwise convolution is performed again to restore the number of channels of the features (i.e., C×2H×2W), and batch normalization and modified linear activation functions are applied after each convolution and transpose convolution. During the upsampling process, feature element-wise addition is used to aggregate multi-layer semantic information. After the feature resolution is restored to the same as the input image, two convolutional units are used to compress the number of channels to 1, and an activation function is added between the two convolutional units. Finally, the Sigmoid function is used to scale the features to [0,1] to generate a change probability map.

[0020] Preferably, step 4 is as follows: the preprocessed data from step 2 is input into the network constructed in step 3, the loss function between the network prediction and the actual change label is calculated, and the network weight parameters are gradually optimized through backpropagation until the network converges; wherein, the binary cross-entropy loss function is used to optimize the network parameters.

[0021] Preferably, step 5 is as follows: read the test set constructed in step 1, perform normalization and tensor transformation, input the preprocessed data into the trained change detection network, obtain the predicted probability map, and finally convert it into change detection results.

[0022] This invention provides an improved remote sensing image change detection method that combines convolutional neural networks and Transformers, which has the following improvements and advantages compared with the prior art:

[0023] Firstly, this invention extracts bi-temporal features from a ResNet18 residual network with the max-pooling layer removed, thereby improving feature resolution and preserving detailed information. Then, the bi-temporal features of each layer are fused, and the fused features are downsampled to four dimensions. By flattening them into a one-dimensional sequence matrix in the spatial dimension and concatenating them in the channel dimension, a pyramid semantic token is formed.

[0024] Secondly, this invention models global information in the feature space of PToken by inputting PToken into the Transformer encoder, and then uses PToken with global information to calculate the global dependency of the bi-temporal fusion feature in the Transformer decoder.

[0025] Thirdly, this invention restores resolution by upsampling layer by layer, aggregates features at different levels by summing features element by element, and obtains a change probability map using a prediction head. This method preserves the detailed information in high-resolution features and provides the model with a global receptive field through the Transformer structure, which improves the problem of limited receptive field in convolution operations, ultimately making the change detection results more effective and reliable. Attached Figure Description

[0026] The present invention will be further explained below with reference to the accompanying drawings and embodiments:

[0027] Figure 1 This is a flowchart of the method of the present invention;

[0028] Figure 2 This is a network structure diagram of the method of the present invention;

[0029] Figure 3 This is the pyramid semantic tag generation diagram of the present invention;

[0030] Figure 4 This is a structural diagram of the Transformer of the present invention;

[0031] Figure 5 This is a structural diagram of the upsampling module of the present invention;

[0032] Figure 6 This is a schematic diagram of some detection results of the present invention on the LEVIR-CD dataset. Detailed Implementation

[0033] The present invention will now be described in detail, and the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] This invention provides an improved method for detecting changes in remote sensing images that combines convolutional neural networks and Transformers. The technical solution of this invention is as follows:

[0035] like Figures 1-6 As shown, a remote sensing image change detection method combining convolutional neural networks and Transformers includes the following steps:

[0036] Step 1: Construct a change detection dataset. First, crop the collected large-format change detection data (including T1 time phase, T2 time phase, and real change label images) into 256×256 images. Then, divide the dataset into training set, validation set, and test set in a 7:1:2 ratio. The training set is used for network training, the validation set is used to check the convergence of the model and adjust the model hyperparameters during training, and the test set is used to test the model and evaluate its generalization ability after training.

[0037] Step 2: Read the change detection dataset constructed in Step 1 and perform data augmentation. This includes reading the training and validation sets during the network training phase. To improve the network's generalization ability, it is necessary to increase the diversity of the training data. Data augmentation operations such as horizontal flipping, vertical flipping, random scale cropping, Gaussian blurring, and color transformation are used to process the training set data. Since the validation set, which is used to check the convergence of the model, does not participate in the training process, no data augmentation operations are performed on it. However, both the training and validation sets must be normalized and converted into tensor form.

[0038] Step 3: Construct a change detection network and train it, which includes the following steps:

[0039] Step 3.1: Construct a feature extraction network and use a Siamese network (i.e., two branches with identical structures and shared weights) to extract features from dual-temporal images. Considering that accurate change detection requires detailed information in high-resolution features, this invention removes the ResNet18 residual network structure with max pooling layers as the branch structure of the Siamese network. Compared with the standard ResNet18 network, the size of the features extracted by the improved network is doubled to alleviate the information loss caused by the pooling layer. After extracting features from four stages, a 3×3 convolutional block is used in each stage to fuse dual-temporal features.

[0040] Step 3.2: Generate Pyramid Semantic Tokens. The pyramid semantic token generation module is used to represent features as compact semantic tokens. To enable these semantic tokens to store richer information, this invention uses spatial pyramid pooling to generate semantic tokens with multi-scale information. Specifically, the fused features from Step 3.1 are downsampled to four sizes: 1×1, 2×2, 4×4, and 8×8 through pooling operations. Then, the four pooled features are flattened into a one-dimensional sequence along the spatial dimension, and then the four one-dimensional sequences are concatenated along the channel dimension to form a pyramid semantic token (PToken). This process can be represented as:

[0041] PToken=Concat(Flatten(SPP(F)))

[0042] In the formula, F represents the input fusion feature, SPP(·) represents spatial pyramid pooling, Flatten(·) represents flattening the four pooling features into a one-dimensional sequence matrix, and Concat(·) represents concatenating the four flattened sequence matrices along the channel dimension.

[0043] Step 3.3: Modeling the global context information of the fused features. Due to the introduction of the multi-head attention mechanism, the Transformer structure can effectively model the global context information of the features. The Transformer structure includes an encoder and a decoder, which are basically the same in structure. Both are composed of layer normalization plus multi-head self-attention mechanism and layer normalization plus feedforward neural network in series. Among them, layer normalization can accelerate network convergence. In order to reduce the overall number of parameters and computational cost of the network, both the Transformer encoder and decoder are set to one layer.

[0044] First, a Transformer encoder is used to model global information in the PToken feature space. After layer normalization, the PToken is input into a multi-head self-attention mechanism, and a query vector Q is generated by performing a linear transformation on the PToken. E Key vector K E Sum vector V E , can be represented as:

[0045] Q E =LN(PToken)W Q

[0046] K E =LN(PToken)W K

[0047] V E =LN(PToken)W V

[0048] In the formula, LN(·) represents layer normalization, and W Q W K and W V They represent Q respectively E K E and V E Weighting coefficients for linear projection;

[0049] The self-attention mechanism is used to multiply the Q vector and K vector to calculate the correlation between each pixel. After scaling and applying the Softmax function, attention weights are generated. Finally, these attention weights are applied to the V vector to complete the modeling of the global information of the PToken. This process can be represented as:

[0050]

[0051] In the formula, d represents the scaling factor;

[0052] The multi-head self-attention mechanism maps the three vectors Q, K, and V to multiple spaces to extract the multi-semantic features. By concatenating the outputs of the multi-head self-attention mechanism and then performing a linear transformation, the result is projected back onto the original feature space. The calculation formula is as follows:

[0053]

[0054] MSA = MSA(T) () )=Concat(head1, head2,..., headn)W 0

[0055] In the formula, Let W represent the Q, K, and V weight matrices of the j-th self-attention head, respectively. 0 The weights represent the linear transformation coefficients, MSA(·) represents the multi-head self-attention mechanism, Concat(·) represents the concatenation operation, and n represents the n self-attention heads.

[0056] After the global information model of PToken is completed, its input features are linearly projected into a high-dimensional space, and effective information is filtered through the Gaussian error linear activation function (GELU). Finally, it is linearly projected into a low-dimensional space.

[0057] This process can be represented as:

[0058] FFN = GELU(T () W1)W2

[0059] In the formula, FFN represents a feedforward neural network, W1 and W2 are linear projection matrices, and GELU(·) represents the Gaussian error linear activation function;

[0060] The Transformer decoder uses PTokens, which contain global information, to establish global dependencies on the fused feature map; in the self-attention mechanism, QD is generated from the fused features, while KD and VD are generated from PTokens.

[0061] Right now:

[0062] Q D =LN(FF)W Q

[0063] K D =LN(PToken)W K

[0064] V D =LN(PToken)W V

[0065] In the formula, FF represents the fused feature; at the same time, the Transformer decoder also uses a feedforward neural network to filter information.

[0066] Step 3.4: Upsample the deep features to the same size as the input image and output the change probability map; starting from the deepest layer features, through upsampling of each layer and element-wise addition of features, the purpose of aggregating multi-level semantic information and restoring feature resolution is achieved; to avoid a significant increase in the computational load and parameter count of the network, this invention designs the upsampling module as a bottleneck structure, first compressing the feature dimension C×H×W to a smaller size through pointwise convolution. Then, a transposed convolution with a 3×3 kernel is used to restore the feature resolution. Finally, perform pointwise convolution to recover the number of feature channels (i.e., C×2H×2W); and use batch normalization (BN) and modified linear activation function (ReLU) after each convolution or transpose convolution;

[0067] The calculation process for this module is as follows:

[0068] F - =ReLU(BN(Conv1(F in )))

[0069] F up =ReLU(BN(TransConv(F) - )))

[0070] F + =ReLU(BN(Conv2(F up )))

[0071] In the formula, F - F represents the feature after channel compression, and Fup represents the feature after upsampling. + The features represent channel recovery, BN(·) represents batch normalization, ReLU(·) represents modified linear activation function, Conv1(·) represents convolution operation with compressed channel number, TransConv(·) represents transposed convolution, and Conv2(·) represents convolution operation with restored channel number.

[0072] After restoring the feature resolution, two convolutional units are used to compress the number of channels to 1, and a ReLU activation function is added between the two convolutional units. Finally, the Sigmoid function is used to scale the features to [0,1] to generate the predicted probability map. This process can be represented as:

[0073] Output=Sigmoid(Conv2(Relu(Conv1(F))))

[0074] In the formula, Conv1(·) represents the convolution operation that initially reduces the number of feature channels, Conv2(·) represents the convolution operation that reduces the number of feature channels to 1, and Sigmoid(·) represents the sigmoid function.

[0075] Step 4: Input the preprocessed data from Step 2 into the network constructed in Step 3 for training and validation. Calculate the loss function between the network's predictions and the actual change labels. Through continuous backpropagation iterations, gradually optimize the network weight parameters until the network converges.

[0076] Specifically, after the network is built, data can be input into the network for training. During the training phase, the binary cross-entropy loss function is used to calculate the error between the actual change and the predicted change to guide the direction of network training. The formula for calculating the binary cross-entropy loss is as follows:

[0077]

[0078] In the formula, N represents the number of samples in the image; y n y represents the true value of pixel n. If the pixel belongs to the change category, y n If y is 1, otherwise y n =0; and These represent the probabilities that pixel n belongs to the changing class and the invariant class after being predicted by the model, respectively;

[0079] Step 5: Read the test set constructed in Step 1, and after normalization and tensor transformation, input the preprocessed data into the trained change detection model. Use the trained model to perform change detection on the preprocessed data to obtain a predicted probability map, and finally convert it into a change detection result. That is, determine the category of each pixel based on its probability value, and finally generate a change detection map.

[0080] Furthermore, the superiority of the method of this invention is verified below with specific experiments. This experiment uses the LEVIR-CD dataset as the experimental data. After fixed-scale cropping, the training set, validation set, and test set contain 7120, 1024, and 2048 images of size 256×256, respectively. This experiment is conducted in the PyTorch deep learning framework, using the AdamW optimizer to optimize the training process. It iterates for 100 epochs, with an initial learning rate set to 0.0001, decaying by 0.000001 in each epoch, and a batch size of 8.

[0081] To verify the superiority of the method of this invention, it was compared with two other change detection networks that also introduced the Transformer structure, namely BIT_CD and ChangeFormer. The accuracy evaluation indicators of the detection results were F1 score, intersection-over-union ratio (IoU), and overall accuracy (OA); the specific detection results are shown in Table 1.

[0082] Model F1-score / % IoU / % OA / % BIT_CD 89.96 81.75 98.89 ChangeFormer 90.40 82.48 99.04 Method of the present invention 91.79 84.83 99.17

[0083] Table 1 Performance comparison of different change detection methods

[0084] As can be seen from Table 1, the method of the present invention achieves the best results in all three accuracy indicators, and compared with other methods, all three accuracy indicators are significantly improved, which demonstrates the effectiveness and superiority of the method of the present invention.

[0085] Figure 6 The results of change detection using different methods are shown in the figure. It can be seen that the method of the present invention has the best visual effect of change detection results, can more accurately detect the irregular structure of buildings, and can avoid the adhesion of detection results of dense buildings, while effectively dealing with the interference of spectral changes.

[0086] The foregoing description enables those skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting changes in remote sensing images combining convolutional neural networks and Transformers, characterized in that: Includes the following steps: Step 1: Construct a change detection dataset. Crops the large-format remote sensing images and their corresponding change label maps according to a uniform scale. Then, divide the cropped T1 time-phase images, T2 time-phase images, and change label maps into training set, validation set, and test set in a ratio of 7:1:

2. Step 2: Read the change detection dataset constructed in Step 1 and perform data augmentation. During the network training phase, read the image data of the training set and the validation set. Data augmentation is performed only on the training set data, while the training set data and the validation set data are normalized and converted into tensor form. Step 3: Construct a change detection network, including: a feature extraction network, a pyramid semantic label generation module, a Transformer structure, an upsampling module, and a network output head. The feature extraction network is designed as a Siamese network structure, with the two branches of the Siamese network having identical structures and sharing weights. The Siamese network uses a ResNet18 residual network to extract multi-scale features from the T1 and T2 time-phase images, respectively. In the ResNet18 network, in order to preserve the detailed information in the high-resolution features, the max pooling layer is removed to improve the change detection effect. The pyramid semantic token generation module is used to represent features as some compact semantic tokens. Specifically, the fused features of the dual-temporal features are downsampled into four features of different scales through spatial pyramid pooling. The four pooled features are flattened into a one-dimensional sequence matrix along the spatial dimension, and the four sequence matrices are spliced ​​together along the channel dimension to form a pyramid semantic PToken. The Transformer structure is used to model global information of features. The Transformer structure includes an encoder and a decoder, and both the encoder and decoder are set as a single layer. Specifically, the encoder is used to model global information in the feature space of PToken. The Transformer encoder consists of layer normalization, multi-head self-attention mechanism and feedforward neural network. Layer normalization is used to accelerate network convergence, multi-head self-attention mechanism is used to model the global dependency of features, and feedforward neural network is used to filter information in high-dimensional space. The decoder is used for information interaction between the output of the encoder and the input features of the decoder. Step 4: Input the data read in Step 2 into the change detection network constructed in Step 3, and train and validate it. Step 5: Read the test set constructed in Step 1, input the preprocessed data into the trained change detection model, and obtain the change detection results.

2. The remote sensing image change detection method combining convolutional neural networks and Transformer as described in claim 1, characterized in that: The upsampling module is used to restore the resolution of deep features. The upsampling module is designed as a bottleneck structure, and the specific process is as follows: first, the feature dimensions are reduced through pointwise convolution. Compress to Then use convolution kernel for Transposed convolution restores feature resolution to Finally, perform one more pointwise convolution to recover the number of feature channels. Furthermore, batch normalization and modified linear activation functions are applied after each convolution and transpose convolution; during the upsampling process, feature element-wise addition is used to aggregate multi-layer semantic information. After restoring the feature resolution to the same level as the input image, two convolutional units are used to compress the number of channels to 1, and an activation function is added between the two convolutional units. Finally, the Sigmoid function is used to scale the features to [0,1] to generate a probability map of change.

3. The remote sensing image change detection method combining convolutional neural networks and Transformer as described in claim 1, characterized in that: The specific process of step 4 is as follows: input the preprocessed data from step 2 into the network constructed in step 3, calculate the loss function between the network prediction and the actual change label, and gradually optimize the network weight parameters through continuous backpropagation until the network converges; wherein, the binary cross-entropy loss function is used to optimize the network parameters.

4. The remote sensing image change detection method combining convolutional neural networks and Transformer according to claim 1, characterized in that: The specific process of step 5 is as follows: read the test set constructed in step 1, perform normalization and tensor transformation, input the preprocessed data into the trained change detection network, obtain the predicted probability map, and finally convert it into change detection results.

Images