A remote sensing image change detection method

Abstract

The present invention belongs to the technical field of remote sensing image change detection, and specifically relates to a remote sensing image change detection method. The method first obtains two remote sensing images of different phases, and then inputs them into a remote sensing image change detection model to obtain a change detection result corresponding to the original remote sensing image; wherein the remote sensing image change detection model includes a feature extraction module and a nested U-Net network, the inputs of the two deep hierarchical Transformer networks in the feature extraction module are respectively the two inputs of the remote sensing image change detection model, and the node X in the nested U-Net network 0,4 The output of is the output of the remote sensing image change detection model, and the output of the rth stage of the two deep hierarchical Transformer networks is concatenated and connected to the node X in the nested U-Net network. r‑1,0 This model makes full use of features of different scales. In the nested U-Net structure, there is information exchange between features of different scales, which can make more full use of features of different scales to improve the accuracy of change detection.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image change detection, and in particular relates to a remote sensing image change detection method. Background Art

[0002] High-resolution remote sensing image change detection has a wide range of applications in land use surveys, geospatial data updates, disaster monitoring and assessment, and is also one of the important research topics in remote sensing image processing and analysis. For registered dual-temporal remote sensing images, the purpose of change detection is to distinguish whether the corresponding pixel pairs have changed. Usually, the changed area is marked with 1, and the unchanged area is marked with 0. Remote sensing image change detection methods can be roughly divided into four types, namely direct pixel comparison method, object-oriented image analysis method, machine learning-based method, and the latest deep learning-based method. The most simple change idea is to judge whether the pixels at the corresponding position have changed by analyzing and comparing each pixel. Representative works in this regard include direct comparison method and image transformation-based method. Object-oriented change detection method does not directly analyze pixels, but takes spatially adjacent and spectrally similar objects as basic processing units. Compared with the method of directly comparing and analyzing pixels, object-oriented change detection method has better detection effect, but how to determine the optimal scale of the object has always been an urgent problem to be solved.

[0003] In recent years, with the continuous improvement of computing power and data acquisition capabilities, learning-based change detection methods have greatly improved the accuracy of change detection. In the early days, shallow models represented by support vector machines and random forests were widely used in remote sensing image change detection tasks. However, the different imaging conditions of remote sensing images at different times will cause the real change area to be confused with the background and difficult to identify, which will lead to the detection effect of directly applying these shallow models is often not ideal. Therefore, these shallow models often require artificial design of complex feature extraction rules to ensure the detection effect of changes. Typical feature extraction methods include texture features, statistical features, and spatial structure features. However, artificially designed feature extraction rules often rely on expert knowledge, and need to set a large number of hyperparameters according to different data to ensure the detection effect.

[0004] Compared with shallow models such as support vector machines and random forests, deep learning models can automatically learn to extract abstract features for downstream tasks, and can achieve higher recognition accuracy with the support of sufficient labeled training data. Therefore, they have received widespread attention in the field of remote sensing image change detection. There are generally two ideas for change detection methods based on deep learning. One idea is to directly splice the dual-phase remote sensing images, and then input them into the full convolutional network to output change information. Classic full convolutional network models include FCN, U-Net, DeepLab series, PsPNet+, UperNet, etc. Another idea is to use a twin network structure to adapt to the dual-branch input of the change detection task, and then splice the features output by the twin network, and then output the change detection results. The attention mechanism enables the deep learning model to ignore irrelevant information, so as to pay more attention to the areas that are important to the task, thereby improving the recognition performance of the deep learning model. To this end, researchers have explored a variety of attention mechanisms suitable for change detection tasks. In addition, models such as high-resolution networks, recurrent neural networks, and generative adversarial networks have also been introduced into the change detection task to better identify the change area.

[0005] Although deep learning methods have made great progress in change detection tasks, most existing methods use convolutional neural networks as the backbone network for feature extraction. However, the receptive field of convolutional neural networks is limited, which makes these models unable to perceive a wider range of contextual information. The self-attention mechanism represented by the visual Transformer has a wider range of receptive fields. Therefore, some scholars introduced the Transformer as a decoder based on CNN to overcome the disadvantage of the limited receptive field of CNN. However, using the Transformer as a decoder cannot give full play to the advantages of the self-attention mechanism, the change detection accuracy is low, and the original Transformer has a high computational complexity. Summary of the invention

[0006] The purpose of the present invention is to provide a remote sensing image change detection method to solve the problem of low accuracy of remote sensing image change detection in the prior art.

[0007] In order to solve the above technical problems, the technical solutions provided by the present invention and the corresponding beneficial effects of the technical solutions are as follows:

[0008] A remote sensing image change detection method of the present invention comprises the following steps:

[0009] 1) Obtain two remote sensing images of different phases and call them two original remote sensing images;

[0010] 2) Inputting the two original remote sensing images into the remote sensing image change detection model to obtain the change detection results corresponding to the two original remote sensing images;

[0011] The remote sensing image change detection model is trained using remote sensing image pairs of different historical phases and corresponding change detection results as training sets; the remote sensing image change detection model includes a feature extraction module and a nested U-Net network, and the feature extraction module includes two deep hierarchical Transformer networks with the same structure and shared weights;

[0012] The deep hierarchical Transformer network includes patch partition units connected in sequence and four stages, the four stages are stage 1, stage 2, stage 3 and stage 4 respectively;

[0013] Node X in the nested U-Net network i,j Output feature map x i,j The calculation formula is: In the formula, i represents the number of layers, j represents the jth convolutional layer of the current layer, and h(·) represents the convolution operation on the feature first and then the ReLU activation function operation. represents an upsampling layer, [] represents the connection operation of features;

[0014] The inputs of the two deep hierarchical Transformer networks in the feature extraction module are the two inputs of the remote sensing image change detection model, and the node X in the nested U-Net network 0,4 The output of is the output of the remote sensing image change detection model, and the output of the rth stage of the two deep hierarchical Transformer networks is concatenated and connected to the node X in the nested U-Net network. r -1,0 , r=1,2,3,4.

[0015] The beneficial effects of the above technical solution are as follows: the remote sensing image change detection model of the present invention utilizes a deep hierarchical Transformer network to extract the features of the remote sensing image. The deep hierarchical Transformer network can take into account both local and global information at the same time, thereby improving the accuracy of change detection; moreover, in order to make full use of features of different scales, the extracted features are spliced ​​and input into a nested U-Net network to complete change detection. In the nested U-Net structure, there is information exchange between features of different scales, and features of different scales can be more fully utilized to improve the accuracy of change detection.

[0016] The above method is further improved, and after step 2), it also includes steps 3) to 5): 3) performing the same enhancement processing on two original remote sensing images of different phases to obtain two enhanced remote sensing images; 4) inputting both the two enhanced remote sensing images into the remote sensing image change detection model, and performing the inverse operation corresponding to the enhancement processing on the output results, so as to obtain the change detection results corresponding to the two enhanced remote sensing images; 5) determining the change of the two remote sensing images of different phases based on the change detection results corresponding to the two original remote sensing images and the change detection results corresponding to the two enhanced remote sensing images.

[0017] The beneficial effect of the above technical solution is that in addition to using the change detection results corresponding to the two original remote sensing images, the change detection results of the enhanced image pair are also used to integrate multiple detection results to improve the final change detection accuracy.

[0018] The above method is further improved. In step 3), the enhancement processing includes at least one of horizontal mirroring, vertical mirroring, rotation of 90 degrees and rotation of 270 degrees. Then, in step 4), the inverse operations corresponding to the enhancement processing are horizontal mirroring, vertical mirroring, rotation of 270 degrees and rotation of 90 degrees respectively.

[0019] The above method is further improved, and the enhancement processing is horizontal mirroring, vertical mirroring, rotation of 90 degrees and rotation of 270 degrees. Then in step 5), the change detection results corresponding to the two original remote sensing images and the change detection results corresponding to the two enhanced remote sensing images include a total of 5 change detection results; if more than 2 of the 5 change detection results at a certain position are predicted to be changes, the change detection result at that position is determined to be a change.

[0020] The beneficial effects of the above technical solution are: comprehensive use of four enhancement processes to improve the change detection accuracy.

[0021] The above method is further improved, and the first stage, the second stage, the third stage and the fourth stage all include Transformer Block modules, and each Transformer Block module is expressed as:

[0022]

[0023]

[0024]

[0025]

[0026] In the formula, z l-1 Represents the input of the Transformer Block module, zl Represents the intermediate variable of the Transformer Block module, z l+1 Represents the output of the Transformer Block module; W-MSA() represents a local window self-attention calculation, which refers to dividing the input into multiple windows and performing self-attention calculation on the feature map in each window; SW-MSA() represents a moving window self-attention calculation, which refers to moving the window based on the local window self-attention, and performing self-attention calculation on the feature map in the moved window; LN() represents an LN operation, which refers to the LayerNorm layer that performs normalization operation in the channel direction; MPL() represents an MLP operation, which refers to a fully connected layer used to enhance the nonlinearity of the model.

[0027] The beneficial effects of the above technical solution are as follows: the self-attention unit in the Transformer Block module includes a local window self-attention calculation, which can effectively reduce the overall model complexity and improve computational efficiency; moreover, the self-attention unit in the TransformerBlock module also includes a moving window self-attention calculation, which can enable information interaction between different windows.

[0028] The above method is further improved. When training the remote sensing image change detection model, the loss function used is a combination of the Focal loss function and the Dice loss function. The calculation formula of the Focal loss function is: FL(p t )=-η t (1-p t ) ρ log(p t ), where p t represents the probability of the model predicting the category, η t represents the weight parameter that controls the contribution of positive and negative samples to the total loss, and ρ is the manually set focus parameter; the calculation formula of the Dice loss function is: Where s represents the Dice coefficient, which is used to measure the similarity between two samples; |X∩Y| is the intersection between two sets, and |X| and |Y| represent the union of two sets respectively.

[0029] The beneficial effects of the above technical solution are: Focal loss and Dice loss are both suitable for the situation where the sample distribution is uneven, and can effectively improve the training stability and training effect of the model, so as to achieve the purpose of better training the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a structural diagram of the remote sensing image change detection model of the present invention;

[0031] FIG2( a ) is a structural diagram of the deep hierarchical Transformer model used in the present invention;

[0032] FIG2( b ) is a structural diagram of the Transformer Block module used in the present invention;

[0033] Figure 3(a) is a schematic diagram of MSA;

[0034] Figure 3(b) is a schematic diagram of the W-MSA mechanism;

[0035] Figure 3(c) is a schematic diagram of the SW-MSA mechanism;

[0036] Figure 4 It is a structural diagram of the nested U-Net network used in the present invention;

[0037] Figure 5 It is a schematic diagram of model fusion of the present invention;

[0038] Figure 6 This is a schematic diagram of the impact of different Focal loss weights on change detection accuracy;

[0039] Figure 7 (1a) is the input image 1 of the first image pair in the LEVIR-CD dataset;

[0040] Figure 7 (1b) is the input image 2 of the first image pair in the LEVIR-CD dataset;

[0041] Figure 7 (1c) is the ground reference image of the first image pair in the LEVIR-CD dataset;

[0042] Figure 7 (1d) is the result of using DeepLabV3 to perform change detection on the first image pair in the LEVIR-CD dataset;

[0043] Figure 7 (1e) is the result of using U-Net to perform change detection on the first image pair in the LEVIR-CD dataset;

[0044] Figure 7 (1f) is the result of using PSPNet to perform change detection on the first image pair in the LEVIR-CD dataset;

[0045] Figure 7 (1g) is the result of using UperNet to perform change detection on the first image pair in the LEVIR-CD dataset;

[0046] Figure 7 (1h) is the result of using MSPSNet to perform change detection on the first image pair in the LEVIR-CD dataset;

[0047] Figure 7 (1i) is the result of using SESNet to perform change detection on the first image pair in the LEVIR-CD dataset;

[0048] Figure 7 (1j) is the result of using F3SNet to perform change detection on the first image pair in the LEVIR-CD dataset;

[0049] Figure 7 (1k) is the result of using DSAMNet to perform change detection on the first image pair in the LEVIR-CD dataset;

[0050] FIG7 (11) is a graph showing the result of change detection on the first image pair in the LEVIR-CD dataset using TrmNU-NetF;

[0051] Figure 7 (2a) is the input image 1 of the second image pair in the LEVIR-CD dataset;

[0052] Figure 7 (2b) is the input image 2 of the second image pair in the LEVIR-CD dataset;

[0053] Figure 7 (2c) is the ground reference image of the second image pair in the LEVIR-CD dataset;

[0054] Figure 7 (2d) is a graph showing the result of using DeepLabV3 to perform change detection on the second image pair in the LEVIR-CD dataset;

[0055] Figure 7 (2e) is the result of using U-Net to perform change detection on the second image pair in the LEVIR-CD dataset;

[0056] Figure 7 (2f) is the result of using PSPNet to perform change detection on the second image pair in the LEVIR-CD dataset;

[0057] Figure 7 (2g) is the result of using UperNet to perform change detection on the second image pair in the LEVIR-CD dataset;

[0058] Figure 7 (2h) is the result of using MSPSNet to perform change detection on the second image pair in the LEVIR-CD dataset;

[0059] Figure 7 (2i) is the result of using SESNet to perform change detection on the second image pair in the LEVIR-CD dataset;

[0060] Figure 7 (2j) is the result of using F3SNet to perform change detection on the second image pair in the LEVIR-CD dataset;

[0061] Figure 7 (2k) is the result of using DSAMNet to perform change detection on the second image pair in the LEVIR-CD dataset;

[0062] FIG7 (2l) is a graph showing the result of change detection on the second image pair in the LEVIR-CD dataset using TrmNU-NetF;

[0063] Figure 7 (3a) is the input image 1 of the third image pair in the LEVIR-CD dataset;

[0064] Figure 7 (3b) is the input image 2 of the third image pair in the LEVIR-CD dataset;

[0065] Figure 7 (3c) is the ground reference image of the third image pair in the LEVIR-CD dataset;

[0066] Figure 7 (3d) is a graph showing the result of change detection on the third image pair in the LEVIR-CD dataset using DeepLabV3;

[0067] Figure 7 (3e) is the result of using U-Net to perform change detection on the third image pair in the LEVIR-CD dataset;

[0068] Figure 7 (3f) is the result of using PSPNet to perform change detection on the third image pair in the LEVIR-CD dataset;

[0069] Figure 7 (3g) is the result of using UperNet to perform change detection on the third image pair in the LEVIR-CD dataset;

[0070] Figure 7 (3h) is the result of using MSPSNet to perform change detection on the third image pair in the LEVIR-CD dataset;

[0071] Figure 7 (3i) is the result of using SESNet to perform change detection on the third image pair in the LEVIR-CD dataset;

[0072] Figure 7 (3j) is the result of using F3SNet to perform change detection on the third image pair in the LEVIR-CD dataset;

[0073] Figure 7 (3k) is the result of using DSAMNet to perform change detection on the third image pair in the LEVIR-CD dataset;

[0074] FIG7 (3l) is a graph showing the result of change detection on the third image pair in the LEVIR-CD dataset using TrmNU-NetF;

[0075] Figure 7 (4a) is the input image 1 of the 4th image pair in the LEVIR-CD dataset;

[0076] Figure 7 (4b) is the input image 2 of the 4th image pair in the LEVIR-CD dataset;

[0077] Figure 7 (4c) is the ground reference image of the 4th image pair in the LEVIR-CD dataset;

[0078] Figure 7 (4d) is the result of using DeepLabV3 to perform change detection on the 4th image pair in the LEVIR-CD dataset;

[0079] Figure 7 (4e) is the result of using U-Net to perform change detection on the 4th image pair in the LEVIR-CD dataset;

[0080] Figure 7 (4f) is the result of using PSPNet to perform change detection on the 4th image pair in the LEVIR-CD dataset;

[0081] Figure 7 (4g) is the result of using UperNet to perform change detection on the 4th image pair in the LEVIR-CD dataset;

[0082] Figure 7 (4h) is the result of using MSPSNet to perform change detection on the 4th image pair in the LEVIR-CD dataset;

[0083] Figure 7 (4i) is the result of using SESNet to perform change detection on the 4th image pair in the LEVIR-CD dataset;

[0084] Figure 7 (4j) is the result of using F3SNet to perform change detection on the 4th image pair in the LEVIR-CD dataset;

[0085] Figure 7 (4k) is the result of using DSAMNet to perform change detection on the 4th image pair in the LEVIR-CD dataset;

[0086] Fig. 7 (4l) is a graph showing the result of change detection on the 4th image pair in the LEVIR-CD dataset using TrmNU-NetF;

[0087] Figure 7 (5a) is the input image 1 of the 5th image pair in the LEVIR-CD dataset;

[0088] Figure 7 (5b) is the input image 2 of the 5th image pair in the LEVIR-CD dataset;

[0089] Figure 7 (5c) is the ground reference image of the 5th image pair in the LEVIR-CD dataset;

[0090] Figure 7 (5d) is the result of using DeepLabV3 to perform change detection on the 5th image pair in the LEVIR-CD dataset;

[0091] Figure 7 (5e) is the result of using U-Net to perform change detection on the 5th image pair in the LEVIR-CD dataset;

[0092] Figure 7 (5f) is the result of using PSPNet to perform change detection on the 5th image pair in the LEVIR-CD dataset;

[0093] Figure 7 (5g) is the result of using UperNet to perform change detection on the 5th image pair in the LEVIR-CD dataset;

[0094] Figure 7 (5h) is the result of using MSPSNet to perform change detection on the 5th image pair in the LEVIR-CD dataset;

[0095] Figure 7 (5i) is the result of using SESNet to perform change detection on the 5th image pair in the LEVIR-CD dataset;

[0096] Figure 7 (5j) is the result of using F3SNet to perform change detection on the 5th image pair in the LEVIR-CD dataset;

[0097] Figure 7 (5k) is the result of using DSAMNet to perform change detection on the 5th image pair in the LEVIR-CD dataset;

[0098] Figure 7 (5l) is a graph showing the result of change detection on the fifth image pair in the LEVIR-CD dataset using TrmNU-NetF;

[0099] Figure 8 (1a) is the input image 1 of the first image pair in the SYSU-CD dataset;

[0100] Figure 8 (1b) is the input image 2 of the first image pair in the SYSU-CD dataset;

[0101] Figure 8 (1c) is the ground reference image of the first image pair in the SYSU-CD dataset;

[0102] Figure 8 (1d) is the result of using DeepLabV3 to perform change detection on the first image pair in the SYSU-CD dataset;

[0103] Figure 8 (1e) is the result of using U-Net to perform change detection on the first image pair in the SYSU-CD dataset;

[0104] Figure 8 (1f) is the result of using PSPNet to perform change detection on the first image pair in the SYSU-CD dataset;

[0105] Figure 8 (1g) is the result of using UperNet to perform change detection on the first image pair in the SYSU-CD dataset;

[0106] Figure 8 (1h) is the result of using MSPSNet to perform change detection on the first image pair in the SYSU-CD dataset;

[0107] Figure 8 (1i) is the result of using SESNet to perform change detection on the first image pair in the SYSU-CD dataset;

[0108] Figure 8 (1j) is the result of using F3SNet to perform change detection on the first image pair in the SYSU-CD dataset;

[0109] Figure 8 (1k) is the result of using DSAMNet to perform change detection on the first image pair in the SYSU-CD dataset;

[0110] Figure 8 (1l) is a graph showing the result of using TrmNU-NetF to perform change detection on the first image pair in the SYSU-CD dataset;

[0111] Figure 8 (2a) is the input image 1 of the second image pair in the SYSU-CD dataset;

[0112] Figure 8 (2b) is the input image 2 of the second image pair in the SYSU-CD dataset;

[0113] Figure 8 (2c) is the ground reference image of the second image pair in the SYSU-CD dataset;

[0114] Figure 8 (2d) is the result of using DeepLabV3 to perform change detection on the second image pair in the SYSU-CD dataset;

[0115] Figure 8 (2e) is the result of using U-Net to perform change detection on the second image pair in the SYSU-CD dataset;

[0116] Figure 8 (2f) is the result of using PSPNet to perform change detection on the second image pair in the SYSU-CD dataset;

[0117] Figure 8 (2g) is the result of using UperNet to perform change detection on the second image pair in the SYSU-CD dataset;

[0118] Figure 8 (2h) is the result of using MSPSNet to perform change detection on the second image pair in the SYSU-CD dataset;

[0119] Figure 8 (2i) is the result of using SESNet to perform change detection on the second image pair in the SYSU-CD dataset;

[0120] Figure 8 (2j) is the result of using F3SNet to perform change detection on the second image pair in the SYSU-CD dataset;

[0121] Figure 8 (2k) is the result of using DSAMNet to perform change detection on the second image pair in the SYSU-CD dataset;

[0122] Figure 8 (2l) is a graph showing the result of using TrmNU-NetF to perform change detection on the second image pair in the SYSU-CD dataset;

[0123] Figure 8 (3a) is the input image 1 of the third image pair in the SYSU-CD dataset;

[0124] Figure 8 (3b) is the input image 2 of the third image pair in the SYSU-CD dataset;

[0125] Figure 8 (3c) is the ground reference image of the third image pair in the SYSU-CD dataset;

[0126] Figure 8 (3d) is the result of change detection on the third image pair in the SYSU-CD dataset using DeepLabV3;

[0127] Figure 8 (3e) is the result of using U-Net to perform change detection on the third image pair in the SYSU-CD dataset;

[0128] Figure 8 (3f) is the result of using PSPNet to perform change detection on the third image pair in the SYSU-CD dataset;

[0129] Figure 8 (3g) is the result of change detection on the third image pair in the SYSU-CD dataset using UperNet;

[0130] Figure 8 (3h) is the result of using MSPSNet to perform change detection on the third image pair in the SYSU-CD dataset;

[0131] Figure 8 (3i) is the result of using SESNet to perform change detection on the third image pair in the SYSU-CD dataset;

[0132] Figure 8 (3j) is the result of using F3SNet to perform change detection on the third image pair in the SYSU-CD dataset;

[0133] Figure 8 (3k) is the result of change detection on the third image pair in the SYSU-CD dataset using DSAMNet;

[0134] Figure 8 (3l) is a graph showing the result of change detection on the third image pair in the SYSU-CD dataset using TrmNU-NetF;

[0135] Figure 8 (4a) is the input image 1 of the 4th image pair in the SYSU-CD dataset;

[0136] Figure 8 (4b) is the input image 2 of the 4th image pair in the SYSU-CD dataset;

[0137] Figure 8 (4c) is the ground reference image of the 4th image pair in the SYSU-CD dataset;

[0138] Figure 8 (4d) is the result of change detection on the 4th image pair in the SYSU-CD dataset using DeepLabV3;

[0139] Figure 8 (4e) is the result of using U-Net to perform change detection on the 4th image pair in the SYSU-CD dataset;

[0140] Figure 8 (4f) is the result of using PSPNet to perform change detection on the 4th image pair in the SYSU-CD dataset;

[0141] Figure 8 (4g) is the result of change detection using UperNet on the 4th image pair in the SYSU-CD dataset;

[0142] Figure 8 (4h) is the result of using MSPSNet to perform change detection on the 4th image pair in the SYSU-CD dataset;

[0143] Figure 8 (4i) is the result of using SESNet to perform change detection on the 4th image pair in the SYSU-CD dataset;

[0144] Figure 8 (4j) is the result of using F3SNet to perform change detection on the 4th image pair in the SYSU-CD dataset;

[0145] Figure 8 (4k) is the result of change detection on the 4th image pair in the SYSU-CD dataset using DSAMNet;

[0146] Figure 8 (4l) is a graph showing the result of change detection on the 4th image pair in the SYSU-CD dataset using TrmNU-NetF;

[0147] Figure 8 (5a) is the input image 1 of the 5th image pair in the SYSU-CD dataset;

[0148] Figure 8 (5b) is the input image 2 of the 5th image pair in the SYSU-CD dataset;

[0149] Figure 8 (5c) is the ground reference image of the 5th image pair in the SYSU-CD dataset;

[0150] Figure 8 (5d) is the result of change detection on the fifth image pair in the SYSU-CD dataset using DeepLabV3;

[0151] Figure 8 (5e) is the result of using U-Net to perform change detection on the 5th image pair in the SYSU-CD dataset;

[0152] Figure 8 (5f) is the result of using PSPNet to perform change detection on the 5th image pair in the SYSU-CD dataset;

[0153] Figure 8 (5g) is the result of change detection using UperNet on the 5th image pair in the SYSU-CD dataset;

[0154] Figure 8 (5h) is the result of using MSPSNet to perform change detection on the 5th image pair in the SYSU-CD dataset;

[0155] Figure 8 (5i) is the result of using SESNet to perform change detection on the 5th image pair in the SYSU-CD dataset;

[0156] Figure 8 (5j) is the result of using F3SNet to perform change detection on the 5th image pair in the SYSU-CD dataset;

[0157] Figure 8 (5k) is the result of using DSAMNet to perform change detection on the 5th image pair in the SYSU-CD dataset;

[0158] Figure 8 (5l) is the result of change detection on the fifth image pair in the SYSU-CD dataset using TrmNU-NetF. DETAILED DESCRIPTION

[0159] The present invention inputs two remote sensing images of different phases into a deep hierarchical Transformer network to extract abstract features, and then splices the features extracted by the four stages of the feature extraction backbone network in the deep hierarchical Transformer model and inputs them into the four stages of the nested U-Net network, and finally outputs the change detection results. In conjunction with the accompanying drawings and embodiments, a remote sensing image change detection method of the present invention is described in detail.

[0160] Method Example:

[0161] The remote sensing image change detection model for remote sensing image change detection in the present invention is based on a deep hierarchical Transformer network and a nested U-Net network. The deep hierarchical Transformer network and the nested U-Net network are first introduced below, and then the remote sensing image change detection model TrmNU-NetF of the present invention is introduced.

[0162] 1) Deep hierarchical Transformer network.

[0163] The quality of the backbone network used for feature extraction in tasks such as image classification, target detection, and semantic segmentation can greatly affect the performance of the entire model. To this end, researchers have designed a series of high-performance feature extraction backbone networks based on CNN for various computer vision tasks, such as VGG series models, ResNet series models, DenseNet series models, Hrnet series models, etc. The application of these models in change detection tasks has greatly improved the accuracy of change detection, but the CNN model faces the problem of limited receptive field, so it cannot make full use of global context information. In order to address this problem, the present invention uses Transformer as a feature extractor.

[0164] Transformer was first used in the field of natural language processing, and its core is the self-attention mechanism. This self-attention mechanism can model long-term dependency information well, so Transformer can be used to capture global information in the image. However, directly applying the original Transformer based on global attention to the image will result in high computational complexity. To this end, a deep hierarchical Transformer network shown in Figure 2(a) is designed as the feature extraction backbone network of the entire change detection model. The entire feature extraction backbone network can be divided into four stages, as shown in Stage 1, Stage 2, Stage 3, and Stage 4 in Figure 2(a). The first is the patch partition operation, which divides the entire image into non-overlapping patches of size 4×4. Since the image contains three bands, the dimension of the one-dimensional feature vector formed after each patch is flattened is 48. Then enter the first stage Stage 1, apply a linear transformation layer (Linear Embedding) to transform the dimension of the feature vector from 48 to C. These transformed feature vectors can be regarded as a feature vector sequence input into the Transformer Block module to extract more abstract features. The first stage contains two Transformer Block modules, and maintains the dimension (C) and number (H / 4×W / 4) of the feature vector. In order to extract hierarchical features, the second stage, Stage 2, is followed by the first stage, Stage 1. Stage 2 first uses a Patch Merging layer to aggregate features and reduce the number of feature vectors. Specifically, the adjacent 2×2 feature vectors with a dimension of C are merged into a feature vector with a dimension of 4C, so that the number of feature vectors is reduced to H / 8×W / 8. Then we apply a linear transformation layer to transform the dimension of the feature vector from 4C to 2C, and then input the feature vector sequence into the two Transformer Block modules of the second stage, Stage 2. The output of the second stage, Stage 2, is input into the third stage, Stage 3. The output of the second stage, Stage 2, is also passed through a Patch Merging layer and then input into the six Transformer Block modules of the third stage, Stage 3. Similarly, the output of Stage 3 is also passed through a PatchMerging layer before being input into the two Transformer Block modules of Stage 4. The four stages together produce a hierarchical feature representation, which mimics the hierarchical structure in CNN, so that the receptive field expands with the increase of network depth, thus being able to better take into account both global and local information.

[0165] The structure of the Transformer Block module in the above four stages is shown in Figure 2(b), which includes four attention units. The first attention unit includes an LN layer and a W-MSA layer connected in sequence, the second attention unit includes an LN layer and an MLP layer (multi-layer perceptron layer) connected in sequence, the third attention unit includes an LN layer and an SW-MSA layer connected in sequence, and the fourth attention unit includes an LN layer and an MLP layer connected in sequence. Among them, the W-MSA layer is used to perform Windows Multi-head Self-Attention operations, the SW-MSA layer is used to perform Shifted Windows Multi-head Self-Attention operations, the MLP layer is a fully connected layer used to enhance the nonlinearity of the model, and the LN layer is a LayerNorm layer, that is, a normalization operation is performed in the channel direction. Moreover, the connection method of these four attention units is: when k = 1, 2, 3, the output of the kth attention unit and the input of the kth attention unit are concatenated as the input of the k+1th attention unit; the output of the 4th attention unit and the input of the 4th attention unit are concatenated as the output of the entire Transformer Block module; the input of the 1st attention unit is the input of the entire Transformer Block module. That is, as shown in Figure 2(b), a Transformer Block module can be formally described as follows:

[0166]

[0167]

[0168]

[0169]

[0170] Wherein, W-MSA() indicates W-MSA operation; LN() indicates LN operation; SW-MSA() indicates SW-MSA operation; MPL indicates MLP operation; l-1 Represents the input of Transformer Block; z l+1 Represents the output of TransformerBlock.

[0171] The W-MSA layer and SW-MSA layer are introduced below. The original Transformer uses the global self-attention mechanism MSA as shown in Figure 3(a). The specific calculation process is shown in the following formula:

[0172]

[0173] Wherein, Q, K, V represent query matrix, key matrix and value matrix respectively; d represents the dimension of Q, K, V. That is to say, the feature vector corresponding to each image block and the feature vectors of all other image blocks are self-attention calculated. In order to extract richer features, Transformer also adopts a multi-head mechanism, that is, initialize multiple groups of Q, K, V parameters, and this embodiment uses 8 groups. The global attention mechanism obviously has a high computational complexity. In order to cope with the problem of high computational complexity of global self-attention, as shown in Figure 3(b), the self-attention calculation in each Transformer Block is limited to a local window, that is, the W-MSA mechanism is adopted, which can effectively reduce the complexity of the model, but in this way, the features between different windows cannot communicate information with each other, which reduces the Transformer's ability to model long-term dependencies. To this end, this embodiment further adopts the SW-MSA mechanism as shown in Figure 3(c), that is, the window calculated by the self-attention mechanism is translated to the lower right corner, so that information can be exchanged between different windows. Both the W-MSA mechanism and the SW-MSA mechanism are prior art and will not be repeated here.

[0174] 2) Nested U-Net network.

[0175] The nested U-Net network is as follows Figure 4 As shown in Figure 1, the biggest difference from the original U-Net network is that the encoder and decoder are densely connected. This dense connection operation is conducive to the comprehensive use of feature information of different scales in the model, thus improving the accuracy of change detection. Let x i,j For node X i,j The output feature map, i represents the index corresponding to the downsampling layer along the encoder direction, that is, the number of layers, and j represents the index along the skip connection direction, that is, the jth convolutional layer of the current layer. Then the feature map x output by any node i,j The calculation formula is as follows:

[0176]

[0177] In the formula, h(·) means that the feature is first convolved and then the Relu activation function is performed; Indicates an upsampling layer; [] represents the connection operation of the feature. The node with j=0 only receives one input from the previous layer of the encoder sub-network; the node with j=1 receives input from two consecutive layers of the encoder sub-network; the node with j>1 receives j+1 inputs, of which j inputs are the outputs of the previous j nodes on the current skip connection path, and the other input is the upsampled output of the next skip path.

[0178] 3) TrmNU-NetF model.

[0179] Remote sensing image change detection model TrmNU-NetF model is as follows Figure 1 As shown in the figure, it includes a feature extraction module and a nested U-Net network. The feature extraction module includes two deep hierarchical Transformer networks with the same structure and shared weights. The inputs of the two deep hierarchical Transformer networks in the feature extraction module are the two inputs of the remote sensing image change detection model. The node X in the nested U-Net network 0,4 The output of is the output of the remote sensing image change detection model, and the output of the first stage of the two deep hierarchical Transformer networks is concatenated and connected to the node X in the nested U-Net network. 0,0 , the output of the second stage is concatenated and connected to node X in the nested U-Net network 1,0 , the output of stage 3 is concatenated and connected to node X in the nested U-Net network 2,0 , the output of stage 4 is concatenated and connected to node X in the nested U-Net network 4,0 .

[0180] Based on the TrmNU-NetF model, a remote sensing image change detection method of the present invention can be implemented, which is described in detail below.

[0181] Step 1: Build Figure 1 The TrmNU-NetF model shown.

[0182] Step 2: Obtain a public remote sensing image change detection dataset and divide it into training data, verification data, and test data. Train, verify, and test the constructed TrmNU-NetF model to finally obtain a remote sensing image change detection model.

[0183] During the training process, the loss function used is a combination of the Focal loss function and the Dice loss function. This is because both Focal loss and Dice loss are suitable for situations where sample distribution is uneven, and can effectively improve the training stability and training effect of the model. The Focal loss function and Dice loss loss function are:

[0184] FL(p t )=-η t (1-p t ) ρ log(p t )

[0185] In the formula, p t represents the probability of the model predicting the category, η t represents the weight parameter that controls the contribution of positive and negative samples to the total loss, and ρ is the manually set focus parameter;

[0186]

[0187] Where s represents the Dice coefficient, which is used to measure the similarity between two samples; |X∩Y| is the intersection between two sets, and |X| and |Y| represent the union of two sets respectively.

[0188] A large number of studies and practices have shown that data augmentation operations help improve the training effect of the model. Therefore, in this embodiment, horizontal mirroring, vertical mirroring, random rotation, and random pixel erasure are randomly applied, where the number of randomly erased pixel regions is 4, and the maximum width and height of the randomly erased pixel region is 50.

[0189] Step three, obtain two remote sensing images of different phases, and call them two original remote sensing images, input the two original remote sensing images into the remote sensing image change detection model, and obtain the change detection results corresponding to the two original remote sensing images.

[0190] Step 4: Figure 5 As shown, two remote sensing images of different phases are enhanced to obtain two enhanced remote sensing images, both enhanced images are input into the remote sensing image change detection model, and the output results are subjected to the reverse operation corresponding to the enhancement process, thereby obtaining the change detection results corresponding to the two enhanced remote sensing images. Specifically, it includes:

[0191] Perform horizontal mirror enhancement processing on two original remote sensing images of different time phases, input them into the remote sensing image change detection model after processing, perform horizontal mirror processing on the output results, and obtain the change detection results corresponding to the horizontal mirror enhancement processing;

[0192] The original remote sensing images of two different phases are processed by vertical mirror enhancement, and then input into the remote sensing image change detection model. The output result is processed by vertical mirror enhancement to obtain the change detection result corresponding to the vertical mirror enhancement.

[0193] The original remote sensing images of two different phases are rotated 90 degrees for enhancement processing, and then input into the remote sensing image change detection model. The output result is rotated 270 degrees to obtain the change detection result corresponding to the 90-degree rotation enhancement processing;

[0194] The original remote sensing images of two different phases are rotated 270 degrees for enhancement processing, and then input into the remote sensing image change detection model. The output result is rotated 90 degrees to obtain the change detection result corresponding to the 270-degree rotation enhancement processing.

[0195] Step 5: According to step 3 and step 4, five change detection results can be obtained. The final change detection result is obtained by voting on the five change detection results at the same position, that is, if more than two of the five change detection results are predicted to be changes, then the position is determined to be a change.

[0196] The following experimental analysis is conducted to illustrate the effectiveness of the method of the present invention. The hardware environment of the experiment is: NVIDIA A100 graphics card, 40Gb video memory, 256Gb memory, and the software environment is: Ubuntu 18.04, using PyTorch to implement the relevant deep learning model.

[0197] Two large public remote sensing image change detection datasets, LEVIR-CD and SYSU-CD, are used for change detection experiments. The LEVIR-CD (LEarning, VIsion and Remote sensing) change detection dataset contains 637 images with a resolution of 0.5m, an image size of 1024×1024 pixels, and a total of 31,333 change instances. The number of training data, validation data, and test data is 445, 64, and 128, respectively, according to the official division method. Due to the limitation of video memory, all images are divided into image blocks of size 512×512 and do not overlap with each other, so that the number of image pairs used for training, validation, and testing is 1780, 256, and 512, respectively. The SYSU-CD (Sun Yat-Sen University, Change Detection) change detection dataset contains a total of 20,000 aerial remote sensing image pairs with a size of 256×256 pixels and an image resolution of 0.5 meters. According to the official division method, the number of training data, validation data, and test data are 12,000, 4,000, and 4,000 respectively.

[0198] In order to quantitatively evaluate the performance of different change detection models, F1 (F1-score), precision, Recall, Iou and overall classification accuracy (OA) are selected as evaluation indicators. The confusion matrix of the change detection results is shown in Table 1, and the calculation formula of the relevant evaluation indicators is as follows:

[0199] precision=TP / (TP+FP)

[0200] recall=TP / (TP+FN)

[0201] IoU = TP / (TP+FN+FP)

[0202] OA=(TP+TN) / (TP+TN+FN+FP)

[0203] F1=(2×recall×precision) / (recall+precision)

[0204] Table 1. Change detection confusion matrix

[0205]

[0206] The Adam optimizer is used to train the model, and the number of epochs is set to 50. The entire training is divided into two stages. The learning rate in the first stage (the first 30 epochs) is set to 0.0001, and the learning rate in the second stage (the last 20 epochs) is 0.00001. Since this embodiment uses a combination of Dice and Focal as the final loss function, the weight coefficients of Dice and Focal will directly affect the accuracy of change detection. In order to study the impact of the two loss functions on the final change detection results, the weight coefficient of the Dice loss is fixed to 1.0, and then the weight coefficients of the Focal loss are set to 0.5, 1.0, 2.0, and 4.0, respectively. The F1-score on the two change detection datasets LEVIR-CD and SYSU-CD is shown in Figure 2. Figure 6 Observation Figure 6 From the experimental results in , we can find that when the Focal loss weight is large (2, 4), the change detection accuracy will be reduced. When the Focal loss weight is 0.5, the detection accuracy of LEVIR-CD data is slightly improved, while the detection accuracy of SYSU-CD data is reduced. Therefore, the weights of both loss functions are set to 1.0.

[0207] In order to explore the influence of different data enhancement strategies and the model fusion method proposed in the present invention on the change detection accuracy, ablation experiments were conducted on two change detection datasets, and the experimental results are shown in Table 2. From the experimental results in Table 2, it is easy to find that the change detection accuracy obtained without data enhancement is the lowest; both horizontal + vertical mirroring and random rotation data enhancement strategies can slightly improve the change detection accuracy; while the random erasing enhancement strategy can greatly improve the change detection accuracy, which can be improved by more than 0.5% on both sets of data sets; introducing the model fusion method proposed in the present invention on the basis of data enhancement can further improve the change detection accuracy, which can be improved by 0.71% on LEVIR-CD and 0.32% on SYSU-CD, which illustrates the necessity of using data enhancement in the change detection model training process, and also proves the effectiveness of the model fusion method proposed in the present invention.

[0208] Table 2 Ablation experiment of F1-Score (%) of LEVIR-CD and SYSU-CD datasets

[0209]

[0210] In order to verify the effectiveness of the method of the present invention, the classic DeepLabV3, U-Net, PSPNet and UperNet were selected as comparison methods. At the same time, the latest MSPSNet, SESNet, F3SNet and DSAMNet were implemented as comparison methods according to the open source code. It should be pointed out that in order to ensure fairness, the training, confirmation and test data in the method of the present invention and the comparison method are exactly the same. The change detection results of the two data sets of LEVIR-CD and SYSU-CD are shown in Tables 3 and 4. In order to facilitate comparative analysis, the change detection results of the method of the present invention without model fusion (TrmNU-Net) are also given. On the LEVIR-CD data set, the F1, Recall, IoU and OA of the TrmNU-Net proposed in the present invention are better than other methods, and the Precision is close to the highest SESNet, which shows that the model proposed in the present invention can improve the accuracy of change detection. The TrmNU-NetF, which further adopts the single model fusion strategy, can achieve the highest accuracy in all five evaluation indicators, which shows the effectiveness of the single model fusion strategy proposed in the present invention. On the SYSU-CD dataset, the F1, Recall, IoU and OA of the proposed method TrmNU-NetF all achieved the highest recognition accuracy, while the Precision was slightly lower than that of UNet. In general, the TrmNU-NetF of the proposed method obtained the most balanced detection results, which fully demonstrated the effectiveness of the proposed method.

[0211] Table 3 Change detection accuracy of LEVIR-CD dataset (%)

[0212]

[0213]

[0214] Table 4 Change detection accuracy of SYSU-CD dataset (%)

[0215]

[0216] In order to observe the detection effects of different methods, 5 image pairs were randomly selected from each of the two datasets, and the change detection results were visualized. Figure 7(1a) to Figure 7(5l)As shown in Figure 7, the image pairs of the LEVIR-CD dataset are given, as shown in Figure 7(xa) and Figure 7(xb) (x represents the xth image pair, x = 1, 2, 3, 4, 5), and the corresponding ground references, as shown in Figure 7(xc). Figures 7(xd)-(xl) are the change detection results corresponding to DeepLabV3, U-Net, PSPNet, UperNet, MSPSNet, SESNet, F3SNet, DSAMNet and TrmNU-NetF, respectively. Each algorithm basically has different degrees of false detection and missed detection. Figure 7(2a) to Figure 7(2l) The comparison methods all have misdetected areas, while the detection results of the method of the present invention are more accurate; Figure 7 (5a) to Figure 7 (5l) The comparison method has a large number of missed detection areas, while the method of the present invention has a more complete detection. Figure 8(1a) to Figure 8(5l) As shown, compared with the comparison method, the method of the present invention has fewer false detection and missed detection areas, which further illustrates the effectiveness of the method of the present invention.

[0217] In summary, the method of the present invention has the following characteristics: 1) A hierarchical Transformer structure is used to better extract the features of remote sensing images. This hierarchical Transformer structure can take into account both local and global information, thereby improving the accuracy of change detection. 2) In order to make full use of features of different scales, the extracted features are spliced ​​and input into the nested U-Net to complete change detection. In the nested U-Net structure, there is information exchange between features of different scales, so this nested structure can make more full use of features of different scales to improve the accuracy of change detection. 3) A simple and effective single model fusion method is proposed, which specifically improves the final change detection accuracy by fusing the change detection results of the enhanced image pairs.

[0218] In the above embodiment, four enhancement processing methods are used, and combined with the original remote sensing image, five change detection results are finally obtained. Correspondingly, it is finally determined to be a change when more than two are predicted to be changes. As other implementations, two of the enhancement processing methods can be used. In this case, combined with the original remote sensing image, three change detection results are finally obtained. Correspondingly, it is finally determined to be a change when more than one is predicted to be a change. It should be noted that the specific number of enhancement processing methods used is not limited here, and can be less than four or more than four. Moreover, what specific results are determined to be changes also varies with the number of enhancement processing methods used.

Claims

1. A remote sensing image change detection method, characterized in that: The steps include: 1) Obtain two remote sensing images of different phases and call them two original remote sensing images; 2) Inputting the two original remote sensing images into the remote sensing image change detection model to obtain the change detection results corresponding to the two original remote sensing images; The remote sensing image change detection model is trained using remote sensing image pairs of different historical phases and corresponding change detection results as training sets; the remote sensing image change detection model includes a feature extraction module and a nested U-Net network, and the feature extraction module includes two deep hierarchical Transformer networks with the same structure and shared weights; The deep hierarchical Transformer network includes patch partition units connected in sequence and four stages, the four stages are stage 1, stage 2, stage 3 and stage 4 respectively; Node X in the nested U-Net network i,j Output feature map x i,j The calculation formula is: In the formula, i represents the number of layers, j represents the jth convolutional layer of the current layer, and h(·) represents the convolution operation on the feature first and then the ReLU activation function operation. represents an upsampling layer, [] represents the connection operation of features; The inputs of the two deep hierarchical Transformer networks in the feature extraction module are the two inputs of the remote sensing image change detection model, and the node X in the nested U-Net network 0,4 The output of is the output of the remote sensing image change detection model, and the output of the rth stage of the two deep hierarchical Transformer networks is concatenated and connected to the node X in the nested U-Net network. r-1,0 , r=1,2,3,4.

2. The remote sensing image change detection method according to claim 1, characterized in that: After step 2), steps 3) to 5) are also included: 3) Performing the same enhancement processing on two original remote sensing images of different phases to obtain two enhanced remote sensing images; 4) inputting both enhanced remote sensing images into the remote sensing image change detection model, and performing an inverse operation corresponding to the enhancement process on the output result, thereby obtaining change detection results corresponding to the two enhanced remote sensing images; 5) According to the change detection results corresponding to the two original remote sensing images and the change detection results corresponding to the two enhanced remote sensing images, the change of the two remote sensing images in different phases is determined.

3. The remote sensing image change detection method according to claim 2, characterized in that: In step 3), the enhancement processing includes at least one of horizontal mirroring, vertical mirroring, rotating 90 degrees and rotating 270 degrees. Then the inverse operations corresponding to the enhancement processing in step 4) are horizontal mirroring, vertical mirroring, rotating 270 degrees and rotating 90 degrees.

4. The remote sensing image change detection method according to claim 3, characterized in that: The enhancement processing is horizontal mirroring, vertical mirroring, rotation of 90 degrees and rotation of 270 degrees. Then in step 5), the change detection results corresponding to the two original remote sensing images and the change detection results corresponding to the two enhanced remote sensing images include a total of 5 change detection results; if more than 2 of the 5 change detection results at a certain position are predicted to be changes, the change detection result at that position is determined to be a change.

5. The remote sensing image change detection method according to claim 1, characterized in that: Stage 1, stage 2, stage 3, and stage 4 all include Transformer Block modules, and each Transformer Block module is represented as: In the formula, z l-1 Represents the input of the Transformer Block module, z l Represents the intermediate variable of the Transformer Block module, z l+1 Represents the output of the Transformer Block module; W-MSA() represents a local window self-attention calculation, which refers to dividing the input into multiple windows and performing self-attention calculation on the feature map in each window; SW-MSA() represents a moving window self-attention calculation, which refers to moving the window based on the local window self-attention, and performing self-attention calculation on the feature map in the moved window; LN() represents an LN operation, which refers to the LayerNorm layer that performs normalization operation in the channel direction; MPL() represents an MLP operation, which refers to a fully connected layer used to enhance the nonlinearity of the model.

6. The remote sensing image change detection method according to claim 1, characterized in that: When training the remote sensing image change detection model, the loss function used is a combination of the Focal loss function and the Dice loss function; The calculation formula of the Focal loss function is: FL(p t )=-η t (1-p t ) ρ log(p t ), where p t represents the probability of the model predicting the category, η t represents the weight parameter that controls the contribution of positive and negative samples to the total loss, and ρ is the manually set focus parameter; The calculation formula of the Dice loss function is: Where s represents the Dice coefficient, which is used to measure the similarity between two samples; |X∩Y| is the intersection between two sets, and |X| and |Y| represent the union of two sets respectively.

Images