Remote sensing image change detection method combining u-shaped network with self-attention mechanism

Abstract

This invention proposes a remote sensing image change detection method combining a U-shaped network and a self-attention mechanism, mainly addressing the problems of large memory consumption and complex network structure in existing technologies. The implementation steps are as follows: 1) Obtain training and testing sample sets; 2) Construct a U-shaped change detection network model, which includes a cascaded multi-scale feature extraction module, a hierarchical self-attention mechanism, a fusion upsampling module, and a classifier; 3) Input the training sample set into the U-shaped change detection network model and iteratively train it using backpropagation; 4) Input the testing sample set into the trained U-shaped change detection network model to obtain a visualization of the remote sensing image change detection and numerical results. This invention achieves lightweight change detection, obtains multi-scale fine-grained features, reduces network structure complexity, improves detection accuracy, and reduces computational resources. It can be used for geological disaster detection, urban planning, and deforestation detection.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method for detecting changes in remote sensing images, which can be used for geological disaster detection, urban planning, and deforestation detection. Background Technology

[0002] Change detection is the process of quantitatively analyzing surface changes between images from different time phases, and it has important significance in many fields.

[0003] Traditional change detection methods primarily utilize spectral information from remote sensing images. Change Vector Analysis (CVA) is a commonly used method, providing information on change intensity and direction. Similar methods include Principal Component Analysis (PCA) and Multivariate Change Detection (MAD). However, these methods often require optimal threshold selection during the decision-making phase, making them scene-dependent and time-consuming. Machine learning algorithms, on the other hand, can learn from partially labeled samples to derive automatic decision models and have therefore been widely applied to change detection in remote sensing images. These include Support Vector Machines (SVMs), Decision Trees, and Random Forests. However, these methods heavily rely on hand-crafted features, making it difficult to effectively capture high-level feature representations, resulting in lower change detection accuracy.

[0004] In recent years, deep learning (DL) has proven highly successful in various computer vision and remote sensing image interpretation applications. Deep learning-based models can capture fine image details and complex texture features. For example, Deep Belief Networks (DBNs), Stacked Autoencoders (SAEs), and Convolutional Neural Networks (CNNs) extract abstract depth features for comparison, generating difference images or transforming vectors of bi-temporal images, and then obtaining the final transformation map based on clustering or classification methods. However, since DBNs and SAEs are fully connected networks, their one-dimensional input format is limited, resulting in a significant loss of spatial information. While CNN architectures can extract local information and utilize weight-sharing mechanisms, demonstrating a remarkable ability to acquire rich local contextual features, purely CNN-based models are inherently limited by the size of their receptive field.

[0005] Self-attention (SA) is a mechanism that weights the input before outputting, assigning greater weight to areas the network wants to focus on and less weight to areas it doesn't want the network to pay attention to. It's used to solve sequence-to-sequence tasks and was first proposed in the field of Natural Language Processing (NLP). In recent years, due to its powerful representational capabilities, SA has been widely applied to various downstream tasks in computer vision and has attracted significant attention for overcoming the limited receptive field of CNNs. However, because SA's output is a uniform global representation at different stages, there is significant redundancy between the shallow and deep features extracted. To effectively encode local and global features, CNNs and SA can be combined into a unified network architecture. Existing methods mainly fall into two categories: one uses CNNs for feature extraction and then uses attention mechanisms to aggregate global contextual information; the other uses CNNs for feature extraction and then combines channel attention, spatial attention, and other methods to reweight the extracted features in the channel / spatial dimensions to model global information. Although both methods produce models with good detection performance, they consume a lot of computational memory, making them impractical for real-world applications.

[0006] Chen Hao et al. published STANet, a remote sensing image change detection method based on spatiotemporal attention, in RemoteSensing 12.10, 2020:1662. The paper designs two self-attention modules: a basic spatiotemporal attention module (BAM) and a pyramidal spatiotemporal attention module (PAM). BAM calculates the spatiotemporally independent attention weights between any two locations and computes the response at each location by weighted summing of features from all locations in the spatiotemporal space. PAM embeds BAM into a pyramidal structure to generate multi-scale attention representations. Although this method has achieved some success, the feature extraction module's feature representation capability is relatively poor, hindering further improvements in network performance.

[0007] Chen Hao et al. proposed a transformer-based method for remote sensing image change detection in IEEE, 2021, 60:1-14. The method first extracts features from two temporal images using a CNN; then, the feature pairs are concatenated along the channels and input into a Transformer encoder. After encoding, the features are split into two, and each is input into a Transformer decoder to generate its own feature map. Finally, the two feature maps are subtracted to find the changed parts, and then processed through several layers of convolutional neural networks to generate a change map. Although this method combines the advantages of CNN and self-attention mechanisms, and the model's performance is improved to some extent, the insufficient feature fusion in this method limits the performance of remote sensing image change detection.

[0008] Fang Sheng et al. published SNUNet, a method for change detection in ultra-high resolution remote sensing images using dense skip connections, in IEEE, 2021, 19:1-14. This method first inputs two-phase images into a parameter-shared Siamese network, extracts features from both phases through the Siamese network, and then concatenates the features from the two branches to ensure information integrity. To obtain discriminative high-resolution features and fine-grained location texture information, this method uses a dense skip connection mechanism between the encoder and decoder. Although SNUNet can suppress semantic gaps and localization errors to some extent, its high resource consumption hinders its application in practical projects. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of the prior art by proposing a remote sensing image change detection method that combines a U-shaped network with a self-attention mechanism. This method reduces the computational complexity of the network structure, fully integrates high-level semantic features, improves detection accuracy, and reduces computational resources.

[0010] To achieve the above objectives, the technical solution of the present invention includes the following steps:

[0011] (1) Obtain the pre-divided training sample set D from the remote sensing change detection database. train and test sample set D test The training set and the test set each contain a T1 phase image x1, a T2 phase image x2, and a label y;

[0012] (2) Constructing a U-shaped change detection network model:

[0013] (2a) Establish a multi-scale feature extraction module containing multiple convolutional layers with different dilation rates, a direct connection layer, a residual connection layer, and an activation function;

[0014] (2b) Establish a twin network consisting of N-layer multi-scale feature extraction modules and N-1-layer downsampling layers cascaded together;

[0015] (2c) Establish a hierarchical self-attention mechanism that includes four overlapping patch partitioning modules, two multilayer perceptron modules and two self-attention modules;

[0016] (2d) Establish a fusion upsampling module containing four transposed convolutional layers, four feature splicing layers, and four feature fusion layers;

[0017] (2e) Construct a classifier containing four 1×1 convolutional layers and four upsampling layers;

[0018] (2f) The Siamese network, hierarchical self-attention mechanism, fusion upsampling module and classifier are cascaded in sequence to form a U-shaped change detection network model;

[0019] (2g) Combining binary cross-entropy loss and dice loss as the loss function of this U-shaped change detection network model.

[0020]

[0021] (3) Based on the training set, the U-shaped change detection network model is iteratively trained using the backpropagation method to obtain the trained U-shaped change detection network model;

[0022] (4) Input the test set into the trained U-shaped change detection network model to obtain the change detection results of the remote sensing image.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] First, because the present invention incorporates a Siamese network in the U-shaped change detection network model, the receptive field can be effectively increased through multiple 3×3 convolutional layers with different hole ratios in the Siamese network, thereby capturing multi-scale fine-grained features. Compared with the prior art, this invention highlights the changing pixels of interest and obtains more discriminative feature information.

[0025] Second, because the present invention has a hierarchical self-attention mechanism in the U-shaped change detection network model, the computational load can be reduced through the hierarchical self-attention mechanism, and global context dependencies can be modeled. Compared with the prior art, it effectively avoids the huge computational cost and memory occupation caused by self-attention of high-resolution feature maps, while ensuring the modeling of global context information.

[0026] Third, because the present invention has a fusion upsampling module in the U-shaped change detection network model, the features can be efficiently fused through the fusion upsampling module, presenting the multi-scale remote details required for accurate change detection. Compared with the prior art, it effectively avoids the problem of large computational load caused by complex feature fusion.

[0027] Fourth, since the present invention has four classifiers in the U-shaped change detection network model, four change prediction maps at different levels can be obtained through the four parallel classifiers, and the loss value can be calculated by the loss function for each. Compared with the prior art, this accelerates the convergence of the change detection network model. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0029] Figure 2 This is a schematic diagram of the U-shaped change detection network model structure in this invention;

[0030] Figure 3 This is a schematic diagram of the multi-scale feature extraction module structure in this invention;

[0031] Figure 4 This is a schematic diagram of the hierarchical self-attention mechanism structure in this invention;

[0032] Figure 5 This is a schematic diagram of the fusion upsampling module structure in this invention;

[0033] Figure 6 This is a schematic diagram of the classifier structure in this invention;

[0034] Figure 7 This is a simulation result diagram of the change detection performed on the CDD dataset according to the present invention;

[0035] Figure 8 This is a simulation result diagram of the change detection performed on the LEVIR dataset according to the present invention;

[0036] Figure 9 This is a simulation result diagram of the change detection performed on the SYSU dataset according to the present invention. Detailed Implementation

[0037] The effects of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Reference Figure 1 The implementation steps for this example are as follows:

[0039] Step 1: Obtain the training sample set and the test sample set.

[0040] Obtain the pre-defined training sample set D from the remote sensing image change detection database. train and test sample set D test Each sample set contains a T1 phase image x1, a T2 phase image x2, and a label y;

[0041] This example extracts the corresponding training and test sets from the CDD dataset at a ratio of 10:3; from the LEVIR dataset at a ratio of 7:2; and from the SYSU dataset at a ratio of 3:1.

[0042] Step 2, construct a U-shaped change detection network model, such as Figure 2 As shown.

[0043] 2.1) Establish a multi-scale feature extraction module containing multiple convolutional layers with different dilation rates, a direct connection layer, a residual connection layer, and an activation function:

[0044] The aforementioned convolutional layers with different dilation rates include three parallel convolutional layers with a kernel size of 3×3 and dilation rates of 1, 3, and 5, respectively. The 3×3 convolutional layer with a dilation rate of 1 is used to replace the 1×1 convolutional layer to reduce channel features, and the two 3×3 convolutional layers with dilation rates of 3 and 5 are used to replace the 5×5 convolutional layer to reduce parameters. The outputs of these convolutional layers with different dilation rates are spliced ​​along the channels to achieve the fusion of different features.

[0045] The direct-connect layer contains a 1×1 convolutional layer;

[0046] The residual connection layer is used to perform residual connections between the output of the direct connection layer and the fused features;

[0047] The activation function mentioned is the ReLU activation function, which is expressed as:

[0048] This example uses three convolutional layers and one direct-connected layer. These four layers are then connected in parallel, followed by a residual connection layer and an activation function layer, forming a multi-scale feature extraction module. Figure 3 As shown;

[0049] 2.2) Establish a Siamese network consisting of N-layer multi-scale feature extraction modules and N-1-layer downsampling layers cascaded sequentially:

[0050] The N-layer multi-scale feature extraction modules have the same structure, N=5. The output image sizes of these five layers are 256×256, 128×128, 64×64, 32×32, and 16×16, respectively, with channel numbers of 16, 24, 32, 96, and 320, respectively, resulting in multi-level fine-grained feature pairs f. i 1 ,f i 2 i = 1, 2, 3, 4, 5;

[0051] The N-1 downsampling layers described above have the same structure, all employing max pooling downsampling. The kernel size of each downsampling layer is 2, and the stride is 2.

[0052] The twin network has two identical branch structures, and the specific structure of each branch is as follows: First multi-scale feature extraction module → First downsampling layer → Second multi-scale feature extraction module → Second downsampling layer → Third multi-scale feature extraction module → Third downsampling layer → Fourth multi-scale feature extraction module → Fourth downsampling layer → Fifth multi-scale feature extraction module.

[0053] 2.3) Establish a hierarchical self-attention mechanism comprising four overlapping patch partitioning modules, two multilayer perceptron modules, and two self-attention modules to achieve more accurate change detection:

[0054] The four overlapping patch partitioning modules have the same structure. Each overlapping patch partitioning module contains a convolutional layer, a flattening layer, a transpose layer, and a layer normalization layer connected in sequence. The convolutional layer parameters in the first overlapping patch partitioning module are set to K=7, S=4, P=3, while the convolutional layer parameters in the other three overlapping patch partitioning modules are set to K=3, S=2, P=1. K represents the kernel size, S represents the stride, and P represents the padding.

[0055] The two multilayer perceptron modules have identical structures. Each multilayer perceptron module includes a layer normalization layer with residual connections and a multilayer perceptron layer. The multilayer perceptron layer contains two fully connected layers, a depthwise separable convolutional layer, and a GELU activation function. The structural relationship is: first fully connected layer → depthwise separable convolutional layer.

[0056] →GELU activation function→Second fully connected layer, used to reduce computational cost;

[0057] The two self-attention modules have the same structure. Each self-attention module contains a multi-head attention layer and a feedforward layer connected in sequence. The multi-head attention layer contains a layer normalization layer and a self-attention layer connected in residual form. The feedforward layer contains a layer normalization layer and a multilayer perceptron layer connected in residual form, which is used to reduce computational consumption and model global context information.

[0058] The connection relationship of the above modules is as follows: First overlapping patch partitioning module → First multilayer perceptron module → Second overlapping patch partitioning module → Second multilayer perceptron module → Third overlapping patch partitioning module → First self-attention module → Fourth overlapping patch partitioning module → Second self-attention module, forming a self-attention mechanism, such as... Figure 4 As shown.

[0059] The implementation of aggregating global context information in each module of the above self-attention mechanism is as follows:

[0060] 2.3.1) For each multilayer perceptron module, feature extraction is performed to reduce computational load:

[0061] First, the output x of the overlapping patch partitioning module is taken as input, and the dimension is expanded by a ratio of n through the first fully connected layer, and the expanded feature x′ is output.

[0062] Secondly, depthwise separable convolution is used to learn positional information. That is, a ring of zeros is added around the feature map x′ and it is input into the depthwise separable convolutional layer, so that the convolutional layer can learn positional information based on the zeros and output a feature x″ with positional information.

[0063] Next, x″ is input into the activation function to activate the changing features and suppress the invariant features, thus obtaining the output feature x″′;

[0064] Finally, the dimension of feature x″′ is reduced to the initial dimension through a second fully connected layer, resulting in the updated feature.

[0065] 2.3.2) Each self-attention module, based on the principle of traditional self-attention mechanism, processes the feature map... Add a scaling factor R to the key K to reduce the number of channels:

[0066] First, the feature map dimension is transformed from N×C to N / R×C·R through a transformation operation to obtain the feature map.

[0067]

[0068] Where N represents the total number of patches, C represents the dimension of each patch, and Reshape(·) represents the deformation operation;

[0069] Secondly, the feature map of dimension N / R×C·R is processed through a fully connected layer. Reduced to N / R×C, the updated feature map K′ is obtained:

[0070]

[0071] Where Linear(·) represents a fully connected layer;

[0072] Finally, the above features are sequentially input into the multi-head attention layer and the feedforward layer to obtain the skip connection features S that model global information. j j = 1, 2, 3, 4;

[0073] 2.4) Establish a fusion upsampling module containing four transposed convolutional layers, four feature concatenation layers, and four feature fusion layers:

[0074] The four transposed convolutional layers have the same structure, with each layer having a kernel size of 4 and a stride of 2. They only upsample the image without changing the number of channels; that is, the number of input channels equals the number of output channels. The number of channels in the four layers is set to 320, 96, 32, and 24 respectively, resulting in the upsampled feature Up. j j = 1, 2, 3, 4;

[0075] The four feature concatenation layers have the same structure, and each feature concatenation layer upsamples the feature Up. j Skip connection features S obtained from hierarchical self-attention mechanism j j = 1, 2, 3, 4 are spliced ​​along the channel dimension;

[0076] The four feature fusion layers have the same structure. Each feature fusion layer consists of a flattening layer, a transpose layer, and a mapping layer connected sequentially. The first feature fusion layer maps the feature dimension from 416 to 96, the second feature fusion layer maps the feature dimension from 128 to 32, the third feature fusion layer maps the feature dimension from 56 to 24, and the fourth feature fusion layer maps the feature dimension from 40 to 16, resulting in four fused features x. j j = 1, 2, 3, 4;

[0077] The connection relationship of the above modules is as follows: First transposed convolutional layer → First feature concatenation layer → First feature fusion layer → Second transposed convolutional layer → Second feature concatenation layer → Second feature fusion layer → Third transposed convolutional layer → Third feature concatenation layer → Third feature fusion layer → Fourth transposed convolutional layer → Fourth feature concatenation layer → Fourth feature fusion layer, as shown. Figure 5 As shown;

[0078] 2.5) Construct a classifier containing four 1×1 convolutional layers and four upsampling layers:

[0079] The four 1×1 convolutional layers described above have identical structures and are used to fuse the corresponding features x. j The number of channels is changed to 1, resulting in four mask feature maps. j j = 1, 2, 3, 4;

[0080] The four upsampling layers have the same structure, and each upsampling layer uses bilinear interpolation to mask the four decoded feature maps. j Upsampling to 256×256 yielded four prediction maps at different levels, p. j j = 1, 2, 3, 4, used to calculate the loss value between the image label and the network, accelerating the network convergence process;

[0081] The four convolutional layers and four upsampling layers mentioned above are connected to each other to form four parallel classifiers, such as... Figure 6 As shown;

[0082] 2.6) The Siamese network, hierarchical self-attention mechanism, fusion upsampling module and classifier are cascaded in sequence to form a U-shaped change detection network model;

[0083] 2.7) Combine the binary cross-entropy loss and the dice loss as the loss function of this U-shaped change detection network model.

[0084]

[0085] The loss function Represented as:

[0086]

[0087] in, Cross-entropy loss;

[0088] Loss due to dice;

[0089] p j Let represent the j-th predicted change map, where j = 1, 2, 3, 4, g represents the corresponding label, and · represents the L1 paradigm.

[0090] Step 3: Iteratively train the U-shaped change detection network model using the backpropagation method.

[0091] 3.1) Initialize the parameters θ of the U-shaped change detection network model, and initialize the detection accuracy dictionary C, which includes precision P, recall R, F1 score F1, intersection-over-union ratio (IoU), and accuracy OA. That is, set C[P] = 0, C[R] = 0, C[F1] = 0, C[IoU] = 0, and C[OA] = 0. Similarly, initialize the optimal accuracy dictionary B, that is, set B[P] = 0, B[R] = 0, B[F1] = 0, B[IoU] = 0, and B[OA] = 0. Set the current iteration number e to 0 and the maximum iteration number max_e to 200.

[0092] 3.2) The training sample set D train The changes are predicted by forward propagation using the input to the network model. Combine it with the label image y e Through the loss function Calculate the loss value of the current network model.

[0093] 3.3) Regarding the loss value The backpropagation method is used to calculate the parameter gradient of the network model, and the gradient descent algorithm is used to update the parameter θ of the network model.

[0094] 3.4) Based on the change prediction map and label image y e By analyzing the number of changing pixels and the number of unchanged pixels at the same location, four categories of labeling results are obtained:

[0095] The first type is if the pixel is in the predicted image. The middle represents the changing pixels, and in the label image y e If the pixels in the middle are variable pixels, then the number of pixels of this type is denoted as TP;

[0096] The second category is if the pixel is in the predicted image. The middle represents the changing pixels, and in the label image y e If the pixels in the middle are constant, then the number of pixels of this type is denoted as FP;

[0097] The third category is if the pixel is in the predicted image. The middle pixel is an invariant pixel, and it is in the label image y. e If the pixels in the middle are constant, then the number of pixels of this type is denoted as TN;

[0098] The fourth category is if the pixel is in the predicted image. The middle pixel is an invariant pixel, and it is in the label image y. e If the pixels in the middle are variable pixels, then the number of pixels of this type is denoted as FN;

[0099] Then, based on TP, FP, TN, and FN, calculate the precision P, recall R, F1 score F1, intersection-over-union ratio (IoU), and accuracy OA for the current iteration number.

[0100]

[0101]

[0102] The detection accuracy dictionary C is composed of P, R, F1, IoU, and OA as described above;

[0103] 3.5) Compare the F1 values ​​in dictionary C with the F1 values ​​in the optimal precision dictionary B, and determine whether to update the optimal precision dictionary B:

[0104] If C[F1] > B[F1], then let B[F1] = C[F1], B[P] = C[P], B[R] = C[R], B[IoU] = C[IoU], B[OA] = C[OA];

[0105] Otherwise, proceed to step 3.6);

[0106] 3.6) Save the network model every 10 generations of training and check if (e+1)%10=0 is true:

[0107] If so, save the current network model in the specified path;

[0108] Otherwise, proceed to step 3.7);

[0109] 3.7) Determine if e = max_e - 1 is true:

[0110] If so, end the training, obtain the trained network model, and proceed to step 4;

[0111] Otherwise, let e = e + 1 and return to step 3.2).

[0112] Step 4: Obtain the change detection results of the remote sensing image.

[0113] 4.1) Test sample set D test Each pair of biphase images k = 1, 2, 3, ... is used as input to the U-shaped change detection network model for forward propagation to obtain the change prediction map.

[0114] 4.2) Visualize the change detection results, displaying the T1 time phase image. T2 phase image The corresponding label y k and change prediction chart After splicing along the channels, save the data to the specified path;

[0115] 4.3) Based on the corresponding label y k With change prediction map Calculate precision P, recall R, F1 score F1, intersection-over-union ratio (IoU), and accuracy OA to obtain the detection precision dictionary C′ on the test set, and print it out.

[0116] The technical effects of this invention can be further illustrated by the following simulation experiments.

[0117] I. Simulation Conditions

[0118] The simulation environment of this invention uses the Python 3.8 + PyTorch 1.9 framework and is implemented on a workstation with a GeForce RTX 2080 Ti and 11GB of memory.

[0119] The simulation used three datasets: the CDD dataset, the LEVIR dataset, and the SYSU dataset.

[0120] The CDD dataset is a set of remote sensing images acquired on Google Earth (Digital Earth), showing seasonal variations in the same area. It includes seven pairs of images (4725×2700 pixels) and four pairs of images (1900×1000 pixels). The spatial resolution of the acquired images ranges from 3 to 100 cm / pixel. The original dataset contains 16,000 pairs of images (256×256 pixels), with 10,000 pairs used for the training set, 3,000 pairs for the validation set, and 3,000 pairs for the test set.

[0121] The LEVIR dataset contains 637 pairs of 1024×1024 pixel images from Google Earth with a spatial resolution of 0.5 m / pixel. These images were collected from 20 different regions in Texas, USA, and contain many pseudo-variations caused by seasonality and lighting. Considering the limitations of GPU memory during the experiments, the images were cut into small, non-overlapping 256×256 patches, resulting in 7120 / 1024 / 2048 patch pairs for training / validation / testing, respectively.

[0122] The SYSU dataset is a challenging dataset for general change detection. It contains 20,000 pairs of labeled, two-time image patches, each 256×256 in size. There are 12,000 / 4,000 / 4,000 pairs of instances and corresponding labels for training / validation / testing.

[0123] II. Simulation Content

[0124] Simulation 1: A visualization change detection simulation was performed on the CDD dataset using this invention. The visualization results are as follows: Figure 7 As shown, the four rows represent the T1 phase image x1, the T2 phase image x2, the image label y, and the predicted image, respectively.

[0125] from Figure 7 As can be seen, the detection results of this invention are basically consistent with the labels, indicating that this invention, by combining a U-shaped network and a self-attention mechanism, can efficiently learn the changing features.

[0126] Simulation 2: The present invention and four existing methods, STANet, BIT, SNUNet and ISNet, were used to perform numerical change detection simulation on the CDD dataset. The numerical results are shown in Table 1.

[0127] Table 1 Results of CDD Dataset

[0128] CDD P(％) R(％) F1(％) IOU (%) OA (%) STANet 88.97 94.31 91.56 84.44 97.95 BIT 95.86 94.59 95.22 90.88 98.88 SNUNet 95.32 95.02 95.17 90.74 98.84 ISNet 95.18 94.43 94.8 90.12 98.78 This invention 96.49 96.51 96.5 93.24 99.1

[0129] As can be seen from Table 1, the present invention significantly outperforms other comparative models in all indicators. In terms of F1 score and Intersection over Union (IOU), the present invention is 1.28% and 2.36% higher than the existing best method, respectively, demonstrating the superiority of the present invention.

[0130] Simulation 3: A visualization change detection simulation was performed on the LEVIR dataset using this invention. The visualization results are as follows: Figure 8 As shown, the four rows represent the T1 phase image x1, the T2 phase image x2, and the predicted image, respectively. And image label y.

[0131] from Figure 8 As can be seen, the present invention can accurately detect the edge and texture information of changing targets, proving the effectiveness of multi-scale feature extraction.

[0132] Simulation 4: The present invention and four existing methods, STANet, BIT, SNUNet and ISNet, were used to perform numerical change detection simulation on the LEVIR dataset. The numerical results are shown in Table 2.

[0133] Table 2 Results of the LEVIR dataset

[0134] LEVIR P(％) R(％) F1(％) IOU (%) OA (%) STANet 80.99 91.21 85.79 75.12 98.46 BIT 91.95 88.57 90.23 82.19 99.02 SNUNet 91.66 88.48 90.04 81.89 99 ISNet 92.46 88.27 90.32 82.35 99.04 This invention 89.47 91.95 90.69 82.97 99.06

[0135] As can be seen from Table 2, the present invention outperforms the existing best method by 0.37% and 0.62% in F1 score and Intersection over Union (IOU), respectively, indicating that the present invention can obtain more granular change information and has fewer false alarms (FP).

[0136] Simulation 5: A visualization change detection simulation was performed on the SYSU dataset using this invention. The visualization results are as follows: Figure 9 As shown, the four rows represent the T1 phase image x1, the T2 phase image x2, and the predicted image, respectively. And image label y.

[0137] from Figure 9 As can be seen, the detection results of the present invention are basically consistent with the labels, indicating that the present invention can learn the semantic information of the changing target well.

[0138] Simulation 6: Numerical change detection simulation was performed on the SYSU dataset using the present invention and four existing methods: STANet, BIT, SNUNet, and ISNet. The numerical results are shown in Table 3.

[0139] Table 3 Results of the SYSU Dataset

[0140] SYSU P(％) R(％) F1(％) IOU (%) OA (%) STANet 81.08 74.3 77.54 63.32 \ BIT 82.32 76.68 79.4 65.84 \ SNUNet 80.3 79.79 80.04 66.73 \ ISNet 80.27 76.41 78.29 64.44 90.01 This invention 80.56 81.89 81.22 68.37 91.07

[0141] As can be seen from Table 3, the present invention outperforms the existing best method by 1.18% and 1.64% in F1 score and Intersection over Union (IOU), respectively, demonstrating that the present invention can achieve good performance in change detection in complex scenes.

[0142] The sources of the four prior art technologies mentioned above:

[0143] STANet is a method for remote sensing image change detection published by Chen Hao et al. in RemoteSensing, namely: Chen, Hao, and Zhenwei Shi. "Aspatial-temporalattention-based method and a new dataset for remote sensing image change detection." RemoteSensing 12.10 (2020): 1662;

[0144] BIT is a method for detecting changes in remote sensing images published by Chen Hao et al. in IEEE TGRS, namely: Chen, Hao, Zipeng Qi, and Zhenwei Shi. "Remote sensing image change detection with transformers." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-14;

[0145] SNUNet is a method for change detection in remote sensing images published by Fang Sheng et al. in IEEE, namely: Fang, Sheng, et al. "SNUNet-CD: Adensely connected Siamese network for change detection of VHR images." IEEE Geoscience and Remote Sensing Letters 19 (2021): 1-5;

[0146] ISNet is a method for detecting changes in remote sensing images published by Cheng et al. in IEEE, namely: Cheng, Gong, Guangxing Wang, and Junwei Han. "Isnet: Towards improving separation for remote sensing image change detection." IEEE Transactions on Geoscience and Remote Sensing 60 (2022): 1-11.

Claims

1. A method for detecting changes in remote sensing images by combining a U-shaped network with a self-attention mechanism, characterized in that, Includes the following steps: (1) Obtain the pre-divided training sample set D from the remote sensing change detection database. train and test sample set D test The training set and the test set each contain a T1 phase image x1, a T2 phase image x2, and a label y; (2) Constructing a U-shaped change detection network model: (2a) Establish a multi-scale feature extraction module containing multiple convolutional layers with different dilation rates, a direct connection layer, a residual connection layer, and an activation function; (2b) Construct a Siamese network consisting of N layers of multi-scale feature extraction modules and N-1 layers of downsampling layers, as shown in the following structure: The N-layer multi-scale feature extraction modules have the same structure, N=5. The five layers of image sizes are 256×256, 128×128, 64×64, 32×32, and 16×16, respectively, and the number of channels are 16, 24, 32, 96, and 320, respectively, resulting in multi-level fine-grained feature pairs f. i 1 ,f i 2 (i = 1, 2, 3, 4, 5), further calculate the absolute difference between feature pairs to obtain the difference feature d. i ; The N-1 downsampling layers described above have the same structure, all employing max pooling downsampling. The kernel size of each downsampling layer is 2, and the stride is 2. The twin network has two identical branch structures, and the specific structure of each branch is as follows: First multi-scale feature extraction module → First downsampling layer → Second multi-scale feature extraction module → Second downsampling layer → Third multi-scale feature extraction module → Third downsampling layer → Fourth multi-scale feature extraction module → Fourth downsampling layer → Fifth multi-scale feature extraction module. (2c) Establish a hierarchical self-attention mechanism that includes four overlapping patch partitioning modules, two multilayer perceptron modules and two self-attention modules; (2d) Establish a fusion upsampling module containing four transposed convolutional layers, four feature splicing layers, and four feature fusion layers; (2e) Construct a classifier containing four 1×1 convolutional layers and four upsampling layers; (2f) The Siamese network, hierarchical self-attention mechanism, fusion upsampling module and classifier are cascaded in sequence to form a U-shaped change detection network model; (2g) Combining binary cross-entropy loss and dice loss as the loss function of this U-shaped change detection network model. (3) Based on the training set, the U-shaped change detection network model is iteratively trained using the backpropagation method to obtain the trained U-shaped change detection network model; (4) Input the test set into the trained U-shaped change detection network model to obtain the change detection results of the remote sensing image.

2. The method according to claim 1, characterized in that, The parameters and activation functions of each layer in the multi-scale feature extraction module established in step (2a) are as follows: The multiple convolutional layers with different dilation rates include three parallel convolutional layers with a kernel size of 3×3 and dilation rates of 1, 3, and 5, respectively. The 3×3 convolutional layer with a dilation rate of 1 is used to replace the 1×1 convolutional layer to reduce channel features, and the two 3×3 convolutional layers with dilation rates of 3 and 5 are used to replace the 5×5 convolutional layer to reduce parameters. The outputs of these convolutional layers with different dilation rates are spliced ​​along the channels to achieve the fusion of different features. The direct-connect layer contains a 1×1 convolutional layer; The residual connection layer, as described above, performs a residual connection between the output of the direct connection layer and the fused features; The activation function mentioned is the ReLU activation function.

3. The method according to claim 1, characterized in that, The hierarchical self-attention mechanism established in step (2c) consists of four overlapping patch partitioning modules, two multilayer perceptron modules, and two self-attention modules, with the following structure: The four overlapping patch partitioning modules have the same structure. Each overlapping patch partitioning module contains a convolutional layer, a flattening layer, a transpose layer, and a layer normalization layer connected in sequence. The convolutional layer parameters in the first overlapping patch partitioning module are set to K=7, S=4, P=3, and the convolutional layer parameters in the other three overlapping patch partitioning modules are set to K=3, S=2, P=1, where K represents the kernel size, S represents the stride, and P represents the padding. The two multilayer perceptron modules have the same structure. Each multilayer perceptron module includes a layer normalization layer with residual connections and a multilayer perceptron layer. The multilayer perceptron layer includes two fully connected layers, a depthwise separable convolutional layer and a GELU activation function. The structural relationship is: first fully connected layer → depthwise separable convolutional layer → GELU activation function → second fully connected layer, which is used to reduce the amount of computation. The two self-attention modules have the same structure. Each self-attention module contains a multi-head attention layer and a feedforward layer connected in sequence. The multi-head attention layer contains a layer normalization layer and a self-attention layer connected in residual form. The feedforward layer contains a layer normalization layer and a multilayer perceptron layer connected in residual form, which is used to reduce computational consumption and model global context information. The connection relationship of the above modules is as follows: First overlapping patch partitioning module → First multilayer perceptron module → Second overlapping patch partitioning module → Second multilayer perceptron module → Third overlapping patch partitioning module → First self-attention module → Fourth overlapping patch partitioning module → Second self-attention module.

4. The method according to claim 3, characterized in that, Each of the aforementioned multilayer perceptron modules performs feature extraction with reduced computational load, as follows: First, the output x of the overlapping patch partitioning module is taken as input, and the dimension is expanded by a ratio of n through the first fully connected layer, and the expanded feature x′ is output. Secondly, depthwise separable convolution is used to learn positional information. That is, a ring of zeros is added around the feature map x′ and it is input into the depthwise separable convolutional layer, so that the convolutional layer can learn positional information based on the zeros and output a feature x″ with positional information. Next, x″ is input into the activation function to activate the changing features and suppress the invariant features, thus obtaining the output feature x″′; Finally, the dimension of feature x″′ is reduced to the initial dimension size through a second fully connected layer, resulting in the updated feature.

5. The method according to claim 3, characterized in that, Each self-attention module performs global context information modeling to reduce computational cost. This is based on the principle of traditional self-attention mechanisms and modifies the feature map. To reduce the number of channels, a scaling factor R is added to the key K, as follows: First, the feature map dimension is transformed from N×C to N / R×C·R through a transformation operation to obtain the feature map. Where N represents the total number of patches, C represents the dimension of each patch, and Reshape(·) represents the deformation operation; Secondly, the feature map of dimension N / R×C·R is processed through a fully connected layer. Reduced to N / R×C, the updated feature map K′ is obtained: Where Linear(·) represents a fully connected layer; Finally, the above features are sequentially input into the multi-head attention layer and the feedforward layer to obtain the skip connection features S that model global information. j j = 1, 2, 3, 4.

6. The method according to claim 1, characterized in that, The structure and parameters of each layer in the fusion upsampling module established in step (2d) are as follows: The four transposed convolutional layers have the same structure, with each layer having a kernel size of 4 and a stride of 2. They only upsample the image without changing the number of channels; that is, the number of input channels equals the number of output channels. The number of channels in the four layers is set to 320, 96, 32, and 24 respectively, resulting in the upsampled feature Up. j j = 1, 2, 3, 4; The four feature concatenation layers have the same structure, and each feature concatenation layer upsamples the feature Up. j With skip connection feature S j Stitch along the channel dimension; The four feature fusion layers have the same structure. Each feature fusion layer consists of a flattening layer, a transpose layer, and a mapping layer connected sequentially. The first feature fusion layer maps the feature dimension from 416 to 96, the second feature fusion layer maps the feature dimension from 128 to 32, the third feature fusion layer maps the feature dimension from 56 to 24, and the fourth feature fusion layer maps the feature dimension from 40 to 16, resulting in four fused features x. j j = 1, 2, 3, 4; The connection relationship of the above modules is as follows: First transposed convolutional layer → First feature splicing layer → First feature fusion layer → Second transposed convolutional layer → Second feature splicing layer → Second feature fusion layer → Third transposed convolutional layer → Third feature splicing layer → Third feature fusion layer → Fourth transposed convolutional layer → Fourth feature splicing layer → Fourth feature fusion layer.

7. The method according to claim 1, characterized in that, The structure and parameters of each layer in the classifier established in step (2e) are as follows: The four 1×1 convolutional layers described above have identical structures and are used to fuse the corresponding features x. j The number of channels is changed to 1, resulting in four mask feature maps. j j = 1, 2, 3, 4; The four upsampling layers have the same structure, and each upsampling layer uses bilinear interpolation to mask the four decoded feature maps. j Upsampling to 256×256 yielded four prediction maps at different levels, p. j j = 1, 2, 3, 4, used to calculate the loss value between the image label and the network, accelerating the network convergence process; The four convolutional layers and four upsampling layers are connected to each other to form four parallel classifiers.

8. The method according to claim 1, characterized in that, The loss function L(p,g) in step (2g) is expressed as follows: in, Cross-entropy loss; Loss due to dice; p j Let represent the j-th predicted change map, j = 1, 2, 3, 4, g represent the corresponding label, and ||·|| represent the L1 paradigm.

9. The method according to claim 1, characterized in that, In step (3), the U-shaped change detection network model is iteratively trained using the backpropagation method, as follows: (3a) Initialize the parameters θ of the U-shaped change detection network model, and initialize the detection accuracy dictionary C, which includes precision P, recall R, F1 score F1, intersection-over-union ratio (IoU), and accuracy OA. That is, set C[P] = 0, C[R] = 0, C[F1] = 0, C[IoU] = 0, and C[OA] = 0. Similarly, initialize the optimal accuracy dictionary B, that is, set B[P] = 0, B[R] = 0, B[F1] = 0, B[IoU] = 0, and B[OA] = 0. Set the current iteration number e to 0 and the maximum iteration number max_e to 200. (3b) The training sample set D train The changes are predicted by forward propagation using the input to the network model. Combine it with the label image y e Through the loss function Calculate the loss value of the current network model. (3c) Regarding the loss value The backpropagation method is used to calculate the parameter gradient of the network model, and the gradient descent algorithm is used to update the parameter θ of the network model. (3d) Based on the change prediction map and label image y e The number of changing pixels and the number of unchanged pixels at the same position are used to obtain four types of labeling results: TP, FP, TN, and FN. Then, based on TP, FP, TN, and FN, the precision P, recall R, F1 score F1, intersection-over-union ratio (IoU), and accuracy OA of the current iteration number are calculated. The detection accuracy dictionary C is composed of P, R, F1, IoU, and OA. (3e) Compare the F1 values ​​in dictionary C with the F1 values ​​in the optimal precision dictionary B to determine whether to update the optimal precision dictionary B: If C[F1]>B[F1], then let B[F1]=C[F1], B[P]=C[P], B[R]=C[R], B[IoU]=C[IoU], B[OA]=C[OA]; Otherwise, execute (3f); (3f) Save the network model every 10 generations of training and determine whether (e+1)%10=0 is true: If so, save the current network model in the specified path; Otherwise, execute (3g); (3g) Determine whether e = max_e-1 is true: If so, then end the training and obtain the trained network model; Otherwise, let e = e + 1 and return to step (3b).

Images