A remote sensing image change detection method based on local-global Transformer network

Abstract

The present invention discloses a remote sensing image change detection method based on a local-global Transformer network, which includes the following steps: preprocessing multiple groups of image sample data to obtain multiple groups of image data; inputting each group of standardized image data into a local-global Transformer network; performing a differential operation on the output of the front-phase remote sensing image and the output of the back-phase remote sensing image in each stage, and then inputting the differential operation into a high-frequency enhancement unit to obtain high-frequency features of the edges of the change areas in each stage; inputting the high-frequency features of the second stage, the third stage and the fourth stage into a multi-scale fusion attention unit to obtain fine-grained fusion features of each stage; inputting the high-frequency features of the edges of the change areas in the first stage and the fine-grained fusion features obtained in the following three stages into a deep feature guidance unit; in order to solve the serious problem of false detection and missed detection of feature boundaries, the present invention proposes a plug-and-play high-frequency enhancement unit to replace the inflexible U-shaped structure to optimize the detection boundary.

Description

Technical Field

[0001] The present invention belongs to the technical field of change detection networks, and in particular to a remote sensing image change detection method based on a local-global Transformer network. Background Art

[0002] The evolution of the natural environment and human production activities leads to continuous changes in the Earth's surface. Therefore, regular monitoring and timely detection of land cover changes are crucial for the harmonious coexistence of humans and nature. Remote sensing image change detection research aims to use multi-temporal remote sensing images and geospatial data captured over the same surface area at different times to identify and analyze changes in land features, including changes in their extent and state. This technology is an important approach and means for achieving surface observation. Currently, this technology has been widely used in urban planning, land use monitoring, agriculture and forestry monitoring, natural disaster monitoring, and many other fields.

[0003] Existing deep learning-based change detection methods for remote sensing images mostly rely on convolutional networks. Due to the inherent properties of convolutional kernels, convolutional networks inherently lack the ability to model long-range dependencies, which may limit their further development in the field of change detection.

[0004] In recent years, some work has begun to utilize non-local relationship modeling to effectively extract global relationships between pixels. Compared to convolutional network-based change detection methods, this method can fully exploit the connections between arbitrary pixels, making the limited receptive field no longer a key factor affecting model performance. However, most existing non-local modeling methods struggle to capture the information flow between multi-scale features. They typically cascade a non-local feature extraction module at the end of the network, but ignore a more critical issue in the field of change detection: the multi-scale geospatial problem.

[0005] Furthermore, the effectiveness of detecting the boundaries of changing regions directly impacts the final detection accuracy. In real-world aerial remote sensing images, the specific boundaries of some changing regions can be difficult to define due to spectral variations or shadows. Existing change detection algorithms typically employ a U-shaped structure to capture detailed information about change boundaries layer by layer. However, this approach inevitably sacrifices flexibility in model design. Summary of the Invention

[0006] Purpose of the invention: The purpose of the present invention is to overcome the shortcomings of existing methods and provide a remote sensing image change detection method based on a local-global Transformer network to solve the problem of low accuracy of remote sensing image change detection.

[0007] Technical solution: The present invention discloses a remote sensing image change detection method based on a local-global Transformer network, which includes the following steps:

[0008] S1 preprocesses multiple sets of image sample data to obtain multiple sets of standardized image data, wherein each set of image sample data includes a dual-temporal remote sensing image, and the dual-temporal remote sensing image includes a forward-temporal remote sensing image and a backward-temporal remote sensing image;

[0009] S2 inputs each set of standardized image data into the local-global Transformer network in sequence, wherein the local-global Transformer network includes image block embedding and a backbone network, wherein the backbone network includes four stages, namely the first stage, the second stage, the third stage and the fourth stage, wherein the forward-temporal remote sensing image and the backward-temporal remote sensing image after the image block embedding are input into the first stage, and the output of the previous stage is used as the input of the next stage, and each stage includes two Siamese Transformer models, wherein the first Siamese Transformer model is used to limit the calculation of self-attention to a local window to model the local pixel relationship of the input image, and the second Siamese Transformer model is used to perform attention calculation on the entire pixel of the image to model the global pixel relationship of the input image, and there is an image block merging operation between each two stages, i.e., the size of the feature map is halved and the number of channels is doubled;

[0010] S3 performs a differential operation on the output of the remote sensing image in the front time phase and the output of the remote sensing image in the back time phase in each stage, and inputs the result into the high-frequency enhancement unit to obtain the high-frequency features of the edge of the change area in each stage;

[0011] S4 inputs the high-frequency features of the second, third and fourth stages into the multi-scale fusion attention unit to obtain the fine-grained fusion features of each stage;

[0012] S5 inputs the high-frequency features of the edge of the changed area in the first stage, the second stage, the third stage, and the fine-grained fusion features obtained in the third stage into the deep feature guidance unit to obtain a more refined detection map;

[0013] S6 performs feature fusion on the output of the deep feature guidance unit and the second stage, the third stage, and the fine-grained fusion features obtained in the third stage to obtain the model output features;

[0014] S7 uses the training data to train the above model and then conducts data testing to obtain the final prediction result.

[0015] Further, including:

[0016] The first stage includes two Siamese Transformer models, and both Siamese Transformer models are used to limit the calculation of self-attention to a local window to model the local pixel relationship of the input image.

[0017] Further, including:

[0018] The image block embedding specifically includes a two-dimensional convolution with a convolution kernel of 4×4 and a stride of 4, i.e., a bi-temporal image I 1 , D, H, and W represent the number of channels, height, and width, respectively, and are converted into image token sequences through image patch embedding operations.

[0019] Further, including:

[0020] The Siamese Transformer model includes two standard Transformer blocks with shared weights, which are divided into a token mixer and a channel mixer according to different regional functions. The token mixer is used to capture the spatial feature representation of the bi-temporal image. First, the feature pairs of the bi-temporal image are layer-normalized, and then three input features Q, K, and V are obtained through linear transformation. The three input features are input into the self-attention calculation. Finally, the output features of the token mixer are obtained by adding the output of the self-attention calculation and the skip connection of the features of the bi-temporal image to the output of the self-attention calculation to obtain the corresponding output features.

[0021] The channel mixer is used to fuse features in the channel dimension. First, the output features of the token mixer are normalized, and then enter the first multi-layer perceptron for depth convolution. The results of the depth convolution and the results of the linear transformation are activated and input into the second multi-layer perceptron. The output features of the second multi-layer perceptron are added to the output features of the token mixer to obtain the output features Y of the channel mixer. Both the first multi-layer perceptron and the second multi-layer perceptron are linear transformations.

[0022] Further, including:

[0023] The token mixer uses the self-attention mechanism to fully capture the global spatial feature relationship of the image. Its mathematical expression is shown in formulas (1)-(2):

[0024]

[0025] MHSA k (Q,K,V)=Φ(Concat(SA1,SA2,...,SA k )) (2)

[0026] Among them, SA i (·) and MGSA k (·) denotes the output features of the i-th self-attention head and the output features of the multi-head self-attention with k attention heads, Q(Q i )、K(K i ) and V(V i) represent the three input features of query, key and value respectively, c represents the feature dimension of each attention head, and Φ represents the linear transformation; the channel mixer uses a multi-layer perceptron to effectively fuse the features on the channel dimension, and its mathematical expression is shown in formulas (3)-(5):

[0027] X′ m =Φ1(norm(X m )) (3)

[0028] pos=T2(DW(T1(X′ m ))) (4)

[0029] Y=Φ2(σ(X′ m +pos))+X m (5)

[0030] Among them, X′ m , pos and Y represent the transition features, i.e., the output features of the first multi-layer perceptron, the conditional position encoding features, and the output features of the channel mixer, respectively. m represents the input feature of the channel mixer, m∈{1,2}, Φ represents the linear transformation, where T represents the reshape operation, where T1∈(1D→2D), T2∈(2D→1D), DW represents the depthwise convolution, and σ represents the GELU activation.

[0031] Further, including:

[0032] The high frequency enhancement unit comprises:

[0033] The input feature X first undergoes a 3×3 convolution to perform differential feature optimization to obtain the shallow differential feature E s In order to obtain high-frequency features, we first use average pooling and multi-head self-attention to capture low-frequency features, then use bilinear interpolation for upsampling to obtain intermediate features, and finally shallow differential features E s Subtract the intermediate features to get the high-frequency features E H , high-frequency feature E H and shallow differential features E s They are connected along the channel dimension and fused using 3×3 convolution to obtain the output feature Y of the final high-frequency enhancement unit;

[0034] Specifically expressed as:

[0035] E s =W1X (6)

[0036] E H =E s -Up(T2(MHSA(T1(Avg(E s ))))) (7)

[0037] Y=W2Concat(E s , E H ) (8)

[0038] Among them, X represents the input feature, Y represents the output feature, and E s and E H Represent shallow differential features and high-frequency features respectively, W1 and W2 represent two 3×3 convolutions, T1 and T2 represent two reshaping operations, MHSA represents multi-head self-attention calculation, and Up represents upsampling operation.

[0039] Further, including:

[0040] The multi-scale fusion attention unit includes a first path that first performs 1×1 convolution on the three input features respectively, and then uses a reshaping operation to convert the three two-dimensional image features into a one-dimensional token sequence. In order to fully interact the features between the multi-scale tokens, the one-dimensional token sequence is connected along the spatial dimension, and the spatial relationship of the features of different scales is continuously aggregated through N multi-head self-attentions; the output features of the multi-head self-attention are divided along the spatial dimension, and then the one-dimensional token sequence is restored to a two-dimensional image feature through a reshaping operation. The two-dimensional image features are subjected to 1×1 convolution, batch normalization and Sigmoid activation to obtain the corresponding attention weights. In another path, the three input features of different scales are fused in three different spatial dimensions using average pooling and upsampling to obtain three coarse-grained fusion features. Then, these three coarse-grained fusion features are optimized through 3×3 convolution, batch normalization and ReLU activation function to obtain the corresponding calibration features. Finally, the three calibration features are respectively subjected to attention weighted calculation using the corresponding attention weights to obtain the final fine-grained fusion features.

[0041] Further, including:

[0042] The multi-scale fusion attention unit is expressed as:

[0043]

[0044] in, represents the calibration characteristic, Represents the output features of multi-head self-attention, Y i represents the output features of the multi-scale fusion attention unit, represents k×k convolution, * represents matrix Hadamard product, i represents the number of stages, i∈{2,3,4}, and BN represents the normalization operation of batch samples.

[0045] Further, including:

[0046] The deep feature guidance unit includes scale calibration of the second stage, the third stage and the fine-grained fusion features obtained in the third stage to obtain the calibrated deep fusion features, and then cross-attention calculation with the high-frequency features of the edge of the change area in the first stage, and then 1×1 convolution, batch normalization and Sigmoid activation to obtain the corresponding attention weights, and after the calibrated deep fusion features are added to the high-frequency features of the edge of the change area in the first stage, they are optimized through 3×3 convolution, batch normalization and ReLU activation function to obtain the corresponding optimized features. Finally, the attention weights and optimized features are used to perform attention weighted calculation to obtain a more refined detection map.

[0047] Further, including:

[0048] The deep feature guidance unit is expressed as shown in formulas (10)-(12):

[0049] ReLU(BN(W 3×3 (X 1 +M)))*Sigmoid(BN(W 1×1 MHCA(X 1 ,M))) (10)

[0050]

[0051] MHCA k (Q,K,V)=Φ(Concat(CA1,CA2,...,CA k )) (12)

[0052] Among them, X 1 represents the high-frequency features of the first stage, M represents the fused calibration features of the last three stages, MHCA represents multi-head cross attention, CA j (·) and MHCA k (·) denote the output features of the j-th cross attention head and the output features of the multi-head cross attention with k attention heads, Q(Q j )、K(K j ) and V(V j ) represent the three input features of query, key and value respectively, where Q is composed of X 1 K and V are obtained by linear transformation. c represents the feature dimension of each attention head, and Φ represents linear transformation.

[0053] Beneficial effects: The method described in the present invention not only uses Transformer to extract local and global features of the image, but also takes into account the problem of multi-scale feature modeling of objects. The method proposes a local-global Siamese Transformer as the backbone network to extract semantic discriminative features; first, in order to solve the serious problem of false detection and missed detection of feature boundaries, a plug-and-play high-frequency enhancement unit is proposed to replace the inflexible U-shaped structure to optimize the detection boundary; secondly, in view of the problem of multi-scale modeling of objects, a multi-scale fusion attention unit is proposed to integrate the multi-scale information flow into the calculation process of self-attention; finally, the deep feature guidance unit is used to optimize the shallow detail feature information to obtain refined detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of the detection method steps described in an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of the structure of a remote sensing image change detection model based on a local-global Transformer network according to an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of the structure of the Siamese Transformer network according to an embodiment of the present invention;

[0057] Figure 4 Schematic diagram of the structure of the high-frequency enhancement unit according to an embodiment of the present invention;

[0058] Figure 5 Schematic diagram of the structure of the multi-scale fusion attention unit according to an embodiment of the present invention;

[0059] Figure 6 Schematic diagram of the structure of the deep feature guidance unit according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0061] The present invention discloses a remote sensing image change detection method based on a local-global Transformer network. First, considering that existing change detection methods rely heavily on CNN to extract features and are limited by the size of the convolution kernel receptive field, these methods often find it difficult to effectively capture the long-range dependencies of images. Therefore, the present invention proposes a local-global Transformer as the backbone network to simultaneously take into account the local and global features of dual-phase images; secondly, to address the problem of loss of detailed information on the boundaries of changed areas and the problem of multi-scale feature modeling of remote sensing objects, a plug-and-play high-frequency enhancement unit and a multi-scale fusion attention unit are proposed; finally, a deep feature guidance unit is proposed to use deep abstract semantic features to guide the optimization of shallow fine-grained features of the network to achieve more refined detection results; the above contents are unified in an end-to-end Transformer-based remote sensing image change detection network, without the need for tedious and complex step-by-step training process.

[0062] like Figure 1 As shown, the present invention includes the following steps:

[0063] Step (1): A set of dual-temporal remote sensing images is input as samples to test the algorithm of the present invention. Since neural networks are very sensitive to data distribution, data preprocessing operations are first performed on the input dual-temporal remote sensing images. To facilitate subsequent batch training, all dual-temporal remote sensing images are cropped to a uniform size of 256×256 pixels; random horizontal flipping, random vertical flipping, and random 0-180° rotation are used as data augmentation methods to train a more powerful change detection model; before the data is fed into the model, the dual-temporal images are subjected to a data normalization step to obtain standardized image data.

[0064] Step (2): Obtain the standardized dual-temporal remote sensing image data from step (1), and construct a local-global Transformer-based backbone network to simultaneously focus on the local high-frequency features and global low-frequency features of the dual-temporal remote sensing image in the feature extraction stage.

[0065] like Figure 2 As shown in the figure, the local-global Transformer network is divided into four stages. Each time the image feature passes through a stage, its scale will be reduced by half and the feature dimension will be doubled, thereby constructing a more flexible feature expression method. Specifically, first, the dual-phase image I 1 , (D, H, and W represent the number of channels, height, and width, respectively) are converted into image token sequences T through image block embedding operations. 1 , In the present invention, a two-dimensional convolution with a convolution kernel of 4×4 and a stride of 4 is used to implement the image block embedding step. Then, T1 and T 2 After four consecutive stages, in each stage, in order to effectively extract local features and global features of the bi-temporal image, the present invention proposes a local-global attention.

[0066] It consists of two Siamese Transformers: the first restricts self-attention computation to a local window to model local pixel relationships in the image, while the second performs attention computation across all pixels in the image to model global pixel relationships. Given the high cost of directly computing self-attention on shallow features in the network, only local attention is performed in the first stage. Furthermore, a patch merging operation is performed between each stage, halving the feature map size and doubling the number of channels.

[0067] The structural diagram of Siamese Transformer is as follows Figure 3 As shown in , the module consists of two standard Transformer blocks with shared weights, and each Transformer block is divided into a token mixer and a channel mixer according to the different regional functions. Assume that the feature pair of the bi-phase image is represented by X 1 , N represents the length of the token sequence, and C represents the number of feature channels. 1 and X 2 First, the spatial feature representation of the bi-phase image is captured by the token mixer, and then the feature information in the channel dimension is fused by the channel mixer.

[0068] Specifically, for token mixers, X 1 First, the layer normalization operation is performed, and then the three input features Q, K, and V are obtained through linear transformation. Q, K, and V are used to calculate the multi-head self-attention to capture the connection between all image tokens. Finally, X 1 The output features of the token mixer are obtained by skip connection and output addition. 2 After the same calculation process, the key to the above process lies in the calculation of multi-head self-attention, and its mathematical expression is shown in formulas (1)-(2):

[0069]

[0070] MHSA k (Q, K, V) = Φ (Concat (SA1, SA2, SA k )) (2)

[0071] Among them, SA i (·) and MHSA k (·) denotes the output features of the i-th self-attention head and the output features of the multi-head self-attention with k attention heads, Q(Q i)、K(K i ) and V(V i ) represent the three input features of query, key and value respectively, c represents the feature dimension of each attention head, and ≤ represents linear transformation.

[0072] The core of the channel mixer lies in the multi-layer perceptron, which consists of two linear transformation layers and an intermediate activation function to model the characteristic relationship of the channel dimension. In addition, to solve the problem that the Transformer is insensitive to spatial position, the present invention introduces conditional position coding based on deep convolution into the multi-layer perceptron. The mathematical expression of the channel mixer is shown in formulas (3)-(5):

[0073] X′ m =Φ1(norm(X m )) (3)

[0074] pos=T2(DW(T1(X′ m ))) (4)

[0075] Y=Φ2(σ(X′ m +pos))+X m (5)

[0076] Among them, X′ m , pos and Y represent the transition feature, conditional position encoding feature and output feature of channel mixer respectively, X m represents the input features of the channel mixer, norm represents layer normalization, Φ represents linear transformation, T represents the reshape operation, T1∈(1D→2D), T2∈(2D→1D), DW represents the depthwise convolution, and σ represents the GELU activation.

[0077] Step (3): Obtain the local features and global features of the dual-phase image from step (2), obtain the difference map using differential algebraic operations, construct a high-frequency enhancement unit, a multi-scale fusion attention unit, and a deep feature guidance unit, and use this to construct a complete remote sensing image change detection model based on the local-global Transformer network.

[0078] Step (3.1): Build a high-frequency enhancement unit. In the change detection task, the detection effect of the edge of the change area has a great impact on the performance indicators of the final model. Most existing change detection methods use a complex U-shaped structure combined with an attention mechanism to refine the boundary of the change area. They usually optimize features from a global perspective, which lacks pertinence for boundary information modeling. In addition, the U-shaped structure is often more complex in network design. Therefore, the present invention proposes a simple but effective high-frequency enhancement unit, which uses the self-attention mechanism and the correlation between the previous and next features to effectively extract the high-frequency feature information of the edge of the change area.

[0079] like Figure 4 As shown, assuming that the input feature is represented as X first undergoes a 3×3 convolution for differential feature optimization. To obtain high-frequency features, low-frequency features are first captured using average pooling and multi-head self-attention, then upsampled using bilinear interpolation, and finally subtracted from shallow features using differential algebraic operations. The high-frequency features and shallow features are concatenated along the channel dimension and fused using a 3×3 convolution to obtain the output features of the final high-frequency enhancement unit. The mathematical expressions of the above process are shown in formulas (6)-(8):

[0080] E s =W1X (6)

[0081] E H =E s -Up(T2(MHSA(T1(Avg(E s ))))) (7)

[0082] Y=W2Concat(E s , E H ) (8)

[0083] Among them, X represents the input feature, Y represents the output feature, and E s and E H Represent shallow differential features and high-frequency features respectively, W1 and W2 represent two 3×3 convolutions, T1 and T2 represent two reshaping operations, MHSA represents multi-head self-attention calculation, and Up represents upsampling operation.

[0084] Step (3.2): Build a multi-scale fusion attention unit. There are many different scales of objects in remote sensing images. Therefore, multi-scale modeling capability is one of the important indicators for measuring remote sensing image change detection models. Most existing change detection algorithms usually use multi-scale convolution or pooling, U-shaped fusion modules and other methods to solve multi-scale problems. Different from these methods, this paper proposes a new multi-scale fusion attention unit based on multi-head self-attention, and its structural diagram is shown in the figure below. Figure 5 shown.

[0085] The input features of this module come from the output of the last three stages of the local-global Transformer backbone network in step (2), which is expressed as To calibrate the channels, we first use a 1×1 convolution to transform the three input feature dimensions into C. Then, we use a reshape operation to transform the three 2D image features into a 1D token sequence, represented as In order to fully interact with the feature information between multi-scale tokens, Tk iThe tokens are connected along the spatial dimension, and N multi-head self-attention processes continuously aggregate the spatial relationships between features at different scales. Thanks to self-attention's ability to model long-range dependencies, all connections between tokens are unified. Considering that the original feature distribution is easily altered after multi-head self-attention, this paper proposes an attention weighting mechanism to effectively mitigate this phenomenon.

[0086] Specifically, the output features of the multi-head self-attention are first split along the spatial dimension, and then the one-dimensional token sequence is restored to a two-dimensional image feature through a reshaping operation. The two-dimensional image feature is subjected to 1×1 convolution, batch normalization and Sigmoid activation to obtain the attention weight. In order to more fully aggregate multi-scale features, the present invention proposes a scale calibration operation, that is, using average pooling and upsampling to fuse features of three different scales in three different spatial dimensions to obtain three coarse-grained fusion features. Then, the three features are optimized by 3×3 convolution, batch normalization and ReLU activation function. Finally, the corresponding attention weights are used to perform attention weighted calculation on the three features to obtain the final fine-grained fusion feature, which is expressed as The mathematical expression of the above process is shown in formula (9):

[0087]

[0088] in, represents the calibration characteristic, Represents the output features of multi-head self-attention, Y i represents the output features of the multi-scale fusion attention unit, represents k×k convolution, BN represents batch normalization, * represents matrix Hadamard product, and i represents the number of stages;

[0089] Step (3.3): Build a deep feature guidance unit. The shallow features of the network usually contain more fine-grained information, such as texture, color, boundary, etc., and also contain more background interference information. The deep features of the network are semantically directed and usually contain more abstract semantic information. From this perspective, using deep semantic features to guide shallow detail features can achieve better detection accuracy. Existing work has incorporated this idea into model design, but they often only consider deep semantic features at a single scale. The present invention proposes a new deep feature guidance unit that can use deep multi-scale semantic features to effectively optimize shallow fine-grained features to obtain a more refined detection map. Its structural diagram is shown as follows Figure 6 shown.

[0090] The overall structure is somewhat similar to the multi-scale fusion attention unit in step (3.2). Similarly, the scale calibration operation is used to fuse the three output features of the multi-scale fusion attention unit in the spatial dimension of stage 1. In order to transfer the semantic information of deep features to shallow features, a similar attention weighting mechanism is introduced. The difference is that the multi-head self-attention is replaced by multi-head cross attention.

[0091] The input of multi-head cross attention is also three features, denoted as Q, K, Q comes from the shallow features of stage 1, and K and V come from the calibrated deep fusion features. Through the interaction of Q, K, and V, multi-head cross attention can fully model the semantic connection between shallow features and multi-scale deep features. Its mathematical expression is shown in formulas (10)-(12):

[0092] ReLU(BN(W 3×3 (X 1 +M)))*Sigmoid(BN(W 1×1 MHCA(X 1 ,M))) (10)

[0093]

[0094] MHCA k (Q,K,V)=Φ(Concat(CA1,CA2,...,CA k )) (12)

[0095] Among them, X 1 represents the high-frequency features of the first stage, M represents the fused calibration features of the last three stages, MHCA represents multi-head cross attention, CA j (·) and MHCA k (·) denote the output features of the j-th cross attention head and the output features of the multi-head cross attention with k attention heads, Q(Q j )、K(K j ) and V(V j ) represent the three input features of query, key and value respectively, where Q is composed of X 1 K and V are obtained by linear transformation. c represents the feature dimension of each attention head, and Φ represents linear transformation.

[0096] Step (4): Construct a remote sensing image change detection model based on the local-global Transformer network and use the standardized dual-temporal image data for model training. Figure 2The figure shows the model proposed by the present invention. The present invention uses four binary remote sensing image change detection datasets, CDD, BTCDD, LEVIR-CD and Google, to conduct experiments. All experiments are based on the Pytorch framework and are equipped with an NVIDIA TITAN RTX GPU. Binary cross entropy is used as the training loss function, AdamW is used as the training optimizer, the weight decay coefficient is 0.01, the learning rate is 0.0001, the batch size is set to 16, and the training cycle is set to 200. Finally, the model output features are activated by Sigmoid to obtain the final prediction results, and all its prediction values ​​are compressed to between 0-1. Among them, the area where the pixel value is greater than or equal to the threshold of 0.5 is located is determined to be the change area (represented by white), and the area where the pixel value is less than the threshold of 0.5 is located is determined to be the no-change area (represented by black).

[0097] Step (5): Model change detection performance evaluation. 200 trained change detection models can be obtained from step (4). The best model is selected from these 200 models based on the five evaluation indicators: precision (PR), recall (RC), F1 score (F1), intersection over union (IoU), and overall accuracy (OA). The performance of the best model is evaluated using these five evaluation indicators. The mathematical expressions of the five evaluation indicators are shown in formulas (13)-(17):

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] Among them, TP, TN, FP and FN represent true positive, true negative, false positive and false negative, respectively.

[0104] By implementing the model of the present invention, the final change detection accuracy is greatly improved. The present invention proposes a local-global Transformer as the backbone network to perform feature extraction on dual-phase remote sensing images, effectively capturing the local and global features of the dual-phase images; proposes a high-frequency enhancement unit to replace the complex U-shaped network structure, and optimizes the high-frequency features of the boundary of the change area in a simple but efficient way; proposes a multi-scale fusion attention unit, and uses a multi-head self-attention mechanism to fully interact with the connection between multi-scale features; finally, proposes a deep feature guidance unit, which uses deep multi-scale semantic features to guide the optimization of the shallow fine-grained features of the network based on the multi-head cross-attention mechanism, and achieves more refined detection results. The above contents together constitute the remote sensing image change detection method based on the local-global Transformer network of this patent.

Claims

1. A remote sensing image change detection method based on a local-global Transformer network, characterized in that: The method comprises the following steps: S1 preprocesses multiple sets of image sample data to obtain multiple sets of standardized image data, wherein each set of image sample data includes a dual-temporal remote sensing image, and the dual-temporal remote sensing image includes a forward-temporal remote sensing image and a backward-temporal remote sensing image; S2 inputs each set of standardized image data into the local-global Transformer network in sequence, wherein the local-global Transformer network includes image block embedding and a backbone network, wherein the backbone network includes four stages, namely the first stage, the second stage, the third stage and the fourth stage, wherein the forward-temporal remote sensing image and the backward-temporal remote sensing image after the image block embedding are input into the first stage, and the output of the previous stage is used as the input of the next stage, and each stage includes two Siamese Transformer models, wherein the first Siamese Transformer model is used to limit the calculation of self-attention to a local window to model the local pixel relationship of the input image, and the second Siamese Transformer model is used to perform attention calculation on the entire pixel of the image to model the global pixel relationship of the input image, and there is an image block merging operation between each two stages, i.e., the size of the feature map is halved and the number of channels is doubled; The Siamese Transformer model consists of two standard Transformer blocks with shared weights, which are divided into a token mixer and a channel mixer according to the different regional functions. The token mixer is used to capture the spatial feature representation of the bi-temporal image. First, the feature pairs of the bi-temporal image are layer-normalized and then linearly transformed to obtain three input features. , the three input features are input into the self-attention calculation. Finally, the output features of the token mixer are obtained by adding the output of the self-attention calculation through the skip connection of the features of the bi-temporal image to obtain the corresponding output features; The channel mixer is used to fuse features in the channel dimension. First, the output features of the token mixer are normalized, and then enter the first multi-layer perceptron for depth convolution. The results of the depth convolution and the results of the linear transformation are activated and input into the second multi-layer perceptron. The output features of the second multi-layer perceptron are added to the output features of the token mixer to obtain the output features Y of the channel mixer. Both the first multi-layer perceptron and the second multi-layer perceptron are linear transformations. S3 performs a differential operation on the output of the remote sensing image in the front time phase and the output of the remote sensing image in the back time phase in each stage, and inputs the result into the high-frequency enhancement unit to obtain the high-frequency features of the edge of the change area in each stage; S4 inputs the high-frequency features of the second, third and fourth stages into the multi-scale fusion attention unit to obtain the fine-grained fusion features of each stage; S5 inputs the high-frequency features of the edge of the changed area in the first stage, the second stage, the third stage, and the fine-grained fusion features obtained in the third stage into the deep feature guidance unit to obtain a more refined detection map; S6 performs feature fusion on the output of the deep feature guidance unit and the second stage, the third stage, and the fine-grained fusion features obtained in the third stage to obtain the model output features; S7 uses the training data to train the above model and then conducts data testing to obtain the final prediction result.

2. The method according to claim 1, characterized in that Also includes: The first stage includes two Siamese Transformer models, and both Siamese Transformer models are used to limit the calculation of self-attention to a local window to model the local pixel relationship of the input image.

3. The method according to claim 1 or 2, characterized in that The image block embedding specifically includes a two-dimensional convolution with a convolution kernel of 4×4 and a stride of 4, i.e., a bi-temporal image , 、 and They represent the number of channels, height and width respectively, and are converted into image token sequences through image block embedding operations.

4. The method according to claim 1, wherein The token mixer uses the self-attention mechanism to fully capture the global spatial feature relationship of the image. Its mathematical expression is shown in formulas (1)-(2): （1） （2） in, ( )and Respectively represent The output features of the self-attention head and The output features of the multi-head self-attention of the attention heads, 、 and Represent the three input features of query, key and value respectively, represents the feature dimension of each attention head, represents a linear transformation; the channel mixer utilizes a multi-layer perceptron to effectively fuse features in the channel dimension, and its mathematical expressions are shown in formulas (3)-(5): （3） （4） （5） in, 、 and They represent the transition features, i.e., the output features of the first multi-layer perceptron, the conditional position coding features, and the output features of the channel mixer, respectively. represents the input characteristics of the channel mixer, , represents a linear transformation, where , , represents a reshape operation, where , , represents depthwise convolution, Indicates GELU activation.

5. The method according to claim 1 or 2, characterized in that The high frequency enhancement unit comprises: Input features First, a 3×3 convolution is performed to optimize the differential features to obtain shallow differential features. In order to obtain high-frequency features, we first use average pooling and multi-head self-attention to capture low-frequency features, then use bilinear interpolation for upsampling to obtain intermediate features, and finally shallow differential features. Subtract the intermediate features to get the high-frequency features , high-frequency features and shallow differential features They are connected along the channel dimension and fused using 3×3 convolution to obtain the output feature Y of the final high-frequency enhancement unit; Specifically expressed as: （6） （7） （8） in, represents the input features, represents the output features, and Represent shallow differential features and high-frequency features respectively, and represents two 3×3 convolutions, and represents two reshape operations, represents multi-head self-attention calculation, Represents an upsampling operation.

6. The method according to claim 1 or 2, characterized in that The multi-scale fusion attention unit includes the first path of first performing 1×1 convolution on the three input features, and then using the reshaping operation to convert the three two-dimensional image features into a one-dimensional token sequence. In order to fully interact with the features between the multi-scale tokens, the one-dimensional token sequence is connected together along the spatial dimension and passed through Multi-head self-attention continuously aggregates the spatial relationship of features of different scales; the output features of the multi-head self-attention are divided along the spatial dimension, and then the one-dimensional token sequence is restored to a two-dimensional image feature through a reshaping operation. The two-dimensional image feature obtains the corresponding attention weights after 1×1 convolution, batch normalization and Sigmoid activation. On the other hand, the input features of three different scales are fused in three different spatial dimensions using average pooling and upsampling to obtain three coarse-grained fusion features. Then, these three coarse-grained fusion features are optimized through 3×3 convolution, batch normalization and ReLU activation function to obtain the corresponding calibration features. Finally, the three calibration features are respectively weighted by attention using the corresponding attention weights to obtain the final fine-grained fusion features.

7. The method according to claim 6, characterized in that The multi-scale fusion attention unit is expressed as: （9） in, represents the calibration characteristic, represents the output features of multi-head self-attention, represents the output features of the multi-scale fusion attention unit, represents k×k convolution, represents the matrix Hadamard product, Indicates the number of stages, , BN represents the normalization operation of batch samples.

8. The method according to claim 1 or 2, characterized in that The deep feature guidance unit includes scale calibration of the second stage, the third stage and the fine-grained fusion features obtained in the third stage to obtain the calibrated deep fusion features, and then cross-attention calculation with the high-frequency features of the edge of the change area in the first stage, and then 1×1 convolution, batch normalization and Sigmoid activation to obtain the corresponding attention weights, and after the calibrated deep fusion features are added to the high-frequency features of the edge of the change area in the first stage, they are optimized through 3×3 convolution, batch normalization and ReLU activation function to obtain the corresponding optimized features. Finally, the attention weights and optimized features are used to perform attention weighted calculation to obtain a more refined detection map.

9. The method according to claim 8, characterized in that The deep feature guided unit is expressed as shown in formulas (10)-(12): （10） （11） （12） in, represents the high-frequency characteristics of the first stage, represents the fused calibration features of the last three stages, Indicates multi-headed cross attention, ( )and Respectively represent The output features of the cross attention head and The output features of the multi-head cross attention of the attention heads, 、 and Represent the three input features of query, key and value respectively, where Depend on The linear transformation is: Depend on The linear transformation is: represents the feature dimension of each attention head, Represents a linear transformation.

Images