A remote sensing image change detection method based on spatiotemporal attention enhancement

Abstract

The present invention discloses a remote sensing image change detection method based on spatiotemporal attention enhancement, comprising: inputting a training set obtained through a remote sensing image group into a twin network to obtain a feature map after feature extraction; wherein the features to be extracted include edge features, line features, shape features, texture features, scene features and semantic features; inputting the feature map after feature extraction into a spatiotemporal information enhancement module to obtain a spatiotemporal information enhanced feature map; obtaining a change binary map based on the spatiotemporal information enhanced feature map; and training the twin network, the spatiotemporal information enhancement module and the decoding module; obtaining a final change binary map to detect changes in remote sensing images. The present invention improves the accuracy of the model without increasing model parameters, fully captures local features in the input data, has obvious effects in detecting small changing targets and identifying the edges of changing objects, and can significantly improve the performance and accuracy of the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image change detection, and in particular relates to a remote sensing image change detection method based on spatiotemporal attention enhancement. Background Art

[0002] Change detection is a key technology in remote sensing image analysis. Its goal is to detect changes in local features by analyzing multi-temporal images of the same area or target acquired at different times. It has important applications in many fields, such as environmental monitoring, disaster assessment, and urban planning. The key to change detection lies in identifying differences between two or more temporal images while eliminating differences caused by variations in image acquisition conditions (such as lighting and weather). The earliest change detection methods relied primarily on image difference operations, which are sensitive to changes in lighting and viewpoint. Subsequent methods have included mutual information methods, change vector analysis, and classification-based approaches. These methods have improved the accuracy and robustness of change detection to a certain extent. In recent years, with the development of deep learning, it has also begun to be applied to change detection tasks. Deep learning methods can automatically learn matching relationships and change patterns between images, significantly improving detection accuracy and performance. Common deep learning frameworks include Siamese networks, ConvGRU, and multi-temporal convolutional neural networks. These deep learning networks can detect both incremental and decremental changes between images and exhibit strong adaptability to complex changes. In summary, change detection technology has evolved from traditional image processing methods to deep learning methods. Different methods have their own advantages and disadvantages. Deep learning methods are the mainstream direction of contemporary change detection research due to their powerful feature learning and matching capabilities.

[0003] Transformer-based change detection methods have achieved significant progress in achieving high accuracy. Traditional change detection methods typically use small kernel convolution as a feature extractor to extract feature information from input data. However, small kernel convolution has certain limitations in perceiving small changes in objects and cannot fully capture local features in the input data. Furthermore, it still has limitations in detecting small changes in objects and identifying edges of changing objects. Summary of the Invention

[0004] This paper proposes a remote sensing image change detection method based on spatiotemporal attention enhancement to overcome the above technical problems;

[0005] A remote sensing image change detection method based on spatiotemporal attention enhancement includes the following steps:

[0006] S1: Acquire a plurality of remote sensing image groups, wherein the remote sensing image groups include a pre-phase image map and a post-phase image map acquired in the same area, and acquire a training set and a test set based on the remote sensing image groups;

[0007] The preceding phase image is a remote sensing image taken at a first time, and the following phase image is a remote sensing image taken at a second time, wherein the first time and the second time are not equal;

[0008] S2: Input the training set into the Siamese network, and perform feature extraction through the multi-scale convolutional attention module of the Siamese network to obtain a feature map after feature extraction; wherein the features to be extracted include edge features, line features, shape features, texture features, scene features and semantic features;

[0009] S3: Input the feature map after feature extraction into the spatiotemporal information enhancement module to obtain the spatiotemporal information enhanced feature map;

[0010] S4: Inputting the spatiotemporal information enhanced feature map into a decoding module for upsampling to obtain a change binary map;

[0011] S5: According to the change binary graph, based on the cross entropy loss function, the twin network, the spatiotemporal information enhancement module and the decoding module are trained;

[0012] S6: The test set is sequentially input into the trained twin network structure, spatiotemporal information enhancement module and decoding module to obtain the final change binary map to detect changes in the remote sensing image.

[0013] Furthermore, in step S2, the Siamese network includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a fourth feature extraction layer and an MLP perception layer;

[0014] The first feature extraction layer is used to extract edge features of the front phase image and the back phase image respectively, and obtain a first forward feature map F0 and a first backward feature map P0;

[0015] The second feature extraction layer is used to extract line features and shape features of the first forward feature map and the first backward feature map according to the first forward feature map F0 and the first backward feature map P0, respectively, to obtain a second forward feature map F1 and a second backward feature map P1; and to splice the first forward feature map and the second forward feature map, and to splice the first backward feature map and the second backward feature map;

[0016] The third feature extraction layer is used to extract texture features of the second forward feature map and the second backward feature map according to the second forward feature map F1 and the second backward feature map P1, respectively, to obtain a third forward feature map F2 and a third backward feature map P2; and to splice the third forward feature map with the spliced ​​first forward feature map and the second forward feature map, and to splice the third backward feature map with the spliced ​​first backward feature map and the second backward feature map;

[0017] The fourth feature extraction layer is used to extract scene features of the third forward feature map and the third backward feature map according to the third forward feature map F2 and the third backward feature map P2, respectively, to obtain a fourth forward feature map F3 and a fourth backward feature map P3; and to splice the fourth forward feature map with the spliced ​​first forward feature map, the second forward feature map, and the third forward feature map to obtain a fifth forward feature map F4, and to splice the fourth backward feature map with the spliced ​​first backward feature map, the second backward feature map, and the third backward feature map to obtain a fifth backward feature map P4;

[0018] The MLP perception layer is used to extract semantic features of the fifth forward feature map F4 and the fifth backward feature map P4 according to the fifth forward feature map F4 and the fifth backward feature map P4 to obtain a feature map D1 after feature extraction.

[0019] Furthermore, the network structures of the first feature extraction layer, the second feature extraction layer, the third feature extraction layer, and the fourth feature extraction layer are the same, and all include a downsampling module and a multi-scale convolutional attention module;

[0020] The calculation formula of the multi-scale convolutional attention module is as follows:

[0021]

[0022]

[0023] Where: a represents the multi-scale convolutional attention feature map; Conv 1×1 (·) represents a 1*1 convolution operation; LeakyReLU(·) represents an activation function; BN(·) represents a batch normalization function; DConv(·) represents a depth-wise separable convolution function; H in represents the input feature map; λ represents the regularization coefficient; represents the L2 regularization term, W represents the attention weight matrix; H out Represents the output of the multi-scale convolutional attention module; Represents the matrix multiplication operator; Scale i (·) denotes the convolution operation in the i-th branch.

[0024] Furthermore, in S3, the method for obtaining the spatiotemporal information enhanced feature map is as follows:

[0025] S31: Obtain a query matrix q, a key matrix k, and a value matrix v according to the feature graph after feature extraction;

[0026] S32: performing global information enhancement on the feature map after feature extraction according to the query matrix q, the key matrix k and the value matrix v to obtain a feature map after global information enhancement;

[0027] S33: Obtain the spatiotemporal key matrix k(S) according to the feature map enhanced by the global information t ) and the space-time value matrix v(S t );

[0028] S34: Obtaining an attention mask of the feature map after global information enhancement according to the query matrix and the spatiotemporal key matrix;

[0029] S35: Obtain the feature map A after global information enhancement according to the attention mask of the feature map after global information enhancement t The matching degree with the query matrix q;

[0030] S36: Feature map A enhanced according to the global information t The matching degree with the query matrix q is used to obtain the spatiotemporal information enhancement graph.

[0031] Furthermore, in S31, the query matrix q, key matrix k and value matrix v are obtained as follows:

[0032]

[0033] Where: LinearWithPosEnc means using the hyperbolic tangent function to generate the position encoding vector; x is the feature map after feature extraction, W q is the query weight matrix; W k is the bond weight matrix; W v is the value weight matrix; P q is the query position encoding vector; P k is the key position encoding vector; P v is the value position encoding vector.

[0034] Furthermore, in S32, the method for obtaining the feature map after global information enhancement is as follows:

[0035]

[0036] Where: N represents the total number of pixels of the feature map after global information enhancement, C represents the number of channels of the feature map after feature extraction, and A tIt is the feature map after global information enhancement; Softmax represents the normalization function.

[0037] Furthermore, in S33, the spatiotemporal key matrix k(S t ) and the space-time value matrix v(S t ) is obtained as follows:

[0038]

[0039] Furthermore, in S34, the attention mask of the feature map after global information enhancement is obtained as follows;

[0040]

[0041] Furthermore, in said S35, the feature map A after global information enhancement is obtained t The matching degree with the query matrix q;

[0042]

[0043] Where n represents the pixel number of the feature map after global information enhancement.

[0044] Furthermore, in said S36, the spatiotemporal information enhancement map is obtained as follows;

[0045]

[0046] Among them, ⊙ represents the pixel-by-pixel product of the matrix, and D2 represents the spatiotemporal information enhanced feature map.

[0047] Beneficial effects: The present invention proposes a remote sensing image change detection method based on spatiotemporal attention enhancement. After feature extraction of the remote sensing image in the twin network, the spatiotemporal information enhancement module is used to improve the accuracy of the model without increasing the model parameters, fully capture the local features in the input data, and have obvious effects in the detection of small changing targets and the identification of the edges of changing objects, which can significantly improve the performance and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 This is a flow chart of the remote sensing image change detection method of the present invention;

[0050] Figure 2 Schematic diagram of the twin network structure in an embodiment of the present invention;

[0051] Figure 3 Schematic diagram of the structure of the spatiotemporal information enhancement module in an embodiment of the present invention.

[0052] Figure 4 Schematic diagram of the detection results of the detection method of the present application and other methods in the embodiments of the present invention on the public data sets LEVIR-CD and DSFIN-CD;

[0053] Figure 5 4 is a flow chart of a remote sensing image change detection method in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0055] The present invention provides a remote sensing image change detection method based on spatiotemporal attention enhancement, comprising the following steps: Figure 1 and Figure 5 As shown;

[0056] S1: Acquire a plurality of remote sensing image groups, wherein the remote sensing image groups include a pre-phase image map and a post-phase image map acquired in the same area, and acquire a training set and a test set based on the remote sensing image groups;

[0057] The preceding phase image is a remote sensing image taken at a first time, and the following phase image is a remote sensing image taken at a second time, wherein the first time and the second time are not equal;

[0058] Specifically, in this embodiment, two remote sensing images taken at different times in the same area are selected, one as the pre-phase image and the other as the post-phase image. The pre-phase image and the post-phase image (i.e., the dual-phase high-resolution remote sensing image) are cropped and the dataset is divided according to a preset ratio to obtain a training set, a test set, and a validation set.

[0059] S2: Input the training set into the Siamese network (i.e., multi-scale large-kernel convolutional attention encoder), perform feature extraction through the multi-scale convolutional attention module of the Siamese network, and obtain a feature map after feature extraction;

[0060] S3: The feature map after feature extraction is input into the spatiotemporal information enhancement module for information enhancement to obtain the spatiotemporal information enhanced feature map to better integrate the deep features and shallow features;

[0061] S4: Inputting the spatiotemporal information enhanced feature map into a decoding module for upsampling to obtain a change binary map;

[0062] S5: According to the change binary graph, based on the cross entropy loss function, the twin network, the spatiotemporal information enhancement module and the decoding module are trained;

[0063] S6: Input the test set into the trained twin network structure, spatiotemporal information enhancement module and decoding module in sequence to obtain the final change binary map to detect changes in remote sensing images.

[0064] In step S2, the twin network includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a fourth feature extraction layer and an MLP perception layer; Figure 2 As shown;

[0065] The first feature extraction layer is used to extract edge features of the front phase image and the back phase image respectively, and obtain a first forward feature map F0 and a first backward feature map P0; wherein, the first forward feature map F0 is obtained by extracting the front phase image through the first feature extraction layer, and the first backward feature map F0 is obtained by extracting the back phase image through the first feature extraction layer;

[0066] Specifically, the training set is input into the first feature extraction layer of the Siamese network to obtain a first forward feature map F0 and a first backward feature map P0;

[0067] The first forward feature map F0 and the first backward feature map P0 are downsampled by the downsampling module of the second feature extraction layer with a convolution kernel size of 3, a step size of 2, and a padding of 1, and then the multi-scale convolution attention module of the second feature extraction layer is used to perform feature extraction to generate the second forward feature map F1 and the second backward feature map P1.

[0068] The second feature extraction layer is used to extract line features and shape features of the first forward feature map and the first backward feature map according to the first forward feature map F0 and the first backward feature map P0, respectively, to obtain a second forward feature map F1 and a second backward feature map P1; and to splice the first forward feature map and the second forward feature map, and to splice the first backward feature map and the second backward feature map;

[0069] The third feature extraction layer is used to extract texture features of the second forward feature map and the second backward feature map according to the second forward feature map F1 and the second backward feature map P1, respectively, to obtain a third forward feature map F2 and a third backward feature map P2; and to splice the third forward feature map with the spliced ​​first forward feature map and the second forward feature map, and to splice the third backward feature map with the spliced ​​first backward feature map and the second backward feature map;

[0070] Specifically, after the second forward feature map F1 and the second backward feature map P1 are downsampled, the multi-scale convolutional attention module of the third feature extraction layer of the twin network is used to generate the third forward feature map F2 and the third backward feature map P2.

[0071] The fourth feature extraction layer is used to extract scene features of the third forward feature map and the third backward feature map according to the third forward feature map F2 and the third backward feature map P2, respectively, to obtain a fourth forward feature map F3 and a fourth backward feature map P3; and to splice the fourth forward feature map with the spliced ​​first forward feature map, the second forward feature map, and the third forward feature map to obtain a fifth forward feature map F4, and to splice the fourth backward feature map with the spliced ​​first backward feature map, the second backward feature map, and the third backward feature map to obtain a fifth backward feature map P4; the scene features therein include streets, buildings, forests, etc.

[0072] Specifically, after the third forward feature map F2 and the third backward feature map P2 are downsampled, the multi-scale convolutional attention module of the fourth feature extraction layer of the twin network is used to generate the fourth forward feature map F3 and the fourth backward feature map P3.

[0073] The MLP perception layer is used to extract semantic features of the fifth forward feature map F4 and the fifth backward feature map P4 according to the fifth forward feature map F4 and the fifth backward feature map P4 to obtain a feature map D1 after feature extraction.

[0074] Specifically, in this embodiment, the front phase image and the back phase image are respectively upsampled by the convolution layers in the first feature extraction layer, the second feature extraction layer, the third feature extraction layer, and the fourth feature extraction layer, thereby improving the resolution of the front phase image and the back phase image. Finally, the front phase image and the back phase image with improved resolution are passed through the MLP layer (perception layer) to obtain the feature map D1 after feature extraction;

[0075] The MLP layer consists of a fully connected layer, a GELU activation function, and a fully connected layer. In the MLP layer, the feature map is first linearly mapped through a fully connected layer, where each input feature is multiplied by a weight and a bias term is added to produce a new feature vector. The new feature vector is nonlinearly transformed through a GELU activation function and then linearly transformed through another fully connected layer to produce the final output. The methods used by various parts of the twin network to splice images are all existing technologies and are not described in detail here.

[0076] The network structures of the first feature extraction layer, the second feature extraction layer, the third feature extraction layer, and the fourth feature extraction layer are the same, and all include a downsampling module and a convolutional attention module; the convolutional attention module is a multi-scale convolutional attention module;

[0077] The calculation formula of the multi-scale convolutional attention module is as follows:

[0078]

[0079]

[0080] Where: a represents the multi-scale large kernel convolution attention feature map; Conv 1×1 (·) represents a 1*1 convolution operation; LeakyReLU(·) represents an activation function; BN(·) represents a batch normalization function; DConv(·) represents a depth-wise separable convolution function; H in represents the input feature map; λ represents the regularization coefficient; represents the L2 regularization term, W represents the attention weight matrix; H out Represents the output of the multi-scale convolutional attention module; Represents the matrix multiplication operator; Scale i (·) represents the convolution operation in the i-th branch. In this embodiment, i∈{0,1,2,3,4}, where i represents the index of the branch in the multi-scale large kernel convolution attention.

[0081] In S3, the method for obtaining the spatiotemporal information enhancement feature map is as follows: Figure 3 As shown:

[0082] S31: According to the feature map after feature extraction, the query matrix q, the key matrix k and the value matrix v are obtained through the following linear transformation;

[0083] Get the query matrix q, key matrix k and value matrix v as follows:

[0084]

[0085] Where: LinearWithPosEnc means using the hyperbolic tangent function to generate the position encoding vector; x is the feature map after feature extraction, W q is the query weight matrix; W k is the bond weight matrix; W v is the value weight matrix; P q is the query position encoding vector; P k is the key position encoding vector; P v is the value position encoding vector.

[0086] S32: performing global information enhancement on the feature map after feature extraction according to the query matrix q, the key matrix k and the value matrix v to obtain a feature map after global information enhancement;

[0087] In S32, the method for obtaining the feature map after global information enhancement is as follows:

[0088]

[0089] Where: N represents the total number of pixels of the feature map after global information enhancement, C represents the number of channels of the feature map after feature extraction, and A t It is the feature map after global information enhancement; Softmax represents the normalization function.

[0090] S33: Obtain the spatiotemporal key matrix k(S) according to the feature map enhanced by the global information t ) and the space-time value matrix v(S t );

[0091] In S33, the space-time key matrix k(S t ) and the space-time value matrix v(S t ) is obtained as follows:

[0092]

[0093] Specifically, the feature map after global information enhancement is linearly transformed again to transform it into a spatiotemporal key matrix k(S t ) and the space-time value matrix v(S t ), q represents mapping the input features to the query Query, k(S t ) represents mapping the input feature to the key value Key, v(S t ) means mapping the input feature to the value Value;

[0094] S34: Obtaining an attention mask of the feature map after global information enhancement according to the query matrix and the spatiotemporal key matrix;

[0095]

[0096] S35: Obtain the feature map A after global information enhancement according to the attention mask of the feature map after global information enhancement t The matching degree with the query matrix q;

[0097] Through the attention mask M t To calculate the feature map A after global information enhancement t The similarity with the query matrix q.

[0098]

[0099] Among them, n represents the pixel number of the feature map after global information enhancement. For pixel n, by t Sum each element and calculate the total matching degree Z t . Z t It reflects the total matching of all pixels around pixel n.

[0100] S36: Feature map A enhanced according to the global information t The matching degree with the query matrix q is used to obtain the spatiotemporal information enhancement graph;

[0101]

[0102] Among them, ⊙ represents the pixel-by-pixel product of the matrix, and D2 represents the spatiotemporal information enhanced feature map.

[0103] Specifically, in this embodiment, the spatiotemporal information enhancement feature map D2 is quadrupled by fusing the feature maps through four upsampling layers, and outputting a change binary map with a resolution of H×W×2, where H and W both represent the resolution of the change binary map.

[0104] Specifically, for the change binary graph, the loss cross entropy function is calculated using the change binary graph and the labels in the training data set, the network is converged through error back propagation, and the trained network model is obtained. The generalization ability and performance of the model are evaluated on the validation set to avoid overfitting of the model on the training set. The model parameters are saved according to the indicators of the validation set and the twin network structure training is completed. The test set is finally used to evaluate the performance of the model and finally output the change binary graph.

[0105] The present invention preprocesses the two-phase remote sensing image to obtain training data and test data; the training data is input into the twin neural network model for training, and the error back propagation is performed using the change detection label and the output result graph of the method to obtain a trained twin neural network model; the test data is input into the trained twin neural network model to obtain a binary prediction image for remote sensing image change detection to represent the changed area in the remote sensing image. Figure 1 As shown,

[0106] An embodiment of the present invention is as follows:

[0107] The proposed method is evaluated on two widely popular datasets, including the LEVIR-CD dataset and the DSIFN-CD dataset:

[0108] The LEVIR-CD dataset, acquired from Google Earth, is 1024×1024 pixels in size. These bi-temporal images, spanning 5 to 14 years, feature significant land use change, particularly building changes. LEVIR-CD covers a wide range of building types, including residential villas, high-rise apartments, small garages, and large warehouses. Here, we focus on building-related changes, including building growth (changes from soil / grass / ground or buildings under construction to new construction areas). These bi-temporal images are annotated with binary labels (1 for change and 0 for unchanged) from remote sensing images.

[0109] The DSIFN-CD dataset consists of six large, bi-temporal, high-resolution image pairs covering six cities in China (i.e., Beijing, Chengdu, Shenzhen, Chongqing, Wuhan, and Xi'an). The five large image pairs (i.e., Beijing, Chengdu, Shenzhen, Chongqing, and Wuhan) were cropped into 394 sub-image pairs of size 512×512. After data augmentation, a collection of 3940 bi-temporal image pairs was collected. The Xi'an image pairs were cropped into 48 image pairs for model testing. There are 3600 image pairs in the training dataset, 340 in the validation dataset, and 48 in the test dataset.

[0110] A. In this example, a multi-scale large kernel convolution attention module is used to extract features. The network uses attention to extract remote sensing image features. Table 1 shows the specific network structure. Specifically, the multi-scale convolution attention module consists of four pairs of strip convolutions of different sizes and a 1×1 convolution to extract image features.

[0111] Table 1

[0112]

[0113] B. Experimental data sets LEVIR-CD and DSFIN-CD were selected, and the proposed BIT model and ChangFormer model were selected as references. The experimental results are shown in Table 2:

[0114] Table 2

[0115]

[0116] The evaluation indicators represent the overall accuracy, recall, the harmonic mean of precision and recall, and the intersection area of ​​the true target area and the detected target area divided by their union area.

[0117] C. In order to more objectively evaluate the role of each module in the model, an ablation experiment is added to illustrate the results. The experimental results are shown in Table 3:

[0118] Table 3

[0119]

[0120]

[0121] Among them, CDCA represents the multi-scale large kernel convolution attention module, and SIEM represents the spatiotemporal information enhancement module.

[0122] The following conclusions can be drawn from the above experiments:

[0123] (1) The experimental results in Table 2 show that on the LEVIR-CD and DSIFN-CD datasets, it provides better overall performance in change detection compared with other high-performance models such as ChangeFormer and BIT.

[0124] (2) Table 3 shows the ablation results of the experiments. It can be observed that the proposed model with both CDCA and SIEM produces the best performance in terms of precision, recall, F1, and IoU. The implementation with SIEM is more competitive than the model without it, with precision promoted to 87.18% and 87.78%, respectively. Although the accuracy of using CDCA or SIEM is improved, the method equipped with both CDCA and SIEM produces the best performance on the four criteria. Specifically, the precision, recall, F1, and IoU reach 92.11%, 87.78%, 89.89%, and 81.65%, respectively. It can be more clearly seen that after adding SIEM to better fuse deep features and shallow features, the effect of change detection is better, which proves that the fusion of shallow features and deep features of the spatiotemporal information enhancement module plays an important role.

[0125] (3) Figure 4 The graphs of the change detection results of different models are shown. According to the graphs of the change detection results of the model output, our method has the following advantages over ChangeFormer and BIT, which makes the results closer to the true labels. Figure 4As can be seen in the figure, the upper right corner of our output (D) is closer to the actual change, while the outputs of the ChangeFormer method (C) and the BIT method (B) appear to have overlooked some of the change. Our output (H) has better spatial consistency, indicating that the detected change areas are well connected, reflecting the continuity of the actual change. However, the outputs of the ChangeFormer method (H) and the BIT method (F) are more fragmented.

[0126] The present invention provides a remote sensing image change detection method based on spatiotemporal attention enhancement, which uses multi-scale large-kernel convolutional attention bands to regularize attention weights for feature extraction to better perceive small changes in targets. Specifically, in the process of calculating the attention weights, each element of the attention weight matrix can be squared and accumulated, and added to the loss function for optimization. This can limit the size of the attention weights and make them smoother. This method improves the performance and accuracy of the change detection model. The design of this module not only reduces the feature differences between deep features and shallow features, but also improves the representation ability and robustness of the model. This feature fusion mechanism can effectively capture the spatiotemporal information in the input remote sensing image, thereby improving the change detection model's edge perception ability for changing targets.

[0127] This paper uses a plug-and-play spatiotemporal information enhancement module based on a multi-scale large-kernel convolutional attention feature extractor. This module is designed for change detection and improves model accuracy without increasing model parameters. This module's contextual information enhancement (i.e., global information enhancement) aims to expand the receptive field of feature maps, improve the model's nonlinear capabilities, and better represent temporal changes in images.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing image change detection method based on spatiotemporal attention enhancement, characterized in that: The following steps are involved: S1: Acquire a plurality of remote sensing image groups, wherein the remote sensing image groups include a pre-phase image map and a post-phase image map acquired in the same area, and acquire a training set and a test set based on the remote sensing image groups; The preceding phase image is a remote sensing image taken at a first time, and the following phase image is a remote sensing image taken at a second time, wherein the first time and the second time are not equal; S2: Input the training set into the Siamese network, and perform feature extraction through the multi-scale convolutional attention module of the Siamese network to obtain a feature map after feature extraction; wherein the features to be extracted include edge features, line features, shape features, texture features, scene features and semantic features; In step S2, the twin network includes a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a fourth feature extraction layer and an MLP perception layer; The first feature extraction layer is used to extract edge features of the front phase image and the back phase image respectively, and obtain a first forward feature map F0 and a first backward feature map P0; The second feature extraction layer is used to extract line features and shape features of the first forward feature map and the first backward feature map according to the first forward feature map F0 and the first backward feature map P0, respectively, to obtain a second forward feature map F1 and a second backward feature map P1; and to splice the first forward feature map and the second forward feature map, and to splice the first backward feature map and the second backward feature map; The third feature extraction layer is used to extract texture features of the second forward feature map and the second backward feature map according to the second forward feature map F1 and the second backward feature map P1, respectively, to obtain a third forward feature map F2 and a third backward feature map P2; and to splice the third forward feature map with the spliced ​​first forward feature map and the second forward feature map, and to splice the third backward feature map with the spliced ​​first backward feature map and the second backward feature map; The fourth feature extraction layer is used to extract scene features of the third forward feature map and the third backward feature map according to the third forward feature map F2 and the third backward feature map P2, respectively, to obtain a fourth forward feature map F3 and a fourth backward feature map P3; and to splice the fourth forward feature map with the spliced ​​first forward feature map, the second forward feature map, and the third forward feature map to obtain a fifth forward feature map F4, and to splice the fourth backward feature map with the spliced ​​first backward feature map, the second backward feature map, and the third backward feature map to obtain a fifth backward feature map P4; The MLP perception layer is used to extract semantic features of the fifth forward feature map F4 and the fifth backward feature map P4 according to the fifth forward feature map F4 and the fifth backward feature map P4 to obtain a feature map D1 after feature extraction; S3: Input the feature map after feature extraction into the spatiotemporal information enhancement module to obtain the spatiotemporal information enhanced feature map; S4: Inputting the spatiotemporal information enhanced feature map into a decoding module for upsampling to obtain a change binary map; S5: According to the change binary graph, based on the cross entropy loss function, the twin network, the spatiotemporal information enhancement module and the decoding module are trained; S6: Input the test set into the trained twin network structure, spatiotemporal information enhancement module and decoding module in sequence to obtain the final change binary map to detect changes in remote sensing images.

2. The remote sensing image change detection method based on spatiotemporal attention enhancement according to claim 1, characterized in that: The network structures of the first feature extraction layer, the second feature extraction layer, the third feature extraction layer, and the fourth feature extraction layer are the same, and all include a downsampling module and a multi-scale convolutional attention module; The calculation formula of the multi-scale convolutional attention module is as follows: Where: a represents the multi-scale convolutional attention feature map; Conv 1×1 (·) represents a 1*1 convolution operation; LeakyReLU(·) represents an activation function; BN(·) represents a batch normalization function; DConv(·) represents a depth-wise separable convolution function; H in represents the input feature map; λ represents the regularization coefficient; represents the L2 regularization term, W represents the attention weight matrix; H out Represents the output of the multi-scale convolutional attention module; Represents the matrix multiplication operator; Scale i (·) denotes the convolution operation in the i-th branch.

3. The remote sensing image change detection method based on spatiotemporal attention enhancement according to claim 1, characterized in that: In S3, the method for obtaining the spatiotemporal information enhancement feature map is as follows: S31: Obtain a query matrix q, a key matrix k, and a value matrix v according to the feature graph after feature extraction; S32: performing global information enhancement on the feature map after feature extraction according to the query matrix q, the key matrix k and the value matrix v to obtain a feature map after global information enhancement; S33: Obtain the spatiotemporal key matrix k(S) according to the feature map enhanced by the global information t ) and the space-time value matrix v(S t ); S34: Obtaining an attention mask of the feature map after global information enhancement according to the query matrix and the spatiotemporal key matrix; S35: Obtain the feature map A after global information enhancement according to the attention mask of the feature map after global information enhancement t The matching degree with the query matrix q; S36: Feature map A enhanced according to the global information t The matching degree with the query matrix q is used to obtain the spatiotemporal information enhancement graph.

4. The remote sensing image change detection method based on spatiotemporal attention enhancement according to claim 3 is characterized in that: In S31, the query matrix q, key matrix k and value matrix v are obtained as follows: Where: LinearWithPosEnc means using the hyperbolic tangent function to generate the position encoding vector; x is the feature map after feature extraction, W q is the query weight matrix; W k is the bond weight matrix; W v is the value weight matrix; P q is the query position encoding vector; P k is the key position encoding vector; P v is the value position encoding vector.

5. The remote sensing image change detection method based on spatiotemporal attention enhancement according to claim 4 is characterized in that: In the above S32, the method for obtaining the feature map after global information enhancement is as follows: Where: N represents the total number of pixels of the feature map after global information enhancement, C represents the number of channels of the feature map after feature extraction, and A t It is the feature map after global information enhancement; Softmax represents the normalization function.

6. The remote sensing image change detection method based on spatiotemporal attention enhancement according to claim 5, characterized in that: In S33, the space-time key matrix k(S t ) and the space-time value matrix v(S t ) is obtained as follows:

7. The remote sensing image change detection method based on spatiotemporal attention enhancement according to claim 6, characterized in that: In the S34, the attention mask of the feature map after the global information enhancement is obtained as follows; 8. The remote sensing image change detection method based on spatiotemporal attention enhancement according to claim 7, characterized in that: In the S35, the feature map A after global information enhancement is obtained t The matching degree with the query matrix q; Where n represents the pixel number of the feature map after global information enhancement.

9. The remote sensing image change detection method based on spatiotemporal attention enhancement according to claim 8, characterized in that: In said S36, obtaining the spatiotemporal information enhancement map is as follows; Among them, ⊙ represents the pixel-by-pixel product of the matrix, and D2 represents the spatiotemporal information enhanced feature map.

Images