A remote sensing image change detection method and device with spatiotemporal multi-granularity integration

Abstract

A method and device for remote sensing image change detection using spatiotemporal multi-granularity fusion, comprising: inputting an acquired remote sensing image dataset into a constructed spatiotemporal multi-granularity fusion network (STMINet) for training, obtaining and saving a trained STMINet; and inputting a remote sensing image to be processed, performing prediction using the trained STMINet, and obtaining a dual-temporal change prediction map.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and more specifically, relates to a remote sensing image change detection method and device with spatiotemporal multi-granularity fusion. Background Art

[0002] The concept of modern artificial intelligence was proposed by John McCarthy at the Dartmouth Conference, which marked the birth of artificial intelligence.

[0003] Change detection in remote sensing images is crucial for urban development, disaster assessment, and environmental protection. Current methods for change detection in remote sensing images primarily employ convolutional neural networks (CNNs) and transformer networks. These methods enhance the network's change detection capabilities by adding bi-temporal feature interactions and fusion. However, they focus solely on the interaction of temporal information at the same scale and fail to consider interference from ambiguous regions. Summary of the Invention

[0004] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention proposes a remote sensing image change detection method with spatiotemporal multi-granularity fusion. By focusing on designing a spatiotemporal multi-granularity module to increase the interaction of multi-granularity features, and designing multi-branch differential acquisition to further explore unclear areas, the problem of difficult identification of change areas with diverse types and scales in remote sensing images is overcome, and the spatiotemporal multi-granularity fusion network is successfully applied to the field of remote sensing image analysis.

[0005] To achieve the above objectives, according to one aspect of the present invention, a remote sensing image change detection method with spatiotemporal multi-granularity integration is provided, comprising:

[0006] The obtained remote sensing image dataset is input into the constructed spatiotemporal multi-granularity fusion network (STMINet) for training, and the trained STMINet is obtained and saved;

[0007] Input the remote sensing image to be processed, perform prediction through the trained STMINet, and obtain the dual-temporal change prediction map.

[0008] In some optional embodiments, the obtained remote sensing image dataset includes:

[0009] The remote sensing image dataset includes bi-temporal remote sensing images and corresponding change masks. Ensure that each pair of bi-temporal remote sensing images and corresponding change masks in the remote sensing image dataset have a one-to-one correspondence and have the same file name. The one-to-one correspondence means that the changed areas or unchanged areas in each pair of bi-temporal remote sensing images need to have the same size, shape, and position in the corresponding change mask. In addition, the pixel value of the changed area in each change mask in the remote sensing image dataset is 1 or 255, and the pixel value of the unchanged area is 0.

[0010] In some optional embodiments, the input is trained into a constructed spatiotemporal multi-granularity fusion network (STMINet), including:

[0011] Construct STMINet, input the obtained dataset into STMINet, set the batch size and training rounds for each iteration to train STMINet.

[0012] In some optional implementation schemes, the STMINet construction includes: an encoder, a spatiotemporal multi-granularity fusion module, a multi-branch differential acquisition, a decoder, and a prediction head.

[0013] In some optional embodiments, the encoder includes: the encoder is a dual-branch feature extractor constructed by a parameter-sharing pre-trained residual network ResNet, and a pair of dual-temporal remote sensing images are input into the structure to generate dual-temporal features F ai 、F bi , then F ai 、F bi Enter the spatiotemporal multi-granularity fusion module.

[0014] In some optional implementation schemes, the spatiotemporal multi-granularity fusion module includes: temporal information fusion, multi-granularity information aggregation and spatial information interaction.

[0015] In some optional embodiments, the time information fusion includes: firstly, ai 、F bi Linear projection and reshape operations are performed, and then concatenated to generate the query (Q), which makes Q have dual-time feature information.

[0016] In some optional embodiments, the multi-granularity information aggregation includes: setting different spatial aggregation rates S j Processing F ai 、F bi Aggregating information to obtain keys (K) and values ​​(V) of different granularities is obtained using the following formulas (1) and (2):

[0017]

[0018] In some optional embodiments, the spatial information interaction includes: using time information fusion and multi-granularity information aggregation to obtain Q, K and V to perform spatial interaction to obtain spatiotemporal multi-granularity information, and then splicing and reshaping it and adding it to F ai 、F bi Generate spatiotemporal multi-granularity features TS ai TS bi It is obtained from the following formulas (3)(4)(5)(6):

[0019]

[0020]

[0021] In some optional implementation schemes, the multi-branch differential acquisition includes: a fuzzy branch, a difference branch, a fuzzy-guided difference branch, and a fusion branch.

[0022] In some optional embodiments, the fuzzy branch includes: ai TS bi Generate TS by reducing the channel dimension through 1×1 convolution operation ai TS` bi . After that, calculate TS` ai 、tS` bi The cosine similarity matrix S between the two, the similarity matrix is ​​multiplied by tS` ai and TS` bi And connected with TS through residual ai TS` bi After addition, subtraction is performed to obtain the fuzzy feature A i From formula (7) and (8), we can get:

[0023] S=Cos(TS′ ai ,TSb′ i ) (7)

[0024] A i =|(S·TS′ ai +TS′ ai )-(S·TS′ bi +TS′ bi )| (8)

[0025] In some optional embodiments, the difference branch includes: first, obtaining a dissimilarity matrix D by subtracting the similarity matrix extracted in the fuzzy branch from 1. Then, multiplying TS by the dissimilarity matrix respectively. ai TS` bi And connected with TS through residual ai TS` biAfter adding, the rough difference feature is obtained. Finally, the difference feature is concatenated in the channel dimension and a 1×1 convolution operation is applied to obtain the precise difference feature D i From formula (9) (10), we can get:

[0026] D=1-S (9)

[0027] D i =Conv(Cat[D·TS′ ai +TS′ ai ,D·TS′ bi +TS′ bi ]) (10)

[0028] In some optional implementations, the fuzzy guided difference branch includes: processing the difference features simultaneously through parallel residual, 3×3 convolution, and 5×5 convolution to obtain multi-scale difference features. Then, through the convolution operation, we reduce the dimension of the fuzzy features and the multi-scale difference enhancement features. The score matrix between the two feature vectors is obtained through multiplication operation and Softmax function. Finally, the score matrix is ​​multiplied with the fuzzy region after convolution to achieve the effective fusion of the fuzzy region information and the enhanced difference information to obtain the fuzzy guided difference feature D` i From formulas (11)-(15), we can get:

[0029] Sc3=Sfmax(Vec(Conv(D 3i ))×Vec(Conv(A i ))) (11)

[0030] Sc5=Sfmax(Vec(Conv(D 5i ))×Vec(Conv(A i ))) (12)

[0031] D′ 3i =Sc3×Conv(A i )+D 3i (13)

[0032] D′ 5i =Sc5×Conv(A i )+D 5i (14)

[0033] D` i =Conv(Cat[D′ 3i ,D′ 5i ])+D i (15)

[0034] In some optional embodiments, the fusion branch comprises: TS ai TS bi The fusion features are obtained by splicing, and then the fusion features are processed by convolution operation to obtain spatiotemporal multi-granularity fusion features. i From formula (16), we can get:

[0035] Fusion i =Cov(Cat(TS ai ,TS bi )) (16)

[0036] In some optional implementations, the decoder and prediction head include: the decoder adopts the decoder in [Change Detection on Remote Sensing Images Using Dual-Branch Multilevel Intertemporal Network], and the prediction head adopts the prediction head in [Remote Sensing Image Change Detection with Transformers], and there is no change, so the specific structures of the two are not repeated. i and Fusion i The decoded features are processed to obtain three scales. After that, the decoded features are upsampled to the size of the original input through bilinear interpolation and then converted into a bi-temporal change prediction map through two convolutional layers and batch normalization operations.

[0037] In some optional implementation schemes, obtaining and saving the trained STMINet includes: calculating the loss and updating the network parameters through step-by-step iterations, completing the network training and saving the network parameter file.

[0038] In some optional embodiments, the prediction is performed by a trained STMINet to obtain a dual-phase change prediction map, which is characterized in that the dual-time remote sensing image pairs that need to be changed are processed into a resolution size that can be accepted by the model, and then input into the trained STMINet for prediction to obtain a dual-phase change prediction map, thereby completing the method.

[0039] The second aspect of the present invention provides a remote sensing image change detection device with spatiotemporal multi-granularity fusion, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a remote sensing image change detection method with spatiotemporal multi-granularity fusion of the present invention.

[0040] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a spatiotemporal multi-granularity fusion remote sensing image change detection method of the present invention.

[0041] The working principle of the present invention is:

[0042] The core working principle of the present invention is to realize remote sensing image change detection by constructing a spatiotemporal multi-granularity fusion network (STMINet). The STMINet network consists of an encoder, a spatiotemporal multi-granularity fusion module, a multi-branch differential acquisition, a decoder and a prediction head. First, the dual-phase remote sensing image is input into the encoder, the features are extracted and input into the spatiotemporal multi-granularity fusion module, which enhances the spatiotemporal information of the features through temporal information fusion, multi-granularity information aggregation and spatial information interaction. Then, the multi-branch differential acquisition module further mines the features of the unclear areas, including fuzzy branches, difference branches, fuzzy-guided difference branches and fusion branches, to extract more detailed change features. Finally, the decoder and prediction head decode and predict the extracted features to generate a dual-phase change prediction map to complete the change detection of the remote sensing image.

[0043] The advantages of the present invention are:

[0044] This paper proposes an innovative spatiotemporal multi-granularity fusion network (STMINet), whose design cleverly integrates multi-granularity feature interaction with a multi-branch differential acquisition strategy, significantly improving the accuracy and robustness of remote sensing image change detection. Through the spatiotemporal multi-granularity fusion module, the present invention effectively enhances the interaction between image features at different time points, achieving accurate identification of changed regions. In addition, the introduction of the multi-branch differential acquisition module, in-depth exploration of features in ambiguous regions, further optimizes the model's sensitivity to subtle changes. The present invention's automated processing flow reduces manual intervention and improves work efficiency. At the same time, its highly accurate detection results benefit from the application of advanced deep learning technology. The flexibility and scalability of the network structure enable the adjustment and optimization of the present invention in different application scenarios, ensuring its broad application prospects in the field of remote sensing image change detection. In summary, the present invention not only theoretically demonstrates the potential of deep learning in remote sensing image analysis, but also demonstrates its excellent performance and wide applicability in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Flowchart of the method of the present invention.

[0046] Figure 2 This is a diagram of the spatiotemporal multi-granularity fusion network model of the present invention.

[0047] Figure 3This is a diagram showing the effect of change detection on the spatiotemporal multi-granularity fusion network of the present invention. Figure 3 (a) is the image at time T1, Figure 3 (b) is the image at time T2, Figure 3 (c) is the prediction result of ICIFNet. Figure 3 (d) is the prediction result of BiT, Figure 3 (e) is the prediction result of DMINet, Figure 3 (f) is the prediction result of SEIFNet, Figure 3 (g) is the prediction result of STMINet, Figure 3 (h) is the true change mask. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit the scope of protection of this application.

[0049] Example 1

[0050] like Figure 1 This embodiment provides a remote sensing image change detection method with spatiotemporal multi-granularity integration, including:

[0051] Step S1: Obtain a remote sensing image dataset; adjust the dataset to an input image size of 256×256.

[0052] Step S2: Construct spatiotemporal multi-granularity fusion STMINet, such as Figure 2 STMINe consists of an encoder, a spatiotemporal multi-granularity fusion module, a multi-branch differential acquisition, a decoder, and a prediction head.

[0053] The encoder is a dual-branch feature extractor constructed by a parameter-sharing pre-trained residual network ResNet. A pair of dual-temporal remote sensing images is input into the structure to generate dual-temporal features F ai 、F bi , then F ai 、F bi Enter the spatiotemporal multi-granularity fusion module.

[0054] The spatiotemporal multi-granularity fusion module includes: temporal information fusion, multi-granularity information aggregation and spatial information interaction. Temporal information fusion is achieved by integrating F ai 、F bi Perform linear projection and reshape operations, and then splice to generate query (Q), which makes Q have dual-time feature information at the same time. Multi-granularity information aggregation is achieved by setting different spatial aggregation rates S j Processing F ai 、Fbi Aggregate information to obtain keys (K) and values ​​(V) of different granularities. Spatial information interaction uses Q, K and V obtained by temporal information fusion and multi-granularity information aggregation to perform spatial interaction to obtain spatiotemporal multi-granularity information, which is then spliced ​​and reshaped and added to F. ai 、F bi Generate spatiotemporal multi-granularity features TS ai TS bi .

[0055] Multi-branch differential acquisition, including: fuzzy branch, difference branch, fuzzy guided difference branch and fusion branch. Fuzzy branch will TS ai TS bi Generate TS by reducing the channel dimension through 1×1 convolution operation ai TS` bi . After that, calculate TS` ai TS` bi The cosine similarity matrix S between the two, the similarity matrix is ​​multiplied by TS` ai and TS` bi And connected with TS through residual ai TS` bi After addition, subtraction is performed to obtain the fuzzy feature A i The difference branch includes: first, obtaining the dissimilarity matrix D by subtracting the similarity matrix extracted in the fuzzy branch from 1. Then, multiplying TS by the dissimilarity matrix respectively. ai TS` bi And connected with TS through residual ai TS` bi After adding, the rough difference feature is obtained. Finally, the difference feature is concatenated in the channel dimension and a 1×1 convolution operation is applied to obtain the precise difference feature D i The fuzzy guided difference branch processes the difference features simultaneously through parallel residual, 3×3 convolution, and 5×5 convolution to obtain multi-scale difference features. Then, through the convolution operation, we reduce the dimensions of the blurred features and the multi-scale difference enhanced features. The score matrix between the two feature vectors is obtained through multiplication and Softmax function. Finally, the score matrix is ​​multiplied with the blurred area after convolution to achieve the effective fusion of the blurred area information and the enhanced difference information to obtain the fuzzy guided difference feature D` i The fusion branch combines TS ai TS bi The fusion features are obtained by splicing, and then the fusion features are processed by convolution operation to obtain spatiotemporal multi-granularity fusion features. i .

[0056] The decoder uses the decoder in [Change Detection on Remote Sensing Images Using Dual-Branch Multilevel Intertemporal Network], and the prediction head uses the prediction head in [RemoteSensing Image Change Detection with Transformers], with no changes. The specific structures of the two are not described here. i and Fusion i The decoded features are processed to obtain three scales. After that, the decoded features are upsampled to the size of the original input through bilinear interpolation and then converted into a bi-temporal change prediction map through two convolutional layers and batch normalization operations.

[0057] Step S3: Input the obtained remote sensing image dataset into the constructed STMINet for training. Calculate the loss and update the network parameters for each batch until all batches of training data are trained. Finally, the trained weights are obtained, and all updated parameters are saved in the weight file. The trained STMINet is obtained and saved.

[0058] Step S4: Input the remote sensing image to be processed, perform prediction using the trained STMINet, and obtain a dual-phase change prediction map.

[0059] Example 2

[0060] This embodiment provides a remote sensing image change detection device with spatiotemporal multi-granularity fusion, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a remote sensing image change detection method with spatiotemporal multi-granularity fusion of embodiment 1.

[0061] Example 3

[0062] This embodiment provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a remote sensing image change detection method with spatiotemporal multi-granularity fusion according to the present invention is implemented.

[0063] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the scope of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A remote sensing image change detection method integrating spatiotemporal and multi-granularity, characterized in that: include: The obtained remote sensing image dataset is input into the constructed spatiotemporal multi-granularity fusion network STMINet for training, and the trained STMINet is obtained and saved; Input the remote sensing image to be processed, perform prediction through the trained STMINet, and obtain the dual-temporal change prediction map; The constructed spatiotemporal multi-granularity fusion network STMINet includes an encoder, a spatiotemporal multi-granularity fusion module, a multi-branch differential acquisition, a decoder and a prediction head; The encoder is a dual-branch feature extractor constructed by a parameter-sharing pre-trained residual network ResNet. A pair of dual-temporal remote sensing images is input into the structure to generate dual-temporal features. 、 , and then 、 Input spatiotemporal multi-granularity fusion module; The spatiotemporal multi-granularity fusion module includes time information fusion, multi-granularity information aggregation and spatial information interaction; The time information fusion includes: firstly 、 Perform linear projection and reshape operations, and then concatenate to generate the query (Q), which allows Q to have both bi-temporal feature information; The multi-granularity information aggregation includes: setting different spatial aggregation rates deal with 、 Aggregate information to obtain keys (K) and values ​​(V) of different granularities; The spatial information interaction includes: using time information fusion and multi-granularity information aggregation to obtain Q, K and V to perform spatial interaction to obtain spatiotemporal multi-granularity information, and then splicing and reshaping them and adding them to 、 Generate spatiotemporal multi-granularity features 、 After that, input multi-branch differential acquisition; The multi-branch differential acquisition includes: fuzzy branch, difference branch, fuzzy guided difference branch and fusion branch; The fuzzy branch includes: 、 pass Convolution operation reduces channel dimension generation 、 ; Then, calculate 、 The cosine similarity matrix between , the similarity matrix is ​​multiplied by and and connected with 、 After addition, subtraction is performed to obtain fuzzy features ; The difference branch includes: first, obtaining a dissimilarity matrix by subtracting the similarity matrix extracted in the fuzzy branch from 1 ; Then multiply by the dissimilarity matrix 、 and connected with 、 After adding, the rough difference features are obtained; finally, the difference features are concatenated in the channel dimension and applied The convolution operation obtains accurate difference features ; The fuzzy guided precise difference branch includes: through parallel residual, convolution, Convolution simultaneously processes the difference features to obtain multi-scale difference features; then through the convolution operation, we reduce the dimensions of the blurred features and the multi-scale difference enhancement features; through the multiplication operation and the Softmax function, we obtain the score matrix between the two feature vectors; finally, the score matrix is ​​multiplied with the blurred area after convolution, realizing the effective fusion of the blurred area information and the enhanced difference information to obtain the fuzzy guidance difference feature. The design of the above branches ensures that the model is more adaptable and robust when dealing with uncertain and significantly changing areas. The fusion branch includes: through a simple channel splicing operation 、 The fusion features are obtained by splicing, and then the fusion features are processed by convolution operation to obtain spatiotemporal multi-granularity fusion features. ; Finally, the multi-branch differential acquisition extracted and Input to the decoder and prediction head for change prediction.

2. The method according to claim 1, characterized in that The obtained remote sensing image dataset includes: The remote sensing image dataset includes dual-temporal remote sensing images and corresponding change masks, ensuring that each pair of dual-temporal remote sensing images and corresponding change masks in the remote sensing image dataset correspond one-to-one and have the same file name, wherein the one-to-one correspondence means that the changed area or unchanged area in each pair of dual-temporal remote sensing images needs to have the same size, shape, and position in the corresponding change mask; in addition, the pixel value of the changed area in each change mask in the remote sensing image dataset is 1 or 255, and the pixel value of the unchanged area is 0; the input is trained in the constructed spatiotemporal multi-granularity fusion network STMINet, including: Construct STMINet, input the obtained dataset into STMINet, set the batch size and training rounds for each iteration to train STMINet.

3. The method according to claim 1, characterized in that The decoder and prediction head pair and Processing is performed to obtain decoding features at three scales; after, The decoded features are upsampled to the size of the original input via bilinear interpolation and then converted into a bi-temporal change prediction map via two convolutional layers and batch normalization operations.

4. The method according to claim 1, wherein Obtaining and saving the trained STMINet includes: calculating the loss and updating the network parameters through step-by-step iteration, completing the network training and saving the network parameter file; The trained STMINet is used to perform prediction to obtain a dual-temporal change prediction map, including: processing the dual-temporal remote sensing image pairs that need to be changed into a resolution size that can be accepted by the model, and then inputting them into the trained STMINet for prediction to obtain the dual-temporal change prediction map, thereby completing the method.

5. A remote sensing image change detection device with spatiotemporal multi-granularity integration, characterized in that: The invention comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a remote sensing image change detection method with spatiotemporal multi-granularity fusion according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, a remote sensing image change detection method with spatiotemporal multi-granularity fusion as claimed in any one of claims 1 to 4 is implemented.

Images