Remote sensing image change detection method and device based on space-time multi-granularity fusion

Abstract

The invention discloses a time-space multi-granularity fusion remote sensing image change detection method and device, and the method comprises the steps: inputting an obtained remote sensing image data set into a constructed time-space multi-granularity fusion network (STMINet) for training, obtaining a trained STMINet, and storing the trained STMINet; and inputting a remote sensing image to be processed, and performing prediction through the trained STMINet to obtain a dual-time-phase change prediction map.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and more specifically, relates to a remote sensing image change detection method and device with spatiotemporal multi-granularity fusion. Background Art

[0002] The concept of modern artificial intelligence was proposed by John McCa rthy at the Dartmouth Conference, which marked the birth of artificial intelligence.

[0003] Change detection of remote sensing images is of great significance for urban construction, disaster assessment, and environmental protection. Current change detection methods for remote sensing images mainly focus on the use of convolutional neural networks and transformer networks. The above two methods enhance the change detection ability of the network by increasing the interaction and fusion of dual-time features, but there are problems such as focusing only on the interaction of time information at the same scale and not considering the interference of unclear areas. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention proposes a remote sensing image change detection method with spatiotemporal multi-granularity fusion, which increases the interaction of multi-granularity features by focusing on designing a spatiotemporal multi-granularity module and designs multi-branch differential acquisition to further explore unclear areas. It overcomes the problem that the changing areas of various types and scales in remote sensing images are difficult to identify, and successfully applies the spatiotemporal multi-granularity fusion network to the field of remote sensing image analysis.

[0005] To achieve the above object, according to one aspect of the present invention, a remote sensing image change detection method with spatiotemporal multi-granularity fusion is provided, comprising:

[0006] The obtained remote sensing image data set is input into the constructed spatiotemporal multi-granularity fusion network (STMINet) for training, and the trained STMINet is obtained and saved;

[0007] Input the remote sensing image to be processed, perform prediction through the trained STMINet, and obtain the dual-temporal change prediction map.

[0008] In some optional embodiments, the remote sensing image dataset obtained includes:

[0009] The remote sensing image dataset includes dual-temporal remote sensing images and corresponding change masks, ensuring that each pair of dual-temporal remote sensing images and corresponding change masks in the remote sensing image dataset correspond one-to-one and have the same file name, where one-to-one correspondence means that the changed area or unchanged area in each pair of dual-temporal remote sensing images needs to have the same size, shape, and position in the corresponding change mask; in addition, the pixel value of the changed area in each change mask in the remote sensing image dataset is 1 or 255, and the pixel value of the unchanged area is 0.

[0010] In some optional implementation schemes, the input is trained into a constructed spatiotemporal multi-granularity fusion network (STMINet), including:

[0011] Construct STMINet, input the obtained dataset into STMINet, set the batch size and training rounds of each iteration to train STMINet.

[0012] In some optional implementation schemes, the construction of STMINet includes: an encoder, a spatiotemporal multi-granularity fusion module, a multi-branch differential acquisition, a decoder and a prediction head.

[0013] In some optional embodiments, the encoder includes: the encoder is a dual-branch feature extractor constructed by a parameter-sharing pre-trained residual network ResNet, a pair of dual-phase remote sensing images is input into the structure to generate a dual-time feature F ai 、F bi , and then F ai 、F bi Enter the spatiotemporal multi-granularity fusion module.

[0014] In some optional implementation schemes, the spatiotemporal multi-granularity fusion module includes: time information fusion, multi-granularity information aggregation and spatial information interaction.

[0015] In some optional implementation schemes, the time information fusion includes: firstly, by ai 、F bi Linear projection and reshape operations are performed, and then concatenation is performed to generate the query (Q), which makes Q have dual-time feature information at the same time.

[0016] In some optional implementations, the multi-granularity information aggregation includes: setting different spatial aggregation rates S j Processing F ai 、F bi Aggregate information to obtain keys (K) and values ​​(V) of different granularities using the following formulas (1) and (2):

[0017]

[0018] In some optional implementation schemes, the spatial information interaction includes: using the Q, K and V obtained by temporal information fusion and multi-granularity information aggregation to perform spatial interaction to obtain spatiotemporal multi-granularity information, and then splicing and reshaping them and adding them to F ai 、F bi Generate spatiotemporal multi-granularity features TS ai TS bi It is obtained from the following formulas (3)(4)(5)(6):

[0019]

[0020]

[0021] In some optional implementation schemes, the multi-branch differential acquisition includes: a fuzzy branch, a difference branch, a fuzzy-guided difference branch and a fusion branch.

[0022] In some optional embodiments, the fuzzy branch includes: ai TS bi Generate TS by reducing the channel dimension through 1×1 convolution operation ai TS` bi . After that, calculate TS` ai ,tS` bi The cosine similarity matrix S between the two, the similarity matrix is ​​multiplied by tS` ai and TS` bi And connected with TS through residual ai TS` bi After addition, subtraction is performed to obtain the fuzzy feature A i From formula (7) and (8), we can get:

[0023] S=Cos(TS′ ai ,TSb′ i ) (7)

[0024] A i =|(S·TS′ ai +TS′ ai )-(S·TS′ bi +TS′ bi )| (8)

[0025] In some optional embodiments, the difference branch includes: first, obtaining a dissimilarity matrix D by subtracting the similarity matrix extracted in the fuzzy branch from 1. Then, multiplying TS by the dissimilarity matrix respectively. ai TS` bi And connected with TS through residual ai TS` biAfter adding, the rough difference feature is obtained. Finally, the difference feature is concatenated in the channel dimension and a 1×1 convolution operation is applied to obtain the precise difference feature D i From formula (9) (10) we get:

[0026] D=1-S (9)

[0027] D i =Conv(Cat[D·TS′ ai +TS′ ai ,D·TS′ bi +TS′ bi ]) (10)

[0028] In some optional implementations, the fuzzy guided difference branch includes: processing the difference features simultaneously through parallel residual, 3×3 convolution, and 5×5 convolution to obtain multi-scale difference features. Then, through the convolution operation, we reduce the dimensions of the fuzzy features and the multi-scale difference enhancement features. The score matrix between the two feature vectors is obtained through multiplication operation and Softmax function. Finally, the score matrix is ​​multiplied with the fuzzy area after convolution to achieve the effective fusion of the fuzzy area information and the enhanced difference information to obtain the fuzzy guided difference feature D` i From formulas (11)-(15), we can get:

[0029] Sc3=Sfmax(Vec(Conv(D 3i ))×Vec(Conv(A i ))) (11)

[0030] Sc5=Sfmax(Vec(Conv(D 5i ))×Vec(Conv(A i ))) (12)

[0031] D′ 3i =Sc3×Conv(A i )+D 3i (13)

[0032] D′ 5i =Sc5×Conv(A i )+D 5i (14)

[0033] D` i =Conv(Cat[D′ 3i ,D′ 5i ])+D i (15)

[0034] In some optional embodiments, the fusion branch comprises: TS ai TS bi The fusion features are obtained by splicing, and then the fusion features are processed by convolution operation to obtain the spatiotemporal multi-granularity fusion feature Fusion i From formula (16), we get:

[0035] Fusion i =Cov(Cat(TS ai ,TS bi )) (16)

[0036] In some optional implementation schemes, the decoder and prediction head include: the decoder adopts the decoder in [Change Detection on Remote Sensing Images Using Dual-Branch Multilevel Intertemporal Network], and the prediction head adopts the prediction head in [Remote Sensing Image Change Detection with Transformers], and there is no change, so the specific structures of the two are not repeated. i and Fusion i The decoded features are processed to obtain three scales. After that, the decoded features are upsampled to the size of the original input through bilinear interpolation, and then converted into a bi-phase change prediction map through two convolutional layers and batch normalization operations.

[0037] In some optional implementation schemes, the step of obtaining and saving the trained STMINet includes: calculating the loss and updating the network parameters through step-by-step iterations, completing the network training and saving the network parameter file.

[0038] In some optional implementation schemes, the prediction is performed through the trained STMINet to obtain a dual-phase change prediction map, which is characterized in that the dual-time remote sensing image pairs that need to be changed are processed into a resolution size that can be accepted by the model, and then input into the trained STMINet for prediction to obtain a dual-phase change prediction map, thereby completing the method.

[0039] The second aspect of the present invention provides a remote sensing image change detection device with spatiotemporal multi-granularity fusion, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a remote sensing image change detection method with spatiotemporal multi-granularity fusion of the present invention.

[0040] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a remote sensing image change detection method of the present invention that integrates time and space multi-granularity.

[0041] The working principle of the present invention is:

[0042] The core working principle of the present invention is to realize remote sensing image change detection by constructing a spatiotemporal multi-granularity fusion network (STMINet). The STMINet network consists of an encoder, a spatiotemporal multi-granularity fusion module, a multi-branch differential acquisition, a decoder and a prediction head. First, the dual-phase remote sensing image is input into the encoder, the features are extracted and input into the spatiotemporal multi-granularity fusion module, which enhances the spatiotemporal information of the features through time information fusion, multi-granularity information aggregation and spatial information interaction. Then, the multi-branch differential acquisition module further mines the features of the unclear area, including fuzzy branches, difference branches, fuzzy-guided difference branches and fusion branches, to extract more detailed change features. Finally, the decoder and the prediction head decode and predict the extracted features, generate a dual-phase change prediction map, and complete the change detection of the remote sensing image.

[0043] The advantages of the present invention are:

[0044] The present invention proposes an innovative spatiotemporal multi-granularity fusion network (STMINet), whose design cleverly integrates multi-granularity feature interaction and multi-branch differential acquisition strategy, significantly improving the accuracy and robustness of remote sensing image change detection. Through the spatiotemporal multi-granularity fusion module, the present invention effectively enhances the interaction between image features at different time points and realizes the accurate identification of the change area. In addition, the introduction of the multi-branch differential acquisition module, in particular, conducts in-depth mining of the features of unclear areas, further optimizing the sensitivity of the model to subtle changes. The automated processing flow of the present invention reduces manual intervention and improves work efficiency. At the same time, its high-accuracy detection results benefit from the application of advanced deep learning technology. The flexibility and scalability of the network structure make it possible to adjust and optimize the present invention in different application scenarios, ensuring its broad application prospects in the field of remote sensing image change detection. In summary, the present invention not only theoretically demonstrates the potential of deep learning in remote sensing image analysis, but also demonstrates its excellent performance and wide applicability in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Flow chart of the method of the present invention.

[0046] Figure 2 This is a diagram of the spatiotemporal multi-granularity fusion network model of the present invention.

[0047] Figure 3This is a diagram showing the effect of detecting changes in the spatiotemporal multi-granularity fusion network of the present invention. Figure 3 (a) is the image at time T1, Figure 3 (b) is the image at time T2, Figure 3 (c) is the prediction result of ICIFNet. Figure 3 (d) is the prediction result of BiT. Figure 3 (e) is the prediction result of DMINet. Figure 3 (f) is the prediction result of SEIFNet. Figure 3 (g) is the prediction result of STMINet. Figure 3 (h) is the true change mask. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the scope of protection of the present application.

[0049] Example 1

[0050] like Figure 1 This embodiment provides a remote sensing image change detection method with spatiotemporal multi-granularity integration, including:

[0051] Step S1, obtain a remote sensing image dataset; adjust the dataset to an input image size of 256×256.

[0052] Step S2: construct spatiotemporal multi-granularity fusion STMINet, such as Figure 2 STMINe includes an encoder, a spatiotemporal multi-granularity fusion module, a multi-branch differential acquisition, a decoder, and a prediction head.

[0053] The encoder is a dual-branch feature extractor constructed by a parameter-sharing pre-trained residual network ResNet. A pair of dual-temporal remote sensing images is input into the structure to generate dual-temporal features F ai 、F bi , and then F ai 、F bi Enter the spatiotemporal multi-granularity fusion module.

[0054] The spatiotemporal multi-granularity fusion module includes: temporal information fusion, multi-granularity information aggregation and spatial information interaction. Temporal information fusion is achieved by integrating F ai 、F bi Linear projection and reshape operations are performed, and then concatenated to generate query (Q), which makes Q have dual-time feature information at the same time. Multi-granularity information aggregation is achieved by setting different spatial aggregation rates S j Processing F ai 、Fbi Aggregate information to obtain keys (K) and values ​​(V) of different granularities. Spatial information interaction uses Q, K and V obtained by temporal information fusion and multi-granularity information aggregation to obtain spatial and temporal multi-granularity information, which is then spliced ​​and reshaped and added to F. ai 、F bi Generate spatiotemporal multi-granularity features TS ai TS bi .

[0055] Multi-branch differential acquisition, including: fuzzy branch, difference branch, fuzzy guided difference branch and fusion branch. Fuzzy branch will TS ai TS bi Generate TS by reducing the channel dimension through 1×1 convolution operation ai TS` bi . After that, calculate TS` ai TS` bi The cosine similarity matrix S between the two, the similarity matrix is ​​multiplied by TS` ai and TS` bi And connected with TS through residual ai TS` bi After addition, subtraction is performed to obtain the fuzzy feature A i The difference branch includes: first, the dissimilarity matrix D is obtained by subtracting the similarity matrix extracted in the fuzzy branch from 1. Then, the dissimilarity matrix is ​​multiplied by TS respectively. ai TS` bi And connected with TS through residual ai TS` bi After adding, the rough difference feature is obtained. Finally, the difference feature is concatenated in the channel dimension and a 1×1 convolution operation is applied to obtain the precise difference feature D i The fuzzy guided difference branch processes the difference features simultaneously through parallel residual, 3×3 convolution, and 5×5 convolution to obtain multi-scale difference features. Then, through the convolution operation, we reduce the dimensions of the blurred features and the multi-scale difference enhanced features. The score matrix between the two feature vectors is obtained through multiplication and Softmax function. Finally, the score matrix is ​​multiplied with the blurred area after convolution to achieve the effective fusion of the blurred area information and the enhanced difference information to obtain the fuzzy guided difference feature D` i The fusion branch combines TS ai TS bi The fusion features are obtained by splicing, and then the fusion features are processed by convolution operation to obtain the spatiotemporal multi-granularity fusion feature Fusion i .

[0056] The decoder uses the decoder in [Change Detection on Remote Sensing Images Using Dual-Branch Multilevel Intertemporal Network], and the prediction head uses the prediction head in [RemoteSensing Image Change Detection with Transformers], and there is no change, so I will not repeat the specific structure of the two. i and Fusion i The decoded features are processed to obtain three scales. After that, the decoded features are upsampled to the size of the original input through bilinear interpolation, and then converted into a bi-phase change prediction map through two convolutional layers and batch normalization operations.

[0057] Step S3: Input the obtained remote sensing image data set into the constructed STMINet for training, calculate the loss for each batch and update the network parameters until all batches of training data are trained, and finally obtain the trained weights, and all updated parameters will be saved in the weight file. Obtain the trained STMINet and save it.

[0058] Step S4: input the remote sensing image to be processed, perform prediction using the trained STMINet, and obtain a dual-phase change prediction map.

[0059] Example 2

[0060] The present embodiment provides a remote sensing image change detection device with spatiotemporal multi-granularity fusion, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a remote sensing image change detection method with spatiotemporal multi-granularity fusion of embodiment 1.

[0061] Example 3

[0062] This embodiment provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a remote sensing image change detection method with spatiotemporal multi-granularity fusion of the present invention is implemented.

[0063] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A remote sensing image change detection method with spatiotemporal multi-granularity integration, characterized in that: include: The obtained remote sensing image data set is input into the constructed spatiotemporal multi-granularity fusion network STMINet for training, and the trained STMINet is obtained and saved; Input the remote sensing image to be processed, perform prediction through the trained STMINet, and obtain the dual-temporal change prediction map.

2. The method according to claim 1, characterized in that The remote sensing image data set obtained includes: The remote sensing image dataset includes dual-phase remote sensing images and corresponding change masks, ensuring that each pair of dual-phase remote sensing images and the corresponding change masks in the remote sensing image dataset correspond one-to-one and have the same file name, wherein the one-to-one correspondence means that the changed area or unchanged area in each pair of dual-phase remote sensing images needs to have the same size, shape, and position in the corresponding change mask; in addition, the pixel value of the changed area in each change mask in the remote sensing image dataset is 1 or 255, and the pixel value of the unchanged area is 0; the input is trained in the constructed spatiotemporal multi-granularity fusion network STMINet, including: Construct STMINet, input the obtained dataset into STMINet, set the batch size and training rounds of each iteration to train STMINet.

3. The method according to claim 3, characterized in that: The constructed spatiotemporal multi-granularity fusion network STMINet includes an encoder, a spatiotemporal multi-granularity fusion module, multi-branch differential acquisition, a decoder and a prediction head.

4. The method according to claim 4, characterized in that: The encoder is a dual-branch feature extractor constructed by a parameter-sharing pre-trained residual network ResNet. A pair of dual-phase remote sensing images is input into the structure to generate dual-time features F ai 、F bi , and then F ai 、F bi Enter the spatiotemporal multi-granularity fusion module.

5. The method according to claim 4, characterized in that The spatiotemporal multi-granularity fusion module includes time information fusion, multi-granularity information aggregation and spatial information interaction; The time information fusion includes: firstly, by ai 、F bi Perform linear projection and reshape operations, and then concatenate to generate a query (Q), which enables Q to have dual-time feature information at the same time; The multi-granularity information aggregation includes: setting different spatial aggregation rates S j Processing F ai 、F bi Aggregate information to obtain keys (K) and values ​​(V) of different granularities; The spatial information interaction includes: using the Q, K and V obtained by time information fusion and multi-granularity information aggregation to obtain spatial and temporal multi-granularity information, and then splicing and reshaping them and adding them to F ai 、F bi Generate spatiotemporal multi-granularity features TS ai TS bi Then, input multi-branch differential acquisition.

6. The method according to claim 4, characterized in that The multi-branch differential acquisition includes: fuzzy branch, difference branch, fuzzy guided difference branch and fusion branch; The fuzzy branch includes: ai TS bi Reduce the channel dimension through 1×1 convolution operation Afterwards, calculate The cosine similarity matrix A between the two, the similarity matrix is ​​multiplied by and And through residual connection with After addition, subtraction is performed to obtain the fuzzy feature A i ; The difference branch includes: firstly, obtaining a dissimilarity matrix D by subtracting the similarity matrix extracted in the fuzzy branch from 1; then, multiplying the dissimilarity matrix by And through residual connection with After adding, the rough difference feature is obtained; finally, the difference feature is concatenated in the channel dimension and a 1×1 convolution operation is applied to obtain the precise difference feature D i ; The fuzzy guided precise difference branch includes: processing the difference features simultaneously through parallel residual, 3×3 convolution, and 5×5 convolution to obtain multi-scale difference features; then through convolution operation, we reduce the dimensions of the fuzzy features and the multi-scale difference enhancement features; through multiplication operation and Softmax function, we obtain the score matrix between the two feature vectors; finally, multiply the score matrix with the fuzzy area after convolution, and realize the effective fusion of the fuzzy area information and the enhanced difference information to obtain the fuzzy guided difference feature D` i ; The design of the above branches ensures that the model is more adaptable and robust when dealing with uncertain and significantly changing areas; The fusion branch includes: TS ai TS bi The fusion features are obtained by splicing, and then the fusion features are processed by convolution operation to obtain the spatiotemporal multi-granularity fusion feature Fusion i ; Finally, the multi-branch differential acquisition extracted and Fusion i Input to decoder and prediction head for change prediction.

7. The method according to claim 4, characterized in that The decoder and prediction head are and Fusion i The decoded features at three scales are processed; then, the decoded features are upsampled to the size of the original input through bilinear interpolation, and then converted into a dual-phase change prediction map through two convolutional layers and batch normalization operations.

8. The method according to claim 1, characterized in that The step of obtaining and saving the trained STMINet includes: calculating the loss and updating the network parameters through step-by-step iteration, completing the network training and saving the network parameter file. The trained STMINet is used to perform prediction to obtain a dual-phase change prediction map, including: processing the dual-time remote sensing image pairs that need to be changed into a resolution size that can be accepted by the model, and then inputting them into the trained STMINet for prediction to obtain a dual-phase change prediction map, thereby completing the method.

9. A remote sensing image change detection device with spatiotemporal multi-granularity integration, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a remote sensing image change detection method with spatiotemporal multi-granularity fusion as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a remote sensing image change detection method with spatiotemporal multi-granularity fusion as described in any one of claims 1 to 8 is implemented.

Images