A remote sensing image change detection method based on space selection and time evolution

Abstract

The application belongs to the technical field of image processing, and discloses a remote sensing image change detection method based on space selection and time sequence evolution, a space and time sequence double-branch encoder structure is constructed, a first branch accurately explores the spatial positioning of a change region in a double-time image pair, and another branch generates a virtual video from the double-time image through a specific data enhancement method to supplement local details of the change region. The method has excellent performance through a small amount of parameters and calculation.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to a method for detecting changes in remote sensing images based on spatial selection and temporal evolution. Background Technology

[0002] Remote sensing image change detection refers to the quantitative analysis of changes in the Earth's surface scene at the same geographical location from two remote sensing images taken at different times. Change detection is an important research direction in the field of remote sensing and a crucial task for monitoring changes in the Earth's surface environment. With the continuous advancement of satellite technology and optical imaging sensors, remote sensing image change detection has been widely applied in various fields, including land cover assessment, land use monitoring, natural disaster management, and water resource management. In recent years, the performance of change detection has been significantly improved due to the powerful representation capabilities provided by deep neural networks. Deep learning-based algorithms can learn features that identify changes from a large number of supervised samples, thus surpassing traditional manual feature extraction methods.

[0003] However, despite significant progress in deep learning-based methods, several challenges remain. First, current Change Detection (CD) methods fall short in extracting semantic associations between bitemporal images when dealing with extensive datasets containing complex external disturbances such as seasonal changes, environmental differences, and illumination variations. Therefore, a method is needed to facilitate the interaction between semantic features of bitemporal images and effectively summarize contextual information from both temporal perspectives. Second, traditional attention mechanisms face significant computational and storage challenges when processing long-sequence data, and the computation of global attention often leads to a waste of computational resources due to the inherent sparsity of change detection regions. Third, existing methods typically prioritize accurate global change localization or focus on capturing detailed features. This emphasis can sometimes make predictions of change regions vague and inaccurate. Therefore, this application proposes a remote sensing image change detection method based on spatial selection and temporal evolution to improve model detection performance. Summary of the Invention

[0004] The purpose of this application is to provide a remote sensing image change detection method based on spatial selection and temporal evolution, aiming to reveal the semantic correlation between two-time images from both spatial and temporal perspectives. Specifically, this application constructs a spatial and temporal dual-branch encoder structure. The first branch accurately explores the spatial localization of changed regions in the two-time image pair, while the other branch generates virtual videos from the two-time images using specific data augmentation methods to supplement the local details of the changed regions. This method achieves excellent performance with a small number of parameters and computational cost.

[0005] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0006] A remote sensing image change detection method based on spatial selection and temporal evolution, comprising:

[0007] Acquire two remote sensing images taken at the same location at different times, and divide the two remote sensing images into several sub-images of the same size. The sub-images corresponding to the two remote sensing images are denoted as F1 and F2, respectively.

[0008] In the spatial decoder branch, subgraphs F1 and F2 are input into a weight-shared feature encoding network to extract features, and three multi-scale feature maps C are extracted from subgraphs F1 and F2 respectively. 1_img1 C 2_img1 C 3_img1 and three multi-scale feature maps C 1_img2 C 2_img2 C 3_img2 ; feature map C 1_img1 and C 1_img2 Feature map C 2_img1 and C 2_img2 Feature map C 3_img1 and C 3_img2 The inputs are fed in pairs into the dynamically selected interaction attention module to obtain the spatial interaction feature map cf. 1_img1 and CF 1_img2 Spatial interaction feature map cf 2_img1 and CF 2_img2 Spatial interaction feature map cf 3_img1 and CF 3_img2 ;

[0009] In the temporal decoder branch, subgraphs F1 and F2 are used to generate a virtual video frame T0 through data augmentation; and the virtual video frame T0 is input into the 3D temporal encoder to obtain multi-scale temporal feature maps ct1, ct2, and ct3.

[0010] In the feature decoder branch, the spatial interaction feature map (cf) is used. 3_img1 cf 3_img2 The temporal feature map ct3 is input into the spatiotemporal feature fusion module to obtain the spatiotemporal fusion feature map df1, and the spatial interaction feature map cf is then input into the spatiotemporal feature fusion module. 2_img1 cf 2_img2 The temporal feature map ct2 is input into the spatiotemporal feature fusion module to obtain the spatiotemporal fusion feature map df2, and the spatial interaction feature map cf is then input into the module. 1_img1 cf 1_img2 The spatiotemporal fusion feature map df3 is obtained by inputting the temporal feature map ct1 into the spatiotemporal feature fusion module;

[0011] The multi-scale spatiotemporal fusion feature maps df1, df2, and df3 are passed through a residual network and progressive upsampling to obtain the final change detection prediction map G1 for sub-maps F1 and F2. This process is repeated for all sub-maps to obtain the final change detection prediction map for the two remote sensing images.

[0012] Several alternative methods are provided below, but they are not intended as additional limitations on the overall solution above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.

[0013] Preferably, the sub-graphs F1 and F2 are input into a weight-shared feature encoding network to extract features, and three multi-scale feature maps C are extracted from the sub-graphs F1 and F2 respectively. 1_img1 C 2_img1 C 3_img1 and three multi-scale feature maps C 1_img2 C 2_img2 C 3_img2 ,include:

[0014] The weight-sharing feature encoding network uses the first three layers of a ResNet18 network, and feature maps C are extracted from the first layers of the two weight-sharing ResNet18 networks respectively. 1_img1 and C 1_img2 Feature maps C are extracted from the second layer of two ResNet18 networks with shared weights. 2_img1 and C 2_img2 Feature maps C are extracted from the third layer of two ResNet18 networks with shared weights. 3_img1 and C 3_img2 .

[0015] Preferably, the feature map C 1_img1 and C 1_img2 Feature map C 2_img1 and C 2_img2 Feature map C 3_img1 and C 3_img2 The inputs are fed in pairs into the dynamically selected interaction attention module to obtain the spatial interaction feature map cf. 1_img1 and CF 1_img2 Spatial interaction feature map cf 2_img1 and CF 2_img2 Spatial interaction feature map cf 3_img1 and CF 3_img2 ,include:

[0016] There are three dynamic selection interaction attention modules, and the processing procedure of each dynamic selection interaction attention module is as follows:

[0017] Let C be the feature map input to the dynamically selected interactive attention module. img1 and C img2 Its size is w×h×c, where w, h, and c are the feature map C, respectively. img1 and C img2 Width, height, and number of channels;

[0018] Feature map C img1 and C img2 Through a single block division and projection operation, it is reshaped into a size of The feature vectors of feature map C img1 and C img2 The reshaped feature vectors are represented as Q1, K1, V1 and Q2, K2, V2, respectively, where s 2 This represents the number of small regions in the block, and Q1 and Q2 represent the feature map C, respectively. img1 and C img2 The query features, where K1 and K2 represent feature maps C respectively. img1 and C img2 The key features, V1 and V2 represent feature maps C respectively. img1 and C img2 Value characteristics;

[0019] Based on Q1, K1, V1 and Q2, K2, V2, the operation proceeds through two branches, with the first branch operation as follows:

[0020] (1) First, calculate the average value of each small region along the wh dimension for features Q1, K2, and V2 to obtain the average feature Q1. m K2 m and V2 m Then average feature Q1 m and K2 m Performing matrix multiplication yields a matrix of size s. 2 ×c is the adjacency matrix representing the similarity between features Q1 and K2.

[0021] (2) For each small region in query feature q1 In the adjacency matrix The indices of the top K most relevant small regions are recorded into a vector. In, then according to the vector The index in the key feature K2 and the value feature V2 are grouped together to obtain a new key feature. Sum value characteristics Where K is the hyperparameter;

[0022] (3) Each small region in query feature Q1 and its corresponding new key features Perform a dot product operation, then combine it with the new feature value. Performing the dot product operation yields a value of eigenvectors

[0023] (4) s 2 eigenvectors The combination yields an attention output cf of size wh×c. img1 ;

[0024] The second branch operation is as follows:

[0025] (1) First, calculate the average value of each small region along the wh dimension for features Q2, K1, and V1 to obtain the average feature Q2. m K1 m and V1 m Then average feature Q2 m and K1 m Performing matrix multiplication yields a matrix of size s. 2 ×c is the adjacency matrix representing the similarity between Q2 and K1.

[0026] (2) For each small region in query feature Q2 In the adjacency matrix The indices of the top K most relevant small regions are recorded into a vector. In, then according to the vector The index in the key feature K1 and the value feature V1 are grouped together to obtain a new key feature. Sum value characteristics

[0027] (3) Each small region in query feature Q2 and its corresponding new key features Perform a dot product operation, then combine it with the new feature value. Performing the dot product operation yields a value of eigenvectors

[0028] (4) s 2 eigenvectors The combination yields an attention output cf of size wh×c. img2 .

[0029] Preferably, the step of generating virtual video frame T0 from subgraphs F1 and F2 using data augmentation includes:

[0030] The virtual video constructed from subgraph F1 to subgraph F2 consists of a total of N frames, where the nth frame f n The calculation formula is as follows:

[0031] f n = (1-λ)F1+λF2

[0032]

[0033] A single video frame with shape W×H×C is f n Expanding the frame length dimension yields a tensor of shape 1×W×H×C. Concatenating N video frames along the frame length dimension results in a final video frame tensor of shape N×W×H×C, i.e., the virtual video frame T0, where 0≤n<N, and W, H, and C represent the values ​​of a single video frame f. n The width, height, and number of channels are given, with λ representing the weight.

[0034] Preferably, the step of inputting the virtual video frame T0 into the 3D temporal encoder to obtain multi-scale temporal feature maps ct1, ct2, and ct3 includes:

[0035] The virtual video frame T0 with shape N×W×H×C is first compressed through a bottleneck structure to obtain the temporal feature ct0. This bottleneck structure consists of a 3D convolution with a kernel size of 3×9×9, a depthwise separable convolution with a kernel size of 3×1×1, a BatchNorm layer, and a ReLU activation function.

[0036] The time feature ct0 is input into the first layer of the 3D-residual network to obtain the time feature ct1. The time feature ct1 is input into the second layer of the 3D-residual network to obtain the time feature ct2. The time feature ct2 is input into the third layer of the 3D-residual network to obtain the time feature ct3. The structure of the 3D-residual network from input to output consists of a 3D convolutional layer with a kernel size of 1×1×1, a 3D convolutional layer with a kernel size of 3×3×3, and a 3D convolutional layer with a kernel size of 1×1×1. The output of the first 3D convolutional layer is added to the output of the last 3D convolutional layer through a skip connection to obtain the output of the 3D-residual network.

[0037] Preferably, the spatiotemporal feature fusion module performs the following operations:

[0038] The input to the spatiotemporal feature fusion module is the spatial interaction feature map (cf). img1 and CF img2 And time feature map ct;

[0039] spatial interaction feature map cf img1 and CF img2 The channel dimensions are concatenated, and then passed through a 3×3 convolutional layer, a linear layer, and a ReLU activation function to finally obtain the feature vector O. s ;

[0040] The temporal feature map ct is simultaneously subjected to max pooling and average pooling along the video frame length dimension. The max pooling and average pooling results are then concatenated and passed through a linear layer and a ReLU activation function to finally obtain the feature vector T. s ;

[0041] The eigenvector O s and eigenvector T s The spatiotemporal fusion feature map df is obtained by cascading the channel dimensions.

[0042] Preferably, the step of obtaining the final change detection prediction map G1 for sub-maps F1 and F2 by passing the multi-scale spatiotemporal fusion feature maps df1, df2, and df3 through a residual network and progressive upsampling includes:

[0043] The spatiotemporal fusion feature map df1 is upsampled to the size of the spatiotemporal fusion feature map df2 after passing through the residual network. It is then concatenated with the spatiotemporal fusion feature map df2 and passed through the residual network together. Then, it is upsampled to the size of the spatiotemporal fusion feature map df3 and concatenated with the spatiotemporal fusion feature map df3 and passed through the residual network together to obtain the final change detection prediction map G1.

[0044] This application provides a remote sensing image change detection method based on spatial selection and temporal evolution, utilizing parallel encoding of spatial Transformer and temporal CNN. To alleviate the problems of insufficient semantic interaction between two-temporal images and the high computational resource requirements of traditional attention mechanisms, this application leverages the inherent sparsity in change and dynamically employs sparse selection of interactive attention to locate the semantically most relevant regions between the two-temporal images, thereby guiding attention to the truly changed regions. To improve the problems of blurred and inaccurate boundary of changed regions, this application uses specific data augmentation methods to construct video frame sequences between the two-temporal images, and then uses 3D-CNN to obtain rich temporal dimension information, thereby improving the accuracy of changed region boundaries. Accurate change detection of heterogeneous remote sensing images is achieved while improving operational efficiency and generalization performance. Attached Figure Description

[0045] Figure 1 This is an architecture diagram of the remote sensing image change detection method based on spatial selection and temporal evolution proposed in this application;

[0046] Figure 2 A schematic diagram illustrating the dynamic selection of the interactive attention module in this application;

[0047] Figure 3 This is a schematic diagram of the spatiotemporal feature fusion module of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application.

[0050] One embodiment provides a remote sensing image change detection method based on spatial selection and temporal evolution, aiming to reveal the semantic correlation between two-time images from both spatial and temporal perspectives. A dual-branch encoder structure combining a spatial Transformer and a temporal CNN is employed. The first branch accurately explores the spatial localization of centering changes in the two-time images, while the other branch generates pseudo-videos from the two-time images using specific data augmentation methods to supplement local details of the changed regions. This method achieves excellent performance with a small number of parameters and computational cost.

[0051] Specifically, such as Figure 1 As shown, the remote sensing image change detection method based on spatial selection and temporal evolution in this embodiment includes the following steps:

[0052] Step S1: Acquire two remote sensing images taken at the same location at different times. Divide the two remote sensing images into several sub-images of the same size. The sub-images corresponding to the two remote sensing images are denoted as F1 and F2, respectively.

[0053] For ease of operation, the size of the sub-graph is set to 256×256 in this embodiment. In other embodiments, the size of the sub-graph can be adjusted according to actual needs.

[0054] Step S2: In the spatial decoder branch, sub-graphs F1 and F2 are input into the weight-shared feature encoding network to extract features, and three multi-scale feature maps C are extracted from sub-graphs F1 and F2 respectively. 1_img1 C 2_img1 C 3_img1 and three multi-scale feature maps C 1_img2 C 2_img2 C 3_img2 .

[0055] In this embodiment, the weight-sharing feature encoding network uses the first three layers of a ResNet18 network. Feature maps C are extracted from the first layers of the two weight-sharing ResNet18 networks, respectively. 1_img1 and C1_img2 Feature maps C are extracted from the second layer of two ResNet18 networks with shared weights. 2_img1 and C 2_img2 Feature maps C are extracted from the third layer of two ResNet18 networks with shared weights. 3_img1 and C 3_img2 .

[0056] Step S21: Let the size of remote sensing image sub-images F1 and F2 be W×H×C, where W, H and C are the width, height and number of channels of the image, respectively.

[0057] Step S22: Input subgraphs F1 and F2 into the ResNet18 network, and obtain a result of size [value missing] through convolutional layers and max pooling layers. Feature map C 1_img1 and C 1_img2 Where L is the third-layer multi-scale feature map C 3_img1 and C 3_img2 The number of channels is [number], the kernel size is 7×7×C, the stride is 2, and the number of output channels is [number]. Feature map C 1_img1 and C 1_img2 A convolutional layer with a kernel size of 3×3×C and a stride of 1 is used to obtain a size of Feature map C 2_img1 and C 2_img2 , feature map C 2_img1 and C 2_img2 A convolutional layer with a kernel size of 3×3×C and a stride of 1 is used to obtain a size of Feature map C 3_img1 and C 3_img2 The next two convolutional layers will reduce the height and width of the image to half of their original values, and double the number of channels.

[0058] Step S3: Extract the feature maps C from the two branches. 1_img1 and C 1_img2 Feature map C 2_img1 and C 2_img2 Feature map C 3_img1 and C 3_img2 The inputs are fed in pairs into the dynamically selected interaction attention module to obtain the spatial interaction feature map cf. 1_img1 and CF 1_img2 Spatial interaction feature map cf 2_img1 and CF 2_img2 Spatial interaction feature map cf 3_img1 and CF 3_img2 .

[0059] Dynamic selection of interactive attention effectively reduces computation by introducing sparsity into the network. To promote semantic interaction between bi-temporal images, spatial dimensionality reduction is first used to calculate the semantic relevance between each region of the bi-temporal images at a coarse-grained level. The indices of the K most relevant regions of sub-image F1 in sub-image F2 are found. These indices are then used to group the selected regions into a set of key-value pairs. At a fine-grained level, attention scores are calculated between each subquery q1 of sub-image F1 and the corresponding key-value pair sets Kg2 and Vg2 of sub-image F2. Interactive attention can be defined as:

[0060] CrossAtt = (softmax(q1*Kg2)) T )*Vg2)

[0061] In the formula, CrossAtt is the attention score, softmax is the softmax activation function, and Kg2 is the value of Kg2. T It is the transpose of Kg2.

[0062] The three dynamically selected interactive attention modules provided in this embodiment have the same structure, therefore let the feature map input to the dynamically selected interactive attention module be C. img1 (representing C) 1_img1 C 2_img1 Or C 3_img1 ) and C img2 (representing C) 1_img2 C 2_img2 Or C 3_img2 The feature map C has a size of w×h×c, where w, h, and c are the feature maps respectively. img1 and C img2 Width, height, and number of channels. For example... Figure 2 As shown, the details are as follows:

[0063] Feature map C img1 and C img2 Through a single block and projection operation, it is reshaped into a shape of size [size missing]. The feature vectors of feature map C img1 and C img2 The reshaped feature vectors are represented as Q1, K1, v1 and Q2, K2, V2, respectively, where s 2 This represents the number of small regions in the block, and Q1 and Q2 represent the feature map C, respectively. img1 and C img2 The query features, where K1 and K2 represent feature maps C respectively. img1 and C img2 The key features, V1 and V2 represent feature maps C respectively. img1 and C img2 The value characteristics.

[0064] Based on Q1, K1, V1 and Q2, K2, V2, the operation proceeds through two branches, with the first branch operation as follows:

[0065] (1) First, calculate the average value (Mean) of each small region along the wh dimension for features Q1, K2, and V2 to obtain the average feature Q1. m K2 m and V2 m Then average feature Q1 m and K2 m Performing matrix multiplication yields a matrix of size s. 2 ×c is the adjacency matrix representing the similarity between features Q1 and K2.

[0066] (2) For each small region in query feature Q1 In the adjacency matrix The indices of the top K most relevant small regions are recorded into a vector. In, then according to the vector The index in the key feature K2 and the value feature V2 are grouped together to obtain a new key feature. Sum value characteristics Where K is a hyperparameter, and and Parameter sharing;

[0067] (3) Each small region in query feature Q1 and its corresponding new key features Perform a dot product operation, then combine it with the new feature value. Performing the dot product operation yields a value of eigenvectors

[0068] (4) s 2 eigenvectors The combination yields an attention output cf of size wh×c. img1 (representing cf) 1_img1 cf 2_img1 or cf 3_img1 ).

[0069] The second branch operation is as follows:

[0070] (1) First, calculate the average value of each small region along the wh dimension for features Q2, K1, and V1 to obtain the average feature Q2. m K1 m and V1 m Then average feature Q2 m and K1 m Performing matrix multiplication yields a matrix of size s. 2×c is the adjacency matrix representing the similarity between Q2 and K1.

[0071] (2) For each small region in query feature Q2 In the adjacency matrix The indices of the top K most relevant small regions are recorded into a vector. In, then according to the vector The index in the key feature K1 and the value feature V1 are grouped together to obtain a new key feature. Sum value characteristics and Parameter sharing;

[0072] (3) Each small region in query feature Q2 and its corresponding new key features Perform a dot product operation, then combine it with the new feature value. Performing the dot product operation yields a value of eigenvectors

[0073] (4) s 2 eigenvectors The combination yields an attention output cf of size wh×c. img2 (representing cf) 1_img2 cf 2_img2 or cf 3_img2 ).

[0074] Step S4: In the time decoder branch, subgraphs F1 and F2 are used to generate a virtual video frame T0 with a size of N×W×H×C through data augmentation, where N is the length of the video frame. The default number of frames N is 8, which can be adjusted according to actual needs.

[0075] Step S41: The virtual video constructed from subgraph F1 to subgraph F2 consists of a total of N frames. Using the original images F1 and F2 and the frame index n, derive the nth (0≤n<N) frame f. n The calculation formula is as follows:

[0076] f n = (1-λ)F1+λF2

[0077]

[0078] Where λ is the weight. And F1 and F2 in the formula refer to taking all pixel values ​​in the image to participate in the calculation, that is, taking the pixel values ​​at corresponding positions in image F1 and F2 and adding them together with coefficients 1-λ and λ to obtain a new fused image of the original input size. One frame represents one image.

[0079] Step S42: Extract a single video frame f of shape W×H×C. n Expanding by one frame length dimension yields a tensor of shape 1×W×H×C. Concatenating N video frames along the frame length dimension results in a final video frame tensor T0 of shape N×W×H×C, which is the virtual video frame T0.

[0080] In this embodiment, a transition is made between an image before the change and another image after the change, with virtual video frames used to fill the gaps instead of real transition images, thus obtaining rich temporal information in a low-cost manner.

[0081] Step S5: Input the virtual video frame T0 into the 3D time encoder to obtain multi-scale time feature maps ct1, ct2, and ct3.

[0082] Step S51: In order to focus attention on the temporal dimension features, the video frame tensor T0 with shape N×W×H×C is first compressed through a bottleneck structure to obtain the temporal feature ct0. The bottleneck structure consists of a 3D convolution with a kernel size of 3×9×9, a 3×1×1 depthwise separable convolution, a BatchNorm layer and a ReLU activation function.

[0083] Step S52: Input the time feature ct0 into the first layer of the 3D-residual network (Conv3d) to obtain the time feature ct1. Input the time feature ct1 into the second layer of the 3D-residual network to obtain the time feature ct2. Input the time feature ct2 into the third layer of the 3D-residual network to obtain the time feature ct3. The structure of the 3D-residual network from input to output consists of a 3D convolutional layer with a kernel size of 1×1×1, a 3D convolutional layer with a kernel size of 3×3×3, and a 3D convolutional layer with a kernel size of 1×1×1. The output of the first 3D convolutional layer is added to the output of the last 3D convolutional layer through a skip connection to obtain the output of the 3D-residual network. Each convolutional layer is followed by a BatchNorm layer and a ReLU activation function, which will not be elaborated here.

[0084] In this embodiment, each 3D-residual network has the same structure. By adjusting the step size, temporal feature outputs with different spatial resolutions are obtained. Specifically, the step size of the first layer of the 3D-residual network is 1×1×1, and the step size of the second and third layers of the 3D-residual network is 2×2×2.

[0085] Step S6: In the feature decoder branch, the spatial interaction feature map cf 3_img1 cf 3_img2 The temporal feature map ct3 is input into the spatiotemporal feature fusion module to obtain the spatiotemporal fusion feature map df1, and the spatial interaction feature map cf is then input into the spatiotemporal feature fusion module.2_img1 cf 2_img2 The temporal feature map ct2 is input into the spatiotemporal feature fusion module to obtain the spatiotemporal fusion feature map df2, and the spatial interaction feature map cf is then input into the module. 1_img1 cf 1_img2 The spatiotemporal fusion feature map df3 is obtained by inputting the time feature map ct1 into the spatiotemporal feature fusion module.

[0086] Since the three spatiotemporal feature fusion modules operate identically, this embodiment sets the input of the spatiotemporal feature fusion module to the spatial interaction feature map cf. img1 (representing cf) 1_img1 cf 2_img1 or cf 3_img1 ) and cf img2 (representing cf) 1_img2 cf 2_img2 or cf 3_img2 ), and time feature maps ct (representing ct1, ct2, or ct3), such as Figure 3 As shown, perform the following operations:

[0087] (1) Spatial interaction feature map cf img1 and CF img2 The channel dimensions are concatenated (Concat), then passed through a 3×3 convolutional layer (Conv), a linear layer (Layer Normalization), and a ReLU activation function to finally obtain the feature vector O. s .

[0088] (2) The temporal feature map ct is simultaneously subjected to max pooling and average pooling along the video frame length dimension. The max pooling and average pooling results are then concatenated. Finally, through a linear layer (Layer Normalization) and ReLU activation function, a feature vector T of shape W×H×C is obtained. s .

[0089] (3) The eigenvector O s and eigenvector T s The spatiotemporal fusion feature map df (representing df3, df2 or df1) is obtained by cascading the channel dimensions.

[0090] Step S7: The multi-scale spatiotemporal fusion feature maps df1, df2, and df3 are passed through a residual network and progressive upsampling to obtain the final change detection prediction map G1 with a shape of W×H×2 for sub-images F1 and F2. This is a black and white image with the original input image size and the number of channels changed to 2. All sub-images are repeated to obtain the final change detection prediction map for the two remote sensing images.

[0091] In this embodiment, upsampling is performed using 2x bilinear interpolation. The spatiotemporal fusion feature map df1 is upsampled to the size of the spatiotemporal fusion feature map df2 after passing through a residual network. This upsampled map is then concatenated with df2 and passed through the residual network again, upsampled to the size of the spatiotemporal fusion feature map df3, and then concatenated with df3 before being passed through the residual network again to obtain the final change detection prediction map G1. Dividing the completed remote sensing image into multiple pairs of sub-images for separate operations not only reduces the computational load per operation but also allows for greater attention to image details, resulting in a more accurate change detection map.

[0092] This application encodes two identical remote sensing images to be detected in parallel using a dual-branch encoder employing both a spatial Transformer and a temporal CNN. To alleviate the insufficient semantic interaction between the two temporal images and the high computational resource requirements of traditional attention mechanisms, this application leverages the inherent sparsity of change and dynamically employs sparse selection with interactive attention to locate the semantically most relevant regions between the two temporal images, thereby guiding attention to the truly changing regions. To improve the blurry and inaccurate boundaries of changing regions, this application utilizes specific data augmentation methods to construct a video frame sequence between the two temporal images, and then uses a 3D-CNN to extract rich temporal dimension feature information, thereby improving the accuracy of changing region boundaries. Finally, the multi-scale spatiotemporal feature map is processed through a residual network and upsampling to obtain the final change prediction map. This application achieves accurate change detection of heterogeneous remote sensing images while improving operational efficiency and generalization performance.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for detecting changes in remote sensing images based on spatial selection and temporal evolution, characterized in that, The remote sensing image change detection method based on spatial selection and temporal evolution includes: Acquire two remote sensing images taken at the same location at different times, and divide the two remote sensing images into several sub-images of the same size. The sub-images corresponding to the two remote sensing images are denoted as F1 and F2, respectively. In the spatial decoder branch, subgraphs F1 and F2 are input into a weight-shared feature encoding network to extract features, and three multi-scale feature maps C are extracted from subgraphs F1 and F2 respectively. 1_img1 C 2_img1 C 3_img1 and three multi-scale feature maps C 1_img2 C 2_img2 C 3_img2 ; feature map C 1_img1 and C 1_img2 Feature map C 2_img1 and C 2_img2 Feature map C 3_img1 and C 3_img2 The inputs are fed in pairs into the dynamically selected interaction attention module to obtain the spatial interaction feature map cf. 1_img1 and CF 1_img2 Spatial interaction feature map cf 2_img1 and CF 2_img2 Spatial interaction feature map cf 3_img1 and CF 3_img2 ; In the temporal decoder branch, subgraphs F1 and F2 are used to generate a virtual video frame T0 through data augmentation; and the virtual video frame T0 is input into the 3D temporal encoder to obtain multi-scale temporal feature maps ct1, ct2, and ct3. In the feature decoder branch, the spatial interaction feature map (cf) is used. 3_img1 cf 3_img2 The temporal feature map ct3 is input into the spatiotemporal feature fusion module to obtain the spatiotemporal fusion feature map df1, and the spatial interaction feature map cf is then input into the spatiotemporal feature fusion module. 2_img1 cf 2_img2 The temporal feature map ct2 is input into the spatiotemporal feature fusion module to obtain the spatiotemporal fusion feature map df2, and the spatial interaction feature map cf is then input into the module. 1_img1 cf 1_img2 The spatiotemporal fusion feature map df3 is obtained by inputting the temporal feature map ct1 into the spatiotemporal feature fusion module; The multi-scale spatiotemporal fusion feature maps df1, df2, and df3 are passed through a residual network and progressive upsampling to obtain the final change detection prediction map G1 for sub-maps F1 and F2. This process is repeated for all sub-maps to obtain the final change detection prediction map for the two remote sensing images.

2. The remote sensing image change detection method based on spatial selection and temporal evolution according to claim 1, characterized in that, The subgraphs F1 and F2 are input into a weight-shared feature encoding network to extract features, and three multi-scale feature maps C are extracted from subgraphs F1 and F2 respectively. 1_img1 C 2_img1 C 3_img1 and three multi-scale feature maps C 1_img2 C 2_img2 C 3_img2 ,include: The weight-sharing feature encoding network uses the first three layers of a ResNet18 network, and feature maps C are extracted from the first layers of the two weight-sharing ResNet18 networks respectively. 1_img1 and C 1_img2 Feature maps C are extracted from the second layer of two ResNet18 networks with shared weights. 2_img1 and C 2_img2 Feature maps C are extracted from the third layer of two ResNet18 networks with shared weights. 3_img1 and C 3_img2 .

3. The remote sensing image change detection method based on spatial selection and temporal evolution according to claim 1, characterized in that, The feature map C 1_img1 and C 1_img2 Feature map C 2_img1 and C 2_img2 Feature map C 3_img1 and C 3_img2 The inputs are fed in pairs into the dynamically selected interaction attention module to obtain the spatial interaction feature map cf. 1_img1 and CF 1_img2 Spatial interaction feature map cf 2_img1 and CF 2_img2 Spatial interaction feature map cf 3_img1 and CF 3_img2 ,include: The dynamic selection interaction attention module consists of three parts, and the processing procedure of each dynamic selection interaction attention module is as follows: Let C be the feature map input to the dynamically selected interactive attention module. img1 and C img2 Its size is w×h×c, where w, h, and c are the feature map C, respectively. img1 and C img2 Width, height, and number of channels; Feature map C img1 and C img2 Through a single block division and projection operation, it is reshaped into a size of The feature vectors of feature map C img1 and C immg2 The reshaped feature vectors are represented as Q1, K1, V1 and Q2, K2, V2, respectively, where s 2 This represents the number of small regions in the block, and Q1 and Q2 represent the feature map C, respectively. img1 and C img2 The query features, where K1 and K2 represent feature maps C respectively. img1 and C img2 The key features, V1 and V2 represent feature maps C respectively. img1 and C img2 Value characteristics; Based on Q1, K1, V1 and Q2, K2, V2, the operation proceeds through two branches, with the first branch operation as follows: (1) First, calculate the average feature Q1 of each small region along the wh dimension by taking the features Q1, K2 and V2 respectively. m K2 m and V2 m Then average feature Q1 m and K2 m Performing matrix multiplication yields a matrix of size s. 2 ×c is the adjacency matrix representing the similarity between features Q1 and K2. (2) For each small region in query feature Q1 In the adjacency matrix The indices of the top K most relevant small regions are recorded into a vector. In, then according to the vector The index in the key feature K2 and the value feature V2 are grouped together to obtain a new key feature. Sum value characteristics Where K is the hyperparameter; (3) Each small region in query feature Q1 and its corresponding new key features Perform a dot product operation, then combine it with the new feature value. Performing the dot product operation yields a value of eigenvectors (4) s 2 eigenvectors The combination yields an attention output cf of size wh×c. img1 ; The second branch operation is as follows: (1) First, calculate the average value of each small region along the wh dimension for features Q2, K1, and V1 to obtain the average feature Q2. m K1 m and V1 m Then average feature Q2 m and K1 m Performing matrix multiplication yields a matrix of size s. 2 ×c is the adjacency matrix representing the similarity between Q2 and K1. (2) For each small region in query feature Q2 In the adjacency matrix The indices of the top K most relevant small regions are recorded into a vector. In, then according to the vector The index in the key feature K1 and the value feature V1 are grouped together to obtain a new key feature. Sum value characteristics (3) Each small region in query feature Q2 and its corresponding new key features Perform a dot product operation, then combine it with the new feature value. Performing the dot product operation yields a value of eigenvectors (4) s 2 eigenvectors The combination yields an attention output cf of size wh×c. img2 .

4. The remote sensing image change detection method based on spatial selection and temporal evolution according to claim 1, characterized in that, The step of generating a virtual video frame T0 from subgraphs F1 and F2 using data augmentation includes: The virtual video constructed from subgraph F1 to subgraph F2 consists of a total of N frames, where the nth frame f n The calculation formula is as follows: f n =(1-λ)F1+λF2 A single video frame with shape W×H×C is f n Extending the frame length dimension yields a tensor of shape 1×W×H×C. Concatenating N video frames along the frame length dimension results in a final video frame tensor of shape N×W×H×C, i.e., the virtual video frame T0, where 0≤n<N, and W, H, and C represent the values ​​of a single video frame f. n The width, height, and number of channels are given, with λ representing the weight.

5. The remote sensing image change detection method based on spatial selection and temporal evolution according to claim 4, characterized in that, The step of inputting the virtual video frame T0 into the 3D temporal encoder to obtain multi-scale temporal feature maps ct1, ct2, and ct3 includes: The virtual video frame T0 with shape N×W×H×C is first compressed through a bottleneck structure to obtain the temporal feature ct0. This bottleneck structure consists of a 3D convolution with a kernel size of 3×9×9, a depthwise separable convolution with a kernel size of 3×1×1, a BatchNorm layer, and a ReLU activation function. The time feature ct0 is input into the first layer of the 3D-residual network to obtain the time feature ct1. The time feature ct1 is input into the second layer of the 3D-residual network to obtain the time feature ct2. The time feature ct2 is input into the third layer of the 3D-residual network to obtain the time feature ct3. The structure of the 3D-residual network from input to output consists of a 3D convolutional layer with a kernel size of 1×1×1, a 3D convolutional layer with a kernel size of 3×3×3, and a 3D convolutional layer with a kernel size of 1×1×1. The output of the first 3D convolutional layer is added to the output of the last 3D convolutional layer through a skip connection to obtain the output of the 3D-residual network.

6. The remote sensing image change detection method based on spatial selection and temporal evolution according to claim 1, characterized in that, The spatiotemporal feature fusion module performs the following operations: The input to the spatiotemporal feature fusion module is the spatial interaction feature map (cf). img1 and CF img2 And time feature map ct; spatial interaction feature map cf img1 and CF img2 The channel dimensions are concatenated, and then passed through a 3×3 convolutional layer, a linear layer, and a ReLU activation function to finally obtain the feature vector O. s ; The temporal feature map ct is simultaneously subjected to max pooling and average pooling along the video frame length dimension. The max pooling and average pooling results are then concatenated and passed through a linear layer and a ReLU activation function to finally obtain the feature vector T. s ; The eigenvector O s and eigenvector T s The spatiotemporal fusion feature map df is obtained by cascading the channel dimensions.

7. The remote sensing image change detection method based on spatial selection and temporal evolution according to claim 1, characterized in that, The step of passing the multi-scale spatiotemporal fusion feature maps df1, df2, and df3 through a residual network and progressive upsampling to obtain the final change detection prediction map G1 for sub-maps F1 and F2 includes: The spatiotemporal fusion feature map df1 is upsampled to the size of the spatiotemporal fusion feature map df2 after passing through the residual network. It is then concatenated with the spatiotemporal fusion feature map df2 and passed through the residual network together. Then it is upsampled to the size of the spatiotemporal fusion feature map df3 and concatenated with the spatiotemporal fusion feature map df3 and passed through the residual network together to obtain the final change detection prediction map G1.

Images