Mask-based self-supervised multi-modal remote sensing image change detection method and system

Abstract

The present application relates to the technical field of remote sensing image processing and computer vision, in particular to a mask-based self-supervised multi-modal remote sensing image change detection method and system, wherein the method comprises the following steps: obtaining a first modal remote sensing image and a second modal remote sensing image with known image change detection true values; inputting the first modal remote sensing image and the second modal remote sensing image with known image change detection true values into a change detection network, training the change detection network, and obtaining a trained change detection network; obtaining a first modal remote sensing image and a second modal remote sensing image to be detected; inputting the first modal remote sensing image and the second modal remote sensing image to be detected into the trained change detection network, and obtaining a change detection result of the first and second modal remote sensing images. The present application can realize high-precision and low-complexity unsupervised change detection.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and computer vision technology, and in particular to a mask-based self-supervised multimodal remote sensing image change detection method and system. Background Technology

[0002] Multimodal remote sensing imagery, by integrating observational data from different sensors (such as optical, SAR, and multispectral), can overcome the information limitations of single-modal imaging, providing multidimensional and complementary information for dynamic monitoring of land cover. Optical imagery, with its rich spectral and textural features, is suitable for fine-grained classification of land cover and vegetation monitoring, but its observation is limited under cloudy and rainy weather conditions. SAR imagery has all-weather, all-time imaging capabilities and is sensitive to changes in land surface deformation and water content, but its interpretation is more challenging. The collaborative use of multimodal data can significantly improve the completeness of information in complex scenarios. With the rapid increase in the number of Earth observation satellites, multimodal remote sensing data now covers multi-scale spatial resolutions from meters to kilometers, providing a data foundation for major needs such as global change research, resource management, and national security.

[0003] Multimodal remote sensing change detection identifies dynamic changes in land cover by analyzing the differences between multi-temporal and multi-source remote sensing data. Its core advantage lies in overcoming the spatiotemporal constraints of single-modal data. Traditional single-modal methods are limited by long data acquisition cycles and cloud cover, making it difficult to meet the needs of high-frequency monitoring. Multimodal methods can compensate for the lack of data from a single sensor by flexibly utilizing heterogeneous data from different temporal phases. Furthermore, the differences in physical characteristics between multimodal data can reveal change information in different dimensions. For example, optical imagery captures changes in vegetation cover, while SAR data reflects changes in land surface structure. The synergy between the two can reduce missed and false detections and improve the interpretability of change detection.

[0004] However, existing multimodal change detection methods still face significant challenges. While feature transformation-based methods (such as modality mapping and domain adaptation) can map heterogeneous data to a common space using generative adversarial networks (GANs), they rely on a large number of paired samples from unchanged regions to construct pseudo-labels, and the transformation process is prone to introducing blur artifacts, leading to the loss of detailed information. Contrastive learning-based methods enhance feature discriminativity by constructing positive and negative sample pairs, but traditional strategies assume that features at the same location are necessarily similar, ignoring the possibility that the location may have changed in the real scene, resulting in a large number of erroneous negative samples interfering with model training. Furthermore, directly comparing features from the original multimodal data introduces noise due to domain shift. Existing methods often handle modality transformation and feature alignment independently, failing to achieve end-to-end joint optimization, and lack adaptive suppression mechanisms for changed regions, causing the reconstruction task to be affected by changed regions and resulting in insufficient feature space alignment. Therefore, how to leverage image reconstruction for modality transformation and perform unified mapping of multimodal features is a key challenge in current research. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a mask-based self-supervised multimodal remote sensing image change detection method and system. This method drives mode transformation and unified feature mapping through a self-supervised task, and combines dynamic masking to suppress interference in change regions, thereby achieving high-precision, low-complexity unsupervised change detection.

[0006] On the one hand, a mask-based self-supervised multimodal remote sensing image change detection method is provided, including: First, acquire a first-modality remote sensing image and a second-modality remote sensing image with known ground truth values ​​for image change detection. Input these two images into a change detection network for training. During training, the change detection network first extracts features from the image patches corresponding to the first and second modalities, obtaining first and second modal features. These first and second modal features are then further extracted to obtain first and second modal sub-features. The first and second modal sub-features are then weighted and fused to obtain fused features. Change detection is performed on the fused features to generate a change probability map. Based on the change probability map, the image patches corresponding to the first and second modalities are masked to obtain first and second modal image patch sets. The first and second modal image patch sets are then sequentially reconstructed using encoding, feature mapping, and reconstruction decoding to obtain predicted image change values. Based on the ground truth and predicted image change values, the total loss function of the network is calculated. The network is then trained based on the total loss function value. Acquire the first and second modal remote sensing images to be detected; input the first and second modal remote sensing images to be detected into the trained change detection network to obtain the change detection results of the first and second modal remote sensing images.

[0007] On the other hand, a mask-based self-supervised multimodal remote sensing image change detection system is provided, including: The training module is configured to: acquire a first-modality remote sensing image and a second-modality remote sensing image with known ground truth values ​​for image change detection; input the first-modality remote sensing image and the second-modality remote sensing image with known ground truth values ​​for image change detection into the change detection network to train the change detection network, resulting in a trained change detection network; during network training, the change detection network first extracts features from the image patches corresponding to the first and second modality remote sensing images respectively, obtaining first and second modality features, and then extracts features from the first and second modality features respectively, obtaining first and second modality sub-features; weightedly fuse the first and second modality sub-features to obtain fused features; perform change detection on the fused features to generate a change probability map; based on the change probability map, perform masking processing on the image patches corresponding to the first and second modality remote sensing images to obtain first and second modality image patch sets; perform reconstruction encoding, feature mapping, and reconstruction decoding processing on the first and second modality image patch sets respectively to obtain image change detection prediction values; calculate the total loss function value of the network based on the image change detection ground truth values ​​and image change detection prediction values, and complete the training of the network based on the total loss function value; The detection module is configured to: acquire a first modal remote sensing image and a second modal remote sensing image to be detected; input the first modal remote sensing image and the second modal remote sensing image to be detected into a trained change detection network to obtain change detection results of the first and second modal remote sensing images.

[0008] The above technical solution has the following advantages or beneficial effects: This method employs image reconstruction to perform modal transformation on remote sensing images and combines it with a dynamic change mask to eliminate interference from changed regions. It enhances modal invariance by uniformly mapping multimodal features through methods such as contrastive learning. A self-supervised approach is used to reduce the model's dependence on label data. This method effectively improves the adaptability of change detection to multimodal remote sensing images and enhances change detection accuracy, which is of great significance for fields such as land use monitoring, ecological environment assessment, and disaster detection.

[0009] By employing a dynamic masking mechanism to precisely suppress the interference of changed regions on image reconstruction, and combining cross-modal contrastive learning with unified feature mapping, the challenges of insufficient feature alignment and spurious negative sample interference in multimodal remote sensing data are effectively addressed. Self-supervised design reduces reliance on labeled data, and modality transformation and feature consistency constraints enhance the model's adaptability to heterogeneous data, thereby improving change detection accuracy. Attached Figure Description

[0010] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0011] Figure 1 This is a flowchart of the method in Example 1. Detailed Implementation

[0012] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0013] Example 1 This embodiment provides a mask-based self-supervised multimodal remote sensing image change detection method; like Figure 1 As shown, the mask-based self-supervised multimodal remote sensing image change detection method includes: S101: Obtain the first modal remote sensing image and the second modal remote sensing image of the known image change detection ground truth value; input the first modal remote sensing image and the second modal remote sensing image of the known image change detection ground truth value into the change detection network, train the change detection network, and obtain the trained change detection network; During network training, the change detection network first extracts features from image patches corresponding to the first and second modalities of remote sensing images, obtaining first and second modal features. These first and second modal features are then further extracted to obtain first and second modal sub-features. The first and second modal sub-features are then weighted and fused to obtain fused features. Change detection is performed on the fused features to generate a change probability map. Based on the change probability map, the image patches corresponding to the first and second modalities of remote sensing images are masked to obtain first and second modal image patch sets. The first and second modal image patch sets are then sequentially reconstructed, encoded, mapped, and decoded to obtain image change detection prediction values. Based on the actual and predicted image change detection values, the total loss function of the network is calculated, and the network training is completed based on the total loss function value. S102: Acquire the first modal remote sensing image and the second modal remote sensing image to be detected; input the first modal remote sensing image and the second modal remote sensing image to be detected into the trained change detection network to obtain the change detection results of the first and second modal remote sensing images.

[0014] The beneficial effects of the above technical solution are: training is carried out using a self-supervised method, which saves the manpower and material resources required for annotation; it supports change detection of multimodal remote sensing images, reducing the application limitations of change detection methods.

[0015] Further, in S101: acquire a first modal remote sensing image and a second modal remote sensing image of known image change detection true value; wherein, the first modal remote sensing image refers to an optical image, and the second modal remote sensing image is a SAR image, wherein the acquisition time of the first modal remote sensing image is earlier than the acquisition time of the second modal remote sensing image.

[0016] Further, the step of inputting the first modality remote sensing image and the second modality remote sensing image, which contain known ground truth values ​​for image change detection, into the change detection network to train the change detection network and obtain the trained change detection network specifically includes: Construct a dataset consisting of several image pairs containing known image change detection ground truth values. Each image pair includes: a first modality remote sensing image and a second modality remote sensing image. The dataset is divided into a training set and a test set; The training set is input into the change detection network to train it. When the total loss function value of the network no longer decreases, the training is stopped, and the trained change detection network is obtained. The test set is input into the trained change detection network for testing. If the test results all meet the preset test index values, the training is considered complete and the currently trained change detection network is the final trained change detection network; otherwise, the training set is changed and retraining is performed.

[0017] The beneficial effects of the above technical solution are: by fusing remote sensing images of the first and second modalities, the complementary features of different modal data can be utilized to significantly improve the adaptability of the change detection model to complex scenes.

[0018] It should be understood that the collection of multi-temporal and multi-modal remote sensing images (such as optical images and SAR images) achieves sub-pixel-level registration (registration error RMSE < 0.5 pixels) through SIFT feature matching or phase correlation algorithms, and bilinear interpolation is used to correct misaligned areas. Data augmentation includes random cropping (256×256), rotation (0°–90°), and flipping. Simultaneously, histogram matching is performed on optical images, Lee filtering is applied to SAR images for denoising, and all images are uniformly normalized to the [-1, 1] interval. The dataset is divided into training and testing sets in a 7:3 ratio to ensure a balanced distribution of ground cover categories.

[0019] Furthermore, the change detection network has the following network structure: The feature extractor, cross-modal attention layer, change detector, mask adder, and classifier are connected in sequence. The feature extractor includes: a first branch and a second branch in parallel; The first branch includes: a first feature extraction module and a second feature extraction module connected in series; The second branch includes: a third feature extraction module and a fourth feature extraction module connected in series; The first feature extraction module, the second feature extraction module, the third feature extraction module and the fourth feature extraction module have the same internal structure. The first feature extraction module includes four TransformerBlock units connected in sequence.

[0020] The Transformer Block unit includes: a multi-head attention mechanism layer, a first normalization layer, a feedforward neural network (FFN) layer, and a second normalization layer connected in sequence; wherein, the input of the first normalization layer is residually connected to the input of the multi-head attention mechanism layer; the input of the second normalization layer is residually connected to the input of the feedforward neural network (FFN) layer; the input of the multi-head attention mechanism layer is the input of the Transformer Block unit, and the output of the second normalization layer is the output of the Transformer Block unit.

[0021] Furthermore, the change detection network first extracts features from the image patches corresponding to the first and second modality remote sensing images, respectively, to obtain first and second modality features. Then, it further extracts features from the first and second modality features to obtain first and second modality sub-features, including: The first modality remote sensing image is cropped to obtain several image patches corresponding to the first modality remote sensing image; The second modality remote sensing image is cropped to obtain several image patches corresponding to the second modality remote sensing image; The first feature extraction module is used to extract features from the image patches corresponding to the first modality remote sensing image to obtain the first modality features; The third feature extraction module is used to extract features from the image patches corresponding to the second modality remote sensing image to obtain the second modality features; The second feature extraction module is used to extract features from the first modality features to obtain the first modality sub-features; The fourth feature extraction module is used to extract features from the second modality features to obtain the second modality sub-features.

[0022] The beneficial effects of the above technical solution are: the independent first and third feature extraction modules are optimized for different modal physical characteristics, which can improve detection accuracy; cropping the remote sensing image before extracting features can greatly reduce memory consumption. The first and third feature extraction modules share parameters with each other.

[0023] For example, in the feature extraction stage, a ViT-based encoder architecture is used to process optical and SAR multimodal image pairs. First, the input image is segmented into 16×16 local blocks, and each image block is mapped to a 384-dimensional embedding vector through linear projection:

[0024] in, and These represent optical images from the first period and SAR images from the second period, respectively. The first four Transformer Block layers of the encoder employ a parameter-sharing strategy to extract common features (such as edges and textures) from the two modes, while the last four layers independently process different modes and learn mode-specific features (such as spectral information in optics and scattering characteristics in SAR).

[0025] Furthermore, the weighted fusion of the first and second modality sub-features to obtain the fused features specifically includes: A cross-modal attention layer is used to weight and fuse the sub-features of the first and second modalities to obtain fused features.

[0026] Furthermore, the weighted fusion of the first and second modality sub-features using a cross-modal attention layer to obtain fused features specifically includes: First modality sub-features as query vector The second modality sub-feature as a key vector Calculate the cross-attention weight matrix :

[0027] in, ; This represents the dimension of the input vector.

[0028] The weight matrix is ​​applied to the concatenated first modality sub-features. Second mode sub-features Weighted fusion is performed to obtain fusion features. :

[0029] in, This indicates a splicing operation along the channel dimension.

[0030] It should be understood that the beneficial effects of the above technical solution are: this process enhances the response of spatiotemporally consistent regions through an attention mechanism, suppressing spurious changes in features caused by sensor differences or noise. The final output is a fused feature. This serves as the basis for subsequent change detection.

[0031] Furthermore, the step of performing change detection on the fused features and generating a change probability map includes: A change detector is used to detect changes in the fused features and generate a change probability map.

[0032] Furthermore, the step of detecting changes in the fused features using a change detector to generate a change probability map includes: First, a 3×3 convolutional layer is used. Compress the channel dimension to 128 dimensions, combined with batch normalization. The first enhanced feature is obtained by enhancing the nonlinear expression of the feature with the ReLU activation function. : ; Next, the first enhancement feature The feature map is then upsampled using a channel attention module to highlight key channel information; subsequently, bilinear interpolation is used to upsample the feature map. The spatial resolution is doubled, then 3×3 convolutions are used to further refine edge features, and finally 1×1 convolutions are used to compress the number of channels to 1. A change probability map is then generated using the Sigmoid function. The overall process can be formally represented as follows: .

[0033] It should be understood that the channel attention module dynamically learns the importance weights of each channel through global average pooling, fully connected layers, and nonlinear activation. Finally, the weights are multiplied with the original feature map channel by channel to highlight key channels and suppress redundant information, thereby adaptively improving the model's ability to focus on task-related features.

[0034] Furthermore, the process of masking the image blocks corresponding to the first and second modal remote sensing images based on the change probability map to obtain the first and second modal image block sets is achieved through a mask adder.

[0035] Further, based on the change probability map, the image patches corresponding to the first and second modal remote sensing images are masked to obtain sets of first and second modal image patches, wherein the masking process includes: The value of each pixel in the probability map is compared with a set threshold 𝑡. If the current pixel value is greater than the set threshold 𝑡, the current pixel value in the probability map is set to 1; otherwise, it is set to 0. If the current pixel value is 1, it means that the image block corresponding to the current pixel contains a changed pixel, so set all pixel values ​​in the image block corresponding to the current pixel to zero; If the current pixel value is 0, it means that the image block corresponding to the current pixel does not contain any changed pixels, and all pixel values ​​in the image block corresponding to the current pixel will remain unchanged.

[0036] Furthermore, the image patches corresponding to the first modality remote sensing image are masked to obtain the first modality image patch set; the image patches corresponding to the second modality remote sensing image are masked to obtain the second modality image patch set.

[0037] It should be understood that, in obtaining the probability diagram of change Then, the probability map is first binarized into a mask of the changing regions using a global threshold λ (usually set based on validation set optimization or empirical values). Then, a 16×16 image block is taken from the previous division. If the image block contains changing pixels, a mask is applied to the image block and all its corresponding modal regions: all pixel values ​​in the image block are set to 0. Image blocks that do not contain changing pixels remain unchanged, ultimately generating a set of multimodal image blocks with dynamic masks.

[0038] Furthermore, the first and second modality image block sets are sequentially reconstructed, encoded, mapped, and decoded to obtain image change detection prediction values, which are then implemented using a classifier.

[0039] Further, the first and second modality image block sets are sequentially reconstructed through encoding, feature mapping, and reconstruction through decoding to obtain image change detection prediction values, including: For the first modality image block set, the first reconstruction encoder is used for encoding processing to obtain the first encoding result. The first encoding result is input into the feature mapper for mapping processing to obtain the first mapping result. The first encoding result is input into the first reconstruction decoder for decoding processing to obtain the first decoding result. For the second modality image block set, the second reconstruction encoder is used for encoding processing to obtain the second encoding result. The second encoding result is input into the feature mapper for mapping processing to obtain the second mapping result. The second encoding result is input into the first reconstruction decoder for decoding processing to obtain the second decoding result. Based on the first mapping result and the second mapping result, the value of the first loss function is calculated; Based on the first decoding result and the second decoding result, the value of the second loss function is calculated; The total loss function value is obtained by summing the first loss function value and the second loss function value.

[0040] Furthermore, the internal structure of the first reconstruction encoder and the second reconstruction encoder is the same. The first reconstruction encoder includes N Transformer Block units connected in sequence.

[0041] Furthermore, for the first Layer Transformer Block unit:

[0042] in, The input vector. This indicates that the focus is on the bulls.

[0043] Understandably, feature learning is performed using a mask-based autoencoder based on a set of multimodal image patches after dynamic masking. First, each local patch is flattened into a vector and mapped to the latent space through a learnable linear mapping layer. The encoder consists of multiple stacked standard Transformer Blocks, each containing a multi-head self-attention and feedforward network. Since the input has had its variable regions suppressed by the dynamic mask, the features extracted by the encoder are representations of the non-variable regions.

[0044] Furthermore, the feature mapper includes four feature mapping layers connected in sequence, each feature mapping layer including a multi-head attention mechanism layer and a gated feedforward neural network layer connected in sequence.

[0045] The feature mapper captures a unified feature representation of multimodal features, while the special modules refine the subtle differences within each modality. It consists of four layers, each containing a multi-head self-attention mechanism and a gated feedforward network.

[0046]

[0047] in For activation function, This is element-wise multiplication. For weight parameters, .

[0048] Furthermore, the first loss function is:

[0049] in, For the set of positive sample pairs, and For the set of negative sample pairs, Indicates the corresponding negative sample. For cosine similarity, Temperature parameter. Modality sensing weights. Cross-modal alignment capability is enhanced by dynamically adjusting the KL (Kullback-Leibler) divergence of the inter-modal feature distribution. Positive samples represent pixels at the same coordinate position in two remote sensing images, while negative samples represent pixels at different coordinate positions in two remote sensing images. The temperature parameter, used to adjust the distribution pattern of similarity scores and control the model's attention to positive and negative samples, is set to 0.1.

[0050] To obtain a unified feature representation of multimodal images, the extracted bi-temporal features are fed into a contrastive learning sub-network. Cross-modal positive sample pairs are defined as bi-temporal features from the same spatial location, while negative sample pairs are random combinations from different locations or different modalities. Traditional contrastive learning uses InfoNCE loss, but it does not consider the heterogeneity of multimodal data.

[0051] Furthermore, the reconstruction decoder includes four Transformer Block units connected in sequence.

[0052] Based on the multimodal features extracted by the reconstruction encoder, a cross-modal reconstruction decoder is designed to generate a reconstructed image of the target modality. By calculating the loss function with the original target modality image, the model's ability to model cross-modal consistency and differences is further enhanced. Similar to the reconstruction encoder structure, the decoder adopts a hierarchical Transformer structure, consisting of four stacked Transformer Blocks, each containing cross-modal attention and a feedforward network. The decoder ultimately outputs the feature matrix H of the target modality, which is mapped back to pixel space through a projection layer. All local blocks are then reconstructed into a complete image according to their spatial locations.

[0053] Furthermore, the second loss function For: Calculate pixel-level L1 loss;

[0054] in, For the reconstructed image, The image is the original image, where H represents the height of the feature and W represents the width of the feature. The number of channels representing the feature. This represents the corresponding coordinate value.

[0055] Furthermore, based on the ground truth and predicted values ​​of image change detection, the total loss function of the network is calculated, and the network is trained based on the total loss function. The total loss function... Its formula expression is: .

[0056] Further, in step S102: acquire the first modality remote sensing image and the second modality remote sensing image to be detected; input the first modality remote sensing image and the second modality remote sensing image to be detected into the trained change detection network to obtain the change detection results of the first and second modality remote sensing images; wherein, the change detection network used in the current step only includes: The third and fourth branches; The third branch includes: a first reconstruction encoder and a first feature mapper connected in sequence; The fourth branch includes: a second reconstruction encoder and a second feature mapper connected in sequence; The input of the first reconstruction encoder is used to input image patches of the first modality remote sensing image to be detected; the image patches of the first modality remote sensing image to be detected are obtained by segmenting the first modality remote sensing image to be detected; through linear mapping, the image patches of the first modality remote sensing image are mapped into a first embedding vector of several dimensions; the first embedding vector is input into the first reconstruction encoder to obtain the first reconstruction feature; the first reconstruction feature is input into the first feature mapper to obtain the first feature representation; The input of the second reconstruction encoder is used to input image patches of the second modality remote sensing image to be detected; the image patches of the second modality remote sensing image to be detected are obtained by segmenting the second modality remote sensing image to be detected; through linear mapping, the image patches of the second modality remote sensing image are mapped into several-dimensional second embedding vectors; the second embedding vectors are input into the second reconstruction encoder to obtain the second reconstruction features; the second reconstruction features are input into the second feature mapper to obtain the second feature representation; The first feature representation is subtracted from the second feature representation pixel by pixel to obtain a change difference map. An activation function is applied to the change difference map to obtain a change probability map to be processed. The pixel value of each pixel in the change probability map to be processed is compared with a set threshold t. If it is greater than the set threshold t, the pixel value of the current pixel is set to 1; otherwise, the pixel value of the current pixel is set to 0, resulting in a binary change map. Thresholding segmentation, closing operation, and connected component analysis are performed on the binary change map in sequence to obtain the change detection results of the first and second modal remote sensing images.

[0057] It should be understood that in practical use, only the reconstruction encoder and feature mapper are retained. The input image is segmented into 16×16 image blocks, and each image block is mapped to a 384-dimensional embedding vector through linear projection. Then, without masking, the vectors are directly fed into the reconstruction encoder to obtain bi-temporal reconstruction features. The bi-temporal reconstruction features are then fed into the feature mapper to generate a unified feature representation. After pixel-by-pixel subtraction, a change difference map is obtained, which is then processed by the Sigmoid function to generate a change probability map, and the threshold from step 4 is taken. After filtering, a binary transformation image is obtained. Post-processing uses the Otsu algorithm for adaptive threshold segmentation (default 0.6), a closing operation (kernel size 5×5) to fill holes, and connected component analysis to filter out noise with an area less than 50 pixels.

[0058] Example 2 This embodiment provides a mask-based self-supervised multimodal remote sensing image change detection system, including: The training module is configured to: acquire a first-modality remote sensing image and a second-modality remote sensing image with known ground truth values ​​for image change detection; input the first-modality remote sensing image and the second-modality remote sensing image with known ground truth values ​​for image change detection into the change detection network to train the change detection network, resulting in a trained change detection network; during network training, the change detection network first extracts features from the image patches corresponding to the first and second modality remote sensing images respectively, obtaining first and second modality features, and then extracts features from the first and second modality features respectively, obtaining first and second modality sub-features; weightedly fuse the first and second modality sub-features to obtain fused features; perform change detection on the fused features to generate a change probability map; based on the change probability map, perform masking processing on the image patches corresponding to the first and second modality remote sensing images to obtain first and second modality image patch sets; perform reconstruction encoding, feature mapping, and reconstruction decoding processing on the first and second modality image patch sets respectively to obtain image change detection prediction values; calculate the total loss function value of the network based on the image change detection ground truth values ​​and image change detection prediction values, and complete the training of the network based on the total loss function value; The detection module is configured to: acquire a first modal remote sensing image and a second modal remote sensing image to be detected; input the first modal remote sensing image and the second modal remote sensing image to be detected into a trained change detection network to obtain change detection results of the first and second modal remote sensing images.

[0059] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A mask-based self-supervised multimodal remote sensing image change detection method, characterized in that, include: Acquire the first and second modal remote sensing images of known image change detection ground truth values; The first and second modal remote sensing images, which are known to be the true values ​​of image change detection, are input into the change detection network to train the change detection network and obtain the trained change detection network. During network training, the change detection network first extracts features from the image patches corresponding to the first and second modal remote sensing images, respectively, to obtain the first and second modal features. Then, it extracts features from the first and second modal features again to obtain the first and second modal sub-features. Finally, it performs weighted fusion of the first and second modal sub-features to obtain the fused features. Change detection is performed on the fused features to generate a change probability map. Based on the change probability map, image patches corresponding to the first and second modal remote sensing images are masked to obtain first and second modal image patch sets. The first and second modal image patch sets are then subjected to reconstruction encoding, feature mapping, and reconstruction decoding processes in sequence to obtain image change detection prediction values. Based on the actual and predicted image change detection values, the total loss function value of the network is calculated. Based on the total loss function value, the network is trained. Acquire the first and second modal remote sensing images to be detected; input the first and second modal remote sensing images to be detected into the trained change detection network to obtain the change detection results of the first and second modal remote sensing images.

2. The mask-based self-supervised multimodal remote sensing image change detection method as described in claim 1, characterized in that, The change detection network has the following network structure: The feature extractor, cross-modal attention layer, change detector, mask adder, and classifier are connected in sequence. The feature extractor includes: a first branch and a second branch in parallel; The first branch includes: a first feature extraction module and a second feature extraction module connected in series; The second branch includes: a third feature extraction module and a fourth feature extraction module connected in series; The first feature extraction module, the second feature extraction module, the third feature extraction module, and the fourth feature extraction module all have the same internal structure. The first feature extraction module includes four Transformer Block units connected in sequence. The Transformer Block unit includes: a multi-head attention mechanism layer, a first normalization layer, a feedforward neural network (FFN) layer, and a second normalization layer connected in sequence; wherein, the input of the first normalization layer is residually connected to the input of the multi-head attention mechanism layer; the input of the second normalization layer is residually connected to the input of the feedforward neural network (FFN) layer; the input of the multi-head attention mechanism layer is the input of the Transformer Block unit, and the output of the second normalization layer is the output of the Transformer Block unit.

3. The mask-based self-supervised multimodal remote sensing image change detection method as described in claim 1, characterized in that, The change detection network first extracts features from the image patches corresponding to the first and second modality remote sensing images, respectively, to obtain first and second modality features. Then, it further extracts features from the first and second modality features to obtain first and second modality sub-features, including: The first modality remote sensing image is cropped to obtain several image patches corresponding to the first modality remote sensing image; The second modality remote sensing image is cropped to obtain several image patches corresponding to the second modality remote sensing image; The first feature extraction module is used to extract features from the image patches corresponding to the first modality remote sensing image to obtain the first modality features; The third feature extraction module is used to extract features from the image patches corresponding to the second modality remote sensing image to obtain the second modality features; The second feature extraction module is used to extract features from the first modality features to obtain the first modality sub-features; The fourth feature extraction module is used to extract features from the second modality features to obtain the second modality sub-features.

4. The mask-based self-supervised multimodal remote sensing image change detection method as described in claim 1, characterized in that, The weighted fusion of the first and second modal sub-features to obtain fused features specifically includes: weighted fusion of the first and second modal sub-features using a cross-modal attention layer to obtain fused features; the weighted fusion of the first and second modal sub-features using a cross-modal attention layer to obtain fused features specifically includes: First modality sub-features as query vector The second modality sub-feature as a key vector Calculate the cross-attention weight matrix : ； in, ; Represents the dimension of the input vector; The weight matrix is ​​applied to the concatenated first modality sub-features. Second mode sub-features Weighted fusion is performed to obtain fusion features. : ； in, This indicates a splicing operation along the channel dimension.

5. The mask-based self-supervised multimodal remote sensing image change detection method as described in claim 4, characterized in that, The step of detecting changes in the fused features and generating a change probability map includes: detecting changes in the fused features using a change detector and generating a change probability map; the step of detecting changes in the fused features using a change detector and generating a change probability map includes: First, a 3×3 convolutional layer is used. Compress the channel dimension to 128 dimensions, combined with batch normalization. The first enhanced feature is obtained by enhancing the nonlinear expression of the feature with the ReLU activation function. : ； Next, the first enhancement feature The feature map is then upsampled using a channel attention module to highlight key channel information; subsequently, bilinear interpolation is used to upsample the feature map. The spatial resolution is doubled, then 3×3 convolutions are used to further refine edge features, and finally 1×1 convolutions are used to compress the number of channels to 1. A change probability map is then generated using the Sigmoid function. The overall process can be formally represented as follows: 。 6. The mask-based self-supervised multimodal remote sensing image change detection method as described in claim 1, characterized in that, Based on the change probability map, the image patches corresponding to the first and second modal remote sensing images are masked to obtain sets of first and second modal image patches. The masking process includes: Compare the value of each pixel in the probability map with a set threshold. A comparison is made; if the current pixel value is greater than a set threshold... If the value is positive, the current pixel value in the probability map is set to 1; otherwise, it is set to 0. If the current pixel value is 1, it means that the image block corresponding to the current pixel contains a changed pixel, so set all pixel values ​​in the image block corresponding to the current pixel to zero; If the current pixel value is 0, it means that the image block corresponding to the current pixel does not contain any changed pixels, so all pixel values ​​in the image block corresponding to the current pixel will not be changed. The image patches corresponding to the first modality remote sensing image are masked to obtain the first modality image patch set; the image patches corresponding to the second modality remote sensing image are masked to obtain the second modality image patch set.

7. The mask-based self-supervised multimodal remote sensing image change detection method as described in claim 1, characterized in that, For the first and second modality image patch sets, reconstruction encoding, feature mapping, and reconstruction decoding are performed sequentially to obtain image change detection prediction values, including: For the first modality image block set, the first reconstruction encoder is used for encoding processing to obtain the first encoding result. The first encoding result is input into the feature mapper for mapping processing to obtain the first mapping result. The first encoding result is input into the first reconstruction decoder for decoding processing to obtain the first decoding result. For the second modality image block set, the second reconstruction encoder is used for encoding processing to obtain the second encoding result. The second encoding result is input into the feature mapper for mapping processing to obtain the second mapping result. The second encoding result is input into the first reconstruction decoder for decoding processing to obtain the second decoding result. Based on the first mapping result and the second mapping result, the value of the first loss function is calculated; Based on the first decoding result and the second decoding result, the value of the second loss function is calculated; The total loss function value is obtained by summing the first loss function value and the second loss function value.

8. The mask-based self-supervised multimodal remote sensing image change detection method as described in claim 7, characterized in that, The first loss function is: ； in, For the set of positive sample pairs, and For the set of negative sample pairs, Indicates the corresponding negative sample. For cosine similarity, Temperature parameter; modal sensing weights Enhance cross-modal alignment capability by dynamically adjusting the KL divergence of intermodal feature distributions; Second loss function For: Calculate pixel-level L1 loss; ； in, For the reconstructed image, The image is the original image, where H represents the height of the feature and W represents the width of the feature. The number of channels representing the feature. This represents the corresponding coordinate value.

9. A mask-based self-supervised multimodal remote sensing image change detection system, characterized in that, include: The training module is configured to: acquire a first-modality remote sensing image and a second-modality remote sensing image of known image change detection ground truth values; The first and second modal remote sensing images, which are known to be the true values ​​of image change detection, are input into the change detection network to train the change detection network and obtain the trained change detection network. During network training, the change detection network first extracts features from the image patches corresponding to the first and second modal remote sensing images, respectively, to obtain the first and second modal features. Then, it extracts features from the first and second modal features again to obtain the first and second modal sub-features. Finally, it performs weighted fusion of the first and second modal sub-features to obtain the fused features. Change detection is performed on the fused features to generate a change probability map. Based on the change probability map, image patches corresponding to the first and second modal remote sensing images are masked to obtain first and second modal image patch sets. The first and second modal image patch sets are then subjected to reconstruction encoding, feature mapping, and reconstruction decoding processes in sequence to obtain image change detection prediction values. Based on the actual and predicted image change detection values, the total loss function value of the network is calculated. Based on the total loss function value, the network is trained. The detection module is configured to: acquire a first modal remote sensing image and a second modal remote sensing image to be detected; input the first modal remote sensing image and the second modal remote sensing image to be detected into a trained change detection network to obtain change detection results of the first and second modal remote sensing images.

Images