Mask-based self-supervised multi-modal remote sensing image change detection method and system

Abstract

The invention relates to the technical field of remote sensing image processing and computer vision, in particular to a mask-based self-supervised multi-modal remote sensing image change detection method and system, and the method comprises the steps: obtaining a first modal remote sensing image and a second modal remote sensing image with known image change detection true values; inputting the first modal remote sensing image and the second modal remote sensing image with known image change detection true values into a change detection network, and training the change detection network to obtain a trained change detection network; obtaining a to-be-detected first modal remote sensing image and a to-be-detected second modal remote sensing image; and inputting a to-be-detected first modal remote sensing image and a to-be-detected second modal remote sensing image into the trained change detection network to obtain change detection results of the first modal remote sensing image and the second modal remote sensing image. According to the invention, high-precision and low-complexity unsupervised change detection can be realized.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing and computer vision technology, and in particular to a mask-based self-supervised multimodal remote sensing image change detection method and system. Background Art

[0002] Multimodal remote sensing images, by integrating observation data from different sensors (such as optical, SAR, and multispectral), can overcome the information limitations of a single modality and provide multi-dimensional complementary information for dynamic monitoring of surface cover. Optical images, with their rich spectral and texture features, are suitable for fine classification of land features and vegetation monitoring, but observation is limited in cloudy and rainy weather. SAR images have all-day, all-weather imaging capabilities and are sensitive to surface deformation and changes in water content, but are difficult to interpret. The coordinated use of multimodal data can significantly improve information completeness in complex scenarios. With the rapid growth in the number of Earth observation satellites, multimodal remote sensing data has covered multi-scale spatial resolutions from meters to kilometers, providing a data foundation for major needs such as global change research, resource management, and national security.

[0003] Multimodal remote sensing change detection identifies dynamic changes in surface cover by analyzing differences in multi-temporal and multi-source remote sensing data. Its core advantage lies in breaking through the spatiotemporal constraints of single-modal data. Traditional single-modal methods are limited by long data acquisition cycles and cloud cover, making it difficult to meet high-frequency monitoring needs. Multimodal methods can compensate for the lack of single-sensor data by flexibly calling heterogeneous data from different temporal phases. In addition, the differences in physical properties between multimodal data can reveal change information in different dimensions. For example, optical images capture changes in vegetation cover, while SAR data reflects changes in surface structure. The synergy between the two can reduce missed detections and false detections, improving the interpretability of change detection.

[0004] However, existing multimodal change detection methods still face significant challenges. While feature transformation-based methods (such as modality mapping and domain adaptation) can map heterogeneous data into a common space using generative adversarial networks (GANs), they rely on a large number of paired samples from unchanged regions to construct pseudo-labels. Furthermore, the transformation process is prone to introducing blurring artifacts, leading to loss of detailed information. Contrastive learning-based methods enhance feature discriminability by constructing positive and negative sample pairs. However, traditional strategies assume that features at the same location are necessarily similar, ignoring the possibility that such locations may have changed in real scenes. This leads to a large number of erroneous negative samples that interfere with model training. Furthermore, directly performing feature comparison on raw multimodal data introduces noise due to domain shift. Existing methods often independently address modality transformation and feature alignment, failing to achieve end-to-end joint optimization. Furthermore, they lack adaptive suppression mechanisms for changed regions, resulting in reconstruction being affected by these regions and insufficient feature space alignment. Therefore, leveraging image reconstruction to perform modality transformation and uniformly map multimodal features remains a key challenge in current research. Summary of the Invention

[0005] In order to address the shortcomings of the existing technology, the present invention provides a mask-based self-supervised multimodal remote sensing image change detection method and system; this method drives modal conversion and feature unified mapping through self-supervised tasks, combines dynamic masks to suppress interference in changing areas, and realizes high-precision, low-complexity unsupervised change detection.

[0006] On the one hand, a mask-based self-supervised multimodal remote sensing image change detection method is provided, including: Obtain a first modality remote sensing image and a second modality remote sensing image with known true values for image change detection; input the first modality remote sensing image and the second modality remote sensing image with known true values for image change detection into a change detection network, train the change detection network, and obtain a trained change detection network; during the network training process, the change detection network first performs feature extraction on the image blocks corresponding to the first and second modality remote sensing images respectively to obtain first and second modality features, and then performs feature extraction on the first and second modality features respectively to obtain first and second modality sub-features; weighted fusion is performed on the first and second modality sub-features to obtain fused features; change detection is performed on the fused features to generate a change probability map; based on the change probability map, mask processing is performed on the image blocks corresponding to the first and second modality remote sensing images to obtain a first and second modality image block set; reconstruction coding, feature mapping, and reconstruction decoding are performed on the first and second modality image block sets in sequence to obtain an image change detection prediction value; based on the image change detection true value and the image change detection prediction value, a total loss function value of the network is calculated, and based on the total loss function value, the network training is completed; A first modality remote sensing image and a second modality remote sensing image to be detected are obtained; the first modality remote sensing image and the second modality remote sensing image to be detected are input into a trained change detection network to obtain change detection results of the first and second modality remote sensing images.

[0007] On the other hand, a mask-based self-supervised multimodal remote sensing image change detection system is provided, including: The training module is configured to: obtain a first modality remote sensing image and a second modality remote sensing image with known true values for image change detection; input the first modality remote sensing image and the second modality remote sensing image with known true values for image change detection into a change detection network, train the change detection network, and obtain a trained change detection network; during the network training process, the change detection network first performs feature extraction on the image blocks corresponding to the first and second modality remote sensing images respectively to obtain first and second modality features, and then performs feature extraction on the first and second modality features respectively to obtain first and second modality sub-features; perform weighted fusion on the first and second modality sub-features to obtain fused features; perform change detection on the fused features to generate a change probability map; based on the change probability map, perform masking on the image blocks corresponding to the first and second modality remote sensing images to obtain first and second modality image block sets; perform reconstruction coding, feature mapping, and reconstruction decoding on the first and second modality image block sets in sequence to obtain image change detection prediction values; calculate the total loss function value of the network based on the true value of image change detection and the image change detection prediction value, and complete the training of the network based on the total loss function value; The detection module is configured to: obtain a first modality remote sensing image and a second modality remote sensing image to be detected; input the first modality remote sensing image and the second modality remote sensing image to be detected into a trained change detection network to obtain change detection results of the first and second modality remote sensing images.

[0008] The above technical solution has the following advantages or beneficial effects: This method uses image reconstruction to transform the modality of remote sensing images and combines it with a dynamic change mask to eliminate interference in changing regions. It also uses contrastive learning and other methods to uniformly map multimodal features, enhancing modal invariance. Self-supervision is employed to reduce the model's reliance on labeled data. This method effectively improves the adaptability of change detection for multimodal remote sensing imagery and enhances change detection accuracy, offering significant implications for land use monitoring, ecological and environmental assessment, and disaster detection.

[0009] A dynamic masking mechanism precisely suppresses interference from changing regions on image reconstruction. Combined with cross-modal contrastive learning and unified feature mapping, this approach effectively addresses the challenges of insufficient feature alignment and pseudo-negative interference in multimodal remote sensing data. The self-supervised design reduces reliance on labeled data and leverages modality conversion and feature consistency constraints to enhance the model's adaptability to heterogeneous data, improving change detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0011] Figure 1 This is a flow chart of the method of embodiment 1. DETAILED DESCRIPTION

[0012] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0013] Example 1 This embodiment provides a mask-based self-supervised multimodal remote sensing image change detection method; like Figure 1 As shown in FIG, a mask-based self-supervised multimodal remote sensing image change detection method includes: S101: Acquire a first modality remote sensing image and a second modality remote sensing image for which true values of image change detection are known; input the first modality remote sensing image and the second modality remote sensing image for which true values of image change detection are known into a change detection network, train the change detection network, and obtain a trained change detection network; During the network training process, the change detection network first extracts features from the image blocks corresponding to the first and second modal remote sensing images respectively to obtain the first and second modal features, and then extracts features from the first and second modal features again to obtain the first and second modal sub-features; the first and second modal sub-features are weightedly fused to obtain fused features; change detection is performed on the fused features to generate a change probability map; based on the change probability map, mask processing is performed on the image blocks corresponding to the first and second modal remote sensing images to obtain the first and second modal image block sets; the first and second modal image block sets are respectively subjected to reconstruction coding, feature mapping and reconstruction decoding processing to obtain the image change detection prediction value; based on the image change detection true value and the image change detection prediction value, the total loss function value of the network is calculated, and the network training is completed based on the total loss function value; S102: Acquire a first modality remote sensing image and a second modality remote sensing image to be detected; input the first modality remote sensing image and the second modality remote sensing image to be detected into a trained change detection network to obtain change detection results of the first and second modality remote sensing images.

[0014] The beneficial effects of the above technical solution are: using self-supervision for training, saving manpower and material resources required for labeling; supporting change detection of multimodal remote sensing images, and reducing the application limitations of change detection methods.

[0015] Furthermore, the S101: obtains a first modality remote sensing image and a second modality remote sensing image of known image change detection true values; wherein the first modality remote sensing image refers to an optical image, and the second modality remote sensing image is a SAR image, wherein the acquisition time of the first modality remote sensing image is earlier than the acquisition time of the second modality remote sensing image.

[0016] Furthermore, the first modality remote sensing image and the second modality remote sensing image, whose true values of image change detection are known, are input into the change detection network, and the change detection network is trained to obtain a trained change detection network, which specifically includes: Constructing a data set, wherein the data set is a plurality of image pairs with known true values of image change detection, each image pair including: a first modality remote sensing image and a second modality remote sensing image; Divide the dataset into training and testing sets; Input the training set into the change detection network and train the change detection network. When the total loss function value of the network no longer decreases, stop training and obtain the trained change detection network. The test set is input into the trained change detection network for testing. If the test results all meet the preset test index values, the training is completed and the currently trained change detection network is the final trained change detection network; otherwise, the training set is replaced and retrained.

[0017] The beneficial effect of the above technical solution is that by fusing the first modality and the second modality remote sensing images, the complementary features of different modal data can be utilized to significantly improve the adaptability of the change detection model to complex scenes.

[0018] It should be understood that the multi-temporal and multi-modal remote sensing images (such as optical and SAR images) were collected and registered at the sub-pixel level (RMSE < 0.5 pixel) using SIFT feature matching or phase correlation algorithms. Misaligned areas were corrected using bilinear interpolation. Data augmentation included random cropping (256×256), rotation (0°–90°), and flipping. Histogram matching was performed on the optical images, and Lee filtering was used to denoise the SAR images. The images were then normalized to the range [-1, 1]. The dataset was split into training and test sets in a 7:3 ratio to ensure a balanced distribution of object types.

[0019] Furthermore, the change detection network has a network structure comprising: A feature extractor, a cross-modal attention layer, a change detector, a mask appender, and a classifier connected in sequence; The feature extractor includes: a first branch and a second branch arranged in parallel; The first branch includes: a first feature extraction module and a second feature extraction module connected in series; The second branch includes: a third feature extraction module and a fourth feature extraction module connected in series; The first feature extraction module, the second feature extraction module, the third feature extraction module and the fourth feature extraction module all have the same internal structure. The first feature extraction module includes: four TransformerBlock units connected in sequence.

[0020] The Transformer Block unit includes: a multi-head attention mechanism layer, a first normalization layer, a feedforward neural network FFN layer and a second normalization layer connected in sequence; wherein the input end of the first normalization layer is residually connected to the input end of the multi-head attention mechanism layer; the input end of the second normalization layer is residually connected to the input end of the feedforward neural network FFN layer; the input end of the multi-head attention mechanism layer is the input end of the Transformer Block unit, and the output end of the second normalization layer is the output end of the Transformer Block unit.

[0021] Furthermore, the change detection network first performs feature extraction on image blocks corresponding to the first and second modal remote sensing images respectively to obtain first and second modal features, and then performs feature extraction on the first and second modal features respectively to obtain first and second modal sub-features, including: Performing image cropping on the first modality remote sensing image to obtain a plurality of image blocks corresponding to the first modality remote sensing image; Performing image cropping on the second modality remote sensing image to obtain a plurality of image blocks corresponding to the second modality remote sensing image; Using a first feature extraction module, extracting features from an image block corresponding to the first modality remote sensing image to obtain first modality features; Using a third feature extraction module, feature extraction is performed on the image block corresponding to the second modality remote sensing image to obtain the second modality feature; Using the second feature extraction module, extract the first modal feature to obtain the first modal sub-feature; The fourth feature extraction module is used to extract features from the second modal features to obtain second modal sub-features.

[0022] The above technical solution has the following beneficial effects: the independent first and third feature extraction modules are optimized for the physical characteristics of different modalities, which can improve detection accuracy; cropping the remote sensing image before extracting features can significantly reduce video memory consumption; and the first and third feature extraction modules share parameters.

[0023] For example, in the feature extraction stage, a ViT-based encoder architecture is used to process optical and SAR multimodal image pairs. First, the input image is divided into 16×16 local blocks, and each image block is mapped to a 384-dimensional embedding vector through linear projection:

[0024] in, and The first four layers of the encoder's Transformer Blocks use a parameter-sharing strategy to extract common features (such as edges and textures) from both modalities. The last four layers independently process each modality and learn modality-specific features (such as spectral information in optics and scattering characteristics in SAR).

[0025] Furthermore, the weighted fusion of the first and second modal sub-features to obtain the fused feature specifically includes: A cross-modal attention layer is used to perform weighted fusion of the first and second modal sub-features to obtain the fused features.

[0026] Furthermore, the cross-modal attention layer is used to perform weighted fusion on the first and second modal sub-features to obtain fused features, specifically including: The first modal sub-feature As a query vector , the second modal sub-feature As a key vector , calculate the cross attention weight matrix :

[0027] in, ; Represents the dimension of the input vector.

[0028] The weight matrix is the first modal sub-feature after concatenation and the second modal sub-feature Perform weighted fusion to obtain fusion features :

[0029] in, Represents a concatenation operation along the channel dimension.

[0030] It should be understood that the beneficial effect of the above technical solution is that this process strengthens the response of the spatiotemporal consistency area through the attention mechanism and suppresses the pseudo-change features caused by sensor differences or noise. The final output fusion feature Serves as the basis for subsequent change detection.

[0031] Furthermore, performing change detection on the fused features to generate a change probability map includes: Through the change detector, the fused features are subjected to change detection to generate a change probability map.

[0032] Furthermore, the method of performing change detection on the fused features by using a change detector to generate a change probability map includes: First, pass through a 3×3 convolutional layer , compress the channel dimension to 128 dimensions, combined with batch normalization The nonlinear expression of the enhanced feature is enhanced with the activation function ReLU to obtain the first enhanced feature : ; Next, the first enhancement feature The channel attention module is used to process and highlight the key channel information; then the feature map is upsampled by bilinear interpolation. Double the spatial resolution to restore the spatial resolution, then use 3×3 convolution to further refine the edge features, and finally use 1×1 convolution to compress the number of channels to 1, and generate a change probability map through the Sigmoid function. , the overall process is formally expressed as: .

[0033] It should be understood that the channel attention module dynamically learns the importance weights of each channel through global average pooling, fully connected layers and nonlinear activation, and finally multiplies the weights with the original feature map channel by channel to highlight key channels and suppress redundant information, thereby adaptively improving the model's ability to focus on task-related features.

[0034] Furthermore, performing mask processing on image blocks corresponding to the first and second modal remote sensing images based on the change probability map to obtain the first and second modal image block sets is achieved by a mask adder.

[0035] Furthermore, the image blocks corresponding to the first and second modal remote sensing images are masked based on the change probability map to obtain the first and second modal image block sets, wherein the masking includes: Compare the value of each pixel in the change probability map with the set threshold 𝑡. If the current pixel value is greater than the set threshold 𝑡, set the current pixel value in the change probability map to 1; otherwise, set it to 0; If the current pixel value is 1, it means that the image block corresponding to the current pixel contains changed pixels, then all pixel values of the image block corresponding to the current pixel are set to zero; If the current pixel value is 0, it means that the image block corresponding to the current pixel does not contain any changed pixels, and all pixel values of the image block corresponding to the current pixel will remain unchanged.

[0036] Furthermore, mask processing is performed on the image blocks corresponding to the first modality remote sensing image to obtain a first modality image block set; and mask processing is performed on the image blocks corresponding to the second modality remote sensing image to obtain a second modality image block set.

[0037] It should be understood that when obtaining the change probability map After that, the probability map is first binarized into a change area mask through a global threshold 𝑡 (usually optimized based on the validation set or set based on the empirical value) Then, we take the divided 16×16 image blocks. If the image blocks contain changed pixels, we mask the image blocks and all the corresponding modal areas: all pixel values in the image blocks are set to 0. The image blocks without changed pixels remain unchanged, and finally a set of multimodal image blocks with dynamic masks is generated.

[0038] Furthermore, the first and second modal image block sets are respectively subjected to reconstruction coding, feature mapping and reconstruction decoding processing in sequence to obtain image change detection prediction values, which are implemented through a classifier.

[0039] Furthermore, the first and second modal image block sets are respectively subjected to reconstruction coding, feature mapping, and reconstruction decoding processing in sequence to obtain an image change detection prediction value, including: Encoding the first modality image block set using a first reconstruction encoder to obtain a first encoding result, inputting the first encoding result into a feature mapper for mapping processing to obtain a first mapping result, and inputting the first encoding result into a first reconstruction decoder for decoding processing to obtain a first decoding result; The second modality image block set is encoded using a second reconstruction encoder to obtain a second encoding result, the second encoding result is input into a feature mapper for mapping processing to obtain a second mapping result, and the second encoding result is input into a first reconstruction decoder for decoding processing to obtain a second decoding result; Calculating a first loss function value based on the first mapping result and the second mapping result; Calculating a second loss function value based on the first decoding result and the second decoding result; The first loss function value is summed with the second loss function value to obtain the total loss function value.

[0040] Furthermore, the internal structures of the first reconstruction encoder and the second reconstruction encoder are consistent, and the first reconstruction encoder includes N Transformer Block units connected in sequence.

[0041] Furthermore, for the Layer Transformer Block unit:

[0042] in, is the input vector. Represents multi-head self-attention.

[0043] It should be understood that feature learning is performed using a masked autoencoder based on a collection of dynamically masked multimodal image patches. Each local patch is first flattened into a vector and mapped to a latent space via a learnable linear mapping layer. The encoder consists of multiple layers of standard Transformer blocks, each containing a multi-head self-attention and feedforward network. Because the input has already suppressed changing regions through the dynamic mask, the features extracted by the encoder represent the features of the unchanged regions.

[0044] Furthermore, the feature mapper includes: four feature mapping layers connected in sequence, and each feature mapping layer includes: a multi-head attention mechanism layer and a gated feedforward neural network layer connected in sequence.

[0045] The feature mapper is used to capture a unified representation of multimodal features, while the specialized module refines the differences in modal details. It has four layers, each containing a multi-head self-attention mechanism and a gated feedforward network:

[0046]

[0047] in is the activation function, is element-wise multiplication. is the weight parameter, .

[0048] Furthermore, the first loss function is:

[0049] in, is the set of positive sample pairs, and is a set of negative sample pairs, represents the corresponding negative sample, is the cosine similarity, is the temperature parameter. Modal perception weight Dynamically adjust the KL (Kullback-Leibler) divergence of inter-modal feature distributions to enhance cross-modal alignment capabilities. Positive samples represent pixels at the same coordinate location in two remote sensing images, while negative samples represent pixels at different coordinate locations. The temperature parameter adjusts the distribution of similarity scores and controls the model's emphasis on positive and negative samples. The value is set to 0.1.

[0050] To obtain a unified feature representation for multimodal images, the extracted bi-temporal features are fed into a contrastive learning subnetwork. Cross-modal positive pairs are defined as bi-temporal features at the same spatial location, while negative pairs are random combinations of features at different locations or modalities. Traditional contrastive learning uses the InfoNCE loss, but this does not account for the heterogeneity of multimodal data.

[0051] Furthermore, the reconstruction decoder includes: four Transformer Block units connected in sequence.

[0052] Based on the multimodal features extracted by the reconstruction encoder, a cross-modal reconstruction decoder is designed to generate a reconstructed image of the target modality. By computing a loss function with the original target modality image, the model's ability to model cross-modal consistency and differences is further enhanced. Similar to the reconstruction encoder structure, the decoder adopts a layered Transformer architecture consisting of four stacked layers of Transformer blocks, each of which incorporates cross-modal attention and a feedforward network. The decoder ultimately outputs the feature matrix H of the target modality, which is then mapped back to pixel space via a projection layer. All local blocks are reconstructed into a complete image based on their spatial location.

[0053] Furthermore, the second loss function To: Calculate pixel-level L1 loss;

[0054] in, is the reconstructed image, is the original image, H represents the height of the feature, W represents the width of the feature, The number of channels representing features, represents the corresponding coordinate value.

[0055] Furthermore, the total loss function value of the network is calculated based on the true value of the image change detection and the predicted value of the image change detection, and the training of the network is completed based on the total loss function value, wherein the total loss function , its formula expression is: .

[0056] Furthermore, in step S102, obtaining a first modality remote sensing image and a second modality remote sensing image to be detected; inputting the first modality remote sensing image and the second modality remote sensing image to be detected into a trained change detection network to obtain change detection results of the first and second modality remote sensing images; wherein the change detection network used in the current step only includes: the third and fourth branches; The third branch includes: a first reconstruction encoder and a first feature mapper connected in sequence; The fourth branch includes: a second reconstruction encoder and a second feature mapper connected in sequence; An input end of the first reconstruction encoder is used to input an image block of a first modality remote sensing image to be detected; the image block of the first modality remote sensing image to be detected is obtained by segmenting the first modality remote sensing image to be detected; the image block of the first modality remote sensing image is mapped into a first embedding vector of multiple dimensions through linear mapping; the first embedding vector is input into the first reconstruction encoder to obtain a first reconstructed feature; the first reconstructed feature is input into the first feature mapper to obtain a first feature representation; The input end of the second reconstruction encoder is used to input an image block of a second modality remote sensing image to be detected; the image block of the second modality remote sensing image to be detected is obtained by segmenting the second modality remote sensing image to be detected; the image block of the second modality remote sensing image is mapped into a second embedding vector of multiple dimensions through linear mapping; the second embedding vector is input to the second reconstruction encoder to obtain a second reconstructed feature; the second reconstructed feature is input to the second feature mapper to obtain a second feature representation; The first feature representation is subtracted from the second feature representation pixel by pixel to obtain a change difference map; the change difference map is processed using an activation function to obtain a change probability map to be processed; the pixel value of each pixel point in the change probability map to be processed is compared with a set threshold t. If it is greater than the set threshold t, the pixel value of the current pixel point is set to 1; otherwise, the pixel value of the current pixel point is set to 0 to obtain a binary change map; the binary change map is subjected to threshold segmentation, closing operation and connected domain analysis in sequence to obtain the change detection results of the first and second modal remote sensing images.

[0057] It should be understood that in the actual use stage, only the reconstruction encoder and feature mapper are retained. The input image is divided into 16×16 image blocks, and each image block is mapped to a 384-dimensional embedding vector through linear projection. Then, without masking, it is directly sent to the reconstruction encoder to obtain the dual-phase reconstruction feature. The dual-phase reconstruction feature is then sent to the feature mapper to generate a unified feature representation. After pixel-by-pixel subtraction, the change difference map is obtained, and then the change probability map is generated through the Sigmoid function, and the threshold in step 4 is taken. After screening, a binary change map was obtained. Post-processing used the Otsu algorithm for adaptive threshold segmentation (default 0.6), a closing operation (kernel size 5×5) to fill holes, and a connected domain analysis to filter out noise with an area smaller than 50 pixels.

[0058] Example 2 This embodiment provides a mask-based self-supervised multimodal remote sensing image change detection system, including: The training module is configured to: obtain a first modality remote sensing image and a second modality remote sensing image with known true values for image change detection; input the first modality remote sensing image and the second modality remote sensing image with known true values for image change detection into a change detection network, train the change detection network, and obtain a trained change detection network; during the network training process, the change detection network first performs feature extraction on the image blocks corresponding to the first and second modality remote sensing images respectively to obtain first and second modality features, and then performs feature extraction on the first and second modality features respectively to obtain first and second modality sub-features; perform weighted fusion on the first and second modality sub-features to obtain fused features; perform change detection on the fused features to generate a change probability map; based on the change probability map, perform masking on the image blocks corresponding to the first and second modality remote sensing images to obtain first and second modality image block sets; perform reconstruction coding, feature mapping, and reconstruction decoding on the first and second modality image block sets in sequence to obtain image change detection prediction values; calculate the total loss function value of the network based on the true value of image change detection and the image change detection prediction value, and complete the training of the network based on the total loss function value; The detection module is configured to: obtain a first modality remote sensing image and a second modality remote sensing image to be detected; input the first modality remote sensing image and the second modality remote sensing image to be detected into a trained change detection network to obtain change detection results of the first and second modality remote sensing images.

[0059] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A mask-based self-supervised multimodal remote sensing image change detection method, characterized by: include: Acquire a first modality remote sensing image and a second modality remote sensing image with known true values for image change detection; Inputting a first modality remote sensing image and a second modality remote sensing image, both of which have known true values for image change detection, into a change detection network, training the change detection network, and obtaining a trained change detection network; During the network training process, the change detection network first extracts features from the image blocks corresponding to the first and second modal remote sensing images to obtain the first and second modal features, then extracts features from the first and second modal features to obtain the first and second modal sub-features; and then performs weighted fusion on the first and second modal sub-features to obtain the fused features. Perform change detection on the fused features to generate a change probability map; based on the change probability map, perform mask processing on the image blocks corresponding to the first and second modal remote sensing images to obtain a set of first and second modal image blocks; perform reconstruction coding, feature mapping, and reconstruction decoding on the first and second modal image blocks, respectively, to obtain an image change detection prediction value; calculate the total loss function value of the network based on the true image change detection value and the image change detection prediction value, and complete the network training based on the total loss function value; A first modality remote sensing image and a second modality remote sensing image to be detected are obtained; the first modality remote sensing image and the second modality remote sensing image to be detected are input into a trained change detection network to obtain change detection results of the first and second modality remote sensing images.

2. The mask-based self-supervised multimodal remote sensing image change detection method according to claim 1, wherein: The change detection network has a network structure comprising: A feature extractor, a cross-modal attention layer, a change detector, a mask appender, and a classifier connected in sequence; The feature extractor includes: a first branch and a second branch arranged in parallel; The first branch includes: a first feature extraction module and a second feature extraction module connected in series; The second branch includes: a third feature extraction module and a fourth feature extraction module connected in series; The first feature extraction module, the second feature extraction module, the third feature extraction module and the fourth feature extraction module all have the same internal structure. The first feature extraction module includes: four Transformer Block units connected in sequence; The Transformer Block unit includes: a multi-head attention mechanism layer, a first normalization layer, a feedforward neural network FFN layer and a second normalization layer connected in sequence; wherein the input end of the first normalization layer is residually connected to the input end of the multi-head attention mechanism layer; the input end of the second normalization layer is residually connected to the input end of the feedforward neural network FFN layer; the input end of the multi-head attention mechanism layer is the input end of the Transformer Block unit, and the output end of the second normalization layer is the output end of the Transformer Block unit.

3. The mask-based self-supervised multimodal remote sensing image change detection method according to claim 1, wherein: The change detection network first performs feature extraction on image blocks corresponding to the first and second modal remote sensing images respectively to obtain first and second modal features, and then performs feature extraction on the first and second modal features respectively to obtain first and second modal sub-features, including: Performing image cropping on the first modality remote sensing image to obtain a plurality of image blocks corresponding to the first modality remote sensing image; Performing image cropping on the second modality remote sensing image to obtain a plurality of image blocks corresponding to the second modality remote sensing image; Using a first feature extraction module, extracting features from an image block corresponding to the first modality remote sensing image to obtain first modality features; Using a third feature extraction module, feature extraction is performed on the image block corresponding to the second modality remote sensing image to obtain the second modality feature; Using the second feature extraction module, extract the first modal feature to obtain the first modal sub-feature; The fourth feature extraction module is used to extract features from the second modal features to obtain second modal sub-features.

4. The mask-based self-supervised multimodal remote sensing image change detection method according to claim 1, wherein: The weighted fusion of the first and second modal sub-features to obtain the fused feature specifically includes: using a cross-modal attention layer to weightedly fuse the first and second modal sub-features to obtain the fused feature; the weighted fusion of the first and second modal sub-features to obtain the fused feature specifically includes: The first modal sub-feature As a query vector , the second modal sub-feature As a key vector , calculate the cross attention weight matrix : ； in, ; Represents the dimension of the input vector; The weight matrix is the first modal sub-feature after concatenation and the second modal sub-feature Perform weighted fusion to obtain fusion features : ； in, Represents a concatenation operation along the channel dimension.

5. The mask-based self-supervised multimodal remote sensing image change detection method according to claim 4, wherein: The performing change detection on the fused features to generate a change probability map includes: performing change detection on the fused features by a change detector to generate a change probability map; the performing change detection on the fused features by a change detector to generate a change probability map includes: First, pass through a 3×3 convolutional layer , compress the channel dimension to 128 dimensions, combined with batch normalization The nonlinear expression of the enhanced feature is enhanced with the activation function ReLU to obtain the first enhanced feature : ； Next, the first enhancement feature The channel attention module is used to process and highlight the key channel information; then the feature map is upsampled by bilinear interpolation. Double the spatial resolution to restore the spatial resolution, then use 3×3 convolution to further refine the edge features, and finally use 1×1 convolution to compress the number of channels to 1, and generate a change probability map through the Sigmoid function. , the overall process is formally expressed as: 。 6. The mask-based self-supervised multimodal remote sensing image change detection method according to claim 1, wherein: The masking process is performed on the image blocks corresponding to the first and second modal remote sensing images according to the change probability map to obtain the first and second modal image block sets, wherein the masking process includes: Compare the value of each pixel in the change probability map with the set threshold 𝑡. If the current pixel value is greater than the set threshold 𝑡, set the current pixel value in the change probability map to 1; otherwise, set it to 0; If the current pixel value is 1, it means that the image block corresponding to the current pixel contains changed pixels, then all pixel values of the image block corresponding to the current pixel are set to zero; If the current pixel value is 0, it means that the image block corresponding to the current pixel does not contain any changed pixels, and all pixel values of the image block corresponding to the current pixel will remain unchanged; Masking is performed on image blocks corresponding to the first modality remote sensing image to obtain a first modality image block set; and masking is performed on image blocks corresponding to the second modality remote sensing image to obtain a second modality image block set.

7. The mask-based self-supervised multimodal remote sensing image change detection method according to claim 1, wherein: The first and second modal image block sets are respectively subjected to reconstruction coding, feature mapping, and reconstruction decoding processing to obtain image change detection prediction values, including: Encoding the first modality image block set using a first reconstruction encoder to obtain a first encoding result, inputting the first encoding result into a feature mapper for mapping processing to obtain a first mapping result, and inputting the first encoding result into a first reconstruction decoder for decoding processing to obtain a first decoding result; The second modality image block set is encoded using a second reconstruction encoder to obtain a second encoding result, the second encoding result is input into a feature mapper for mapping processing to obtain a second mapping result, and the second encoding result is input into a first reconstruction decoder for decoding processing to obtain a second decoding result; Calculating a first loss function value based on the first mapping result and the second mapping result; Calculating a second loss function value based on the first decoding result and the second decoding result; The first loss function value is summed with the second loss function value to obtain the total loss function value.

8. The mask-based self-supervised multimodal remote sensing image change detection method according to claim 7, wherein: The first loss function is: ； in, is the set of positive sample pairs, and is a set of negative sample pairs, represents the corresponding negative sample, is the cosine similarity, is the temperature parameter; modal perception weight Enhance cross-modal alignment capabilities by dynamically adjusting the KL divergence of inter-modal feature distributions; The second loss function To: Calculate pixel-level L1 loss; ； in, is the reconstructed image, is the original image, H represents the height of the feature, W represents the width of the feature, The number of channels representing features, represents the corresponding coordinate value.

9. The mask-based self-supervised multimodal remote sensing image change detection method according to claim 1, wherein: Obtain a first modality remote sensing image and a second modality remote sensing image to be detected; input the first modality remote sensing image and the second modality remote sensing image to be detected into a trained change detection network to obtain change detection results of the first and second modality remote sensing images; wherein the change detection network used in the current step only includes: the third and fourth branches; The third branch includes: a first reconstruction encoder and a first feature mapper connected in sequence; The fourth branch includes: a second reconstruction encoder and a second feature mapper connected in sequence; An input end of the first reconstruction encoder is used to input an image block of a first modality remote sensing image to be detected; the image block of the first modality remote sensing image to be detected is obtained by segmenting the first modality remote sensing image to be detected; the image block of the first modality remote sensing image is mapped into a first embedding vector of multiple dimensions through linear mapping; the first embedding vector is input into the first reconstruction encoder to obtain a first reconstructed feature; the first reconstructed feature is input into the first feature mapper to obtain a first feature representation; The input end of the second reconstruction encoder is used to input an image block of a second modality remote sensing image to be detected; the image block of the second modality remote sensing image to be detected is obtained by segmenting the second modality remote sensing image to be detected; the image block of the second modality remote sensing image is mapped into a second embedding vector of multiple dimensions through linear mapping; the second embedding vector is input to the second reconstruction encoder to obtain a second reconstructed feature; the second reconstructed feature is input to the second feature mapper to obtain a second feature representation; The first feature representation is subtracted from the second feature representation pixel by pixel to obtain a change difference map; the change difference map is processed using an activation function to obtain a change probability map to be processed; the pixel value of each pixel point in the change probability map to be processed is compared with a set threshold t. If it is greater than the set threshold t, the pixel value of the current pixel point is set to 1; otherwise, the pixel value of the current pixel point is set to 0 to obtain a binary change map; the binary change map is subjected to threshold segmentation, closing operation and connected domain analysis in sequence to obtain the change detection results of the first and second modal remote sensing images.

10. A mask-based self-supervised multimodal remote sensing image change detection system, characterized by: include: A training module is configured to: obtain a first modality remote sensing image and a second modality remote sensing image with known true values for image change detection; Inputting a first modality remote sensing image and a second modality remote sensing image, both of which have known true values for image change detection, into a change detection network, training the change detection network, and obtaining a trained change detection network; During the network training process, the change detection network first extracts features from the image blocks corresponding to the first and second modal remote sensing images to obtain the first and second modal features, then extracts features from the first and second modal features to obtain the first and second modal sub-features; and then performs weighted fusion on the first and second modal sub-features to obtain the fused features. Perform change detection on the fused features to generate a change probability map; based on the change probability map, perform mask processing on the image blocks corresponding to the first and second modal remote sensing images to obtain a set of first and second modal image blocks; perform reconstruction coding, feature mapping, and reconstruction decoding on the first and second modal image blocks, respectively, to obtain an image change detection prediction value; calculate the total loss function value of the network based on the true image change detection value and the image change detection prediction value, and complete the network training based on the total loss function value; The detection module is configured to: obtain a first modality remote sensing image and a second modality remote sensing image to be detected; input the first modality remote sensing image and the second modality remote sensing image to be detected into a trained change detection network to obtain change detection results of the first and second modality remote sensing images.

Images