Heterogeneous remote sensing image change detection system and method based on Mama model

Abstract

The invention relates to a heterogeneous remote sensing image change detection system and method based on a Mama model. The system comprises a feature extractor, a codec network and a change detector. The feature extractor is used for extracting first feature maps of an optical mode and an SAR mode respectively; the codec network is used for mapping and reconstructing the first feature map based on a Mama model to generate a second feature map of an optical mode and an SAR mode; the change detector subtracts the second feature maps of the same mode and calculates an L2 norm along the channel dimension, and weighted fusion is carried out on calculation results of the L2 norms of different modes, so that a difference map is obtained; the difference map processor optimizes the difference map based on a full-connection conditional random field method, and segments the optimized difference map into a varying region and a non-varying region based on an optimal threshold selected by an adaptive threshold segmentation algorithm. According to the method, a feature alignment process and difference chart generation are coupled into an end-to-end optimization task, and the problem of error accumulation caused by two-stage decoupling in a traditional unsupervised method is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image change detection, and in particular to a heterogeneous remote sensing image change detection system and method based on a Mamba model. Background Art

[0002] Remote sensing image change detection technology identifies changes in land cover or target status by analyzing the differences in spectral or spatial structural characteristics of images of the same area at different times. It has important application value in urban planning, environmental monitoring, and disaster assessment. Traditional remote sensing image change detection methods mainly design algorithms for homologous remote sensing images (such as optical or radar images acquired by a single sensor), leveraging the natural consistency of their spectral and spatial characteristics to extract differences. However, in disaster emergency scenarios such as floods and earthquakes, post-disaster images are often limited by factors such as cloud cover, and only synthetic aperture radar images can be obtained. Pre-disaster archived data are mostly optical images. There is an urgent need to develop cross-modal heterogeneous remote sensing image change detection technology to support rapid rescue decision-making.

[0003] Existing heterogeneous change detection methods often employ unsupervised learning frameworks to address the technical challenge of scarce labeled data. Typical solutions include the symmetric convolutional coupled network (SCCN), which maps heterogeneous images into a shared feature space for difference analysis via a two-stream convolutional network; and modality conversion methods based on conditional generative adversarial networks (cGANs), which unify heterogeneous images into a new modality for direct comparison. The core of these methods lies in establishing a feature alignment mechanism between images, but their performance faces multiple limitations. First, existing network architectures overly focus on the modality alignment process while neglecting targeted optimization for the change detection task itself, resulting in insufficient feature discrimination. Second, methods based on convolutional neural networks (CNNs) are limited by their local receptive field characteristics, making it difficult to effectively model global information correlations between images. Furthermore, heterogeneous images exhibit significant domain shifts in spectral response, texture structure, and geometric deformation due to differences in sensor imaging mechanisms (e.g., reflectivity characteristics for optical images versus backscattering characteristics for radar images), directly impacting the comparability of the changed regions.

[0004] Current technical deficiencies are primarily manifested in three aspects: First, unsupervised methods lack optimal design for change detection tasks, decoupling feature alignment from the change discrimination objective, resulting in low detection stability. Second, traditional CNN architectures are limited in capturing long-range dependencies across modal imagery, restricting the model's adaptability to complex scenarios. Third, the highly specialized nature of annotated data for heterogeneous change detection makes it difficult to acquire on a large scale, and unsupervised methods, lacking effective learning guidance mechanisms, are prone to falling into local optimal solutions. These bottlenecks collectively limit the practical deployment of heterogeneous remote sensing change detection technology in time-sensitive scenarios such as emergency response.

[0005] For example, CN107480620A proposes a method for automatic recognition of remote sensing targets based on heterogeneous feature fusion, which includes a training phase and a testing phase. The training phase includes the following steps: Step 1: Construct a training set of multi-class remote sensing image target samples, classify the target samples to be trained, and create labels; Step 2: Based on the analysis of the special imaging mechanism of remote sensing images, extract remote sensing target samples to train multiple heterogeneous features, including: improved dense SURF-BOF (Speeded Up Robust Feature-Bag of Features) features, improved Mean-Std (Mean-Standard Deviation) features, and LBP (Local Binary Patterns) features; Step 3: Select and construct a complete multi-kernel space, map different types of target features to a unified kernel space, design a multi-kernel fusion algorithm, and realize adaptive fusion of heterogeneous features in the kernel space; Step 4: Design an MKL-SVM classifier based on heterogeneous feature fusion; The testing phase includes the following steps:

[0006] Step 5: Construct a multi-class remote sensing image target test set; Step 6: Extract improved dense SURF-BOF features, improved Mean-Std features, and LBP features from the remote sensing image target test set; Step 7: Input the extracted multiple heterogeneous features of the test image into the MKL-SVM classifier trained in Step 4 to classify and recognize the test image. This technical solution relies on manually designed feature extraction methods (such as improved SURF-BOF, Mean-Std, and LBP). Its feature representation capabilities are limited by human experience and cannot adaptively capture the complex texture, scale, and radiometric differences in remote sensing images. In particular, when processing heterogeneous remote sensing data (such as optical and SAR imagery), manual features cannot effectively align cross-modal feature distributions, leading to feature mismatch. Summary of the Invention

[0007] Current technical deficiencies are primarily manifested in three areas: First, unsupervised methods lack sufficient optimization objectives for change detection tasks, decoupling feature alignment from the change discrimination objective, resulting in low detection stability. Second, traditional CNN architectures are limited in capturing long-range dependencies across modal imagery, restricting the model's adaptability to complex scenarios. Third, the highly specialized nature of annotated data for heterogeneous change detection makes it difficult to acquire on a large scale, and unsupervised methods, lacking effective learning guidance mechanisms, are prone to falling into local optimal solutions. These bottlenecks collectively limit the practical deployment of heterogeneous remote sensing change detection technology in time-sensitive scenarios such as emergency response.

[0008] To address the shortcomings of the prior art, the present invention provides, from a first aspect, a heterogeneous remote sensing image change detection system based on the Mamba model, comprising a feature extractor, a codec network, and a change detector. The feature extractor is used to extract first feature maps of the optical modality and the SAR modality, respectively; the codec network is used to map and reconstruct the first feature maps based on the Mamba model to generate second feature maps of the optical modality and the SAR modality; the change detector subtracts the second feature maps of the same modality and calculates the L2 norm along the channel dimension, and weightedly fuses the L2 norm calculation results of different modalities to obtain a difference map. The difference map processor optimizes the difference map based on a fully connected conditional random field method and segments the optimized difference map into changed regions and unchanged regions based on an optimal threshold selected by an adaptive threshold segmentation algorithm.

[0009] The encoder-decoder network unifies the modeling of feature alignment and difference detection through a shared encoder and dual decoder architecture. The shared encoder extracts global information based on the Mamba model, forcing the distribution of optical and SAR modal features to align within a high-dimensional common domain. This technical solution couples the feature alignment process (common domain mapping) with the change detection objective (difference map generation) as an end-to-end optimization task, avoiding the error accumulation problem caused by the two-stage decoupling in traditional unsupervised methods.

[0010] The codec network includes a shared encoder, an optical decoder, and a SAR decoder. The shared encoder extracts global information from the first feature map of the optical modality and the first feature map of the SAR modality based on the global receptive field of the Mamba model and maps the global information to a high-dimensional common domain. The optical decoder reconstructs the first feature map of the optical modality in the high-dimensional common domain into a second feature map of the optical modality and maps the first feature map of the SAR modality into a second feature map of the optical modality. The SAR decoder reconstructs the first feature map of the SAR modality in the high-dimensional common domain into a second feature map of the SAR modality and maps the first feature map of the optical modality into a second feature map of the SAR modality.

[0011] A sequence modeling method (such as the Mamba model) is introduced into the shared encoder, and its global information modeling capability is used to capture the long-range dependencies of cross-modal images, and combined with residual connections to achieve cross-layer feature fusion.

[0012] According to a preferred embodiment, the difference map processor includes a fully connected conditional random field module and a threshold segmentation module. The fully connected conditional random field module establishes the correlation constraints between the global pixels of the difference map and calculates the energy function based on the fully connected conditional random field method, and calculates the optimal solution that makes the energy function approach the minimum value, so as to optimize the difference map; the threshold segmentation module traverses all possible segmentation thresholds between the changed area and the unchanged area based on the adaptive threshold segmentation algorithm, calculates the intra-class variance and inter-class variance of the pixel value set under each possible segmentation threshold, selects the optimal threshold that minimizes the intra-class variance and maximizes the inter-class variance, and segments the optimized difference map into the changed area and the unchanged area based on the optimal threshold.

[0013] Compared with the local receptive field limitations of traditional CNNs, the Mamba model can simultaneously model the global correlation between the image spatial domain and the channel domain, effectively capturing the complex domain offset patterns caused by sensor differences between heterogeneous images (such as the differences in the spectral reflectance characteristics of optical images and the backscattering characteristics of SAR images), thereby improving the model's adaptability to complex scenes.

[0014] According to a preferred embodiment, the feature extractor includes cascaded sub-encoders and sub-decoders, each level includes a sub-encoder and a sub-decoder connected to it, and the sub-encoder of the first-level structure is also connected to the sub-encoder and sub-decoder of the next-level structure respectively, so that the sub-decoder of the next-level structure outputs a feature map with gradually reduced dimension based on the mean square error loss function output by the sub-encoder of the previous-level structure; the sub-encoder of the next-level structure performs dimensionality reduction on the feature map output by the sub-decoder of the previous-level structure.

[0015] Preferably, the feature extractor adopts a three-level cascade structure, achieving cross-modal noise suppression and abstract feature extraction through layer-by-layer feature compression and nonlinear fitting. The first-level structure extracts the first-dimensional feature vector, and the second and third-level structures gradually reduce the dimensionality to the third-dimensional feature vector, forcing the network to ignore modality-specific noise and retain cross-modal shared semantic information. The third-dimensional feature vector has fewer dimensions than the first-dimensional feature vector.

[0016] The stacked structure implicitly constructs semantic consistency constraints between heterogeneous images through layer-by-layer denoising and feature abstraction, providing implicit guidance for unsupervised learning. Combining the fully connected conditional random field optimization (energy function constraint) and adaptive threshold segmentation (within-class and between-class variance criterion) of the difference map processor further avoids local optimal solutions and ensures the physical plausibility of detection results.

[0017] According to a preferred embodiment, the shared encoder in the codec network includes an input layer, an intermediate layer and an output layer. The input layer expands the number of channels of the first feature map and assigns channel weights and spatial weights, capturing the complex dependencies of the sequence along the channel dimension to extract global information; the intermediate layer performs residual connection on the global information output by the input layer to achieve cross-layer feature fusion; the output layer receives the information after the residual connection of the intermediate layer, and integrates and refines the global information based on the Mamba model while keeping the size of the first feature map unchanged.

[0018] According to a preferred embodiment, the input layer of the shared encoder in the codec network expands the first feature map channel for the first time based on the first convolutional layer, allocates channel weights and spatial weights based on the first convolutional attention module, and extracts the global information of the first feature map based on the Mamba model in the first Mamba module; the intermediate layer of the shared encoder expands the local receptive field of the second convolutional layer based on the maximum pooling layer, and enhances the fitting ability of the intermediate layer based on the residual connection; the output layer of the shared encoder expands the first feature map channel for the second time based on the third convolutional layer, allocates channel weights and spatial weights again based on the second convolutional attention module, and extracts the global information of the first feature map again based on the Mamba model in the second Mamba module.

[0019] The attention mechanism dynamically assigns feature weights, enabling the network to prioritize cross-modal common features relevant to change detection during channel weight calibration while suppressing irrelevant noise. This technical solution directly links the optimization goal of channel weight calibration with the identification of changed regions, reducing the loss of semantic information caused by over-alignment.

[0020] According to a preferred embodiment, the optical decoder or SAR decoder in the codec network includes an input layer, an intermediate layer, an output layer and a Sigmoid activation function module; the input layer includes a fourth convolutional layer and a third convolutional attention module; the intermediate layer includes a fifth convolutional layer and a fourth convolutional attention module; the output layer includes a sixth convolutional layer, a fifth convolutional attention module and a third Mamba module; the Sigmoid activation function module normalizes the feature map output by the third Mamba module, and is used to output the second feature map of the optical modality in the optical decoder and to output the second feature map of the SAR modality in the SAR decoder.

[0021] The architecture of the codec network can improve the quality and efficiency of image decoding by combining multi-level feature extraction, attention mechanism, special function modules and appropriate activation functions, and has a certain flexibility to cope with different types of data.

[0022] According to a second aspect, the present invention provides a method for heterogeneous remote sensing image change detection based on a Mamba model. The method comprises: extracting first feature maps of the optical modality and the SAR modality, respectively; mapping and reconstructing the first feature maps based on the Mamba model to generate second feature maps of the optical modality and the SAR modality; subtracting the second feature maps of the same modality and calculating the L2 norm along the channel dimension; and weightedly fusing the L2 norm calculation results of different modalities to obtain a difference map. The difference map is optimized based on a fully connected conditional random field method, and the optimized difference map is segmented into changed regions and unchanged regions based on an optimal threshold selected by an adaptive threshold segmentation algorithm.

[0023] The proposed method extracts global information based on the Mamba model, forcing the distribution alignment of optical and SAR modal features within a high-dimensional common domain. This technical solution couples the feature alignment process (common domain mapping) with the change detection objective (difference map generation) as an end-to-end optimization task, avoiding the error accumulation problem caused by the two-stage decoupling in traditional unsupervised methods.

[0024] Preferably, the step of mapping and reconstructing the first feature map based on the Mamba model to generate the second feature maps of the optical modality and the SAR modality includes: extracting global information from the first feature map of the optical modality and the first feature map of the SAR modality based on the global receptive field of the Mamba model and mapping them to a high-dimensional common domain; reconstructing the first feature map of the optical modality in the high-dimensional common domain into the second feature map of the optical modality, and mapping the first feature map of the SAR modality into the second feature map of the optical modality; reconstructing the first feature map of the SAR modality in the high-dimensional common domain into the second feature map of the SAR modality, and mapping the first feature map of the optical modality into the second feature map of the SAR modality.

[0025] The shared encoder forces the bimodal features to be aligned within a high-dimensional common domain. L2 norm fusion unifies feature alignment and difference detection into an end-to-end optimization task, avoiding the error accumulation caused by two-stage decoupling in traditional methods and improving detection stability. Furthermore, the Mamba model effectively captures complex domain offset patterns in cross-modal imagery (such as the long-range correlation between spectral reflectance in optical images and backscatter in SAR images) by modeling global information correlation in the channel dimension through structured state space sequences, improving the model's adaptability to complex scenarios.

[0026] According to a preferred embodiment, the steps of respectively extracting the first feature maps of the optical mode and the SAR mode include: extracting a first-dimensional feature vector representing more advanced and abstract features from the input original remote sensing images of each modality; approximately fitting the first-dimensional feature vector to output a second-dimensional feature vector; approximately fitting the second-dimensional feature vector to output a first feature map of the third-dimensional feature vector.

[0027] The stacked structure uses nonlinear compression to filter modality-specific noise (such as speckle noise in SAR images and illumination variations in optical images), preserving cross-modal shared semantic information (such as flooded areas and fire-damaged areas), providing a reliable feature foundation for unsupervised alignment. The unsupervised dimensionality reduction process of the feature extractor implicitly constructs semantic consistency constraints, guiding the model to learn cross-modal invariant features in the absence of labeled data, thereby reducing its reliance on labeled data.

[0028] According to a preferred embodiment, the shared encoder, optical decoder and SAR decoder in the codec network are all provided with a convolutional force attention module; the convolutional force attention module includes a channel weight calibration unit and a spatially significant area enhancement unit; the channel weight calibration unit extracts the maximum pooling features and the average pooling features of each channel in the feature map of the original input, and fits the maximum pooling features and the average pooling features based on the multi-layer perceptron to obtain the channel weights reflecting the relative importance of each channel in the current task, and multiplies the channel weights with the feature map of the original input to obtain the weight-calibrated output; the spatially significant area enhancement unit performs maximum pooling and average pooling operations on different spatial regions of the feature map of the original input to obtain the maximum pooling features and the average pooling features of each region, and splices and fits the maximum pooling features and the average pooling features of each region based on the convolution kernel to obtain a fitted feature map; calculates the spatial weight of each spatial position for the fitted feature map; multiplies the spatial weight with the feature map of the original input to obtain the output after regional enhancement.

[0029] The channel weight calibration unit extracts the maximum pooling features and average pooling features from each channel and uses a multi-layer perceptron to fit these features to calculate channel weights. This allows the model to identify which channels are more important for the current task. This approach effectively enhances the model's focus on useful features while suppressing irrelevant ones. The channel weight calibration unit weights the original input feature map based on the calculated channel weights, dynamically adjusting the contributions of different channels and thus improving the model's expressiveness and flexibility.

[0030] The Spatial Salient Region Enhancement Unit performs maximum and average pooling operations on different spatial regions of the feature map and fits these features using convolution kernels to generate fitted feature maps. This process helps identify and emphasize important local details or salient regions in the image. The Spatial Salient Region Enhancement Unit calculates spatial weights for each spatial location and applies these weights to the original input feature map. This can more accurately highlight areas that are critical to the task, thereby improving the model's ability to locate and classify target objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1This is a simplified module connection diagram of a heterogeneous remote sensing image change detection system based on the Mamba model provided by the present invention;

[0032] Figure 2 This is another simplified module connection diagram of the heterogeneous remote sensing image change detection system based on the Mamba model provided by the present invention;

[0033] Figure 3 is a schematic diagram of the changes before and after processing of the first characteristic graph of the optical modality provided by the present invention;

[0034] Figure 4 is a schematic diagram of the connection between the change detector and the difference map processor provided by the present invention;

[0035] Figure 5 is a schematic diagram of a data set of image samples provided by the present invention;

[0036] Figure 6 1 is a schematic diagram of the cascade structure of the optical feature extractor provided by the present invention;

[0037] Figure 7 This is a schematic diagram of the structure of one of the convolutional attention modules provided by the present invention;

[0038] Figure 8 It is a structural diagram of the Mamba model provided by the present invention;

[0039] Figure 9 It is a schematic diagram of the experimental results of the heterogeneous remote sensing image change detection system provided by the present invention and the traditional algorithm;

[0040] Figure 10 Schematic diagram of the structure of the shared encoder provided by the present invention;

[0041] Figure 11 A table comparing the experimental results of the image sample dataset provided by the present invention with the experimental results of the traditional algorithm;

[0042] Figure 12 It is a structural schematic diagram of the optical decoder or SAR decoder provided by the present invention.

[0043] Reference Signs List

[0044] 100: Feature extractor; 110: Optical feature extractor; 111: First sub-encoder; 112: First sub-decoder; 120: SAR feature extractor; 121: Second sub-encoder; 122: Second sub-decoder; 131: Third sub-encoder; 132: Third sub-decoder; 140: Tensor converter; 200: Encoder-decoder network; 210: Shared encoder; 220: Optical decoder; 221: Channel weight calibration unit; 222: Spatial salient region enhancement unit; 230: SAR decoder; 300: Change detector; 400: Difference map processor; 410: Fully connected conditional random field module; 420: Threshold segmentation module; 500: Original remote sensing image. DETAILED DESCRIPTION

[0045] The following is a detailed description with reference to the accompanying drawings.

[0046] The present invention first explains the noun terms.

[0047] First feature map: The feature map extracted by the feature extractor 100 (SCDAE), which removes some redundant pixels while retaining the features of the main part. An example of the first feature map is as follows Figure 3 As shown in FIG, the original remote sensing image 500 of the three channels is converted into the first feature map of the optical modality after being processed by the feature extractor 100. Figure 3 As shown in the right half of , the first feature map of the optical modality has 20 channels, and each channel retains the features of the main part of the original remote sensing image 500. For example, the first channel removes redundant pixels.

[0048] Second feature map: an image generated by the codec network 200 processing the first feature map.

[0049] Optical modality: refers to remote sensing image data obtained through optical sensors. Its imaging relies on the reflected energy of electromagnetic bands such as visible light and near-infrared, and can characterize the spectral reflectance characteristics and multispectral information of the ground objects.

[0050] SAR mode: refers to remote sensing image data obtained by Synthetic Aperture Radar (SAR) sensors, which actively transmit electromagnetic waves in the microwave band and receive backscattered signals from ground objects, and have all-day and all-weather imaging capabilities.

[0051] Feature extractor 100: A deep network architecture consisting of multiple cascaded structures, which realizes feature extraction through layer-by-layer feature compression and denoising.

[0052] Residual connections are a network design technique used in deep learning. They are primarily used to address the vanishing / exploding gradients and model degradation (i.e., accuracy decreases instead of improving with deeper networks) that occur during deep neural network training. The core idea is to introduce skip connections, which allow the neural network to learn a residual mapping between input and output, rather than directly learning the original mapping.

[0053] Mamba model: The neural network framework based on structured state space sequence modeling in this invention is good at capturing long-range dependencies and global information.

[0054] Shared encoder 210: A feature extraction module that shares weight parameters in multimodal processing, used to map heterogeneous data into a unified feature space.

[0055] Channel weight calibration unit 221: a submodule that generates channel attention weights through statistical fusion to achieve channel weight calibration.

[0056] Spatial salient region enhancement unit 222: A submodule based on dual-path pooling and spatial attention mechanism, used to enhance the feature response of change-sensitive areas.

[0057] Fully Connected Conditional Random Field: An energy function optimization method based on global correlation between pixels to improve the spatial consistency of difference maps.

[0058] Adaptive threshold segmentation algorithm: an unsupervised method that dynamically determines the optimal segmentation threshold by minimizing the intra-class variance and maximizing the inter-class variance.

[0059] Dual-channel statistics: channel-level feature distribution descriptors extracted by parallel max pooling and average pooling operations.

[0060] Global information: long-range semantic association features covering the entire spatial range of the image.

[0061] High-dimensional common domain: the unified high-dimensional feature space where heterogeneous features are located after mapping.

[0062] Italy Dataset: This dataset was collected from a flood-affected area in northern Italy. The pre-disaster remote sensing data consisted of multispectral optical imagery acquired by the Landsat-8 satellite (with a spatial resolution of 30 meters, covering visible and near-infrared bands), and the post-disaster data consisted of synthetic aperture radar (SAR) imagery acquired by the Sentinel-1 satellite (C-band, with a resolution of 10 meters). The imagery covers rivers, farmland, and residential areas, with changes marked as vegetation damage and road interruptions caused by flooding. This dataset was used to validate the robustness of cross-modal change detection algorithms based on optical and SAR remote sensing imagery in natural disaster emergency response scenarios.

[0063] Gurast Dataset: This dataset is derived from forest fire monitoring scenarios in the Gurast region of Northern Europe. Pre-disaster imagery is provided by the Sentinel-2 satellite, providing high-resolution multispectral optical data (10-meter resolution, covering the red edge and shortwave infrared bands). Post-disaster imagery is provided by the ALOS-2 PALSAR-2 radar sensor, using L-band SAR imagery (15-meter resolution). The data annotates areas of vegetation loss and exposed ground due to the fire, with significant differences in texture between optical and radar imagery. This dataset is suitable for testing algorithms' sensitivity to dramatic texture changes in heterogeneous imagery.

[0064] Ottawa Dataset: This dataset was developed to monitor urban expansion in Ottawa, Canada. The pre-disaster data consists of C-band SAR imagery (25-meter resolution) acquired by the RADARSAT-1 satellite in 2002, and the post-disaster data consists of multispectral optical imagery (0.3-meter resolution, covering visible and near-infrared bands) acquired by the WorldView-3 satellite in 2022. Annotated areas of change include new buildings, road expansions, and the conversion of green spaces to artificial surfaces. This dataset focuses on validating the effectiveness of algorithms for geometric deformation correction and channel weight calibration in high-resolution imagery.

[0065] Example 1

[0066] Current technical deficiencies are primarily manifested in three areas: First, unsupervised methods lack sufficient optimization objectives for change detection tasks, decoupling feature alignment from the change discrimination objective, resulting in low detection stability. Second, traditional CNN architectures are limited in capturing long-range dependencies across modal imagery, restricting the model's adaptability to complex scenarios. Third, the highly specialized nature of annotated data for heterogeneous change detection makes it difficult to acquire on a large scale, and unsupervised methods, lacking effective learning guidance mechanisms, are prone to falling into local optimal solutions. These bottlenecks collectively limit the practical deployment of heterogeneous remote sensing change detection technology in time-sensitive scenarios such as emergency response.

[0067] In view of the shortcomings of the existing technology, the present invention provides a heterogeneous remote sensing image change detection system based on the Mamba model, including a feature extractor 100, a codec network 200, a change detector 300 and a difference map processor 400.

[0068] The feature extractor 100 is used to extract the first feature map of the optical mode and the SAR mode respectively. The number of the feature extractors 100 is preferably two, an optical feature extractor 110 and a SAR feature extractor 120, such as Figure 1 and 2 The optical feature extractor 110 is used to extract a first feature map of the optical modality. The SAR feature extractor 120 is used to extract a first feature map of the SAR modality.

[0069] The codec network 200 is used to map and reconstruct the first feature map based on the Mamba model to generate the second feature map of the optical mode and the SAR mode. Figure 1 and 2 As shown, a novel loss function is designed in the data processing process within the codec network 200 to guide the fitting direction of the codec network 200, thereby achieving guidance for unsupervised learning.

[0070] like Figure 1 and 2 As shown, the codec network 200 includes a shared encoder 210, an optical decoder 220, and a SAR decoder 230. The shared encoder 210 captures the complex dependencies of the sequence along the channel dimension using the Mamba model, extracts global information, and maps the first feature map of the optical modality and the first feature map of the SAR modality onto a high-dimensional common domain. The optical decoder 220 reconstructs the first feature map of the optical modality in the high-dimensional common domain into a second feature map of the optical modality and maps the first feature map of the SAR modality into a second feature map of the optical modality. The SAR decoder 230 reconstructs the first feature map of the SAR modality in the high-dimensional common domain into a second feature map of the SAR modality and maps the first feature map of the optical modality into a second feature map of the SAR modality.

[0071] Since the shared weights of the same shared encoder 210 are used, that is, the first feature map of the optical modality and the optical feature map of the SAR modality are sequentially sent to the shared encoder 210 for encoding, the distribution characteristics of the value range (high-dimensional public domain) of the shared encoder 210 are naturally consistent.

[0072] Compared to using two encoders to process first feature maps of different modalities separately, the shared weights of shared encoder 210 reduce the computational complexity of codec network 200. This makes the network more lightweight, and the consistent high-dimensional common domain variables facilitate change detection after reconstruction and mapping into second feature maps. For second feature maps of the same modality, the benefit of consistency is reflected in the fact that the features of unchanged areas of the reconstructed and mapped second feature map are similar, while the features of changed areas are clearly distinct.

[0073] Preferably, if Figure 2 As shown, the shared encoder 210 may be configured as two physical hardwares for processing the first feature map of the optical modality extracted by the optical feature extractor 110 and the first feature map of the SAR modality extracted by the SAR feature extractor 120 respectively.

[0074] The change detector 300 is used to subtract the second feature maps of the same modality and calculate the L2 norm along the channel dimension, and then weightedly fuse the L2 norm calculation results of different modalities to obtain a difference map.

[0075] Specifically, if Figure 4 As shown, the second feature map of the reconstructed optical modality is subtracted from the second feature map of the mapped optical modality to obtain a feature difference map of the optical modality (still having 20-dimensional channels). The L2 norm of the feature difference map of the optical modality is calculated along the channel dimension to obtain a difference map of the optical modality.

[0076] like Figure 4 As shown, the second feature map of the reconstructed SAR modality is subtracted from the second feature map of the mapped SAR modality to obtain a feature difference map of the SAR modality. The L2 norm of the feature difference map of the SAR modality is calculated along the channel dimension to obtain a difference map of the SAR modality.

[0077] The formula for calculating the L2 norm is:

[0078] Among them, C represents the number of channels (C=20), H represents the vertical resolution of the feature difference map, W represents the horizontal resolution of the feature difference map; i represents the channel dimension index; h represents the spatial height index; w represents the spatial width index.

[0079] like Figure 4 As shown in FIG, the difference map of the optical mode is weightedly fused with the difference map of the SAR mode to obtain the difference map.

[0080] The formula for weighted fusion is:

[0081] in, a second characteristic map representing the reconstructed optical modality; a second characteristic map representing the mapped optical modality; A second eigenmap representing the reconstructed SAR modality; Represents the second characteristic map of the mapped SAR modality. Characteristic difference maps representing optical modes; Characteristic difference map representing the SAR modalities. is a weight parameter, indicating that the difference in contributions between optical and SAR modes is treated equally. i Represents the fused difference map, which is a two-dimensional matrix. ‖·‖2 represents the calculation of the L2 norm of the channel dimension of the feature difference map.

[0082] Preferably, if Figure 1 、 Figure 2 and Figure 4 As shown, the difference map obtained by the change detector 300 is a single-channel grayscale map, where the portion with a pixel value close to 0 is a black area, indicating that no change has occurred, and the portion with a pixel value close to 1 is a white area, indicating that a change has occurred.

[0083] Since the boundary between changes and no changes on the difference map is not clear, which is reflected by the presence of some light white areas, the blurred boundary and light white areas of the difference map will significantly reduce the accuracy and reliability of change detection in heterogeneous data scenarios. Based on this defect, the system of the present invention also provides a difference map processor 400, such as Figure 1 、 Figure 2 and Figure 4 The difference map processor 400 further optimizes the difference map based on the fully connected conditional random field (Dense CRF), and then generates the final change detection result based on the adaptive threshold segmentation algorithm (OSTU).

[0084] According to a preferred embodiment, Figure 2 As shown, the difference map processor 400 includes a fully connected conditional random field module 410 and a threshold segmentation module 420.

[0085] The fully connected conditional random field module 410 establishes the association constraints between the global pixels of the difference map based on the fully connected conditional random field (calculation method) and calculates the energy function, and calculates the optimal solution that makes the energy function approach the minimum value to optimize the difference map.

[0086] Fully connected conditional random fields are a pixel-level optimization method based on probabilistic graphical models. They are used to establish association constraints between pixels across the entire image and achieve refined predictions through the energy minimization criterion.

[0087] The calculation formula of the fully connected conditional random field is as follows:

[0088] Energy function:

[0089]

[0090] Among them, (x i ,x j ) represents the pixel value at the corresponding position in the difference map, λ1 and λ2 represent the balance coefficient, ψ u represents a single potential energy term, ψ g represents the Gaussian spatial potential energy term, which can ensure spatial continuity; ψ b represents the bilateral potential energy term, which can maintain the sharpness of feature edges. V represents the node set consisting of all pixels in the image; E represents the set of all possible edges under the full connection condition.

[0091] The balancing coefficient λ1 controls the weight of the Gaussian smoothing term, which is used to encourage neighboring pixels to have similar labels, especially those that are spatially close.

[0092] The balance coefficient λ2 controls the weight of the bilateral smoothing term. This term considers not only spatial proximity but also pixel value similarity, thereby better preserving edge information. The balance coefficients λ1 and λ2 can range from 1 to 10. In the present invention, both λ1 and λ2 are set to 1.

[0093]

[0094] In the above formula, d i represents a difference map; represents the probability distribution of pixel i belonging to category c in the difference map, where c∈{0(unchanged),1(changed)}, given by the difference map d i Directly mapped to p i represents the position of pixel i in the spatial coordinate system; p j represents the position of pixel j in the spatial coordinate system; σ g Represents the standard deviation of the Gaussian spatial potential energy term, controlling the smoothing range of the space; I i I represents the multispectral or SAR intensity value of pixel i in the original input image (such as the RGB channel of the optical image or the SAR backscatter coefficient); j represents the multispectral or SAR intensity value of pixel j in the original input image; σ s represents the kernel standard deviation of the spatial distance in the bilateral potential energy term; σ c Represents the kernel standard deviation of the color / intensity differences in the bilateral potential term.

[0095] Fully Connected Conditional Random Field (FCRF) module 410 processes the difference map using a FCRF, eliminating isolated pseudo-change points caused by heterogeneous image registration errors. This smoothes homogeneous areas (such as farmland and water) while preserving sharp edges (such as roads and buildings). FCRF module 410 iteratively optimizes the energy function, bringing the difference map closer to the global optimal solution, significantly improving the spatial consistency and physical plausibility of the detection results.

[0096] The energy function is used to define the evaluation criteria for the system state, and the iterative equation is used to describe the optimization path of the energy function. The iterative equation gradually updates the probability distribution Q so that the energy function E approaches the minimum value, thereby approaching the optimal solution:

[0097]

[0098] represents the state distribution probability of pixel i at iteration step t+1 (such as the probability of belonging to a certain category), which is updated by the normalized exponential function. k represents the index parameter, which traverses two potential function types. k∈{g,b}, corresponding to the smooth potential function ψ g (xi ,x j ) and the boundary potential function ψ b (x i ,x j ). Represents the historical state distribution probability of adjacent pixel j at iteration step t, which is used to constrain the update of pixel i.

[0099] The threshold segmentation module 420 traverses all possible segmentation thresholds between the changed area and the unchanged area based on the adaptive threshold segmentation algorithm (OSTU), calculates the intra-class variance and inter-class variance of the pixel value set under each possible segmentation threshold, selects the optimal threshold that minimizes the intra-class variance and maximizes the inter-class variance, and segments the optimized difference map into the changed area and the unchanged area based on the optimal threshold.

[0100] The threshold segmentation module 420 determines the optimal threshold to segment the foreground and background of the image by maximizing the inter-class variance or minimizing the intra-class variance. In the inter-class variance and intra-class variance in the Otsu method, the foreground is located in the changing area and the background is the non-changing area. The Otsu method here is a classic unsupervised threshold segmentation method that automatically determines the segmentation threshold by maximizing the inter-class difference between the foreground and the background. In remote sensing change detection, it is suitable for scenes with high contrast in the difference map, but in the case of weak changes or noise interference, it needs to be combined with other methods (such as context analysis or deep learning) to improve robustness.

[0101] The intra-class variance represents the degree of discreteness of the grayscale values ​​of pixels within the same class. The calculation formula performed by the threshold segmentation module 420 is:

[0102]

[0103] w0 and w1 represent the proportion of foreground and background pixels, respectively (w0+w1=1); and denote the variance of foreground and background categories respectively.

[0104] The inter-class variance represents the degree of difference between two classes, and the calculation formula performed by the threshold segmentation module 420 is:

[0105] μ0 and μ1 represent the average grayscale of the foreground and background respectively; μ T Indicates the overall average grayscale of the image.

[0106] Total variance relationship:

[0107] represents the total variance, represents minimizing the intra-class variance, It means maximizing the between-class variance.

[0108] The goal of Otsu's method is to maximize the between-class variance Minimize the intra-class variance Due to total variance It is a fixed value.

[0109] The gray level of the difference map processed by the fully connected conditional random field module 410 is L, and the following statistics are defined.

[0110] The normalized distribution of grayscale value histogram is:

[0111] Among them, n k represents the number of pixels of gray level k, Indicates the total number of pixels, p k It represents the probability of occurrence of gray level k in the entire difference map, that is, the proportion of the number of pixels of this gray level to the total number of pixels.

[0112] Cumulative distribution function:

[0113] P(t) represents the cumulative probability from gray level 0 to t; S(t) represents the cumulative intensity weighted sum from gray level 0 to t.

[0114] The threshold segmentation module 420 calculates the optimal threshold using the following optimization formula.

[0115] Category weight recursion:

[0116] Represents the category weight of gray level 0 to t, that is The probability of the corresponding change.

[0117] Represents the category weights of gray levels t+1 to L-1, that is The probability of no change.

[0118] Category mean recursion:

[0119] Represents the category mean of gray levels 0 to t, corresponding to the average intensity of the changing area.

[0120] Represents the category mean of gray levels t+1 to L-1, corresponding to the average intensity of the unchanged area.

[0121] Between-class variance optimization objective:

[0122] Optimal threshold:

[0123] According to a preferred embodiment, the feature extractor 100 includes a cascade of sub-encoders and sub-decoders. Each level of structure includes a sub-encoder and a sub-decoder connected thereto. The sub-encoder of a first level structure is further connected to the sub-encoder and sub-decoder of the next level structure, respectively, so that the sub-decoder of the next level structure outputs a feature map with a gradually reduced dimension based on the mean square error loss function output by the sub-encoder of the previous level structure. The sub-encoder of the next level structure performs dimensionality reduction on the feature map output by the sub-decoder of the previous level structure.

[0124] The present invention uses the feature extractor 100 for feature extraction and denoising, by encoding the input data and noise into a high-dimensional representation, and then decoding it back to the original data through the guidance of the mean square error loss function (MSE Loss), thereby realizing data reconstruction. After training to sufficient fit, the feature extractor 100 can extract more abstract and advanced feature maps, which eliminate redundant pixels. Figure 3 The first feature map of the optical modality shown is similar to edge features, color features, etc. These features can be decoded back into the input image by the decoder part of the feature extractor 100. In essence, they are compressed information, so they are more advanced and abstract.

[0125] like Figure 6 As shown, the first-level structure includes a first sub-encoder 111 and a first sub-decoder 112 connected to each other. The second-level structure includes a second sub-encoder 121 and a second sub-decoder 122 connected to each other. The third-level structure includes a third sub-encoder 131 and a third sub-decoder 132 connected to each other.

[0126] The first layer of the first sub-encoder 111 is a convolutional layer. The convolutional layer has a kernel size of 7*7, a stride of 3, a padding of 2, and an output channel of 40. The second layer is a Sigmoid activation layer.

[0127] The first layer of the first sub-decoder 112 is a convolutional layer with a kernel size of 7*7, a stride of 3, a padding of 2, and an output channel of 30. The second layer is a Sigmoid activation layer.

[0128] The first-level structure of the optical feature extractor 110 is trained for 5,000 rounds using a mean square error loss function. The first-level structure of the optical feature extractor 110 performs a near-fitting step to extract a first-dimensional feature vector, i.e., a 40-dimensional feature vector. This 40-dimensional feature vector can represent more advanced and abstract features.

[0129] like Figure 6 As shown, the first-level structure of the optical feature extractor 110 inputs the 40-dimensional feature vector into the second-level structure for extraction. After receiving the 40-dimensional feature vector, the second-level structure performs approximate fitting on the 40-dimensional feature vector and outputs a second-dimensional feature vector, namely a 30-dimensional feature vector.

[0130] The first layer of the second sub-encoder 121 of the second-level structure is a convolutional layer. The convolutional layer has a kernel size of 1*1, a stride of 1, a padding of 0, and an output channel of 30. The second layer is a Sigmoid activation layer.

[0131] The first layer of the second sub-decoder 122 is a convolutional layer. The convolutional layer has a kernel size of 1*1, a stride of 1, zero padding, and 40 output channels. The second layer is a sigmoid activation layer. The second-level structure is guided by the mean squared error loss function and is also trained for 5000 epochs. The second-level structure is nearly fitted, extracting the second-dimensional feature vector output by the second encoder and feeding it into the third-level structure.

[0132] The third-level structure performs approximate fitting on the second-dimensional feature vector and outputs the third-dimensional feature vector, i.e., the first feature map of the optical mode of the 20-dimensional feature vector.

[0133] The first layer of the third sub-encoder 131 of the third-level structure is a convolutional layer. The convolutional layer has a kernel size of 1*1, a stride of 1, a padding of 0, and an output channel of 20. The second layer is a Sigmoid activation layer.

[0134] The first layer of the third sub-decoder 132 is a convolutional layer. The kernel size of the convolutional layer is 1*1, the stride is 1, the padding is 0, and the output channels are 30. The second layer is a sigmoid activation layer. The third-level structure is guided by the mean square error loss function and is also trained for 5000 epochs.

[0135] The third-level structure is similar to the second-level structure, except that the convolutional layer output channels in the third-level structure are 20, while the convolutional layer output channels in the second-level structure are 30. The third-level structure is trained for 5000 rounds to reach a fitting state, and the extracted 20-dimensional feature vector is the first feature map of the optical modality. After the three-level cascade structure of the optical feature extractor 110 is trained sequentially, the extracted feature map retains the main features in different channels and eliminates most redundant pixels, which facilitates subsequent training and change detection.

[0136] like Figure 6 As shown, the tensor converter 140 is connected to the first sub-encoder 111 and the first sub-decoder 112 in the first-level structure, respectively. The original remote sensing image 500 is converted into a tensor form by the tensor converter 140 and input into the first sub-encoder 111. The first sub-decoder 112 receives the mean square error loss function output by the tensor converter 140 and outputs a feature map with the first dimension reduction. In other words, the tensor converter 140 converts the original remote sensing image 500 into a form suitable for network processing, and then inputs it into the first sub-encoder 111 for preliminary feature extraction. The first sub-decoder 112 receives information from the tensor converter 140 and generates a feature map after the first dimension reduction.

[0137] The first sub-encoder 111 is respectively connected to the second sub-encoder 121 and the second sub-decoder 122 of the second-level structure, so that the second sub-decoder 122 of the second-level structure outputs a feature map with a second reduced dimension based on the mean square error loss function output by the first sub-encoder 111 of the first-level structure. At the same time, the first sub-decoder 112 is connected to the second sub-encoder 121, and the second sub-encoder 121 further performs dimensionality reduction processing on the feature map output by the first sub-decoder 112 of the first-level structure. In other words, this level of structure receives the feature map from the first-level structure and further extracts features while reducing the dimension of the feature map. The second-level structure not only relies on the loss function of the first-level structure to guide training, but also directly uses the feature map of the first-level structure to perform deeper feature extraction.

[0138] The second sub-encoder 121 is connected to the third sub-encoder 131 and the third sub-decoder 132 of the third-level structure, respectively, so that the third sub-decoder 132 of the third-level structure outputs the first feature map with a third reduced dimension based on the mean square error loss function output by the second sub-encoder 121 of the second-level structure. At the same time, the second sub-decoder 122 is connected to the third sub-encoder 131, and the third sub-encoder 131 further reduces the dimension of the feature map output by the second sub-decoder 122 of the second-level structure. In other words, the third-level structure continues to obtain feature maps from the second-level structure and performs more detailed feature extraction and dimensionality reduction on them.

[0139] At each level, the feature map is optimized and adjusted using a sigmoid activation function and convolution operations (e.g., 7x7Conv or 1x1Conv). Ultimately, after step-by-step dimensionality reduction and optimization across the three-level structure, the first feature map of the optical modality is output. This first feature map of the optical modality contains key information extracted from the original remote sensing image 500.

[0140] The cascade structure of the SAR feature extractor 120 is the same as that of the optical feature extractor 110, so the description thereof will not be repeated. The SAR feature extractor 120 outputs a first feature map of the SAR modality.

[0141] According to a preferred embodiment, the shared encoder 210 includes an input layer, an intermediate layer, and an output layer. Figure 10As shown, the input layer of the shared encoder 210 includes a first convolutional layer, a first convolutional attention module, and a first Mamba module. The Mamba module is used to execute the Mamba model. The intermediate layer includes a second convolutional layer and a maximum pooling layer. The output layer includes a third convolutional layer, a second convolutional attention module, and a second Mamba module. The first Mamba module of the input layer is connected to the second convolutional layer of the intermediate layer. The maximum pooling layer of the intermediate layer is connected to the third convolutional layer of the output layer. A residual connection is established between the input data of the second convolutional layer of the intermediate layer and the output data of the maximum pooling layer.

[0142] The input layer expands the number of channels in the first feature map and assigns channel weights and spatial weights, capturing the complex dependencies of the sequence along the channel dimension to extract global information. The input layer first expands the channels of the first feature map generated in the first stage to 64 through the first convolutional layer with a kernel size of 1×1 and a stride of 1, increasing the network depth while integrating channel information. The first convolutional block attention module (CBAM) then assigns channel weights and spatial weights, enabling the model to focus on important features and suppress unnecessary ones. Finally, the first Mamba module captures the complex dependencies of the sequence along the channel dimension to extract global information.

[0143] The middle layer makes a residual connection to the global information output by the input layer to achieve cross-layer feature fusion. Preferably, the middle layer first passes through a second convolutional layer with a kernel size of 3×3, a stride of 1, and a padding of 1, and then passes through a max pooling layer with a kernel size of 3×3, a stride of 1, and a padding of 1. The purpose of the middle layer is to refine the information without changing the channel and spatial dimensions. It makes a residual connection with the output of the input layer to achieve cross-layer feature fusion, enhancing the representation capability of features while preventing gradient vanishing.

[0144] The output layer receives residual-connected information from the intermediate layers and, using the Mamba model, integrates and refines global information while preserving the size of the first feature map. Similar to the input layer, the output layer first expands the channels to 128 a second time through a third convolutional layer with a kernel size of 1×1 and a stride of 1. Global information is then integrated and refined through a second convolutional attention module and a second Mamba module. During the encoding process, the size of the first feature map remains unchanged.

[0145] The shared encoder 210 increases the dimension of the feature map generated by the feature extractor 100 layer by layer, learning and representing more complex features in the high-dimensional common domain.

[0146] The data mapping of the shared encoder 210 is E1(·):

[0147] Z1 and Z2 represent variables in a high-dimensional public domain with consistent statistical distributions, and They are respectively the first feature map of the optical mode and the first feature map of the SAR mode obtained by mapping the original remote sensing image 500 through processing by the feature extractor 100.

[0148] The core innovation of the present invention is to construct a Mamba model based on the Selective Structured State Space Model (Mamba), such as Figure 8 As shown in Figure 2. The Mamba model effectively achieves global semantic integration through a dynamic parameter adjustment mechanism. Figure 8 As shown in Figure 2, the Mamba model is optimized in the UM-Net architecture through the following innovative principles:

[0149] 1) Sequence reconstruction: Expand the three-dimensional input feature map into a sequence structure to form a sequence of length [C, H × W] (C represents the input feature map dimension, H represents the input feature map height, and W represents the input feature map width).

[0150] 2) Mamba processing sequence: Construct a state transition equation SSM with input dependence, which consists of the input sequence x, the potential state space h and the output sequence y:

[0151] y k =Ch k .

[0152] Among them, the dynamic parameter matrix The projection matrix C is generated by linear gating network mapping, which is based on the current input x k The parameters of h are adaptively adjusted to form a state transition mechanism with memory-forgetting characteristics. k represents the position index of the sequence element (time step or spatial position). k represents the hidden state at step k; x k represents the input vector of step k; y k Represents the output vector of the k-th step.

[0153] 3) Stabilized feature processing architecture: A layer normalization (Lnorm) unit is deployed after Mamba to control the stability of feature distribution through the normalization-activation function structure and introduce nonlinear expression capabilities using the GeLU activation function.

[0154] 4) Information Path Protection Mechanism: This mechanism constructs a residual connection, which fuses the original input feature map with the output processed by the stabilized feature processing architecture to produce a fitted sequence and fitted feature map. This design effectively overcomes the information attenuation issue of the Mamba model and ensures the integrity of key features.

[0155] According to a preferred embodiment, the optical decoder 220 assigns channel weights and spatial weights based on a convolutional attention module, extracts global information of the first feature map based on a Mamba model, and outputs a second feature map for calculating differences based on a Sigmoid activation function.

[0156] The optical decoder 220 and the SAR decoder 230 map and reconstruct the second feature maps of the optical and SAR modes, respectively. The optical decoder 220 processes the feature maps sequentially through the input layer, intermediate layer, and output layer, gradually reducing the feature map dimension to maintain consistency between the input and output.

[0157] like Figure 12 As shown, the optical decoder 220 and the SAR decoder 230 have the same structure, and the optical decoder 220 is taken as an example for structural description.

[0158] The input layer of the optical decoder 220 includes a convolutional layer and a convolutional attention module. The intermediate layer includes a convolutional layer and a convolutional attention module. The output layer includes a convolutional layer, a convolutional attention module, and a Mamba module. The output layer is connected to a Sigmoid activation function module.

[0159] First, the input layer includes the fourth convolutional layer and the third convolutional attention module. The fourth convolutional layer with a kernel size of 1×1 and a stride of 1 reduces the number of channels in the encoded first feature map to 64. The third convolutional attention module then performs the first distribution of spatial weights and channel weights.

[0160] The middle layer includes the fifth convolutional layer and the fourth convolutional attention module. The fifth convolutional layer with a kernel size of 3×3, a stride of 1, and a padding of 1 learns abstract features, and then the fourth convolutional attention module performs a second distribution of spatial weights and channel weights.

[0161] The output layer consists of the sixth convolutional layer, the fifth convolutional attention module, and the third Mamba module. The sixth convolutional layer, with a kernel size of 1×1 and a stride of 1, matches the number of channels to the input feature map. The fifth convolutional attention module then performs a third distribution of spatial and channel weights. The third Mamba module integrates global information into the feature map after this third distribution and then uses a sigmoid activation function to output the second optical modality feature map used for disparity calculation.

[0162] The data mapping of the optical decoder 220 is D1(·):

[0163] represents the reconstructed second characteristic map of the optical modality; Represents the mapped second characteristic map of the optical modality.

[0164] The structure of the SAR decoder 230 is identical to that of the optical decoder 220 and will not be further described here. However, the SAR decoder 230 is trained to have different weight parameters than the optical decoder 220 in order to distinguish the optical modality second feature map from the SAR modality second feature map.

[0165] The data mapping of the SAR decoder 230 is D2(·):

[0166] Represents the reconstructed second characteristic map of SAR mode; Represents the mapped SAR modality second characteristic map.

[0167] According to a preferred embodiment, the first, second, third, fourth and fifth convolutional attention modules in the codec network 200 belong to the convolutional attention modules of the same structure. Figure 7 As shown, the convolutional attention module includes a channel weight calibration unit 221 and a spatial salient area enhancement unit 222.

[0168] The channel weight calibration unit 221 extracts the maximum pooling features and the average pooling features of each channel in the feature map of the original input, and fits the maximum pooling features and the average pooling features based on the multi-layer perceptron to obtain the channel weights that reflect the relative importance of each channel in the current task. The channel weights are multiplied by the feature map of the original input to obtain the output after weight calibration. In other words, the input feature map of the channel weight calibration unit 221 is the output of the global maximum pooling along the spatial dimension and the output of the global average pooling along the spatial dimension. The output is the global pooling information of (C*1*1). The multi-layer perceptron is used to learn the two outputs. The learned outputs are added and activated by the Sigmoid activation function to obtain the channel weights, which are multiplied by the original input to obtain the weight calibration output.

[0169] The spatially salient region enhancement unit 222 performs maximum pooling and average pooling operations on different spatial regions of the original input feature map, respectively, to obtain the maximum pooling features and the average pooling features of each region, and performs splicing and fitting on the maximum pooling features and the average pooling features of each region based on the convolution kernel to obtain a fitting feature map (7*7 convolution fitting feature map); calculates the spatial weight of each spatial position of the fitting feature map; and multiplies the spatial weight with the feature map of the original input to obtain the output after region enhancement (region enhancement output).

[0170] For the first convolutional attention module in the shared encoder 210, its channel weight calibration unit 221 is used to extract the two-way channel statistics in the first feature map, perform nonlinear fusion on the two-way channel statistics, and establish the correlation attention coefficient between the channels. After the channel weight calibration unit 221 weights and normalizes the correlation attention coefficient, it generates a channel attention weight matrix, performs channel-level product with the feature map input by the convolution layer, and generates an adaptively calibrated feature map (weight calibration output).

[0171] The step of extracting the two-way channel statistics from the first feature map by the channel weight calibration unit 221 includes: performing global maximum pooling (GlobalMax Pooling) and global average pooling (GlobalAveragePooling) along the spatial dimension respectively to generate two feature vectors with channel discrimination.

[0172] Specifically, the spatially salient region enhancement unit 222 is used to perform maximum pooling and average pooling along the channel dimension to obtain dual-channel features, which are then concatenated to form a composite feature map. Cross-channel feature fusion is performed by mapping the feature map using a 7*7 convolution fit, and a spatial attention distribution heat map is generated through nonlinear mapping. The spatial attention distribution heat map is normalized using a Sigmoid activation function to form a spatial weight matrix. The spatial weight matrix is ​​then point-by-point multiplied by the adaptively calibrated feature map in the spatial domain to enhance focus on the changing region.

[0173] "Concatenation" specifically refers to the feature fusion operation of connecting tensors along the channel dimension in deep learning. Specifically, after applying max pooling and average pooling to the input feature map, two single-channel feature maps with the same spatial dimensions (H×W) but a channel dimension of 1 are obtained. These two feature maps are then superimposed along the channel axis (usually the first dimension of the tensor) to generate a composite feature map with the channel dimension expanded to 2 (2×H×W).

[0174] This operation overcomes the representational limitations of single pooling methods, fully preserving the complementary information between the maximum response feature and the global statistical features, and providing an explicit correlation structure of heterogeneous features for cross-channel interaction in the subsequent 1×1 convolutional layer. It is worth noting that this channel concatenation, unlike the implicit fusion of element-wise addition, explicitly maintains feature independence through a dimensionality expansion mechanism, enabling neural networks to more effectively learn nonlinear mapping relationships between different features.

[0175] The advantages achieved through the operation of the convolutional attention module include: first, the dual-path pooling strategy synchronously captures the extreme value and mean distribution characteristics of features, enhancing the robustness of statistical representation; second, the channel-spatial attention cascade architecture realizes multi-granularity optimization of feature selection, breaking through the limitations of a single attention mechanism; third, the lightweight design ensures that the module can be embedded in any level of the existing network, achieving an average accuracy improvement of 2.7% with only a maximum increase of 0.5% in the number of parameters; finally, the adaptive weight allocation mechanism enables the network to dynamically suppress noise interference and enhance effective features.

[0176] In order to accurately detect the changed area, the performance robustness of the mapping and reconstruction of the second stage codec network 200 is crucial. The present invention designs a set of loss functions for the network to guide unsupervised learning. The calculation formula of the loss function is:

[0177]

[0178] The input of the codec network 200 is the first feature map of the optical mode extracted by the optical feature extractor 110 and the first feature map of the SAR mode extracted by the SAR feature extractor 120. The modalities and contents are different. It is not advisable to directly detect the changes in the content. The modalities should be unified first. Based on the idea of ​​same-domain comparison, It is the self-mapping loss in the optical domain and SAR domain, which is used to guide the encoder-decoder to have the ability to map the original domain feature map based on the original domain.

[0179]

[0180] It is the cross-mapping loss of the conversion between the optical domain and the SAR domain, which is used to guide the codec to have the ability to map foreign domain feature maps based on the original domain.

[0181]

[0182] middle, and They are and The mapping of different modalities is formed again through the codec network 200.

[0183]

[0184] is the reconstruction loss guided by the difference map DI,

[0185]

[0186] in, The third characteristic map of the optical modality is guided by the difference map DI, and the change area indicated by DI is reconstructed using the second characteristic map of the optical modality Filling, unchanged area uses the mapped optical modality second characteristic map filling, The same applies to the combination of .

[0187]

[0188]

[0189] exist and In the case of convergence, the reconstructed feature map mapped to the optical domain and SAR domain and The comparison conditions are in place. Taking the SAR domain as an example, the core idea of ​​reconstruction loss is explained. The unchanged area indicated by the difference map DI is in the label With label are the same in With label are the same, based on this, take The unchanged area and The reconstructed feature map is composed of the change regions With label The advantage of doing so is that the difference map participates in the back-propagation process, which promotes the convergence of the difference map in unsupervised learning and obtains stable and reliable change detection results.

[0190] The present invention describes the experimental results of this system as follows.

[0191] For quantitative analysis and evaluation indicators:

[0192] Based on the final change detection result image, its pixels can be classified into four categories: true positive TP, false positive FP, false negative FN, and true negative TN. Based on these four categories, the following quantitative analysis indicators are available:

[0193] Overall accuracy (OA) is the overall proportion of correctly classified pixels:

[0194]

[0195] Precision (Pre) is the ratio of pixels correctly classified as changed to all pixels classified as changed:

[0196]

[0197] Recall (Rec) is the ratio of pixels that are correctly classified as changed to all actual changed pixels:

[0198]

[0199] F1-score (F1) is the harmonic mean of Pre and Rec, which is used to measure the overall performance of the classification algorithm:

[0200]

[0201] Kappa coefficient (KC) is a more comprehensive evaluation of change detection results. It takes into account the consistency of random classification probabilities. A higher KC indicates a better classification effect:

[0202]

[0203] The calculation method of PR is as follows:

[0204]

[0205] The experimental contents and analysis of the present invention are as follows.

[0206] Simulation experiments were conducted on different heterogeneous remote sensing image datasets using existing methods and the method provided by this invention. Existing unsupervised methods include convolutional neural network methods (SCCN) and CAA, and graph neural network methods (SDA), VDF, and IST-CRF.

[0207] SCCN (Symmetric Convolutional Coupling Network) is a symmetric convolutional coupling network that achieves feature space alignment of heterogeneous remote sensing images through a two-branch convolutional network. CAA (Context-Aware Attention) is a context-aware attention mechanism that dynamically assigns feature weights based on local context associations. SDA (Spectral Domain Adaptation) is a spectral domain adaptation method that uses adversarial training to reduce the spectral distribution differences of heterogeneous images. VDF (Variational Deep Feature) is a variational deep feature generation method that improves cross-modal feature robustness through probabilistic modeling. IST-CRF (Iterative Spatial-Temporal Conditional Random Field) is an iterative spatiotemporal conditional random field that jointly optimizes the differences between the spatiotemporal and spectral domains.

[0208] Figure 5 Is a diagram of a dataset of image samples. Figure 5In the figure, the first column of images (a) are images before the event, the second column of images (b) are images after the event, and the third column of images (c) are labels of manually marked change areas. The first to third rows are the Italian dataset, the Gulast dataset, and the Ottawa dataset, respectively.

[0209] Figure 11 Table 1 shows the experimental results of the method of the present invention and the prior art method. Figure 9 It is a schematic diagram of the experimental results of change detection and traditional algorithms. Figure 9 In the figure, TP pixels are represented in white, TN pixels are represented in black, FP pixels are represented in green, and FN pixels are represented in red. Each column of images shows the detection results of different methods using the Italian, Gulast, and Ottawa datasets, respectively. The first row of images shows the detection results of different methods on the Italian dataset. The second row of images shows the detection results of different methods on the Gulast dataset. The third row of images shows the detection results of different methods on the Ottawa dataset.

[0210] The first row of images in the first column is a reference image of the Italian dataset, the second row of images in the first column is a reference image of the Gulast dataset, and the third row of images in the first column is a reference image of the Ottawa dataset. The second column of images (b) is a detection result image using the CAA method. The third column of images (c) is a detection result image using the SCCN method. The fourth column of images (d) is a detection result image using the SDA method. The fifth column of images (e) is a detection result image using the VDF method. The sixth column of images (f) is a detection result image using the IST-CRF method. The seventh column of images (g) is a detection result image using the method of the present invention.

[0211] against Figure 11 An in-depth analysis of the quantitative indicators in Table 1 is as follows.

[0212] Sardinia dataset:

[0213] The proposed method's KC value (75.46%) is close to that of IST-CRF (79.22%), but its Precision (88.48%) is significantly higher than that of IST-CRF (83.35%), demonstrating that the proposed method is more accurate in determining change regions and has a lower false positive rate (FP). The slightly lower KC value may be due to the concentrated distribution of change regions in the Sardinia data, making traditional CRF post-processing more direct in optimizing spatial continuity.

[0214] Technical Relevance: The energy function of Dense CRF constrains spatial continuity through Gaussian potential energy, but its modeling of global information is limited. The long-range modeling capability of the Mamba model has greater potential in complex scenarios.

[0215] Gurast Dataset:

[0216] The KC value (82.95%) of the proposed method is 6.1% higher than that of IST-CRF (76.85%), and the F1-score (84.78%) is 5.93% higher, indicating that the model has better comprehensive performance in complex surfaces (such as vegetation cover changes and local building damage).

[0217] Attribution analysis: The three-stage denoising of the feature extractor 100 (Figure d) effectively eliminates speckle noise in SAR images, and the shared weight design of the codec (Figure b) forces the optical and SAR features to be aligned in the common domain, reducing feature offsets caused by modal differences.

[0218] Ottawa Dataset:

[0219] The KC value (93.22%) and F1-score (94.27%) of the present invention are fully ahead, and the Rec (92.80%) is close to perfect, indicating that the model has an extremely low missed detection rate (FN) in extreme scenarios (such as flooded areas).

[0220] Core advantage: reconstruction loss function driven by difference map The global modeling capability of the Mamba model ensures the consistent detection of large-scale changes (such as water diffusion) by directly optimizing the localization of change areas through dynamic reconstruction.

[0221] against Figure 11 A fine-grained comparison of the visualization results is as follows.

[0222] CAA and SCCN (CNN-like methods):

[0223] Problem: Green false positives (FP) are concentrated in areas with similar textures (such as the boundary between farmland and bare land), and red missed positives (FN) are mostly due to the loss of local features (such as damage to small buildings).

[0224] Root cause: The local receptive field of CNN cannot model long-range dependencies, resulting in insensitivity to global information.

[0225] Improvements and limitations of SDA, VDF, and IST-CRF (graph neural network and CRF-like methods): Graph structure modeling (SDA, VDF) improves spatial correlation, but sporadic green false detections (such as road edges) expose sensitivity to noise; IST-CRF optimizes spatial continuity through post-processing, but relies on manual parameter adjustment, and its generalization is limited.

[0226] A typical example: In the Gulast dataset, IST-CRF mistakenly judges shadows as changes (green spots) in vegetation coverage areas, while the present invention avoids such errors due to Mamba's global modeling.

[0227] The method of the present invention is Figure 9 The advantages reflected in column (g) include:

[0228] TP region coherence: The change boundary is clear, thanks to the global information capture of the Mamba model and the edge sharpening of the Dense CRF.

[0229] FP / FN is significantly reduced: Feature Extractor 100 denoising ( Figure 7 The column a in the image contains more noise) to eliminate the coherent speckle noise of the SAR image, and the difference map loss Suppress light interference in optical images.

Claims

1. A heterogeneous remote sensing image change detection system based on the Mamba model, characterized in that: include: A feature extractor (100) is used to extract first feature maps of the optical mode and the SAR mode respectively; A codec network (200) is configured to map and reconstruct the first feature map based on a Mamba model to generate second feature maps of an optical mode and a SAR mode; A change detector (300) subtracts the second feature maps of the same modality and calculates the L2 norm along the channel dimension, and performs weighted fusion of the L2 norm calculation results of different modalities to obtain a difference map; A difference map processor (400) optimizes the difference map based on a fully connected conditional random field method, and segments the optimized difference map into a changed area and a unchanged area based on an optimal threshold selected by an adaptive threshold segmentation algorithm.

2. The system according to claim 1, wherein: The codec network (200) includes a shared encoder (210), an optical decoder (220) and a SAR decoder (230), The first feature map of the optical mode and the first feature map of the SAR mode are extracted global information by the shared encoder (210) based on the global receptive field of the Mamba model and mapped to a high-dimensional common domain; The optical decoder (220) reconstructs the first feature map of the optical modality in the high-dimensional public domain into a second feature map of the optical modality, and maps the first feature map of the SAR modality into the second feature map of the optical modality; The SAR decoder (230) reconstructs the first feature map of the SAR modality in the high-dimensional public domain into a second feature map of the SAR modality, and maps the first feature map of the optical modality into the second feature map of the SAR modality.

3. The system according to claim 1 or 2, characterized in that The feature extractor (100) comprises a cascaded sub-encoder and a sub-decoder, Each level of structure includes a sub-encoder and a sub-decoder connected to it. The sub-encoder of the first-level structure is connected to the sub-encoder and sub-decoder of the next-level structure respectively, so that the sub-decoder of the next-level structure outputs a feature map with a gradually reduced dimension based on the mean square error loss function output by the sub-encoder of the previous-level structure; The sub-encoder of the next-level structure performs dimensionality reduction on the feature map output by the sub-decoder of the previous-level structure.

4. The system according to any one of claims 1 to 3, characterized in that: The shared encoder (210) in the codec network (200) includes an input layer, an intermediate layer, and an output layer, The input layer expands the number of channels of the first feature map and assigns channel weights and spatial weights, capturing complex dependencies of the sequence along the channel dimension to extract global information; The intermediate layer performs a residual connection on the global information output by the input layer to achieve cross-layer feature fusion; The output layer receives the residual connected information of the intermediate layer, and integrates and refines the global information based on the Mamba model while keeping the size of the first feature map unchanged.

5. The system according to any one of claims 1 to 4, characterized in that: The input layer of the shared encoder (210) in the codec network (200) first expands the first feature map channel based on the first convolutional layer, assigns channel weights and spatial weights based on the first convolutional attention module, and extracts global information of the first feature map based on the Mamba model in the first Mamba module; The middle layer of the shared encoder (210) expands the local receptive field of the second convolutional layer based on the maximum pooling layer and enhances the fitting ability of the middle layer based on the residual connection; The output layer of the shared encoder (210) expands the first feature map channel for a second time based on the third convolutional layer, redistributes channel weights and spatial weights based on the second convolutional attention module, and extracts global information of the first feature map again based on the Mamba model in the second Mamba module.

6. The system according to any one of claims 1 to 5, characterized in that: The optical decoder (220) or SAR decoder (230) in the codec network (200) comprises: Input layer, including the fourth convolutional layer and the third convolutional attention module; The middle layer includes the fifth convolutional layer and the fourth convolutional attention module; The output layer includes the sixth convolutional layer, the fifth convolutional attention module, and the third Mamba module; A Sigmoid activation function module normalizes the feature map output by the third Mamba module, and is used in the optical decoder (220) to output a second feature map of the optical modality, and in the SAR decoder (230) to output a second feature map of the SAR modality.

7. A heterogeneous remote sensing image change detection method based on the Mamba model, characterized in that: The method comprises: Extracting the first feature maps of the optical mode and the SAR mode respectively; Mapping and reconstructing the first feature map based on the Mamba model to generate second feature maps of the optical mode and the SAR mode; Subtracting the second feature maps of the same modality and calculating the L2 norm along the channel dimension, and weightedly fusing the L2 norm calculation results of different modalities to obtain a difference map; The difference map is optimized based on the fully connected conditional random field method, and the optimized difference map is segmented into changed areas and unchanged areas based on the optimal threshold selected by the adaptive threshold segmentation algorithm.

8. The method according to claim 7, characterized in that The method comprises: The first feature map of the optical mode and the first feature map of the SAR mode are used to extract global information based on the global receptive field of the Mamba model and mapped to a high-dimensional common domain; Reconstructing the first feature map of the optical modality in the high-dimensional public domain into a second feature map of the optical modality, and mapping the first feature map of the SAR modality into the second feature map of the optical modality; The first feature map of the SAR modality in the high-dimensional public domain is reconstructed into a second feature map of the SAR modality, and the first feature map of the optical modality is mapped into the second feature map of the SAR modality.

9. The method according to claim 7 or 8, characterized in that The steps of extracting the first feature maps of the optical mode and the SAR mode respectively include: Extracting a first-dimensional feature vector from the input original remote sensing image (500) of each modality; Approximately fitting the first-dimensional feature vector to output a second-dimensional feature vector; The second-dimensional feature vector is approximately fitted to output a first feature map of the third-dimensional feature vector.

10. The method according to any one of claims 7 to 9, characterized in that The shared encoder (210), optical decoder (220) and SAR decoder (230) in the codec network (200) are all provided with a convolutional attention module; The convolutional attention module includes: A channel weight calibration unit (221) extracts the maximum pooling feature and the average pooling feature of each channel in the feature map of the original input, fits the maximum pooling feature and the average pooling feature based on a multi-layer perceptron, obtains a channel weight reflecting the relative importance of each channel in the current task, and multiplies the channel weight with the feature map of the original input to obtain a weight-calibrated output; The spatially significant region enhancement unit (222) performs maximum pooling and average pooling operations on different spatial regions of the original input feature map to obtain maximum pooling features and average pooling features of each region, and performs splicing and fitting on the maximum pooling features and average pooling features of each region based on the convolution kernel to obtain a fitting feature map; calculates the spatial weight of each spatial position of the fitting feature map; and multiplies the spatial weight with the feature map of the original input to obtain the output after region enhancement.

Images