A remote sensing semantic change detection method based on mixed Mamba-Transformer and cross attention fusion

Abstract

The application discloses a remote sensing semantic change detection method based on mixed Mamba-Transformer and cross attention fusion, inputs a double-time-phase remote sensing image into a weight-shared mixed Mamba-Transformer encoder to extract multi-level double-time-phase features; a complementary space-time interaction fusion module performs space-time projection, parallel Mamba scanning and cross attention calculation on the features to generate change perception fusion features; a change detection decoder combines the fusion features and the multi-level features to output a binary change mask, and two parallel semantic segmentation decoders generate double-time-phase land cover semantic probability maps respectively; and the change mask is used to constrain the semantic probability maps in space to obtain a semantic change detection result. The application considers global modeling efficiency and local detail description ability through a mixed architecture, effectively suppresses environmental pseudo changes through parallel Mamba and cross attention fusion, and guarantees the consistency of change areas and semantic categories from the mechanism through cross-task collaborative decoding, so that the robustness and precision of remote sensing semantic change detection in a complex environment are significantly improved.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and computer vision technology, and in particular to a remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion. Background Technology

[0002] In Earth observation applications, accurately locating regions affected by semantic changes and identifying their evolutionary nature is a core requirement. Semantic Change Detection (SCD) aims to simultaneously extract changed regions and identify "from-to" land cover transition information from dual-temporal remote sensing data. In recent years, deep learning has greatly promoted the development of this field, with Convolutional Neural Networks (CNNs) becoming the mainstream backbone network due to their powerful local feature extraction capabilities.

[0003] However, existing models still face the following serious challenges in practical applications: First, limited by the local receptive field of the convolutional kernel, CNNs struggle to capture complex global spatiotemporal dependencies. While the introduction of the Transformer addresses the global modeling problem through self-attention, its computational complexity increases quadratically with image resolution. When processing large-scale, high-resolution remote sensing sequences, the computational bottleneck becomes severe. Therefore, balancing global modeling capabilities with computational efficiency is a key challenge.

[0004] Secondly, remote sensing images are highly susceptible to interference from irrelevant environmental factors such as fluctuations in lighting conditions, seasonal vegetation succession, and occlusion. These visual differences can easily be confused with true semantic changes, leading to a large number of false change detections or misclassifications in traditional models.

[0005] Furthermore, existing methods lack sufficient consideration of the semantic correlation between two time phases, often leading to contradictory output results. For example, a region may be marked as "changed" in a binary mask, but the semantic categories predicted by the two time phases are exactly the same.

[0006] Therefore, there is an urgent need for a semantic change detection technology that can achieve robust global modeling while being computationally efficient, effectively suppressing environmental interference, and ensuring consistency of semantic output across multiple time phases. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a remote sensing semantic change detection method (HMTNet) based on a hybrid Mamba-Transformer dual-branch architecture and cross-attention fusion. The purpose of this invention is to overcome the shortcomings of existing remote sensing change detection models in balancing global modeling efficiency and fine-grained feature extraction, and to design an end-to-end automatic detection technology with strong environmental robustness and high semantic consistency.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion, comprising: Acquire the dual-temporal remote sensing images to be detected, and input the dual-temporal remote sensing images into the weight-shared hybrid Mamba-Transformer encoder to extract multi-level dual-temporal features; The multi-level dual-temporal features extracted by the encoder are input into the complementary spatiotemporal interactive fusion module. Through spatiotemporal projection, parallel Mamba scanning and complementary cross-attention calculation, long-distance change perception fusion features are extracted. The long-distance change perception fusion feature and multi-level dual-temporal feature are input into the change detection decoder to generate a binary change probability map, and the Argmax operation is performed on the binary change probability map to obtain a binary change mask. The multi-level dual-temporal features extracted by the encoder are input into two parallel semantic segmentation decoder branches to generate land cover semantic probability maps corresponding to the two observation time phases respectively; The binary change mask is used to spatially constrain the land cover semantic probability map, so that the semantic change category is only identified within the detected binary change area, and the final remote sensing image semantic change detection result is output.

[0009] In one embodiment of the present invention, the hybrid Mamba-Transformer encoder employs a four-stage hierarchical architecture to extract features, specifically including: The original input image first passes through the Stem module, which consists of consecutive convolutional layers with a stride of 2, mapping the image from pixel space to embedding space and outputting downsampled feature blocks. The downsampled features pass through the first and second stages of the encoder sequentially. These stages consist of residual blocks based on convolutional neural networks, used to extract local texture and geometric structures from the high-resolution feature maps. The calculation formula for each residual block is as follows:

[0010] Where GELU represents the Gaussian error linear unit activation function, and BN represents batch normalization; After the second stage of processing, the features are sequentially processed through the third and fourth stages of the encoder. The third and fourth stages consist of Mamba-Transformer blocks. Each Mamba-Transformer block includes a Mamba block based on a state-space model and a Transformer block containing a global self-attention mechanism. The features first enter the Mamba block based on the state-space model for global information aggregation, and then enter the Transformer block containing a global self-attention mechanism to extract features.

[0011] In one embodiment of the present invention, the feature extraction process of the complementary spatiotemporal interaction fusion module includes: Spatiotemporal projection: By performing feature concatenation along the channel dimension and combining it with spatial interleaving, bi-temporal features are mapped onto a unified coding space to obtain projected features; Parallel Mamba scanning and feature aggregation: The projected features are input into two parallel Mamba blocks for long sequence modeling. Subsequently, the features extracted from the two parallel branches are channel-wise concatenated to synthesize enhanced change-aware joint features. G ; Cross-attention mapping: Through skip connections, the original bi-temporal features bypass the Mamba block and are directly input into the complementary cross-attention unit to generate keys. K Sum V Vectors, while simultaneously incorporating the enhanced change-aware joint features G As a query Q The vector input to the complementary cross-attention unit is calculated using the following formula:

[0012] in, is the scaling factor for the dimension of the key vector; Feature reconstruction: The output of the cross-attention unit is input into the convolutional layer for dimensionality reduction to obtain the reconstructed fused features, which are then input into the subsequent decoder network.

[0013] In one embodiment of the present invention, model training is also included, wherein the model is iteratively optimized based on the training set data during the training phase, and the total loss function used for training is... Binary change detection loss and bi-temporal semantic classification loss , The weighted sum is calculated using the following formula:

[0014] in, The weighting parameters correspond to the loss; binary change detection loss. Both the bi-temporal semantic classification loss and the loss are composed of cross-entropy loss and Lovasz-softmax loss.

[0015] In one embodiment of the present invention, the binary change detection loss The calculation formula is:

[0016] in, Loss detection for binary changes Cross-entropy loss, Loss detection for binary changes The Lovasz-softmax loss; The calculation formula is:

[0017] in, N Total number of pixels This represents the unique heat representation of the label. This represents the classification probability value output by the network.

[0018] In one embodiment of the present invention, the cross-entropy loss of the bi-temporal semantic classification The calculation formula is:

[0019] in, N Total number of pixels This represents the unique heat representation of the label. The classification probability value output by the network. This represents the total number of land cover categories.

[0020] In one embodiment of the present invention, the parallel Mamba scan includes: inputting the projection features into a forward Mamba block and a backward Mamba block respectively, wherein the forward Mamba block performs sequence modeling along the original spatial flattening direction, and the backward Mamba block performs sequence modeling along the reverse spatial flattening direction, so as to capture long-distance dependencies in different spatial directions.

[0021] Secondly, the present invention provides a remote sensing semantic change detection system based on hybrid Mamba-Transformer and cross-attention fusion, for implementing the aforementioned remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion, the system comprising: A hybrid Mamba-Transformer encoder is used to receive a dual-temporal remote sensing image to be detected and extract multi-level dual-temporal features; wherein, the hybrid Mamba-Transformer encoder shares weights when processing dual-temporal images; The complementary spatiotemporal interactive fusion module is used to receive multi-level dual-temporal features and extract long-distance change perception fusion features through spatiotemporal projection, parallel Mamba scanning and complementary cross-attention calculation. A change detection decoder is used to receive the long-distance change-aware fusion features and multi-level dual-temporal features, and generate a binary change mask; A semantic segmentation decoder is used to receive multi-level dual-temporal features and generate a land cover semantic probability map corresponding to the two observation time phases. The mask constraint unit is used to spatially constrain the land cover semantic probability map using the binary change mask, so that the semantic change category is only identified within the detected binary change area, and outputs the final remote sensing image semantic change detection result.

[0022] Thirdly, the present invention provides a computer-readable storage medium storing computer instructions that are executed by a processor as described in the remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion.

[0023] Fourthly, the present invention provides a computer program product storing computer instructions, which are executed by a processor as described in the remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion.

[0024] Compared with the prior art, the present invention has at least the following beneficial effects: 1. Balancing Global Modeling Efficiency with Local Detail Representation: The hybrid Mamba-Transformer encoder of this invention utilizes the state-space model (Mamba) to achieve a long-range global receptive field with linear computational complexity, completely overcoming the computational bottleneck of pure Transformer structures when processing high-resolution remote sensing images. Simultaneously, the Transformer's self-attention mechanism enhances local details of the global features output by Mamba, enabling the model to accurately capture the boundaries and fine-grained textures of complex terrain features. Thus, the network achieves a good balance between global contextual modeling and local fine-grained feature extraction, significantly improving the completeness and accuracy of multi-scale terrain feature change perception.

[0025] 2. Significantly Enhanced Robustness and Resistance to False Changes in Complex Environments: The Complementary Spatiotemporal Interaction Fusion Module (CSTI-FM) performs multi-directional long-sequence scanning along different spatial flattening directions using parallel Mamba, fully aggregating complementary global contextual information to form a change-aware prior. Subsequently, a cross-attention mechanism guided by joint features is used, with this prior as the query vector and the original dual-temporal features as the key-value vector, to achieve deep semantic alignment in the original high-resolution space. This design effectively suppresses false change noise caused by external factors such as illumination fluctuations, seasonal vegetation succession, and occlusion, significantly enhancing the detection robustness and reliability of the method under different imaging conditions and natural environments.

[0026] 3. Enhanced semantic consistency in multi-task collaboration: This invention employs a cross-task collaborative dual-branch decoding architecture, enabling the change detection branch and the semantic segmentation branch to share the same multi-level features extracted by the same encoder. Furthermore, spatial constraints are imposed on the bi-temporal semantic probability map using a binary change mask, ensuring that semantic change categories are identified only within the detected change regions. This forced interaction strategy eliminates the logical contradiction of "regions labeled as changed but with the same semantic category before and after" at the structural level, significantly improving the spatiotemporal consistency and logical self-consistency of semantic change detection results, ultimately achieving industry-leading detection accuracy. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the overall process of the present invention; Figure 2 This is a schematic diagram of the overall network architecture of HMTNet of the present invention; Figure 3 This is a schematic diagram of the hybrid Mamba-Transformer encoder in this invention; Figure 4 This is a schematic diagram of the internal structure of the Mamba block in this invention; Figure 5 This is a schematic diagram of the Complementary Spatiotemporal Interaction Fusion Module (CSTI-FM) in this invention; Figure 6 This is a schematic diagram of the cross-attention unit structure guided by joint features in this invention; Figure 7 This is a schematic diagram of the decoder network structure in this invention; Figure 8This is a visualization of the method of the present invention on the SECOND dataset; Figure 9 This is a visualization of the method of the present invention on the Landsat-SCD dataset. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited to the following embodiments. For those skilled in the art, equivalent substitutions, combinations or conventional modifications made without departing from the concept of the present invention should all fall within the scope of protection of the present invention.

[0030] This invention utilizes the latest State Space Models (SSMs) technology. In particular, the Selective Scan Mechanism, represented by Mamba, employs an improved selective state space model (S6) to maintain linear computational complexity proportional to the sequence length. Simultaneously, it achieves global receptive field modeling. In the semantic change detection (SCD) task, Mamba can compress and propagate cross-temporal context states with low memory overhead, and extract local details by combining the self-attention mechanism of Transformer, providing a solid technical foundation for processing strongly spatiotemporally correlated bi-temporal remote sensing sequences.

[0031] like Figure 1 and Figure 2 As shown, this invention provides a remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion, comprising the following steps: Step 1. Preprocessing and Input of Dual-Temporal Remote Sensing Images Step 1.1 In the Semantic Change Detection (SCD) task, the training dataset is first formally defined as follows: ,in and Represents a dual-temporal image. This represents the corresponding land cover label. Here, This represents the total number of land cover categories. and These represent the height and width of the image, respectively.

[0032] Step 1.2 The input image first passes through the Stem module, which consists of two consecutive 3x3 convolutional layers with a stride of 2. Its function is to transform the original input image from pixel space... Mapped to dimension The embedded space has an output resolution of The feature blocks. This design provides rich low-level visual representations for subsequent layers while maintaining computational efficiency.

[0033] Step 2. Feature extraction using a hybrid Mamba-Transformer encoder The encoder employs a hierarchical architecture to extract multi-level features from the input image, referencing... Figure 3 .

[0034] Step 2.1 The first and second stages utilize residual blocks based on convolutional neural networks (CNNs). These stages aim to efficiently extract local texture and geometry from high-resolution feature maps. The calculation formula for each residual block is as follows:

[0035]

[0036] in, Represents the activation function of the Gaussian error linear unit. This indicates batch normalization. This represents a 3x3 convolution operation. The transition between stages is achieved through a 3x3 convolution downsampler with a stride of 2, which halves the spatial resolution while doubling the channel dimension, laying the foundation for deep semantic perception.

[0037] Step 2.2 Introduces the Mamba-Transformer block in the third and fourth stages. First, the Mamba block is based on the selective state-space model (S6) with linear complexity, whose structure references... Figure 4 This allows for the aggregation of global information within an extremely wide receptive field, capturing the spatiotemporal correlations of large-scale features (such as large building complexes, rivers, and forests). The State-Space Model (SSM) utilizes latent states... Input sequence Mapped to output Its continuous-time dynamic equation is expressed as:

[0038]

[0039] To accommodate the discrete digital signals of remote sensing images, the S6 model introduces a time step. Transforming continuous parameters into discrete parameters and The calculation process for the discretized selective scanning mechanism is as follows:

[0040]

[0041] in, , Through this linear scanning mechanism, Mamba efficiently captures the spatiotemporal correlations of large-scale land features such as large building complexes, rivers, and forests.

[0042] Subsequently, the Transformer block employs a global self-attention mechanism to apply the features of the Mamba output. Perform local detail enhancements. Generate query, key, and value matrices using linear mapping:

[0043] Furthermore, by employing self-attention operations, it precisely focuses on the most distinctive semantic regions, refining the complex boundaries of object evolution:

[0044] in, The weight matrix is ​​a learnable matrix. This is the feature dimension scaling factor.

[0045] Step 3. Extraction and optimization of the Complementary Spatiotemporal Interaction Fusion Module (CSTI-FM) This step aims to achieve deep cross-scale feature interaction and fusion through complementary spatiotemporal interaction, referencing... Figure 5 .

[0046] Step 3.1 Spatiotemporal Projection: Breaking temporal isolation early in feature processing by concatenating elements along the channel dimension. and Spatial interlacing in dimensions provides an initial mapping of bi-temporal features, forcing the model to perceive different temporal states at the same location within a unified encoding space.

[0047] Step 3.2 Parallel Mamba Scan and Feature Aggregation: The joint features after spatiotemporal projection are denoted as... This feature is fed into two parallel Mamba scan paths. Leveraging the linear sequence modeling capabilities of SSM, the model scans along different spatial flattening directions (e.g., forward and backward) to capture long-range global dependencies. The parallel scanning process can be represented as:

[0048]

[0049] in, and These represent independent state-space scanning mechanisms in different directions. Subsequently, these two sets of complementary long-range semantic features are concatenated along the channel dimension to extract enhanced change-aware joint features. :

[0050] This fusion feature By incorporating multi-directional contextual information, it provides more robust prior guidance for subsequent cross-attention mechanisms.

[0051] Step 3.3 Joint Feature-Guided Cross-Attention: Introducing a skip connection mechanism to prevent the loss of original spatial details. Original bi-temporal features ( Bypassing the Mamba block, it directly feeds into the complementary cross-attention unit, the structure of which is referenced. Figure 6 Here, Mamba-enhanced union features. As a query vector ( ), original feature generation key ( ) and value ( Vector. The core calculation formula is:

[0052] in, is the dimensionality scaling factor for the key vectors. Guided by a changing prior, this design performs precise semantic alignment and complementary mapping in the original high-resolution feature space. Finally, dimensionality reduction and feature reconstruction are achieved through convolutional layers, effectively suppressing non-semantic environmental noise caused by illumination and seasonal changes.

[0053] Step 4. Decode and output the cross-task collaborative dual-branch code. This step aims to use a decoder to decode high-dimensional features into detection results, referencing... Figure 7 .

[0054] Step 4.1. Change Detection Branch (CD): Receives multi-level bi-temporal features from the encoder. and The fused features are then used to learn the evolutionary relationship between the two temporal features through a change decoder, generating a binary change probability map. :

[0055] Applying the Argmax operation to the probabilistic graph yields the final binary transformation graph. .

[0056] Step 4.2 Semantic Segmentation Branch (SS): The multi-level features extracted by the encoder are simultaneously fed into two parallel semantic decoder branches (Dec_seg1 and Dec_seg2) to extract detailed semantic attributes for each temporal phase:

[0057] Step 4.3 Finally, apply the masking operation, using a binary transformation mask. Spatially constrain the semantic graph to ensure that semantic changes are identified only within the detected regions of change, thereby obtaining the final "from-to" semantic change result.

[0058] The effects of this invention can be further illustrated by the following experiments.

[0059] 1. Simulation conditions This invention is based on the SECOND dataset (containing 4662 pairs) Validated on high-resolution aerial images and the Landsat-SCD dataset (long-time sequence Landsat images with 30-meter spatial resolution containing up to 10 fine-grained semantic variation types).

[0060] Experimental environment: The operating system is Linux, and NVIDIA A100 GPU is used for training and validation. The programming framework is PyTorch. The Adam optimizer is used, and the initial learning rate is [missing value]. Momentum parameters and The values ​​were set to 0.9 and 0.999 respectively. The batch size was set to 8 pairs of images, and the total number of training steps was 30,000.

[0061] The total loss function for model optimization is composed of the binary change detection (BCD) loss and the semantic change detection (SCD) loss.

[0062] BCD cross-entropy loss formula:

[0063] Combining Lovasz-softmax loss to mitigate class imbalance, the total BCD loss is... .

[0064] For the dual-temporal semantic classification task, calculate the semantic cross-entropy loss:

[0065] Final total loss function L total It is a weighted sum of BCD loss and bi-temporal semantic loss:

[0066] The evaluation metrics used are overall accuracy (OA), mean intersection-over-union ratio (mIoU), separation Kappa coefficient (SeK), and... Fractions. The formulas for calculating mIoU and SeK are:

[0067]

[0068]

[0069]

[0070] 2. Quantitative analysis To verify the effectiveness of the proposed remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion, comprehensive quantitative comparative experiments were conducted on two publicly available benchmark datasets (SECOND and Landsat-SCD). Evaluation metrics included overall accuracy (OA), mean intersection-over-union ratio (mIoU), separation Kappa coefficient (SeK), and semantic change detection F-score (…). ).

[0071] 2.1 Comparative analysis on high-resolution aerial imagery (SECOND dataset) Table 1. Performance comparison of different models on the SECOND dataset

[0072] The SECOND dataset contains 4662 pairs of high-resolution aerial images, covering highly complex urban environments and 30 combinations of ground feature variations. The method described in this invention was compared with nine mainstream baseline models (including HRSCD-S4, SSCD-1, BiSRNet, SCANNet, LSAFNet, etc.), and the experimental results are shown in Table 1. Experimental data: The method described in this invention achieved 87.82% OA, 73.85% mIoU, 23.12% SeK, and 62.84% [missing data - likely a percentage of accuracy]. Technical performance analysis: In terms of mIoU, SeK, and other metrics reflecting the core capabilities of semantic change detection... In terms of metrics, this invention achieved optimal results. In particular, compared to the second-best performing LSAFNet model, the SeK score of this invention was improved by 0.22%. The improvement was 0.32%. This indicates that the Complementary Spatiotemporal Interaction Fusion Module (CSTI-FM) in this invention effectively solves the problem of aligning dual-temporal features at complex terrain boundaries by capturing long-distance change priors through parallel Mamba blocks and combining them with a cross-attention mechanism, significantly improving the ability to accurately distinguish between "semantic categories" and identify "change regions".

[0073] 2.2 Comparative analysis on long-term medium-resolution satellite imagery (Landsat-SCD dataset) Table 2. Performance comparison of different models on the Landsat dataset

[0074] The Landsat-SCD dataset contains long-term Landsat imagery, but has low spatial resolution and is highly susceptible to seasonal phenology and atmospheric fluctuations. It includes 10 fine-grained "From-To" semantic variation types. Experimental results are shown in Table 2. Experimental data: The method of this invention achieved 96.15% OA, 88.74% mIoU, 57.32% SeK, and 87.51% [missing data]. Technical Performance Analysis: On the most challenging Landsat-SCD dataset, this invention achieves state-of-the-art performance in two key metrics: OA (96.15%) and mIoU (88.74%). Compared to the second-best STS-FINet model, this invention leads in mIoU by approximately 0.92%. This fully demonstrates the superior ability of the hybrid Mamba-Transformer architecture in perceiving large-scale land evolution patterns and robustly capturing complex spatiotemporal variation features in noisy environments. Furthermore, the highest OA score also intuitively reflects the significant technical advantage of this invention in suppressing non-semantic noise (such as spectral fluctuations and seasonal changes), ensuring global consistency of classification results.

[0075] 3. Qualitative Analysis To visually verify the feature extraction and anti-interference capabilities of the proposed hybrid Mamba-Transformer network architecture and complementary spatiotemporal interaction fusion module (CSTI-FM) in complex real-world scenarios, this invention conducted a visual qualitative comparison experiment on the SECOND dataset and the Landsat-SCD dataset, and compared it with existing mainstream baseline models.

[0076] 3.1 Performance Validation in Complex Urban Environments (Based on the SECOND Dataset) In the SECOND dataset, which contains highly complex urban environments, this invention demonstrates significant advantages in handling details and background noise. (Reference) Figure 8 .

[0077] High internal coherence of large-scale features: In vegetation area detection, this invention can accurately distinguish between "trees" and "low vegetation." Compared to the sporadic misclassification spots and blurred semantic boundaries of baseline models (LSAFNet and SSCD), this invention maintains high internal coherence. Technical principle verification: This directly reflects the advantages of parallel Mamba blocks in large-scale, long-distance modeling. Its global context scanning mechanism ensures macroscopic identification of target attributes, thereby guaranteeing the semantic consistency of the region.

[0078] High-fidelity detail in complex geometric edges: In areas with complex contours, such as buildings and playgrounds, this invention demonstrates excellent spatial resolution preservation. The predicted results are highly consistent with the ground truth (GT), with clear and sharp boundaries, and no severe aliasing or geometric distortion as seen in the baseline model. Technical principle verification: This proves that the skip connections in the CSTI-FM module of this invention successfully and effectively transfer high-frequency spatial details from the encoder to the decoder.

[0079] Effective suppression of pseudo-change noise: In areas where water and land are intricately interwoven, this invention not only accurately identifies genuine semantic transitions (such as vegetation transforming into ground) but also successfully filters out non-semantic changes caused by seasonal color fluctuations. In contrast, LSAFNet completely loses structural coherence in this region. Technical principle verification: This effect is attributed to the cross-attention mechanism of this invention. Guided by the change prior extracted by Mamba, the model completes bi-temporal semantic alignment in the original feature space, significantly suppressing pseudo-change noise caused by differences in illumination and viewpoint.

[0080] 3.2 Performance Validation in Large-Scale Land Evolution Scenarios (Based on Landsat-SCD Dataset) The Landsat-SCD dataset is characterized by low image resolution and large land evolution scale. This invention also performs excellently in this scenario, referencing... Figure 9 .

[0081] Consistency Extraction of Large-Area Features: In large-scale areas such as farmland and deserts, the prediction results of this invention exhibit excellent regional consistency, effectively avoiding the "salt-and-pepper noise" and intra-class holes that severely affect the results of LSAFNet and SSCD. Technical Principle Verification: This strongly validates the long-range modeling capability of Mamba blocks. Its linear scanning mechanism overcomes the limitations of local receptive fields, capturing global spatial dependencies, thereby maintaining high semantic consistency in large-scale target recognition.

[0082] Keen Perception of Subtle Targets: In areas containing small water veins and scattered buildings, despite the limited resolution of Landsat imagery, this invention accurately extracted these subtle semantic components, with contours consistent with the ground truth labels. In contrast, baseline models either completely missed these minute details or severely misclassified them. Technical Principle Verification: This high sensitivity in "small target, large background" scenarios is primarily due to the skip connections and cross-attention mechanisms of CSTI-FM: skip connections preserve the positional details of underlying features, while cross-attention establishes accurate feature mappings across time phases.

[0083] 4. Ablation test 4.1 Validation of the Hybrid Encoder Architecture This embodiment verifies the beneficial effects of the encoder design by comparing the pure Transformer (based on multi-head self-attention MHSA) architecture, the pure Mamba (based on state space model SSM) architecture, and the hybrid architecture of the present invention. The results are shown in Tables 3 and 4.

[0084] On the SECOND dataset, the hybrid architecture of this invention achieves good mIoU and In terms of metrics, the hybrid architecture achieved improvements of 2.0% and 4.72% in mIoU and SeK respectively compared to the pure Transformer, with a 2.74% improvement in SeK compared to the pure Mamba. On the Landsat-SCD dataset, the hybrid architecture achieved improvements of 1.92% and 2.91% in mIoU and SeK respectively compared to the pure Transformer. It is 1.17% better than pure Mamba.

[0085] Table 3 Ablation experiments of encoders on the SECOND dataset

[0086] Table 4 Ablation experiments of encoders on the Landsat-SCD dataset

[0087] Experimental data shows that while the pure Transformer architecture excels at global dependency modeling, it tends to overlook local texture details when processing high-resolution images; while the pure Mamba architecture, although computationally efficient, lacks flexibility in modeling non-aligned spatial relationships in complex scenes. The hybrid architecture of this invention successfully achieves complementary advantages, effectively considering both global context and refining local boundary features, significantly reducing semantic gaps in large-area target recognition, and achieving a leap in core detection metrics.

[0088] 4.2 Validation of the Feature Fusion Method This embodiment verifies the technical advantages of the feature fusion strategy by comparing element-wise addition, channel-wise concat, and the complementary spatiotemporal interaction fusion module (CSTI-FM) of the present invention. The results are shown in Tables 5 and 6.

[0089] Table 5 Ablation experiments of feature fusion methods on the SECOND dataset.

[0090] Table 6 Ablation experiments of feature fusion methods on the Landsat-SCD dataset

[0091] On the SECOND dataset, compared to the Concat method which lacks dynamic alignment capabilities, the CSTI-FM of this invention improves the SeK metric to 23.12% (an improvement of 3.02%). The performance improved to 62.84% (an improvement of 3.60%), while the Add method performed the worst. On the Landsat-SCD large-scale modeling dataset, CSTI-FM achieved an mIoU of 88.74%, outperforming the Concat and Add methods by 1.58% and 2.96%, respectively; its SeK was better than the Concat method by 3.87%.

[0092] The above results intuitively demonstrate that, in long-distance modeling scenarios, the cross-attention mechanism designed in this invention achieves semantic alignment with significantly better technical results than simple linear summation or convolutional splicing methods. CSTI-FM can effectively solve the semantic inconsistency problem caused by differences in lighting or viewpoint in complex urban backgrounds.

[0093] 4.3 Validation of the effectiveness of cross-task synergistic interaction This embodiment designs three variants to verify the technical effect of the cross-task collaborative interaction mechanism: 1) Independent task (semantic segmentation SS and change detection CD branches run independently without feature interaction); 2) One-way guidance (SS prediction is guided only by CD features); 3) Bidirectional collaborative interaction of the present invention. The results are shown in Tables 7 and 8.

[0094] Table 7 Interactive ablation experiments on the SECOND dataset

[0095] Table 8 Interactive ablation experiments on the Landsat-SCD dataset

[0096] On the SECOND dataset, the bidirectional interaction model of this invention significantly improves all metrics compared to the independent task variant (OA improved by 0.88%, mIoU improved by 1.67%, SeK improved by 2.27%). It improves performance by 2.42%, and its SeK score also outperforms the unidirectional guided model by 1.08%. On the Landsat-SCD dataset, bidirectional interaction outperforms the independent task by 0.92% and 2.46% on OA and SeK, respectively.

[0097] Experiments have shown that treating the SS and CD tasks in isolation leads to suboptimal feature representations, failing to leverage the intrinsic correlation between semantic categories and regions of change. The bidirectional feature sharing mechanism of this invention not only guides the SS task to locate semantic shifts within regions of change using the CD task, but also utilizes the semantic context of the SS task to inversely optimize the change boundary accuracy of the CD task. This effectively suppresses spurious change predictions caused by seasonal phenology in datasets with strong environmental noise.

[0098] In some embodiments, the present invention provides a remote sensing semantic change detection system based on hybrid Mamba-Transformer and cross-attention fusion, for implementing the aforementioned remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion, the system comprising: A hybrid Mamba-Transformer encoder is used to receive a dual-temporal remote sensing image to be detected and extract multi-level dual-temporal features; wherein, the hybrid Mamba-Transformer encoder shares weights when processing dual-temporal images; The complementary spatiotemporal interactive fusion module is used to receive multi-level dual-temporal features and extract long-distance change perception fusion features through spatiotemporal projection, parallel Mamba scanning and complementary cross-attention calculation. A change detection decoder is used to receive the long-distance change-aware fusion features and multi-level dual-temporal features, and generate a binary change mask; A semantic segmentation decoder is used to receive multi-level dual-temporal features and generate a land cover semantic probability map corresponding to the two observation time phases. The mask constraint unit is used to spatially constrain the land cover semantic probability map using the binary change mask, so that the semantic change category is only identified within the detected binary change area, and outputs the final remote sensing image semantic change detection result.

[0099] In some embodiments, the present invention provides a computer-readable storage medium storing computer instructions that are executed by a processor as described in any of the above embodiments, for the remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion.

[0100] Computer-readable storage media can take the form of any combination of one or more readable media. A readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (not an exhaustive list) may include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD). ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0101] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] Embodiments of the present invention may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion according to various embodiments of the present invention, as described in the "Exemplary Methods" section above.

[0103] The steps of the method of the present invention are not limited to the specific order described above, unless otherwise specifically stated. Furthermore, in some embodiments, the invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the method according to the invention. Therefore, the invention also covers recording media storing programs for performing the method according to the invention.

[0104] Although the invention has been described with reference to preferred embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, the technical features mentioned in the various embodiments can be combined in any manner as long as there is no structural conflict. The invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion, characterized in that, include: Acquire the dual-temporal remote sensing images to be detected, and input the dual-temporal remote sensing images into the weight-shared hybrid Mamba-Transformer encoder to extract multi-level dual-temporal features; The multi-level dual-temporal features extracted by the encoder are input into the complementary spatiotemporal interactive fusion module. Through spatiotemporal projection, parallel Mamba scanning and complementary cross-attention calculation, long-distance change perception fusion features are extracted. The long-distance change perception fusion feature and multi-level dual-temporal feature are input into the change detection decoder to generate a binary change probability map, and the Argmax operation is performed on the binary change probability map to obtain a binary change mask. The multi-level dual-temporal features extracted by the encoder are input into two parallel semantic segmentation decoder branches to generate land cover semantic probability maps corresponding to the two observation time phases respectively; The binary change mask is used to spatially constrain the land cover semantic probability map, so that the semantic change category is only identified within the detected binary change area, and the final remote sensing image semantic change detection result is output.

2. The method according to claim 1, characterized in that, The hybrid Mamba-Transformer encoder employs a four-stage hierarchical architecture to extract features, specifically including: The original input image first passes through the Stem module, which consists of consecutive convolutional layers with a stride of 2, mapping the image from pixel space to embedding space and outputting downsampled feature blocks. The downsampled features pass through the first and second stages of the encoder sequentially. These stages consist of residual blocks based on convolutional neural networks, used to extract local texture and geometric structures from the high-resolution feature maps. The calculation formula for each residual block is as follows: Where GELU represents the Gaussian error linear unit activation function, and BN represents batch normalization; After the second stage of processing, the features are sequentially processed through the third and fourth stages of the encoder. The third and fourth stages consist of Mamba-Transformer blocks. Each Mamba-Transformer block includes a Mamba block based on a state-space model and a Transformer block containing a global self-attention mechanism. The features first enter the Mamba block based on the state-space model for global information aggregation, and then enter the Transformer block containing a global self-attention mechanism to extract features.

3. The method according to claim 1, characterized in that, The feature extraction process of the complementary spatiotemporal interaction fusion module includes: Spatiotemporal projection: By performing feature concatenation along the channel dimension and combining it with spatial interleaving, bi-temporal features are mapped onto a unified coding space to obtain projected features; Parallel Mamba scanning and feature aggregation: The projected features are input into two parallel Mamba blocks for long sequence modeling. Subsequently, the features extracted from the two parallel branches are channel-wise concatenated to synthesize enhanced change-aware joint features. G ; Cross-attention mapping: Through skip connections, the original bi-temporal features bypass the Mamba block and are directly input into the complementary cross-attention unit to generate keys. K Sum V Vectors, while simultaneously incorporating the enhanced change-aware joint features G As a query Q The vector input to the complementary cross-attention unit is calculated using the following formula: in, is the scaling factor for the dimension of the key vector; Feature reconstruction: The output of the cross-attention unit is input into the convolutional layer for dimensionality reduction to obtain the reconstructed fused features, which are then input into the subsequent decoder network.

4. The method according to claim 1, characterized in that, It also includes model training, where the model iteratively optimizes itself based on the training set data during the training phase, and the total loss function used for training. Binary change detection loss and bi-temporal semantic classification loss , The weighted sum is calculated using the following formula: in, The weighting parameters correspond to the loss; binary change detection loss. Both the bi-temporal semantic classification loss and the loss are composed of cross-entropy loss and Lovasz-softmax loss.

5. The method according to claim 4, characterized in that, The binary change detection loss The calculation formula is: in, Loss detection for binary changes Cross-entropy loss, Loss detection for binary changes The Lovasz-softmax loss; The calculation formula is: in, N Total number of pixels This represents the unique heat representation of the label. This represents the classification probability value output by the network.

6. The method according to claim 4, characterized in that, The cross-entropy loss of the bi-temporal semantic classification The calculation formula is: in, N Total number of pixels This represents the unique heat representation of the label. The classification probability value output by the network. This represents the total number of land cover categories.

7. The method according to claim 3, characterized in that, The parallel Mamba scan includes: inputting the projection features into a forward Mamba block and a backward Mamba block respectively; the forward Mamba block performs sequence modeling along the original spatial flattening direction; and the backward Mamba block performs sequence modeling along the reverse spatial flattening direction, in order to capture long-distance dependencies in different spatial directions.

8. A remote sensing semantic change detection system based on hybrid Mamba-Transformer and cross-attention fusion, characterized in that, The system for implementing the remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion as described in any one of claims 1-7, the system comprising: A hybrid Mamba-Transformer encoder is used to receive a dual-temporal remote sensing image to be detected and extract multi-level dual-temporal features; wherein, the hybrid Mamba-Transformer encoder shares weights when processing dual-temporal images; The complementary spatiotemporal interactive fusion module is used to receive multi-level dual-temporal features and extract long-distance change perception fusion features through spatiotemporal projection, parallel Mamba scanning and complementary cross-attention calculation. A change detection decoder is used to receive the long-distance change-aware fusion features and multi-level dual-temporal features, and generate a binary change mask; A semantic segmentation decoder is used to receive multi-level dual-temporal features and generate a land cover semantic probability map corresponding to the two observation time phases. The mask constraint unit is used to spatially constrain the land cover semantic probability map using the binary change mask, so that the semantic change category is only identified within the detected binary change area, and outputs the final remote sensing image semantic change detection result.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are executed by a processor as described in any one of claims 1-7: a remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion.

10. A computer program product, characterized in that, The computer program product stores computer instructions, which are executed by a processor as described in any one of claims 1-7, for the remote sensing semantic change detection method based on hybrid Mamba-Transformer and cross-attention fusion.

Images