Complex remote sensing image segmentation method and system based on base model decoupling and multi-modal spatio-temporal fusion

Abstract

The application discloses a complex remote sensing image segmentation method and system based on basic model decoupling and multi-modal space-time fusion, and belongs to the technical field of remote sensing AI vision. The application constructs a decoupling reasoning framework of "rough positioning first and precise segmentation later": a second-order prompter is cascaded to orthogonally decompose text features into category and attribute dual semantic subspaces, and two-level cross-modal interaction suppresses multi-entity target false activation; a LiDAR space geometry + Sentinel-2 time series multi-spectral double-flow encoder is built to extract highly physically constrained and vegetation phenology features; an asymmetric cross-attention bottleneck is used to fuse multi-modal features, and high-resolution features of LiDAR are used to guide boundary refinement; Softmax hard classification is abandoned, and Dirichlet distribution is used to predict output sub-pixel vegetation mixed proportion soft segmentation maps, and LoRA low-rank fine-tuning is used to light-weight adapt CLIP and SAM basic models. The application solves the three major pain points of positioning and segmentation coupling error, remote sensing fuzzy boundary and vegetation foreign object same spectrum in the prior art, and compared with a CLIP+SAM baseline, mIoU is improved by 11.44%, the proportion of multi-entity false activation pixels is reduced to 2.15%, the ecological gradual change boundary can be accurately restored, and the application is suitable for land survey and ecological monitoring scenes.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing, computer vision and artificial intelligence, and in particular to a high-precision segmentation method and system for remote sensing images oriented towards Referring Remote Sensing Image Segmentation (RRSIS), which is particularly suitable for vegetation and ground feature segmentation tasks in complex natural environments such as blurred boundaries and ecological transition zones. Background Technology

[0002] Semantic segmentation of remote sensing images is one of the core technologies in fields such as natural resource surveys, ecological environment monitoring, and urban planning. In recent years, with the rapid development of visual basic models, general segmentation models, represented by SAM (Segment Anything Model), have made breakthrough progress in the field of natural images. Researchers have begun to try to transfer these basic models to remote sensing image segmentation tasks.

[0003] In existing technologies, indicative remote sensing image segmentation (RRSIS) typically employs a coupled three-stage pipeline architecture of "bimodal coding - cross-modal interaction - pixel decoding". This architecture couples the target localization and boundary delineation subtasks into the same coding-decoding process, which can achieve certain results in simple scenarios, but has significant drawbacks in complex semantic and terrain environments.

[0004] First, there is a problem of target misactivation in multi-entity scenarios. Native CLIP (Contrastive Language-Image Pre-training) and other cross-modal models tend to activate multiple similar targets or incorrect targets simultaneously when processing complex descriptions involving multiple entities, making it difficult to accurately locate the specific object specified by the user. Existing cross-modal interaction mechanisms have limited ability to model complex semantics and logical relationships, resulting in systematic biases in image-text alignment.

[0005] Secondly, the adaptability to blurred boundaries is poor. The prior knowledge of basic models such as SAM mainly comes from the clear and sharp object boundaries in natural images. However, low-contrast targets and blurred boundaries are common in remote sensing images, such as dam outlines, port shorelines, and the boundary between water bodies and wetlands. Existing methods are unable to accurately delineate their true boundaries, and the segmentation results generally suffer from edge drift problems.

[0006] The most prominent challenge is the "heterogeneous yet similar spectral composition" problem in natural vegetation segmentation. In complex natural ecosystems, natural forests, shrublands, and grasslands often intertwine and gradually transition between each other in the form of ecological transition zones, lacking clear physical boundaries. Single RGB optical images are highly susceptible to the "heterogeneous yet similar spectral composition" phenomenon, meaning that different vegetation types exhibit highly similar spectral characteristics. Traditional binarized hard-boundary segmentation methods and techniques that rely solely on optical features cannot effectively distinguish mixed pixels or handle spectral ambiguity, resulting in a long-standing bottleneck in vegetation classification accuracy.

[0007] In summary, existing remote sensing image segmentation methods suffer from technical problems such as error propagation caused by the coupling of target localization and boundary segmentation, poor adaptability of the basic model domain, and insufficient segmentation accuracy of fuzzy boundaries and ecological transition zones. A new technical solution is urgently needed to address these issues. Summary of the Invention

[0008] (a) Purpose of the invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a complex remote sensing image segmentation method and system based on decoupling of the basic model and multimodal spatiotemporal fusion, which solves the error propagation problem caused by the coupling between target localization and boundary segmentation, as well as the technical problem of low segmentation accuracy of the basic model in remote sensing image scenarios with blurred boundaries, low-contrast targets and complex ecological transition zones.

[0010] (II) Overall Technical Solution

[0011] On the one hand, this invention provides a remote sensing image segmentation method based on basic model decoupling and multimodal spatiotemporal fusion. The core inference process includes, in sequence: multi-source remote sensing data acquisition, text semantic orthogonal decoupling and two-level cascaded localization, LiDAR spatial geometry + multispectral temporal dual-stream feature extraction, asymmetric cross-attention cross-modal fusion, and Dirichlet distribution subpixel soft segmentation output; and is accompanied by an offline LoRA parameter efficient fine-tuning training process.

[0012] On the other hand, the present invention provides a segmentation system adapted to the above method, along with supporting storage media and electronic devices, fully covering the software implementation and rights protection chain.

[0013] (III) Detailed Explanation of the Four Core Innovation Modules

[0014] (1) Semantic decomposition of cascaded second-order indicator prompter (target coarse localization module): The original indicator text features are decomposed by orthogonal projection matrix. It is split into two types of orthogonal semantic subspaces: one representing category identity. Characterizing attributes / spatial relationships The two semantic categories do not interfere with each other. A two-level cascaded interaction: first, candidate regions are coarsely screened using category features to generate... reuse After modulating visual features and combining them with attribute features for fine screening, a precise localization mask is output. This is converted into SAM-compatible sparse and dense cues to suppress false activation of multiple entities.

[0015] (2) Spatial geometry branch of multimodal spatiotemporal dual-stream feature extraction module: LiDAR point cloud is rasterized into canopy height model (CHM) and intensity map, and HRNet extracts high-resolution structural features such as height, roughness, and porosity. Provides physical height constraints for fuzzy boundaries; Temporal phenology branch: Sentinel-2 temporal multispectral extraction of NDRE (Normalized Red Edge Index) and IRECI (Inverted Red Edge Chlorophyll Index), 1D-CNN+Transformer extraction of annual vegetation phenological features. Distinguish spectrally similar vegetation from a time dimension.

[0016] (3) The asymmetric cross-attention bottleneck fusion module uses high-resolution spatial geometric features as query vectors. Temporal phenological characteristics as The scaling dot product attention calculation is completed to enable proactive querying of temporal spectral information at spatial boundaries; combined with guided filter upsampling and demixing, the fused features simultaneously possess the ability to distinguish fine spatial contours and vegetation phenology.

[0017] (4) The Dirichlet distribution subpixel soft segmentation output module abandons the traditional Softmax hard binary classification, fits the Dirichlet distribution to output the component ratio of various vegetation in each pixel, and outputs a gradual soft segmentation map that fits the real surface features of the continuous gradual change in the ecological transition zone; combined with LoRA low-rank fine-tuning, it achieves lightweight adaptation of CLIP and SAM remote sensing domains.

[0018] (iv) Beneficial effects of the present invention

[0019] (1) Eliminating error coupling and multi-entity interference. This invention proposes a decoupling paradigm of "coarse localization followed by fine segmentation" to decouple the target localization and boundary delineation subtasks, fundamentally avoiding error propagation in the coupled pipeline. The two-level cascaded interaction of the orthogonal semantic subspace reduces the proportion of falsely activated pixels in multi-entity scenes from 13.21% in the prior art to 2.15%, significantly improving localization accuracy. (2) Overcoming the limitations of blurred boundaries and foreign objects sharing the same spectrum. LiDAR height features provide hard physical boundary constraints, while time-series phenological features highlight the differences between similar spectra. Dual-modal fusion simultaneously solves the two major pain points of low-contrast jagged boundaries and vegetation spectrum confusion. The mIoU of the general remote sensing dataset is improved by 11.44 percentage points compared with the mainstream CLIP+SAM baseline. (3) It conforms to the sub-pixel accuracy of ecological reality. The Dirichlet distribution predicts the proportion of mixed vegetation in the output pixels, replacing the hard boundary of black and white. The RMSE of the mixed pixels in the ecological transition zone prediction is reduced from 0.217 to 0.062, and the error is reduced by 69%. The segmentation results can directly support ecological quantitative monitoring. (4) Parameter-efficient basic model adaptation. LoRA low-rank fine-tuning is adopted, which only trains a small number of low-rank matrix parameters. There is no need to update the CLIP and SAM massive backbone weights. The training computing power and annotation data requirements are greatly reduced, and it is suitable for lightweight terminal deployment in the field. Attached Figure Description

[0020] Figure 1 is a schematic diagram of the overall system architecture of the present invention. It includes a text command input terminal, an optical image input terminal, a multimodal data input terminal (LiDAR and temporal multispectral), a decoupled prompt generation module, a cross-modal fusion decoder, and a soft segmentation output terminal;

[0021] Figure 2 is a schematic diagram of the structure of a cascaded second-order indicator cue. It shows how text features are decomposed into two subspaces through orthogonal attention and then cascaded and multiplied twice with visual features (Mask 1 and Mask 2) to ultimately generate sparse and dense cues.

[0022] Figure 3 shows the bottleneck structure of asymmetric cross-modal fusion and cross-attention. It illustrates the computational flow of the cross-attention mechanism, which uses high-dimensional LiDAR spatial features as the query and multispectral temporal features as the key and value.

[0023] Figure 4 shows the sub-pixel-level Dirichlet distribution prediction head structure, illustrating how the feature map is mapped to the concentration parameter. And generate a soft mask with gradient changes in component proportions;

[0024] Figure 5 shows a comparison of the qualitative segmentation results of multiple models: it includes three sets of comparison sub-images, see Figure 5(a), Figure 5(b), and Figure 5(c): ① Comparison of multi-entity text prompt scene (original CLIP+SAM misactivation in multiple regions, this invention accurately locks the target); ② Comparison of low-contrast blurred boundary of dam (SAM boundary jagged and broken, this invention has continuous and complete outline); ③ Comparison of forest-shrub-grassland ecological transition zone (traditional model has hard boundary missegmentation, this invention has gradual soft segmentation that fits the actual vegetation ratio). Detailed Implementation

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0026] This invention proposes a complex remote sensing image segmentation method based on basic model decoupling and multimodal spatiotemporal fusion. The core idea is to adopt a decoupling framework of "coarse localization followed by fine segmentation", combine it with a cascaded second-order cue to achieve accurate target localization, and deeply fuse the spatial geometric features of lidar and multispectral temporal phenological features. Finally, the method outputs soft segmentation results that conform to ecological reality through a Dirichlet distribution prediction head.

[0027] 5.1 Example 1: Remote Sensing Image Segmentation Inference Method

[0028] Figure 1 illustrates the overall system architecture of the present invention. This embodiment provides a complete inference flow, which executes the following steps sequentially:

[0029] Step S1: Acquisition and Preprocessing of Multi-Source Remote Sensing Data. Acquire three types of data from the same geographic area, all uniformly registered: RGB optical remote sensing images, airborne LiDAR point cloud data, and Sentinel-2 (ESA Sentinel-2 satellite) multispectral time-series imagery throughout the year. LiDAR Preprocessing: Point cloud filtering and rasterization to generate a canopy height model (CHM) and laser intensity map, aligning the resolution with the optical images. Multispectral Temporal Preprocessing: Extract NDRE and IRECI red-edge vegetation indices, stacking them along the time dimension to construct a temporal feature tensor.

[0030] Step S2: Independent encoding of visual and linguistic features. The backbone model adopts the open-source CLIP pre-trained image-text base model, and the image encoder extracts visual features from optical images. The text encoder extracts the original features of the user-instructed text. The model's attention layer is pre-set with a LoRA low-rank matrix. Remote sensing domain adaptation is completed during the offline training phase, and the backbone is frozen during the inference phase, using only the adapted features.

[0031] Step S3: Text Feature Semantic Decomposition and Coarse Target Localization (See Figure 2) (1) Semantic Decoupling Projection: Orthogonal projection matrix will Decompose into orthogonal, non-redundant components. (Category Subspace) (Attribute subspace); (2) First-stage cross-activation: (2) Coarse screening of candidate regions for target categories; (3) Visual feature modulation: (3) Suppress feature responses in non-candidate regions; (4) Second-stage cross-activation: (5) Prompt generation: Use attribute information to remove interfering ground features to obtain a precise positioning mask; Convert the cues into sparse cues (bounding boxes, keypoint embeddings) and dense cues (coarse segmentation masks), and input them into the subsequent decoder.

[0032] Step S4: Multimodal Spatiotemporal Feature Dual-Stream Extraction Spatial Geometric Encoder (HRNet) inputs CHM + intensity map, outputs high-resolution spatial geometric features. The temporal phenology encoder (1D-CNN+Transformer) takes a multispectral temporal tensor as input and outputs vegetation phenological feature signatures. .

[0033] Step S5: Asymmetric cross-modal feature fusion (see Figure 3) Linear transformation into query vector , Generate separately The scaled dot product attention calculation is performed to obtain the fused features; the LiDAR spatial features are used as a guide filter to complete the upsampling and demixing, restoring the original image resolution.

[0034] Step S6: Dirichlet distribution fine soft mask generation (see Figure 4) The segmentation cue from S3 and the fusion feature from S5 are fed into the SAM decoder, and the softmax classification head is replaced at the end with the Dirichlet distribution prediction layer: (1) The concentration parameter is obtained by fusing the feature mapping. (2) Based on Solve the expectation of the Dirichlet distribution and output the proportion of each type of vegetation for each pixel; (3) Generate a gradient soft segmentation map to intuitively reflect the mixed state of continuous vegetation in the ecological transition zone.

[0035] 5.2 Example 2: Offline Model Training Optimization Process

[0036] This embodiment corresponds to the offline training step of claim 8, using an NVIDIA A100 80G graphics card for training, selecting the AdamW adaptive optimizer, setting the rank of the LoRA low-rank matrix to 8, training epochs of 200, and batch size of 32.

[0037] Basic model frozen: CLIP and SAM backbone weights are frozen throughout, only LoRA matrix, cross-modal fusion module, and Dirichlet prediction head parameter updates are open;

[0038] Multi-task joint loss: The localization branch uses Dice Loss to optimize the two-level mask; the soft segmentation branch uses an improved KL divergence loss to match the actual surface vegetation cover score labels; the total loss is a weighted sum of the two losses.

[0039] After iterative convergence, a lightweight adaptation model is saved, which can be directly used for inference on field remote sensing terminals.

[0040] 5.3 Model Comparison Experiment and Ablation Experiment

[0041] All experimental datasets and baseline models in this chapter are publicly available open-source resources. The data can be fully reproduced and are used to quantitatively demonstrate the technological advancements of this invention.

[0042] 5.3.1 Basic Experimental Configuration

[0043] 1. Publicly available test dataset

[0044] ISPRS Potsdam Dataset: The official ISPRS 2D remote sensing segmentation benchmark, released in 2012, open source address: ISPRS Official Dataset Platform; 5cm aerial RGB imagery, 6 land cover classes, containing a large number of ecological transition blurry boundaries, divided into a training set of 24 images and a test set of 14 images, with standard pixel binary mask ground truth.

[0045] Jornada Ecotone Ecodataset: The USDA releases a temporal vegetation LiDAR + multispectral dataset specifically annotated with ground truth pixel fractional cover of natural forest / shrub / grassland, targeting "heterogeneous objects with the same spectrum" gradient boundary tests.

[0046] LoveDA Open Vocabulary Remote Sensing Dataset: An open-source urban and rural remote sensing image segmentation dataset from Wuhan University, featuring multi-entity mixed ground object scenes, used to verify the effectiveness of multi-entity false activation suppression.

[0047] 2. Compare with baseline model

[0048] Native SAM: Kirillov et al., "Segment Anything" (2023); only RGB optical input, no multimodal, no semantic decoupling;

[0049] CLIP+SAM Naive Coupling: Radford et al. 2021 CLIP Basic Graphical Pipeline, Positioning and Segmentation Coupling, No Two-Level Hints;

[0050] GeoRSCLIP+SAM: Fine-tuning CLIP baseline in the remote sensing domain, with only single-stage image and text interaction;

[0051] HRNet-SAM: Sun et al.'s HRNet high-resolution network only integrates LiDAR single-modality data, without temporal phenology coding or Dirichlet output;

[0052] TransUNet: A traditional hard classification and segmentation baseline for remote sensing, without large-scale model collaboration.

[0053] 3. Standardize evaluation indicators

[0054] (1) Hard segmentation metrics (Potsdam, LoveDA) mIoU: Average Intersection over Union (intersection over union) (core segmentation metric, unit %); mF1: Average F1 score (unit %); OA: Overall pixel accuracy (unit %)

[0055] (2) Sub-pixel soft segmentation index (Jornada ecological dataset) RMSE: Root Mean Square Error (prediction error of mixed vegetation ratio, dimensionless, the smaller the value, the higher the accuracy); KL divergence: KL distribution difference (deviation between predicted and actual vegetation distribution, dimensionless).

[0056] 4. Training Hardware and Hyperparameters

[0057] NVIDIA A100 80G, AdamW optimizer, LoRA rank = 8, 200 epochs, batch_size = 32; loss combination: Dice Loss (localization) + improved KL divergence (Dirichlet soft segmentation).

[0058] 5.3.2 Results of cross-sectional comparison experiment on public dataset baseline

[0059] Table 1. Comparison of hard segmentation accuracy for general remote sensing segmentation (Potsdam dataset) (unit: %)

[0060] TransUNet 66.44 70.12 82.01 Optical characteristics alone: ​​foreign objects with the same spectrum, poor accuracy of blurred boundaries. Native SAM 69.75 73.46 84.35 Poor text and image alignment, false activation of multiple entities, and poor boundary adaptation. CLIP+SAM (Naive Coupling) 72.18 76.09 86.72 Location / segmentation coupling, single-stage text interaction is prone to accidental activation. GeoRSCLIP+SAM 74.53 78.61 88.16 Without LiDAR, the temporal dual-stream has no distinguishing ability in the ecological transition zone. HRNet-SAM (Spatial Features Only) 76.27 80.34 89.40 The lack of temporal phenological characteristics and the inability to resolve spectral confusion are issues that need to be addressed. Complete framework of the invention 83.62 87.95 94.28 Decoupled localization + multimodal spatiotemporal fusion + Dirichlet soft segmentation

[0061] Conclusion 1: Compared with the industry mainstream CLIP+SAM coupling baseline, the present invention improves mIoU by 11.44 percentage points and mF1 by 11.86%, directly solving the two major background technical pain points of fuzzy boundaries and multi-entity false activation, corresponding to the beneficial effects (1) (2).

[0062] Table 2 Comparison of multi-entity localization accuracy for open vocabulary indicator segmentation (LoveDA dataset) (unit: %)

[0063] Metric: Target Recall (measures the ability to suppress false negatives / false activations in multi-entity scenarios)

[0064] Native SAM 81.27 63.51 18.74% CLIP+SAM Naive Coupling 84.63 69.28 13.21% GeoRSCLIP+SAM 86.19 72.45 10.68% This invention (cascaded second-order prompter) 92.84 88.73 2.15%

[0065] Conclusion 2: The orthogonal semantic two-level cascaded reasoning of this invention reduces the proportion of falsely activated pixels in multi-entity scenes from 13.21% to 2.15%, completely solving the problems of CLIP image-text alignment deviation and false activation of multi-entity, and achieving beneficial matching effects (1).

[0066] Table 3. Accuracy comparison of subpixel soft segmentation of the ecological transition zone (Jornada dataset) (RMSE and KL divergence are dimensionless values, without units)

[0067] A lower RMSE indicates a more accurate prediction of vegetation mixing ratios, while a smaller KL divergence indicates that the gradient boundary distribution closely matches the actual ecology.

[0068] CLIP+SAM (Hard Sorting Head) 0.217 0.426 Hard boundary breaks the transition zone, resulting in large-area spectral confusion. HRNet-SAM (Softmax Binary Header) 0.183 0.371 Unable to output blending ratio, gradient region classification is disordered. This invention (Dirichlet distribution prediction head) 0.062 0.089 Continuous gradient soft segmentation accurately recreates the vegetation transition zone.

[0069] Conclusion 3: Existing hard classification models cannot adapt to the ecological transition zone. The Dirichlet pixel prediction of this invention reduces the mixed pixel error by 69%, which is a unique innovation that existing technologies do not have, and has the corresponding beneficial effect (3).

[0070] 5.3.3 Module Ablation Experimental Analysis

[0071] Using the Potsdam dataset mIoU as the evaluation benchmark, the performance degradation of validation was removed module by module, proving that each innovative module is indispensable and there is no simple superposition:

[0072] Complete framework of this invention 83.62 — All innovation modules work together Remove the Cascaded Second-Order Indicator (single-stage text interaction) 75.19 -8.43% Loss of semantic decoupling capability leads to a surge in false activations of multiple entities. Remove the LiDAR Spatial Geometric Encoder (Temporal Multispectral Only) 77.45 -6.17% Missing height physical boundaries, low contrast contour segmentation fails Remove the [Temporal Phenology Encoder] (LiDAR Single Mode Only) 78.21 -5.41% Without phenological spectral differentiation, vegetation with different spectra cannot be distinguished. Replace "Cross-Attention Bottleneck" with simple splicing and fusion. 79.06 -4.56% Spatial-temporal features cannot accurately align with boundary pixels. Replace the Dirichlet prediction head with Softmax hard classification. 76.84 -6.78% Loss of subpixel blending ratio output capability, and a sharp drop in transition band accuracy. Remove LoRA parameter for efficient fine-tuning (full fine-tuning) 81.37 -2.25% The basic model domain offset is not adapted, resulting in degraded remote sensing feature extraction.

[0073] The core conclusions of the ablation experiment are: all six innovative modules bring significant accuracy improvement, and the overall framework works together to produce unexpected technical effects beyond the superposition of individual modules, which meets the criteria for the inventiveness evaluation of invention patents; LoRA lightweight fine-tuning brings a 2.25% accuracy improvement, corresponding to the beneficial effect (4).

[0074] 5.3.4 Qualitative Visual Comparison (See Figure 5)

[0075] Multi-entity text prompt scenario (Figure 5 (a)): The native CLIP+SAM simultaneously activates multiple unrelated similar plots, resulting in a large area of ​​mis-segmentation mask; the cascaded second-order prompter of this invention only locks the text specified target, without any extra mis-activated areas.

[0076] Low-contrast blurred boundary scene of the dam (Fig. 5(b)): The original SAM output segmentation boundary is jagged and has fracture drift; the LiDAR height feature of this invention provides physical cutting constraints, and the outline of the ground features is complete and continuous.

[0077] Forest-shrub-grassland ecological transition zone scene (Fig. 5 (c)): Traditional hard classification model outputs black and white cut boundaries, and the entire transition zone is misclassified; the Dirichlet soft segmentation of this invention outputs a continuous gradient of gray level, and the gray level corresponds to the proportion of various vegetation types, which perfectly matches the mixed state of vegetation in the actual field.

[0078] 5.3.5 Comprehensive Demonstration of Experimental Data

[0079] Table 2 Quantitatively verifies that "orthogonal semantic cascade decoupling suppresses multi-entity false activation," thus addressing the first defect in the background technology;

[0080] Table 1 Quantitatively verifies that "LiDAR + temporal multimodal fusion improves the accuracy of fuzzy boundaries and general ground feature segmentation", thus addressing the second defect of the background technology;

[0081] Table 3 Quantitatively verifies "the soft segmentation of the Dirichlet distribution adapts to the ecological transition zone and eliminates the homospectral dissemination of foreign matter", thus solving the third defect of the background technology;

[0082] The ablation scale demonstrates that each innovative module independently contributes to the accuracy gain, and the solution is not a simple patchwork of existing technologies, demonstrating sufficient creativity.

[0083] All datasets and baseline models are publicly available open-source resources. Experimental procedures and hyperparameters are fully disclosed, and those skilled in the art can fully reproduce the experiments, meeting the requirements for full disclosure of patents.

[0084] It can be directly applied to scenarios such as natural resource ownership confirmation surveys, land spatial planning, forest carbon sequestration, and wetland ecological restoration assessment. Compared with the industry mainstream CLIP+SAM baseline, the mIoU of general remote sensing segmentation is improved by 11.44%, the proportion of falsely activated pixels in multi-entity scenes is reduced by 11.06 percentage points, and the prediction error of mixed pixels in ecological transition zones is reduced by 69%. It can significantly reduce the workload of manual visual interpretation, has clear economic value and ecological and social benefits, and the technological improvement effect can be fully reproduced.

Claims

1. A remote sensing image segmentation method based on basic model decoupling and multimodal spatiotemporal fusion, characterized in that, The following inference steps are executed sequentially: S1. Acquire the optical remote sensing image, lidar point cloud data, and multispectral time-series image of the region to be segmented; S2. Perform semantic decoupling on the instruction text input by the user, and project the features of the instruction text onto the orthogonal basic category subspace and attribute relation subspace to obtain category semantic features and attribute semantic features respectively; S3. Perform a first-stage cross-activation on the visual features of the optical remote sensing image and the category semantic features to generate a preliminary candidate region mask; Based on the preliminary candidate region mask, the visual features are modulated, and then cross-activated with the attribute semantic features in the second stage to generate a fine localization mask, and the fine localization mask is converted into segmentation prompt information; S4. Extract features from the lidar point cloud data and the multispectral time-series image respectively to obtain spatial geometric features and time-series phenological features; S5. Using the spatial geometric features as the query vector and the temporal phenological features as the key and value vectors, perform cross-attention fusion to obtain a fused feature map; S6. Input the segmentation hint information obtained in step S3 and the fused feature map obtained in step S5 into the segmentation decoder, and output the component proportion of each pixel belonging to each category through the Dirichlet distribution prediction head to generate a soft segmentation result.

2. The remote sensing image segmentation method according to claim 1, characterized in that, The semantic decoupling of the user-input instruction text specifically includes: extracting the original text features of the instruction text through a pre-trained text encoder; and projecting the original text features onto the basic category subspace and the attribute relation subspace respectively through an orthogonal projection matrix, wherein the basic category subspace is used to represent the category identity of the target object, and the attribute relation subspace is used to represent the attribute description and spatial relationship of the target object.

3. The remote sensing image segmentation method according to claim 1, characterized in that, The first-stage cross-activation and the second-stage cross-activation are calculated using the following formulas: in, As a preliminary candidate region mask, For precise mask positioning, As a visual feature, For category semantic features, For attribute semantic features, The projection matrix is ​​learnable. The Sigmoid activation function is used. This is for element-wise multiplication.

4. The remote sensing image segmentation method according to claim 1, characterized in that, The feature extraction of the lidar point cloud data specifically includes: rasterizing the lidar point cloud data to generate a canopy height model (CHM) and an intensity map; and extracting spatial geometric features including absolute height, porosity, and surface roughness from the canopy height model and intensity map using a two-dimensional convolutional neural network.

5. The remote sensing image segmentation method according to claim 1, characterized in that, The feature extraction of multispectral time-series images specifically includes: extracting red-edge band indices from multispectral time-series images to construct a time-series feature tensor; and extracting phenological feature signatures representing the seasonal change patterns of vegetation from the time-series feature tensor using a one-dimensional convolutional neural network combined with a Transformer encoder.

6. The remote sensing image segmentation method according to claim 1, characterized in that, The cross-attention fusion specifically includes: obtaining a query vector from high-resolution spatial geometric features through a linear transformation. The key vectors are obtained by linear transformation of the temporal phenological characteristics. Sum value vector Calculate the attention weights and sum them with the value vector to obtain the fused features: in The feature dimension is defined as follows: spatial geometric features are used as a guiding filter to upsample and demix the fused features, restoring them to the original resolution.

7. The remote sensing image segmentation method according to claim 1, characterized in that, The calculation process of the Dirichlet distribution prediction head is as follows: the concentration parameter is obtained by mapping the fused feature map through the output layer. Based on the concentration parameter Calculate the expected proportion of each pixel belonging to each category to generate a soft segmentation map, wherein the sum of the proportions of each pixel in each category in the soft segmentation map is 1, which is used to characterize the gradual boundary of the ecological transition zone.

8. The remote sensing image segmentation method according to claim 1, characterized in that, It also includes offline model training steps: inserting low-rank adaptation (LoRA) matrices into the pre-trained CLIP visual language base model and SAM segmentation base model, and using a parameter-efficient fine-tuning strategy for remote sensing domain adaptation; based on the surface component proportion label, using the KL divergence loss function to optimize the Dirichlet distribution prediction head, and combining it with Dice loss to optimize the localization branch; During training, only the low-rank adaptation matrix and decoder parameters are updated, while the backbone weights of the base model are frozen.

9. A remote sensing image segmentation system based on fundamental model decoupling and multimodal spatiotemporal fusion, characterized in that, The remote sensing image segmentation method according to any one of claims 1 to 8 comprises: a data input module for acquiring optical remote sensing images, lidar point cloud data, and multispectral temporal images of the region to be segmented; a decoupling prompt generation module for semantically decoupling user-input prompt text to obtain category semantic features and attribute semantic features, and generating a fine-grained localization mask through cascaded two-stage cross-activation, which is then converted into segmentation prompt information; a multimodal spatiotemporal feature extraction module, including a spatial geometric encoder and a temporal phenological encoder, for extracting spatial geometric features from lidar point cloud data and temporal phenological features from multispectral temporal images, respectively; a cross-modal fusion decoding module for performing cross-attention fusion using spatial geometric features as query vectors and temporal phenological features as key and value vectors to generate a fused feature map; and a soft segmentation output module for outputting the component proportions of each pixel belonging to each category through a Dirichlet distribution prediction head to generate soft segmentation results; the modules are executed sequentially in the order of "text localization → multimodal feature extraction → cross-attention fusion → Dirichlet soft segmentation".

10. The remote sensing image segmentation system according to claim 9, characterized in that, The decoupled cue generation module includes: a semantic decomposition unit, used to decompose the original text features into basic category subspace features and attribute relation subspace features through an orthogonal projection matrix; a cascaded activation unit, used to sequentially perform a first-stage category localization and a second-stage attribute refinement to generate a two-level mask; and a cue conversion unit, used to convert the fine-grained localization mask into sparse cue and dense cue, wherein the sparse cue includes bounding box or point coordinate embedding, and the dense cue includes a coarse segmentation mask.

11. The remote sensing image segmentation system according to claim 9, characterized in that, The spatial geometric encoder is built on the HRNet architecture, with the input being a canopy height model and intensity map, and the output being high-resolution structural features; the temporal phenological encoder is composed of stacked one-dimensional convolutional layers and Transformer coding layers, with the input being a multispectral time series, and the output being a feature vector representing phenological patterns.

12. The remote sensing image segmentation system according to claim 9, characterized in that, The cross-modal fusion decoding module is equipped with a cross-attention bottleneck structure, which performs spatial feature-guided temporal feature queries at multiple scales during the decoding stage, thereby achieving multi-scale cross-modal feature fusion.

13. The remote sensing image segmentation system according to claim 9, characterized in that, The soft segmentation output module abandons the traditional Softmax binary classification head and uses a Dirichlet distribution to fit the class mixing ratio of each pixel, outputting a soft segmentation map with sub-pixel accuracy to characterize the gradual boundary of the ecological transition zone.

14. The remote sensing image segmentation system according to claim 13, characterized in that, It also includes a parameter-efficient fine-tuning module, which inserts a low-rank adaptation (LoRA) matrix into the attention layer of the CLIP visual language model and the SAM segmentation model to achieve domain adaptation in the remote sensing field. During training, only the low-rank matrix parameters and decoder parameters are updated.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the remote sensing image segmentation method according to any one of claims 1 to 8.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the remote sensing image segmentation method according to any one of claims 1 to 8.

Images