Low-altitude remote sensing feature extraction method with DSM side edge adapter

Abstract

The application discloses a low-altitude remote sensing ground feature extraction method fusing a DSM side edge adapter. The method takes a pre-trained visual large model with frozen parameters as a backbone to extract visual features, and constructs a trainable DSM side edge adapter in parallel to extract ground object elevation geometric features; a cross-modal attention fusion mechanism is used to dynamically guide the injection of height information, realize content-aware adaptive modal fusion, and take the visual features as queries and the elevation geometric features as keys. In addition, the application adopts a parallel prompt decoding strategy, uses a batch stacking technology to construct a virtual task batch, reconstructs multi-class semantic segmentation into a high-dimensional parallel prompt-driven binary segmentation task, breaks through the limitation of the large model original paradigm under the premise of less structural modification, and combines a channel mutual exclusion and competition mechanism for end-to-end fine tuning. The application significantly improves the ground feature extraction precision and boundary integrity in complex scenes, and has the advantages of low training cost, strong generalization and flexible deployment.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing and computer vision technology, specifically relating to a method and system for extracting remote sensing multimodal ground features based on a large visual segmentation model and side-fitting technology. Background Technology

[0002] Semantic segmentation and feature extraction of high-resolution remote sensing imagery are core technologies for automated interpretation of land cover information and supporting intelligent geospatial perception. In recent years, with the rapid development of UAV (Unmanned Aerial Vehicle) technology, low-altitude remote sensing has become an important means of acquiring high spatiotemporal resolution land surface information. Compared to traditional satellite remote sensing, UAV imagery possesses ultra-high spatial resolution at the sub-meter or even centimeter level, and its acquisition methods are flexible and cost-effective, demonstrating enormous application potential in areas such as urban 3D modeling, refined land use surveys, disaster emergency assessment, and the construction of digital twin cities.

[0003] However, in practical engineering applications that utilize UAV imagery for the extraction of various types of ground features, existing technical solutions still face significant challenges.

[0004] First, the interpretation capability of a single optical mode in complex scenarios encounters a bottleneck.

[0005] While the extremely high spatial resolution of UAV imagery provides rich texture details, it also significantly increases the spectral heterogeneity within ground features (increased intra-class variance), leading to frequent salt-and-pepper noise. In complex urban or natural scenes, severe "same spectral, different objects" and "same object, different spectral" phenomena are prevalent. Relying solely on visual information is insufficient to capture the differences in the three-dimensional spatial structure of ground features, easily causing confusion and limiting further improvements in classification accuracy. Although introducing elevation geometry information provided by DSM (Digital Surface Model) can effectively help distinguish these ground features, existing fusion methods mostly employ simple channel stitching or element-wise addition, lacking adaptive suppression mechanisms for optical flow matching noise, holes, and jagged edges commonly found in UAV DSM data, making it difficult to achieve efficient complementarity of multimodal features.

[0006] Secondly, existing large-scale visual segmentation models are difficult to directly adapt to multi-class semantic segmentation tasks.

[0007] SAM3, as a large-scale pre-trained segmentation model based on a cue-response mechanism, has demonstrated powerful zero-shot generalization and object segmentation capabilities in natural image segmentation tasks. Trained on massive amounts of images, SAM3 can accurately segment foreground targets based on cues such as points, bounding boxes, and text, exhibiting good general visual feature extraction capabilities. However, directly transferring SAM3 to remote sensing multi-class ground feature segmentation tasks presents the following structural limitations:

[0008] Task paradigm mismatch: SAM3's native architecture is a binary classification model based on a prompt-response mechanism (i.e., distinguishing foreground objects from background), designed to segment specific objects of interest. Low-altitude remote sensing feature extraction, however, is a multi-class pixel-level classification task, requiring each pixel in the image to be assigned a unique and mutually exclusive category label (simultaneously outputting buildings, water bodies, roads, farmland, woodland, low vegetation, and others). SAM3 cannot natively support parallel output of multi-channel mutually exclusive semantic masks, limiting its application in full-feature mapping.

[0009] Modal input limitations: SAM3 is pre-trained on massive amounts of natural images, and its visual encoder only supports three-channel optical information input, making it unable to directly process single-channel DSM elevation data. Retraining it with all parameters to adapt to multimodal input would not only destroy the general visual representations already learned by the pre-trained model, but also be extremely computationally expensive.

[0010] In summary, how to construct a framework that can retain the powerful feature extraction capabilities of the SAM3 visual segmentation model, efficiently integrate DSM geometric information, and transform the binary classification paradigm of the large model into a semantic segmentation framework for multi-class land cover elements is a key technical problem that urgently needs to be solved in the field of intelligent interpretation of low-altitude remote sensing. Summary of the Invention

[0011] This invention aims to address the technical problems in existing remote sensing ground feature segmentation technologies, such as insufficient interpretation capability of single optical modal, low efficiency of DSM data fusion, and difficulty in adapting large visual models to multi-category segmentation tasks. It provides a method for extracting remote sensing multimodal ground feature elements based on large visual segmentation models and side-fitting technology.

[0012] This invention relates to a technique for refining the extraction of multiple types of ground features from remote sensing images containing elevation information by using a pre-trained visual segmentation large model SAM3 (Segment Anything Model 3, third-generation visual basic model) as a visual feature extractor, constructing an asymmetric two-stream network architecture by introducing a trainable lightweight DSM side adapter, and combining a cross-modal attention fusion mechanism and a parallel cue decoding strategy.

[0013] This invention constructs an asymmetric dual-stream network architecture, which retains the general feature extraction capability of the pre-trained visual large model SAM3, and introduces a trainable DSM side adapter and a cross-modal attention fusion mechanism to achieve efficient injection of elevation geometric information. At the same time, by using a parallel cue decoding strategy and a channel mutual exclusion competition mechanism, it breaks through the limitations of the original binary classification paradigm of the large model and realizes the refined extraction of multi-class land feature elements.

[0014] The specific technical innovations of this invention are as follows:

[0015] 1. DSM Side Adapter: To address the issue that large visual models cannot directly process elevation data, a lightweight, trainable DSM side adapter is designed. This adapter extracts DSM elevation geometry features and aligns the elevation geometry feature space to the visual feature space through a linear projection layer, enabling multimodal data access without disrupting the pre-trained weights of the large model.

[0016] 2. Attention Fusion Module: To address the difficulty of suppressing DSM noise in traditional fusion methods, a content-aware attention fusion module was used. This module uses the general visual features of the backbone network as queries and the extracted elevation geometric features as key / value pairs. It adaptively calculates the weights of the elevation geometric features at different spatial locations and dynamically injects the weighted elevation geometric information into the visual feature stream through gated residual connections, significantly enhancing the model's 3D perception capability for "different objects with the same spectrum".

[0017] 3. Parallel Hint Decoding Strategy: To address the limitation of large models natively supporting only single-task binary segmentation, a parallel hint decoding strategy based on virtual batch construction is proposed. By broadcasting multimodal fusion features at the batch dimension and constructing multi-class semantic hints as a set of parallel query vectors, the decoder can process segmentation requests of all classes in parallel during a forward propagation process, thereby transforming "multi-task serial" into "single-time parallel inference".

[0018] 4. Channel Mutual Exclusion and Competition Mechanism: To address the issues of class imbalance and blurred boundaries in remote sensing images, a joint loss function incorporating multi-class cross-entropy (CE) and Dice coefficients was constructed. Utilizing the normalization property of the Softmax function, multiple independently generated binary masks are forced to compete probabilistically at the pixel level, thereby transforming non-mutually exclusive binary predictions into mutually exclusive multi-class semantic distributions. This effectively improves the segmentation accuracy of small sample classes and the integrity of ground feature edges.

[0019] The specific technical solution is as follows:

[0020] A method for extracting low-altitude remote sensing ground features by integrating a DSM side adapter includes the following steps:

[0021] Step 1: Acquire low-altitude imagery and DSM data from different regions and perform preprocessing.

[0022] Step 2: Construct a feature extraction network that includes a main visual branch and side adaptation branches:

[0023] The main visual branch uses a pre-trained and frozen SAM3 image encoder to extract the visual feature pyramid; the side adaptation branch uses a DSM side adapter to extract elevation geometry features from DSM data and align them to the visual feature space through an end projection layer.

[0024] Step 3: Construct a cross-modal fusion module. Use the visual features of the main visual branch as the query and the elevation geometric features of the side adaptation branch as the key and value. Calculate the spatial location-related attention weight map through a multi-head attention mechanism to dynamically aggregate effective elevation geometric information. Then, smoothly inject the aggregated elevation geometric information into the visual features through gated residual connections to generate multimodal enhanced features.

[0025] Step 4: Combining text prompts and prompt encoders, a parallel decoding task is constructed based on virtual batches to obtain matching features;

[0026] Step 5: Input the matched features into the mask decoder, construct the joint loss for training, and obtain multi-class semantic segmentation results based on mutual exclusion competition during backpropagation.

[0027] Furthermore, in step 1, the low-altitude image data is preprocessed and the corresponding ground truth labels for ground features are annotated; the DSM data is linearly normalized; and a semantic segmentation dataset containing training, validation, and test sets is constructed by combining the manually interpreted ground truth labels for ground features.

[0028] Furthermore, the DSM side adapter contains four consecutive convolutional blocks, with a convolutional projection layer connected at the end of each block. This DSM side adapter processes single-channel normalized DSM data. Hierarchical encoding for elevation geometric features It aligns with the visual features of the main visual branches in terms of dimension.

[0029] Furthermore, the cross-modal fusion module performs cross-modal fusion on each layer of visual features output by the image encoder. Where i is the visual feature index, the following fusion operation is performed:

[0030] (1) Spatial alignment: using bilinear interpolation to align elevation geometric features Adjust to Same spatial resolution;

[0031] (2) Feature normalization: Features are layer-normalized to eliminate dimensional differences in feature distribution and ensure the numerical stability of attention calculation, thus obtaining... and Where Q represents the query, and K and V represent the key and value, respectively. Representation layer normalization;

[0032] (3) Attention convergence: That is, using visual features as queries, adaptively weighting the DSM elevation geometry features, and introducing... As a scaling factor, the dot product result is adjusted to a numerically stable range. Indicates the activation function; This represents the elevation geometric context features after weighting, enabling dynamic aggregation of elevation information guided by visual content.

[0033] (4) Gated residual connection: ,in Learnable parameters initialized to 0. This is a multimodal enhancement feature.

[0034] Furthermore, a CLIP-based text encoder is used as the cue encoder to transform the natural language descriptions of land cover categories, i.e., text cue words, into high-dimensional semantic vectors. For N land cover categories, N text cue words are input, and the cue encoder maps these text cue words into text cue embeddings. , where D is the embedding dimension.

[0035] Furthermore, a lightweight Transformer decoder architecture is adopted as the mask decoder, including a self-attention layer and a mutual attention layer. The self-attention layer processes the text embedding vector from the cue encoder, and uses the semantic vector enhanced by self-attention as the query vector. Multimodal enhancement features come from the cross-modal feature fusion module. The key and value are assigned to each other. By calculating the dot product similarity between the query vector and the key, the deep alignment of cross-modal information is achieved, and the final output is an object query vector containing semantic information. The mask decoder is connected to a dynamic mask prediction head at the end to generate the final mask.

[0036] Furthermore, the parallel decoding task refers to transforming a single-graph multi-class task into a multi-graph single-class parallel task, assuming the physical training batch size is... The number of categories is The algorithm generates text cue embeddings for N categories using a cue encoder; and utilizes tensor broadcasting to integrate the fused multimodal augmented features. Copy N times in the batch dimension to generate a virtual batch size. The extended feature tensor; the text prompts of N categories are extended to... To ensure a matching of prompts, N text prompts are embedded and matched one-to-one with the extended feature tensors in the virtual batch.

[0037] Furthermore, during training, the learning rate used by the DSM side adapter and the cross-modal feature fusion module is set to... To facilitate rapid learning and alignment of elevation geometry features; for both the mask decoder and the dynamic mask prediction head, a learning rate of less than [value missing] is used. The value is fine-tuned.

[0038] Furthermore, step 5 constructs a joint loss function based on channel mutual exclusion and competition mechanisms, including:

[0039] Multi-class cross-entropy loss By category dimension N Upward introduction Softmax The normalization operation establishes a mutual exclusion mechanism between channels, and the calculation formula is as follows:

[0040] ;

[0041] in, M The total number of pixels in the image. For pixels i Category c The true label, For pixels i Category c The probability of;

[0042] Multiclass Dice loss The calculation formula is:

[0043] ;

[0044] in, To prevent the denominator from being zero, a smoothing coefficient is used;

[0045] Final joint loss function Defined as:

[0046] ;

[0047] in and As weight.

[0048] The present invention also provides a low-altitude remote sensing feature extraction system integrating a DSM side adapter, comprising: a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a low-altitude remote sensing feature extraction method integrating a DSM side adapter as described in the above technical solution.

[0049] Compared with the prior art, the technical advantages of the present invention are as follows:

[0050] 1. Non-intrusive architecture extension for efficient multimodal fusion: The side adapter proposed in this invention adopts a lightweight design independent of the backbone network, enabling non-intrusive extension of pre-trained large visual segmentation models. It eliminates the need for full parameter fine-tuning of the visual backbone with hundreds of millions of parameters; only a side network with a minimal number of parameters is required to complete feature extraction and injection of the DSM elevation geometry modality. This design significantly reduces computational resource consumption and training time costs, achieving low-cost, high-efficiency extension of multimodal capabilities.

[0051] 2. Cross-modal content-aware fusion, breaking through the bottleneck of "different objects with the same spectrum": This invention achieves content-aware feature fusion by using a cross-modal attention mechanism, with visual features as queries and elevation geometric features as keys. The model can adaptively and dynamically weight elevation information in different land feature areas, effectively alleviating the phenomena of "different objects with the same spectrum" and "different spectra of the same object", and improving the semantic extraction capability of land feature elements in complex scenarios.

[0052] 3. Innovative Task Paradigm Shift, Reducing the Need for Modification and Adaptation of Large Model Structures: The parallel cue decoding strategy proposed in this invention utilizes tensor broadcasting and virtual batch construction techniques to reconstruct the traditional multi-class semantic segmentation task into a high-dimensional parallel binary cue segmentation task. This strategy achieves synchronous output and mutual exclusion optimization of multi-class semantic masks without altering the original encoder-decoder topology of the large visual model. This provides a general and efficient solution for rapidly migrating large models natively used for binary classification to multi-class remote sensing interpretation tasks.

[0053] 4. Joint Constraints and Smooth Injection: Balancing Training Stability and Segmentation Accuracy: The mutually exclusive competition mechanism constructed in this invention jointly utilizes multi-class CE loss and Dice loss. The former forces inter-channel competition through Softmax, ensuring the uniqueness of pixel classification; the latter directly optimizes region overlap, solving the boundary fragmentation problem caused by class imbalance in remote sensing images. Simultaneously, a gated residual injection mechanism (initialized to zero) is employed. The parameters ensure that the model can smoothly reuse the pre-trained feature distribution of the large model in the early stage of training, avoid optimization oscillations caused by random initialization of side networks, and improve the stability and convergence speed of the training process. Attached Figure Description

[0054] Figure 1 This is an overall process architecture diagram of an embodiment of the present invention;

[0055] Figure 2 This is a diagram of the asymmetric two-stream feature extraction network structure in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the cross-modal feature fusion module structure based on attention mechanism and gated residual method in an embodiment of the present invention;

[0057] Figure 4 This is a logical diagram of the parallel prompt decoding strategy in an embodiment of the present invention;

[0058] Figure 5 This is a detailed network structure diagram of the DSM side adapter in an embodiment of the present invention;

[0059] Figure 6 This is a schematic diagram of the ground feature extraction results according to an embodiment of the present invention. Detailed Implementation

[0060] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0061] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for extracting ground features from low-altitude remote sensing images by fusing a large visual segmentation model with a DSM side adapter, comprising the following steps:

[0062] S1) Data collection and preprocessing; acquiring low-altitude image data and DSM data from different cities or regions; preprocessing the low-altitude image data and labeling the corresponding ground truth labels of ground features; performing linear normalization on the DSM data; and constructing a semantic segmentation dataset containing training, validation, and test sets by combining the manually interpreted ground truth labels of ground features.

[0063] S2) Construct a feature extraction network that includes a main visual branch and a side adaptation branch; the main visual branch uses a pre-trained and frozen SAM3 image encoder to extract visual feature pyramids; the side adaptation branch uses the lightweight DSM side adapter designed in this invention to extract elevation geometric features from DSM data and align them to the visual feature space through the end projection layer.

[0064] S3) Cross-modal feature fusion based on attention mechanisms and gated residuals: At each level of the feature pyramid, cross-modal fusion modules are used for feature enhancement. Specifically, visual features of the main branches are used as queries, and elevation geometric features of the side branches are used as keys and values. A multi-head attention mechanism is used to calculate spatially relevant attention weight maps, dynamically aggregating effective elevation geometric information. Subsequently, a learnable gated scalar is introduced, and the aggregated elevation geometric information is smoothly injected into the visual feature stream through gated residual connections, generating multi-modal enhanced features.

[0065] S4) Parallel decoding task based on virtual batches; the total number of land cover categories is set to [number] based on the parallel cue decoding strategy. N Generate by prompting the encoder N Text prompts corresponding to each category are embedded; tensor broadcasting is used to replicate the multimodal enhancement features in the batch dimension. N Next, construct virtual task batches. Then, N Each semantic text embedding is matched one-to-one with the feature samples in the virtual batch, and multi-task feature extraction is achieved by using a multi-class single-task model in parallelization.

[0066] S5) Parallel decoding and semantic generation based on mutual exclusion competition; the matched features are input into the mask decoder for parallel generation. N An independent binary mask is used; the mask is redimensionally reorganized to construct a joint loss function containing multi-class CE (cross-entropy) and Dice coefficients. By utilizing the normalization property of the Softmax function, N channels at the same pixel position are forced to compete for probability during backpropagation, thereby obtaining mutually exclusive and boundary-complete multi-class semantic segmentation results.

[0067] like Figure 2 As shown, this invention proposes an asymmetric dual-stream remote sensing feature extraction network based on the SAM3 visual segmentation large model. This network mainly consists of a frozen backbone visual branch, a trainable side-adaptive branch, a cross-modal fusion module, and a cue-driven decoder (including a cue encoder and a mask decoder).

[0068] 1. Main Vision Branch: Employs the image encoder from the pre-trained SAM3 large-scale vision model. This encoder, based on the ViT-Large (Vision Transformer) network architecture, is used to process the input UAV optical imagery. Deep semantic features are extracted and a feature pyramid is output through Patch Embedding (block encoding) layer and multiple Transformer Blocks. .in The resolution is the same as the original image. The number of channels is uniformly mapped to .

[0069] 2. Side-adaptive branch: To address the segmentation bottleneck of a single optical modality in complex low-altitude remote sensing scenarios, elevation information is introduced. This invention designs a DSM side adapter, responsible for transforming the physical elevation information of the DSM into a high-dimensional elevation geometric representation that can interact with the feature space of the large visual model, and achieving adaptive fusion through a content-aware mechanism. The specific implementation steps are as follows:

[0070] like Figure 5 As shown, a parallel-training side adapter is constructed, containing four consecutive convolutional blocks (Conv-BatchNorm-GELU, i.e., convolution-batch normalization-activation function, stride of 2), with a concatenation function connected at the end of each block. The Conv projection layer. This network projects single-channel normalized DSM data. Step-by-step encoding into elevation geometric feature map The resolution is the same as the original image. The number of channels is projected to 256, thereby aligning with the main visual features in terms of dimensions.

[0071] 3. Cross-modal fusion module: such as Figure 3 As shown, for each layer of visual features output by the image encoder Perform the following fusion operations:

[0072] (1) Spatial alignment: using bilinear interpolation to align elevation geometric features Adjust to Same spatial resolution.

[0073] (2) Feature normalization: Features are subjected to layer normalization (LayerNorm) to eliminate dimensional differences in feature distribution and ensure the numerical stability of attention calculation, resulting in... and .

[0074] (3) Attention convergence: This step utilizes visual features as the query, adaptively weights the DSM elevation geometry features, and introduces... As a scaling factor, the dot product result is adjusted to a numerically stable range, thereby ensuring the gradient stability of the model during multimodal feature alignment training. This represents the weighted elevation geometric context features, enabling dynamic aggregation of elevation information guided by visual content. This mechanism achieves content-aware fusion: for example, at building edges, abrupt changes in visual features lead to increased attention weights, significantly introducing elevation abrupt changes in the DSM; while in flat water areas, the weights automatically decrease, thus suppressing DSM noise.

[0075] (4) Gated residual connection: ,in These are learnable parameters initialized to 0. This ensures the model can be "warm-started" in the optimal state of the pre-trained SAM3, avoiding the disruption of the general feature distribution already learned by the large model due to random initialization of side networks. As training progresses, Gradually learn non-zero values ​​to achieve smooth injection of elevation geometry information.

[0076] 4. The cue-driven decoder includes: (1) a cue encoder: a text encoder based on the CLIP (Contrastive Language-Image Pre-training) architecture, which transforms the natural language descriptions of land cover categories ("buildings", "water bodies", "roads", "farmland", "woodland", "low vegetation" and "others") into high-dimensional semantic vectors. For N Enter the land cover category. N Each text prompt. The encoder maps these texts into text embedding vectors. ,in D For the embedding dimension. These vectors serve as semantic anchors, guiding the decoder to accurately locate visual regions that match the text description within a complex feature space.

[0077] (2) Mask Decoder: A lightweight Transformer Decoder architecture is adopted, including a self-attention layer and a cross-attention layer. First, the text embedding vectors from the cue encoder are processed by the self-attention layer, and these vectors are fully connected to each other for interactive computation. This process aims to establish the contextual dependencies and feature reorganization within the cue, enabling the semantic query vector to enhance itself according to its own semantic attributes, thereby preparing the representation for accurate localization in the complex feature space. Subsequently, the cross-attention layer is used to perform the mapping retrieval from the "semantic space" to the "visual-geometric space". In this process, the semantic vector enhanced by self-attention serves as the query vector, while the multimodal enhanced features from the cross-modal feature fusion module are used. The key and value are assigned to each element. By calculating the dot product similarity between the query vector and the key, the model can accurately "retrieve" spatial regions on the fused feature map that are semantically consistent with the text description. It then aggregates the visual texture and elevation geometry features of these regions into the query vector, achieving deep alignment of cross-modal information and ultimately outputting an object query vector containing semantic information. A dynamic mask prediction head is connected to the decoder to generate the final mask.

[0078] Since the native SAM3 architecture only supports binary segmentation and most parameters are frozen, this invention employs a parallel cue decoding strategy based on batch stacking, combined with hierarchical fine-tuning for training, such as... Figure 4 As shown. Specifically includes:

[0079] 1. Parallel hint decoding strategy: Transform the "single-image multi-class" task into a parallel "multi-image single-class" task. Let the physical training batch size be... Number of categories (Buildings, water bodies, roads, farmland, woodland, low vegetation, and others). Using tensor operations, the fused multimodal feature maps are... Batch-level copying N Next, generate virtual batch size The extended feature tensor. N The text embeddings of each category are expanded to To ensure prompt matching, a prompt-driven decoder is used, which combines 28 sets of features with the prompt input. The mask decoder treats these as 28 independent binary classification tasks and performs parallel computation. Multi-class segmentation output is achieved without modifying the underlying model structure.

[0080] 2. Layered Learning Rate Optimization Strategy: Given that this invention employs an asymmetric two-stream architecture, including a frozen main visual branch, a side-adaptation branch, and a cue-driven decoder, a layered learning rate strategy is adopted to balance training stability and convergence speed. The DSM side-adaptor and the cross-modal feature fusion module utilize higher learning rates. This facilitates rapid learning and alignment of elevation geometry features; in this embodiment, the mask decoder and dynamic mask prediction head employ... Fine-tuning can be performed; in practical applications, a value smaller than [a certain value] can be used. The value is fine-tuned. This strategy ensures rapid learning of new modal features while preventing catastrophic forgetting of pre-trained knowledge.

[0081] To achieve high-precision multi-class semantic segmentation without altering the SAM3 binary segmentation architecture and effectively address the prevalent class imbalance problem in remote sensing images, this invention constructs a joint loss function based on channel mutual exclusion and competition mechanisms. The specific implementation steps are as follows:

[0082] 1. Predictive Tensor Reconstruction: The original tensor shape output by the dynamic mask prediction head is as follows: ,in The dimension represents the virtual batch size for parallel processing. For multi-class semantic supervision, it is first reorganized into... Standard semantic segmentation prediction tensor ,in B Physical batch size N This represents the total number of land feature categories.

[0083] 2. Multi-class CE loss: Although parallel prompt decoding physically performs... N Each independent binary mask generation task outputs... Logits value They are unrelated, but this invention achieves this through the category dimension. N Upward introduction Softmax The normalization operation artificially creates a mutual exclusion mechanism between channels. The calculation formula is as follows:

[0084] ,

[0085] in, M The total number of pixels in the image. For pixels i Category c The true label, probability The calculation is the core:

[0086] ,

[0087] use Softmax The normalization property of the same pixel position superior N Each independent Logits value is forced to map to a probability distribution that sums to 1, compelling the model to suppress the probability of non-target classes and increase the probability of target classes at the same pixel location. This forces the same pixel location to be mapped to a probability distribution that sums to 1 during backpropagation. N Each category channel generates probability competition, achieving mutual exclusion of categories.

[0088] 3. Multi-class Dice loss: Used to optimize region overlap, it is highly robust to small sample classes (such as narrow roads or small bodies of water). The calculation formula is:

[0089] ,

[0090] in, The predicted probability after Softmax normalization. To prevent smoothing coefficients with zero denominators, this loss directly optimizes the upper bound of IoU, effectively mitigating the inundation effect of large-area background on the gradient of small sample targets.

[0091] 4. Joint optimization strategy: The final total loss function Defined as the weighted sum of the two above. The calculation formula is:

[0092] ,

[0093] In this embodiment, the following settings are provided: Pixel classification accuracy and region integrity are optimized with equal weight.

[0094] Figure 6 This paper demonstrates the multi-class segmentation performance of the proposed DSM-fused side adapter method in processing low-altitude remote sensing imagery. The results are compared with current mainstream segmentation networks (U-Net, Swin-Transformer) and the original SAM3 model without DSM fusion, showcasing the significant advantages of this invention in using DSM information to help distinguish 'same-spectrum, different-object' features and maintain the integrity of ground object edges. Compared to the boundary blurring and internal salt-and-pepper noise issues observed in traditional methods, the extraction results of this invention show more complete ground object edges, with smoother and more consistent internal regions, closely resembling the ground truth mask. This proves the effectiveness of the mutual exclusion mechanism and multimodal fusion strategy in improving refined extraction.

[0095] This invention also provides a low-altitude remote sensing feature extraction system integrating a DSM side adapter, comprising: a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a low-altitude remote sensing feature extraction method integrating a DSM side adapter as described above.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A low-altitude remote sensing feature extraction method of a fusion DSM side edge adapter, characterized in that, Includes the following steps: Step 1: Acquire low-altitude imagery and DSM data from different regions and perform preprocessing. Step 2: Construct a feature extraction network that includes a main visual branch and side adaptation branches: The main visual branch uses a pre-trained and frozen SAM3 image encoder to extract the visual feature pyramid; the side adaptation branch uses a DSM side adapter to extract elevation geometry features from DSM data and align them to the visual feature space through an end projection layer. Step 3: Construct a cross-modal fusion module. Use the visual features of the main visual branch as the query and the elevation geometric features of the side adaptation branch as the key and value. Calculate the spatial location-related attention weight map through a multi-head attention mechanism to dynamically aggregate effective elevation geometric information. Then, smoothly inject the aggregated elevation geometric information into the visual features through gated residual connections to generate multimodal enhanced features. Step 4: Combining text prompts and prompt encoders, a parallel decoding task is constructed based on virtual batches to obtain matching features; Step 5: Input the matched features into the mask decoder, construct the joint loss for training, and obtain multi-class semantic segmentation results based on mutual exclusion competition during backpropagation.

2. The method for low-altitude remote sensing feature extraction of a fusion DSM side adapter according to claim 1, characterized in that: In step 1, the low-altitude image data is preprocessed and the corresponding ground truth labels for ground features are annotated; the DSM data is linearly normalized; and a semantic segmentation dataset containing training, validation and test sets is constructed by combining the manually interpreted ground truth labels for ground features.

3. The method for extracting low-altitude remote sensing ground features by integrating a DSM side adapter as described in claim 1, characterized in that: The DSM side adapter contains four consecutive convolutional blocks, with a convolutional projection layer connected to the end of each block. This DSM side adapter processes single-channel normalized DSM data. Hierarchical encoding for elevation geometric features It aligns with the visual features of the main visual branches in terms of dimension.

4. The method for extracting low-altitude remote sensing ground features by integrating a DSM side adapter as described in claim 1, characterized in that: The cross-modal fusion module provides visual features for each layer output by the image encoder. Where i is the visual feature index, the following fusion operation is performed: Spatial alignment: using bilinear interpolation to align elevation geometry features Adjust to Same spatial resolution; Feature normalization: Features are layer-normalized to eliminate dimensional differences in feature distribution, ensuring numerical stability of attention computation, and obtaining... and Where Q represents the query, and K and V represent the key and value, respectively. Representation layer normalization; Attention convergence: That is, using visual features as queries, adaptively weighting the DSM elevation geometry features, and introducing... As a scaling factor, the dot product result is adjusted to a numerically stable range. Indicates the activation function; This represents the elevation geometric context features after weighting, enabling dynamic aggregation of elevation information guided by visual content. Gated residual connection: ,in Learnable parameters initialized to 0. This is a multimodal enhancement feature.

5. The method for extracting low-altitude remote sensing ground features by integrating a DSM side adapter as described in claim 1, characterized in that: A CLIP-based text encoder is used as the cue encoder to transform the natural language descriptions of land cover categories, i.e., text cue words, into high-dimensional semantic vectors. For N land cover categories, N text cue words are input, and the cue encoder maps these text cue words into text cue embeddings. , where D is the embedding dimension.

6. The method for extracting low-altitude remote sensing ground features by integrating a DSM side adapter as described in claim 1, characterized in that: A lightweight Transformer decoder architecture is employed as the mask decoder, comprising self-attention layers and mutual attention layers. The self-attention layer processes the text embedding vector from the cue encoder, and uses the self-attention-enhanced semantic vector as the query vector. Multimodal enhancement features are derived from the cross-modal feature fusion module. It serves as both key and value, and achieves deep alignment of cross-modal information by calculating the dot product similarity between the query vector and the key, ultimately outputting an object query vector containing semantic information; The mask decoder is connected to a dynamic mask prediction head at the end, which is used to generate the final mask.

7. The method for extracting low-altitude remote sensing ground features by integrating a DSM side adapter as described in claim 1, characterized in that: Parallel decoding tasks refer to transforming a single-graph multi-class task into a multi-graph single-class parallel task. Let the physical training batch size be... The number of categories is The algorithm generates text cue embeddings for N categories using a cue encoder; and utilizes tensor broadcasting to integrate the fused multimodal augmented features. Copy N times in the batch dimension to generate a virtual batch size. The extended feature tensor; Expand the embedding of text prompts in N categories to: To ensure a matching of prompts, N text prompts are embedded and matched one-to-one with the extended feature tensors in the virtual batch.

8. The method for extracting low-altitude remote sensing ground features by integrating a DSM side adapter as described in claim 6, characterized in that: During training, the learning rate used by the DSM side adapter and the cross-modal feature fusion module was set to... To facilitate rapid learning and alignment of elevation geometry features; for both the mask decoder and the dynamic mask prediction head, a learning rate of less than [value missing] is used. The value is fine-tuned.

9. The method for extracting low-altitude remote sensing ground features by integrating a DSM side adapter as described in claim 1, characterized in that: Step 5 constructs a joint loss function based on channel mutual exclusion and contention mechanisms, including: Multi-class cross-entropy loss By category dimension N Upward introduction Softmax The normalization operation establishes a mutual exclusion mechanism between channels, and the calculation formula is as follows: ； in, M The total number of pixels in the image. For pixels i Category c The true label, For pixels i Category c The probability of; Multiclass Dice loss The calculation formula is: ； in, To prevent the denominator from being zero, a smoothing coefficient is used; Final joint loss function Defined as: ； in and As weight.

10. A low-altitude remote sensing ground feature extraction system integrating a DSM side adapter, characterized in that, include: A processor and memory, wherein the memory is used to store program instructions, and the processor is used to execute the stored instructions as described in claim 1. The method for extracting low-altitude remote sensing features by incorporating the DSM side adapter as described in any one of the nine claims.

Images