Remote sensing semantic segmentation method coupled with multi-modal and visual large model and product

Abstract

This invention provides a remote sensing semantic segmentation method and product that couples multimodal data with a large visual model, relating to the field of remote sensing image processing technology. In this invention, the additional modalities are no longer treated as inputs equal to the native modalities (RGB images) of the large visual model, but rather as cues for the primary modality. Specifically, by treating multimodal data as cues with different remote sensing attributes, the perceptual dimensions of the desired modality can be dynamically injected according to the specific downstream task, and corresponding segmentation results can be provided. This invention utilizes a hybrid expert model mechanism to construct a multimodal cue, which sparsifies the model structure while extracting multimodal features, thus controlling the computational load.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a remote sensing semantic segmentation method and product that couples multimodal and large visual models. Background Technology

[0002] Semantic segmentation is a pixel-level classification method for images, playing a crucial role in downstream remote sensing tasks such as building extraction, land use mapping, environmental monitoring, and precision agriculture. To overcome the performance limitations of segmentation, pioneers have focused on optimizing algorithm structures to improve segmentation performance in specific remote sensing scenarios. Examples include constructing network architectures that benefit from multi-scale features and global context, deploying combinations of operators with global-local extraction capabilities, and designing novel attention mechanisms. These studies continue to drive the development of semantic segmentation technology and remain a major research area within the remote sensing community.

[0003] In recent years, with the rapid development of artificial intelligence technology, general-purpose visual foundational models, represented by large-model technology, have opened up new tracks for visual tasks, including semantic segmentation. Unlike dedicated semantic segmentation models built for specific task scenarios, large-model visual systems benefit from rich visual priors, easily adapting to downstream tasks with only simple fine-tuning, without needing to train from scratch. SAM is the pioneering work of large-model visual systems; it uses self-supervised learning and mask generation as proxy tasks and has undergone extensive pre-training on massive amounts of natural data, exhibiting strong versatility and generalization capabilities. SAM2 is the latest version of SAM, providing a new option for remote sensing semantic segmentation technology driven by large models. However, its advantages diminish when applied to specialized fields such as remote sensing, mainly in three aspects: cross-domain model adaptation, multi-classification requirements, and end-to-end automatic segmentation.

[0004] First, the upstream pre-training of SAMs (SAM and SAM2) mostly uses natural images, which differ significantly from remote sensing images captured by spaceborne and airborne sensors in terms of inductive biases such as field of view, scale, and angle. Therefore, optimization for downstream datasets is necessary. Regardless of the cues used, SAMs have limited zero-shot capability when directly processing remote sensing images. Second, the proxy task of SAMs is essentially a cue-based binary mask prediction task. When using sparse cues such as points or boxes for segmentation, even with multiple land cover categories labeled, the entire image is only divided into foreground and background. When using general image segmentation operations, the model uniformly samples the image to be segmented and cuts it into multiple semantically unknown classes based on a threshold. For atypical land cover, the model struggles to distinguish them; and for the same category, it further segments it into multiple unnecessary image patches based on the threshold, resulting in a significant deviation from the desired semantic segmentation result. This cue-based interactive approach is not end-to-end, greatly hindering its application in large-scale automatic remote sensing image segmentation tasks. To address these issues, while it is natural to follow the pre-training-fine-tuning paradigm of existing large models to adapt to downstream remote sensing tasks, how to combine it with remote sensing multimodalities, especially in modality-agnostic scenarios, remains an area for further exploration.

[0005] Multimodal (or multi-source) semantic segmentation can leverage remote sensing images from various sensors providing different perceptual information to obtain more comprehensive and accurate results. For example, combining RGB images with Digital Surface Model (DSM) data can enhance inter-class differences for highly sensitive objects, resulting in more accurate semantic segmentation. Benefiting from its advantages in improving performance in complex scenes, research on multimodal semantic segmentation models has also made significant progress in recent years, but limitations still exist. Most previous multimodal semantic segmentation models required training from scratch or initialization based on pre-trained weights from ImageNet-1k, resulting in extremely limited prior visual knowledge available to the models. Therefore, how to combine multimodal and large-scale visual models, enjoying the large-scale visual priors of SAM2 while possessing multiple perceptual dimensions, to maximize the performance boundaries of semantic segmentation, is an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a remote sensing semantic segmentation method and product that couples multimodal and large visual models, in order to at least partially solve the above-mentioned problems.

[0007] The first aspect of this invention provides a remote sensing semantic segmentation method coupled with a multimodal and large visual model, the method comprising:

[0008] Acquire multimodal data, and input the RGB image from the multimodal data into a pre-trained image encoder in a remote sensing semantic segmentation model; the remote sensing semantic segmentation model includes: an image encoder, a multimodal cue, and a category decoder;

[0009] The image encoder of the remote sensing semantic segmentation model encodes the RGB images in the multimodal data to obtain image embeddings;

[0010] The multimodal prompter integrates the multimodal data into a multimodal feature matrix and provides prompts to the output of the image encoder in the form of channel concatenation weighting, thereby obtaining the prompted image embedding;

[0011] The category decoder obtains the image embedding after the prompt and outputs the probability distribution results for multi-class classification.

[0012] Optionally, the image encoder inherits from a large visual model, and the image encoder is trained using any one of the following five fine-tuning methods:

[0013] Only fine-tune the decoder part; do not update any parameters of the image encoder.

[0014] Only fine-tune the bias terms of all linear layers;

[0015] Add parallel, trainable mini-adaptors after the MLP layer of each Transformer architecture and adjust the embedding layer of the image encoder.

[0016] Reparameterize the query, key, and value matrices of the self-attention layer using low-rank matrices;

[0017] Update all parameters of the image encoder.

[0018] Optionally, the method further includes:

[0019] The remote sensing semantic segmentation model receives a list provided by the user containing information on the number of channels for each modality. It then identifies the number of modalities in the input multimodal data and initializes the corresponding multimodal cue.

[0020] Optionally, the multimodal prompter includes a hard hybrid expert model and a soft hybrid expert model; the multimodal prompter integrates the multimodal data into a multimodal feature matrix, and provides prompts for the output of the image encoder in the form of channel concatenation weighting, including:

[0021] The hard hybrid expert model activates a suitable expert network to learn features based on the data characteristics of each modality, thereby obtaining a feature map for each modality;

[0022] The feature maps of each modality are connected and normalized to obtain a multimodal feature map;

[0023] The soft hybrid expert model obtains a multimodal cue mask based on the heterogeneous aggregation of complementary features of the multimodal feature map.

[0024] Optionally, the hard hybrid expert model includes a gating network and several expert networks; the hard hybrid expert model activates a suitable expert network based on the data characteristics of each modality to perform feature learning, obtaining a feature map for each modality, including:

[0025] The gated network performs global max pooling and average pooling in parallel in the spatial dimension on the feature maps of each input modality, and weights the results of max pooling and average pooling and performs dimensional transformation through a reshaping operation.

[0026] For each modality, the expected probability assigned to each expert network is calculated using trainable gating weights, and the top scorers are grouped together. k One expert network is selected as the proposed activation expert network. The probability of the proposed activation expert network is retained, and the probabilities of other expert networks are filled with -∞. After a Softmax operation, the final activation is obtained. k Normalized weight scores of each expert network The activated expert network is used as the feature extractor for the current modality to obtain the feature map of the current modality.

[0027] Optionally, the soft hybrid expert model aggregates complementary features based on the heterogeneity of the multimodal feature maps, including:

[0028] The soft hybrid expert model concatenates multimodal feature maps along the channel dimension, resulting in three feature representations: , , ,calculate and The cosine distance score between them is then assigned to the corresponding value. Adaptive aggregation based on modal heterogeneity:

[0029]

[0030] in, Indicates divided into n The group performs a linear projection operation, for and First M × M Perform max pooling on small blocks and then linear transformation; This indicates that it is expanded into a vector along the channel dimension. ; This represents the relative positional bias between modalities; two Softmax operations are used to normalize the weights; the final result is reshaped into... ;

[0031] The final multimodal cue mask is obtained by performing full-path fusion and using residual connections.

[0032] A second aspect of this invention provides a remote sensing semantic segmentation device coupled with a multimodal and visual large model, the device comprising:

[0033] The acquisition module is used to acquire multimodal data and input the RGB image from the multimodal data into a pre-trained image encoder in the remote sensing semantic segmentation model; the remote sensing semantic segmentation model includes: an image encoder, a multimodal cue, and a category decoder;

[0034] The encoding module is used by the image encoder of the remote sensing semantic segmentation model to encode the RGB images in the multimodal data to obtain image embeddings;

[0035] The prompting module is used by the multimodal prompter to integrate the multimodal data into a multimodal feature matrix, and to provide prompts to the output result of the image encoder in the form of channel concatenation and weighting, so as to obtain the image embedding after prompting;

[0036] The decoding module is used by the category decoder to obtain the image embedding after the prompt and output the probability distribution results of the multi-class classification.

[0037] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes a remote sensing semantic segmentation method coupled with a multimodal and visual large model as described in the first aspect of the present invention.

[0038] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the remote sensing semantic segmentation method coupled with a multimodal and visual large model as described in the first aspect of the present invention.

[0039] A fifth aspect of the present invention provides a computer program product, including a computer program / instructions, which are implemented by a processor as the steps in the remote sensing semantic segmentation method coupled with a multimodal and visual large model as described in the first aspect of the present invention.

[0040] In this embodiment of the invention, the additional modalities are no longer treated as inputs equal to the native modalities (RGB images) of the large visual model, but rather as cues for the main modality. Specifically, in the native large visual model, the user randomly provides cues in the form of points and boxes, and the model can dynamically provide different outputs. Therefore, by treating multimodal data as cues with different remote sensing attributes, the perceptual dimensions of the desired modality can be dynamically injected according to the specific downstream task, and corresponding segmentation results can be provided. To balance modal heterogeneity and computational efficiency, this embodiment of the invention introduces a hybrid expert model mechanism, constructing a hard hybrid expert model and a soft hybrid expert model to form the multimodal cue for the remote sensing semantic segmentation model. The hard hybrid expert model consists of multiple heterogeneous modal expert networks and a gating network. The expert networks can extract the heterogeneous features of the multimodal data accordingly; the gating network is used to ensure the sparsity of the cue to control the incremental computational overhead. The soft hybrid expert model selects complementary features to the greatest extent possible based on the vector relationships between the multimodal feature maps. Furthermore, to meet the cross-domain requirements of large-scale visual models from natural scenes to remote sensing, this invention introduces several general large-scale model fine-tuning methods to adapt to the image encoder. Simultaneously, considering that the point- and box-based interactive working mode of large-scale visual models is not suitable for large-scale automated segmentation applications, this invention also introduces a standard category decoder for end-to-end modification, thereby reducing manual costs. Therefore, the remote sensing semantic segmentation method provided by this invention operates in a one-step manner, requiring no pre-processor, and only needs to update a small portion of the model's weight increments during training. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of the steps of the remote sensing semantic segmentation method coupled with a multimodal and visual large model provided in the embodiments of the present invention;

[0043] Figure 2 This is a schematic diagram of the architecture of the remote sensing semantic segmentation model in the remote sensing semantic segmentation method that couples multimodal and visual large models provided in the embodiments of the present invention.

[0044] Figure 3 This is a schematic diagram of the multimodal cue in the remote sensing semantic segmentation model of the remote sensing semantic segmentation method coupled with a large visual model provided in the embodiments of the present invention. Detailed Implementation

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Constructing a general approach that combines multimodal and large-scale models is challenging, requiring overcoming modality agnosticity. Specifically, this invention defines modality agnosticity as the unpredictable number and types of modalities involved in different downstream tasks. These typically depend on the user's available data resources and whether the modalities carry semantics beneficial to the segmented object. Previous multimodal semantic segmentation models built for specific tasks largely assumed known modalities, limiting their general applicability. Furthermore, the pre-training of large-scale visual models (e.g., SAM2) is conducted without emphasizing specific downstream scenarios, and constructing additional proxy tasks upstream to pre-cover an infinite number of spatiotemporally consistent remote sensing modalities is impractical. In sufficiently broad scenarios, there will inevitably be situations where certain downstream task modalities are inaccessible during upstream pre-training. Therefore, as a general approach, it must be dynamically compatible with downstream tasks without knowing the modality types; more specifically, it must be usable across randomized, customized multimodal datasets. Additionally, it is necessary to consider how to control the computational overhead under the dual support of multimodality and large-scale models.

[0047] Multimodal semantic segmentation aims to extract and fuse broader segmentation features from data with different perceptual dimensions. Previous multimodal semantic segmentation methods mainly included early fusion, late fusion, and mid-fusion. Early fusion directly concatenates multimodal data of the same resolution along the channel paths and adjusts the dimension of the linear input layer of the network, thus transforming multimodal segmentation into a multi-channel form of single-modal segmentation. However, this method ignores the differences in variance, sparsity, and data distribution among multimodal data, resulting in homogeneous computation of all modalities in the first linear layer of the network, leading to poor performance. To consider the heterogeneity between modalities, some studies have developed late fusion methods based on extracting modal features using independent encoders and decoders and fusing the prediction results at the end. However, in this method, the rich features of multimodalities are gradually compressed or even lost during the long gradient flow. To address these issues, CMX, MCANet, NLFNet, and others have adopted mid-fusion methods that use independent encoders to extract multimodal features and fuse them based on a single decoder, which has become the current mainstream approach. Such methods typically only consider the case of simultaneous input of two modalities. When extrapolating to more modalities, the significant computational overhead of the encoder is often prohibitive for consumer users. To address this, MultiSenseSeg has developed a feasible method for extrapolation at a low cost.

[0048] While the aforementioned methods have significantly improved the performance margins of semantic segmentation, the agnostic nature of multimodal data often necessitates de novo training for specific scenarios, lacking the necessary large-scale upstream prior support. Therefore, this invention proposes a pre-trained large-scale visual model-based approach to investigate an efficient downstream fine-tuning method for modally agnostic remote sensing scenarios.

[0049] SAM is one of the representative large-scale visual foundational models. Pre-trained on massive datasets containing billions of masks and 11 million images, SAM integrates a wealth of visual knowledge. Its rich visual priors have driven a series of studies applying SAM to specific downstream tasks. SAM2 is a recently released upgrade to SAM, optimizing model accuracy and efficiency and adding video segmentation capabilities. When dealing with image segmentation, SAM2 exhibits the exact same behavior as SAM: given a set of images and cue inputs, the model outputs possible masks. Furthermore, SAM2 offers a lighter variant, further reducing the hardware costs required for deployment, making it possible to use large models in everyday tasks (although strictly speaking, a large model equals a large dataset with a large number of parameters, extremely large parameter sets are indeed difficult for consumer users to adopt).

[0050] However, due to the fixed paradigm of SAM and SAM2—that is, inputting a 3-band image and outputting an expected mask—most existing works can be summarized as paradigm-invariant cross-domain problems. In other words, the model input only differs in the data domain, but remains a 3-band single-modal image, making it difficult to leverage the multimodal advantages of diverse remote sensing data sources. Therefore, this invention proposes that additional modalities are no longer treated as inputs equal to the native modality (RGB image) of the large visual model, but rather as cues to the main modality, thus injecting multimodal remote sensing features into the visual base model.

[0051] Specifically, embodiments of the present invention provide a feasible method for extending a single-modal (or finite-modal) visual large model to modality-agnostic remote sensing scenarios. Embodiments of the present invention propose a remote sensing semantic segmentation framework coupled with a multimodal and visual large model, which not only possesses the necessary functions of single-modal remote sensing semantic segmentation but can also introduce more arbitrary remote sensing image modalities to further obtain performance gains.

[0052] Specifically, such as Figure 1 The diagram illustrates a flowchart of the remote sensing semantic segmentation method coupled with a multimodal and visual large model provided in an embodiment of the present invention. The method includes the following steps:

[0053] S101, acquire multimodal data, and input the RGB image in the multimodal data into the pre-trained image encoder in the remote sensing semantic segmentation model; the remote sensing semantic segmentation model includes: an image encoder, a multimodal cue, and a category decoder.

[0054] In this embodiment of the invention, the image encoder can be inherited from any large visual model.

[0055] This invention provides an efficient downstream fine-tuning method for modality-agnostic remote sensing scenarios based on a pre-trained large visual model.

[0056] Fine-tuning aims to optimize performance on a specific task by adjusting model weights, building upon prior knowledge learned during pre-training. Fine-tuning is divided into full fine-tuning and Parametrically Efficient Fine-Tune (PEFT). Considering the limitations of large models in terms of computational and storage resources, PEFT is typically used, where only a small subset of parameters is fine-tuned while keeping most parameters unchanged. Common PEFT methods include incremental fine-tuning, selective fine-tuning, and reparameterization-based fine-tuning. Incremental fine-tuning adds additional trainable parameters to better adapt the model to downstream tasks. Selective fine-tuning aims to selectively adjust a small subset of model parameters to balance computational efficiency, such as adjusting only the decoder or bias terms. Reparameterization-based fine-tuning methods utilize low-rank representations to reorganize model weights with minimal training parameters. The most well-known example is Low-Rank Adaptation (LoRA), which uses simple low-rank matrix factorization to perform reparameterized weight updates on the self-attention layers of the Transformer architecture.

[0057] The key to combining multimodal representations and large-scale visual models lies in how to inject multimodal data into them. Due to the agnostic nature of modalities, it is impractical to integrate multimodal representations using large-scale pre-training support, as the computational burden is prohibitive. Therefore, the technical solution proposed in this invention adopts the PEFT approach, requiring only simple fine-tuning on the desired dataset incorporating the intended multimodal data to adapt to downstream tasks. This approach has the following advantages: 1) minimized computational overhead; 2) avoidance of large-scale multimodal data collection; 3) the simplest operational flow, i.e., the entire process is a one-step process, requiring no additional proxy tasks such as self-supervision and knowledge distillation, and the large model base used can contain only single-modal information.

[0058] The remote sensing semantic segmentation model provided in this embodiment of the invention includes an image encoder, a multimodal cue, and a category decoder. For example... Figure 2The diagram illustrates the architecture of the remote sensing semantic segmentation model provided in this embodiment of the invention. The image encoder inherits from a large visual model (e.g., SAM2), carrying a large amount of prior knowledge and further optimizing its performance on specific tasks through fine-tuning, processing the native modality (RGB image). The multimodal cue integrates multimodal data (modality 1 to modality n) into a feature matrix and provides cue to the image encoder's output in a channel-weighted concatenation form, offering additional perceptual dimensions. The decoder acquires the cueed input and outputs the multi-class probability distribution results. To simplify user operation, the entire remote sensing semantic segmentation model is implemented end-to-end, and extrapolation to any modality can be achieved by changing only one hyperparameter of the framework. During training, a fine-tuning method is used to fine-tune the image encoder, train the multimodal cue, and train the category decoder. Figure 2 An exemplary method for fine-tuning an image encoder is shown in the example.

[0059] To achieve cross-domain processing from natural scenes to remote sensing, this embodiment of the invention introduces a variety of fine-tuning methods for image encoders in the remote sensing semantic segmentation model.

[0060] In this embodiment of the invention, the image encoder is trained using any one of the following five fine-tuning methods:

[0061] 1) Only fine-tune the decoder part, without updating any parameters of the image encoder.

[0062] 2) Fine-tune the bias terms of all linear layers only.

[0063] 3) Add parallel, trainable small adapters after the MLP layer of each Transformer architecture and adjust the embedding layer of the image encoder; for the , l The mathematical form of the adapter's operation, defined by a Transformer structure, can be expressed as follows:

[0064]

[0065] The first linear layer Input data (token) Dimensional reduction After being activated by the Gaussian Error Linear Unit (GELU), and then... Restore dimensions. Because... Therefore, the parameter increments for fine-tuning only account for a small portion. It is worth noting that the embodiments of the present invention... Kamin uniform initialization is used, while... Initialize all values ​​to 0. Initializing the adapter is crucial. During the initial training phase, it is essential to ensure that the weight increments of the image encoder are 0; otherwise, the consistency of the pre-trained weights in the chain rule will be compromised.

[0066] 4) Reparameterize the query, key, and value matrices of the self-attention layer using low-rank matrices;

[0067] The low-rank decomposition process can be represented as The weight initialization method is the same as the method described in section 3 above.

[0068] 5) Update all parameters of the image encoder.

[0069] For ease of understanding, from a macro perspective, any fine-tuning method can be considered as a system based on... The RGB image is used as input, and the output is... The higher-order nonlinear transformation function of the image embedding, i.e., Depending on the model size, SAM2 sets Tiny( T Small S ), Base+( B+ ) and Large ( L Four variant versions are available. In this embodiment of the invention, the image encoder can be any one of these versions.

[0070] S102, the image encoder of the remote sensing semantic segmentation model encodes the RGB image in the multimodal data to obtain the image embedding.

[0071] In this embodiment of the invention, the image encoder only encodes the native modality (RGB image) of the large visual model to obtain the image embedding. Instead of treating the additional modality as an input equal to the native modality (RGB image) of the large visual model, it uses it as a cue for the main modality and injects multimodal remote sensing features into the visual base model through a multimodal cue.

[0072] S103, the multimodal prompter integrates the multimodal data into a multimodal feature matrix, and provides prompts to the output result of the image encoder in the form of channel concatenation weighting, thereby obtaining the image embedding after prompting.

[0073] The visual large model SAM's cue encoder works by accepting two types of cues: dense cues (2D masks) and sparse cues (dots, boxes). Dense cues are then expanded into sparse cues by a smaller encoder. Element-wise summation is then performed along the same dimension to obtain the excited image embedding. ; sparse hints are then added after embedding into a vector. Cross-attention is performed to introduce dynamic interaction capabilities between the model and the user. It's easy to see that, among the two types of cues mentioned above, dense cues have a spatial correspondence with the image, which happens to be similar to the data format of remote sensing multimodal imagery. Inspired by this, embodiments of the present invention mimic the logic of dense cues to inject multimodal cues into large visual models. On the other hand, although sparse cues bring flexible interaction capabilities to the model, they do hinder automatic prediction in an end-to-end manner in practical applications. To achieve automation on the user end, embodiments of the present invention remove all sparse cues.

[0074] Compared to using unimodal dense cues, injecting multimodal representations presents the following challenges: 1) How to employ appropriate encoders for each modality given its heterogeneity (sparse points and dense spectra), specifically the characteristics of each modality. 2) How to control the linearly increasing computational overhead as the number of modalities grows, facilitating deployment on consumer-grade hardware. 3) How to search for complementary features across the entire modal space to provide the model with the most diverse perceptual dimension classes possible. To address these challenges, this invention introduces the concept of hybrid expert models, constructing both hard and soft hybrid expert models to maximize the driving force of multimodal representations. Specifically, as... Figure 3 The diagram illustrates a multimodal cueing mechanism for a remote sensing semantic segmentation model provided in this embodiment of the invention. The multimodal input first passes through a specific patch embedding layer. Next, a hard hybrid expert model activates a suitable encoder for feature learning based on the modal characteristics. Finally, a soft hybrid expert model aggregates complementary features based on the heterogeneity of the encoded multimodal feature stack. This process reuses features from the original RGB input to ensure synchronization of semantic granularity.

[0075] It is worth noting that, considering the uncertainty of the number of modalities, and to simplify the extrapolation of the multimodal cue, this embodiment of the invention uses only one hyperparameter to control this process in the remote sensing semantic segmentation model. The user only needs to provide a list containing information on the number of channels for each modality, and the remote sensing semantic segmentation model can automatically identify the number of input modalities and initialize the corresponding multimodal cue, without any additional operations. When cueing is not used, the remote sensing semantic segmentation model will degenerate into an end-to-end single-modal visual large language model fine-tuning framework and continue operating.

[0076] In this embodiment of the invention, the method further includes:

[0077] S11, receive a list provided by the user containing information on the number of channels for each modality, the remote sensing semantic segmentation model identifies the number of modalities in the input multimodal data, and initializes the corresponding multimodal cue.

[0078] In this embodiment of the invention, the multimodal cue includes a hard hybrid expert model and a soft hybrid expert model, and step S103 includes:

[0079] S1031, the hard hybrid expert model activates a suitable expert network to perform feature learning based on the data characteristics of each modality, and obtains the feature map of each modality.

[0080] S1032, perform connection and normalization operations on the feature maps of each modality to obtain multimodal feature maps.

[0081] S1033, the soft hybrid expert model obtains a multimodal cue mask based on the heterogeneous aggregation of complementary features of the multimodal feature map.

[0082] In this embodiment of the invention, the hard hybrid expert model aims to extract intramodal features based on modal characteristics while avoiding excessive computational complexity increments. The hard hybrid expert model consists of a gating network and several expert networks, where the gating network is used to adjust the sparsity of the model. This is a rigid constraint that routes between different expert nodes and selects the top k expert networks by score for activation.

[0083] In this embodiment of the invention, the expert network can be any end-to-end combination of operators. It has a high degree of freedom and can be designed in a targeted manner according to the modal categories involved in the specific downstream task. For example, it can be an MLP or a few simple convolutional layers, or even a complex Transformer or Mamba network.

[0084] In this embodiment of the invention, step S1031 includes:

[0085] The gated network performs global max pooling and average pooling in parallel along the spatial dimension on the feature maps of each input modality. The results of max pooling and average pooling are weighted and dimensionality transformed through a reshaping operation. For each modality, the expected probability assigned to each expert network is calculated using trainable gated weights, and the top scorers are grouped together. k One expert network is selected as the proposed activation expert network. The probability of the proposed activation expert network is retained, and the probabilities of other expert networks are filled with -∞. After a Softmax operation, the final activation is obtained. k Normalized weight scores of each expert network ; with the first i Taking one mode as an example, the above calculation process can be expressed as:

[0086]

[0087]

[0088]

[0089] in, Indicates the first i Feature maps of various modalities after feature block embedding layers. , , and These represent batch size, number of channels, height, and width, respectively. This represents the weight coefficient used to adjust the two-way pooling; it is set to 0.3 based on experiments. This is to control the computational cost of the model to the greatest extent possible.

[0090] The activated expert network is used as the feature extractor for the current modality to obtain the feature map of the current modality.

[0091] In this embodiment of the invention, it can be k Setting it to 1 means that only the expert network with the highest score is activated as the feature extractor for the current modality.

[0092] In this embodiment of the invention, the expert networks are designed in a heterogeneous form. In this embodiment, only two layers of convolutions with different specifications are used to construct an extremely lightweight expert network. The four types of experts constructed are: 1) 3×3 ordinary convolution, suitable for most modalities; 2) 3×3 dilated convolution, which has a larger receptive field while reducing sensitivity to neighborhoods to prevent overfitting; 3) MLP, consistent with the idea in the transformer block, treating the image as a token and extracting nonlinear relationships; 4) upsampling before convolution to improve the expressive power of coarse-grained modalities. Indicates the first i The convolutional specifications adopted by various experts can be simplified to the following: This balances computational efficiency and represents the performance gain of the technical solution provided in this embodiment of the invention under conservative estimation conditions. Correspondingly, the remote sensing semantic segmentation model provided in this embodiment of the invention also allows users to use a larger-capacity expert network as an alternative.

[0093] In this embodiment of the invention, step S1033 includes:

[0094] The soft hybrid expert model concatenates multimodal feature maps along the channel dimension, resulting in three feature representations: , , ,calculate and The cosine distance score between them is then assigned to the corresponding value. Adaptive aggregation based on modal heterogeneity:

[0095]

[0096] in, Indicates divided into n The group performs a linear projection operation, for and First M × M Perform max pooling on small blocks and then linear transformation; This indicates that the vector is flattened along the channel. ; This represents the relative positional bias between modalities; two Softmax operations are used to normalize the weights; the final result is reshaped into... Perform full-path fusion and use residual connections to obtain the final multimodal cue mask.

[0097] In this embodiment of the invention, the soft hybrid expert model aims to impose soft constraints on the multimodal feature stack to aggregate complementary features. Its core idea is to vectorize multimodal features and construct cosine distance relationships between modalities to represent modal heterogeneity, such as... Figure 3 The sub-parts are shown.

[0098] In this embodiment of the invention, all operations in formulas (5), (6), and (7) are implemented in groups according to modality categories. This means that the weights learned by the network in this part are still within the modality (which can be regarded as expert weights), and the features between modalities are not interactively fused. Next, the fusion of the full-channel paths is performed and residual connections are used to obtain the final multimodal cue mask.

[0099] S104, the category decoder obtains the image embedding after the prompt and outputs the probability distribution results of the multi-class classification.

[0100] In this embodiment of the invention, considering that the segmentation result output by the native mask decoder of the large visual model is category-independent (i.e., the generated segmentation mask does not contain semantic labels), it cannot be directly supervised fine-tuned. Furthermore, to perform end-to-end modification, this embodiment of the invention must remove the interactive sparse cues, which would cause the original built-in cross-attention relationship between cue tokens and image tokens to disappear. Therefore, this embodiment of the invention considers using a classic decoder from semantic segmentation to replace this part.

[0101] This invention employs FPN as the decoder, and to save computational efficiency, convolutional layers are not deployed at the highest resolution level; instead, bilinear interpolation is used to directly restore the resolution. The loss function used is a combination of cross-entropy loss with class weights and Dice loss, with the background weight set to 1 / 4 of the foreground class weight. Supervised fine-tuning can be completed by directly minimizing the semantic segmentation loss function, without the need for other additional types of losses.

[0102] Based on the same inventive concept, embodiments of the present invention also provide a remote sensing semantic segmentation device coupled with a multimodal and visual large model, the device comprising:

[0103] The acquisition module is used to acquire multimodal data and input the RGB image from the multimodal data into a pre-trained image encoder in the remote sensing semantic segmentation model; the remote sensing semantic segmentation model includes: an image encoder, a multimodal cue, and a category decoder;

[0104] The encoding module is used by the image encoder of the remote sensing semantic segmentation model to encode the RGB images in the multimodal data to obtain image embeddings;

[0105] The prompting module is used by the multimodal prompter to integrate the multimodal data into a multimodal feature matrix, and to provide prompts to the output result of the image encoder in the form of channel concatenation weighting, so as to obtain the image embedding after prompting;

[0106] The decoding module is used by the category decoder to obtain the image embedding after the prompt and output the probability distribution results of the multi-class classification.

[0107] Optionally, the image encoder inherits from a large visual model, and the image encoder is trained using any one of the following five fine-tuning methods:

[0108] Only fine-tune the decoder part; do not update any parameters of the image encoder.

[0109] Only fine-tune the bias terms of all linear layers;

[0110] Add parallel, trainable mini-adaptors after the MLP layer of each Transformer architecture and adjust the embedding layer of the image encoder.

[0111] Reparameterize the query, key, and value matrices of the self-attention layer using low-rank matrices;

[0112] Update all parameters of the image encoder.

[0113] Optionally, the device further includes:

[0114] The receiving module is used to receive a list provided by the user containing information on the number of channels for each modality. The remote sensing semantic segmentation model identifies the number of modalities in the input multimodal data and initializes the corresponding multimodal cue.

[0115] Optionally, the multimodal prompter includes a hard hybrid expert model and a soft hybrid expert model; the prompting module is used for:

[0116] The hard hybrid expert model activates a suitable expert network to learn features based on the data characteristics of each modality, thereby obtaining a feature map for each modality;

[0117] The feature maps of each modality are connected and normalized to obtain a multimodal feature map;

[0118] The soft hybrid expert model obtains a multimodal cue mask based on the heterogeneous aggregation of complementary features of the multimodal feature map.

[0119] Optionally, the hard hybrid expert model includes a gating network and several expert networks; the prompting module is used for:

[0120] The gated network performs global max pooling and average pooling in parallel in the spatial dimension on the feature maps of each input modality, and weights the results of max pooling and average pooling and performs dimensional transformation through a reshaping operation.

[0121] For each modality, the expected probability assigned to each expert network is calculated using trainable gating weights, and the top scorers are grouped together. k One expert network is selected as the proposed activation expert network. The probability of the proposed activation expert network is retained, and the probabilities of other expert networks are filled with -∞. After a Softmax operation, the final activation is obtained. k Normalized weight scores of each expert network The activated expert network is used as the feature extractor for the current modality to obtain the feature map of the current modality.

[0122] Optionally, the prompting module is used to:

[0123] The soft hybrid expert model concatenates multimodal feature maps along the channel dimension, resulting in three feature representations: , , ,calculate and The cosine distance score between them is then assigned to the corresponding value. Adaptive aggregation based on modal heterogeneity:

[0124]

[0125] in, Indicates divided into n The group performs a linear projection operation, for and First M × M Perform max pooling on small blocks and then linear transformation; This indicates that it is expanded into a vector along the channel dimension. ; This represents the relative positional bias between modalities; two Softmax operations are used to normalize the weights; the final result is reshaped into... ;

[0126] The final multimodal cue mask is obtained by performing full-path fusion and using residual connections.

[0127] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the remote sensing semantic segmentation method coupled with a multimodal and visual large model as described in any of the above embodiments.

[0128] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the remote sensing semantic segmentation method coupled with multimodal and visual large model as described in any of the above embodiments.

[0129] Based on the same inventive concept, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the remote sensing semantic segmentation method coupled with multimodal and visual large models as described in any of the above embodiments.

[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0131] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable terminal device to cause a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0135] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0136] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0137] The foregoing has provided a detailed description of a remote sensing semantic segmentation method coupled with a large visual model. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A remote sensing semantic segmentation method coupled with a multimodal and large visual model, characterized in that, The method includes: Acquire multimodal data, and input the RGB image from the multimodal data into a pre-trained image encoder in a remote sensing semantic segmentation model; the remote sensing semantic segmentation model includes: an image encoder, a multimodal cue, and a category decoder; The image encoder of the remote sensing semantic segmentation model encodes the RGB images in the multimodal data to obtain image embeddings; The multimodal prompter integrates the multimodal data into a multimodal feature matrix and provides prompts to the output of the image encoder in the form of channel concatenation weighting, thereby obtaining the prompted image embedding; The category decoder obtains the image embedding after the prompt and outputs the probability distribution results of multi-class classification; The multimodal cueing mechanism includes a hard hybrid expert model and a soft hybrid expert model; the multimodal cueing mechanism integrates the multimodal data into a multimodal feature matrix, and provides cueing to the output of the image encoder in the form of channel concatenation weighting, including: The hard hybrid expert model activates a suitable expert network to learn features based on the data characteristics of each modality, thereby obtaining a feature map for each modality; The feature maps of each modality are connected and normalized to obtain a multimodal feature map; The soft hybrid expert model obtains a multimodal cue mask based on the heterogeneous aggregation of complementary features of the multimodal feature map.

2. The remote sensing semantic segmentation method coupled with a multimodal and visual large model according to claim 1, characterized in that, The image encoder is inherited from the large visual model and is trained using any one of the following five fine-tuning methods: Only fine-tune the decoder part; do not update any parameters of the image encoder. Only fine-tune the bias terms of all linear layers; Add parallel, trainable mini-adaptors after the MLP layer of each Transformer architecture and adjust the embedding layer of the image encoder. Reparameterize the query, key, and value matrices of the self-attention layer using low-rank matrices; Update all parameters of the image encoder.

3. The remote sensing semantic segmentation method coupled with a multimodal and visual large model according to claim 1, characterized in that, The method further includes: The remote sensing semantic segmentation model receives a list provided by the user containing information on the number of channels for each modality. It then identifies the number of modalities in the input multimodal data and initializes the corresponding multimodal cue.

4. The remote sensing semantic segmentation method coupled with a multimodal and visual large model according to claim 1, characterized in that, The hard hybrid expert model includes a gating network and several expert networks; the hard hybrid expert model activates a suitable expert network based on the data characteristics of each modality to perform feature learning, obtaining a feature map for each modality, including: The gated network performs global max pooling and average pooling in parallel in the spatial dimension on the feature maps of each input modality, and weights the results of max pooling and average pooling and performs dimensional transformation through a reshaping operation. For each modality, the expected probability assigned to each expert network is calculated using trainable gating weights. The top k expert networks with the highest scores are selected as the proposed activation expert networks. The probabilities of the proposed activation expert networks are retained, and the probabilities of other expert networks are padded with -∞. After a Softmax operation, the normalized weight scores of the final k activated expert networks are obtained. The activated expert network is used as the feature extractor for the current modality to obtain the feature map of the current modality.

5. The remote sensing semantic segmentation method coupled with a multimodal and visual large model according to claim 1, characterized in that, The soft hybrid expert model aggregates complementary features based on the heterogeneity of multimodal feature maps, including: The soft hybrid expert model concatenates multimodal feature maps along the channel dimension, resulting in three feature representations: , , ,calculate and The cosine distance score between them is then assigned to the corresponding value. Adaptive aggregation based on modal heterogeneity: in, This indicates that the linear projection operation is performed in n groups. and First, perform max pooling on M × M blocks and then perform linear transformation; This indicates that it is expanded into a vector along the channel dimension. ; This represents the relative positional bias between modalities; two Softmax operations are used to normalize the weights; the final result is reshaped into... ; The final multimodal cue mask is obtained by performing full-path fusion and using residual connections.

6. A remote sensing semantic segmentation device coupled with a multimodal and visual large model, characterized in that, The device includes: The acquisition module is used to acquire multimodal data and input the RGB image from the multimodal data into a pre-trained image encoder in the remote sensing semantic segmentation model; the remote sensing semantic segmentation model includes: an image encoder, a multimodal cue, and a category decoder; The encoding module is used by the image encoder of the remote sensing semantic segmentation model to encode the RGB images in the multimodal data to obtain image embeddings; The prompting module is used by the multimodal prompter to integrate the multimodal data into a multimodal feature matrix, and to provide prompts to the output result of the image encoder in the form of channel concatenation weighting, so as to obtain the image embedding after prompting; The decoding module is used by the category decoder to obtain the image embedding after the prompt and output the probability distribution results of multi-classification. The multimodal prompter includes a hard hybrid expert model and a soft hybrid expert model; the prompting module is used for: The hard hybrid expert model activates a suitable expert network to learn features based on the data characteristics of each modality, thereby obtaining a feature map for each modality; The feature maps of each modality are connected and normalized to obtain a multimodal feature map; The soft hybrid expert model obtains a multimodal cue mask based on the heterogeneous aggregation of complementary features of the multimodal feature map.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the remote sensing semantic segmentation method that couples multimodal and visual large model as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the remote sensing semantic segmentation method that couples multimodal and visual large models as described in any one of claims 1-5.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps in the remote sensing semantic segmentation method coupled with a multimodal and visual large model as described in any one of claims 1-5.

Images