A multi-modal semantic segmentation method for open vocabulary and remote sensing image interaction

Abstract

The present application belongs to the technical field of remote sensing image processing, and particularly relates to a multi-modal semantic segmentation method for open vocabulary and remote sensing image interaction. The present application introduces a double-branch text encoder to extract scene-level semantics from various text data to realize dynamic cross-modal fusion. These text-derived features dynamically interact with visual embeddings through the proposed text-guided visual semantic fusion module to realize domain-aware feature refinement and human-interpretable decision-making. In addition, the text integration opens up the way for open vocabulary semantic segmentation, enabling the system to identify and classify invisible classes through natural language descriptions, thereby overcoming the strict constraints of the pre-defined class classification method.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a multimodal semantic segmentation method for interaction between open vocabulary and remote sensing images. Background Technology

[0002] Semantic segmentation is an advanced remote sensing technique that achieves pixel-level image segmentation by classifying each pixel in an image. It plays a crucial role in key Earth observation tasks such as environmental monitoring, urban planning, disaster response, and land use classification. Advances in sensor technology and the surge in multi-source remote sensing data, including optical, synthetic aperture radar (SAR), and lidar (LiDAR) data, have driven the development of Earth observation paradigms but also brought challenges to data interpretation. Deep learning has become a powerful tool for solving complex tasks over the past decade, but most research has focused on single-modal semantic segmentation, which has limited ability to distinguish spectrally similar categories. To address this issue, research has introduced multimodal methods, such as combining optical and SAR imagery to mitigate the effects of adverse weather and noise. However, existing methods primarily rely on visual modes, neglecting the potential of non-visual modes such as textual data. Textual data can provide real-world knowledge such as geographical context and material properties, enhancing model robustness and paving the way for open-vocabulary semantic segmentation, enabling systems to identify and classify invisible categories through natural language descriptions, overcoming the limitations of predefined categories.

[0003] In recent years, Visual Language Models (VLMs) have attracted attention due to their excellent open-vocabulary object recognition capabilities; however, their feature representations lack pixel-level granularity, making them difficult to directly apply to dense prediction tasks. To address this, research has proposed a series of methods, such as fast learning and adapters, to enhance the performance of VLMs in open-vocabulary semantic segmentation tasks. In land cover mapping, this approach eliminates the need for a fixed label set, allowing users to define desired categories through language and generate personalized maps that meet specific needs, providing convenience for non-remote sensing experts. Summary of the Invention

[0004] To address the problem that existing semantic segmentation methods based on multimodal remote sensing images struggle to overcome predefined category limitations, this invention proposes a multimodal semantic segmentation method for interaction between open vocabulary and remote sensing images. This method innovatively integrates fine-grained features from image and text patterns and achieves accurate semantic segmentation in open vocabulary scenarios through a multimodal feature interaction mechanism.

[0005] The technical solution adopted in this invention is as follows:

[0006] A multimodal semantic segmentation method for interaction between open vocabulary and remote sensing images, characterized by the following steps:

[0007] S1. Register the original optical images and SAR images, manually describe each group of images with text to obtain a multimodal visual language dataset, and preprocess it to obtain training set and validation set data;

[0008] S2. Perform data augmentation on the training and validation sets;

[0009] S3. Construct a text-supervised multimodal open-vocabulary semantic segmentation network, including a pseudo-Siamese feature extraction network module, a multimodal feature correction module, a multimodal feature fusion module, a text feature extraction module, a dual-branch image-text fusion module, and a decoder module;

[0010] The pseudo-Siamese feature extraction network module uses ViT as its backbone. It segments the input image into fixed-size image blocks and flattens them. These blocks are then concatenated with learnable location codes and category tokens used to carry classification information to form a sequence. The sequence is then input into a Transformer encoder composed of a multi-head self-attention mechanism and a feedforward network for global feature modeling. Global feature extraction is completed using the ViT backbone module. Finally, only the encoded output of the category tokens is linearly mapped to obtain feature description maps for the optical image and the SAR image, respectively.

[0011] The multimodal feature correction module takes the feature descriptors of the obtained optical image and SAR image as input. First, it inputs the feature descriptors to the channel attention module to generate two sets of channel attention weight maps, which are then multiplied with the corresponding input features to obtain the channel-corrected feature maps. Then, it introduces the spatial attention module, which takes the channel-corrected feature maps as input to generate two sets of spatial attention weight maps, which are then multiplied with the input features to obtain the spatially corrected feature descriptors.

[0012] The multimodal feature fusion module takes the spatially corrected optical image and SAR image feature descriptor map as input, performs multi-level fusion operation on multiple feature layers at different scales, realizes multi-level feature aggregation through cross-modal attention mechanism and dual-branch interaction strategy, and integrates multi-level fused features into multi-scale feature pyramid in the output aggregation stage, and dynamically assigns weights to features at each scale to obtain multimodal fused features output by cross-modal attention mechanism.

[0013] The text feature extraction module preprocesses the original text description data, uses a pre-trained language model as the backbone network for text feature extraction, inputs the preprocessed category text into the model encoder, and directly outputs the semantic feature vector of the category name; for long text descriptions, the sequence features output by the encoder are reduced in dimension and integrated to generate a fixed-length text feature representation.

[0014] The text feature extraction module preprocesses the two types of raw text description data, text description and category text, respectively. It uses a pre-trained language model as the backbone network for text feature extraction. The preprocessed category text is input into the model encoder, which directly outputs the semantic feature vector of the category name. For long text descriptions, the preprocessed text is input into the encoder, and the sequence features output by the encoder are reduced in dimensionality and integrated to generate a fixed-length text feature representation.

[0015] The dual-branch image-text fusion module comprises two parts: image-text object-level label feature fusion and image-text scene-level semantic feature fusion. It takes two types of text description features and multimodal fusion features as input. In the image-text object-level label feature fusion, the semantic feature vector of the category name and multimodal fusion features are taken as input. A template-based text prompt is used, with the template "a photo of a..." as the input. The placeholder [CLS] in "[CLS]" is replaced with K class names to construct text prompts, which are then encoded using the CLIP text encoder. After encoding, a context-aware strategy is introduced to refine the text features. The text features generated by the template prompts are directly used as query input to the Transformer decoder to obtain text features. Subsequently, by measuring the compatibility between the language-aligned multimodal fusion feature map and the text features, a pixel-text score map is calculated. This map quantitatively reflects the matching relationship between pixel-level visual patterns and text semantics. It can be used as a segmentation result to assist in calculating the segmentation loss, or it can be concatenated with the final feature map to introduce language prior knowledge. Finally, the score map is input into the neck module based on the Feature Pyramid Network (FPN) to output multimodal features. In the fusion of image and text scene-level semantic features, the text feature representation and multimodal features of the text description are used as input. A "align first, then fuse" strategy is adopted. First, the image-text contrast loss is applied to reduce the modal gap between image and text features. Then, the global text features are encoded and scaled through the image-text fusion module based on the cross-modal attention mechanism. The image features and text features are fused layer by layer using the cross-modal attention mechanism to output multi-scale fusion features.

[0016] The decoder module receives the processed multi-scale fusion features, first performs dimensionality reduction and adaptation of the low-level features through 1×1 convolution, then concatenates the high-level features after bilinear interpolation upsampling with the dimensionality-reduced low-level feature channels, refines the features through 3×3 convolution blocks, and then upsamples them to the input image size through bilinear interpolation to output a high-precision segmentation result.

[0017] The loss function is designed as follows:

[0018] ,

[0019] ,

[0020] ,

[0021] in, It is object-level label loss. It is the total number of classes. It is a class of model predictions probability , It is a model to class The original output; It is a scene-level semantic loss. and These are image and text embeddings, respectively. It is a similarity function. It's a temperature parameter. It is the number of negative samples; It is cross-entropy loss. It is a Dice loss;

[0022] The method for open-vocabulary semantic segmentation is to encode the input text category through a text encoder to obtain the target embedding, calculate the cosine similarity between the semantic features and fused features at each pixel position of the target embedding, generate a similarity map for each target text, and obtain the final segmentation mask by selecting the label corresponding to the maximum value of all similarity values ​​at each pixel position.

[0023] S4. Input the training set data into the constructed semantic segmentation network for training to obtain the learned text-supervised multimodal open-vocabulary semantic segmentation network model. Then, tune the parameters on the validation set until the optimal model and its corresponding parameters are obtained, thus obtaining the trained semantic segmentation network.

[0024] S5. Input the preprocessed optical and SAR images and their corresponding text descriptions into the trained semantic segmentation network to obtain the segmentation results.

[0025] Furthermore, the preprocessing described in S1 involves unifying the image resolution to 256×256 and normalizing the labels to the [0,255] range.

[0026] Furthermore, the data augmentation methods described in S2 include rotation, translation, cropping, flipping, scaling, adding noise, color jittering, random erasure, and Gaussian blur.

[0027] Furthermore, the input layer of the pseudo-Siamese feature extraction network module described in S3 adopts an H×W×C sample format, where H and W are the image height and width, and C is the number of image channels; the Transformer encoder, as the core component of the ViT backbone, consists of a multi-head self-attention mechanism layer and a feedforward network layer, and the input of each layer is the feature sequence output by the preceding layer of the Transformer encoder.

[0028] Furthermore, the preprocessing operations on the original text description data described in S3 include word segmentation, stop word removal, and stemming / lexical reconstruction; the pre-trained language model is a BERT or GPT pre-trained language model.

[0029] The beneficial effects of this invention are as follows:

[0030] (1) This invention develops a multimodal semantic segmentation network (TSMNet) for interaction between open vocabulary and remote sensing images. It innovatively integrates fine-grained features of optical images, SAR images and text, and achieves accurate semantic segmentation in open vocabulary scenarios through a multimodal feature interaction mechanism.

[0031] (2) The present invention designs a dual-branch image-text fusion module (DITF), which integrates image features with scene-level semantics and object-level label features of text by optimizing text embedding, effectively integrating heterogeneous graphic features, enhancing intra- and inter-pattern dependencies, and thus enriching semantic information.

[0032] (3) This invention does not rely on fixed category labels. By constructing a joint loss function based on image-text similarity and segmentation consistency, it supports accurate semantic segmentation in open vocabulary scenarios and significantly improves the segmentation generalization ability and robustness in complex scenarios. Attached Figure Description

[0033] Figure 1 This is a flowchart of a multimodal semantic segmentation method for interaction between open vocabulary and remote sensing images, provided in an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of the multimodal semantic segmentation network (TSMNet) structure provided in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of the structure of the Multimodal Feature Correction Module (MFRM) provided in an embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of the structure of the multimodal feature fusion module (MFFM) provided in an embodiment of the present invention.

[0037] Figure 5A schematic diagram of an object-level label feature fusion framework for images and text provided in an embodiment of the present invention.

[0038] Figure 6 This is a schematic diagram of a scene-level semantic feature fusion framework for images and text provided in an embodiment of the present invention. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] Example:

[0041] like Figure 1 As shown, this example includes the following steps:

[0042] S1. Register the original optical images and SAR images, manually add text descriptions to each image group to obtain a multimodal visual language dataset, and preprocess it to obtain training and validation set data; the specific execution method is as follows:

[0043] 1) Register the original optical image and the SAR image;

[0044] 2) Data labels and their corresponding text descriptions are created by a professional remote sensing team;

[0045] 3) Divide the original data into training, validation, and test sets in an 8:1:1 ratio; do the same for labels and text descriptions.

[0046] 4) Unify the image resolution to 256*256; and process the labels to normalize them to the [0,255] range.

[0047] S2. Perform data augmentation on the training and validation sets, including techniques such as rotation, translation, cropping, flipping, scaling, adding noise, color jittering, random erasure, and Gaussian blur.

[0048] S3. Construct a text-supervised multimodal open-vocabulary semantic segmentation network (e.g., Figure 2 (as shown), including:

[0049] 1) Pseudo-Siam Feature Extraction Network Module: The input layer uses an H×W×C sample format, where H and W are the image height and width, and C is the number of image channels. The pseudo-Siam feature extraction network, with ViT as its backbone module, performs feature processing. First, the input image is segmented into fixed-size image blocks and flattened. These blocks are then concatenated with learnable positional codes and category tokens specifically used to carry classification information to form a sequence. This sequence serves as input to a Transformer encoder composed of a multi-head self-attention mechanism and a feedforward network for global feature modeling. The Transformer encoder, as the core component of the ViT backbone module, consists of a multi-head self-attention mechanism layer and a feedforward network layer. The input to each layer is the feature sequence output from the preceding layer of the Transformer encoder. The pseudo-Siam feature extraction network utilizes the ViT backbone module to complete global feature extraction, ultimately performing a linear mapping only on the encoded output of the category tokens to obtain feature description maps for both the optical and SAR images.

[0050] 2) Multimodal feature correction module: such as Figure 3 As shown, the input layer uses dual-modal feature maps (optical and SAR) as input. First, it is input to the channel attention module to generate two sets of channel attention weight maps, which are then multiplied with the corresponding input features to obtain channel-corrected feature maps. The spatial attention module is then introduced, which takes the channel-corrected feature maps as input to generate two sets of spatial attention weight maps, which are then multiplied with the input features to obtain spatial-corrected feature maps. This module achieves fine-grained correction of multimodal features through a "channel-space" serial cascade method.

[0051] 3) Multimodal feature fusion module: such as Figure 4 As shown, a cross-modal attention mechanism is introduced to achieve deep semantic interaction. The input features are subjected to multi-level fusion operations on multiple feature layers at different scales. Multi-level feature aggregation is achieved through the cross-modal attention mechanism and the dual-branch interaction strategy. In the output aggregation stage, the multi-level fused features are integrated into a multi-scale feature pyramid, and weights are dynamically assigned to features at each scale to obtain the output features of the cross-modal attention mechanism.

[0052] 4) Text Feature Extraction Module: First, preprocessing is performed on the raw text data. The preprocessing process includes core steps such as word segmentation, stop word removal, stemming / lemma reconstruction, etc. For multilingual text scenarios, additional language detection and unified translation operations are introduced to ensure consistent feature extraction standards across different languages. Pre-trained language models such as BERT and GPT are selected as the backbone network for text feature extraction. The preprocessed categorized text is input into the model encoder, which directly outputs semantic feature vectors of the category names. For long text descriptions (e.g., "A river flows through a forest..."), strategies such as mean pooling and max pooling are used to reduce the dimensionality of the sequence features output by the encoder and integrate them to generate fixed-length text feature representations.

[0053] 5) Dual-branch image-text fusion module: This module includes image and text object-level label feature fusion (e.g., ... Figure 5 (as shown) and image and text scene-level semantic feature fusion (such as...) Figure 6 (As shown) Two parts.

[0054] In the fusion of image and text object-level label features, a template-based text prompt is employed. This involves replacing the placeholder "[CLS]" in the template "a photo of a [CLS]" with K class names to construct text prompts (e.g., road, tree, water, etc.), and then encoding them using the CLIP text encoder. The contextual features used in this invention consist of global image representation and local spatial features. Simultaneously, a context-aware method is employed to enhance text features, rather than relying solely on traditional predefined templates. Therefore, the input to the text encoder becomes:

[0055] ,

[0056] in It is a learnable textual context, and It is the first The name of the class is embedded.

[0057] Following the text encoder, a context-aware strategy is introduced to refine the text features—a post-model hint method. Specifically, text features are generated using template hints and directly used as query input to the Transformer decoder. This mechanism enables text features to effectively capture the most relevant visual information and achieve feature updates through subsequent connections. Template hints are used to generate text features, which are then directly used for queries in the Transformer decoder.

[0058] ,

[0059] in It is a text feature. It is a language-compatible multimodal fusion image feature.

[0060] This method encourages text features to find the most relevant visual cues, and then update the text features using the remaining links:

[0061] ,

[0062] in, It is a learnable parameter used to control the scaling of the residuals, and it is initialized to a very small value (e.g., 10). -4 This is to preserve, to the greatest extent possible, the prior linguistic knowledge derived from text features.

[0063] Subsequently, the pixel-text score mapping is calculated by measuring the compatibility between the language-aligned feature map z and the text feature t.

[0064] ,

[0065] in and yes and Along the channel dimensions Normalized version.

[0066] This mapping quantitatively reflects the matching relationship between pixel-level visual patterns and text semantics. On the one hand, this mapping can be directly used as a segmentation result and to assist in the calculation of segmentation loss; on the other hand, it is concatenated with the final feature map to incorporate prior linguistic knowledge. Finally, it is fed into the neck module of a Feature Pyramid Network (FPN) to fuse and enhance features at different levels, thereby better supporting object detection and segmentation tasks.

[0067] In the fusion of image and text scene-level semantic features, a "align-then-fuse" strategy is adopted to handle heterogeneous image and text features. First, image-text contrast loss is applied to reduce the modal gap between image and text features. This operation generates an initial relationship between the two features, promoting image-text feature fusion in subsequent processing. Then, addressing the issue that text features such as climate information and geographic object features can serve as global prior information for cross-modal feature fusion, an image-text fusion module based on a cross-modal attention mechanism is proposed. This module first encodes global text features, dynamically generates multimodal text features, and scales the text features to match them with image features at different levels. Then, the cross-modal attention mechanism is used to fuse image and text features layer by layer, capturing their semantic relationships. Specifically, for each level of image features, the module interacts with text features at the corresponding scale through the cross-modal attention mechanism to generate a fused feature representation. Finally, the module outputs multi-scale fused features, providing rich cross-modal information for downstream tasks. Experiments show that this module can significantly improve the performance of joint image and text representation, providing an effective feature fusion solution for multimodal tasks.

[0068] 6) Decoder module: Receives the processed high-level semantic features and low-level features. First, it uses 1×1 convolution to reduce the dimensionality of the low-level features for adaptation. Then, it concatenates the high-level features after bilinear interpolation upsampling with the low-level feature channels after dimensionality reduction. After refining the features through 3×3 convolution blocks, it uses bilinear interpolation upsampling to the input image size and outputs a high-precision segmentation result.

[0069] 7) Establishing the model's loss function: During the training of the multimodal semantic segmentation network (TSMNet) for open-vocabulary and remote sensing image interaction, three main losses were designed and calculated: object-level label loss, scene-level semantic loss, and semantic segmentation task loss. The object-level label loss guides the optimization of image and label text information, the scene-level semantic loss guides the optimization of image and text description information, and the semantic segmentation task loss ensures that the segmentation mask generated by the model is consistent with the target region. Specifically, we use cross-entropy loss for the object-level label loss, as follows:

[0070] ,

[0071] ,

[0072] in It is the total number of classes. It is a class of model predictions The probability, It is a model to class The original output.

[0073] Scene-level semantic loss typically employs InfoNCE (Noise Contrast Estimation) loss, which is widely used in contrastive learning frameworks. The mathematical formula for this loss function can be expressed as follows:

[0074] ,

[0075] in and These are image and text embeddings, respectively. It is a similarity function (such as cosine similarity), and the temperature parameter is... , It is the number of negative samples.

[0076] At the same time, cross-entropy loss is used ( ) and Dice loss ( Let ) represent the loss for the semantic segmentation task. The total loss of TSMNet is the sum of the three losses, which can be expressed as:

[0077] ;

[0078] 8) Achieve open-vocabulary semantic segmentation: While generating the segmentation map, a similarity mapping between the categorized text and the image can be generated. First, the input text category is encoded by a text encoder to obtain the target embedding. Then, the cosine similarity between the semantic features and fused features at each pixel position of the target embedding is calculated to generate the similarity mapping for each target text. Finally, by selecting the label corresponding to the maximum value of all similarity values ​​at each pixel position, the final segmentation mask is obtained, thus achieving open-vocabulary semantic segmentation.

[0079] S4. Input the training set data into the constructed semantic segmentation network for training to obtain the learned text-supervised multimodal open-vocabulary semantic segmentation network model. Then, tune the parameters on the validation set until the optimal model and its corresponding parameters are obtained, thus obtaining the trained semantic segmentation network.

[0080] S5. Input the preprocessed optical and SAR images and their corresponding text descriptions into the trained semantic segmentation network to obtain the segmentation results.

[0081] Thus, this invention achieves multimodal semantic segmentation for interaction between open vocabulary and remote sensing images.

Claims

1. A multimodal semantic segmentation method for interaction between open vocabulary and remote sensing images, characterized in that, Includes the following steps: S1. Register the original optical images and SAR images, manually describe each group of images with text to obtain a multimodal visual language dataset, and preprocess it to obtain training set and validation set data; S2. Perform data augmentation on the training and validation sets; S3. Construct a text-supervised multimodal open-vocabulary semantic segmentation network, including a pseudo-Siamese feature extraction network module, a multimodal feature correction module, a multimodal feature fusion module, a text feature extraction module, a dual-branch image-text fusion module, and a decoder module; The pseudo-Siamese feature extraction network module uses ViT as its backbone to segment the input image into fixed-size image blocks and flatten them. These blocks are then concatenated with learnable positional codes and category tokens used to carry classification information to form a sequence. The sequence is then input into a Transformer encoder composed of a multi-head self-attention mechanism and a feedforward network for global feature modeling. Global feature extraction is performed using the ViT backbone module. Finally, a linear mapping is performed only on the encoded output of the category token to obtain the feature description maps of the optical image and the SAR image respectively. The multimodal feature correction module takes the feature description maps of the obtained optical image and SAR image as input. First, the feature description maps are input to the channel attention module to generate two sets of channel attention weight maps, which are then multiplied with the corresponding input features to obtain the channel correction feature maps. Then, a spatial attention module is introduced, which takes the channel-corrected feature map as input and generates two sets of spatial attention weight maps. These weight maps are then multiplied by the input features to obtain the spatially corrected feature description map. The multimodal feature fusion module takes the spatially corrected optical image and SAR image feature descriptor map as input, performs multi-level fusion operation on multiple feature layers at different scales, realizes multi-level feature aggregation through cross-modal attention mechanism and dual-branch interaction strategy, and integrates multi-level fused features into multi-scale feature pyramid in the output aggregation stage, and dynamically assigns weights to features at each scale to obtain multimodal fused features output by cross-modal attention mechanism. The text feature extraction module preprocesses the original text description data, uses a pre-trained language model as the backbone network for text feature extraction, inputs the preprocessed category text into the model encoder, and directly outputs the semantic feature vector of the category name; for long text descriptions, the sequence features output by the encoder are reduced in dimension and integrated to generate a fixed-length text feature representation. The text feature extraction module preprocesses the two types of raw text description data, text description and category text, respectively. It uses a pre-trained language model as the backbone network for text feature extraction. The preprocessed category text is input into the model encoder, which directly outputs the semantic feature vector of the category name. For long text descriptions, the preprocessed text is input into the encoder, and the sequence features output by the encoder are reduced in dimensionality and integrated to generate a fixed-length text feature representation. The dual-branch image-text fusion module includes two parts: image and text object-level label feature fusion and image and text scene-level semantic feature fusion, taking two types of text description features and multimodal fusion features as input; In the fusion of image and text object-level label features, the semantic feature vector of the category name and the multimodal fusion feature are used as input. A text prompt based on template is used. The text prompt is constructed by replacing the [CLS] placeholder in the template "a photo of a [CLS]" with K class names. Then, the CLIP text encoder is used for encoding. After encoding, a context-aware strategy is introduced to refine the text features. The text features generated by the template prompt are directly used as query input to the Transformer decoder to obtain the text features. Subsequently, by measuring the compatibility between the language-aligned multimodal fusion feature map and the text features, a pixel-text score map is calculated. This map quantitatively reflects the matching relationship between pixel-level visual patterns and text semantics. It can be used as a segmentation result to assist in calculating the segmentation loss, or it can be concatenated with the final feature map to introduce prior language knowledge. Finally, the score map is input into the neck module based on the Feature Pyramid Network (FPN) to output multimodal features. In the fusion of image and text scene-level semantic features, the text feature representation and multimodal features of the text description are taken as input. The "align first, then fuse" strategy is adopted. First, the image-text contrast loss is applied to reduce the modal gap between image and text features. Then, the global text features are encoded and scaled by the image-text fusion module based on the cross-modal attention mechanism. The image features and text features are fused layer by layer using the cross-modal attention mechanism to output multi-scale fusion features. The decoder module receives the processed multi-scale fusion features, first performs dimensionality reduction and adaptation of the low-level features through 1×1 convolution, then concatenates the high-level features after bilinear interpolation upsampling with the dimensionality-reduced low-level feature channels, refines the features through 3×3 convolution blocks, and then upsamples them to the input image size through bilinear interpolation to output a high-precision segmentation result. The loss function is designed as follows: ， ， ， in, It is object-level label loss. It is the total number of classes. It is a class of model predictions probability , It is a model to class The original output; It is a scene-level semantic loss. and These are image and text embeddings, respectively. It is a similarity function. It's a temperature parameter. It is the number of negative samples; It is cross-entropy loss. It is a Dice loss; The method for open-vocabulary semantic segmentation is to encode the input text category through a text encoder to obtain the target embedding, calculate the cosine similarity between the semantic features and fused features at each pixel position of the target embedding, generate a similarity map for each target text, and obtain the final segmentation mask by selecting the label corresponding to the maximum value of all similarity values ​​at each pixel position. S4. Input the training set data into the constructed semantic segmentation network for training to obtain the learned text-supervised multimodal open-vocabulary semantic segmentation network model. Then, tune the parameters on the validation set until the optimal model and its corresponding parameters are obtained, thus obtaining the trained semantic segmentation network. S5. Input the preprocessed optical and SAR images and their corresponding text descriptions into the trained semantic segmentation network to obtain the segmentation results.

2. The multimodal semantic segmentation method for interaction between open vocabulary and remote sensing images according to claim 1, characterized in that, The preprocessing described in S1 involves unifying the image resolution to 256×256 and normalizing the labels to the [0,255] range.

3. The multimodal semantic segmentation method for interaction between open vocabulary and remote sensing images according to claim 1, characterized in that, The data augmentation methods described in S2 include rotation, translation, cropping, flipping, scaling, adding noise, color jittering, random erasure, and Gaussian blur.

4. The multimodal semantic segmentation method for interaction between open vocabulary and remote sensing images according to claim 1, characterized in that, The input layer of the pseudo-Siamese feature extraction network module described in S3 adopts the sample format of H×W×C, where H and W are the image height and width, and C is the number of image channels. The Transformer encoder, as the core component of the ViT backbone, consists of a multi-head self-attention mechanism layer and a feedforward network layer. The input of each layer is the feature sequence output by the preceding layer of the Transformer encoder.

5. The multimodal semantic segmentation method for interaction between open vocabulary and remote sensing images according to claim 1, characterized in that, The preprocessing operations for the original text description data described in S3 include word segmentation, stop word removal, and stemming / lexical restoration; the pre-trained language model is a BERT or GPT pre-trained language model.

Images