Remote sensing image directional segmentation method based on bidirectional feature enhancement of image and text

Abstract

The application provides a remote sensing image directionality segmentation method based on bidirectional feature enhancement of image and text, and the method comprises the following steps: step one, image features and text features are extracted respectively through a backbone network, and the text features are enhanced by using the image features; step two, based on the original image features and the text features obtained in step one, the image features are supplemented and enhanced in attention by using the text features; step three, the enhanced text features and the enhanced image features obtained in steps one and two are interactively decoded, and finally, a mask result of remote sensing image directionality segmentation is obtained. The application provides a bidirectional feature enhancement method of image and text, fully excavates the similarity and complementarity between cross-modal features, realizes accurate alignment of the features, and significantly improves the accuracy of directionality segmentation.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a remote sensing image directional segmentation method with bidirectional feature enhancement of image and text. Background Technology

[0002] In recent years, large-scale language models have made significant progress in language understanding, analysis, and generation, particularly demonstrating powerful capabilities in contextual reasoning and multi-task processing. These breakthroughs have driven the development of visual language models, which combine visual information with linguistic expression and have been widely applied in tasks such as image generation and description, image text retrieval, and visual question answering. Especially in the field of remote sensing, the fusion of natural language and remote sensing images has become a current research focus, driving the transformation from traditional image processing to intelligent remote sensing analysis.

[0003] Directional segmentation of remote sensing images is an emerging technology that aims to generate pixel-level masks of specific targets in remote sensing images using natural language descriptions. The key to this task is to achieve precise localization and segmentation of specific regions or objects by matching the language descriptions with the image content. Compared with traditional remote sensing image processing methods, directional segmentation not only improves user interactivity and convenience but also overcomes the technical barriers of semantic understanding, enabling non-professional users to quickly retrieve and analyze content in remote sensing images using simple language descriptions.

[0004] Remote sensing image pointing segmentation technology has shown broad application prospects in multiple fields. Firstly, in land use monitoring, it can automatically generate segmentation results for specific areas, helping decision-makers assess land changes in real time. In search and rescue operations, remote sensing image pointing segmentation can quickly locate targets by describing specific environments or objects, improving operational efficiency. In environmental monitoring, it can detect pollution sources or environmental changes through verbal descriptions. Furthermore, fields such as military intelligence, agricultural assessment, and urban planning can also benefit from this technology, achieving more accurate remote sensing data analysis.

[0005] However, directional segmentation of remote sensing images still faces challenges. Remote sensing images typically have high resolution and complex scenes, and efficiently extracting targets that match language from large amounts of data remains a technical difficulty. Summary of the Invention

[0006] The purpose of this invention is to address the problem of low accuracy in existing remote sensing image directional segmentation models, as well as the problems of insufficient utilization of image text information and inadequate interaction of different modal information in existing remote sensing image directional segmentation models. Therefore, this invention proposes a remote sensing image directional segmentation method with bidirectional feature enhancement of image and text.

[0007] This invention is achieved through the following technical solution: This invention proposes a remote sensing image directional segmentation method with bidirectional feature enhancement of image and text, the method comprising the following steps:

[0008] Step 1: Extract image features and text features separately through the backbone network, and use the image features to enhance the text features;

[0009] Step 2: Based on the original image features and text features obtained in Step 1, use the text features to perform attention-based enhancement on the image features;

[0010] Step 3: Based on the enhanced text features and enhanced image features obtained in Step 1 and Step 2, perform interactive decoding to finally obtain the mask result of directional segmentation of the remote sensing image.

[0011] Furthermore, step one specifically includes:

[0012] Step 11: Obtain the remote sensing image V and its corresponding text description L;

[0013] Steps 1 and 2: Process the remote sensing images using a visual backbone network to extract their multi-scale visual features. The visual backbone network extracts features at different scales and semantic levels layer by layer through convolution or self-attention mechanisms; meanwhile, text features are extracted through the text backbone network; these networks can extract relevant contextual information from the input natural language text and generate text features with semantic representation.

[0014] Step 13: Dimensionality reduction of multi-scale visual features is performed using an adaptive pooling layer. The adaptive pooling layer dynamically adjusts the pooling parameters according to the size of the input image features to generate a feature map of a fixed size. By compressing image features of different scales into the same feature space, the size mismatch problem in subsequent operations can be avoided. Subsequently, the dimensionally reduced image features are fused through a stitching operation to form a compact feature vector containing global information.

[0015]

[0016] Step 1, Section 4: Perform interactive attention calculations between the dimensionality-reduced image features and a trainable random cue layer;

[0017]

[0018] Step 15: Add the prompt layer P containing image information v With the extracted text features L v The features are then concatenated to generate enhanced text features.

[0019] Furthermore, step two specifically involves:

[0020] Step 2: Flatten the multi-scale image features obtained in Step 1. The flattened image features contain information in both spatial and channel dimensions, which can capture the spatial structure of the image at different scales.

[0021] V iq =Flatten(ω) iq (V i ))

[0022] Step 22: Combine the flattened image features with the text features L i The interaction attention mechanism is used to fuse the data and calculate the interaction attention scores at different scales. The interaction attention mechanism identifies which image regions are most relevant to the text description by calculating the attention matrix between image and text features.

[0023]

[0024] Step 23: Normalize the interaction attention score calculated in Step 22; use the Softmax function to normalize the attention score, and multiply the normalized attention score by the flattened image features to emphasize the text-related parts of the image.

[0025]

[0026] Step 2.4: Analyze the image features V from Step 1. i The weighting is then performed; by reintegrating the normalized attention scores obtained in steps two and three with the image features, a cross-modal feature map is obtained.

[0027] V li =Conv 1×1 ((Att i )☉V i

[0028] Step 25: Resample the interaction attention scores at different scales, unify the size, and calculate the cross-scale attention difference map. The cross-scale attention difference map can show the correspondence between images and text at different scales. By selecting the top K regions with the greatest differences, the model can be further guided to focus on the image regions that are most relevant to the text description, thereby improving the accuracy of feature extraction.

[0029]

[0030] Step 26: Project and stitch the first K regions selected in Step 25; by projecting these regions onto different scales, a unified attention deficit map can be obtained; the attention deficit map integrates attention information from multiple scales, and can reflect global feature differences while maintaining local details;

[0031]

[0032] Step 27: Compensate for attention deficits in image features at different scales; by combining text-enhanced multi-scale image features with original image features, the expressive power of image features is further improved, ensuring that the final feature map can reflect the different focuses at different scales in the image, thereby achieving the unification of multi-scale feature attention.

[0033]

[0034] Furthermore, step three specifically includes:

[0035] Step 3: First, flatten the multi-scale image features enhanced by text in Step 2. The flattened visual features retain the global information and spatial layout of the image, making the subsequent fusion steps smoother. After flattening the image features, align the text features with the image features.

[0036]

[0037] Step 3.2: Introduce a multi-layered cross-modal feature interaction mechanism; this cross-modal feature interaction mechanism relies on the following three core modules: cross-attention mechanism, self-attention mechanism, and feedforward neural network;

[0038] Step 3: After the text and visual features have undergone multiple layers of interaction through cross-attention, self-attention, and feedforward neural networks, the next step is to align the two types of features pixel by pixel. This alignment employs a multi-scale deformable attention mechanism. Multi-scale deformable attention ensures that the model can effectively capture image details related to text features at all levels by dynamically sampling visual features at different scales. Through this process, visual features and text features achieve fine alignment at the pixel level. The mathematical expression of the above process is as follows:

[0039]

[0040] Steps 3 and 4: After completing the two-way interaction between visual and textual features, we proceed to the final feature fusion and mask generation stage.

[0041] Furthermore, the cross-attention mechanism is used to establish a global association between visual features and text features in each layer of interaction; by calculating the attention weight matrix of visual features and text features, the model can automatically capture which visual regions are most relevant to the text description; the cross-attention mechanism can ensure that the key information contained in the text features can correctly guide the model to focus on the target region in the image.

[0042] Furthermore, the self-attention mechanism strengthens the contextual relationships between text features by calculating attention weights within text features.

[0043] Furthermore, after cross-attention and self-attention processing at each layer, the feedforward neural network performs a non-linear transformation on the fused features, with residual connections and layer normalization introduced into each layer operation. Through this introduction, textual and visual features are progressively enhanced through multiple interactive layers into features that better express multimodal semantics. The mathematical expression of the above steps is as follows:

[0044]

[0045] Furthermore, steps three and four specifically involve:

[0046] First, the visual features enhanced by multi-scale attention and the original visual features are mapped to the same feature space through a 1×1 convolutional layer; the 1×1 convolution is used to compress and map the feature dimensions to ensure the consistency of feature dimensions.

[0047] Then, spatial resampling is used to adjust the mapped visual features to a size that matches the original image, ensuring that the visual features are consistent with the spatial resolution of the image. In this process, text embedding features are also multiplied element-wise with visual features to enhance the guiding role of text features on visual features and ensure that semantic information in the text can be accurately located to the corresponding region of the image.

[0048] Generate a predicted segmentation mask: A preliminary segmentation mask is generated from the output of feature fusion; the mask marks the image regions related to the text description, indicating the boundaries or shapes of the regions;

[0049] Mask upsampling: The predicted mask is upsampled to the same spatial resolution as the original input image using bilinear interpolation;

[0050] Ultimately, the generated masks can represent the position and outline of the target object in the image. These masks can be directly used for subsequent target detection, region recognition, or scene analysis to ensure that the target in the remote sensing image can be accurately identified and processed.

[0051]

[0052] The final result of directional segmentation of the remote sensing image was obtained, which is the mask of the target.

[0053] The beneficial effects of this invention are:

[0054] This invention addresses the problem of insufficient utilization of image and text features in existing directional segmentation of remote sensing images. Previous directional segmentation models typically focus on utilizing only a single modality of text or image features, neglecting the full utilization of features from the other modality. This results in limited improvement in segmentation results compared to using only single-modality data, failing to fully demonstrate the advantages of directional segmentation. This invention significantly improves the accuracy of directional segmentation by fully combining features from both image and text modalities and utilizing cross-modal information for encoding and decoding.

[0055] This invention also addresses the problem of insufficient interaction between image and text features in existing remote sensing image directional segmentation: Previous remote sensing image directional segmentation models have relatively simple cross-modal feature interaction processes, typically employing unidirectional feature interaction, making it difficult to fully utilize the similarity and complementarity between cross-modal features. This deficiency leads to difficulties in effectively aligning image and text modal features, thus limiting the improvement of directional segmentation performance. This invention proposes a bidirectional feature enhancement method for images and text, fully exploring the similarity and complementarity between cross-modal features to achieve precise feature alignment and significantly improve the accuracy of directional segmentation. Attached Figure Description

[0056] Figure 1 This is a flowchart of the remote sensing image directional segmentation method with bidirectional feature enhancement of images and text as described in this invention;

[0057] Figure 2 A schematic diagram of the supplementary module for image-guided text prompts;

[0058] Figure 3 Schematic diagram of the image attention supplementation module for text-guided learning;

[0059] Figure 4 This is a schematic diagram of an image-text interaction decoder. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Specific Implementation Method 1: The specific process of the remote sensing image directional segmentation method with bidirectional text feature enhancement proposed in this implementation method is as follows:

[0062] Step 1: Extract image features and text features separately through the backbone network, and use the image features to enhance the text features;

[0063] Step 2: Based on the original image features and text features obtained in Step 1, use the text features to perform attention-based enhancement on the image features;

[0064] Step 3: Based on the enhanced text features and enhanced image features obtained in Step 1 and Step 2, perform interactive decoding to finally obtain the mask result of directional segmentation of the remote sensing image.

[0065] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that, in step one, image features and text features are extracted separately through a backbone network, and the image features are used to enhance the text features.

[0066] The specific process is as follows:

[0067] Step 11: First, obtain the remote sensing image V and its corresponding text description L. The remote sensing image can come from various data sources such as satellites and drones, while the text description is usually manually annotated or automatically generated to describe the content or geographical environment in the image;

[0068] Steps 1 and 2: Next, use an appropriate visual backbone network (such as ResNet, ViT, Swin, ConvNext, etc.) to process the remote sensing images and extract their multi-scale visual features. Visual backbone networks extract features at different scales and semantic levels layer by layer through convolution or self-attention mechanisms. Meanwhile, text features are extracted using text backbone networks (such as LSTM, CLIP, BERT, etc.). These network models can extract relevant contextual information from the input natural language text to generate text features with semantic representations.

[0069] Step 13: To align image features with text features and thus enhance text features, multi-scale visual features are reduced in dimensionality using an adaptive pooling layer. The adaptive pooling layer dynamically adjusts the pooling parameters based on the size of the input image features to generate a fixed-size feature map. By compressing image features of different scales into the same feature space, size mismatch issues in subsequent operations can be avoided. Subsequently, the dimensionality-reduced image features are fused through a concatenation operation to form a compact feature vector containing global information. This step not only preserves the multi-scale information of the image but also reduces computational complexity.

[0070]

[0071] Step 1, Part 4: Next, the dimensionality-reduced image features are interactively interacted with a trainable PromptLayer. The PromptLayer is a learnable embedding layer designed to capture key information from the image features. Through a cross-attention mechanism, the PromptLayer extracts crucial contextual information from the image features. This mechanism selectively focuses on salient regions in the image by calculating attention weights between the PromptLayer and the image features, thereby improving the efficiency and effectiveness of feature extraction.

[0072]

[0073] Step 15: Add the prompt layer P containing image information v With the extracted text features L v The images are concatenated to generate enhanced text features. This concatenation operation not only improves the semantic expressiveness of the text but also integrates visual information from the image into the text features, thus achieving deep fusion between the image and the text. These enhanced text features can more accurately guide image decoding and segmentation in subsequent processing steps.

[0074] The other steps and parameters are the same as in Specific Implementation Method 1.

[0075] Specific Implementation Method 3: This implementation method differs from Specific Implementation Method 1 or 2 in that, in step 2, based on the original image features and text features obtained in step 1, attention enhancement is performed on the image features using text features.

[0076] The specific process is as follows:

[0077] Step Two: First, flatten the multi-scale image features obtained in Step One. This process converts three-dimensional or four-dimensional image features into two-dimensional vector representations, facilitating alignment and interaction with text features. The flattened image features contain information in both spatial and channel dimensions, allowing the capture of the image's spatial structure at different scales.

[0078] V iq =Flatten(ω) iq (V i ))

[0079] Step 22: Combine the flattened image features with the text features L iThe model employs a cross-attention mechanism to fuse information and calculate cross-attention scores at different scales. This mechanism identifies which image regions are most relevant to the text description by calculating the attention matrix between image and text features. Through this mechanism, the model can guide semantic information from the text to the corresponding image regions, achieving precise modality alignment.

[0080]

[0081] Step 23: Normalize the interactive attention scores calculated in Step 22. First, normalize the attention scores using the Softmax function to ensure that scores at different scales can be effectively compared. The normalized attention scores are then multiplied by the flattened image features to emphasize text-related parts of the image. To further improve feature discriminability, a threshold operation can be used to suppress attention scores below a certain threshold, thus focusing attention on important feature regions.

[0082]

[0083] Step 2.4: Analyze the image features V from Step 1. i Then, reweighting is performed. By reintegrating the normalized attention scores obtained in steps two and three with the image features, a cross-modal feature map can be obtained. This cross-modal feature map integrates multimodal information from text and images, enabling it to better represent key information in images.

[0084] V li =Conv 1×1 (Att i )☉V i

[0085] Step 25: Next, the interaction attention scores at different scales are resampled, and after standardizing the size, a cross-scale attention difference map is calculated. The cross-scale attention difference map can show the correspondence between images and text at different scales. By selecting the top K regions with the greatest differences, the model can be further guided to focus on image regions most relevant to the text description, thereby improving the accuracy of feature extraction.

[0086]

[0087] Step 26: Project and stitch the first K regions selected in Step 25. By projecting these regions onto different scales, a unified attention deficit map can be obtained. This map integrates attention information from multiple scales, reflecting global feature differences while preserving local details.

[0088]

[0089] Step 27: Finally, perform attention deficit compensation on image features at different scales. By combining text-enhanced multi-scale image features with the original image features, the expressive power of image features can be further improved, ensuring that the final feature map can reflect the different focuses at different scales in the image, thereby achieving the unification of multi-scale feature attention.

[0090]

[0091] The other steps and parameters are the same as those in one of the specific implementation methods one or two.

[0092] Specific Implementation Method Four: This implementation method differs from one of the specific implementation methods one to three in that, in step three, interactive decoding is performed based on the enhanced text features and enhanced image features obtained in steps one and two, and finally the mask result of directional segmentation of remote sensing image is obtained.

[0093] The specific process is as follows:

[0094] Step 3: First, flatten the multi-scale image features enhanced by text in Step 2. The purpose of flattening is to convert the original three-dimensional image features (i.e., feature maps containing spatial and channel dimensions) into two-dimensional feature vectors, facilitating cross-modal fusion and alignment with text features. This operation ensures that visual features at different scales can be processed in a unified dimension, avoiding information loss due to scale differences. The flattened visual features retain the global information and spatial layout of the image, making the subsequent fusion steps smoother.

[0095]

[0096] After image features are flattened, text features are aligned with image features. This process ensures consistency in their representation in feature space by modeling the relationship between text and image features. This not only improves the correlation between text and visual information but also provides more accurate feature matching for subsequent mask generation.

[0097] Step 3.2: To effectively capture and fuse the relationship between textual and visual features, this step introduces a multi-layered cross-modal feature interaction mechanism. This mechanism relies on three core modules: Cross-Attention Mechanism, Self-Attention Mechanism, and Feed-Forward Network (FFN).

[0098] Cross-attention mechanism: First, the cross-attention mechanism establishes a global association between visual and textual features at each layer of interaction. By calculating the attention weight matrix of visual and textual features, the model can automatically capture which visual regions are most relevant to the text description. The cross-attention mechanism ensures that the key information contained in the textual features correctly guides the model to focus on the target region in the image. This step is particularly important for the identification and segmentation of fine-grained targets (such as vehicles, buildings, etc.) in remote sensing images.

[0099] Self-attention mechanism: Simultaneously, text features are processed by a self-attention mechanism. This mechanism strengthens the contextual relationships between text features by calculating attention weights within each feature. This process helps the model understand the dependencies between words in the text, thus better capturing the global semantic information. This is crucial for accurately interpreting complex semantic information describing remote sensing images, such as scenes describing "vegetation around a building" or "traffic signals at a road intersection."

[0100] Feedforward Neural Network (FFN): After cross-attention and self-attention processing at each layer, the FFN performs a non-linear transformation on the fused features. This step achieves further feature learning and representation enhancement through the network's hierarchical structure. To ensure the stability of feature updates, residual connections and layer normalization are introduced into each layer. Residual connections help solve the gradient vanishing problem in deep networks, ensuring a smooth transition in feature updates, while layer normalization ensures the network's convergence speed and stability. Through this process, textual and visual features are gradually enhanced through multiple interactive layers into features that better represent multimodal semantics. The mathematical expression of the above steps is as follows:

[0101]

[0102] Step 3: After the text and visual features have undergone multiple layers of interaction through cross-attention, self-attention, and feedforward neural networks, the next step is to align the two types of features pixel by pixel. To accurately capture spatial details at multiple scales in the image, a multi-scale deformable attention mechanism (MSDeformAttn) is employed. This mechanism, through a flexible sampling strategy, can capture spatial information and semantic details at different scales. This mechanism is particularly important in remote sensing images because target objects in remote sensing images are typically distributed across different spatial scales and have different sizes and shapes.

[0103] Multi-scale deformable attention ensures that the model effectively captures image details related to text features at all levels by dynamically sampling visual features at different scales. Specifically, guided by text features, this mechanism adaptively selects the visual regions to focus on, thereby effectively capturing key targets in the image. For example, the model can focus on the edges of buildings at a high-resolution scale, while focusing on the layout of the entire area at a lower-resolution scale.

[0104] Through this process, visual and textual features are finely aligned at the pixel level. This mechanism not only ensures the accurate representation of image details but also enhances the model's ability to understand complex scenes in remote sensing images through multi-scale fusion. The mathematical expression of the above process is as follows.

[0105]

[0106] Steps three and four: After completing the bidirectional interaction between visual and textual features, the final feature fusion and mask generation stage begins. First, the visual features enhanced with multi-scale attention and the original visual features are mapped to the same feature space through a 1×1 convolutional layer. The 1×1 convolution is used to compress and map the feature dimensions, ensuring dimensional consistency.

[0107] Then, spatial resampling is used to adjust the mapped visual features to a size that matches the original image, ensuring that the visual features have the same spatial resolution as the image. During this process, text embedding features are also element-wise multiplied with the visual features to enhance the guiding role of text features on visual features, ensuring that semantic information in the text can be accurately located to the corresponding region in the image.

[0108] Generating a predicted segmentation mask: A preliminary segmentation mask is generated from the output of feature fusion. The mask marks the image regions associated with the text description, representing the boundaries or shape of those regions. This step is crucial for the accurate localization of specific targets in the image (such as buildings, roads, water bodies, etc.).

[0109] Mask Upsampling: To ensure the generated segmentation mask remains consistent with the input high-resolution remote sensing image, bilinear interpolation is used to upsample the predicted mask to the same spatial resolution as the original input image. This process preserves detail integrity at high resolution, ensuring accurate segmentation results with good spatial consistency.

[0110] Ultimately, the generated masks can represent the location and outline of target objects in the image, such as buildings, vehicles, and vegetation. These masks can be directly used for subsequent target detection, region recognition, or scene analysis, ensuring that targets in remote sensing images can be accurately identified and processed.

[0111]

[0112] The final result of directional segmentation of the remote sensing image was obtained, which is the mask of the target.

[0113] The other steps and parameters are the same as those in one of the specific implementation methods one or two.

[0114] The above description merely illustrates preferred embodiments of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A remote sensing image directional segmentation method with bidirectional feature enhancement of image and text, characterized in that: The method includes the following steps: Step 1: Extract image features and text features separately through the backbone network, and use the image features to enhance the text features; Step 2: Based on the original image features and text features obtained in Step 1, use the text features to perform attention enhancement on the image features; Step 3: Based on the enhanced text features and enhanced image features obtained in Step 1 and Step 2, perform interactive decoding to finally obtain the mask result of directional segmentation of the remote sensing image; Step one specifically involves: Step 11: Obtain the remote sensing image and its corresponding text description; Steps 1 and 2: Use a visual backbone network to process remote sensing images and extract their multi-scale visual features. The visual backbone network extracts features at different scales and semantic levels layer by layer through convolution or self-attention mechanisms. At the same time, text features are extracted through a text backbone network. These networks can extract the corresponding contextual information based on the input natural language text and generate text features with semantic representation. Step 13: Dimensionality reduction of multi-scale visual features is performed using an adaptive pooling layer. The adaptive pooling layer dynamically adjusts the pooling parameters according to the size of the input image features to generate a feature map of a fixed size. By compressing image features of different scales into the same feature space, the size mismatch problem in subsequent operations can be avoided. Subsequently, the dimensionally reduced image features are fused through a stitching operation to form a compact feature vector containing global information. Step 1, Section 4: Perform interactive attention calculations between the dimensionality-reduced image features and a trainable random cue layer; Step 15: Concatenate the prompt layer containing image information with the extracted text features to generate enhanced text features; Step two specifically involves: Step 2: Flatten the multi-scale image features obtained in Step 1. The flattened image features contain information in both spatial and channel dimensions, which can capture the spatial structure of the image at different scales. Step 22: The flattened image features and text features are fused through an interactive attention mechanism to calculate interactive attention scores at different scales. The interactive attention mechanism identifies which image regions are most relevant to the text description by calculating the attention matrix between image and text features. Step 23: Normalize the interaction attention score calculated in Step 22; use the Softmax function to normalize the attention score, and multiply the normalized attention score by the flattened image features to emphasize the text-related parts of the image. Step 24: Reweight the image features from Step 1; by reintegrating the normalized attention scores obtained in Step 23 with the image features, a cross-modal feature map is obtained. Step 25: Resample the interaction attention scores at different scales, unify the size, and calculate the cross-scale attention difference map. The cross-scale attention difference map can show the correspondence between images and text at different scales. By selecting the top K regions with the greatest differences, the model can be further guided to focus on the image regions that are most relevant to the text description, thereby improving the accuracy of feature extraction. Step 26: Project and stitch the first K regions selected in Step 25; by projecting these regions onto different scales, a unified attention deficit map can be obtained; the attention deficit map integrates attention information from multiple scales, and can reflect global feature differences while maintaining local details; Step 27: Compensate for attention deficits in image features at different scales; by combining text-enhanced multi-scale image features with original image features, the expressive power of image features is further improved, ensuring that the final feature map can reflect the different focuses at different scales in the image, thereby achieving the unification of multi-scale feature attention.

2. The method according to claim 1, characterized in that, Step three specifically involves: Step 3: First, flatten the multi-scale image features enhanced by text in Step 2. The flattened visual features retain the global information and spatial layout of the image, making the subsequent fusion steps smoother. After flattening the image features, align the text features with the image features. Step 3.2: Introduce a multi-layered cross-modal feature interaction mechanism; this cross-modal feature interaction mechanism relies on the following three core modules: cross-attention mechanism, self-attention mechanism, and feedforward neural network; Step 3: After the text and visual features have undergone multiple layers of interaction through cross-attention, self-attention, and feedforward neural networks, the next step is to align the two types of features pixel by pixel. This alignment employs a multi-scale deformable attention mechanism. Multi-scale deformable attention ensures that the model can effectively capture image details related to text features at all levels by dynamically sampling visual features at different scales. Through this process, visual features and text features achieve fine alignment at the pixel level. The mathematical expression of the above process is as follows: Steps 3 and 4: After completing the two-way interaction between visual and textual features, we proceed to the final feature fusion and mask generation stage.

3. The method according to claim 2, characterized in that, The cross-attention mechanism is used to establish a global association between visual features and text features in each layer of interaction; by calculating the attention weight matrix of visual features and text features, the model can automatically capture which visual regions are most relevant to the text description; the cross-attention mechanism can ensure that the key information contained in the text features can correctly guide the model to focus on the target region in the image.

4. The method according to claim 3, characterized in that, The self-attention mechanism strengthens the contextual relationships between text features by calculating attention weights within text features.

5. The method according to claim 4, characterized in that, After cross-attention and self-attention processing at each layer, the feedforward neural network performs a non-linear transformation on the fused features. Residual connections and layer normalization are introduced into each layer operation. Through this introduction process, textual and visual features are gradually enhanced into features that can better express multimodal semantics through multiple interactive layers.

6. The method according to claim 4, characterized in that, Steps three and four are specifically as follows: First, the visual features enhanced by multi-scale attention and the original visual features are mapped to the same feature space through a 1×1 convolutional layer; the 1×1 convolution is used to compress and map the feature dimensions to ensure the consistency of feature dimensions. Then, spatial resampling is used to adjust the mapped visual features to a size that matches the original image, ensuring that the visual features are consistent with the spatial resolution of the image. In this process, text embedding features are also multiplied element-wise with visual features to enhance the guiding role of text features on visual features and ensure that semantic information in the text can be accurately located to the corresponding region of the image. Generate a predicted segmentation mask: A preliminary segmentation mask is generated from the output of feature fusion; the mask marks the image regions related to the text description, indicating the boundaries or shapes of the regions; Mask upsampling: The predicted mask is upsampled to the same spatial resolution as the original input image using bilinear interpolation; Ultimately, the generated mask can represent the position and outline of the target object in the image. These masks can be directly used for subsequent target detection, region recognition, or scene analysis to ensure that the target in the remote sensing image can be accurately identified and processed. Finally, the result of directional segmentation of the remote sensing image is obtained, which is the target mask.

Images