Remote sensing image directivity segmentation method for image text bidirectional feature enhancement

Abstract

The invention provides a remote sensing image directivity segmentation method for image text bidirectional feature enhancement, and the method comprises the following steps: 1, respectively extracting image features and text features through a backbone network, and carrying out the enhancement of the text features through the image features; step 2, based on the original image features and the text features obtained in the step 1, using the text features to carry out attention supplementary enhancement on the image features; and step 3, performing interactive decoding based on the enhanced text features and the enhanced image features obtained in the step 1 and the step 2, and finally obtaining a mask result of directional segmentation of the remote sensing image. According to the image and text bidirectional feature enhancement method provided by the invention, the similarity and complementarity between cross-modal features are fully mined, accurate alignment of the features is realized, and the precision of directional segmentation is remarkably improved.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing image directional segmentation method for image-text bidirectional feature enhancement. Background Art

[0002] In recent years, large language models have made significant progress in language understanding, analysis, and generation, especially in contextual reasoning and multi-task processing. These breakthroughs have promoted the development of visual language models, which can combine visual information with language expression and have been widely used in tasks such as image generation description, image text retrieval, and visual question answering. Especially in the field of remote sensing, how to integrate natural language with remote sensing images has become the current research focus, promoting the transformation from traditional image processing to intelligent remote sensing analysis.

[0003] The directional segmentation task of remote sensing images is an emerging technology that aims to generate pixel-level masks of specific targets in remote sensing images through natural language descriptions. The key to this task is to achieve accurate positioning and segmentation of specific areas or objects by matching language descriptions with image content. Compared with traditional remote sensing image processing methods, directional segmentation of remote sensing images not only improves user interactivity and convenience, but also breaks through the technical barriers of semantic understanding, allowing non-professional users to quickly retrieve and analyze the content in remote sensing images through simple language descriptions.

[0004] Remote sensing image directional segmentation technology has shown broad application prospects in many fields. First, in land use monitoring, it can automatically generate segmentation results for specific areas to help decision makers evaluate land changes in real time. In search and rescue operations, remote sensing image directional segmentation can quickly locate targets and improve operational efficiency by describing specific environments or objects. In environmental monitoring, it can detect pollution sources or environmental changes through language descriptions. In addition, fields such as military intelligence, agricultural assessments, and urban planning can also benefit from this technology to achieve more accurate remote sensing data analysis.

[0005] However, the directional segmentation of remote sensing images still faces challenges. Remote sensing images usually have high resolution and complex scenes, and how to efficiently extract targets that match language from large amounts of data remains a technical difficulty. Summary of the invention

[0006] The purpose of the present invention is to solve the problem of low precision of the existing remote sensing image directional segmentation model, and also to solve the problem of insufficient utilization of image text information and insufficient interaction of different modal information in the existing remote sensing image directional segmentation model, and to propose a remote sensing image directional segmentation method with image text bidirectional feature enhancement.

[0007] The present invention is implemented by the following technical scheme. The present invention proposes a remote sensing image directional segmentation method with image-text bidirectional feature enhancement, and the method comprises the following steps:

[0008] Step 1: Extract image features and text features through the backbone network respectively, and use image features to enhance text features;

[0009] Step 2: Based on the original image features and text features obtained in step 1, use the text features to perform attention supplementation and enhancement on the image features;

[0010] Step 3: Interactive decoding is performed based on the enhanced text features and enhanced image features obtained in steps 1 and 2, and finally a mask result of directional segmentation of the remote sensing image is obtained.

[0011] Furthermore, the step 1 is specifically as follows:

[0012] Step 11: Obtain a remote sensing image V and its corresponding text description L;

[0013] Step 1 and 2: Use the visual backbone network to process remote sensing images and extract their multi-scale visual features The visual backbone network extracts features of different scales and semantic levels layer by layer through convolution or self-attention mechanism; at the same time, text features are extracted through the text backbone network; these networks can extract corresponding contextual information based on the input natural language text and generate text features with semantic representation.

[0014] Step 1-3: Reduce the dimension of multi-scale visual features through an adaptive pooling layer; the adaptive pooling layer dynamically adjusts the pooling parameters according to the size of the input image features to generate a fixed-size feature map; by compressing image features of different scales into the same feature space, the size mismatch problem in subsequent operations can be avoided; then, the image features after dimensionality reduction are fused through a splicing operation to form a compact feature vector containing global information;

[0015]

[0016] Step 14: Perform interactive attention calculation on the reduced image features and a trainable random hint layer;

[0017]

[0018] Step 15: The prompt layer P containing image information v And the extracted text feature L v Perform splicing to generate enhanced text features.

[0019] Furthermore, the step 2 is specifically as follows:

[0020] Step 21: Flatten the multi-scale image features obtained in step 1; the flattened image features contain information of spatial dimensions and channel dimensions, and can capture the spatial structure of the image at different scales;

[0021] V iq =Flatten(ω iq (V i ))

[0022] Step 22: Compare the flattened image features with the text features L i The interactive attention mechanism is used to fuse and calculate the interactive attention scores at different scales. The interactive attention mechanism calculates the attention matrix between image and text features to identify which image regions are most relevant to the text description.

[0023]

[0024] Step 23: Normalize the interactive attention score calculated in step 22; use the Softmax function to normalize the attention score, and multiply the normalized attention score with the flattened image feature to emphasize the text-related part of the image;

[0025]

[0026] Step 24: For the image feature V in step 1 i Re-weighting; by reintegrating the normalized attention scores obtained in steps 2 and 3 with the image features, a cross-modal feature map is obtained

[0027] V li =Conv 1×1 ((Att i )☉V i

[0028] Step 25: Resample the interactive attention scores at different scales, unify the sizes and calculate the cross-scale attention difference map; the cross-scale attention difference map can show the correspondence between images and texts at different scales. By selecting the top K regions with the largest differences, the model can be further guided to focus on those image regions that are most relevant to the text description, thereby improving the accuracy of feature extraction;

[0029]

[0030] Step 26: Project and concatenate the first K regions selected in step 25; by projecting these regions onto different scales, a unified attention deficit map can be obtained; the attention deficit map integrates attention information from multiple scales and can reflect global feature differences while maintaining local details;

[0031]

[0032] Step 27: Compensate for attention deficits of image features at different scales; by combining the multi-scale image features enhanced by text with the original image features, the expressiveness of the image features is further improved, ensuring that the final feature map can reflect the different focuses at different scales in the image, thereby achieving the unification of multi-scale feature attention;

[0033]

[0034] Furthermore, the step three is specifically as follows:

[0035] Step 3: Flatten the multi-scale image features enhanced by text in step 2; the flattened visual features retain the global information and spatial layout of the image, making the subsequent fusion step smoother; after the image features are flattened, align the text features with the image features;

[0036]

[0037] Step 32: Introduce a multi-level cross-modal feature interaction mechanism; the cross-modal feature interaction mechanism relies on the following three core modules: cross-attention mechanism, self-attention mechanism and feedforward neural network;

[0038] Step 3. After the text and visual features have undergone multi-layer interactions of cross attention, self-attention and feedforward neural networks, the two features need to be aligned pixel by pixel. The alignment uses a multi-scale deformable attention mechanism. The multi-scale deformable attention dynamically samples visual features of different scales to ensure that the model can effectively capture image details related to text features at all levels. Through this process, the visual features and text features are finely aligned at the pixel level. The above process is mathematically expressed as:

[0039]

[0040] Step 3 and 4: After completing the two-way interaction between visual features and text features, enter the final feature fusion and mask generation stage.

[0041] Furthermore, the cross-attention mechanism is used to establish a global association between visual features and text features in each layer of interaction; by calculating the attention weight matrix of visual features and text features, the model can automatically capture which visual areas are most relevant to the text description; the cross-attention mechanism can ensure that the key information contained in the text features can correctly guide the model to focus on the target area in the image.

[0042] Furthermore, the self-attention mechanism strengthens the contextual relationship between text features by calculating the attention weights within the text features.

[0043] Furthermore, after the cross attention and self-attention processing of each layer, the feedforward neural network performs nonlinear transformation on the fused features, and residual connection and layer normalization are introduced into each layer operation; after this introduction processing, the text features and visual features are gradually enhanced through multiple interactive layers to be features that can better express multimodal semantics; the mathematical expression of the above steps is:

[0044]

[0045] Furthermore, the steps three and four are specifically as follows:

[0046] First, the visual features enhanced by multi-scale attention and the original visual features are mapped to the same feature space through a 1×1 convolution layer; the 1×1 convolution is used to compress and map the feature dimensions to ensure the dimensional consistency of the features;

[0047] Then, spatial resampling is used to adjust the mapped visual features to a size that matches the original image, ensuring that the visual features are consistent with the image spatial resolution. In this process, the text embedding features are also multiplied element-wise with the visual features to enhance the guiding role of the text features on the visual features, ensuring that the semantic information in the text can be accurately located in the corresponding area of ​​the image.

[0048] Generate predicted segmentation mask: Generate a preliminary segmentation mask through the output of feature fusion; the mask marks the image area related to the text description, indicating the boundary or shape of the area;

[0049] Mask upsampling: The predicted mask is upsampled to the same spatial resolution as the original input image using bilinear interpolation;

[0050] Finally, the generated masks can indicate the position and outline of the target object in the image. These masks can be directly used for subsequent target detection, region recognition or scene analysis to ensure that the targets of remote sensing images can be accurately identified and processed.

[0051]

[0052] Finally, the result of directional segmentation of the remote sensing image is obtained, that is, the mask of the target.

[0053] The present invention has the following beneficial effects:

[0054] The present invention solves the problem of insufficient utilization of image and text features in existing remote sensing image directional segmentation: previous remote sensing image directional segmentation models usually only focus on utilizing a single modality in text or image features, and the features of the other modality are not fully utilized, resulting in limited improvement in segmentation results compared to using only single-modality data, and failing to fully demonstrate the advantages of directional segmentation. The present invention significantly improves the accuracy of directional segmentation by fully combining the features of both image and text modalities and using cross-modal information for encoding and decoding.

[0055] The present invention also solves the problem of insufficient interaction between image and text features in existing remote sensing image directional segmentation: in previous remote sensing image directional segmentation models, the interaction process of cross-modal features is relatively simple, and usually one-way feature interaction is adopted, which makes it difficult to fully utilize the similarities and complementarities between cross-modal features. This deficiency makes it difficult to effectively align the two modal features of image and text, thereby limiting the improvement of the directional segmentation effect. The present invention proposes a bidirectional feature enhancement method for images and texts, which fully exploits the similarities and complementarities between cross-modal features, realizes precise alignment of features, and significantly improves the accuracy of directional segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of the remote sensing image directional segmentation method with image-text bidirectional feature enhancement according to the present invention;

[0057] Figure 2 A schematic diagram of a module for supplementing image-guided text prompts;

[0058] Figure 3 Schematic diagram of the text-guided image attention supplement module;

[0059] Figure 4 Schematic diagram of the image-text interactive decoder. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Specific implementation method 1: The specific process of the remote sensing image directional segmentation method with image-text bidirectional feature enhancement proposed in this implementation is as follows:

[0062] Step 1: Extract image features and text features through the backbone network respectively, and use image features to enhance text features;

[0063] Step 2: Based on the original image features and text features obtained in step 1, use the text features to perform attention supplementation and enhancement on the image features;

[0064] Step 3: Interactive decoding is performed based on the enhanced text features and enhanced image features obtained in steps 1 and 2, and finally a mask result of directional segmentation of the remote sensing image is obtained.

[0065] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that the step 1 extracts image features and text features respectively through the backbone network, and uses the image features to enhance the text features;

[0066] The specific process is:

[0067] Step 1: First, obtain the remote sensing image V and its corresponding text description L. Remote sensing images can come from a variety of data sources such as satellites and drones, and text descriptions are usually manually annotated or automatically generated to describe the content or geographical environment in the image;

[0068] Step 1 and 2: Next, use an appropriate visual backbone network (such as ResNet, ViT, Swin, ConvNext, etc.) to process the remote sensing image and extract its multi-scale visual features. The visual backbone network extracts features of different scales and semantic levels layer by layer through convolution or self-attention mechanisms. At the same time, text features are extracted through text backbone networks (such as LSTM, CLIP, BERT, etc.). These network models can extract corresponding contextual information based on the input natural language text and generate text features with semantic representation.

[0069] Step 1-3: In order to align image features with text features and enhance text features, the multi-scale visual features are reduced in dimension through the adaptive pooling layer. The adaptive pooling layer dynamically adjusts the pooling parameters according to the size of the input image features to generate a fixed-size feature map. By compressing image features of different scales into the same feature space, the size mismatch problem in subsequent operations can be avoided. Subsequently, the image features that have undergone dimensionality reduction are fused through a splicing operation to form a compact feature vector that contains global information. This step not only retains the multi-scale information of the image, but also reduces the computational complexity.

[0070]

[0071] Step 14: Next, the reduced image features are interactively attentioned with a trainable random prompt layer (PromptLayer). The random prompt layer is a learnable embedding layer designed to capture key information in image features. Through the cross-attention mechanism, the prompt layer can extract key contextual information from image features. This mechanism selectively focuses on salient areas in the image by calculating the attention weights between the prompt layer and the image features, thereby improving the efficiency and effectiveness of feature extraction.

[0072]

[0073] Step 15: The prompt layer P containing image information v And the extracted text feature L v The splicing is performed to generate enhanced text features. This splicing operation not only improves the semantic expression ability of the text, but also integrates the visual information in the image into the text features, thereby achieving a deep fusion between the image and the text. These enhanced text features can more accurately guide the decoding and segmentation of the image in the subsequent processing steps.

[0074] The other steps and parameters are the same as those in the first embodiment.

[0075] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that in step two, based on the original image features and text features obtained in step one, the text features are used to perform attention supplement and enhancement on the image features;

[0076] The specific process is:

[0077] Step 2: First, the multi-scale image features obtained in step 1 are flattened. This process converts three-dimensional or four-dimensional image features into two-dimensional vector representations, which is convenient for alignment and interaction with text features. The flattened image features contain information of spatial dimensions and channel dimensions, and can capture the spatial structure of the image at different scales.

[0078] V iq =Flatten(ω iq (V i ))

[0079] Step 22: Compare the flattened image features with the text features L iThe cross-attention mechanism is used for fusion and the cross-attention scores at different scales are calculated. The cross-attention mechanism calculates the attention matrix between image and text features to identify which image regions are most relevant to the text description. Through this mechanism, the model can guide the semantic information in the text to the corresponding image region and achieve accurate modality alignment.

[0080]

[0081] Step 23: Normalize the interactive attention scores calculated in step 22. First, use the Softmax function to normalize the attention scores to ensure that scores of different scales can be effectively compared. The normalized attention scores are multiplied by the flattened image features to emphasize the text-related parts of the image. To further improve the recognition of features, a threshold operation can be used to suppress attention scores below a certain threshold, thereby focusing on important feature areas.

[0082]

[0083] Step 24: For the image feature V in step 1 i By reintegrating the normalized attention scores obtained in steps 2 and 3 with the image features, a cross-modal feature map can be obtained. This cross-modal feature map combines multimodal information from text and images and can better express the key information in the image.

[0084] V li =Conv 1×1 (Att i )☉V i

[0085] Step 25: Next, resample the interactive attention scores at different scales, unify the size and calculate the cross-scale attention difference map. The cross-scale attention difference map can show the correspondence between images and texts at different scales. By selecting the top K regions with the largest differences, the model can be further guided to focus on those image regions that are most relevant to the text description, thereby improving the accuracy of feature extraction.

[0086]

[0087] Step 26: Project and concatenate the first K regions selected in step 25. By projecting these regions to different scales, a unified attention deficit map can be obtained. This map integrates attention information from multiple scales and can reflect global feature differences while maintaining local details.

[0088]

[0089] Step 27. Finally, compensate for the attention deficit of image features at different scales. By combining the multi-scale image features enhanced by text with the original image features, the expressiveness of the image features can be further improved, ensuring that the final feature map can reflect the different focuses at different scales in the image, thereby achieving the unification of multi-scale feature attention.

[0090]

[0091] The other steps and parameters are the same as those in Specific Implementation 1 to 2.

[0092] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that in step 3, interactive decoding is performed based on the enhanced text features and enhanced image features obtained in step 1 and step 2, and finally a mask result of directional segmentation of the remote sensing image is obtained;

[0093] The specific process is:

[0094] Step 3 1. First, flatten the multi-scale image features that have been enhanced with text in step 2. The purpose of flattening is to convert the original three-dimensional image features (i.e., feature maps containing spatial dimensions and channel dimensions) into two-dimensional feature vectors, which are convenient for cross-modal fusion and alignment with text features. This operation ensures that visual features of different scales can be processed in a unified dimension, avoiding information loss due to scale differences. The flattened visual features retain the global information and spatial layout of the image, making the subsequent fusion step smoother.

[0095]

[0096] After flattening the image features, the text features are aligned with the image features. This process models the relationship between text and image features to ensure that the two are expressed consistently in the feature space. This not only improves the relevance between text and visual information, but also provides more accurate feature matching for subsequent mask generation.

[0097] Step 32: In order to effectively capture and integrate the relationship between text features and visual features, this step introduces a multi-level cross-modal feature interaction mechanism. This mechanism relies on the following three core modules: Cross-Attention Mechanism, Self-Attention Mechanism and Feed-Forward Network (FFN).

[0098] Cross-attention mechanism: First, the cross-attention mechanism is used to establish a global association between visual features and text features in each layer of interaction. By calculating the attention weight matrix of visual features and text features, the model can automatically capture which visual areas are most relevant to the text description. The cross-attention mechanism ensures that the key information contained in the text features can correctly guide the model to focus on the target area in the image. This step is especially important for the recognition and segmentation of fine-grained targets (such as vehicles, buildings, etc.) in remote sensing images.

[0099] Self-attention mechanism: At the same time, text features are processed by the self-attention mechanism. The self-attention mechanism strengthens the contextual relationship between text features by calculating the attention weights within the text features. This process helps the model understand the dependencies between individual words in the text, so as to better capture the global semantic information of the text. This is crucial for accurately interpreting the complex semantic information describing remote sensing images, such as describing scenes such as "vegetation around buildings" or "traffic signals at road intersections".

[0100] Feedforward Neural Network (FFN): After the cross-attention and self-attention processing of each layer, the feedforward neural network performs a nonlinear transformation on the fused features. This step achieves further learning and expression enhancement of features through the hierarchical structure of the network. In order to ensure the stability of feature updates, residual connections (Residual Connection) and layer normalization (Layer Normalization) are introduced into each layer of operation. Residual connections help solve the gradient vanishing problem in deep networks and ensure a smooth transition of feature updates, while layer normalization ensures the convergence speed and stability of the network. After this processing, text features and visual features are gradually enhanced through multiple interactive layers to features that can better express multimodal semantics. The mathematical expression of the above steps is:

[0101]

[0102] Step 3: After the text and visual features have been subjected to multi-layer interactions of cross-attention, self-attention, and feedforward neural networks, the two features need to be aligned pixel by pixel. In order to accurately capture the spatial detail information at multiple scales in the image, a multi-scale deformable attention mechanism (MSDeformAttn) is used. This mechanism can capture spatial information and semantic details at different scales through a flexible sampling strategy. This mechanism is particularly important in remote sensing images because the target objects in remote sensing images are usually distributed at different spatial scales and have different sizes and shapes.

[0103] Multi-scale deformable attention ensures that the model can effectively capture image details related to text features at all levels by dynamically sampling visual features at different scales. Specifically, the mechanism can adaptively select the visual area to focus on based on the guidance of text features, thereby effectively capturing key targets in the image. For example, the model can focus on the edge of a building at a high-resolution scale, while focusing on the layout of the entire area at a lower-resolution scale.

[0104] Through this process, the visual features and text features are finely aligned at the pixel level. This mechanism not only ensures the accurate expression of image details, but also enhances the model's ability to understand complex scenes in remote sensing images through multi-scale fusion. The above process is mathematically expressed as:

[0105]

[0106] Step 3 and 4: After completing the two-way interaction between visual features and text features, enter the final feature fusion and mask generation stage. First, the visual features enhanced by multi-scale attention and the original visual features are mapped to the same feature space through a 1×1 convolution layer. The 1×1 convolution is used to compress and map the feature dimensions to ensure the dimensional consistency of the features.

[0107] Then, spatial resampling is used to adjust the mapped visual features to a size that matches the original image, ensuring that the visual features are consistent with the image spatial resolution. In this process, the text embedding features are also multiplied element-wise with the visual features to enhance the guiding role of the text features on the visual features, ensuring that the semantic information in the text can be accurately located in the corresponding area of ​​the image.

[0108] Generate predicted segmentation mask: Generate a preliminary segmentation mask through the output of feature fusion. The mask marks the image area related to the text description and indicates the boundary or shape of the area. This step is very important for the precise positioning of specific targets in the image (such as buildings, roads, water bodies, etc.).

[0109] Mask upsampling: In order to make the generated segmentation mask consistent with the input high-resolution remote sensing image, the predicted mask is upsampled to the same spatial resolution as the original input image using bilinear interpolation. Through this process, the segmentation mask can maintain detail integrity at high resolution, ensuring that the final segmentation result is accurate and has good spatial consistency.

[0110] Finally, the generated masks can indicate the location and outline of the target objects in the image, such as buildings, vehicles, vegetation, etc. These masks can be directly used for subsequent target detection, region recognition or scene analysis, ensuring that the targets of remote sensing images can be accurately identified and processed.

[0111]

[0112] Finally, the result of directional segmentation of the remote sensing image is obtained, that is, the mask of the target.

[0113] The other steps and parameters are the same as those in Specific Implementation 1 to 2.

[0114] The above only expresses the preferred implementation mode of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A remote sensing image directional segmentation method with bidirectional feature enhancement of image and text, characterized by: The method comprises the following steps: Step 1: Extract image features and text features through the backbone network respectively, and use image features to enhance text features; Step 2: Based on the original image features and text features obtained in step 1, use the text features to perform attention supplementation and enhancement on the image features; Step 3: Interactive decoding is performed based on the enhanced text features and enhanced image features obtained in steps 1 and 2, and finally a mask result of directional segmentation of the remote sensing image is obtained.

2. The method according to claim 1, characterized in that The step 1 is specifically as follows: Step 11: Obtain a remote sensing image V and its corresponding text description L; Step 1 and 2: Use the visual backbone network to process remote sensing images and extract their multi-scale visual features The visual backbone network extracts features of different scales and semantic levels layer by layer through convolution or self-attention mechanism; at the same time, text features are extracted through the text backbone network; these networks can extract corresponding contextual information based on the input natural language text and generate text features with semantic representation. Step 13: Reduce the dimension of multi-scale visual features through the adaptive pooling layer; The adaptive pooling layer dynamically adjusts the pooling parameters according to the size of the input image features in order to generate a fixed-size feature map. By compressing image features of different scales into the same feature space, the size mismatch problem in subsequent operations can be avoided. Subsequently, the image features after dimensionality reduction are fused through a splicing operation to form a compact feature vector that contains global information. Step 14: Perform interactive attention calculation on the reduced image features and a trainable random hint layer; Step 15: The prompt layer P containing image information v And the extracted text feature L v Perform splicing to generate enhanced text features.

3. The method according to claim 2, characterized in that The step 2 is specifically as follows: Step 21: Flatten the multi-scale image features obtained in step 1; the flattened image features contain information of spatial dimensions and channel dimensions, and can capture the spatial structure of the image at different scales; V iq =Flat(ω iq (V i )) Step 22: Compare the flattened image features with the text features L i Fusion is performed through the interactive attention mechanism to calculate interactive attention scores at different scales; The interactive attention mechanism identifies which image regions are most relevant to the text description by calculating the attention matrix between image and text features; Step 23: Normalize the interactive attention score calculated in step 22; use the Softmax function to normalize the attention score, and multiply the normalized attention score with the flattened image feature to emphasize the text-related part of the image; Step 24: For the image feature V in step 1 i Re-weighting; by reintegrating the normalized attention scores obtained in steps 2 and 3 with the image features, a cross-modal feature map is obtained V li =Conv 1×1 (To i )☉V i Step 25: Resample the interactive attention scores at different scales, unify the sizes and calculate the cross-scale attention difference map; the cross-scale attention difference map can show the correspondence between images and texts at different scales. By selecting the top K regions with the largest differences, the model can be further guided to focus on those image regions that are most relevant to the text description, thereby improving the accuracy of feature extraction; Step 26: Project and concatenate the first K regions selected in step 25; by projecting these regions onto different scales, a unified attention deficit map can be obtained; the attention deficit map integrates attention information from multiple scales and can reflect global feature differences while maintaining local details; Step 27: Compensate for attention deficits of image features at different scales; by combining the multi-scale image features enhanced by text with the original image features, the expressiveness of the image features is further improved, ensuring that the final feature map can reflect the different focuses at different scales in the image, thereby achieving the unification of multi-scale feature attention; 4. The method according to claim 3, characterized in that: The step three is specifically as follows: Step 3: Flatten the multi-scale image features enhanced by text in step 2; the flattened visual features retain the global information and spatial layout of the image, making the subsequent fusion step smoother; after the image features are flattened, align the text features with the image features; Step 32: Introduce a multi-level cross-modal feature interaction mechanism; the cross-modal feature interaction mechanism relies on the following three core modules: cross-attention mechanism, self-attention mechanism and feedforward neural network; Step 3. After the text and visual features have undergone multi-layer interactions of cross attention, self-attention and feedforward neural networks, the two features need to be aligned pixel by pixel. The alignment uses a multi-scale deformable attention mechanism. The multi-scale deformable attention dynamically samples visual features of different scales to ensure that the model can effectively capture image details related to text features at all levels. Through this process, the visual features and text features are finely aligned at the pixel level. The above process is mathematically expressed as: Step 3 and 4: After completing the two-way interaction between visual features and text features, enter the final feature fusion and mask generation stage.

5. The method according to claim 4, characterized in that The cross-attention mechanism is used to establish a global association between visual features and text features in each layer of interaction; by calculating the attention weight matrix of visual features and text features, the model can automatically capture which visual areas are most relevant to the text description; the cross-attention mechanism can ensure that the key information contained in the text features can correctly guide the model to focus on the target area in the image.

6. The method according to claim 4, characterized in that The self-attention mechanism strengthens the contextual relationship between text features by calculating the attention weights within the text features.

7. The method according to claim 4, characterized in that After the cross attention and self-attention processing of each layer, the feedforward neural network performs nonlinear transformation on the fused features, and residual connection and layer normalization are introduced into each layer operation; after this introduction, the text features and visual features are gradually enhanced through multiple interactive layers to become features that can better express multimodal semantics; the mathematical expression of the above steps is:

8. The method according to claim 4, characterized in that The steps three and four are specifically: First, the visual features enhanced by multi-scale attention and the original visual features are mapped to the same feature space through a 1×1 convolution layer; the 1×1 convolution is used to compress and map the feature dimensions to ensure the dimensional consistency of the features; Then, spatial resampling is used to adjust the mapped visual features to a size that matches the original image, ensuring that the visual features are consistent with the image spatial resolution. In this process, the text embedding features are also multiplied element-wise with the visual features to enhance the guiding role of the text features on the visual features, ensuring that the semantic information in the text can be accurately located in the corresponding area of ​​the image. Generate predicted segmentation mask: Generate a preliminary segmentation mask through the output of feature fusion; the mask marks the image area related to the text description, indicating the boundary or shape of the area; Mask upsampling: The predicted mask is upsampled to the same spatial resolution as the original input image using bilinear interpolation; Finally, the generated masks can indicate the position and outline of the target object in the image. These masks can be directly used for subsequent target detection, region recognition or scene analysis to ensure that the targets of remote sensing images can be accurately identified and processed. Finally, the result of directional segmentation of the remote sensing image is obtained, that is, the mask of the target.

Images