Remote sensing image anaphora segmentation method and system based on multi-mode prompt

Abstract

The invention relates to a remote sensing image anaphora segmentation method and system based on multi-mode prompt, and belongs to the technical field of computer vision and remote sensing intelligent interpretation. The method comprises the following steps: S1, data preprocessing: dividing a remote sensing image anaphora segmentation data set; s2, constructing a remote sensing image anaphora segmentation model based on multi-modal prompt, wherein the remote sensing image anaphora segmentation model comprises an image encoder, a space prior module, a text encoder, a fine-grained visual language interaction device and a multi-modal prompt encoder; s3, training a remote sensing image anaphora segmentation model based on multi-modal prompt: for the training of the remote sensing image anaphora segmentation model, performing optimization training on the model by using a loss function combining Focal loss, cross entropy loss and Dice loss; and S4, outputting a segmentation result: inputting a remote sensing image to be segmented into the trained model, and then inputting a model output result into a mask decoder to complete anaphora segmentation of the target object. According to the invention, the interpretation efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and remote sensing intelligent interpretation, and relates to a remote sensing image reference segmentation method and system based on multi-modal prompts. BACKGROUND

[0002] Remote sensing image reference segmentation aims to locate and segment the target in the remote sensing image according to the natural language description, and is a core technology in the fields of geographic information interpretation, precision agriculture, disaster monitoring, etc. Remote sensing image reference segmentation is a cross-research task of computer vision and natural language processing, which requires the segmentation method to understand two modal data of input image and text description of the target at the same time, so as to realize the pixel-level segmentation of the target region. For a given image and text description, the reference segmentation process is as follows:

[0003]

[0004] Where x is the given input image, t is the natural language description text, and ε vis (·) and ε text (·) represent the visual and text encoders respectively, represents cross-modal feature interaction, D ref is a mask decoder, and Y ref is the probability map of the output target region.

[0005] SAM is a general image segmentation model that can achieve effective segmentation of any object through simple geometric prompts. However, the geometric prompts supported by SAM by nature mainly include points, boxes and masks, which cannot directly process natural language prompts. The semantic gap between geometric prompts and text prompts makes it difficult for SAM to perform the reference segmentation task that relies on semantic understanding.

[0006] Therefore, there is an urgent need for an image segmentation method that can understand both geometric prompts and text prompts. SUMMARY

[0007] Therefore, the present application aims to provide a remote sensing image reference segmentation method and system based on multi-modal prompts to solve the limitations of single modal prompts in the prior art. Specifically, if the traditional method only relies on geometric prompts, it can locate the target space range through coordinate hard constraints, but it cannot represent semantic attributes. If only text prompts are used, although open semantics can be described, there is a lack of accurate guidance for pixel-level spatial distribution, resulting in insufficient segmentation accuracy. The present solution combines the advantages of geometric prompts and text prompts to build a cross-modal collaborative mechanism, enabling the model to accurately analyze language descriptions, thereby achieving more efficient and intelligent human-computer interaction and improving the automation level and practical value of remote sensing image analysis.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] Solution 1: A remote sensing image referent segmentation method based on multimodal cues, including the following steps:

[0010] S1: Data preprocessing: Divide the remote sensing image into segmentation datasets, including training set, validation set, and test set;

[0011] S2: Build a remote sensing image referent segmentation model based on multimodal cues, including an image encoder, a spatial prior module, a text encoder, a fine-grained visual language interactor, and a multimodal cue encoder;

[0012] S3: Training a remote sensing image referent segmentation model based on multimodal cues: For the training of the remote sensing image referent segmentation model, a loss function combining Focal loss, cross entropy loss, and Dice loss is used to optimize the model training;

[0013] S4: Output segmentation results: Input the remote sensing image to be segmented into the trained model, and then input the model output results into the mask decoder to complete the referential segmentation of the target object.

[0014] Furthermore, in step S2, the image encoder introduces a multi-scale feature fusion mechanism through Vision Transformer (ViT) to enhance the hierarchical extraction of image spatial information. Specifically, the input remote sensing image The image encoder of the general image segmentation model (Segment Anything Model, SAM) is converted into a visual embedding vector F v :

[0015] F v =ε v (x)

[0016] Among them, ε v (·) represents the image encoder, and H, W, and C are the height, width, and number of channels of the remote sensing image, respectively.

[0017] Furthermore, in step S2, the processing flow of the Spatial Prior Module (SPM) is as follows: the remote sensing image x first undergoes a downsampling phase consisting of three convolutional layers and a pooling layer to obtain a feature map After that, f1 is input into several convolutional layers with kernel size 3 and stride 2, which doubles the number of channels while reducing the size of the feature map. Finally, a 1×1 convolution is applied to map the number of channels of all feature maps to a fixed D dimension, i.e.

[0018] f1,f2,f3,f4=SPM(x)

[0019] Among them, SPM represents the spatial prior module; thus we get a feature pyramid {f2,f3,f4}, where the feature map as well as These feature maps are flattened and concatenated to obtain multi-scale spatial features F sp As input to subsequent modules:

[0020]

[0021] Here, dim=0 means concatenation in the first dimension, and Concat means concatenation.

[0022] Furthermore, in step S2, the text encoder is used to encode the referential expression into text features, specifically: for the input referential expression T = {t1, t2, ..., t n} will be converted into a text embedding vector F through the text encoder text :

[0023] F text =E text (T)

[0024] Among them, t n represents the nth expression, E text (·) represents a text encoder, and the Transformer model represented by the pre-trained bidirectional encoder (Bidirectional Encoder Representations from Transformers, BERT) is used as the text encoder.

[0025] Furthermore, in step S2, the processing flow of the fine-grained visual language interactor is as follows: the original contextual text prompt t c , ground object prompts o and spatial location cues p Together through the text encoder ε t After encoding, it is input into the hint enhancement module in the fine-grained visual language interactor and combined with the updated multi-scale spatial feature F sp Interact to highlight relevant guidance information; visual features and multi-scale spatial features will also interact at multiple stages of feature extraction, and finally obtain the enhanced text prompt vector L t and multi-scale spatial features F sp , as shown below:

[0026] L t ,F sp = Ι(F sp ,ε t (t c ),ε t(t o ),ε t (t p ))

[0027] Here, Ι(·) represents the fine-grained vision-language interactor module.

[0028] Furthermore, in step S2, the processing flow of the multimodal prompt encoder is as follows: by receiving point and box geometric prompts, and also receiving enhanced text prompts, in the encoder, all the information brought by the multimodal prompts will be fused and finally mapped into a sparse prompt embedding vector E sparse and dense hint embedding vector E dense , is input into the mask decoder as guidance information. The whole process is shown in the following formula:

[0029] E sparse ,E dense =P(L t ,points,boxes)

[0030] Here, P(·) represents the multimodal prompt encoder, points and boxes represent the input point and box prompts, respectively. When these types of prompts do not exist, the multimodal prompt encoder does not process them, which is consistent with the prompt encoder in SAM.

[0031] Furthermore, in step S3, the loss function is calculated as:

[0032]

[0033] Among them, ω is the adjustment factor that balances the traditional cross entropy loss and Focal loss, D(·) represents the downsampling operation, m is the 0-1 prediction mask obtained by the prediction mask probability matrix p, CE(·) is the cross entropy loss, Focal(·) is the Focal loss, Dice(·) is the Dice loss, m is the segmentation mask, λ1 is the weight controlling the cross entropy loss and Focal loss, and λ2 is the weight controlling the Dice loss.

[0034] Furthermore, in step S4, the processing flow of the mask decoder is as follows: the sparse hint embedding vector E sparse and dense hint embedding vector E dense Will be along with the visual embedding vector F v It is input into the mask decoder to complete the segmentation of the target object:

[0035] m=D m (F v ,E spares ,E dense )

[0036] Among them, D m(·) is the mask decoder, and m is the segmentation mask of the target object.

[0037] Solution 2: A remote sensing image referent segmentation system based on multimodal cues, including a user interaction module, a referent segmentation processing module, and a result analysis module; the referent segmentation processing module includes an image encoder, a spatial prior module, a text encoder, a fine-grained visual language interactor, a multimodal cue encoder, and a mask decoder;

[0038] The user interaction module receives user input through a local graphical interface and supports uploading remote sensing images in JPEG / PNG format, natural language text prompts, and geometric annotation information (such as points or boxes). These inputs are passed to the reference segmentation processing module as prompt information for the reference segmentation task.

[0039] The referent segmentation processing module performs multimodal feature extraction and cross-modal interaction processing in parallel on the local GPU. Image features and text features are extracted through the image encoder and text encoder, respectively. Subsequently, the fine-grained visual language interactor performs deep cross-modal fusion of the image and text to generate enhanced text cue vectors and multi-scale spatial image features. These features are input into the multimodal cue encoder together with other cue information and further fused into sparse and dense cue embedding vectors. Finally, the mask decoder outputs the accurate target object segmentation mask.

[0040] The result analysis module is responsible for visually displaying and quantitatively evaluating the generated segmentation results, and outputting corresponding analysis indicators to help users intuitively understand the segmentation performance and effect of the system.

[0041] The beneficial effects of the present invention are:

[0042] The proposed method and system for remote sensing image referential segmentation based on multimodal prompts, through an early text prompt enhancement strategy, introduces a visual language interactor in each feature extraction stage of the SAM image encoder to achieve multi-level dynamic fusion of image features and text features, thereby enhancing text prompts. In addition, since remote sensing images are more complex, the present invention also divides text prompts into a more fine-grained manner, extracts ground object prompts and spatial position prompts, and can improve the model's recognition and positioning capabilities for target objects during the text enhancement process. The enhanced text prompts will be input into the multimodal prompt encoder designed in this article. The encoder is based on the SAM original prompt encoder and can simultaneously receive text prompts and other set prompts, and ultimately maps all prompts into sparse and dense prompt embedding vectors, which are input into the SAM mask decoder to obtain the segmentation results of the target object, thereby improving the automation level and practical value of remote sensing image analysis and improving the efficiency of referential segmentation.

[0043] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0045] Figure 1 The overall structure of the model based on fine-grained visual language interactor;

[0046] Figure 2 Enhanced module structure for prompts;

[0047] Figure 3 Prompt encoder structure for multimodality;

[0048] Figure 4 It is a visual language prompter structure;

[0049] Figure 5 This is a flowchart of the system processing of the present invention. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0051] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0052] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0053] See also Figures 1 to 5 The embodiment of the present invention provides a remote sensing image reference segmentation method and system based on multimodal prompts, which can realize the reference segmentation task of remote sensing images.

[0054] Example 1:

[0055] The overall method diagram is as follows Figure 1 As shown, the image encoder introduces a multi-scale feature fusion mechanism through the ViT adapter, integrating image representations from different spatial scales. This enables the model to simultaneously capture local details and global semantics, improving its hierarchical understanding of the image's spatial structure. Furthermore, the remote sensing image is processed by the Spatial Prior Module (SPM). All feature maps obtained are flattened and concatenated to obtain multi-scale spatial features, which serve as input to subsequent modules.

[0056] The pre-trained BERT is used as a text encoder to encode referential expressions into text features. Furthermore, the fine-grained visual language interactor converts the original contextual text prompts t c , ground object prompts o and spatial location cues p Together through the text encoder ε t After encoding, it is input into the hint enhancement module in the fine-grained visual language interactor and combined with the updated multi-scale spatial feature F sp Interact to highlight relevant guidance information. Visual features and multi-scale spatial features will also interact with each other at multiple stages of feature extraction, and finally obtain the enhanced text prompt vector L t and multi-scale spatial features F sp .

[0057] The multimodal prompt encoder receives point and box geometric prompts as well as enhanced text prompts. In the encoder, all the information brought by the multimodal prompts will be fused and finally mapped into a sparse prompt embedding vector E sparse and dense hint embedding vector E dense, which is input into the mask decoder as guidance information to complete the segmentation of the target object.

[0058] Specifically, the intermediate features in the i-th stage of the image encoder Before inputting to the next stage, the multi-scale spatial features are Inject into the intermediate features as shown below:

[0059]

[0060] Among them, i∈{1,2,3,4} represents four stages, γ i It is the weight factor used to balance the output and input features of the attention layer at each stage, and the intermediate features Query as attention mechanism, multi-scale spatial features As the key and value, norm(·) refers to the layer normalization function, which will be omitted in the subsequent formulas for simplicity. Will be input to all ViT modules in the current stage of the image encoder to obtain the output features of this stage Then, cross attention is also used to inject the updated intermediate features into the multi-scale spatial features middle:

[0061]

[0062] Among them, multi-scale spatial features serve as queries for the attention mechanism, and intermediate features serve as keys and values. Through the above process, it is possible to achieve fusion updates of visual features and spatial features at different stages and granularities.

[0063] Original contextual text prompt t c , ground object prompts o and spatial location cues p It will first be encoded by the text encoder to obtain the text prompt embedding vector:

[0064] L o =τ(t o ), L p =τ(t p )

[0065] where τ(·) refers to the text encoder, represents the contextual cue embedding vector in the first stage, L o and L p Represent the object hint embedding vector and the position hint embedding vector respectively. Afterwards, all text hint embedding vectors and the updated multi-scale spatial features The prompt enhancement module is fed into the prompt enhancement module to enhance text prompts of different granularities. The prompt enhancement module combines a fine-grained image-text alignment mechanism based on cross-modal feature fusion. Its structure is as follows: Figure 2 As shown, in stage i, the input The contextual hints are injected as keys and values ​​into the embedding vector via the cross-attention mechanism:

[0066]

[0067] in, Represents the contextual text hint feature after combining with the visual feature, Tanh(·) represents the gating mechanism, which includes linear mapping, ReLU activation function, linear mapping and Tanh activation function, and Attn(·) represents self-attention. At the same time, the object hint embedding vector and the position hint embedding vector are also combined with Fusion, as shown below:

[0068]

[0069]

[0070] Among them, L o Represents the object hint feature combined with the visual feature, L p represents the spatial position hint feature after combining with the visual feature, AvgPool(·) and Sigmoid(·) represent the average pooling function and the Sigmoid activation function respectively. After that, the enhanced text hint embedding vector can be obtained:

[0071]

[0072] In addition to enhancing textual hints, the hint enhancement module also retains updates to multi-scale spatial features, with the goal of enhancing key information in visual features based on the current textual information:

[0073]

[0074] in, As query input, As key and value input, It is the feature output to the next stage, and MLP is a multi-layer perceptron.

[0075] Since the SAM image encoder is divided into four stages, there are four corresponding fine-grained vision-language interactors. In the last interactor module, It will be re-split into the feature pyramid {f2,f3,f4}, and then the dimensions of all features will be converted to D through linear mapping m, and then aggregated into the final multi-scale spatial feature F sp , where D m =256 is the feature dimension in the SAM mask decoder. In addition, f3 is the visual embedding vector F that is finally output by the SAM encoder. v The spatial dimensions are the same, so this solution uses f3 to F v To update:

[0076] F v =F v +f3

[0077] Similarly, the enhanced The final text prompt embedding vector L will also be obtained through linear mapping t :

[0078]

[0079] Among them, N t Represents the length of the text prompt sequence, Linear represents the linear layer, and GELU represents the GELU activation function.

[0080] In order to effectively integrate geometric cues and text cues, this embodiment proposes a multimodal cue encoder that can simultaneously receive geometric cues such as points and boxes and enhanced text cues, and aggregate them into the final cue embedding vector. Figure 3 As shown in Figure 3, the multimodal cue encoder retains the processing of geometric cues in the original SAM cue encoder, and introduces a visual language cue to process the enhanced text cues.

[0081] The multimodal cue encoder maps all cues into two vectors: sparse cue embedding vectors and dense cue embedding vectors. Among them, sparse cues include points and boxes, which usually have low spatial density, and each cue corresponds to a specific position or area in the image. Dense cues mainly refer to coarse segmentation masks, which contain complete spatial structure information and can directly correspond to the pixel level of the image. Since the enhanced text cues have target information of different granularities, the visual language prompter processes them to obtain the corresponding sparse cue embedding vectors and coarse masks. The coarse masks are output by the mask embedding module to output the dense cue embedding vectors. Geometric cues such as points and boxes are input into the corresponding point cue embedding module and box cue embedding module to obtain the corresponding sparse cue embedding vectors. The structure of the visual language prompter is as follows Figure 4 As shown, it also uses the enhanced text prompt L t and the updated multi-scale spatial feature F sp , and introduced learnable hint primitives Contains 1 dense marker T d and N sSparsely labeled T s , the primitive is input into two Transformer layers. Each Transformer layer gradually performs self-attention, cue-to-image cross-attention, and cue-to-text cross-attention to aggregate visual-linguistic information, as shown in the following formula:

[0082] T pv =CrossAttn(Attn(T p ),F sp )

[0083]

[0084] Among them, T pv It refers to the prompt primitive after aggregating multi-scale spatial visual information, T p It is a hint primitive that fuses visual and textual information after being updated by Transformer.

[0085] Then, after two Transformer layers, T p Will be split into two parts and The initial sparse embedding vector F sparse By T s After a linear layer, it is generated. In addition, since the multimodal cue encoder retains the ability to process geometric cues such as points and boxes in the SAM cue encoder, the generated sparse embedding vectors will be spliced ​​together to form the final vector:

[0086]

[0087] Among them, N e Depends on the number of point and box hints. If there are no geometric hints, then N e = 0. For dense cue embedding vectors, this embodiment generates a rough segmentation mask based on visual features and dense cue primitives, and then uses the mask embedding module in SAM to map it to a dense cue embedding vector. Specifically, the obtained multi-scale spatial features are first split to obtain f2, and then f2 is upsampled twice using bilinear interpolation, and finally combined with T d Perform matrix multiplication to obtain the rough mask m c :

[0088]

[0089] Among them, Upsample2(·) represents the double upsampling function, is matrix multiplication, (·) T represents the matrix transpose operation. After that, m cIt will be input into the original mask hint embedding module of SAM to obtain the dense hint embedding vector E dense In summary, we can obtain sparse hint embedding vectors and dense hint embedding vectors to guide the model to complete the segmentation of the target object.

[0090] Specific implementation details:

[0091] The entire network architecture and training process were implemented in PyTorch. The experimental environment is as follows: an NVIDIA RTX 3090 GPU. Parameter optimization uses the AdamW optimizer; the optimizer's weight decay value is 0.01; the learning rate is set to 0.00002; and the batch size is set to 8.

[0092] Comparative experiment:

[0093] The proposed method is compared with traditional and specialized remote sensing image referent segmentation methods, including LSTM-CNN (traditional), ConvLSTM (traditional), CMSA (traditional), BRINet (traditional), LAVT (traditional), CrossVLT (traditional), LGCE (specialized), RSMIN (specialized), and FIANet (specialized). The effectiveness of the proposed method on the remote sensing image referent segmentation task is verified by comparison with these methods on the RefSegRS dataset and the RRSIS-D dataset.

[0094] To verify the segmentation performance of different methods, three different metrics were used in the experimental results: overall intersection over union (oIoU), mean intersection over union (mIoU), and the precision threshold metric Pr@X, where X represents the specific precision threshold, to more comprehensively evaluate segmentation quality.

[0095] The experimental results are shown in Tables 1 and 2. Table 1 shows the test evaluation metrics of the proposed method compared with other methods on the RefSegRS dataset. The experimental results show that the proposed method achieves 80.85% and 70.41% in terms of overall intersection-over-union (oIoU) and mean intersection-over-union (mIoU) respectively, achieving the best segmentation results to date.

[0096] On the RRSIS-D dataset, the present invention achieved an overall intersection-over-union (IoU) of 77.56% and an average IoU of 64.49%, which is an improvement of 0.65% and 0.48% compared to the current most advanced method FIANet, becoming the best segmentation result at present.

[0097] Table 1 Experimental results of the methods on the RefSegRS dataset

[0098]

[0099] Table 2 Experimental results of the method on the RRSIS-D dataset

[0100]

[0101]

[0102] In summary, the early text prompt enhancement strategy is adopted in the present application, and the visual language interactor is used to interactively fuse the image features and text features in each feature extraction stage of the image encoder of SAM, so as to enhance the text prompt. In addition, since the remote sensing image is more complex, the present application further divides the text prompt into a more fine-grained granularity, extracts the ground object prompt and the spatial position prompt, and can improve the recognition and positioning ability of the model to the target object in the text enhancement process. The enhanced text prompt is input into the multi-modal prompt encoder designed in the present application, which is based on the original prompt encoder of SAM and can receive the text prompt and other set prompts at the same time, and finally maps all the prompts into sparse and dense embedding vectors, which are input into the mask decoder to obtain the segmentation result of the target object.

[0103] Embodiment 2:

[0104] Figure 5 The structure block diagram of the remote sensing image reference segmentation system based on multi-modal prompts, which comprises a user interaction module, a reference segmentation processing module and a result analysis module; the reference segmentation processing module comprises an image encoder, a spatial prior module, a text encoder, a fine-grained visual language interactor, a multi-modal prompt encoder and a mask decoder.

[0105] The user interaction module receives user input through a local graphical interface, supports uploading remote sensing images in JPEG / PNG format, natural language text prompts and geometric annotation information (such as points or boxes); these inputs are used as prompt information for the reference segmentation task and are transmitted to the reference segmentation processing module for processing.

[0106] The reference segmentation processing module performs parallel multi-modal feature extraction and cross-modal interaction processing on a local GPU; image features and text features are extracted through the image encoder and the text encoder respectively; then, the fine-grained visual language interactor performs deep cross-modal fusion on the image and the text to generate enhanced text prompt vectors and multi-scale spatial image features; these features and other prompt information are input into the multi-modal prompt encoder to further fuse into sparse and dense prompt embedding vectors; finally, the mask decoder outputs the precise target object segmentation mask.

[0107] The result analysis module is responsible for visualizing and quantitatively evaluating the generated segmentation results, and outputs corresponding analysis indicators to assist users in intuitively understanding the segmentation performance and effect of the system.

[0108] Embodiment 3:

[0109] Based on the intelligent computing device, the system of embodiment 2 is embedded in the computing device to realize accurate segmentation of specified ground objects in remote sensing images based on natural language, improve interpretation efficiency, and enrich the interactive means of remote sensing image interpretation.

[0110] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. A remote sensing image referent segmentation method based on multimodal cues, characterized in that: The method comprises the following steps: S1: Data preprocessing: Divide remote sensing images into segmentation datasets, including training sets, validation sets, and test sets; S2: Build a remote sensing image referent segmentation model based on multimodal cues, including an image encoder, a spatial prior module, a text encoder, a fine-grained visual language interactor, and a multimodal cue encoder; S3: Training a remote sensing image referent segmentation model based on multimodal cues: For the training of the remote sensing image referent segmentation model, a loss function combining Focal loss, cross entropy loss, and Dice loss is used to optimize the model training; S4: Output segmentation results: Input the remote sensing image to be segmented into the trained model, and then input the model output results into the mask decoder to complete the referential segmentation of the target object.

2. The remote sensing image reference segmentation method based on multimodal prompts according to claim 1, characterized in that: In step S2, the image encoder introduces a multi-scale feature fusion mechanism through ViT to enhance the hierarchical extraction of image spatial information. Specifically: the input remote sensing image The image encoder after the general image segmentation model SAM is converted into a visual embedding vector F v : F v =e v (x) Among them, ε v (·) represents the image encoder, and H, W, and C are the height, width, and number of channels of the remote sensing image, respectively.

3. The remote sensing image reference segmentation method based on multimodal prompting according to claim 1, characterized in that: In step S2, the processing flow of the spatial prior module is as follows: the remote sensing image x first undergoes a downsampling phase consisting of three convolutional layers and a pooling layer to obtain a feature map After that, f1 is input into several convolutional layers with kernel size 3 and stride 2, which doubles the number of channels while reducing the size of the feature map. Finally, a 1×1 convolution is applied to map the number of channels of all feature maps to a fixed D dimension, i.e. f1,f2,f3,f4=SPM(x) Among them, SPM represents the spatial prior module; a feature pyramid {f2,f3,f4} is obtained, in which the feature map as well as These feature maps are flattened and concatenated to obtain multi-scale spatial features F sp : Here, dim=0 means concatenation in the first dimension, and Concat means concatenation.

4. The remote sensing image reference segmentation method based on multimodal prompting according to claim 1, characterized in that: In step S2, the text encoder is used to encode the referential expression into text features, specifically: for the input referential expression T = {t1, t2, ..., t n } will be converted into a text embedding vector F through the text encoder text : F text =E text (T) Among them, t n represents the nth expression, E text (·) represents the text encoder, and the Transformer model represented by the pre-trained bidirectional encoder is used as the text encoder.

5. The remote sensing image reference segmentation method based on multimodal prompting according to claim 1, characterized in that: In step S2, the processing flow of the fine-grained visual language interactor is as follows: the original contextual text prompt t c , ground object prompts o and spatial location cues p Together through the text encoder ε t After encoding, it is input into the hint enhancement module in the fine-grained visual language interactor and combined with the updated multi-scale spatial feature F sp Interact; visual features and multi-scale spatial features will also interact at multiple stages of feature extraction, and finally obtain the enhanced text prompt vector L t and multi-scale spatial features F sp , as shown below: L t ,F sp =I(F sp ,he t (t c ),e t (t o ),e t (t p )) Here, Ι(·) represents the fine-grained vision-language interactor module.

6. The remote sensing image reference segmentation method based on multimodal prompts according to claim 5, characterized in that: In step S2, the processing flow of the multimodal prompt encoder is as follows: by receiving point and box geometric prompts, and also receiving enhanced text prompts, in the encoder, all the information brought by the multimodal prompts will be fused and finally mapped into a sparse prompt embedding vector E sparse and dense hint embedding vector E dense , is input into the mask decoder as guidance information. The whole process is shown in the following formula: E sparse ,E dense =Ρ(L t ,points,boxes) Here, P(·) represents the multimodal prompt encoder, points and boxes represent the input point and box prompts, respectively. When these types of prompts do not exist, the multimodal prompt encoder does not process them, which is consistent with the prompt encoder in SAM.

7. The remote sensing image reference segmentation method based on multimodal prompting according to claim 1, characterized in that: In step S3, the calculation formula of the loss function is: Among them, ω is the adjustment factor that balances the traditional cross entropy loss and Focal loss, D(·) represents the downsampling operation, m is the 0-1 prediction mask obtained by the prediction mask probability matrix p, CE(·) is the cross entropy loss, Focal(·) is the Focal loss, Dice(·) is the Dice loss, m is the segmentation mask, λ1 is the weight controlling the cross entropy loss and Focal loss, and λ2 is the weight controlling the Dice loss.

8. The remote sensing image reference segmentation method based on multimodal prompts according to claim 5, characterized in that: In step S4, the processing flow of the mask decoder is: sparse hint embedding vector E sparse and dense hint embedding vector E dense Will be along with the visual embedding vector F v It is input into the mask decoder to complete the segmentation of the target object: m=D m (F v ,E spares ,E dense ) Among them, D m (·) is the mask decoder, and m is the segmentation mask of the target object.

9. A remote sensing image reference segmentation system based on multimodal cues, characterized in that: The system includes a user interaction module, a reference segmentation processing module and a result analysis module; the reference segmentation processing module includes an image encoder, a spatial prior module, a text encoder, a fine-grained visual language interactor, a multimodal prompt encoder and a mask decoder; The user interaction module receives user input through a local graphical interface and supports uploading remote sensing images in JPEG / PNG format, natural language text prompts, and geometric annotation information; These inputs are used as prompt information for the referent segmentation task and are passed to the referent segmentation processing module for processing; The referent segmentation processing module performs multimodal feature extraction and cross-modal interaction processing in parallel on the local GPU. Image features and text features are extracted through the image encoder and text encoder respectively. Subsequently, the fine-grained visual language interactor performs deep cross-modal fusion of images and text to generate enhanced text hint vectors and multi-scale spatial image features. These features are fed into a multimodal cue encoder along with other cue information and further fused into sparse and dense cue embedding vectors. Finally, a mask decoder outputs an accurate target object segmentation mask. The result analysis module is responsible for visually displaying and quantitatively evaluating the generated segmentation results, and outputting corresponding analysis indicators to help users intuitively understand the segmentation performance and effect of the system.

Images