Remote sensing image text retrieval method and device based on mamba adapter and shared hint enhancement

Abstract

This invention discloses a method and apparatus for remote sensing image text retrieval based on Mamba adapters and shared cues enhancement. It includes a CLIP enhancement network based on multimodal adapters and shared cues. By introducing a mode-specific adapter based on a Mamba structure into the CLIP backbone model, the modeling ability for long-range spatial features of remote sensing images and semantic associations of text is enhanced, thereby improving the ability to focus on the main subject of the remote sensing image. Simultaneously, a dynamic cue fusion module is proposed. Based on pre-trained CLIP, initial cue feature vectors of images and text are extracted, projected to generate mode-specific dynamic cue. Combined with a learnable shared cue matrix, a composite cue vector fusing intramodal features and cross-modal interaction information is constructed and injected into the 12-layer Mamba adapter of CLIP, guiding the model to optimize cross-modal alignment. This invention improves the efficiency of remote sensing feature extraction through Mamba adapters and achieves intramodal and extramodal information synergy through dynamic cues, significantly improving the accuracy and generalization of remote sensing image-text retrieval.

Description

Technical Field

[0001] This invention relates to a method and apparatus for text retrieval of remote sensing images based on Mamba adapter and shared hint enhancement, belonging to the field of text retrieval technology for remote sensing images. Background Technology

[0002] With the vigorous development of space exploration in China and globally, remote sensing technology and data have been widely applied, becoming a key tool in various fields such as agricultural management, environmental monitoring, and urban planning. Remote sensing images, as the primary means of acquiring information about the Earth's surface, are experiencing unprecedented growth in data volume. However, how to quickly and accurately retrieve this massive amount of remote sensing image data has become an urgent problem to solve. Currently, image-text retrieval technology has shown great potential and effectiveness in addressing this challenge. It can understand image and text content, perform image-text matching, and greatly promote the analysis and application of image data.

[0003] With the rapid development of visual language models, transfer learning based on large-scale pre-trained models has become the mainstream paradigm for remote sensing image text retrieval. These models first extract cross-modal semantic features through pre-training on large-scale natural image-text pairs, and then fine-tune the model using image-text pairing data from the remote sensing domain, such as satellite imagery with geographic tags, disaster monitoring reports with radar images, etc. Because the remote sensing domain possesses abundant image-text pairing data, this method has achieved considerable success in the field of remote sensing image text retrieval.

[0004] However, existing methods still face three key challenges: 1. Visual language models typically contain billions of parameters (e.g., CLIP-ViT-L / 14 has 350 million parameters), and fine-tuning all parameters requires extremely high computational resources and time costs, limiting their lightweight deployment in remote sensing scenarios; 2. Remote sensing images have significant regional focusing characteristics (e.g., farmland monitoring requires attention to specific crop areas, and geological disaster detection requires locating landslide areas), but existing models are easily affected by complex backgrounds (e.g., cloud cover, terrain interference), making it difficult to effectively extract key ground features; 3. Traditional visual language models use independent encoders to process images and text (e.g., CLIP's dual-tower structure), lacking cross-modal interaction mechanisms. In remote sensing tasks, images and text often have strong spatial correlations (e.g., "pollution in the northeast river" requires text to guide image region attention), and independent encoding may lose fine-grained semantic alignment information. Summary of the Invention

[0005] Purpose of the invention: To overcome the shortcomings of existing technologies, this invention provides a method and apparatus for text retrieval of remote sensing images based on Mamba adapters and shared cue enhancement. It comprehensively utilizes adapter and cue technologies to reduce the required training parameters, while using the Mamba adapter structure to enhance the extraction of key features. At the same time, it uses cue with shared content to compensate for the lack of cross-modal interaction in the model, thereby enhancing the accuracy of text retrieval of remote sensing images with as few parameters as possible.

[0006] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0007] A method for text retrieval of remote sensing images based on Mamba adapter and shared cue enhancement includes the following steps:

[0008] Step 1: Obtain the original remote sensing image and its corresponding text description (RS-Text). Crop and standardize the original remote sensing image to obtain a remote sensing image (RS-Image) of a set size.

[0009] Step 2: Divide the remote sensing images and corresponding text descriptions into training and testing sets according to a set ratio. The training set includes remote sensing images I... train and corresponding text description T train The test set includes remote sensing images I test and corresponding text description T test ;

[0010] Step 3: Construct a CLIP-enhanced network based on multimodal adapters and shared cues, including a cue generation network and a feature extraction network. The cue generation network includes a cue vector extraction module and a cue generation module. The cue vector extraction module includes an image cue vector extractor and a text cue vector extractor. The feature extraction network includes a feature extraction module and an adapter module. The adapter module includes a set of image adapters and a set of text adapters. The feature extraction module is designed based on the CLIP model, which includes 12 Transformer Block layers. An image adapter is embedded at the image end of each Transformer Block layer, and an image adapter is embedded at the image end of each Transformer Block layer. A text adapter is embedded at the text end of the Block layer; the CLIP model parameters in the cue vector extraction module and the feature extraction module are replaced with pre-trained CLIP-B / 32 parameters; the image adapter includes a dimensionality reduction linear layer, a data preprocessing layer, two TreeMamba deep feature extraction layers, two residual connections, a gated connection, and an up-dimensional linear layer; the text adapter includes a dimensionality reduction linear layer, two Hydra deep feature extraction layers, a data preprocessing layer, two residual connections, a gated connection, and an up-dimensional linear layer;

[0011] Will I train and T train Input into the prompt generation network, I train Image cue vector features are generated by the image cue vector extractor. p_img T train Text prompt vector feature generated by the text prompt vector extractor p_text , will feature p_img and feature p_text As input to the prompt generation module;

[0012] feature p_img and feature p_text The input prompt generation module generates the final image prompt vector. and the final text prompt vector j = 1, 2, ..., 12;

[0013] Will I train and T train As respectively and The input feature extraction network consists of 12 Transformer Blocks; the image input to the j-th Transformer Block is denoted as... Image output is denoted as Will and Input image adapter, generate image optimization features The text input of the j-th layer TransformerBlock is denoted as... Text output is denoted as Will and Input text adapter, generate text optimization features

[0014] Will The cls_token is used to tag the input linear projection layer and output the image feature vector F after dimensionality reduction. img ,Will The [EOS] token is input to a linear projection layer for dimensionality reduction, and the output is a text feature vector F. text ;

[0015] Step 4: Calculate F using a weighted combination of symmetric cross-modal contrast loss and membership loss. img and F text The loss value;

[0016] Step 5: Freeze the parameters of the prompt vector extraction module and the feature extraction module, and repeat Step 3 to Step 4 until the loss value converges, thus completing the training of the prompt generation module and the adapter module;

[0017] Step 6: Place I test and T test Input prompt generation network, output image final prompt vector and the final text prompt vector Will I test T test , and Input the feature extraction network and output F img and F text ;

[0018] Step 7: Based on F img and F text Perform bidirectional cross-modal retrieval, calculate retrieval performance metrics, and output them.

[0019] Specifically, in the cue vector extraction module, the image cue vector extractor includes a PatchEmbedding layer, a Position Embedding layer, and a learnable embedding vector P. class ', 12 Transformer Encoder layers in sequential concatenation and one linear projection layer; I trainThe input image prompts the vector extractor, which segments and flattens it into a vector sequence P through the PatchEmbedding layer. img ', for P img 'and P class 'Concatenate the components, then output them through the PositionEmbedding layer.' The input to the j-th Transformer Encoder layer is The output of the j-th TransformerEncoder layer is Will The input is a linear projection layer that transforms dimensions, and the output is a feature. p_i ;

[0020] The prompt vector extraction module includes a text prompt vector extractor comprising a tokenizer, a token embedding layer, a position embedding layer, 12 sequentially concatenated Transformer Encoder layers, and a linear projection layer; [The text then repeats the structure of the prompt vector extraction module, which is not translated here.] train The input text prompts a vector extractor, which then segments the text and outputs T. token ', output P after TokenEmbedding layer text ', output through the Position Embedding layer The input to the j-th Transformer Encoder layer is The output of the j-th Transformer Encoder layer is Will The input is a linear projection layer that transforms dimensions, and the output is a feature. p_t .

[0021] Specifically, the prompt generation module includes an image linear layer, a text linear layer, and a shared prompt matrix; it will generate features... p_i and feature p_t Input suggestion generation module, feature p_i Feature generation via image linear layer p_i ', feature p_t Feature generation via text linear layer p_t ', feature p_i 'Concatenate with the shared cue matrix to output the final cue vector of the image feature p_t 'Concatenate with the shared cue matrix to output the final text cue vector'

[0022] Specifically, in the prompt generation module, the dimensions of the image linear layer are [512, 128], the dimensions of the text linear layer are [512, 128], and the dimensions of the shared prompt matrix are [12, 5, 128].

[0023] Specifically, in the feature extraction module, the image end of the Transformer Block layer is called the image feature extractor, and the text end of the Transformer Block layer is called the text feature extractor.

[0024] The image feature extractor includes a Patch Embedding layer, a Position Embedding layer, and a learnable embedding vector P. class A series of 12 Transformer Encoder layers and one linear projection layer; T train The input image feature extractor is segmented and flattened into a vector sequence P by the Patch Embedding layer. img , for P img and P class The components are concatenated and then output through the Position Embedding layer. The input to the j-th Transformer Encoder layer is The output of the j-th Transformer Encoder layer is Will and Input image adapter, generate image optimization features Will The cls_token is input to the linear projection layer to transform dimensions, and the output is the image feature vector F. img ;

[0025] The text feature extractor includes a token segmenter, a token embedding layer, a position embedding layer, 12 Transformer Encoder layers, and a linear projection layer; it will extract the T... train The input is fed into the text feature extractor, and after word segmentation, the output T is generated. token P is output through the Token Embedding layer. text Output via Position Embedding layer The input to the j-th Transformer Encoder layer is The output of the j-th Transformer Encoder layer is Will and Input text adapter, generate text optimization features Will The [EOS] tokens are input into a linear projection layer, where the dimensions are transformed, and the output is a text feature vector F. text .

[0026] Specifically, in the adapter module, the image adapter includes a dimension reduction linear layer, a data preprocessing layer, two TreeMamba deep feature extraction layers, two residual connections, a gated connection, and a dimension increase linear layer. The data preprocessing layer converts the input from a length dimension to two dimensions: length and width.

[0027] Will and Input image adapter, Output after dimensionality reduction linear layer and After concatenation, the output is activated by the GELU function. Output after dimensional transformation by the data preprocessing layer The output is sequentially processed through two TreeMamba deep feature extraction layers. right and Connect and output the gating mechanism. right and Perform residual connection and output Remove China belongs to The token was obtained Output after dimensionality-upgrading linear layer right and Perform residual connection and output

[0028] Specifically, in the adapter module, the text adapter includes a dimension-reducing linear layer, two Hydra deep feature extraction layers, a data preprocessing layer, two residual connections, a gated connection, and an increasing-dimensional linear layer;

[0029] Will and Input text adapter, Output after dimensionality reduction linear layer and After concatenation, the output is activated by the GELU function. The output is sequentially processed through two Hydra deep feature extraction layers. right and Connect and output the gating mechanism. right and Perform residual connection and output Remove China belongs to The token was obtained Output after dimensionality-upgrading linear layer right, and Perform residual connection and output

[0030] Specifically, in the image adapter, the dimension of the dimensionality reduction linear layer is [768, 128], the acceptance dimension of the TreeMamba deep feature extraction layer is 128, the SSM spatial feature dimension is 1, and the dimension of the dimensionality increase linear layer is [128, 768]; in the text adapter, the dimension of the dimensionality reduction linear layer is [512, 128], the acceptance dimension of the Hydra deep feature extraction layer is 128, the SSM spatial feature dimension is 16, and the dimension of the dimensionality increase linear layer is [128, 512].

[0031] A remote sensing image text retrieval device based on Mamba adapter and shared cue enhancement includes a remote sensing image and text acquisition unit, a CLIP enhancement network based on multimodal adapter and shared cue, a training unit and a testing unit;

[0032] The remote sensing image and text acquisition unit is used to acquire the original remote sensing image and the corresponding text description, and to crop and standardize the original remote sensing image to obtain a remote sensing image of a set size.

[0033] The CLIP-based enhanced network with multimodal adapter and shared cues includes a cue generation network and a feature extraction network. The cue generation network includes a cue vector extraction module and a cue generation module. The cue vector extraction module extracts image pre-cue vectors and text pre-cue vectors from the remote sensing image and corresponding text description, respectively. These vectors are then processed by the cue generation module to generate the final image cue vector and the final text cue vector. The feature extraction network includes a feature extraction module and an adapter module. The feature extraction module and the adapter module work together. The feature extraction module extracts shallow features from the remote sensing image and corresponding text description, while the adapter module extracts deep features from the remote sensing image and corresponding text description, ultimately generating feature vectors for the remote sensing image and corresponding text description.

[0034] The training unit uses a loss function to perform supervised training on the CLIP augmentation network based on multimodal adapter and shared cueing;

[0035] The test unit uses a trained CLIP-based augmented network with multimodal adapter and shared cues to perform image-to-text retrieval and text-to-image retrieval on pairs of remote sensing images and corresponding text descriptions.

[0036] Specifically, the CLIP-based enhanced network with multimodal adapters and shared cues includes a cue generation network and a feature extraction network. The cue generation network includes a cue vector extraction module and a cue generation module. The cue vector extraction module includes an image cue vector extractor and a text cue vector extractor. The feature extraction network includes a feature extraction module and an adapter module. The adapter module includes a set of image adapters and a set of text adapters. The feature extraction module is designed based on the CLIP model, which includes 12 Transformer Blocks. An image adapter is embedded at the image end of each Transformer Block, and a text adapter is embedded at the text end of each Transformer Block. The CLIP model parameters in the cue vector extraction module and the feature extraction module are replaced with pre-trained CLIP-B / 32 parameters.

[0037] Specifically, in the CLIP-enhanced network based on multimodal adapters and shared cues:

[0038] Will I train and T train Input into the prompt generation network, I train Image cue vector features are generated by the image cue vector extractor. p_img T train Text prompt vector feature generated by the text prompt vector extractor p_text , will feature p_img and feature p_text As input to the prompt generation module;

[0039] feature p_img and feature p_text The input prompt generation module generates the final image prompt vector. and the final text prompt vector j = 1, 2, ..., 12;

[0040] Will I train and T train As respectively and The input feature extraction network consists of 12 Transformer Blocks; the image input to the j-th Transformer Block is denoted as... Image output is denoted as Will and Input image adapter, generate image optimization features The text input of the j-th layer TransformerBlock is denoted as... Text output is denoted as Will and Input text adapter, generate text optimization features

[0041] Will The cls_token is used to tag the input linear projection layer and output the image feature vector F after dimensionality reduction. img ,Will The [EOS] token is input to a linear projection layer for dimensionality reduction, and the output is a text feature vector F. text .

[0042] Beneficial effects: The remote sensing image text retrieval method and apparatus based on Mamba adapter and shared cue enhancement provided by this invention establishes a joint retrieval model by introducing a Mamba-based adapter and cue with a sharing mechanism into CLIP, and performs image text retrieval on remote sensing images. While greatly reducing the number of trainable parameters, it enables the model to extract the main content features of the image more effectively and is more sensitive to the association between the image and the text, resulting in higher accuracy for remote sensing image text retrieval. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the frame structure of the device of the present invention;

[0044] Figure 2 This is a flowchart illustrating the implementation of the method of the present invention;

[0045] Figure 3 A pair of remote sensing images and corresponding text descriptions;

[0046] Figure 4 The image shows the prediction performance of the CLIP augmentation network based on a multimodal adapter and shared prompts after training. The text in the rightmost box represents the correctly retrieved text. Detailed Implementation

[0047] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1 The image shown is a remote sensing image text retrieval device based on Mamba adapter and shared cue enhancement, including a remote sensing image and text acquisition unit, a CLIP enhancement network based on multimodal adapter and shared cue, a training unit and a testing unit.

[0049] The remote sensing image and text acquisition unit is used to acquire the original remote sensing image and the corresponding text description, and to crop and standardize the original remote sensing image to obtain a remote sensing image of a set size.

[0050] The CLIP-based enhanced network with multimodal adapter and shared cues includes a cue generation network and a feature extraction network. The cue generation network includes a cue vector extraction module and a cue generation module. The cue vector extraction module extracts image pre-cue vectors and text pre-cue vectors from the remote sensing image and corresponding text description, respectively. These vectors are then processed by the cue generation module to generate the final image cue vector and the final text cue vector. The feature extraction network includes a feature extraction module and an adapter module. The feature extraction module and the adapter module work together. The feature extraction module extracts shallow features from the remote sensing image and corresponding text description, while the adapter module extracts deep features from the remote sensing image and corresponding text description, ultimately generating feature vectors for the remote sensing image and corresponding text description.

[0051] The training unit uses a loss function to supervise the training of the CLIP-based augmentation network with multimodal adapter and shared cueing.

[0052] The test unit uses a trained CLIP-based augmented network with multimodal adapter and shared cues to perform image-to-text retrieval and text-to-image retrieval on pairs of remote sensing images and corresponding text descriptions.

[0053] use Figure 1 The illustrated remote sensing image text retrieval device based on Mamba adapter and shared hint enhancement performs multi-source remote sensing image text retrieval as follows: Figure 2 As shown below, the specific steps will be explained in detail.

[0054] Step 1: Obtain remote sensing images and corresponding text descriptions

[0055] The process involves acquiring the original remote sensing image and its corresponding text description (RS-Text), cropping and standardizing the original image to obtain a remote sensing image (RS-Image) of the same size for easier subsequent processing. For example... Figure 3 The image shows the acquired pairs of remote sensing images and their corresponding text descriptions.

[0056] Step 2: Divide the training set and the test set

[0057] The remote sensing images and their corresponding text descriptions were divided into a training set and a test set in a 7:3 ratio, whereby the training set included remote sensing images I. train and corresponding text description T train The test set includes remote sensing images I test and corresponding text description T test.

[0058] Step 3: Construct a CLIP-enhanced network based on multimodal adapters and shared hints

[0059] The CLIP-enhanced network based on multimodal adapter and shared cues includes a cue generation network and a feature extraction network.

[0060] 3.1 Suggestions for generating network structure

[0061] The prompt generation network includes a prompt vector extraction module and a prompt generation module.

[0062] 3.1.1 Structure of the Hint Vector Extraction Module

[0063] The cue vector extraction module includes an image cue vector extractor and a text cue vector extractor.

[0064] The image cue vector extractor includes a Patch Embedding layer, a Position Embedding layer, and a learnable embedding vector P. class ', 12 Transformer Encoder layers in sequential concatenation and one linear projection layer; I train The input image prompts the vector extractor, which segments and flattens it into a vector sequence P after passing through the Patch Embedding layer. img ', for P img 'and P class 'Concatenate the components, then output them through the Position Embedding layer.' The input to the j-th Transformer Encoder layer is The output of the j-th Transformer Encoder layer is Will The input is a linear projection layer that transforms dimensions, and the output is a feature. p_i .

[0065] The text prompt vector extractor includes a token segmenter, a token embedding layer, a position embedding layer, 12 sequentially concatenated Transformer Encoder layers, and a linear projection layer; it extracts the T... train The input text prompts a vector extractor, which then segments the text and outputs T. token ', output P after Token Embedding layer text ', output through the Position Embedding layer The input to the j-th Transformer Encoder layer is The output of the j-th Transformer Encoder layer is Will The input is a linear projection layer that transforms dimensions, and the output is a feature. p_t .

[0066] 3.1.2 Suggestion for the structure of the generation module

[0067] The prompt generation module includes an image linear layer, a text linear layer, and a shared prompt matrix; it will generate features. p_i and feature p_t Input suggestion generation module, feature p_i Feature generation via image linear layer p_i ', feature p_t Feature generation via text linear layer p_t ', feature p_i 'Concatenate with the shared cue matrix to output the final cue vector of the image feature p_t 'Concatenate with the shared cue matrix to output the final text cue vector'

[0068] In the prompt generation module, the dimensions of the image linear layer are [512, 128], the dimensions of the text linear layer are [512, 128], and the dimensions of the shared prompt matrix are [12, 5, 128].

[0069] 3.2 Structure of Feature Extraction Network

[0070] The feature extraction network includes a feature extraction module and an adapter module. The adapter module includes a set of image adapters and a set of text adapters. The feature extraction module is designed based on the CLIP model, which includes 12 Transformer Blocks. An image adapter is embedded at the image end of each Transformer Block layer, and a text adapter is embedded at the text end of each Transformer Block layer.

[0071] 3.2.1 Structure of the Feature Extraction Module

[0072] In the feature extraction module, the image end of the Transformer Block layer is called the image feature extractor, and the text end of the Transformer Block layer is called the text feature extractor.

[0073] The image feature extractor includes a Patch Embedding layer, a Position Embedding layer, and a learnable embedding vector P. class A series of 12 Transformer Encoder layers and one linear projection layer; T train The input image feature extractor is segmented and flattened into a vector sequence P by the Patch Embedding layer. img , for P img and P class The components are concatenated and then output through the Position Embedding layer. The input to the j-th Transformer Encoder layer is The output of the j-th Transformer Encoder layer is Will and Input image adapter, generate image optimization features Will The cls_token is input to the linear projection layer to transform dimensions, and the output is the image feature vector F. img .

[0074] The text feature extractor includes a token segmenter, a token embedding layer, a position embedding layer, 12 Transformer Encoder layers, and a linear projection layer; it will extract the T... train The input is fed into the text feature extractor, and after word segmentation, the output T is generated. token P is output through the Token Embedding layer. text Output via Position Embedding layer The input to the j-th Transformer Encoder layer is The output of the j-th Transformer Encoder layer is Will and Input text adapter, generate text optimization features Will The [EOS] tokens are input into a linear projection layer, where the dimensions are transformed, and the output is a text feature vector F. text .

[0075] 3.2.2 Structure of the adapter module

[0076] The adapter module includes a set of image adapters and a set of text adapters.

[0077] The image adapter comprises a dimension-reducing linear layer, a data preprocessing layer, two TreeMamba deep feature extraction layers, two residual connections, a gated connection, and a dimension-upgrading linear layer. The data preprocessing layer transforms the input from a length dimension to two dimensions: length and width. and Input image adapter, Output after dimensionality reduction linear layer and After concatenation, the output is activated by the GELU function. Output after dimensional transformation by the data preprocessing layer The output is sequentially processed through two TreeMamba deep feature extraction layers. right and Connect and output the gating mechanism. right and Perform residual connection and output Remove China belongs to The token was obtained Output after dimensionality-upgrading linear layer right and Perform residual connection and output

[0078] In the image adapter, the dimension of the dimension reduction linear layer is [768, 128], the acceptance dimension of the TreeMamba deep feature extraction layer is 128, the SSM spatial feature dimension is 1, and the dimension of the dimension increase linear layer is [128, 768].

[0079] The text adapter comprises a dimension-reduced linear layer, two Hydra deep feature extraction layers, a data preprocessing layer, two residual connections, a gated connection, and a dimension-upgrading linear layer; and Input text adapter, Output after dimensionality reduction linear layer and After concatenation, the output is activated by the GELU function. The output is sequentially processed through two Hydra deep feature extraction layers. right and Connect and output the gating mechanism. right and Perform residual connection and output Remove China belongs to The token was obtained Output after dimensionality-upgrading linear layer right, and Perform residual connection and output

[0080] In the text adapter, the dimension of the dimensionality reduction linear layer is [512, 128], the acceptance dimension of the Hydra deep feature extraction layer is 128, the SSM spatial feature dimension is 16, and the dimension of the dimensionality increase linear layer is [128, 512].

[0081] 3.3 Execution process of CLIP-enhanced network based on multimodal adapter and shared hints

[0082] Replace the CLIP model parameters in the cue vector extraction module and the feature extraction module with the pre-trained CLIP-B / 32 parameters.

[0083] Will I train and T train Input into the prompt generation network, I train Image cue vector features are generated by the image cue vector extractor. p_img T train Text prompt vector feature generated by the text prompt vector extractor p_text , will feature p_img and feature p_text As input to the prompt generation module.

[0084] feature p_img and feature p_text The input prompt generation module generates the final image prompt vector. and the final text prompt vector j = 1, 2, ..., 12.

[0085] Will I train and T train As respectively and The input feature extraction network consists of 12 Transformer Blocks; the image input to the j-th Transformer Block is denoted as... Image output is denoted as Will and Input image adapter, generate image optimization features The text input of the j-th layer TransformerBlock is denoted as... Text output is denoted as Will and Input text adapter, generate text optimization features

[0086] Will The cls_token is used to tag the input linear projection layer and output the image feature vector F after dimensionality reduction. img ,Will The [EOS] token is input to a linear projection layer for dimensionality reduction, and the output is a text feature vector F. text .

[0087] Step 4: Supervised training using a loss function

[0088] The training unit uses a weighted combination of symmetric cross-modal contrastive loss and membership loss to perform supervised training on the CLIP-based augmented network with multimodal adapter and shared cueing.

[0089] During the supervised training of the CLIP-based augmentation network with multimodal adapter and shared cues, the cue generation network generates final image cue vectors and final text cue vectors respectively. Then, the remote sensing image and its corresponding text description, along with their corresponding final image and text cue vectors, are input into the feature extraction network, which outputs image and text feature vectors. The similarity matrix logits is calculated using the image and text feature vectors. A weighted combination of symmetric cross-modal contrastive loss and membership loss is calculated using logits and labels to obtain the final loss value.

[0090] L = L c +λ cs L a

[0091]

[0092] Where: L represents the final loss value, L a L represents the membership loss value. c λ represents the comparative loss value. cs Indicates the central scale; V i Let t represent the image feature vector of the i-th sample pair. i Let N represent the text feature vector of the i-th sample pair, and N represent the number of sample pairs. The image center represents the category to which the i-th sample belongs. This represents the text center of the category to which the i-th sample belongs.

[0093] If the final loss value L converges, the CLIP-enhanced network training based on the multimodal adapter and shared cueing is complete; otherwise, a new sample pair I is selected. train and T trainThe image and text prompt vectors are generated again by the prompt generation network, and then the image and text feature vectors are extracted by the feature extraction network. The final loss value is calculated on the results until the final loss value converges.

[0094] Step 6: Test using the trained CLIP-based augmentation network with multimodal adapter and shared cueing.

[0095] like Figure 4 As shown, I test and T test Input prompt generation network, output image final prompt vector and the final text prompt vector Will I test T test , and Input the feature extraction network and output the image feature vector F img and text feature vector F text Calculate F img and F text Based on the similarity between them, output the predicted label. Figure 4 The comparison between the prediction results and the labels shows that the present invention can accurately locate the core ground features in remote sensing images and achieve high-accuracy retrieval results in complex multi-target scenarios. Even when faced with dense distribution or partial occlusion of similar ground features, it can still effectively focus on the subject and effectively avoid background noise interference.

[0096] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A method for remote sensing image text retrieval based on Mamba adapter and shared hint enhancement, characterized in that: Comprising the following steps: Step1: Obtain the original remote sensing image and the corresponding text description, crop and standardize the original remote sensing image to obtain a remote sensing image of a certain size; Step 2: divide the remote sensing image and the corresponding text description into a training set and a test set according to a proportion, wherein: the training set includes remote sensing images I train and corresponding text descriptions T train , and the test set includes remote sensing images I test and corresponding text descriptions T test ; Step3: Construct a CLIP enhanced network based on a multi-modal adapter and a shared prompt, including a prompt generation network and a feature extraction network; the prompt generation network includes a prompt vector extraction module and a prompt generation module, and the prompt vector extraction module includes an image prompt vector extractor and a text prompt vector extractor; the feature extraction network includes a feature extraction module and an adapter module, and the adapter module includes a group of image adapters and a group of text adapters, and the feature extraction module is designed based on a CLIP model, and the CLIP model includes 12 layers of TransformerBlock, an image adapter is embedded at the image end of each Transformer Block layer, and a text adapter is embedded at the text end of each Transformer Block layer; the parameters of the CLIP model in the prompt vector extraction module and the feature extraction module are replaced with the pre-trained CLIP-B / 32 parameters; the image adapter includes a dimension reduction linear layer, a data preprocessing layer, two TreeMamba deep feature extraction layers, two residual connections, a gating mechanism connection and a dimension increasing linear layer; the text adapter includes a dimension reduction linear layer, two Hydra deep feature extraction layers, a data preprocessing layer, two residual connections, a gating mechanism connection and a dimension increasing linear layer; I train and T train are input into the prompt generation network, I train image pre-prompt vectors feature p_img are generated by the image prompt vector extractor, T train text pre-prompt vectors feature p_text are generated by the text prompt vector extractor, feature p_img and feature p_text are input into the prompt generation module; feature p_img and feature p_text input prompt generation module, generating an image final prompt vector and a text final prompt vector I train and T train respectively as and input feature extraction network, the feature extraction network includes 12 layers of Transformer Block; the image end input of the jth layer of Transformer Block is denoted as the image end output is denoted as input and input image adapter, generate image optimization features the text end input of the jth layer of Transformer Block is denoted as the text end output is denoted as input and input text adapter, generate text optimization features The cls_token mark of is input into the linear projection layer to reduce dimension and output an image feature vector F img The [EOS] mark of is input into the linear projection layer to reduce dimension and output a text feature vector F text ; Step4: the loss value of F is calculated by weighted combination of symmetric cross-modal contrastive loss and membership loss img and F text 's loss value; Step5: Freeze the parameters of the prompt vector extraction module and the feature extraction module, repeat Step3-Step4 until the loss value converges, and complete the training of the prompt generation module and the adapter module; Step 6: I test and T test Input prompt generation network, output image final prompt vector and text final prompt vector Step 7: I test , T test , and Input feature extraction network, output F img and F text ; Step 7: Based on F img and F text Perform bidirectional cross-modal retrieval, calculate retrieval performance indicators and output.

2. The remote sensing image text retrieval method based on Mamba adapter and shared prompt enhancement according to claim 1, wherein: In the prompt vector extraction module, the image prompt vector extractor includes a Patch Embedding layer, a Position Embedding layer, and a learnable embedding vector P class 12 TransformerEncoder layers in sequence and a linear projection layer; I train The input image prompt vector extractor is divided into a vector sequence P img ' by the Patch Embedding layer img P class ' and P The input of the jth Transformer Encoder layer is The output of the jth Transformer Encoder layer is P The input of the linear projection layer is transformed in dimension, and the output is feature p_i ; In the prompt vector extraction module, the text prompt vector extractor includes a word segmenter, a TokenEmbedding layer, a Position Embedding layer, 12 Transformer Encoder layers connected in sequence, and a linear projection layer; T train The input text prompt vector extractor is segmented by the word segmenter to output T token , output P text by the TokenEmbedding layer, and output The input of the jth Transformer Encoder layer is The output of the jth Transformer Encoder layer is is input into the linear projection layer to change the dimension, and the output is feature p_t .​ 3. The method for remote sensing image text retrieval based on Mamba adapter and shared prompt enhancement according to claim 1, characterized in that: The prompt generation module includes an image linear layer, a text linear layer, and a shared prompt matrix; feature p_i and feature p_t Input the prompt generation module, feature p_i Generate feature p_i ' through the image linear layer, feature p_t Generate feature p_t ' through the text linear layer, feature p_i ' and the shared prompt matrix are spliced to output the image final prompt vector feature p_t ' and the shared prompt matrix are spliced to output the text final prompt vector 4. The method of remote sensing image text retrieval based on Mamba adapter and shared prompt enhancement according to claim 3, characterized in that: In the prompt generation module, the dimension of the image linear layer is [512, 128], the dimension of the text linear layer is [512, 128], and the dimension of the shared prompt matrix is [12, 5, 128].

5. The method for remote sensing image text retrieval based on Mamba adapter and shared hint enhancement according to claim 1, characterized in that: In the feature extraction module, the image end of the Transformer Block layer is referred to as an image feature extractor, and the text end of the Transformer Block layer is referred to as a text feature extractor; The image feature extractor includes a Patch Embedding layer, a Position Embedding layer, and a learnable embedding vector P. class A series of 12 Transformer Encoder layers and one linear projection layer; T train The input image feature extractor is segmented and flattened into a vector sequence P by the Patch Embedding layer. img , for P img and P class The components are concatenated and then output through the Position Embedding layer. The input to the j-th Transformer Encoder layer is The output of the j-th Transformer Encoder layer is Will and Input image adapter, generate image optimization features Will The cls_token is input to the linear projection layer to transform dimensions, and the output is the image feature vector F. img ; The text feature extractor includes a word segmenter, a Token Embedding layer, a PositionEmbedding layer, 12 Transformer Encoder layers, and a linear projection layer; T train is input into the text feature extractor, segmented by the word segmenter to output T token , output by the Token Embedding layer to output P text , output by the Position Embedding layer The input of the jth Transformer Encoder layer is The output of the jth Transformer Encoder layer is and and are input into the text adapter to generate text optimization features The [EOS] mark of is input into the linear projection layer to transform the dimension, and the output is the text feature vector F text .

6. The remote sensing image text retrieval method based on Mamba adapter and shared prompt enhancement according to claim 3, wherein: In the adapter module, the image adapter includes a dimension reduction linear layer, a data preprocessing layer, two TreeMamba deep feature extraction layers, two residual connections, a gating mechanism connection and a dimension increasing linear layer, and the data preprocessing layer converts the input from a length dimension to a length and width dimension; Will and input image adapter, dimensionality reduction linear layer output with concatenated gelu activation function output dimensionality conversion after data preprocessing layer output two tree mamba deep feature extraction layers in turn output gate mechanism connection and output and residual connection and output and remove Token belonging to get dimensionality increase linear layer output residual connection and output and ​​​ In the adapter module, the text adapter includes a dimension reduction linear layer, two Hydra deep feature extraction layers, a data preprocessing layer, two residual connections, a gated mechanism connection, and a dimension increase linear layer; will be and input text adapter, dimension-reduced linear layer output with concatenated gelu activation function output two hydra deep feature extraction layers in turn output gate mechanism connection and output and residual connection and output and remove Token belonging to get dimension-increased linear layer output residual connection and output and ​​​ 7. The method of remote sensing image text retrieval based on Mamba adapter and shared prompt enhancement according to claim 6, characterized in that: In the image adapter, the dimension of the dimension reduction linear layer is [768, 128], the receptive dimension of the TreeMamba deep feature extraction layer is 128, the SSM spatial feature dimension is 1, and the dimension of the dimension increase linear layer is [128, 768]; in the text adapter, the dimension of the dimension reduction linear layer is [512, 128], the receptive dimension of the Hydra deep feature extraction layer is 128, the SSM spatial feature dimension is 16, and the dimension of the dimension increase linear layer is [128, 512].

8. A device for implementing the method for remote sensing image text retrieval based on Mamba adapter and shared hint enhancement according to any one of claims 1-7, characterized in that: The remote sensing image and text acquisition unit, the CLIP enhanced network based on the multi-modal adapter and shared prompt, the training unit, and the testing unit are included. The remote sensing image and text acquisition unit is configured to acquire original remote sensing images and corresponding text descriptions, and to crop and standardize the original remote sensing images to obtain remote sensing images of a set size. The CLIP enhanced network based on the multi-modal adapter and shared prompt includes a prompt generation network and a feature extraction network. The prompt generation network includes a prompt vector extraction module and a prompt generation module. The prompt vector extraction module extracts image pre-prompt vectors and text pre-prompt vectors from the remote sensing images and corresponding text descriptions, respectively. The prompt generation module generates final image prompt vectors and final text prompt vectors. The feature extraction network includes a feature extraction module and an adapter module. The feature extraction module and the adapter module work together. The feature extraction module extracts shallow features of the remote sensing images and corresponding text descriptions, and the adapter module extracts deep features of the remote sensing images and corresponding text descriptions. Finally, the feature vectors of the remote sensing images and corresponding text descriptions are generated. The training unit uses a loss function to supervise the training of the CLIP enhanced network based on the multi-modal adapter and shared prompt. The testing unit uses the trained CLIP enhanced network based on the multi-modal adapter and shared prompt to perform image-to-text retrieval and text-to-image retrieval on pairs of remote sensing images and corresponding text descriptions.

9. The apparatus of claim 8, wherein: The CLIP enhanced network based on the multi-modal adapter and shared prompt includes a prompt generation network and a feature extraction network. The prompt generation network comprises a prompt vector extraction module and a prompt generation module, the prompt vector extraction module comprises an image prompt vector extractor and a text prompt vector extractor; the feature extraction network comprises a feature extraction module and an adapter module, the adapter module comprises a group of image adapters and a group of text adapters, the feature extraction module is designed based on a CLIP model, the CLIP model comprises 12 layers of Transformer Block, an image adapter is embedded at the image end of each Transformer Block layer, and a text adapter is embedded at the text end of each Transformer Block layer; the parameters of the CLIP model in the prompt vector extraction module and the feature extraction module are replaced by the pre-trained CLIP-B / 32 parameters.

10. The apparatus of claim 9, wherein: In the CLIP enhanced network based on the multi-modal adapter and the shared prompt: I train and T train are input into the prompt generation network, I train image pre-prompt vectors feature p_img , T train text pre-prompt vectors feature p_text , feature p_img and feature p_text are input into the prompt generation module; feature p_img and feature p_text input prompt generation module, generating an image final prompt vector and a text final prompt vector I train and T train are respectively taken as and input into a feature extraction network, the feature extraction network comprising 12 layers of Transformer Blocks; the image-side input of the jth layer of Transformer Blocks is denoted as the image-side output is denoted as and and are input into an image adapter to generate image optimization features the text-side input of the jth layer of Transformer Blocks is denoted as the text-side output is denoted as and and are input into a text adapter to generate text optimization features Will The cls_token is used to tag the input linear projection layer and output the image feature vector F after dimensionality reduction. img ,Will The [EOS] token is input to a linear projection layer for dimensionality reduction, and the output is a text feature vector F. text .

Images