Remote Sensing Image Text Retrieval Method Based on Remote Sensing Multimodal Model

Abstract

This invention relates to the fields of remote sensing image analysis and cross-modal retrieval technology. It discloses a remote sensing image text retrieval method based on a remote sensing multimodal fundamental model. By fine-tuning the CLIP large model, its large-scale pre-training capabilities are applied to the semantic alignment of remote sensing images and text. By introducing a visual saliency calculation module and a visual block fine-grained selection integration module, this invention effectively solves the problem of multi-scale targets and redundant information in remote sensing images, achieving fine-grained semantic alignment between images and text, and improving retrieval accuracy. Especially when images contain multiple salient targets and redundant regions, the proposed cross-modal semantic alignment fine-grained filtering method can accurately identify key information blocks in the image and perform fine matching with the text description.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image analysis and cross-modal retrieval technology, specifically relating to a remote sensing image text retrieval method based on a remote sensing multimodal basic model. Background Technology

[0002] Cross-modal retrieval of remote sensing image-text is an important task in computer vision, aiming to improve the accuracy and efficiency of retrieval between different modalities of data (such as images and text). In the field of remote sensing image interpretation, remote sensing scene classification and image retrieval often face complex image structures and diverse scene content. Each scene contains objects of multiple scales and different types, making it particularly difficult to extract effective and discriminative features from images. In addition, remote sensing image data has multi-scale and multi-object characteristics, with different objects exhibiting different shapes, sizes, and spatial layouts. This makes it difficult for traditional image-text alignment methods to effectively capture the fine-grained semantic relationships between images and text, especially when images contain multiple salient objects and redundant regions. Accurate alignment between images and text becomes a significant challenge.

[0003] Currently, deep learning-based visual-language models, such as CLIP, have demonstrated semantic alignment capabilities between images and text. The CLIP model, pre-trained on large-scale image-text alignment data, can provide effective image-text matching across multiple visual tasks. However, this coarse-grained alignment approach has limitations when processing remote sensing images. Remote sensing images typically contain multi-scale targets and complex backgrounds, making it difficult for coarse-grained alignment methods to capture subtle semantic differences between images and text. This is especially true when images contain multiple redundant regions and details; traditional methods tend to overlook important visual information, thus affecting the accuracy of image-text matching.

[0004] To address this issue, researchers have proposed various methods in recent years to improve the performance of cross-modal image-text retrieval. For example, by designing specialized semantic alignment modules, introducing self-attention mechanisms, and employing multi-scale feature fusion, researchers have attempted to optimize the semantic alignment between images and text. However, most existing methods still face the challenge of effectively handling multi-scale and complex background information in remote sensing images, which limits their application in the field of remote sensing.

[0005] Furthermore, remote sensing image datasets are typically small, and data annotation requires specialized domain knowledge, leading to data scarcity issues in their application in cross-modal retrieval. Although some methods have attempted to address the data annotation difficulties using semi-supervised learning, pseudo-labels, and other strategies, effectively improving the performance of remote sensing image-text cross-modal retrieval with limited data remains a pressing problem. Summary of the Invention

[0006] To address the accuracy and efficiency issues in cross-modal retrieval of remote sensing images and text, this invention proposes a remote sensing image text retrieval method based on a remote sensing multimodal fundamental model. By combining multi-scale features and semantic information of remote sensing images and text, the method can effectively improve the cross-modal matching accuracy of images and text.

[0007] This invention discloses a remote sensing image text retrieval method based on a remote sensing multimodal fundamental model, comprising the following steps:

[0008] Step S1: Collect and preprocess the remote sensing image and text dataset. Divide the remote sensing image and text dataset into a training set, a validation set, and a test set for model training, validation, and evaluation, respectively. The remote sensing image and text dataset includes the RSITD remote sensing image-text dataset, the RSICD remote sensing image-text dataset, and the UCM remote sensing image-text dataset.

[0009] Step S2: Construct a retrieval model based on a remote sensing multimodal basic model. The remote sensing multimodal basic model adopts a pre-trained CLIP image-text large model. The retrieval model also includes an image encoder, a text encoder, a visual saliency calculation module, a visual block fine-grained selection and integration module, and a cross-modal similarity calculation module.

[0010] The image encoder employs CLIP's VisionTransformer architecture to extract image patch features from remote sensing images and map them to a shared multimodal space. Represented as:

[0011] ;

[0012] Indicates the number of visual blocks. The dimension of the feature; This represents the result after the remote sensing image is segmented. Each visual block has its own corresponding feature vector;

[0013] The implementation process of the visual saliency calculation module includes:

[0014] Visual region saliency modeling: Calculating region saliency scores using multi-head self-attention mechanism, feedforward neural network, and layer normalization. The specific formula is as follows:

[0015] ;

[0016] ;

[0017] ;

[0018] In the formula, It is a multi-head self-attention mechanism. This refers to feedforward neural networks, which are used to perform nonlinear transformations on the output of self-attention mechanisms to enhance the expressive power of feature sequences. This refers to layer normalization operation; These are features resulting from multi-head self-attention combined with residual connections; It was through Features after adding residual connections and layer normalization;

[0019] Step S3: Train the retrieval model and improve the matching accuracy and training efficiency of remote sensing images and text by optimizing the output and loss function of the cross-modal similarity calculation module;

[0020] Step S4: Input the remote sensing image or text to be retrieved into the trained retrieval model, and output the text or remote sensing image most relevant to the remote sensing image or text to be retrieved.

[0021] Preferably, the implementation process of the visual block fine-grained selection and integration module includes:

[0022] Cross-modal visual saliency modeling and visual region saliency modeling generate cross-modal alignment scores and region saliency scores for images, respectively, reflecting the semantic matching between images and text;

[0023] Two types of significance scores and Weighted fusion yields a visual comprehensive importance score that reflects both the characteristics of key regions in the image and the degree of semantic matching between the image and text. , This represents the image-text alignment score calculated through cross-modal visual saliency modeling. This represents the region saliency score obtained through visual region saliency modeling. These are the weighting coefficients for adjusting significance fusion; the formula for weighted fusion is as follows:

[0024] ;

[0025] Apply softmax normalization to the visual comprehensive importance score Convert to visual block selection score The formula is:

[0026] ;

[0027] Representing the The selection score of each visual block, i.e. Representing the first The selection score for each visual block; For the first The overall visual importance score of each visual block. For the first The overall visual importance score of each visual block;

[0028] right Sort in descending order, select factors based on visual blocks. The visual block with the highest score is selected to form the filtered visual block feature. ,in The parameter can be a fixed value or a trainable parameter in the range of 0.80-0.95.

[0029] right L2 normalization is performed, followed by average pooling to generate the final visual representation. The formula is ,in .

[0030] Preferably, in step S3: the loss function The triplet loss function is used, and its expression is:

[0031]

[0032] in, express function, Represents the cosine similarity function; The parameters are the triplet loss; the loss function consists of two symmetric parts; the first part applies to all negative text samples. Perform summation, given that the query image is... The second part then examines all negative image samples. Perform summation based on the query text. .

[0033] Preferably, the cross-modal similarity calculation module calculates the matching degree between the remote sensing image and the text using cosine similarity, with the following formula:

[0034] ;

[0035] Represents the similarity function. Indicates the use of calculation The function of cosine similarity Represents the dot product operation. and Representing characteristics respectively and The norm of .

[0036] Preferably, the retrieval task in step S4 includes two sub-tasks:

[0037] Text retrieval: Using remote sensing images as the query term, retrieve the most relevant text from a text database, based on similarity scores. Sort the text;

[0038] Image retrieval: Using text as the query term, retrieves the most relevant images from a remote sensing image database, based on... Sort the images;

[0039] Recall rate is used to evaluate search performance. This indicates that the most relevant samples are among the top results returned by the retrieval. The proportion of each result This is the preset number of results.

[0040] Preferably, the collection and preprocessing of remote sensing image dataset in step S1 can also be carried out in the following way: first adjust the image to 256×256 pixels, and then unify the image size to 224×224 pixels through a center cropping operation to retain the information of the core area of ​​the image.

[0041] Preferably, in step S4, the number of "most relevant text or remote sensing images" returned can be set according to actual needs. Furthermore, in remote sensing scenarios containing multi-scale targets and complex backgrounds, the retrieval model can filter redundant regions through the visual saliency calculation module and the visual block fine-grained selection integration module, thereby achieving fine-grained semantic alignment and improving retrieval accuracy.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] 1. This invention provides a remote sensing image text retrieval method based on a remote sensing multimodal fundamental model, specifically relating to the field of cross-modal retrieval technology for remote sensing images and text. This invention applies the large-scale pre-training capabilities of the CLIP model, through fine-tuning, to the semantic alignment of remote sensing images and text.

[0044] 2. By introducing a visual saliency calculation module and a visual block fine-grained selection and integration module, this invention effectively solves the problem of multi-scale targets and redundant information in remote sensing images, realizes fine-grained semantic alignment between images and text, and improves the accuracy of retrieval.

[0045] 3. When an image contains multiple salient targets and redundant regions, the proposed cross-modal semantic alignment fine-grained filtering method can accurately identify key information blocks in the image and perform fine matching with the text description.

[0046] 4. This invention improves the accuracy and robustness of cross-modal retrieval, and promotes the development of remote sensing image-text cross-modal retrieval technology. Attached Figure Description

[0047] Figure 1 This is a simplified flowchart illustrating a fine-grained alignment method for cross-modal retrieval of remote sensing images and text based on a large image-text model.

[0048] Figure 2 This is a schematic diagram of the network structure of a cross-modal retrieval model for remote sensing images and text based on a large image-text model with fine-grained alignment. Detailed Implementation

[0049] Please refer to Figures 1 to 2 This embodiment discloses a remote sensing image text retrieval method based on a remote sensing multimodal fundamental model, including:

[0050] Step S1: Collect remote sensing image dataset and preprocess it. The remote sensing image dataset is divided into training set, validation set and test set, which are used for subsequent model training, validation and evaluation, respectively.

[0051] Step S2: Construct a fine-grained aligned remote sensing image-text cross-modal retrieval model based on a large image-text model. The model adopts a pre-trained CLIP large image-text model and combines it with a fine-grained alignment method, including an image encoder, a text encoder, a visual saliency calculation module, a visual block fine-grained selection and integration module, and a cross-modal similarity calculation module.

[0052] Step S3: Train the fine-grained aligned remote sensing image-text cross-modal retrieval model based on the large image-text model. During the training process, the accuracy and efficiency of image-text matching are improved by optimizing the cross-modal similarity calculation module and loss function.

[0053] Step S4: Input the remote sensing image or text to be retrieved into the trained image-text model, and output the text or image most relevant to the query image or text.

[0054] Furthermore, three commonly used publicly available remote sensing image-text datasets were collected for experiments: the RSITMD, RSICD, and UCM datasets. The RSITMD dataset contains 4743 remote sensing images distributed across 32 different scenes. The RSICD dataset contains 10,921 remote sensing images covering 31 scenes. The UCM dataset contains 2100 remote sensing images spanning 21 scenes. Each image in these three datasets is accompanied by five corresponding text sentences describing the image content. The three datasets were divided into training (80%), validation (10%), and test (10%) sets.

[0055] The preprocessing of the marine remote sensing image data in step S1 includes: changing the image size of each image in the remote sensing image dataset to 256×256, and cropping it to 224×224 using a center cropping method.

[0056] The structure of a fine-grained aligned remote sensing image-text cross-modal retrieval model based on a large image-text model is as follows: Figure 2 As shown, it comprises five main modules: an image encoder, a text encoder, a visual saliency calculation module, a visual block fine-grained selection and integration module, and a cross-modal similarity calculation module. The overall framework is based on the CLIP pre-trained model, achieving cross-modal semantic alignment between remote sensing images and text through fine-tuning.

[0057] The image encoder adopts CLIP's VisionTransformer (ViT) architecture, extracting image patch features and mapping them to a shared multimodal space. The text encoder extracts text features using CLIP's Transformer model and maps them to the same embedding space. The visual saliency calculation module first identifies the saliency of key regions in the image through visual region saliency modeling, and then further optimizes the semantic alignment of the image by combining cross-modal visual saliency modeling with text information.

[0058] The fine-grained visual block selection and integration module filters key information blocks in the image based on visual saliency scores and performs refined alignment with the text description. Finally, a cross-modal similarity calculation module uses cosine similarity to measure the degree of matching between image and text features. The entire model is trained using triplet loss to improve the performance of remote sensing image-text cross-modal retrieval, and further optimizes the accuracy of image-text alignment through weighted saliency score fusion and a visual block selection strategy.

[0059] Furthermore, the remote sensing image-text cross-modal retrieval task is defined as follows:

[0060] For remote sensing image-text cross-modal retrieval, the query term can be a remote sensing image, and the goal is to retrieve the most relevant text from the database; conversely, the query term can also be text, with the aim of retrieving relevant remote sensing images. This involves two sub-tasks, designated as TextRetrieval and ImageRetrieval. The primary goal of model training is recall. ),in Indicates the previous results returned by the search. Each item. This indicates that the most relevant sample items are located at the top of the returned list. The proportion of each item.

[0061] definition Indicated as containing The training set of remote sensing image-text pairs. Indicates that it is correct, and This indicates a negative pair. Therefore, It is a remote sensing image. The most relevant text samples, and vice versa. During model learning, the similarity function... Used to distinguish between positive and negative sample pairs, with positive samples having higher similarity. In text retrieval, where the query term is an image, the ranking of the text database depends on the similarity score. .index The most relevant text was quantified based on Sort by first The proportion of text candidates. Correspondingly, this metric is also applicable to image retrieval scenarios. Here, cosine similarity between cross-modal features is used as the similarity between samples.

[0062] ;

[0063] In the formula This represents the entire CLIP-FGA model, with remote sensing image-text pairs as input. Mapped to corresponding visual-linguistic features , Indicates model parameters.

[0064] Then, the formula for calculating cross-modal cosine similarity is:

[0065]

[0066] In the formula Indicates the use of calculation The function of cosine similarity Represents the dot product operation. and Representing characteristics respectively and The norm of .

[0067] Furthermore, the image and text encoders are the foundational modules of the entire remote sensing image-text cross-modal retrieval framework. Their main task is to extract rich semantic information from remote sensing images and text descriptions, respectively, and map the features of the two modalities into the same shared space, thus laying the foundation for subsequent fine-grained alignment. CLIP achieves image-text alignment through contrastive learning, making semantically similar image-text pairs closer in high-dimensional space.

[0068] In the vision branch, for the input remote sensing image J, the image patch features are extracted using CLIP's VisionTransformer (ViT) architecture image encoder ViT-B / 32.

[0069] ;

[0070] in, The number of visual blocks is represented by D, where D is the dimension of the feature. This represents the result after the remote sensing image is segmented. Each visual block has its own corresponding feature vector.

[0071] For the input text, [SOS] and [EOS] markers are first added to its sequence to indicate the start and end positions. During encoding, the Transformer model used by CLIP takes the activation values ​​at the [EOS] markers in the last layer as the feature representation of the text. This representation is then normalized and mapped to the multimodal embedding space via linear projection to obtain the text features.

[0072] Furthermore, the visual saliency calculation module is designed to address the semantic matching challenges posed by multiple targets and redundant regions in remote sensing images, refining the coarse-grained alignment results of the CLIP model. To this end, a visual saliency calculation module is proposed, which calculates two types of saliency scores to accurately capture the semantic correspondence between images and text.

[0073] Specifically, the module first uses visual region saliency modeling to generate region saliency scores for each region in the image, thereby identifying the most critical parts of visual semantic communication. Simultaneously, the module employs cross-modal visual saliency modeling, utilizing image-text matching information to generate cross-modal semantic alignment scores, measuring the degree of matching between image content and text description. Finally, by fusing the two types of saliency scores, a comprehensive visual importance score is obtained that reflects both the characteristics of key regions in the image and the degree of image-text semantic matching. This score provides precise guidance for the subsequent selection and integration of fine-grained visual blocks, thereby achieving fine-grained semantic alignment between text and images.

[0074] The visual region saliency representation calculates and generates region saliency scores by analyzing each local region of the image, thereby quantifying the importance of each region in conveying key information.

[0075] This chapter uses the self-attention mechanism to represent the saliency of visual regions, as detailed below:

[0076] ;

[0077] ;

[0078] ;

[0079] In the formula, It is a multi-head self-attention mechanism. This refers to feedforward neural networks, which are used to perform nonlinear transformations on the output of self-attention mechanisms to enhance the expressive power of feature sequences. This refers to layer normalization operation; These are features resulting from multi-head self-attention combined with residual connections; It was through Features after adding residual connections and layer normalization;

[0080] After calculation using the self-attention mechanism, the region saliency score obtained through visual region saliency modeling was obtained. This is used to characterize the semantic importance of each visual region.

[0081] The cross-modal visual saliency representation, by combining text information, helps to determine regions in an image that are highly related to the semantics of the text, thereby improving the accuracy of image-text matching.

[0082] Specifically, two calculation modes were designed to measure the matching degree between images and text. The first mode directly calculates the similarity between image regions and text, using it as the image-text matching score to measure the alignment between the overall image content and the text description. The second mode, based on a cross-attention mechanism, calculates a cross-modal alignment score, thereby dynamically capturing the image regions most critical for semantic matching under text guidance. This leverages textual information to enhance the perception of the overall semantics of the image, uncovering key regions in the image that highly match the text at a fine-grained level.

[0083] The method of directly calculating similarity ensures global semantic consistency, while the method based on cross-attention further improves the alignment accuracy between visual features and text features.

[0084] The design of the image region and text matching alignment mode is as follows:

[0085] ;

[0086] This represents transposition. This represents the image-text alignment score calculated through cross-modal visual saliency modeling. This method of calculating similarity ensures global semantic consistency, and the direct image-text matching score intuitively reflects the relevance between visual regions and text semantics.

[0087] The design of the cross-modal alignment mode based on the cross-attention mechanism is as follows:

[0088] ;

[0089] ;

[0090] ;

[0091] ;

[0092] In the formula, It is a multi-head cross-attention mechanism. The cross-attention approach further enhances the semantic matching capability between images and text at a fine-grained level by dynamically capturing the correlation between visual and textual features. In this way, the cross-attention mechanism can accurately weight key information in the image, thereby making the cross-modal semantic alignment between images and text more detailed and precise.

[0093] Specifically, the saliency score fusion involves cross-modal visual saliency modeling and visual region saliency modeling generating cross-modal alignment scores and region saliency scores for the image, respectively, reflecting the semantic matching between the image and the text. By weighted fusion of these two types of scores, the module can comprehensively consider the multidimensional matching information between the image and the text, thereby forming a comprehensive visual importance score.

[0094] Specifically, the two types of significance scores and Weighted fusion yields a visual comprehensive importance score that reflects both the characteristics of key regions in the image and the degree of semantic matching between the image and text. The formula for weighted fusion is as follows:

[0095] ;

[0096] In the formula It is a weighting coefficient that adjusts the significance level of the fusion, controlling the contribution of the two types of significance scores in the fusion process. . These represent the weight parameters of the visual overall importance score for the i-th visual block. There are two design options: the first is to... The first approach uses a trainable weight parameter, which is automatically optimized by the deep learning network during training. The second approach sets a fixed constant value (ranging from 0 to 1.0) to control the fusion ratio of the two saliency scores. The subsequent experimental section compares these two designs, and the results show that the trainable weight parameter approach is superior in performance.

[0097] In this way, a comprehensive visual importance score XS can be generated by fully utilizing the complementarity between cross-modal visual saliency and regional importance, ensuring a more comprehensive semantic alignment between images and text.

[0098] Furthermore, the visual block fine-grained selection and integration module is designed to achieve more accurate cross-modal semantic alignment by precisely selecting key regions in the image that are highly relevant to the text description. In this module, the most critical visual blocks for image-text matching are first identified using the aforementioned saliency score fusion results. Then, by combining this key information, the feature representations of the visual blocks are further integrated and optimized, thereby enhancing the semantic matching accuracy between the image and text at a fine-grained level.

[0099] Apply softmax normalization to the visual comprehensive importance score Convert to visual block selection score :

[0100] ;

[0101] Representing the The selection score of each visual block, i.e. Representing the first The selection score for each visual block; For the first The overall visual importance score of each visual block. For the first The overall visual importance score of each visual block.

[0102] Next, select a score for the visual block. Sort the data in descending order to obtain the sorted scores and corresponding indices. Based on the sorted indices, analyze the original visual block features. Select the highest-scoring portion to form the filtered visual block feature. ,in As a visual patch selection factor, it determines the number of visual patches to be retained. This chapter also explores the impact of different settings for this parameter, comparing two approaches: a fixed constant value (ranging from 0.80 to 0.95) and trainable weight parameters. In this way, we can not only evaluate the difference in adaptability between fixed values ​​and trainable parameters when selecting visual patches, but also better understand how to suppress redundant information and improve the semantic alignment accuracy between images and text by adjusting the number of retained visual patches.

[0103] Selected visual block features L2 normalization (Norm) is then performed to eliminate the influence of feature vector length, followed by average pooling (Avgpool) to form the final visual representation. Subsequently, cross-modal similarity is calculated with text features:

[0104] ;

[0105] This module effectively filters out the most semantically valuable regions from images and aligns them more precisely with text descriptions, thereby improving the performance of remote sensing image-text cross-modal retrieval tasks.

[0106] Furthermore, a triplet loss function is used to train the entire remote sensing image-text retrieval network. The expression for the loss function is:

[0107]

[0108] in, express function, Represents the cosine similarity function. The parameters of the triplet loss are... Setting this parameter to 0.2 defines a threshold for determining which samples can be considered "far enough" negative samples. During training, this ensures that the model does not minimize loss by simply pulling all negative samples away, but instead focuses on modeling the semantic differences between positive and negative samples.

[0109] The loss function consists of two symmetric parts. The first part applies to all negative text samples. Perform summation, given that the query image is... The second part then examines all negative image samples. Perform summation based on the query text. .

[0110] To improve computational efficiency, during stochastic gradient descent training, only negative samples in the mini-batch are summed, rather than considering all negative samples in the training set.

[0111] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A remote sensing image text retrieval method based on a remote sensing multi-modal base model, characterized in that, The method comprises the following steps: Step S1: Collecting and preprocessing a remote sensing image-text data set, and dividing the remote sensing image-text data set into a training set, a validation set and a test set for model training, validation and evaluation, respectively; the remote sensing image-text data set comprises a RSITMD remote sensing image-text data set, a RSICD remote sensing image-text data set and a UCM remote sensing image-text data set; Step S2: Constructing a retrieval model based on a remote sensing multi-modal base model, wherein the remote sensing multi-modal base model adopts a pre-trained CLIP image-text large model, and the retrieval model further comprises an image encoder, a text encoder, a visual saliency calculation module, a visual block fine-grained selection and integration module and a cross-modal similarity calculation module; The image encoder adopts a VisionTransformer architecture of CLIP, extracts image block features of a remote sensing image and maps the image block features to a shared multi-modal space, and the image block features are represented as: ; representing a number of visual blocks, is a dimension of the feature; representing a number of visual blocks obtained after the remote sensing image is divided each corresponding to a feature vector; The implementation process of the visual saliency calculation module comprises: Visual region saliency modeling: Region saliency scores are computed by multi-head self-attention mechanism, feed-forward neural network and layer normalization The specific formula is: ； ； ; In the formula, is a multi-head self-attention mechanism, refers to a feedforward neural network, which is used for nonlinear transformation of the output of the self-attention mechanism, and enhances the expression ability of the feature sequence, refers to a layer normalization operation; is a feature after multi-head self-attention plus residual connection; is a feature after plus residual connection plus layer normalization; Step S3: Training the retrieval model, optimizing the output and loss function of the cross-modal similarity calculation module, and improving the matching accuracy and training efficiency of the remote sensing image and the text; Step S4: Inputting a remote sensing image or text to be retrieved into the trained retrieval model, and outputting the most relevant text or remote sensing image corresponding to the remote sensing image or text to be retrieved.

2. The remote sensing image text retrieval method based on a remote sensing multi-modal base model according to claim 1, characterized in that, The implementation process of the visual block fine-grained selection and integration module comprises: Cross-modal visual saliency modeling and visual region saliency modeling generate cross-modal alignment scores and region saliency scores of the image, respectively, reflecting the semantic matching between the image and the text; The two types of saliency scores are fused by weighting and A visual comprehensive importance score is obtained, which reflects both the characteristics of key regions in the image and the degree of semantic matching between the image and the text , represents the image-text alignment score calculated by cross-modal visual saliency modeling, represents the region saliency score obtained by visual region saliency modeling, is a weight coefficient for adjusting the saliency fusion; the calculation formula of the weighted fusion is as follows: ; The visual synthesis importance score is converted into a visual block selection score by applying a softmax normalization operation , as follows: , as follows: ; Representing the The selection score of each visual block, i.e. Representing the first The selection score for each visual block; For the first The overall visual importance score of each visual block. For the first The overall visual importance score of each visual block; right Sort in descending order, select factors based on visual blocks. The visual block with the highest score is selected to form the filtered visual block feature. ,in The parameter can be a fixed value or a trainable parameter in the range of 0.80-0.

95. To perform L2 normalization, and then generate the final visual representation through average pooling operation , the formula is where . 3.The remote sensing image text retrieval method based on a remote sensing multi-modal base model according to claim 1, characterized in that, In the step S3, the loss function With the triplet loss function, the expression is: ； wherein, represents the function, represents the cosine similarity function; is a parameter of the triplet loss; the loss function consists of two symmetric parts; the first part sums over all negative text samples given the query image ; the second part sums over all negative image samples given the query text .

4. The remote sensing image text retrieval method based on a remote sensing multi-modal base model according to claim 1, characterized in that, The cross-modal similarity calculation module calculates the matching degree of the remote sensing image and the text by cosine similarity, and the formula is: ； denotes a similarity function, denotes a function for computing cosine similarity, denotes a dot product operation, and denotes a norm of a feature and respectively. 5.The remote sensing image text retrieval method based on a remote sensing multi-modal base model according to claim 1, characterized in that, The retrieval task in step S4 comprises two sub-tasks: Text retrieval: retrieving the most relevant text from a text database given a remote sensing image as a query, based on a similarity score Ranking of text; Image retrieval: retrieve the most relevant images from a remote sensing image database with text as query term, according to Ranking of images; The search performance evaluation employs recall , represents the proportion of the most relevant samples located in the top results returned by the search, is the preset result number. 6.The remote sensing image text retrieval method based on a remote sensing multi-modal base model according to claim 1, characterized in that, In step S1, the remote sensing image-text data set can also be collected and preprocessed in the following manner: first, adjusting the image to 256x256 pixels, and then uniformly cutting the image size to 224x224 pixels by a center cutting operation to retain the core area information of the image. 7.The remote sensing image text retrieval method based on a remote sensing multi-modal base model according to claim 1, characterized in that, In the step S4, the output "most relevant text or remote sensing image" can be set to return a number according to actual needs , and the retrieval model can filter redundant areas through the visual saliency calculation module and the visual block fine-grained selection integration module in a remote sensing scene containing multi-scale targets and complex backgrounds, realize fine-grained semantic alignment, and improve retrieval accuracy.

Images