A Remote Sensing Image-Text Retrieval Method Based on Hierarchical Targeted Semantic Alignment
By employing a hierarchical targeted semantic alignment method, which combines feature extraction and fine-grained alignment of remote sensing images and text, the problem of insufficient fine-grained information capture and redundant alignment in remote sensing image and text retrieval is solved, thereby improving the accuracy and efficiency of remote sensing image and text retrieval and adapting to the characteristics of remote sensing scenarios.
Patent Information
- Application Number
- CN202611349137.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-02
- Publication Date
- 2026-11-17
AI Technical Summary
Existing remote sensing image retrieval technologies suffer from insufficient fine-grained information capture capabilities, high computational complexity, and redundant alignment issues in remote sensing scenarios. Furthermore, pre-trained models are not adapted to the characteristics of remote sensing scenarios, resulting in insufficient retrieval accuracy and efficiency.
We employ a hierarchical targeted semantic alignment approach, extracting features from images and text through image sampling networks and text sampling networks respectively. We combine coarse-grained global semantic alignment and fine-grained local semantic alignment, introduce a targeted semantic alignment mechanism to filter core matching pairs, and improve model adaptability through a hierarchical prompting optimization mechanism. Finally, we construct a joint loss function for end-to-end training.
Multi-dimensional semantic optimization of remote sensing image and text retrieval was achieved, which improved retrieval accuracy and adaptability, reduced redundant computational overhead, adapted to the characteristics of remote sensing scenarios, and improved the transfer performance of pre-trained models.
Smart Images

Figure CN123115453A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing image processing technology, and specifically to a remote sensing image and text retrieval method based on hierarchical targeted semantic alignment. Background Technology
[0002] With the continuous advancement of Earth observation technology, the efficiency and scale of high-resolution remote sensing image acquisition have rapidly increased. How to efficiently and accurately retrieve images that match text descriptions from massive amounts of remote sensing data—that is, remote sensing image-text retrieval technology—has become an important research direction in the field of remote sensing information processing. It has wide application value in scenarios such as geographic positioning, disaster emergency rescue, urban planning and management, and land and resources surveys.
[0003] The core of existing remote sensing image-text retrieval (RSITR) technology lies in achieving cross-modal semantic alignment. Based on the granularity and implementation method of alignment, it can be divided into two main categories: global feature alignment paradigms and local feature alignment paradigms. In recent years, general visual language pre-training techniques, represented by contrastive language-image pre-training (CLIP) models, have achieved significant breakthroughs. Their powerful cross-modal semantic representation capabilities provide a new technical path for remote sensing image-text retrieval, and transfer learning based on pre-trained models has gradually become the mainstream research direction in this field. Among them, the global feature alignment method extracts global semantic features of images and text separately through an encoder, maps them to a unified shared semantic space, and then calculates similarity. This type of method has high computational efficiency and strong generalization ability, but its ability to capture fine-grained ground feature information is insufficient, making it difficult to adapt to the retrieval needs of multi-scale, sparse targets in remote sensing scenes. Local feature alignment methods break down images into multiple region features and text into word-level features. By fine-grained matching between regions and words, richer semantic correspondences can be mined, which can improve the accuracy of detail matching. However, they generally suffer from high computational complexity and are prone to introducing redundant alignment.
[0004] Building upon the aforementioned technological advancements, numerous optimization studies have been conducted in this field focusing on transfer learning efficiency and alignment accuracy. One study disclosed a computationally and memory-efficient remote sensing image retrieval framework. This framework suppresses background pixel interference during small target feature extraction through a side-branched focusing adapter module. Simultaneously, it uses remote sensing scene categories as metadata to design tag enhancement techniques to narrow the retrieval space and expands the negative sample pool for contrastive learning using a negative sample retrieval strategy. This approach effectively reduces GPU memory usage and increases data throughput during the training phase while maintaining competitive retrieval performance. However, this scheme emphasizes computational and storage resource efficiency optimization during the training phase and employs a single-path semantic alignment logic. It does not target redundant regions in the fine-grained matching process, leaving room for improvement in the deep semantic optimization dimension of retrieval accuracy.
[0005] Existing patent document CN121919374A discloses a cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration. It constructs a dual parallel branch architecture with explicit and implicit branches, achieving fine-grained word-region alignment in the explicit branch through a context-aware weighted masking strategy. Then, the contrastive loss generated by the explicit branch supervises the adaptive binary hash representation generation of the implicit branch, balancing semantic alignment accuracy and hash retrieval efficiency. This scheme employs a parallel branch collaboration design, but its fine-grained alignment mechanism lacks adaptation optimization to the imaging characteristics of the remote sensing scene.
[0006] Furthermore, most existing transfer learning schemes directly reuse the structure and parameters of general domain pre-trained models without specifically adapting them to the characteristics of remote sensing scenes, such as land cover morphology and spatial distribution. When transferring pre-trained semantic knowledge to the remote sensing domain, there is a significant performance loss, which fails to fully unleash the representational potential of the pre-trained models.
[0007] Therefore, there is an urgent need in this field for a new technical solution—a remote sensing image and text retrieval method that can adapt to the characteristics of remote sensing scenarios and take into account both retrieval accuracy and operational efficiency, so as to better meet the high-precision retrieval needs under massive remote sensing data. Summary of the Invention
[0008] To alleviate or partially alleviate the above-mentioned technical problems, the solution of the present invention is as follows:
[0009] A remote sensing image and text retrieval method based on hierarchical targeted semantic alignment includes the following steps:
[0010] The input remote sensing image is feature-encoded using an image sampling network, and the output is an image feature sequence containing global image features and local features of multiple image regions; the input descriptive text is feature-encoded using a text sampling network, and the output is a text feature sequence containing global text features and local features corresponding to multiple words.
[0011] Based on the global features of the image and the global features of the text, coarse-grained global semantic alignment is performed to obtain cross-modal classification loss for constraining the discriminative power between feature classes within the modality, and cross-modal matching loss for aligning the distributions of global features of the image and global features of the text.
[0012] The semantic similarity between the local features of each image region and the local features of all words is calculated to construct a region-text similarity matrix. For each word, the semantic matching results of all image regions corresponding to that word are targeted and filtered by a target correction weight. The target correction weight satisfies the following condition: the weight is 1 if and only if the image region is the region with the highest semantic similarity to the word, and the weight of the other image regions is 0. Based on the filtered semantic matching results, the target similarity between the word and the remote sensing image is calculated. The target similarity of all words is aggregated to obtain a fine-grained cross-modal similarity, and a fine-grained alignment loss is obtained.
[0013] The cross-modal classification loss, cross-modal matching loss, and fine-grained alignment loss are summed to construct a joint loss function. The image sampling network and text sampling network are trained end-to-end with the goal of minimizing the joint loss, resulting in a remote sensing image-text retrieval model for bidirectional retrieval of remote sensing images and descriptive text.
[0014] Preferably, the target correction weight is determined as follows:
[0015] ;
[0016] in, The target correction weights are the x-th image region and the y-th word; y is the index of the query word. The word index corresponding to the image region; For the x-th image region and the th The semantic similarity of words; arg max(·) is the operation that takes the index corresponding to the maximum value; otherwise represents the other cases.
[0017] Preferably, the target similarity is calculated as follows:
[0018] The selected image region features are subjected to a normalized dot product operation with the local features of the corresponding word to obtain the target similarity between the word and the remote sensing image; the normalized dot product operation is to calculate the dot product after normalizing the two feature vectors by L2 norm respectively.
[0019] Preferably, the fine-grained cross-modal similarity obtained by aggregating the targeted similarity of all words is achieved using logarithmic exponential pooling, including:
[0020] The target similarity of each word is multiplied by a scaling factor and then exponentially calculated. The sum of the exponential calculation results of all words is then logarithmically calculated and divided by the scaling factor to obtain the fine-grained cross-modal similarity. The scaling factor is used to adjust the weight of the most relevant semantics in similarity aggregation.
[0021] Preferably, the cross-modal matching loss is calculated as follows:
[0022] Calculate the matching loss from text to image direction and the matching loss from image to text direction, respectively;
[0023] The text-to-image matching probability distribution is calculated based on global image features and global text features. KL divergence is used to measure the difference between the text-to-image matching probability distribution and the image-to-text matching probability distribution, and the matching loss in the text-to-image direction is obtained.
[0024] The image-to-text matching probability distribution is calculated based on global image features and global text features. KL divergence is used to measure the difference between the image-to-text matching probability distribution and the text-to-image matching probability distribution, and the matching loss in the image-to-text direction is obtained.
[0025] The cross-modal matching loss is obtained by summing the matching loss from text to image and the matching loss from image to text.
[0026] Preferably, the fine-grained alignment loss employs a bidirectional ranking loss, including:
[0027] For a matched image-text pair, select the negative sample image with the lowest text similarity to the matched image-text pair from the non-matching pairs in the training batch, and the negative sample text with the lowest image similarity to the matched image-text pair;
[0028] Based on the difference between the image-text pair similarity of positive matching and the image-text pair similarity of negative matching, the ranking loss in the text-to-image direction and the ranking loss in the image-to-text direction are calculated respectively. The ranking losses in the two directions are summed to obtain the fine-grained alignment loss.
[0029] Preferably, both the image sampling network and the text sampling network use the corresponding encoder of the pre-trained contrastive language image pre-trained CLIP model for parameter initialization; during the training phase, only the hidden layers of the CLIP model are adapted by generating hierarchical cue vectors to output a contextual representation that integrates remote sensing scene semantic information, without the need for full fine-tuning of the CLIP model.
[0030] Preferably, the semantic matching result is corrected based on contextual representation, including:
[0031] The context representation input conditional random field is used to perform sequence labeling probability modeling to obtain the semantic correction probability of each word; the semantic correction probability is multiplied and fused with the original matching probability element by element to obtain the corrected matching probability, which is used for subsequent cross-modal matching loss calculation.
[0032] Preferably, the remote sensing image-text retrieval model is used to perform text retrieval of images or image retrieval of text:
[0033] In the text retrieval image mode, the query text is encoded to obtain text features, the remote sensing images in the image library are pre-encoded to obtain image features and stored in the feature library, the total similarity between the query text features and each image feature is calculated, sorted from high to low similarity, and the top K images are returned as the retrieval results.
[0034] In the image retrieval text mode, the query image is encoded to obtain image features, and the descriptive text in the text library is pre-encoded to obtain text features. The total similarity between the query image features and each text feature is calculated, and the results are sorted from high to low similarity. The top K descriptive texts are returned as the retrieval results.
[0035] Preferably, the number of Transformer layers in fine-grained feature learning is configured to be 3; and / or, the end-to-end training uses the Adam or AdamW optimizer for parameter updates.
[0036] The technical solution of this invention has one or more of the following beneficial technical effects:
[0037] (1) A hierarchical semantic learning framework integrating coarse and fine granularity was constructed. The overall semantic distribution was constrained by coarse-grained global classification loss and cross-modal matching loss, and fine-grained local alignment loss was combined to mine detailed semantic information, thereby realizing joint optimization learning of multi-dimensional semantics. This not only ensured the generalization ability of cross-modal retrieval, but also improved the accuracy of fine-grained semantic matching, effectively improving the overall performance and scene adaptability of remote sensing image and text retrieval.
[0038] (2) A targeted semantic alignment mechanism is introduced. Fine-grained region-word matching pairs are filtered by target correction weight. Only the matching signals of core words and corresponding key image regions are retained, and redundant semantic interference caused by background regions and non-core words is eliminated. While maintaining the fine-grained matching accuracy, redundant computational overhead is reduced, which is suitable for the scene characteristics of remote sensing images with high background ratio and sparse and scattered ground objects.
[0039] (3) A hierarchical prompting optimization mechanism is introduced. Multi-layer prompting vectors are generated by mapping network and embedded into each hidden layer of the pre-trained Transformer. Only a small number of projection parameters need to be fine-tuned to achieve rapid adaptation of the general visual language pre-trained model to the remote sensing scene. There is no need to perform full fine-tuning of the pre-trained model. Under the premise of reducing training computing power cost, the transfer effect of pre-trained semantic knowledge to the remote sensing field is effectively improved.
[0040] Furthermore, other beneficial effects of the present invention will be mentioned in the specific embodiments. Attached Figure Description
[0041] Figure 1 This is a schematic diagram of the overall architecture of the present invention;
[0042] Figure 2 This is a diagram comparing the effects of different semantic alignment methods;
[0043] Figure 3 This is a schematic diagram of the feature pooling process for targeted semantic alignment in an embodiment of the present invention;
[0044] Figure 4 This is a schematic diagram of the prompting and tuning module of the pre-trained model in an embodiment of the present invention;
[0045] Figure 5 These are convergence curves of three types of losses in embodiments of the present invention;
[0046] Figure 6 This is an example diagram illustrating the effect of performing a text retrieval image task according to an embodiment of the present invention;
[0047] Figure 7 This is an example diagram illustrating the experimental results of image-to-text retrieval in an embodiment of the present invention. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0049] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order.
[0050] This embodiment provides a remote sensing image-text retrieval method based on hierarchical targeted semantic alignment, applied to the Remote Sensing Image-Text Retrieval (RSITR) task. It can retrieve matching images from a remote sensing image database based on text descriptions, and can also retrieve corresponding descriptive text based on remote sensing images. This invention uses a Contrastive Language-Image Pretraining (CLIP) model as its backbone, constructing a hierarchical semantic learning framework that includes coarse-grained global semantics and fine-grained local semantics. Simultaneously, a targeted semantic alignment mechanism is introduced in the fine-grained matching stage to filter core matching pairs to suppress background redundancy interference. A hierarchical prompting optimization mechanism is used to improve the adaptability of the pre-trained model to remote sensing scenes. Finally, end-to-end training with multiple losses is used to improve retrieval performance. Under the premise of reasonable control of computational overhead, the accuracy and generalization ability of RSITR are effectively improved. Figure 1 This is a schematic diagram of the overall architecture of the present invention. (See diagram below.) Figure 1 As shown, after the input image-text case pair is encoded as image and text respectively, the input hierarchical semantic learning module completes multi-dimensional semantic alignment. Combined with targeted semantic alignment optimization and prompt-based tuning and adaptation, the model is iteratively optimized through a joint loss function.
[0051] Before semantic alignment learning, the input remote sensing image and descriptive text are first feature-encoded using an Image Sampling Network (ImgNet) and a Text Sampling Network (TxtNet), respectively, to obtain corresponding multi-scale feature sequences. Specifically, this embodiment defines the remote sensing image-text retrieval dataset as follows: , where d n This represents the original nth image-text pair, where N is the total number of samples in the dataset.
[0052] For the input remote sensing image `img`, the image sampling network, based on the visual Transformer architecture, divides the image into multiple image patches and then extracts features, outputting an image feature sequence containing global features and multiple regional local features, which can be represented as:
[0053] ;
[0054] Among them, I x f represents the image feature sequence corresponding to the input remote sensing image; i (·) represents the feature extraction function of the image sampling network; The global features of the input image correspond to the overall semantic representation of the entire remote sensing image; , … These are the local features corresponding to the 1st, 2nd, ..., wth image regions, respectively. represents the trainable parameters of the image sampling network; the superscript T indicates the vector transpose operation; img is the input remote sensing image.
[0055] Accordingly, for the input descriptive text txt, the text sampling network performs word segmentation and encoding based on the Transformer architecture, outputting a text feature sequence containing global features and multiple word local features, which can be represented as:
[0056] ;
[0057] Among them, T y The input describes the text feature sequence corresponding to the text; f t (·) represents the feature extraction function of the text sampling network; The global features of the input text correspond to the overall semantic representation of the entire description. , … The local features are, in order, the 1st, 2nd, ..., zth words; represents the trainable parameters of the text sampling network; the superscript T indicates the vector transpose operation; txt is the input descriptive text.
[0058] In this embodiment, both the image sampling network and the text sampling network are initialized with the corresponding encoder of the pre-trained CLIP model to transfer the semantic representation capabilities of the pre-trained general visual language.
[0059] The hierarchical semantic learning module of this invention comprises a two-layer structure: coarse-grained global learning and fine-grained local learning. The coarse-grained global learning constrains the semantic distribution of cross-modal features at the overall level, and includes cross-modal classification learning and cross-modal matching learning. The purpose of cross-modal classification learning is to improve the inter-class discriminative power of intra-modal features, constraining the feature distribution of samples of the same class to be more compact and the feature distribution of samples of different classes to be more dispersed through classification loss.
[0060] For a batch of training data containing N image-text pairs, the classification probability is calculated based on the global features of the images and the global features of the text, and the classification loss L is calculated. c It can be represented as:
[0061] ;
[0062] in, Here are the weight parameters of the classifier, and here are the trainable parameters; This represents the softmax activation function, used to map features to classification probabilities; N is the total number of image-text pairs in a training batch; x is the index of the image-text pair in the batch, ranging from 1 to N. The global feature corresponding to the x-th image; Let be the global feature corresponding to the x-th text; c is the indicator matrix, which takes a value of 1 when the corresponding image-text pair is a matching pair (c xy =1), and the value is 0 when there is no matching pair (c xy =0). By constraining the classification loss, the encoder can learn more discriminative global semantic features, reducing semantic confusion between different scene categories.
[0063] Cross-modal matching learning is used to align the feature distributions of image and text modalities, enabling matched image-text pairs to have higher similarity in a shared semantic space. This is first achieved based on global image features. With text global features Calculate the cosine similarity sim(·), and then adjust the sharpness of the probability distribution using a temperature coefficient τ to obtain the text-to-image matching probability p. x,y :
[0064] ;
[0065] Correspondingly, the image-to-text matching probability q x,y It can be represented as:
[0066] ;
[0067] Where, p x,y Let q be the text-to-image matching probability between the x-th image and the y-th text; x,y is the image-to-text matching probability between the x-th image and the y-th text; sim(·) is the cosine similarity calculation function, used to measure the semantic similarity between two feature vectors; τ is the temperature coefficient, used to adjust the sharpness of the matching probability distribution; k is the sequence index of the text / image in the batch, with a value range from 1 to N; exp(·) is the natural exponential function.
[0068] The Kullback-Leibler (KL) divergence is used to measure the difference in matching probability distributions, and the text-to-image matching loss L... m-t2i for:
[0069] ;
[0070] Similarly, the image-to-text matching loss L m-i2t It can be constructed in a symmetrical manner:
[0071] ;
[0072] Among them, L m-t2i L represents the text-to-image matching loss value. m-i2t Image-to-text matching loss; KL(·‖·) is the KL divergence calculation function, used to measure the degree of difference between two probability distributions; p x Let q be the text-to-image probability distribution corresponding to the x-th image; x Let ξ be the probability distribution from image to text corresponding to the x-th image; ξ is a smoothing constant to prevent numerical instability and to avoid cases where the denominator is 0; log(·) is the natural logarithm function.
[0073] The final total matching loss L m The sum of losses in both directions:
[0074] ;
[0075] By using bidirectional matching loss constraints, the global features of images and text can be accurately aligned in a shared semantic space.
[0076] Building upon coarse-grained global learning, the hierarchical semantic learning module further captures richer local semantic correspondences through fine-grained feature learning. Figure 2 This is a diagram comparing the effects of different semantic alignment methods. For example... Figure 2 As shown, the differences in performance between different alignment granularities are clearly illustrated. Coarse-grained semantic alignment performs global feature matching between the image and text, assigning matching weights to semantic elements such as airplanes, airports, and grasslands based on their contribution to the overall scene. This approach is highly efficient but lacks focus on key targets. Fine-grained semantic alignment aligns each region of the image with each word of the text, capturing detailed semantics. However, in remote sensing images where background areas are prevalent and targets are sparse, this mode easily includes background areas such as grasslands and open spaces with non-core words in the matching process, generating redundant semantic interference and diluting the matching weight of key features, thus affecting the final retrieval accuracy. The targeted semantic alignment introduced in this embodiment, on the other hand, only retains the image region with the highest semantic matching degree for each query word in the final similarity calculation, eliminating interference from other background areas. This preserves the accuracy of fine-grained matching while avoiding the dilution of core semantics by redundant information.
[0077] A feasibility study of the mechanism for targeted semantic alignment is conducted from a probabilistic perspective: Let... This indicates that the query word is t. y The probability that the semantics of the input image (img) are consistent. This indicates that the query word is t. y With x image regions i x The probability of semantic consistency. Generally, the probability of semantic consistency is positively correlated with the similarity value, i.e. Furthermore, each image region contributes positively or neutrally to the overall similarity. Assuming that the sampling of each region in the image is independent, we can deduce that:
[0078] ;
[0079] Where i k To match the query word t y The image region with the highest semantic similarity. This formula shows that retaining only the semantic consistency probability of the region with the highest similarity is the lower bound of the semantic consistency probability of the entire image. That is, targeted screening can theoretically guarantee the effectiveness of core semantic matching, while redundant full-region matching will introduce background noise and dilute the matching weight of key features. In practical implementation, each word is equivalent to a detector, performing targeted detection at different locations in the image and recording the best response, which can ensure the matching credibility of local features.
[0080] Specifically, this embodiment introduces a targeted semantic alignment mechanism in fine-grained feature learning. Figure 3 This is a schematic diagram of the feature pooling process for targeted semantic alignment in an embodiment of the present invention; as shown below. Figure 3 As shown. First, the cosine similarity between all image region features and all word features is calculated, resulting in a region-text similarity matrix S of dimension w×z, where the element in the x-th row and y-th column of S corresponds to the similarity between the x-th image region and the y-th word. Then, for each query word, targeted filtering is performed using target correction weights. The calculation method is as follows:
[0081] ;
[0082] in, The target correction weights are the x-th image region and the y-th word; y is the index of the query word. The word index corresponding to the image region; For the x-th image region and the th The semantic similarity of words; arg max(·) is the operation to take the index corresponding to the maximum value; N is the total number of words; otherwise indicates the other cases.
[0083] That is, for the y-th word, only the matching weight of the image region with the highest semantic similarity is retained, and the weights of the remaining regions are reset to 0. Based on the target correction weights, the filtered key query features can be obtained. :
[0084] ;
[0085] in, Let i be the key query feature after filtering for the y-th word;x These are local features of the x-th image region; The target correction weights are the x-th image region and the y-th word.
[0086] Therefore, the word t y Target similarity s(t) between the target image and the corresponding image (img) y (, img) can be represented as:
[0087] ;
[0088] Wherein s(t) y (, img) is the word t y Target similarity with the input image; t y Let be the local feature of the y-th word; represent the vector dot product operation; ‖·‖ represents the L2 norm operation of the vector; k is the value of the vector with respect to the query word t. y The index of the most relevant image region; max(·) is the maximum value operation.
[0089] After calculating the targeted similarity of individual words, the matching signals of all words are aggregated through pooling operations to obtain the fine-grained similarity between the entire text and the entire image.
[0090] Under the theoretically optimal configuration of fine-grained feature aggregation, this embodiment uses log-sum-exponential pooling (LSE pooling) for aggregation. The final similarity s(img, txt) of the fine-grained cross-modal matching is:
[0091] ;
[0092] Where s(img, txt) is the fine-grained final similarity between the image and the text; λ is the scaling factor for adjusting the weights of the most relevant semantics; S is the region-text similarity matrix; the meanings of the remaining variables are the same as those in the previous text.
[0093] Fine-grained alignment uses a bidirectional ranking loss as the training objective. For a matched image-text pair (img, txt), the hardest negative sample in the batch is set to img. ﹣ With txt ﹣ Two-way ranking loss L a It can be represented as:
[0094] ;
[0095] Where γ is the margin parameter, used to control the similarity interval between positive and negative samples; img ﹣ The negative sample image in the batch that is the hardest to match the input text; txt ﹣The negative sample text in the batch that is the hardest to match with the input image; [·] + This represents the hinge loss function, equivalent to max(x, 0), meaning that loss occurs only when the value within the parentheses is greater than 0.
[0096] By using targeted semantic alignment and ranking loss constraints, the model can focus on the semantic matching of core features and suppress the interference of redundant background information.
[0097] To improve the adaptability of the general CLIP pre-trained model to remote sensing scenarios, this invention introduces a hierarchical prompting optimization mechanism during the feature learning stage. Figure 4 This is a schematic diagram of the prompting and tuning module for the pre-trained model in an embodiment of the present invention. Figure 4 As shown, the initial multimodal features M (including image and text features) are first input into the mapping network. Multi-layered cue learning vectors are generated through two layers of learnable projection parameters. Each cue vector corresponds to a Transformer layer in the pre-trained language model. Cue features M P The calculation method is as follows:
[0098] ;
[0099] Among them, M P For the generated multi-layered cue features; W in With W out represents the learnable projection parameters of the mapping network, and represents the input layer weights and output layer weights, respectively; M represents the multimodal features of the input, including image features and text features; ReLU(·) is the linear rectified activation function used to introduce nonlinear transformation.
[0100] The cue vectors from each layer are then embedded into the corresponding hidden layers of the pre-trained Transformer, and together with the features of the original text input, they are used for forward computation. The final output is a contextual representation C that incorporates remote sensing scene adaptation information.
[0101] ;
[0102] in, f represents the parameters of the pre-trained language model. p (·) indicates the forward computation process of the pre-trained model.
[0103] The obtained contextual representation C can be incorporated into the cross-modal matching calculation process to correct the global matching probability. The corrected matching probability for:
[0104] ;
[0105] Among them, P pro The semantic correction probability is obtained based on the context representation:
[0106] ;
[0107] in, Let be the semantic correction probability of the y-th word given a word sequence and its context representation; CRF(·) is the Conditional Random Field (CFR) computation function used to model the probability distribution of sequence annotations. This hinting tuning mechanism requires only a small number of parameters to train the mapping network, without needing to perform full fine-tuning on the CLIP pre-trained model. It allows pre-trained semantic knowledge to quickly adapt to scene characteristics such as remote sensing land cover morphology and spatial distribution, improving transfer learning performance while controlling training costs.
[0108] The total training loss L of this method is composed of the coarse-grained classification loss L. c Coarse-grained matching loss L m fine-grained alignment loss L a The summation of three parts constitutes:
[0109] ;
[0110] A hierarchical semantic learning mechanism is achieved by simultaneously guiding global semantic distribution optimization and local fine-grained alignment optimization through a joint loss function. Model training employs a batch-by-batch iterative optimization approach, as detailed below:
[0111] First, randomly initialize the adaptation parameters of the image sampling network. Adaptation parameters of text sampling network The system includes the following steps: First, it prompts the mapping parameters of the optimization module; second, it initializes the hyperparameters temperature coefficient τ, smoothing constant ξ, and margin parameter γ; third, it sets the learning rate κ and the maximum number of iterations MaxNum. During each iteration, a batch of image-text matching pairs is randomly sampled from the training dataset. Global and local features of the image and text are extracted using the image sampling network and text sampling network, respectively. The optimization module generates contextual representations and corrects the matching probabilities. Finally, the classification loss L is calculated sequentially. c Matching loss L m Alignment loss L a The total loss L is obtained by summing the results. The gradient of the total loss with respect to each trainable parameter is calculated using the backpropagation algorithm. The Adam optimizer is used to update the adaptation parameters of the image sampling network and the text sampling network, as well as the mapping parameters of the prompt tuning module. The above iterative process is repeated until the total loss converges or the preset maximum number of training rounds is reached. The final model parameters are then saved for subsequent retrieval tasks.
[0112] To verify the effectiveness of the proposed hierarchical targeted semantic alignment-based remote sensing image retrieval method (hereinafter referred to as HTSA for convenience), this embodiment conducts comparative and ablation experiments on publicly available standard datasets in the field of remote sensing image retrieval to verify the retrieval performance, the role of each module, and the convergence characteristics of the method.
[0113] The experiment used two publicly available benchmark datasets, RSICD and RSITMD. Each remote sensing image in both datasets corresponds to five manually annotated descriptive texts. The RSICD dataset contains 10,921 remote sensing images, which were divided into training, validation, and test sets in a 7:2:1 ratio, with corresponding sample sizes of 7,862, 1,966, and 1,093 images, respectively. The RSITMD dataset contains 4,743 remote sensing images, also divided in a 7:2:1 ratio, with corresponding sample sizes of 3,435, 856, and 452 images in the training, validation, and test sets, respectively. The experiment used recall@K (Recall@K, R@K) and mean recall mR as evaluation metrics, where R@K represents the proportion of correctly matched results appearing in the top K search results. K takes values of 1, 5, and 10, corresponding to the three metrics R@1, R@5, and R@10, respectively measuring the precision of the first hit, the recall of the top five, and the recall of the top ten. The mean recall (mR) is the average of all R@K metrics in both the image-to-text and text-to-image directions, and is calculated using the following formula:
[0114] ;
[0115] Where, (R@1 + R@5 + R@10) I2T The sum of the three recall metrics in the image-to-text direction is (R@1 + R@5 + R@10). T2I mR is the sum of three recall metrics in the text-to-image direction, used to comprehensively measure the overall retrieval performance of the model.
[0116] The experiment was implemented using Python 3.8.13 and the PyTorch 1.7.1 deep learning framework, powered by the CUDA 11.0 parallel computing architecture, and ran on an NVIDIA A100 Tensor Core GPU with 40GB of VRAM. Both the image sampling network and the text sampling network used a pre-trained CLIP model as their initial backbone. The input image size was uniformly adjusted to 256×256 pixels, and the image patch size was 16×16. The Adam optimizer was used, with the training batch size set according to the VRAM configuration, and the maximum number of training epochs set to 20. The hyperparameters were set as follows: the temperature coefficient τ for matching probability calculation was set to 0.02, and the smoothing constant ξ for KL divergence calculation was set to 1×10⁻. 8The margin parameter γ for ranking loss is set to 1.05.
[0117] To demonstrate the retrieval performance advantages of the method of this invention, several existing remote sensing image and text retrieval methods were selected as comparative baselines, including traditional methods based on deep convolutional neural networks and transfer learning methods based on CLIP pre-trained models. All comparative methods used the same ViT-B-32 visual backbone and corresponding text backbone to ensure the fairness of the comparison. The quantitative comparison results on the two datasets are shown in Tables 1 and 2, respectively.
[0118] Table 1 Quantitative experiments on the RSICD dataset
[0119]
[0120] Table 2 Quantitative experiments on the RSITMD dataset
[0121]
[0122] Further analysis of the experimental data reveals that different feature sampling baselines significantly affect the ability of remote sensing image matching: for example, CLIP-based methods such as CLGSA and CMER generally outperform methods based on deep convolutional neural networks. However, some traditional methods (such as KAMCL) can also outperform some CLIP-based methods on specific metrics. This reflects that relying solely on deep fine-grained classifiers without targeted semantic mining of core objects cannot fully unleash the performance potential of pre-trained models.
[0123] Experimental results show that the CLIP-based pre-trained model outperforms traditional convolutional neural network methods overall, validating the advantages of pre-trained visual language models in cross-modal semantic representation. The proposed method achieves competitive retrieval performance on both datasets. On the RSIMD dataset, the proposed method achieves an average recall (mR) of 47.15%, which is 0.27 percentage points higher than the CMER method with the same backbone architecture and 0.52 percentage points higher than the CLGSA method. On the RSICD dataset, the proposed method also ranks among the top performers. In five out of six sub-indicators for image-to-text and text-to-image retrieval, the proposed method ranks in the top two, demonstrating that hierarchical semantic learning and targeted alignment strategies effectively improve the lower bound of retrieval performance and ensure model stability in different scenarios. Compared to large-scale model methods that require additional large-scale training data, the proposed method achieves performance levels close to these levels through lightweight adaptation and alignment optimization, demonstrating higher resource efficiency and practical value. Meanwhile, stable retrieval performance can still be obtained by replacing the visual backbone with different specifications (such as ViT-B-16), which verifies that the method has good model portability and can be adapted to deployment scenarios with different computing power.
[0124] To further compare the performance differences between our method and large models in the field of remote sensing, we compared it with a series of methods based on the GeoRSCLIP large model on the RSICD dataset. The results are shown in Table 3.
[0125] Table 3. Comparison of HTSA and Large Language Models on the RSICD dataset.
[0126]
[0127] Combining the results in Tables 1 and 3, it can be seen that the GeoRSCLIP series of large models can achieve better retrieval performance through the accumulation of large-scale pre-training data. However, the method of this invention uses lightweight CLIP as the backbone and can approach the performance level of large models by simply performing hierarchical alignment optimization and targeted screening. It achieves competitive retrieval results with a much lower computational and data cost than large models, and has higher resource efficiency and practical value.
[0128] To analyze the function of each module of this invention and the impact of different configurations, ablation experiments were conducted on the RSICD dataset to verify the results from five dimensions: loss function configuration, core components, pooling strategy, optimization strategy, and number of Transformer layers.
[0129] First, we analyzed the impact of different loss function configurations and verified the effects of classification loss alone, classification plus matching loss, and the combination of the three types of loss. The results are shown in Table 4.
[0130] Table 4. Impact of Loss Function Configuration on Model Performance
[0131]
[0132] As shown in Table 4, the overall matching performance was lowest when only classification loss was used. The performance improved after adding matching loss. Furthermore, the retrieval accuracy was significantly improved after introducing fine-grained alignment loss. This verifies that hierarchical multi-loss learning can mine semantic information from multiple dimensions, and targeted fine-grained learning can achieve accurate semantic adjustment, which plays a key role in improving retrieval accuracy.
[0133] Next, the impact of the core components was analyzed, and the performance of two variants, namely, removing target semantic alignment and removing prompts for optimization, was verified. The results are shown in Table 5.
[0134] Table 5 Impact of some core components
[0135]
[0136] As shown in Table 5, the model performance decreased significantly after removing the target semantic alignment, indicating that the targeted screening of the core region can effectively eliminate redundant semantic interference and make an important contribution to improving retrieval accuracy. The model performance also decreased after removing CLIP prompts, which verifies that hierarchical prompts can improve the adaptability of the pre-trained model to remote sensing scenarios and effectively transfer pre-trained semantic knowledge.
[0137] Further analysis was conducted on the impact of different pooling strategies on fine-grained feature aggregation. Five schemes, namely max pooling, average pooling, summation pooling, softmax pooling, and LSE pooling, were compared. The results are shown in Table 6.
[0138] Table 6. Impact of different pooling strategies on model performance
[0139]
[0140] As shown in Table 6, LSE pooling achieves the best aggregation effect by balancing local salient responses and global contextual information while maintaining differentiability to support gradient training. Average pooling performs second best, while other pooling strategies are relatively weaker. To maintain consistency with the experimental configuration of existing public methods and ensure fairness in the comparison, the main comparative experiments adopted the average pooling configuration. Ablation experiments verified that LSE pooling achieves the best aggregation effect by balancing local salient responses and global contextual information while maintaining differentiability. The comparative experimental results for different optimization strategies are shown in Table 7.
[0141] Table 7. Impact of different optimization strategies on model performance
[0142]
[0143] As shown in Table 7, the AdamW optimizer has slightly better retrieval accuracy than optimizers such as Adam, SGD, RMSProp, and Adamamax, and can slightly improve the model's convergence performance and retrieval accuracy. The main experiments used the Adam optimizer to maintain consistency with mainstream methods.
[0144] The ablation experiment results for the number of Transformer layers in fine-grained feature learning are shown in Table 8.
[0145] Table 8. Impact of different Transformer layer numbers on model performance
[0146]
[0147] As shown in Table 8, when the number of layers is 1, the model fails to fully utilize the region information modeling capability of the Transformer, resulting in low accuracy. With increasing layers, performance peaks at the 3rd layer. Further increases in layers lead to overfitting, causing performance to decline. Therefore, choosing a 3-layer Transformer configuration for fine-grained feature learning ensures performance while avoiding overfitting.
[0148] To verify the convergence of the model training process, the variation curves of classification loss, matching loss, and alignment loss during training were recorded on the RSICD dataset. Figure 5 These are convergence curves for three types of losses according to embodiments of the present invention. Figure 5 As shown, all three loss functions converged rapidly to stable values within 20 training epochs. The matching loss exhibited the most significant decrease, indicating rapid optimization of global cross-modal matching. The fine-grained alignment loss, with its relatively small value, decreased at a relatively moderate pace. The classification loss, acting as a coarse-grained constraint, decreased less overall but also showed a stable convergence trend. The synergistic convergence of the three losses validates that the joint loss function effectively guides model training, demonstrating the stable optimization characteristics of the hierarchical semantic learning framework.
[0149] In practical retrieval applications, two modes are supported: text-based image queries and image-based text queries. For text-based image queries, the input query text is encoded using a text sampling network to obtain text features. All remote sensing images in the image database are pre-encoded using an image sampling network to obtain image features and stored in the feature library. The total similarity between the query text features and each image feature is calculated, and the images are sorted from highest to lowest similarity. The top K images are returned as the retrieval results. For image-based text queries, the similarity between the query image features and each text feature in the text database is calculated similarly, and the top K descriptive texts are returned.
[0150] Figure 6 This is an example diagram illustrating the effect of performing a text retrieval image task according to an embodiment of the present invention. For example... Figure 6 As shown, Figure 6 Part (a) represents the search results from the RSICD dataset. Figure 6 Part (b) shows the retrieval results from the RSITMD dataset. In each example, the upper part is the input text description, and the lower part is the top 10 retrieved images sorted by similarity. Green borders represent images with correct semantic matching, and red borders represent images with incorrect semantic matching.
[0151] Figure 7 These are example images illustrating the experimental results of image-to-text retrieval according to an embodiment of the present invention. Figure 7 As shown, Figure 7 Part (a) represents the search results from the RSICD dataset. Figure 7Part (b) shows the retrieval results from the RSITMD dataset. In each example, the left side is the input remote sensing image, and the right side is the top 5 search texts sorted by similarity. A "√" indicates a correct semantic match, and an "×" indicates an incorrect semantic match. Experimental results show that this method can effectively recall matching targets in complex remote sensing scenarios, demonstrating good retrieval accuracy and generalization ability.
[0152] This invention also offers several alternative configuration options, all of which can achieve the core hierarchical targeted semantic alignment function:
[0153] In the fine-grained feature aggregation stage, LSE pooling can be replaced with max pooling, average pooling, summation pooling, or softmax pooling. Average pooling can achieve better performance while ensuring fairness in comparison, while max pooling is suitable for retrieval tasks in strong target scenarios. In the model training stage, the Adam optimizer can be replaced with SGD, RMSProp, Adammax, or AdamW optimizers. The AdamW optimizer can slightly improve the model's convergence effect and retrieval accuracy. The image and text sampling networks can use CLIP pre-trained backbones of different specifications (such as ViT-B-16 and ViT-B-32) to adapt to different computing power and accuracy requirements. The number of Transformer layers in the fine-grained feature learning module can be configured to 1, 3, or 5 layers, with 3 layers achieving the optimal balance between performance and overfitting risk.
[0154] To better illustrate the present invention, numerous specific details have been provided in the detailed embodiments described above. Those skilled in the art should understand that the present invention can be practiced even without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of the present invention.
[0155] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A remote sensing image and text retrieval method based on hierarchical targeted semantic alignment, characterized in that, Includes the following steps: The input remote sensing image is feature-encoded using an image sampling network, and the output is an image feature sequence containing global image features and local features of multiple image regions; the input descriptive text is feature-encoded using a text sampling network, and the output is a text feature sequence containing global text features and local features corresponding to multiple words. Based on the global features of the image and the global features of the text, coarse-grained global semantic alignment is performed to obtain cross-modal classification loss for constraining the discriminative power between feature classes within the modality, and cross-modal matching loss for aligning the distributions of global features of the image and global features of the text. The semantic similarity between the local features of each image region and the local features of all words is calculated to construct a region-text similarity matrix. For each word, the semantic matching results of all image regions corresponding to that word are targeted and filtered by a target correction weight. The target correction weight satisfies the following condition: the weight is 1 if and only if the image region is the region with the highest semantic similarity to the word, and the weight of the other image regions is 0. Based on the filtered semantic matching results, the target similarity between the word and the remote sensing image is calculated. The target similarity of all words is aggregated to obtain a fine-grained cross-modal similarity, and a fine-grained alignment loss is obtained. The cross-modal classification loss, cross-modal matching loss, and fine-grained alignment loss are summed to construct a joint loss function. The image sampling network and text sampling network are trained end-to-end with the goal of minimizing the joint loss, resulting in a remote sensing image-text retrieval model for bidirectional retrieval of remote sensing images and descriptive text.
2. The remote sensing image and text retrieval method based on hierarchical targeted semantic alignment according to claim 1, characterized in that, The target correction weights are determined as follows: ; in, The target correction weights are the x-th image region and the y-th word; y is the index of the query word. The word index corresponding to the image region; For the x-th image region and the th The semantic similarity of words; arg max(·) is the operation that takes the index corresponding to the maximum value; otherwise represents the other cases.
3. The remote sensing image and text retrieval method based on hierarchical targeted semantic alignment according to claim 2, characterized in that, The target similarity is calculated as follows: The selected image region features are subjected to a normalized dot product operation with the local features of the corresponding word to obtain the target similarity between the word and the remote sensing image; the normalized dot product operation is to calculate the dot product after normalizing the two feature vectors by L2 norm respectively.
4. The remote sensing image and text retrieval method based on hierarchical targeted semantic alignment according to claim 3, characterized in that, The fine-grained cross-modal similarity is obtained by aggregating the targeted similarity of all words, and is implemented using logarithmic exponential pooling, including: The target similarity of each word is multiplied by a scaling factor and then exponentially calculated. The sum of the exponential calculation results of all words is then logarithmically calculated and divided by the scaling factor to obtain the fine-grained cross-modal similarity. The scaling factor is used to adjust the weight of the most relevant semantics in similarity aggregation.
5. The remote sensing image and text retrieval method based on hierarchical targeted semantic alignment according to claim 1, characterized in that, The cross-modal matching loss is calculated as follows: Calculate the matching loss from text to image direction and the matching loss from image to text direction, respectively; The text-to-image matching probability distribution is calculated based on global image features and global text features. KL divergence is used to measure the difference between the text-to-image matching probability distribution and the image-to-text matching probability distribution, and the matching loss in the text-to-image direction is obtained. The image-to-text matching probability distribution is calculated based on global image features and global text features. KL divergence is used to measure the difference between the image-to-text matching probability distribution and the text-to-image matching probability distribution, and the matching loss in the image-to-text direction is obtained. The cross-modal matching loss is obtained by summing the matching loss from text to image and the matching loss from image to text.
6. The remote sensing image and text retrieval method based on hierarchical targeted semantic alignment according to claim 1, characterized in that, The fine-grained alignment loss employs a bidirectional ranking loss, including: For a matched image-text pair, select the negative sample image with the lowest text similarity to the matched image-text pair from the non-matching pairs in the training batch, and the negative sample text with the lowest image similarity to the matched image-text pair; Based on the difference between the image-text pair similarity of positive matching and the image-text pair similarity of negative matching, the ranking loss in the text-to-image direction and the ranking loss in the image-to-text direction are calculated respectively. The ranking losses in the two directions are summed to obtain the fine-grained alignment loss.
7. The remote sensing image and text retrieval method based on hierarchical targeted semantic alignment according to claim 1, characterized in that: Both the image sampling network and the text sampling network use the corresponding encoder of the pre-trained contrastive language image pre-trained CLIP model for parameter initialization; during the training phase, only the hidden layers of the CLIP model are adapted by generating hierarchical cue vectors to output the contextual representation that integrates the semantic information of the remote sensing scene, without the need for full fine-tuning of the CLIP model.
8. The remote sensing image and text retrieval method based on hierarchical targeted semantic alignment according to claim 7, characterized in that, The semantic matching results are corrected based on contextual representation, including: The context representation input conditional random field is used to perform sequence labeling probability modeling to obtain the semantic correction probability of each word; the semantic correction probability is multiplied and fused with the original matching probability element by element to obtain the corrected matching probability, which is used for subsequent cross-modal matching loss calculation.
9. The remote sensing image and text retrieval method based on hierarchical targeted semantic alignment according to any one of claims 1 to 8, characterized in that, The remote sensing image-text retrieval model is used to perform text retrieval of images or image retrieval of text. In the text retrieval image mode, the query text is encoded to obtain text features, the remote sensing images in the image library are pre-encoded to obtain image features and stored in the feature library, the total similarity between the query text features and each image feature is calculated, sorted from high to low similarity, and the top K images are returned as the retrieval results. In the image retrieval text mode, the query image is encoded to obtain image features, and the descriptive text in the text library is pre-encoded to obtain text features. The total similarity between the query image features and each text feature is calculated, and the results are sorted from high to low similarity. The top K descriptive texts are returned as the retrieval results.
10. The remote sensing image and text retrieval method based on hierarchical targeted semantic alignment according to any one of claims 1 to 8, characterized in that: In fine-grained feature learning, the Transformer layer is configured with 3 layers; And / or, the end-to-end training uses the Adam or AdamW optimizer for parameter updates.
Citation Information
Patent Citations
Cross-modal remote sensing image-text retrieval method based on explicit and implicit semantic collaborative guidance
CN121919374A
