A cross-modal remote sensing image-text retrieval method with explicit-implicit semantic collaborative guidance

Abstract

The present application relates to the technical field of remote sensing image processing, in particular to a kind of explicit and implicit semantic collaborative guidance's cross-modal remote sensing image-text retrieval method, comprising: text is masked with context-aware TF-IDF weighting strategy processing;Explicit branch will masked text as implicit semantic embedding and image are fine-grained alignment;Implicit branch by the implicit semantic embedding produced during pre-training and image are adaptively fused features, at the end of implicit branch, integrate a lightweight differentiable adaptive binary projector;Contrast loss produced by explicit branch provides alignment supervision signal to implicit branch to perform final retrieval matching;The present application effectively combines the strong generalization ability of implicit alignment and high precision of explicit alignment through the collaborative interaction of two branches, realizes the high-precision, high-efficiency retrieval of remote sensing image and text, and significantly improves the retrieval performance in complex remote sensing scenarios.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration. Background Technology

[0002] With the development of satellite technology, the types of data are constantly increasing. High-resolution images and large-scale geographic data provided by various remote sensing satellites have created unprecedented Earth observation resources. However, efficiently and accurately extracting useful information from these massive amounts of images has become a key challenge in the field of remote sensing. Because remote sensing images themselves contain rich spatial information and geographic details, image-based retrieval methods often face challenges such as high dimensionality, semantic ambiguity, and scene diversity. At the same time, remote sensing data is usually accompanied by rich metadata, descriptive text, and manual annotations. Joint retrieval of image-text data helps to overcome the limitations of single-modal retrieval and improve the effectiveness of information utilization.

[0003] Despite its significant application potential, cross-modal remote sensing image-text retrieval faces core challenges in its development. The differences between image and text modalities lead to difficulties in semantic alignment, thus affecting retrieval accuracy. Effectively bridging the semantic gap and achieving robust cross-modal matching remain key research focuses in this field.

[0004] Current cross-modal remote sensing image retrieval methods typically follow an implicit semantic alignment paradigm, mapping global feature vectors of text and images to a shared semantic space. These methods can learn robust global representations with good generalization ability. However, they still have limitations in capturing fine-grained semantic correspondences, leading to a persistent semantic gap. In recent years, some studies have attempted to introduce explicit alignment mechanisms to enhance local semantic relationships. However, these methods usually treat explicit alignment as an auxiliary module rather than an independent and systematic supervised branch, limiting their improvement effect. Therefore, balancing explicit and implicit semantic information and improving the quality of fine-grained alignment while maintaining generalization ability remains a key challenge in current research. Based on this, there is an urgent need for a remote sensing image retrieval method to improve retrieval performance in complex remote sensing scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration, which effectively combines the powerful generalization ability of implicit alignment with the high precision of explicit alignment, and significantly improves the retrieval performance in complex remote sensing scenarios.

[0006] To achieve the above objectives, this invention provides a cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration, comprising:

[0007] The text is masked using a context-aware TF-IDF weighted strategy;

[0008] Explicit branches align the masked text with the image in a fine-grained manner as an implicit semantic embedding.

[0009] The implicit branch integrates a lightweight, differentiable, adaptive binarization projector at the end of the implicit branch by adaptively fusing the implicit semantic embeddings generated during pre-training with the image features.

[0010] The contrast loss generated by explicit branching provides alignment supervision signals to implicit branching to perform the final retrieval matching.

[0011] The specific steps for masking text using a context-aware TF-IDF weighted strategy include:

[0012] To calculate the context-dependent inverse document frequency (CIDF), let the target word be denoted as t, the set of context-dependent documents be denoted as C, and the current text to be weighted be denoted as d. The formula for calculating the IDF of word t relative to the context C is as follows:

[0013] ;

[0014] in, It is the total number of documents in the context document set C; It is a smoothing factor to prevent the denominator from being 0;

[0015] When calculating the IF (Term Frequency Index), the distribution of words in the context is taken into account. The calculation formula is as follows:

[0016] ;

[0017] in This refers to the number of times word t appears in document d, where t' is a temporary variable used in the summation symbol ∑, representing each distinct word in document d;

[0018] The final context-aware IF-IDF is the product of inverse document frequency and term frequency, calculated as follows:

[0019] .

[0020] The specific steps involved in the explicit branching of fine-grained alignment between the masked text and the image, using the implicit semantic embedding, include:

[0021] Let the mask position be p, and its corresponding hidden vector be... It is an implicit semantic embedding, v j For each image region model, calculations are required. With all image regions The association is used to find the image evidence that best suggests the masked word, which is achieved through cross-modal attention. The formula for calculating the attention weight is as follows:

[0022] ;

[0023] in, For cosine similarity, It is the temperature coefficient.

[0024] Based on attention weights, image information is aggregated into a single... Related vectors, This can be viewed as the most relevant local visual evidence in the image to the masked word, and the calculation formula is as follows:

[0025] ;

[0026] To predict masked words, the aggregated visual information is fused with the implicit semantics of the text to predict the masked words. The calculation formula is as follows:

[0027] ;

[0028] in, For vector concatenation operations, the text semantics are... and visual evidence They are connected along the feature dimension to form a richer joint representation; FFN(.) is a feedforward neural network;

[0029] Then, the predicted distribution is obtained through a classification layer, calculated using the following formula:

[0030] ;

[0031] in, To output the weight matrix, This is the output bias vector;

[0032] The loss function is cross-entropy loss, which encourages the model to correctly predict the masked words based on the image.

[0033] ;

[0034] in, It is the set of all masked positions in the current text batch. For the real word masked at position p, It is the input image. This is the masked text.

[0035] The implicit branching process involves adaptively fusing implicit semantic embeddings generated during pre-training with image features, and integrating a lightweight, differentiable, adaptive binarization projector at the end of the implicit branch. The specific steps include:

[0036] Image features are denoted as v, and implicit semantic embeddings are denoted as t. e , v and t e To fuse these into a more powerful joint representation, we design an adaptive gated fusion module, as shown in the following formula:

[0037] ;

[0038] ;

[0039] in, and For learnable parameters, (.) represents the Sigmoid activation function, with output values ​​between (0,1). For element-wise multiplication, It is an adaptive, element-wise weight vector. Features after fusion;

[0040] The target hash space is projected using a lightweight feedforward network FFN, and the continuous relaxation representation of the hash code is calculated as follows:

[0041] ;

[0042] in, The projection weight matrix is... This is the projection bias matrix;

[0043] A differentiable approximation function is used, and an adaptive threshold is introduced. The formulas for adaptive threshold generation and differentiable binarization are as follows:

[0044] ;

[0045] ;

[0046] in, This is the weight matrix of the threshold generator. This is the bias vector for the threshold generator. This is the sharpness coefficient. Let i be the i-th component of the continuous feature after projection. The i-th component of the adaptive threshold;

[0047] Encourage soft hash codes To approximate binary representation and reduce quantization error, the quantization loss is calculated using the following formula:

[0048] .

[0049] The specific steps involved in providing alignment supervision signals to the implicit branches through the contrastive loss generated by the explicit branches to perform the final retrieval matching include:

[0050] For a batch of N paired samples, the contrast loss is defined in two directions: image to text and text to image. The formula for calculating the contrast loss is as follows:

[0051] ;

[0052] ;

[0053] ;

[0054] in, For image-to-text contrast loss, For text-to-image contrast loss, Total comparative loss;

[0055] Calculate the global features of explicit branches The similarity matrix between them is calculated using the following formula:

[0056] ;

[0057] The similarity matrix between implicit branch outputs is calculated using the following formula:

[0058] ;

[0059] in, , Let be the binary hash codes of the i-th and j-th samples, respectively. This is the scaling factor;

[0060] To make the similarity relationship of implicit branches mimic the similarity relationship of explicit branches as closely as possible, mean squared error loss is used, and the calculation formula is as follows:

[0061] .

[0062] This invention presents a cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration. The proposed framework consists of two parallel branches: an explicit semantic guidance branch and an implicit semantic guidance branch. The explicit branch establishes fine-grained correspondences between words and image regions, providing precise local supervision signals to guide representation learning, while the implicit branch focuses on learning global cross-modal representations and performing the final retrieval matching. Through the collaborative interaction of these two branches, this invention effectively combines the powerful generalization ability of implicit alignment with the high precision of explicit alignment, achieving high-precision and high-efficiency retrieval of remote sensing images and text, and significantly improving retrieval performance in complex remote sensing scenarios. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0064] Figure 1 This is a flowchart of a cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration according to the present invention.

[0065] Figure 2 This is a general framework diagram of a cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration according to the present invention.

[0066] Figure 3 This is a flowchart of the masking process performed by the context-aware TF-IDF weighting strategy of the present invention.

[0067] Figure 4 This is a flowchart of the lightweight, differentiable, adaptive binarization projector of the present invention.

[0068] Figure 5 This is the visual attention map of the model of the present invention in scene one.

[0069] Figure 6 This is the visual attention map of the model of the present invention in scenario two.

[0070] Figure 7 This is the visual attention map of the model of the present invention in scenario three.

[0071] Figure 8 This is the visual attention map of the model of the present invention in scenario four. Detailed Implementation

[0072] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0073] Please see Figures 1-8This invention provides a cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration, comprising:

[0074] S1 uses a context-aware TF-IDF weighted strategy to mask the text;

[0075] The specific steps include:

[0076] S1.1 First, calculate the context-dependent inverse document frequency (CIDF). Let the target word be t, the context-dependent document set be C, and the current text to be weighted be d. The IDF formula for word t relative to context C is as follows:

[0077] ;

[0078] in, It is the total number of documents in the context document set C; It is a smoothing factor to prevent the denominator from being 0;

[0079] S1.2 When calculating the term frequency (IF), the distribution of words in the context is taken into account. The calculation formula is as follows:

[0080] ;

[0081] in This refers to the number of times word t appears in document d, where t' is a temporary variable used in the summation symbol ∑, representing each distinct word in document d;

[0082] In this step, the distribution of words in the context is considered when calculating the word frequency (IF). For example, if word t is concentrated in d (appearing in key paragraphs), it can be given a higher weight.

[0083] S1.3 The final context-aware IF-IDF is the product of inverse document frequency and term frequency, calculated as follows:

[0084] .

[0085] S2 explicit branching uses the masked text as an implicit semantic embedding to perform fine-grained alignment with the image.

[0086] The specific steps include:

[0087] S2.1 Set the mask position to p, and its corresponding hidden vector is... It is an implicit semantic embedding, v j For each image region model, calculations are required. With all image regions The association is used to find the image evidence that best suggests the masked word, which is achieved through cross-modal attention. The formula for calculating the attention weight is as follows:

[0088] ;

[0089] in, For cosine similarity, It is the temperature coefficient.

[0090] S2.2 aggregates image information into a single entity based on attention weights. Related vectors, This can be viewed as the most relevant local visual evidence in the image to the masked word, and the calculation formula is as follows:

[0091] ;

[0092] S2.3 performs masked word prediction by fusing aggregated visual information with implicit semantics of the text to predict the masked words. The calculation formula is as follows:

[0093] ;

[0094] in, For vector concatenation operations, the text semantics are... and visual evidence They are connected along the feature dimension to form a richer joint representation; FFN(.) is a feedforward neural network;

[0095] Then, the predicted distribution is obtained through a classification layer, calculated using the following formula:

[0096] ;

[0097] in, To output the weight matrix, This is the output bias vector;

[0098] The loss function is cross-entropy loss, which encourages the model to correctly predict the masked words based on the image.

[0099] ;

[0100] in, It is the set of all masked positions in the current text batch. For the real word masked at position p, It is the input image. This is the masked text.

[0101] The S3 implicit branch integrates a lightweight differentiable adaptive binarization projector (DABP) at the end of the implicit branch by adaptively fusing the implicit semantic embeddings generated during pre-training with the image features.

[0102] The specific steps include:

[0103] S3.1 The image features are denoted as v, and the implicit semantic embedding is denoted as t. e , v and t e To fuse these into a more powerful joint representation, we design an adaptive gated fusion module, as shown in the following formula:

[0104] ;

[0105] ;

[0106] in, and For learnable parameters, (.) represents the Sigmoid activation function, with output values ​​between (0,1). For element-wise multiplication, It is an adaptive, element-wise weight vector. Features after fusion;

[0107] S3.2 A lightweight feedforward network FFN is used to project u onto the target hash space. The formula for calculating the continuous relaxation representation of the hash code is as follows:

[0108] ;

[0109] in, The projection weight matrix is... This is the projection bias matrix;

[0110] S3.3 Traditional binarization uses b=sign(h), but the derivative of the sign function is almost zero everywhere, making gradient backpropagation impossible. We adopt a differentiable approximation function and introduce an adaptive threshold. The formulas for adaptive threshold generation and differentiable binarization calculation are as follows:

[0111] ;

[0112] ;

[0113] in, This is the weight matrix of the threshold generator. This is the bias vector for the threshold generator. This is the sharpness coefficient. Let i be the i-th component of the continuous feature after projection. It represents the i-th component of the adaptive threshold.

[0114] S3.4 Encourages soft hash codes To approximate binary representation and reduce quantization error, the quantization loss is calculated using the following formula:

[0115] .

[0116] The contrastive loss generated by the S4 explicit branch provides alignment supervision signals to the implicit branch to perform the final retrieval matching.

[0117] The specific steps include:

[0118] S4.1 For a batch of N paired samples, define the contrast loss in two directions: image to text and text to image. The formula for calculating the contrast loss is as follows:

[0119] ;

[0120] ;

[0121] ;

[0122] in, For image-to-text contrast loss, For text-to-image contrast loss, Total comparative loss;

[0123] S4.2 Calculate the global features of explicit branches The similarity matrix between them is calculated using the following formula:

[0124] ;

[0125] S4.3 Calculate the similarity matrix between implicit branch outputs using the following formula:

[0126] ;

[0127] in, , Let be the binary hash codes of the i-th and j-th samples, respectively. This is the scaling factor;

[0128] In this step, the similarity matrix between the implicit branch outputs is calculated. Since hash codes are compact representations used for retrieval, their similarity calculation should simulate the retrieval metric. Here, a normalized inner product is used for approximation, and the calculation formula is as follows: .

[0129] S4.4 To make the similarity relationship of implicit branches mimic the similarity relationship of explicit branches as closely as possible, mean squared error loss is used, and the calculation formula is as follows:

[0130] .

[0131] The following are the experimental data, and the table below compares the performance of different models:

[0132]

[0133] In the table above, all data are expressed as a percentage (%). The terms are as follows: Method: [Method name missing]. Image to Text: Image-to-text retrieval. Text to Image: Text-to-image retrieval. RemoteCLIP: A foundational model for cross-modal retrieval based on the CLIP architecture, continuously pre-trained on remote sensing data. WSSCN: Proposes a complete semantic sparse coding network for constructing comprehensive and reliable feature representations. SGPD: Transforms dense vectors into efficient sparse representations for retrieval through a sparsity-guided partially dense strategy. EISCG: An abbreviation for Explicit-Implicit Semantic Collaborative Guidance, a method proposed in this application. It proposes an Explicit-Implicit Semantic Collaborative Guidance (EISCG) framework that improves retrieval accuracy and efficiency by constructing parallel explicit and implicit semantic branches.

[0134] R@K (Recall@K) represents the proportion of correct matches appearing in the top K search results across all query samples. R@1 (Recall@1) represents whether the correct result appears in the first (highest ranked) position.

[0135] R@5 (Recall@5) indicates whether the correct result appears in the first 5 results.

[0136] R@10 (Recall@10) indicates whether the correct result appears in the first 10 results.

[0137] Specifically, R@K (including R@1, R@5 and R@10) is used to measure whether the top K candidate results returned by the model contain real matches for a given query. In essence, it is a measure of the recall capability of the retrieval system under the condition of ranking truncation.

[0138] This application employs two evaluation metrics, R@K and mR, to assess the performance of our model. R@K (including R@1, R@5, and R@10) measures the proportion of true values ​​appearing in the top K results. The mR metric represents the average of all R@K values, providing a more comprehensive assessment of the overall effectiveness of the model.

[0139] Experimental data show that EISCG demonstrates comprehensive advantages across multiple metrics, particularly in text-to-image retrieval tasks, where its R@5 (67.09%) and R@10 (96.80%) are significantly higher than all comparable methods. Furthermore, EISCG achieves the best performance in mean recall (mR) at 62.36%, outperforming SGPD (58.09%), WSCN (59.03%), and RemoteCLIP (56.36%) by 3.38%, 2.44%, and 6%, respectively.

[0140] Figure 5 The visual attention map of the model proposed in this application is shown in Scene 1, which is a bridge spanning a river with vehicles passing through it and surrounded by buildings on both sides. Figure 5 The four sub-images in the image, from left to right, are: the original remote sensing image showing the overall scene of the bridge spanning the river; the model's attention response to the keyword "bridge"; the model's attention response to "river"; and the model's response to "vehicle".

[0141] Figure 6 The visual attention map of the model proposed in this application is shown in Scene 2, which is a pond surrounded by roads and trees, with residential buildings visible on the right. Figure 6 The four sub-images in the image, from left to right, are: the original image showing the pond and its surrounding environment, the attention map to the "pond", the response map to the "buildings", and the response map to the "road".

[0142] Figure 7 The visual attention map of the model proposed in this application is shown in scenario three, which is a residential area containing multiple buildings, roads running through it, and trees distributed around the houses. Figure 7 The four sub-images in the image, from left to right, are: the original image showing the overall layout of the residential area, a schematic diagram of the model recognizing the "building complex", a response map to "trees", and an attention map to "roads".

[0143] Figure 8 The visual attention map of the model proposed in this application is shown in scene four, which is a church surrounded by buildings and roads. Figure 8 The four sub-images in the image, from left to right, are: the original image showing the church structure at the center, the response image to "church", the response image to "surrounding buildings", and the response image to "road".

[0144] refer to Figures 5-8This highlights the model's response to individual words in the input text. It is clearly observed that EISCG accurately identifies different objects within the same image. For geographic elements such as rivers, ponds, and trees, attention is precisely focused on the corresponding natural areas in the image, indicating the model's strong scene-level geographic semantic awareness. For man-made objects such as buildings, vehicles, and roads, the model effectively distinguishes them and assigns different attention patterns, even if these objects are spatially adjacent or overlapping.

[0145] This invention presents a cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration. The proposed framework consists of two parallel branches: an explicit semantic guidance branch and an implicit semantic guidance branch. The explicit branch establishes fine-grained correspondences between words and image regions, providing precise local supervision signals to guide representation learning, while the implicit branch focuses on learning global cross-modal representations and performing the final retrieval matching. Through the collaborative interaction of these two branches, this invention effectively combines the powerful generalization ability of implicit alignment with the high precision of explicit alignment, achieving high-precision and high-efficiency retrieval of remote sensing images and text, and significantly improving retrieval performance in complex remote sensing scenarios.

[0146] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration, characterized in that, include: The text is masked using a context-aware TF-IDF weighted strategy; Explicit branches align the masked text with the image in a fine-grained manner as an implicit semantic embedding. The specific steps include: Let the mask position be p, and its corresponding hidden vector be... It is an implicit semantic embedding, v j For each image region model, calculations are required. With all image regions The association is used to find the image evidence that best suggests the masked word, which is achieved through cross-modal attention. The formula for calculating the attention weight is as follows: ； in, For cosine similarity, It is the temperature coefficient; Based on attention weights, image information is aggregated into a single... Related vectors, Considered as the most relevant local visual evidence in the image to the masked word, the calculation formula is as follows: ； To predict masked words, the aggregated visual information is fused with the implicit semantics of the text to predict the masked words. The calculation formula is as follows: ； in, For vector concatenation operations, the text semantics are... and visual evidence They are connected along the feature dimension to form a richer joint representation; FFN(.) is a feedforward neural network; Then, the predicted distribution is obtained through a classification layer, calculated using the following formula: ； in, To output the weight matrix, This is the output bias vector; The loss function is cross-entropy loss, which encourages the model to correctly predict the masked words based on the image. ； in, It is the set of all masked positions in the current text batch. For the real word masked at position p, It is the input image. For masked text; The implicit branch integrates a lightweight, differentiable, adaptive binarization projector at the end of the implicit branch by adaptively fusing the implicit semantic embeddings generated during pre-training with image features; the specific steps include: Image features are denoted as v, and implicit semantic embeddings are denoted as t. e , v and t e To fuse these into a more powerful joint representation, we design an adaptive gated fusion module, as shown in the following formula: ； ； in, and For learnable parameters, The activation function is sigmoid, and the output value is between (0,1). For element-wise multiplication, It is an adaptive, element-wise weight vector. Features after fusion; The target hash space is projected using a lightweight feedforward network FFN, and the continuous relaxation representation of the hash code is calculated as follows: ； in, Let be the projection weight matrix. This is the projection bias matrix; ； ； in, The weight matrix of the threshold generator, This is the bias vector for the threshold generator. This is the sharpness coefficient. Let i be the i-th component of the continuous features after projection. The i-th component of the adaptive threshold; Encourage soft hash codes To approximate binary representation and reduce quantization error, the quantization loss is calculated using the following formula: ； The contrast loss generated by explicit branching provides alignment supervision signals to implicit branching to perform the final retrieval matching.

2. The cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration as described in claim 1, characterized in that, The specific steps for masking text using a context-aware TF-IDF weighted strategy include: To calculate the context-dependent inverse document frequency (CIDF), let the target word be denoted as t, the set of context-dependent documents be denoted as C, and the current text to be weighted be denoted as d. The formula for calculating the IDF of word t relative to the context C is as follows: ； in, It is the total number of documents in the context document set C; It is a smoothing factor to prevent the denominator from being 0; When calculating the IF (Term Frequency Index), the distribution of words in the context is taken into account. The calculation formula is as follows: ； in This refers to the number of times word t appears in document d, where t' is a temporary variable used in the summation symbol ∑, representing each distinct word in document d; The final context-aware IF-IDF is the product of inverse document frequency and term frequency, calculated as follows: 。 3. The cross-modal remote sensing image and text retrieval method guided by explicit and implicit semantic collaboration as described in claim 1, characterized in that, The specific steps for performing the final retrieval matching, where the contrastive loss generated by explicit branching provides alignment supervision signals to implicit branching, include: For a batch of N paired samples, the contrast loss is defined in two directions: image to text and text to image. The formula for calculating the contrast loss is as follows: ； ； ； in, For image-to-text contrast loss, For text-to-image contrast loss, Total comparative loss; Calculate the global features of explicit branches The similarity matrix between them is calculated using the following formula: ； The similarity matrix between implicit branch outputs is calculated using the following formula: ； in, Let be the binary hash codes of the i-th and j-th samples, respectively. This is the scaling factor; To make the similarity relationship of implicit branches mimic the similarity relationship of explicit branches as closely as possible, mean squared error loss is used, and the calculation formula is as follows: 。

Images