Zero sample remote sensing image combination retrieval method based on multi-modal fusion

Abstract

The invention particularly relates to a zero-sample remote sensing image combination retrieval method based on multi-modal fusion, which comprises the following steps of: firstly, designing a zero-sample query text generator which can automatically generate a complete query text sample based on attributes so as to construct a remote sensing combination image reference data set containing triples; the problem that an existing data set in the remote sensing field cannot support multi-modal retrieval is solved. Secondly, the invention provides a text-image sequence training combined image retrieval method, through a multi-stage training strategy, bimodal information of the image and the text is effectively utilized for retrieval, and the limitation that a traditional method only depends on the text is avoided. In addition, the text image sequential training method can filter conflict information between the image and the text, the fine-grained features of the image are reserved, and the retrieval precision is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image retrieval technology, and specifically to a zero-sample remote sensing image combination retrieval method based on multimodal fusion. Background Technology

[0002] Currently, remote sensing applications in Earth observation are receiving increasing attention. The vast amount of remote sensing data provides rich research material for computer vision tasks. However, managing and extracting relevant images from this data remains a significant challenge, making the ability to efficiently and rapidly retrieve remote sensing images crucial.

[0003] Composed Image Retrieval (CIR) addresses this challenge by searching and retrieving images from remote sensing image archives. Unlike traditional unimodal queries, CIR is a more challenging visual-linguistic task involving paired images and text. Its goal is to retrieve target images by adapting existing images. Since language is the most natural way to encode human interaction, CIR enhances the retrieval process, enabling users to refine their queries through language and more effectively obtain the images they need. However, in the field of Remote Sensing Composed Image Retrieval (RSCIR), existing technologies face three core challenges: (1) Lack of standardized benchmark datasets, such as the attribute-labeled remote sensing image retrieval dataset Patterncom, which only supports attribute-based retrieval and lacks text query statements and structured triples, making it difficult to evaluate the performance of existing methods in the field of remote sensing combined image retrieval; (2) Existing zero-shot combined image retrieval (ZS-CIR) only uses the text modality for retrieval, ignoring the fine-grained information in the image, resulting in low retrieval accuracy, especially in the field of remote sensing image retrieval; (3) Existing ZS-CIR typically uses a single projection module to process multi-location masks, which fails to fully capture semantic diversity and further reduces retrieval performance.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This invention provides a zero-sample remote sensing image combination retrieval method based on multimodal fusion, a computer-readable storage medium, and a computer program product, which can effectively overcome the defects existing in the prior art.

[0006] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0007] According to a first aspect of the present invention, a zero-shot remote sensing image combination retrieval method based on multimodal fusion is provided, the method comprising: Encode the current query text and the current query image to obtain text embedding information and image embedding information respectively; Fine-grained image information is obtained by extracting fine-grained features of image embedding information based on a pre-built fine-grained image attention model. The fine-grained image attention model is built by training a multi-head self-attention network structure using query image samples and is used to filter image embedding information that does not match the text embedding information. The target image is obtained by matching the embedded text information and fine-grained image information with the candidate images respectively.

[0008] In some exemplary embodiments, the step of extracting fine-grained features of image embedding information based on a pre-built fine-grained image attention model to obtain fine-grained image information includes: Image embedding information is input into a fine-grained image attention model; By using a multi-head self-attention network structure in a fine-grained image attention model, the image embedding information is reweighted to determine the fine-grained image information.

[0009] In some exemplary embodiments, the step of matching the text embedding information and fine-grained image information with candidate images respectively to obtain the target image includes: Candidate images are pre-encoded to obtain candidate embedding information; Calculate the first cosine distance based on text embedding information and candidate embedding information; The second cosine distance is calculated based on fine-grained image information and candidate embedding information; The first cosine distance and the second cosine distance are weighted and fused according to preset weighting coefficients to obtain the comprehensive distance of the candidate images; The target image is selected from the candidate images based on the comprehensive distance.

[0010] In some exemplary embodiments, the method further includes: Based on image similarity, the initial remote sensing image set is divided into multiple image subsets, and image pairs are constructed in each image subset; wherein, the image pair contains a query image sample and a target image sample; A zero-shot query text generator is used to generate query text samples that match image pairs; the zero-shot query text generator includes a text generation sub-model and a contrastive language-image pre-trained sub-model. The first loss value is calculated using query text samples and the first loss function; the multi-self-masking network structure is trained by updating the first loss value to obtain the multi-self-masking projection model; The second loss value is calculated based on the multiple self-masking projection model, query image samples, and the second loss function; the multi-head attention network structure is trained by updating the second loss value to obtain a fine-grained image attention model.

[0011] In some exemplary embodiments, the step of dividing the initial remote sensing image set into multiple image subsets based on image similarity and constructing image pairs in each image subset includes: Feature extraction is performed on each remote sensing image in the initial remote sensing image set to obtain the corresponding image feature vector; The initial remote sensing image set is divided based on the image similarity between image feature vectors to obtain multiple image subsets. The image similarity of each remote sensing image within the same image subset is within a preset similarity range. One remote sensing image is randomly selected from the same image subset as the query image sample, and the remaining remote sensing images in the corresponding image subset are used as target image samples in turn to form an image pair with the query image sample.

[0012] In some exemplary embodiments, generating query text samples that match image pairs using a zero-sample query text generator includes: Input the target image sample and the structured text template into the text generation sub-model to obtain candidate query text and the corresponding fluency score; Cross-modal similarity between candidate query text and target image samples is calculated using a contrastive language-image pre-trained sub-model; Query text samples are selected from candidate query texts based on fluency scores and cross-modal similarity.

[0013] In some exemplary embodiments, calculating the first loss value using the query text sample and the first loss function includes: The query text sample is encoded using a text encoder to generate baseline text embedding information; Noise is added to the baseline text embedding information to generate noisy text embedding information; A multi-self-masking structure is used to reconstruct the embedded information of noisy text using multi-template masking and projection, generating reconstructed query text. The multi-self-masking structure contains multiple parallel self-masking branches, each corresponding to a mask template. Encode the reconstructed query text to generate embedded information in the reconstructed text; Substitute the baseline text embedding information and the reconstructed text embedding information into the first loss function to obtain the first loss value.

[0014] In some exemplary embodiments, calculating the second loss value based on the multiple self-masking projection model, query image samples, and a second loss function includes: Image encoding is performed on the query image samples to obtain the image sample embedding information; The image sample embedding information is input into the multi-self-masking projection model to obtain the conflict information between the query image sample and the query text sample. Image sample embedding information is input into a multi-head attention structure to obtain fine-grained image sample embedding information; The second loss value is obtained by substituting the image sample embedding information, conflict information, and fine-grained image sample information into the second loss function.

[0015] According to a second aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the storage medium is located to execute the above-described zero-sample remote sensing image combination retrieval method based on multimodal fusion.

[0016] According to a third aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the above-described zero-sample remote sensing image combination retrieval method based on multimodal fusion.

[0017] According to a fourth aspect of the present invention, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to implement the above-described zero-sample remote sensing image combination retrieval method based on multimodal fusion when executing the executable instructions.

[0018] The embodiments of this invention provide a zero-shot remote sensing image fusion-based retrieval method. At the data level, the zero-shot query text generator and the benchmark dataset supporting fine-grained retrieval fill the gap in standardized benchmark datasets in the remote sensing field. At the methodological level, the text-image sequential training method innovatively introduces a multiple self-masking projection model and a fine-grained image attention model, achieving conflict information filtering and feature optimization through sequential training. At the application level, the dual-modal retrieval mechanism achieves the synergistic utilization of text and visual features for the first time in remote sensing image fusion retrieval tasks. This end-to-end innovation of "data-model-application" provides a scalable technical framework for remote sensing image understanding, solving the inherent defects of traditional retrieval methods in terms of annotation dependence, single modality, and insufficient generalization. It is suitable for practical application scenarios such as disaster monitoring and land resource surveys, which have high requirements for fine-grained retrieval.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0021] Figure 1 This schematically illustrates a flowchart of a zero-sample remote sensing image combination retrieval method based on multimodal fusion, an exemplary embodiment of the present invention. Figure 2 The illustration shows a schematic diagram of the overall process of a zero-sample remote sensing image combination retrieval method based on multimodal fusion, an exemplary embodiment of the present invention. Figure 3 This illustration shows a schematic diagram of the training process of a zero-shot query text generator for a zero-shot remote sensing image combination retrieval method based on multimodal fusion, an exemplary embodiment of the present invention. Figure 4 The illustration shows a schematic diagram of the training process of a multi-self-masking projection model for a zero-shot remote sensing image combination retrieval method based on multimodal fusion, an exemplary embodiment of the present invention. Figure 5 The illustration shows a fine-grained image attention model training process of a zero-shot remote sensing image combination retrieval method based on multimodal fusion, an exemplary embodiment of the present invention. Figure 6The illustration shows a visualization of the fine-grained image attention model corresponding to a zero-shot remote sensing image combination retrieval method based on multimodal fusion, an exemplary embodiment of the present invention. Figure 7 This schematic diagram illustrates the composition of an electronic device according to an exemplary embodiment of the present invention. Detailed Implementation

[0022] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0023] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0024] Currently, zero-shot combined image retrieval (ZS-CIR) has become a research hotspot due to its independence from labeled triples. Typical methods, such as Pic2Word and SEARLE, utilize pre-trained models (e.g., the contrastive language-image pre-trained model CLIP) to train lightweight projection modules, mapping query images to text representations, which are then concatenated with the query text for retrieval. While these methods exhibit good generalization ability, they rely solely on the text modality, ignoring fine-grained information in the images, resulting in poor performance in remote sensing scenarios. Furthermore, LinCIR further simplifies the process by training the projection module using only text, but it still cannot resolve the problem of intermodal information conflicts.

[0025] Regarding the dataset, the existing Patterncom dataset only provides image attributes as labels and does not construct subsets or triples. .in, Indicates the query image (reference image); Indicates query text / modify description; This refers to the target image (the image in the database that actually matches the "query image + text description").

[0026] To address the shortcomings and deficiencies of existing technologies, this example implementation provides a zero-shot remote sensing image fusion retrieval method based on multimodal fusion. (Reference) Figure 1 As shown, it can specifically include: Step S10: Encode the current query text and the current query image to obtain text embedding information and image embedding information respectively; Step S12: Extract fine-grained features of image embedding information based on a pre-built fine-grained image attention model to obtain fine-grained image information; wherein, the fine-grained image attention model is built based on training a multi-head self-attention network structure using query image samples, and is used to filter image embedding information that does not match the text embedding information; Step S14: Match the text embedding information and fine-grained image information with the candidate images respectively to obtain the target image.

[0027] Based on steps S10 and S14 above, at the data level, the zero-shot query text generator and the benchmark dataset supporting fine-grained retrieval fill the gap in standardized benchmark datasets in the field of remote sensing; at the methodological level, the text-image sequential training method innovatively introduces a multiple self-masking projection model and a fine-grained image attention model, achieving conflict information filtering and feature optimization through sequential training; at the application level, the dual-modal retrieval mechanism achieves the synergistic utilization of text and visual features for the first time in remote sensing combined image retrieval tasks. This full-chain innovation of "data-model-application" provides a scalable technical framework for remote sensing image understanding, solving the inherent defects of traditional retrieval methods in terms of annotation dependence, single modality, and insufficient generalization, and is suitable for practical application scenarios such as disaster monitoring and land resource surveys where fine-grained retrieval requirements are high.

[0028] The following will describe in more detail each step of a zero-sample remote sensing image combination retrieval method based on multimodal fusion in this exemplary embodiment, with reference to the accompanying drawings and embodiments.

[0029] For example, in step S10, the aforementioned current query text is used to characterize the user's retrieval intent or scene modification request for the target remote sensing image. For example, "A satellite image of a white plane with wings in the background of forests".

[0030] The aforementioned current query image is used to represent the visual reference information provided by the user, indicating the basic image content or initial scene features of the retrieval task. For example, the current query image could be a "remote sensing image of a white airplane in a forest background," used to provide visual semantic information that needs to be retained or compared during the retrieval process.

[0031] Specifically, after receiving the current query text and the current query image, the query text is input to the text encoder and the query image is input to the image encoder, respectively. The encoders convert the input text and image into corresponding embedding information. The feature vector output by the text encoder is the text embedding information, and the feature vector output by the image encoder is the image embedding information. Both are used for subsequent attention extraction and retrieval matching processing.

[0032] For example, in step S12, the extraction of fine-grained features of image embedding information based on a pre-built fine-grained image attention model to obtain fine-grained image information includes: Step S121: Input the image embedding information into the fine-grained image attention model; Step S122: Through the multi-head self-attention network structure in the fine-grained image attention model, the image embedding information is reweighted to determine the fine-grained image information.

[0033] Specifically, the image embedding information output by the image encoder is used as input features and fed into the fine-grained image attention model. The multi-head self-attention network structure in the fine-grained image attention model is used to assign weights and reweight features at different positions / channels in the image embedding information, suppress features unrelated to the text embedding information, highlight detailed features related to the text embedding information, and determine the fine-grained image information based on the reweighting results.

[0034] For example, in step S14, matching the text embedding information and fine-grained image information with the candidate images respectively to obtain the target image includes: Step S141: Image encoding is performed on the candidate images in advance to obtain candidate embedding information; Step S142: Calculate the first cosine distance based on the text embedding information and the candidate embedding information; Step S143: Calculate the second cosine distance based on fine-grained image information and candidate embedding information; Step S144: The first cosine distance and the second cosine distance are weighted and fused according to the preset weighting coefficients to obtain the comprehensive distance of the candidate images; Step S145: Select the target image from the candidate images based on the comprehensive distance.

[0035] Specifically, it also utilizes the current query text. and the currently queried image To perform a search, for each image in the remote sensing image database, calculate its relationship with the current query text. and the currently queried image The comprehensive distance is calculated as follows: (1) in, For fine-grained image information; Embed information into text; Embedded information for candidates; Preset weighting coefficients are used to balance the importance of text and images in retrieval; The second cosine distance; The distance is the first cosine distance; The calculation process for cosine distance is shown in the following formula: (2) in, The length is the module length.

[0036] For details, please refer to Figure 2 As shown, the current query text is input into the text encoder to obtain text embedding information; the current query image is input into the image encoder to obtain image embedding information.

[0037] The image embedding information is then input into a pre-trained fine-grained image attention model. The image features are reweighted through the multi-head self-attention structure in the model, and components in the image embedding information that do not match the query text are filtered out, resulting in the output of fine-grained image information.

[0038] Candidate images from the database are sequentially input into an image encoder to obtain candidate embedding information. This process can be completed offline.

[0039] During online retrieval, the first cosine distance between the text embedding information and the candidate embedding information, and the second cosine distance between the fine-grained image information and the candidate embedding information are calculated separately. Finally, the first and second cosine distances are fused according to preset weights to obtain the comprehensive distance. The best-matching target image is selected from the candidate images based on the comprehensive distance.

[0040] For example, the method further includes: Step S21: Based on image similarity, the initial remote sensing image set is divided into multiple image subsets, and an image pair is constructed in each image subset; wherein, the image pair includes a query image sample and a target image sample; In step S21, the aforementioned original remote sensing image is used to characterize the remote sensing observation data sample set.

[0041] Step S22: Generate query text samples that match the image pairs using a zero-shot query text generator; wherein, the zero-shot query text generator includes a text generation sub-model and a contrastive language-image pre-trained sub-model. Step S23: Calculate the first loss value using the query text sample and the first loss function; train the multi-self-masking network structure by updating the first loss value to obtain the multi-self-masking projection model; Step S24: Calculate the second loss value based on the multiple self-masking projection model, query image samples, and the second loss function; train the multi-head attention network structure by updating the second loss value to obtain a fine-grained image attention model.

[0042] Specifically, this process involves the offline training of a fine-grained image attention model. First, the initial remote sensing image set is divided into multiple image subsets based on image similarity, and image pairs consisting of query image samples and target image samples are constructed within each subset. Then, a zero-shot query text generator, incorporating a text generation sub-model and a contrastive language-image pre-trained sub-model, automatically generates matching query text samples for each image pair. Next, using the query text samples as input, the multi-self-masking network structure is optimized through a first loss function to obtain a multi-self-masking projection model capable of reconstructing text semantics. Finally, based on the query image samples, the trained multi-self-masking projection model is introduced, and combined with a second loss function, the multi-head attention network structure is trained to obtain a fine-grained image attention model for filtering conflicting information and highlighting fine-grained visual features.

[0043] For example, in step S21, dividing the initial remote sensing image set into multiple image subsets based on image similarity and constructing image pairs in each image subset includes: Step S211: Extract features from each remote sensing image in the initial remote sensing image set to obtain the corresponding image feature vector; In step S211, the initial remote sensing image set mentioned above is the Patterncom dataset.

[0044] Step S212: The initial remote sensing image set is divided based on the image similarity between image feature vectors to obtain multiple image subsets, wherein the image similarity of each remote sensing image within the same image subset is within a preset similarity range; Step S213: Randomly select one remote sensing image from the same image subset as the query image sample, and sequentially use the remaining remote sensing images in the corresponding image subset as target image samples to form an image pair with the query image sample.

[0045] In step S213, the query image samples and target image samples are used to characterize the training dataset used in the model training process.

[0046] Specifically, to construct the PatternCIR dataset for combined retrieval of remote sensing images, the same method used to construct the CIR benchmark dataset (CIRR) is first employed to extract image features using a residual neural network, such as ResNet152. Next, an image subset is constructed based on similarity, and then image pairs are selected from this subset. Finally, a zero-shot query text generator is used to generate query text, further constructing triples.

[0047] Specifically, the nature of the CIR task requires constructing a set of negative sample images with high visual similarity, and forming triples based on these. Otherwise, distinguishing between reference and target images would become too simplistic. Furthermore, subset recall is a crucial metric for evaluating the performance of CIR methods; therefore, constructing image subsets is a necessary step in creating the PatternCIR dataset. Borrowing from the CIRR approach, multiple subsets of six visually similar images are constructed for each attribute, denoted as... .

[0048] For a given set of attributes Randomly select an image as the initial image. Image feature vectors are extracted using ResNet152 pre-trained on ImageNet, and the remaining images are then compared with... cosine similarity The images are then sorted based on similarity. During subset construction, images with similarity to the original image are removed. Avoid selecting nearly identical images if the similarity exceeds 0.94; add images in descending order of similarity, and skip images if the difference in similarity between an image and an already added image is less than 0.020; discard a subset if it is impossible to construct a complete subset of 6 images, or if there are more than 4 overlapping images with an existing subset.

[0049] In the constructed subset In the process, image pairs are selected to form triples of supervised data. The initial images of the subset are then used. As a query image sample It is then paired with the remaining 5 images in turn to serve as target image samples. Generate image pairs .

[0050] For example, in step S22, generating query text samples that match the image pairs using a zero-sample query text generator includes: Step S221: Input the target image sample and the structured text template into the text generation sub-model to obtain the candidate query text and the corresponding fluency score; In step S221, the above text generation sub-model is the BART text generation model with lexical constraints (CBART).

[0051] The aforementioned structured text template is used to represent a fixed sentence structure for query semantics. By predefining several semantic slots (such as target category slots, attribute description slots, background scene slots, etc.), it guides the text generation model to fill in the corresponding slots with semantic content that matches the image pair, so as to generate query text samples that meet the requirements of combined retrieval.

[0052] Step S222: Calculate the cross-modal similarity between the candidate query text and the target image sample using the contrastive language-image pre-trained sub-model; In step S222, the aforementioned contrastive language-image pre-trained sub-model is the RemoteCLIP visual language model in the field of remote sensing. Based on the general CLIP model, it is retrained or fine-tuned using large-scale remote sensing data (satellite images + text descriptions) to enable the model to understand the semantics of land features such as airports, ports, forests, and farmland in remote sensing images.

[0053] Step S223: Select query text samples from candidate query texts based on fluency score and cross-modal similarity.

[0054] Specifically, the target image sample and the structured text template are first input into the text generation sub-model to obtain candidate query texts and their corresponding fluency scores. The position of the structured text template... Sampling vocabulary At that time, CBART first predicts the conditional probability. , and select the previous candidate words And calculate the corresponding fluency score. The fluency score calculation process is shown in the following formula: (3) in, Indicates that, apart from location In addition, words that remain unchanged.

[0055] Combine candidate words with context to form One candidate sentence .

[0056] Then, the image encoder via RemoteCLIP and text encoder Calculate candidate sentences and target image samples The cross-modal similarity is calculated as follows: (4) in, The feature vector extracted from the target image sample by the image encoder. For the first The feature vectors extracted from the candidate sentences by the text encoder.

[0057] Next, candidate words are selected using a dual ranking mechanism. First, based on the fluency score of the language model... First, ensure the generated text is natural and fluent; second, base it on cross-modal similarity scores. This ensures high alignment between the text and image content. This sorting process is iterative. In each round, the candidate word with the highest overall score is retained and used to generate a better sentence for the next round, while the rest are eliminated. After multiple rounds of this "generation-sorting-refinement" cycle, the final output is the text description that best matches the target image sample. In each round, a comprehensive score is obtained by combining the overall fluency score and cross-modal similarity score, and candidate words are selected based on this comprehensive score. The comprehensive score is shown in the following formula: (5) in, and These are the weighting coefficients for fluency score and cross-modal similarity, respectively.

[0058] For details, please refer to Figure 3 As shown, the initial query text input is first constructed based on a predefined text template (such as "A satellite image of [attribute value], [class name], with [S] in the background of [S]") and then input into the text generation sub-model CBART. After combining the input template with the visual semantics of the corresponding remote sensing image, CBART automatically generates multiple candidate query texts and calculates the fluency score corresponding to each candidate query text. For example, candidate texts include: 1. "A satellite image of white plane with wings in the background offorests"; 2. "A satellite image of white plane with painting in the background of buildings"; 3. "A satellite image of white plane with logos in the background of airport" etc.

[0059] Subsequently, the image encoder using RemoteCLIP was utilized. and text encoder Calculate candidate sentences and target image text Cross-modal similarity In each round, a comprehensive score is obtained by combining the fluency score and cross-modal similarity. The candidate word with the highest comprehensive score is retained and used to generate a better sentence in the next round, while the rest are eliminated, until a query text sample that meets the requirements of combined retrieval is generated.

[0060] Furthermore, a zero-sample query text generator is used to generate query text for each image pair. Ultimately forming a triplet The input template is a fixed sentence: "A satellite image of, [attributevalue], [class name], with, in the background of." Here, [attribute value] is the target image. The attribute values ​​(e.g., white, dense) and [class name]_ represent the category name of the object of interest in the target image (e.g., airplane, tennis court). The zero-shot query text generator expands the template into a complete sentence while retaining keywords from the initial template. Comma-separated phrases are treated as a whole, with their order and content remaining unchanged. Finally, a large number of triples are constructed using this method and divided into test, validation, and training sets according to a certain ratio. The dataset organization is consistent with CIRR, facilitating direct evaluation by existing CIR methods.

[0061] For example, in step S23, calculating the first loss value using the query text sample and the first loss function includes: Step S231: Encode the query text sample using a text encoder to generate baseline text embedding information; Step S232: Add noise to the baseline text embedding information to generate noisy text embedding information; Step S233: The noisy text embedding information is reconstructed by multi-template masking and projection using a multi-self-masking structure to generate reconstructed query text; wherein, the multi-self-masking structure contains multiple parallel self-masking branches, and each self-masking branch corresponds to a mask template. Step S234: Encode the reconstructed query text to generate reconstructed text embedding information; Step S235: Substitute the baseline text embedding information and the reconstructed text embedding information into the first loss function to obtain the first loss value.

[0062] In step S235, the first loss function is as follows: (6) in, Embed information into the baseline text; To reconstruct embedded information in the text; It is the L2 norm (squared Euclidean distance), used to measure the difference between two semantic embedding vectors.

[0063] Specifically, existing technologies typically use single self-mask projection (SMP) to handle multiple masks at different locations, resulting in poor performance. This is because a single SMP struggles to effectively handle the diverse semantic information from different masks. Especially when masks are located in different positions, they carry different semantic nuances, which a single SMP cannot fully capture. Therefore, this invention uses a multiple self-mask projection module to train an independent SMP for each mask. During training, only the corresponding mask is retained, while other parts of the text remain unchanged. This method allows each SMP to focus on extracting the semantic information of a single mask, thus avoiding interference from other parts. By processing each mask in isolation, it is ensured that semantic extraction is not diluted by irrelevant information, thereby generating more reliable and useful embedded information.

[0064] For details, please refer to Figure 4 As shown in the figure, this diagram illustrates the training process of a Multi-Self-Mask Projection (MSMP) model. First, the training text is input into a text encoder to obtain the baseline text embedding. Subsequently, the perturbation vector is sampled from the Gaussian distribution, scaled, and added to... The noisy text embedding is obtained. This noisy text embedding is then fed into a multi-self-masking projection structure, where multiple self-masking branches are used to simultaneously reconstruct different occlusion positions in the training text, generating corresponding local text representations. Finally, the reconstruction results from all positions are concatenated to obtain the complete reconstructed text embedding. Then, by calculating the difference loss between the baseline text embedding and the reconstructed text embedding, the multi-self-masking projection structure is guided to learn how to recover semantic information from the occluded text fragments, thereby improving the model's fine-grained text representation ability.

[0065] For example, in step S24, calculating the second loss value based on the multiple self-masking projection model, the query image sample, and the second loss function includes: Step S241: Image encoding is performed on the query image sample to obtain the image sample embedding information; Step S242: Input the image sample embedding information into the multiple self-masking projection model to obtain the conflict information between the query image sample and the query text sample; Step S243: Input the image sample embedding information into the multi-head attention structure to obtain fine-grained image sample embedding information; Step S244: Substitute the image sample embedding information, conflict information, and fine-grained image sample information into the second loss function to obtain the second loss value.

[0066] In step S244, the second loss function is as follows: (7) in, Embed information into image samples; Conflict information between images and text extracted by a multi-self-masking projection model; Information is embedded in the fine-grained image samples for the fine-grained image attention output (excluding conflicting parts).

[0067] The second loss function is used to adjust , so that it is with Alignment involves preserving features in the image that are consistent with the text description (such as the shape and color of the target object) while filtering out features in the image that conflict with the text description (such as background interference and irrelevant objects). In this way, fine-grained image attention models can generate more accurate image embeddings, enabling them to work better with text embeddings during retrieval.

[0068] For details, please refer to Figure 5 As shown. After training the multi-self-masking projection model, the fine-grained image attention (FGIA) model is further trained. The structure of the fine-grained image attention module is based on a multi-head self-attention mechanism, which can simultaneously focus on different aspects of the embeddings, thereby more effectively extracting fine-grained features of the image. During training, image sample embedding information... The data is fed into a pre-trained MSMP model. This MSMP model has previously been trained on plain text, learning how to extract clean semantic features related to category and attributes from masked text templates. At this point, the MSMP model uses the image sample embedding information... It predicts and outputs a "conflict information embedding". This conflict information embedding reflects visual information in the image that does not match or contradicts the text semantic template. For example, irrelevant objects in the background, colors or textures that do not match the text description, etc.

[0069] To enable the fine-grained image attention model to filter out information that conflicts with the text, a second loss function is used as a constraint to train the multi-head self-attention mechanism network, thus obtaining the fine-grained image attention model.

[0070] Further, refer to Figure 6As shown, the fine-grained image attention model dynamically removes visual features (such as the main object) that are irrelevant to the text description by calculating embedding similarity, while retaining fine-grained background information. Its effectiveness is verified through heatmap visualization. The embedded features filtered by the fine-grained image attention model focus more on background details (such as mesh texture), avoiding the misleading influence of the main object's features.

[0071] The method provided in this invention innovatively employs a two-stage training strategy for text-image sequential training. The first stage trains a multi-self-masking projection model using text sequences to learn cross-modal alignment. The second stage optimizes a fine-grained image attention model using image sequences to enhance fine-grained feature filtering capabilities. The multi-self-masking projection model strengthens semantic understanding through text modality, while the fine-grained image attention model filters out main object features that conflict with the query (such as...). Figure 6 (As shown in the similarity mapping), the model can focus more on background details (such as texture and color), which significantly improves retrieval accuracy.

[0072] Furthermore, the following experiments were designed to demonstrate the model's sophistication.

[0073] Comparative experiment: Six ZS-CIR methods (such as Pic2Word and SEARLE-XL) were compared on PatternCIR. The text-image sequential training method achieved state-of-the-art (SOTA) results in both R@1 and R_subset@1 (Table 1).

[0074] Ablation experiments: Verify the independent contributions of the multiple self-masking projection model and the fine-grained image attention model. When the two models are used together, R@1 is improved by 22.95% (Table 2).

[0075] Visualization Analysis: Demonstrating through heatmaps that fine-grained image attention models can dynamically suppress conflicting features ( Figure 6 ).

[0076] Table 1 Main experimental results

[0077] Table 2 Ablation Experiment Results

[0078] The experiments above show that on the PatternCIR dataset created in this invention, the text-image sequential training method achieves an R@1 of 7.34%, a 12.4% improvement over the best baseline (6.53% for LDRE), and an R_subset@1 of 44.81% (better than WEICOM's 39.24%). Traditional methods such as LinCIR rely solely on text training, resulting in an R@1 as low as 4.53%; while SEARLE suffers from performance limitations due to the ViT-B backbone network (R@1 is only 1.99%). The text-image sequential training method, through dual-modal co-training and the ViT-L backbone, completely solves the problems of single-modal bias and insufficient backbone network capacity.

[0079] Specifically, the fine-grained image attention model dynamically removes visual features (such as the main object) irrelevant to the text description by calculating embedding similarity, retains fine-grained background information, and verifies its effectiveness through heatmap visualization. For example... Figure 6 As shown, the embedded data filtered by the fine-grained image attention model focuses more on background details (such as mesh texture), avoiding misleading results from the main object's features. In similar image subset retrieval, the fine-grained image attention model achieves an R_subset@3 of 82.94%, a 3.03 percentage point improvement over LDRE (79.91%), demonstrating its effectiveness in reducing false detections. The dynamic filtering mechanism of the fine-grained image attention model adapts to diverse queries (such as "replace background" or "modify local texture") without relying on a fixed template, overcoming the template noise problem of methods such as AnomalyCLIP.

[0080] Specifically, this invention utilizes a zero-shot query text generator to automatically generate diverse query texts and constructs the PatternCIR dataset containing similar image triples, covering complex scenarios (such as subset retrieval). Without manual annotation, the generated query texts cover fine-grained requirements (such as "metal surface scratch detection"), making the model more generalizable in RSCIR tasks. PatternCIR is the first to support subset recall (R_subset@K) evaluation, reducing the false negative problem caused by dataset bias in traditional R@K, and becoming a new benchmark in the CIR field.

[0081] Furthermore, this invention achieves collaborative training of text and image sequences, solving the modal fragmentation problem of traditional methods. In scenarios such as e-commerce retrieval and security monitoring, it can accurately match fine-grained user-described needs (such as "finding the same bag but with different prints"), reducing manual screening costs by more than 70%. The text-image sequential training method, through modeling design of multiple self-masking projection and fine-grained image attention, dynamic feature filtering, and automated data construction, achieves a dual breakthrough in accuracy and efficiency in zero-shot CIR tasks, providing a new paradigm for multimodal retrieval.

[0082] The beneficial effects of this invention are as follows: (1) Reduce annotation costs and construct standardized fine-grained datasets A zero-shot query text generator automatically transcribes attribute labels from Patterncom into natural language query text, which, together with similar image pairs, forms the triplet dataset PatternCIR. This eliminates the need for manual annotation of each query and triplet. This dataset, containing both similar subsets and fine-grained text descriptions, provides the first publicly assessable standardized benchmark for remote sensing composite image retrieval, significantly reducing data construction costs and improving reusability. It fills a gap in the field of remote sensing image retrieval and provides a standardized evaluation platform for subsequent research.

[0083] (2) Multiple self-masking projection enhances the semantic expressive power of text. This invention introduces a Multiple Self-Mask Projection (MSMP) module at the text level, training sub-projection networks separately for different mask positions. This allows the model to decouple modeling for fine-grained components such as category, attribute, and background, avoiding semantic confusion caused by single projection processing of multiple masks. By constraining the consistency between the reconstructed embedding and the original text embedding through a first loss function, a cleaner and more aligned text embedding can be obtained, laying a semantic foundation for subsequent cross-modal retrieval.

[0084] (3) Fine-grained image attention filtering to filter conflicting information and improve retrieval accuracy On the image side, this invention uses a fine-grained image attention (FGIA) module to perform multi-head self-attention reweighting on image embeddings, and leverages a trained MSMP to extract "conflicting embeddings," explicitly suppressing main object or background information inconsistent with the query text while preserving detailed features consistent with the text semantics. Experiments show that on the PatternCIR dataset, this method achieves an R@1 of 7.34%, an improvement of approximately 12.4% compared to the best baseline LDRE (6.53%), and an R_subset@1 of 44.81%, significantly outperforming WEICOM's 39.24%, effectively reducing false detections.

[0085] (4) The zero-shot multimodal retrieval framework trained in text-image sequence has good generalization ability. By employing a sequential training strategy of "text first, image later," this invention achieves true zero-sample multimodal combined retrieval without relying on pairwise annotations (query text – query image). It utilizes both text embedding and filtered image embedding for weighted matching. Compared to ZS-CIR methods such as LinCIR and SEARLE, which rely solely on text, this approach demonstrates significant advantages in metrics such as R@1 and R_subset@1. It improves the accuracy and robustness of fine-grained remote sensing image retrieval and is applicable to various remote sensing applications, including disaster monitoring and land resource surveys, demonstrating significant engineering application value.

[0086] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may, for example, be executed synchronously or asynchronously in multiple modules.

[0087] It should be noted that although several modules or units of the device for performing actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0088] Figure 7 A schematic diagram of an electronic device suitable for implementing embodiments of the present invention is shown.

[0089] It should be noted that, Figure 7 The electronic device 1000 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0090] like Figure 7As shown, the electronic device 1000 includes a Central Processing Unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1002 or programs loaded from storage section 1008 into Random Access Memory (RAM) 1003. The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An Input / Output (I / O) interface 1005 is also connected to the bus 1004. Furthermore, the electronic device 1000 also includes an FPGA device and a System-on-a-Chip (SoC) device.

[0091] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1010 as needed so that computer programs read from them can be installed into storage section 1008 as needed.

[0092] In particular, according to embodiments of the present invention, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs various functions defined in the system of this application.

[0093] Specifically, the aforementioned electronic devices can be airborne intelligent electronic devices.

[0094] It should be noted that the storage medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, wherein computer-readable program code is carried. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0096] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0097] It should be noted that, as another aspect, this application also provides a storage medium, which may be included in an electronic device or may exist independently without being assembled into the electronic device. The aforementioned storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the methods described in the following embodiments. For example, the electronic device may perform... Figure 1 The steps of the method shown.

[0098] In one embodiment, this application provides a computer program product including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0099] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0100] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0101] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A zero-shot remote sensing image combination retrieval method based on multimodal fusion, characterized in that, The method includes: Encode the current query text and the current query image to obtain text embedding information and image embedding information respectively; Fine-grained image information is obtained by extracting fine-grained features of image embedding information based on a pre-built fine-grained image attention model. The fine-grained image attention model is built by training a multi-head self-attention network structure using query image samples and is used to filter image embedding information that does not match the text embedding information. The target image is obtained by matching the embedded text information and fine-grained image information with the candidate images respectively.

2. The method according to claim 1, characterized in that, The fine-grained image information is obtained by extracting fine-grained features of image embedding information based on a pre-built fine-grained image attention model, including: Image embedding information is input into a fine-grained image attention model; By using a multi-head self-attention network structure in a fine-grained image attention model, the image embedding information is reweighted to determine the fine-grained image information.

3. The method according to claim 1, characterized in that, The step of matching the text embedding information and fine-grained image information with the candidate images respectively to obtain the target image includes: Candidate images are pre-encoded to obtain candidate embedding information; Calculate the first cosine distance based on text embedding information and candidate embedding information; The second cosine distance is calculated based on fine-grained image information and candidate embedding information; The first cosine distance and the second cosine distance are weighted and fused according to preset weighting coefficients to obtain the comprehensive distance of the candidate images; The target image is selected from the candidate images based on the comprehensive distance.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on image similarity, the initial remote sensing image set is divided into multiple image subsets, and image pairs are constructed in each image subset; wherein, the image pair contains a query image sample and a target image sample; A zero-shot query text generator is used to generate query text samples that match image pairs; the zero-shot query text generator includes a text generation sub-model and a contrastive language-image pre-trained sub-model. The first loss value is calculated using query text samples and the first loss function; the multi-self-masking network structure is trained by updating the first loss value to obtain the multi-self-masking projection model; The second loss value is calculated based on the multiple self-masking projection model, query image samples, and the second loss function; the multi-head attention network structure is trained by updating the second loss value to obtain a fine-grained image attention model.

5. The method according to claim 4, characterized in that, The process of dividing the initial remote sensing image set into multiple image subsets based on image similarity and constructing image pairs in each image subset includes: Feature extraction is performed on each remote sensing image in the initial remote sensing image set to obtain the corresponding image feature vector; The initial remote sensing image set is divided based on the image similarity between image feature vectors to obtain multiple image subsets. The image similarity of each remote sensing image within the same image subset is within a preset similarity range. One remote sensing image is randomly selected from the same image subset as the query image sample, and the remaining remote sensing images in the corresponding image subset are used as target image samples in turn to form an image pair with the query image sample.

6. The method according to claim 4, characterized in that, The method of generating query text samples that match image pairs using a zero-sample query text generator includes: Input the target image sample and the structured text template into the text generation sub-model to obtain candidate query text and the corresponding fluency score; Cross-modal similarity between candidate query text and target image samples is calculated using a contrastive language-image pre-trained sub-model; Query text samples are selected from candidate query texts based on fluency scores and cross-modal similarity.

7. The method according to claim 6, characterized in that, The calculation of the first loss value using the query text sample and the first loss function includes: The query text sample is encoded using a text encoder to generate baseline text embedding information; Noise is added to the baseline text embedding information to generate noisy text embedding information; A multi-self-masking structure is used to reconstruct the embedded information of noisy text using multi-template masking and projection, generating reconstructed query text. The multi-self-masking structure contains multiple parallel self-masking branches, each corresponding to a mask template. Encode the reconstructed query text to generate embedded information in the reconstructed text; Substitute the baseline text embedding information and the reconstructed text embedding information into the first loss function to obtain the first loss value.

8. The method according to claim 4, characterized in that, The calculation of the second loss value based on the multiple self-masking projection model, query image samples, and the second loss function includes: Image encoding is performed on the query image samples to obtain the image sample embedding information; The image sample embedding information is input into the multi-self-masking projection model to obtain the conflict information between the query image sample and the query text sample. Image sample embedding information is input into a multi-head attention structure to obtain fine-grained image sample embedding information; The second loss value is obtained by substituting the image sample embedding information, conflict information, and fine-grained image sample information into the second loss function.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 8.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images