Cross-modal retrieval model training method and remote sensing image text retrieval method

Abstract

The invention provides a cross-modal retrieval model training method and a remote sensing image text retrieval method, and is applied to the technical field of remote sensing cross-modal retrieval. The remote sensing image text retrieval method comprises the following steps: performing feature extraction on a preprocessed remote sensing image by utilizing a trained cross-modal retrieval model to obtain image global features and image local features, and performing feature extraction on a preprocessed target text to obtain text global features and phrase global features; calculating the global similarity between the image global features and the text global features by using the trained cross-modal retrieval model, and calculating the local similarity between the image local features and the word group global features; and carrying out weighted calculation on the global similarity and the local similarity by utilizing the trained cross-modal retrieval model, and screening a matching result between the remote sensing image and the target text based on the weighted similarity to obtain a retrieval result between the remote sensing image and the target text.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing cross-modal retrieval, and in particular to a training method for a cross-modal retrieval model and a remote sensing image text retrieval method. Background Art

[0002] With the continuous development of remote sensing technology, it has become easier to obtain remote sensing images, and a large amount of remote sensing image data has been accumulated. However, how to quickly and accurately retrieve the information required by users from massive remote sensing images is still an important research issue. Therefore, the problem of Remote Sensing Image-Text Retrieval (RSITR) was raised. Remote sensing image text retrieval refers to retrieving matching remote sensing images through natural language description queries, or finding relevant text descriptions through image queries. The diversity and similarity of remote sensing images make the retrieval between images and texts more complicated.

[0003] Among the existing remote sensing image and text retrieval methods, the remote sensing image text retrieval method based on convolutional neural networks has shortcomings in capturing local fine-grained information between images and texts, especially when there are subtle differences between the image and text content, the retrieval performance is often limited; and the remote sensing image text retrieval method based on Transformer still has technical problems such as insufficient alignment of fine-grained features. Summary of the invention

[0004] In view of the above problems, the present invention provides a cross-modal retrieval model training method and a remote sensing image text retrieval method for improving matching accuracy and cross-modal retrieval efficiency.

[0005] According to a first aspect of the present invention, a method for training a cross-modal retrieval model is provided, comprising:

[0006] The visual encoder of the cross-modal retrieval model is used to extract features from the preprocessed remote sensing image samples to obtain the global features and local features of the image samples.

[0007] The text encoder of the cross-modal retrieval model is used to extract features from the preprocessed text samples to obtain global features of the text samples and global features of the phrase samples.

[0008] Based on the contrastive learning method, the global features of the image samples and the global features of the text samples are aligned by calculating the global similarity between the constructed positive sample pairs and the negative sample pairs, and the global alignment loss value is calculated;

[0009] Using the local similarity between the local features of the image sample and the global features of the phrase sample, the local features of the image sample and the global features of the phrase sample are locally aligned, and the local alignment loss value is calculated;

[0010] Perform mask modeling on the preprocessed remote sensing image samples and the preprocessed text samples respectively, and calculate the mask loss values ​​of the remote sensing image samples and the text samples in the mask modeling process respectively;

[0011] The global alignment loss value, local alignment loss value and mask loss value are weighted to obtain a weighted loss value, and the weighted loss value is used to optimize the parameters of the cross-modal retrieval model.

[0012] According to an embodiment of the present invention, the training method of the cross-modal retrieval model further includes:

[0013] Iteratively perform feature extraction operations, weighted loss value calculation operations, and model parameter optimization operations on remote sensing image samples and text samples until the preset training conditions are met, thereby obtaining a trained cross-modal retrieval model;

[0014] The visual encoder using the cross-modal retrieval model extracts features from the preprocessed remote sensing image samples, and obtains global features and local features of the image samples, including:

[0015] Based on the dimensional information of the remote sensing image, the remote sensing image sample is segmented to obtain multiple non-overlapping rectangular area image samples;

[0016] Each rectangular area image sample is projected into a one-dimensional space using the learnable linear mapping module of the cross-modal retrieval model to obtain multiple local features of the initial image samples.

[0017] Adding a special code for aggregating global information to each local feature of the initial image sample to obtain the local feature of the initial image sample with the special code;

[0018] Adding an image position code to each local feature of an initial image sample having a special code to obtain a plurality of encoded local features of the initial image sample;

[0019] The visual encoder of the cross-modal retrieval model is used to process all encoded local features of the initial image samples to obtain local features and global features of the image samples.

[0020] According to an embodiment of the present invention, the text encoder using the cross-modal retrieval model extracts features from the preprocessed text sample to obtain global features of the text sample and global features of the phrase sample, including:

[0021] The text sample is segmented by using a byte pair encoding method to obtain multiple phrases from the text sample, and a start mark word and an end mark word are respectively inserted at the start position and the end position of the text sample;

[0022] Use the encoding matrix of the cross-modal retrieval model to map each phrase into a word vector and add a text position code to each word vector;

[0023] The text encoder of the cross-modal retrieval model is used to extract features from the start marker word, the end marker word, and all word vectors with text position encoding to obtain the global features of the text;

[0024] A predefined natural language processing tool is used to extract multiple noun phrases from a text sample to complete the enhancement operation on the text sample, and a text encoder is used to extract global features of each noun phrase to obtain the global features of the phrase sample.

[0025] According to an embodiment of the present invention, the above-mentioned visual encoder is constructed based on the Vision Transformer model;

[0026] Among them, the text encoder is built based on the Bert model;

[0027] Among them, predefined natural language processing tools include the Spacy library.

[0028] According to an embodiment of the present invention, the above-mentioned contrastive learning method is based on calculating the global similarity between the constructed positive sample pairs and the negative sample pairs to perform global feature alignment on the global features of the image samples and the global features of the text samples, and calculating the global alignment loss value includes:

[0029] Based on the pairing information between remote sensing image samples and text samples, construct positive sample pairs and negative sample pairs;

[0030] Based on the contrastive learning method, the global similarity of positive and negative sample pairs is calculated respectively to obtain positive global similarity and negative global similarity;

[0031] Using positive global similarity and negative global similarity to perform global feature alignment on the global features of image samples and the global features of text samples;

[0032] The global alignment loss value in the global feature alignment process is calculated using a predefined global alignment loss function.

[0033] According to an embodiment of the present invention, the above-mentioned local feature alignment of the local features of the image samples and the global features of the phrase samples using the local similarity between the local features of the image samples and the global features of the phrase samples, and calculating the local alignment loss value includes:

[0034] Calculate the cosine similarity between the global features of the phrase sample and the local features of the image sample;

[0035] All cosine similarities above the preset similarity threshold are averaged to obtain the local similarity;

[0036] Use local similarity to align local features of image samples and global features of phrase samples;

[0037] The predefined local alignment loss function is used to calculate the local-to-partial alignment loss value in the local feature alignment process.

[0038] According to an embodiment of the present invention, the above-mentioned performing mask modeling on the preprocessed remote sensing image samples and the preprocessed text samples respectively, and calculating the mask loss values ​​of the remote sensing image samples and the text samples in the mask modeling process respectively includes:

[0039] Based on a preset image mask ratio, the preprocessed remote sensing image samples are randomly masked to obtain masked remote sensing image samples;

[0040] The visual encoder of the cross-modal retrieval model is used to extract features from the masked remote sensing image samples to obtain multiple mask feature samples;

[0041] All mask feature samples are processed using the visual mask regression head of the cross-modal retrieval model to obtain reconstructed remote sensing image samples;

[0042] The reconstructed remote sensing image samples and remote sensing image samples are processed using a predefined image mask loss function to obtain an image mask loss value.

[0043] According to an embodiment of the present invention, the above-mentioned performing mask modeling on the preprocessed remote sensing image samples and the preprocessed text samples respectively, and calculating the mask loss values ​​of the remote sensing image samples and the text samples in the mask modeling process respectively, further includes:

[0044] Based on the preset image mask ratio, some words in the text sample are randomly replaced with masked specific words to obtain the masked text sample;

[0045] The text encoder of the cross-modal retrieval model is used to extract features from the masked text samples to obtain multiple word features.

[0046] Use the text mask classification head of the cross-modal retrieval model to predict each word feature and obtain the predicted text;

[0047] The text sample and predicted text are processed using the predefined text mask loss function to obtain the text mask loss value.

[0048] According to an embodiment of the present invention, the above-mentioned predefined global alignment loss function and the predefined local alignment loss function are constructed based on the contrastive learning loss function;

[0049] Among them, the predefined image mask loss function is constructed based on the minimum square error function;

[0050] Among them, the predefined text mask loss function is built based on the cross entropy loss function.

[0051] According to a second aspect of the present invention, a remote sensing image text retrieval method is provided, comprising:

[0052] Preprocessing the remote sensing image and the target text respectively using the trained cross-modal retrieval model to obtain a preprocessed remote sensing image and a preprocessed target text, wherein the trained cross-modal retrieval model is trained based on the training method of the cross-modal retrieval model for remote sensing image text retrieval;

[0053] The trained cross-modal retrieval model is used to extract features from the preprocessed remote sensing image to obtain global features and local features of the image, and the preprocessed target text is also extracted to obtain global features of the text and global features of the phrase.

[0054] The trained cross-modal retrieval model is used to calculate the global similarity between the global features of the image and the global features of the text, and the local similarity between the local features of the image and the global features of the phrase is calculated;

[0055] The trained cross-modal retrieval model is used to perform weighted calculation on the global similarity and the local similarity to obtain the weighted similarity. The matching results between the remote sensing image and the target text are screened based on the weighted similarity to obtain the retrieval results between the remote sensing image and the target text.

[0056] The training method of the cross-modal retrieval model provided by the present invention can be used in the field of remote sensing image text retrieval technology. The global and local features of remote sensing image samples and text samples are extracted through the constructed cross-modal retrieval model, and the global features and local features are aligned based on contrast learning. Not only the feature alignment of image and text samples on a large scale is achieved, but also the feature alignment of image and text samples on a fine granularity can be achieved; through image and text sample mask modeling, the correlation learning of single-modal features is enhanced, and the matching ability of images and texts is further improved. In addition, by combining the global similarity and local similarity scores, the accuracy of the trained cross-modal retrieval model in remote sensing image and text retrieval is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0058] Figure 1 is an application scenario diagram of a cross-modal retrieval model training method and a remote sensing image text retrieval method according to an embodiment of the present invention;

[0059] Figure 2 is a flowchart of a method for training a cross-modal retrieval model according to an embodiment of the present invention;

[0060] Figure 3 is a flowchart of a remote sensing image text retrieval method according to an embodiment of the present invention;

[0061] Figure 4 is a structural block diagram of a remote sensing image text retrieval device according to an embodiment of the present invention;

[0062] Figure 5 It is a block diagram of an electronic device suitable for implementing a training method for a cross-modal retrieval model and a remote sensing image text retrieval method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0064] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0065] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0066] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0067] Among the existing remote sensing image and text retrieval methods, most rely on convolutional neural networks (CNNs) to extract image features and recurrent neural networks (RNNs) to extract text features, and achieve image and text retrieval through global feature alignment. However, these methods are insufficient in capturing local fine-grained information between images and texts, especially when there are subtle differences between image and text content, the retrieval performance of existing methods is often limited. Recent Transformer-based image and text retrieval methods use the powerful global information fusion capability of Transformer to improve the matching accuracy of images and texts. Among them, the most prominent one is the CLIP (Contrastive Language-Image Pre-training) model, which has made significant progress in image and text retrieval performance. However, although Transformer can effectively process global features, there is still a problem of insufficient alignment of fine-grained features. In order to make up for this defect, it is necessary to combine the alignment of global features and local features to improve the accuracy and robustness of remote sensing image and text retrieval. How to effectively combine global alignment with fine-grained local alignment has become a key issue in improving the accuracy of remote sensing image and text retrieval.

[0068] In order to solve at least one of the problems in the prior art, the present invention provides a remote sensing image and text retrieval method that combines global alignment with phrase-enhanced fine-grained local alignment (Global and Local Alignment with Phrase Argumentation, GLAPA). The above method provided by the present invention can effectively improve the matching accuracy between remote sensing images and texts and the accuracy of cross-modal retrieval, and can be widely used in disaster monitoring, agricultural production, environmental monitoring and other fields in the remote sensing field.

[0069] The embodiment of the present invention provides a remote sensing image text retrieval method, which relates to the field of remote sensing cross-modal retrieval technology and can be applied to disaster monitoring, agricultural production, environmental monitoring and other fields in the remote sensing field.

[0070] Figure 1 It is an application scenario diagram of a cross-modal retrieval model training method and a remote sensing image text retrieval method according to an embodiment of the present invention.

[0071] like Figure 1As shown, the application scenario 100 according to this embodiment may include disaster monitoring, agricultural production, environmental monitoring and other fields in the field of remote sensing cross-modal retrieval technology. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0072] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0073] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0074] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0075] It should be noted that the training method of the cross-modal retrieval model and the remote sensing image text retrieval method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the remote sensing image text retrieval device provided in the embodiment of the present invention can generally be set in the server 105. The training method of the cross-modal retrieval model and the remote sensing image text retrieval method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the remote sensing image text retrieval device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0076] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0077] The following will be based on Figure 1 The scene described by Figure 2 and Figure 3 The training method of the cross-modal retrieval model and the remote sensing image text retrieval method of the disclosed embodiment are described in detail.

[0078] Figure 2 is a flowchart of a cross-modal retrieval model according to an embodiment of the present invention.

[0079] like Figure 2 As shown, the above-mentioned training of the cross-modal retrieval model based on the multimodal pre-trained neural network to obtain the trained retrieval model includes operations S210 to S270.

[0080] In operation S210, a visual encoder of a cross-modal retrieval model is used to extract features from the preprocessed remote sensing image samples to obtain global features and local features of the image samples.

[0081] The above cross-modal retrieval model is built based on a multimodal pre-trained neural network, which includes CLIP (Contrastive Language-Image Pre-Training). CLIP can perform image classification, image retrieval, text generation, multimodal search, etc.

[0082] In operation S220, a text encoder of a cross-modal retrieval model is used to perform feature extraction on the preprocessed text sample to obtain global features of the text sample and global features of the phrase sample.

[0083] Since a text sample is composed of multiple words or phrases, the global features of the phrase sample can be used to characterize the local features of the text sample.

[0084] In operation S230 , based on a contrastive learning method, global feature alignment is performed on the image sample global features and the text sample global features by calculating the global similarity between the constructed positive sample pairs and the negative sample pairs, and a global alignment loss value is calculated.

[0085] The contrastive learning method can improve the generalization ability of the trained model, reduce the dependence on labeled data, enhance the model performance and adapt to different data types.

[0086] In operation S240, local feature alignment is performed on the local features of the image samples and the global features of the phrase samples using the local similarity between the local features of the image samples and the global features of the phrase samples, and a local alignment loss value is calculated.

[0087] Local feature alignment realizes the alignment of images and texts at a fine-grained level, and the local alignment process is also based on contrastive learning methods.

[0088] In operation S250 , mask modeling is performed on the preprocessed remote sensing image samples and the preprocessed text samples, respectively, and mask loss values ​​of the remote sensing image samples and the text samples in the mask modeling process are calculated, respectively.

[0089] Mask Modeling In computer vision tasks, masks can be used to mark specific areas in an image, such as object detection or image segmentation. A mask is a matrix of the same size as the image, where each element represents the state of the corresponding pixel, which can better process remote sensing images.

[0090] In operation S260, a weighted calculation is performed on the global alignment loss value, the local alignment loss value, and the mask loss value to obtain a weighted loss value, and the weighted loss value is used to optimize the parameters of the cross-modal retrieval model.

[0091] In operation S270, feature extraction operations of remote sensing image samples and text samples, weighted loss value calculation operations, and model parameter optimization operations are iteratively performed until preset training conditions are met, thereby obtaining a trained cross-modal retrieval model.

[0092] The training method of the cross-modal retrieval model provided by the present invention can be used in the field of remote sensing image text retrieval technology. By extracting the global and local features of remote sensing image samples and text samples, and aligning the global features and local features based on contrast learning, not only the feature alignment of image and text samples on a large scale is achieved, but also the feature alignment of image and text samples on a fine granularity can be achieved; through image and text sample mask modeling, the correlation learning of single modal features is enhanced, and the matching ability of images and texts is further improved. In addition, by combining the global similarity and local similarity scores, the accuracy of remote sensing image and text retrieval is significantly improved.

[0093] According to an embodiment of the present invention, the above-mentioned predefined global alignment loss function and the predefined local alignment loss function are constructed based on the contrastive learning loss function; wherein, the predefined image mask loss function is constructed based on the minimum square error function; wherein, the predefined text mask loss function is constructed based on the cross entropy loss function.

[0094] The training process of the cross-modal retrieval model involved in the above embodiment is further explained in detail below through specific embodiments.

[0095] The training process of the cross-modal retrieval model of the present invention mainly involves feature extraction of remote sensing images and texts, global feature alignment, local feature alignment, mask modeling, training and reasoning.

[0096] Among them, global feature alignment: use contrastive learning method to align the global features of images and texts to ensure the consistency of global semantics.

[0097] Among them, local feature alignment: by extracting conceptual phrases in the text and aligning them with local area features in the image, the relationship between the image and text is captured at a fine-grained level.

[0098] Among them, mask modeling: introduce mask modeling strategy to enhance the correlation learning of unimodal features and further improve the matching ability of images and texts.

[0099] Among them, training and reasoning: In the training phase, the training is performed by weighting the global and local alignment losses, and additionally using the mask modeling losses of the image and text. In the reasoning phase, the global similarity and local similarity are combined to obtain the final retrieval similarity score by weighted average, and the influence of the two is balanced by the weight parameter.

[0100] The following is a detailed description of each operation involved in the cross-modal retrieval model training process through specific embodiments.

[0101] According to an embodiment of the present invention, the above-mentioned preprocessing of remote sensing image samples and using the visual encoder of the cross-modal retrieval model to extract features of the preprocessed remote sensing image samples to obtain global features and local features of the image samples include: based on the dimensional information of the remote sensing image, performing image segmentation on the remote sensing image samples to obtain multiple non-overlapping rectangular area image samples; using the learnable linear mapping module of the cross-modal retrieval model to project each rectangular area image sample into a one-dimensional space to obtain multiple initial image sample local features; adding a special code for aggregating global information to each initial image sample local feature to obtain an initial image sample local feature with a special code; adding an image position code to each initial image sample local feature with a special code to obtain multiple encoded initial image sample local features; using the visual encoder of the cross-modal retrieval model to process all encoded initial image sample local features to obtain local features and global features of the image samples.

[0102] According to an embodiment of the present invention, the above-mentioned visual encoder is constructed based on the Vision Transformer model; wherein, the text encoder is constructed based on the Bert model; wherein, the predefined natural language processing tool includes the Spacy library.

[0103] In the above embodiment, the process of extracting global features and local features of remote sensing image samples utilizes a visual encoder based on Vision Transformer.

[0104] Specifically, given an input image , express dimensional Euclidean space (the same below), first divide the image into non-overlapping rectangular regions, where , and Represent the height, width and number of channels of the image respectively. is the side length of the rectangular area. The features of each rectangular area are projected into a one-dimensional vector space through a learnable linear mapping to obtain the feature representation of each rectangular area ,in is the feature dimension, Indicates an image (the same below), Indicates the number or quantity (the same below), that is, Then, add position encoding and special features [CLS] to all rectangular region features. [CLS] features are used to aggregate global information representing the entire image. Then, all feature vectors are input into In the Transformer model of the layer, the global features of the image are obtained And the local features of each rectangular area ,in, That is, global, which means global (the same below), is expressed as follows using formula (1):

[0105] (1),

[0106] in, represents the CLIP visual encoder, It is the global feature obtained by [CLS] feature aggregation, are local features of different rectangular areas.

[0107] According to an embodiment of the present invention, the above-mentioned preprocessing of text samples and using the text encoder of the cross-modal retrieval model to perform feature extraction on the preprocessed text samples to obtain global features of the text samples and global features of phrase samples include: using a byte pair encoding method to segment the text samples and thereby obtain multiple phrases from the text samples, and inserting a start identifier and an end identifier at the start position and the end position of the text sample respectively; using the encoding matrix of the cross-modal retrieval model to map each phrase into a word vector, and adding a text position code to each word vector; using the text encoder of the cross-modal retrieval model to perform feature extraction on the start identifier, the end identifier, and all word vectors with text position codes to obtain global text features; using a predefined natural language processing tool to extract multiple noun phrases from the text sample and thereby complete the enhancement operation on the text sample, and using the text encoder to perform global feature extraction on each noun phrase to obtain global features of the phrase sample.

[0108] The feature extraction process of the above text samples utilizes the text encoder of the cross-model retrieval model, where the text encoder is built based on the Bert model.

[0109] For a given input text First, the text is segmented by Byte Pair Encoding (BPE), and the special words [SOS] and [EOS] are inserted at the beginning and end of the text respectively. Then, each word in the text is mapped to a word vector through the encoding matrix, and the positional encoding is added. Then, all word vectors are passed through The layer Transformer obtains the feature representation of each word ,in, That is, start of sentence, indicating the beginning of the text (the same below), That is, end of sentence, indicating the end of the text (the same below), That is, text, which means text (the same below). As a global feature of text , is the local feature of the text. The process is expressed as follows using formula (2):

[0110] (2)

[0111] in, represents the CLIP text encoder, is the global feature of the text, is the local feature of each word in the text.

[0112] In addition to feature extraction of the entire text, the present invention also performs phrase enhancement (Phrase Argumentation) on each independent phrase in the text. The present invention uses the well-known NLP processing tool spacy library to extract features from the text. Extract Noun phrase , and each noun phrase is fed into the text encoder separately The global and local features of each noun phrase are obtained in . Here, only the global features of each noun phrase are used. For example, for the sentence "an empty port is nearsome buildings and many green trees.", three noun phrases {"an empty port", "somebuildings", "many green trees"} are extracted. Then the three noun phrases are sent to the text encoder respectively to obtain the global features of the three noun phrases. , That is, text phrase, which means a text phrase (the same below). Indicates The above process can be expressed by formula (3) as follows:

[0113] (3),

[0114] in, Represents the noun phrase extraction tool in the SpaCy library.

[0115] According to an embodiment of the present invention, the above-mentioned contrastive learning method is based on calculating the global similarity between the constructed positive sample pairs and the negative sample pairs to perform global feature alignment on the global features of the image samples and the global features of the text samples, and calculating the global alignment loss value in the global feature alignment process, including: constructing positive sample pairs and negative sample pairs based on the pairing information between the remote sensing image samples and the text samples; based on the contrastive learning method, respectively calculating the global similarity of the positive sample and the negative sample pairs to obtain the positive global similarity and the negative global similarity; using the positive global similarity and the negative global similarity to perform global feature alignment on the global features of the image samples and the global features of the text samples; using a predefined global alignment loss function to calculate the global alignment loss value in the global feature alignment process.

[0116] The above-mentioned embodiments involve the global feature alignment operation of the cross-modal retrieval model. The present invention aligns the global features of images and texts through contrastive learning. Contrastive learning methods have been widely used in tasks such as image recognition and natural language processing. Especially in cross-modal learning, they can effectively align feature representations of different modalities. The core idea is to bring similar samples (positive samples) closer and push dissimilar samples (negative samples) farther away. In this process, the similarity between positive samples and negative samples will be pulled apart to ensure that the model can learn more critical features. Specifically in the image-text retrieval task, paired image-text pairs are treated as positive samples, and unpaired images and texts are treated as negative samples. Use cosine similarity to measure the similarity between image and text feature vectors, record is the global similarity between the image and the text, For the Pictures and The global similarity between texts, Indicates Pictures and The global similarity between the texts, then the loss of global alignment As shown in formula (4):

[0117] (4),

[0118] in, Represents the global similarity between positive pairs of images and texts, is the batch size, A hyperparameter for controlling temperature in contrastive learning.

[0119] According to an embodiment of the present invention, the above-mentioned method of using the local similarity between the local features of the image samples and the global features of the phrase samples to perform local feature alignment on the local features of the image samples and the global features of the phrase samples, and calculating the local alignment loss value during the local feature alignment process includes: calculating the cosine similarity between the global features of the phrase samples and the local features of the image samples; averaging all cosine similarities above a preset similarity threshold to obtain the local similarity; using the local similarity to perform local feature alignment on the local features of the image samples and the global features of the phrase samples; and using a predefined local alignment loss function to calculate the local-to-partial alignment loss value during the local feature alignment process.

[0120] The above embodiments involve the local feature alignment operation of cross-modal retrieval model training. In addition to global alignment, the present invention also aligns fine-grained local features to supplement the details ignored in global alignment. The global features of the noun phrases are aligned with the local features of the image, that is, for each noun phrase, find the rectangular area of ​​the image whose similarity is greater than a certain threshold for alignment. Specifically, For the picture The rectangular area and the text The local similarity between noun phrases, where That is, local, which means local (the same below). Noun phrases, their similarity to the image is the average similarity of the rectangular areas with higher similarity, which can be expressed by formula (5) as follows:

[0121] (5),

[0122] in, is a threshold hyperparameter, Indicates Noun phrases are more similar than The local similarity set of Representing a collection Next, we use the average similarity between all name phrases in a text and the image as the local similarity between the text and the image. , as shown in formula (6):

[0123] (6),

[0124] in, is the number of noun phrases in a text. For the Pictures and The local similarity between texts, For the Pictures and The local similarity between the texts is then the local alignment loss As shown in formula (7):

[0125] (7),

[0126] in, Represents the local similarity between the positive pairing of images and texts, is the batch size, is the temperature hyperparameter.

[0127] According to an embodiment of the present invention, the above-mentioned mask modeling is performed on the preprocessed remote sensing image samples and the preprocessed text samples respectively, and the mask loss values ​​of the remote sensing image samples and the text samples in the mask modeling process are calculated respectively, including: based on a preset image mask ratio, randomly masking the preprocessed remote sensing image samples to obtain masked remote sensing image samples; using the visual encoder of the cross-modal retrieval model to extract features of the masked remote sensing image samples to obtain multiple mask feature samples; using the visual mask regression head of the cross-modal retrieval model to process all mask feature samples to obtain reconstructed remote sensing image samples; using a predefined image mask loss function to process the reconstructed remote sensing image samples and remote sensing image samples to obtain image mask loss values.

[0128] The above embodiments involve the image mask modeling operation in the cross-modal retrieval model training process. In order to further capture the correlation of features within a single modality, the present invention introduces a mask modeling strategy. The idea of ​​mask modeling is to randomly mask a part of the input, and then let the model predict the masked part through the unmasked part, so as to learn the internal correlation of the input data.

[0129] In order to better understand the local information in the image and capture the relationship between rectangular areas, the present invention adopts visual mask modeling. The present invention randomly masks 75% of the image area to obtain the masked image .Will Input into the visual encoder to obtain the features of each rectangular area. Then these features are passed through a visual mask regression head to reconstruct the original image, and the reconstructed value is recorded as The least square error is used to optimize the masked part of the reconstructed image and the original image. MVM stands for Masked Visual Modeling. represents the visual mask modeling loss, as shown in formula (8):

[0130] (8),

[0131] in, Represents the set of subscripts that are masked out in the rectangular area of ​​the image.

[0132] According to an embodiment of the present invention, the above-mentioned mask modeling is performed on the preprocessed remote sensing image samples and the preprocessed text samples respectively, and the mask loss values ​​of the remote sensing image samples and the text samples in the mask modeling process are calculated respectively, which also includes: based on a preset image mask ratio, randomly replacing some words in the text sample with mask specific words to obtain masked text samples; using the text encoder of the cross-modal retrieval model to extract features of the masked text sample to obtain multiple word features; using the text mask classification head of the cross-modal retrieval model to predict each word feature to obtain predicted text; using a predefined text mask loss function to process the text sample and the predicted text to obtain a text mask loss value.

[0133] The above embodiment involves the text mask modeling operation in the cross-modal retrieval model training process. Text mask modeling is very common in language models and multimodal models and is used to enhance the contextual understanding ability of the model. The present invention randomly replaces 15% of the words with the special word [MASK] to obtain the masked text Next, Input into the text encoder to get the features of each word. Then pass the features through a text mask classification head to predict the probability of the word. The probability is recorded as ,in yes The number of words in is the size of the word dictionary in CLIP. Use cross entropy loss To optimize the prediction probability, as shown in formula (9):

[0134] (9),

[0135] in, represents the set of subscripts of the masked words, Indicates The index of a word in the vocabulary. Indicates The words are predicted to be probability.

[0136] In the training and reasoning stages of the cross-modal retrieval model, the present invention uses a weighted average of the global alignment loss, the local alignment loss, and the mask modeling loss to train the model, as shown in formula (10):

[0137] (10),

[0138] in, is a hyperparameter used to balance the importance of different loss functions. Specifically, in the experiment, the present invention sets .

[0139] In the reasoning stage, the present invention combines the global similarity and local similarity Calculate the final retrieval similarity score , as shown in formula (11):

[0140] (11),

[0141] in, is a weighting coefficient that controls the importance of global similarity and local similarity. The value of can balance the contribution of global information and local information and optimize the retrieval performance. Usually, The value of can be determined experimentally to maximize the retrieval accuracy.

[0142] The training method of the cross-modal retrieval model of the present invention captures the relationship between image and text at a fine-grained level by extracting conceptual phrases from the text and aligning them with local area features in the image, thus supplementing the neglect of fine-grained features in global alignment. At the same time, a mask modeling strategy is introduced to enhance the correlation learning of unimodal features and further improve the matching ability of images and texts. Finally, the global similarity and local similarity scores are combined to significantly improve the accuracy of remote sensing image and text retrieval.

[0143] Figure 3 4 is a flowchart of a remote sensing image text retrieval method according to an embodiment of the present invention.

[0144] like Figure 3 As shown, the remote sensing image text retrieval method of this embodiment includes operations S310 to S340.

[0145] In operation S310, the remote sensing image and the target text are preprocessed respectively using the trained cross-modal retrieval model to obtain a preprocessed remote sensing image and a preprocessed target text, wherein the trained cross-modal retrieval model is trained based on the above-mentioned training method of the cross-modal retrieval model for remote sensing image text retrieval.

[0146] The present invention firstly constructs a cross-modal retrieval model based on a pre-trained neural network, trains the model, and uses the trained model to realize retrieval between remote sensing images and target text.

[0147] Before retrieving remote sensing images and target texts, they need to be preprocessed to improve matching accuracy and retrieval efficiency.

[0148] In operation S320, the trained cross-modal retrieval model is used to perform feature extraction on the preprocessed remote sensing image to obtain image global features and image local features, and feature extraction is performed on the preprocessed target text to obtain text global features and phrase global features.

[0149] In the process of image feature extraction, a visual encoder based on VisionTransformer is used; in the process of target text feature extraction, a text encoder based on the Bert model is used.

[0150] In operation S330 , the trained cross-modal retrieval model is used to calculate the global similarity between the image global features and the text global features, and to calculate the local similarity between the image local features and the phrase global features.

[0151] Local similarity can achieve fine-grained cross-modal retrieval and can supplement the lack of global similarity in matching detailed features. The present invention achieves fine-grained cross-modal matching, thereby greatly improving matching accuracy and retrieval accuracy.

[0152] In operation S340, the global similarity and the local similarity are weightedly calculated using the trained cross-modal retrieval model to obtain weighted similarity, and the matching results between the remote sensing image and the target text are screened based on the weighted similarity to obtain retrieval results between the remote sensing image and the target text.

[0153] In the process of weighted similarity calculation, those skilled in the art can adjust the weighting coefficient according to actual needs, thereby balancing the contribution of global information and local information, thereby optimizing the retrieval performance. The weighting coefficient can also be determined through experiments.

[0154] The remote sensing image text retrieval method provided by the present invention realizes high-precision matching between remote sensing images and target texts by using a trained cross-modal retrieval model; the remote sensing image text retrieval method provided by the present invention extracts the features of the remote sensing image and the target text, and locally aligns the features of the remote sensing image and the target text, captures the relationship between the image and the text at a fine-grained level, and supplements the neglect of fine-grained features in global alignment; at the same time, due to the combination of global similarity and local similarity, the cross-modal retrieval accuracy between the remote sensing image and the target text is significantly improved.

[0155] Figure 4 4 is a structural block diagram of a remote sensing image text retrieval device according to an embodiment of the present invention.

[0156] like Figure 4As shown, the remote sensing image text retrieval device 400 of this embodiment includes an image and text preprocessing module 410 , a feature extraction module 420 , a similarity calculation module 430 and a weighting and screening module 440 .

[0157] The image and text preprocessing module 410 is used to train the cross-modal retrieval model based on the multimodal pre-trained neural network to obtain a trained cross-modal retrieval model, and use the trained cross-modal retrieval model to preprocess the remote sensing image and the target text respectively to obtain the preprocessed remote sensing image and the preprocessed target text, wherein the trained cross-modal retrieval model is trained based on the above-mentioned training method of the cross-modal retrieval model for remote sensing image text retrieval; in one embodiment, the image and text preprocessing module 410 can be used to perform the operation S310 described above, which will not be repeated here.

[0158] The feature extraction module 420 is used to use the trained cross-modal retrieval model to perform feature extraction on the preprocessed remote sensing image to obtain image global features and image local features, and to perform feature extraction on the preprocessed target text to obtain text global features and phrase global features; in one embodiment, the feature extraction module 420 can be used to perform the operation S320 described above, which will not be repeated here.

[0159] The similarity calculation module 430 is used to calculate the global similarity between the global features of the image and the global features of the text using the trained cross-modal retrieval model, and calculate the local similarity between the local features of the image and the global features of the phrase; in one embodiment, the similarity calculation module 430 can be used to perform the operation S330 described above, which will not be repeated here.

[0160] The weighting and filtering module 440 is used to use the trained cross-modal retrieval model to perform weighted calculations on the global similarity and the local similarity to obtain the weighted similarity, and to filter the matching results between the remote sensing image and the target text based on the weighted similarity to obtain the retrieval results between the remote sensing image and the target text; in one embodiment, the weighting and filtering module 440 can be used to perform the operation S340 described above, which will not be repeated here.

[0161] According to an embodiment of the present invention, any multiple modules of the image and text preprocessing module 410, the feature extraction module 420, the similarity calculation module 430, and the weighting and screening module 440 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the image and text preprocessing module 410, the feature extraction module 420, the similarity calculation module 430, and the weighting and screening module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the image and text preprocessing module 410, the feature extraction module 420, the similarity calculation module 430, and the weighting and screening module 440 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0162] Figure 5 It is a block diagram of an electronic device suitable for implementing a training method for a cross-modal retrieval model and a remote sensing image text retrieval method according to an embodiment of the present invention.

[0163] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0164] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0165] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that the computer program read therefrom is installed into the storage portion 508 as needed.

[0166] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0167] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0168] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0170] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for training a cross-modal retrieval model, characterized in that: The method comprises: The visual encoder of the cross-modal retrieval model is used to extract features from the preprocessed remote sensing image samples to obtain the global features and local features of the image samples. Using the text encoder of the cross-modal retrieval model to extract features from the preprocessed text samples, and obtaining global features of the text samples and global features of the phrase samples; Based on the contrastive learning method, the global features of the image sample and the global features of the text sample are globally aligned by calculating the global similarity between the constructed positive sample pair and the negative sample pair, and the global alignment loss value is calculated; Performing local feature alignment on the local features of the image samples and the global features of the phrase samples by using the local similarity between the local features of the image samples and the global features of the phrase samples, and calculating a local alignment loss value; Performing mask modeling on the preprocessed remote sensing image samples and the preprocessed text samples respectively, and calculating mask loss values ​​of the remote sensing image samples and the text samples in the mask modeling process respectively; The global alignment loss value, the local alignment loss value and the mask loss value are weightedly calculated to obtain a weighted loss value, and the weighted loss value is used to optimize the parameters of the cross-modal retrieval model.

2. The method according to claim 1, characterized in that Also includes: Iteratively performing a feature extraction operation, a weighted loss value calculation operation, and a model parameter optimization operation on the remote sensing image sample and the text sample until a preset training condition is met, thereby obtaining a trained cross-modal retrieval model; The visual encoder of the cross-modal retrieval model is used to extract features from the preprocessed remote sensing image samples to obtain global features and local features of the image samples, including: Based on the dimensional information of the remote sensing image, the remote sensing image sample is segmented to obtain a plurality of non-overlapping rectangular area image samples; Using the learnable linear mapping module of the cross-modal retrieval model, each rectangular area image sample is projected into a one-dimensional space to obtain a plurality of local features of the initial image samples; Adding a special code for aggregating global information to each of the local features of the initial image samples to obtain the local features of the initial image samples having the special code; Adding an image position code to each local feature of the initial image sample having the special code to obtain a plurality of encoded local features of the initial image sample; The visual encoder of the cross-modal retrieval model is used to process all the encoded local features of the initial image samples to obtain local features and global features of the image samples.

3. The method according to claim 1, characterized in that The text encoder of the cross-modal retrieval model is used to extract features from the preprocessed text samples, and the global features of the text samples and the global features of the phrase samples are obtained, including: Segmenting the text sample using a byte pair encoding method to obtain a plurality of phrases from the text sample, and inserting a start identification word and an end identification word at the start position and the end position of the text sample respectively; Mapping each of the phrases into a word vector using the encoding matrix of the cross-modal retrieval model, and adding a text position code to each of the word vectors; Using the text encoder of the cross-modal retrieval model, feature extraction is performed on the start identification word, the end identification word, and all word vectors with text position encoding to obtain the global features of the text sample; A predefined natural language processing tool is used to extract multiple noun phrases from the text sample to complete the enhancement operation of the text sample, and the text encoder is used to extract global features of each noun phrase to obtain the global features of the phrase sample.

4. The method according to claim 2 or 3, characterized in that: The visual encoder is constructed based on the VisionTransformer model; Wherein, the text encoder is constructed based on the Bert model; Wherein, the predefined natural language processing tool includes the Spacy library.

5. The method according to claim 1, characterized in that: Based on the contrastive learning method, the global features of the image sample and the global features of the text sample are analyzed by calculating the global similarity between the constructed positive sample pair and the negative sample pair. Alignment and calculation of the global alignment loss value include: Based on the pairing information between the remote sensing image sample and the text sample, construct a positive sample pair and a negative sample pair; Based on the contrastive learning method, respectively calculating the global similarity of the positive sample and the negative sample pair to obtain a positive global similarity and a negative global similarity; Performing global feature alignment on the image sample global feature and the text sample global feature using the positive global similarity and the negative global similarity; The predefined global alignment loss function is used to calculate the global alignment loss value in the global feature alignment process.

6. The method according to claim 1, characterized in that Performing local feature alignment on the local features of the image sample and the global features of the phrase sample by using the local similarity between the local features of the image sample and the global features of the phrase sample, and calculating the local alignment loss value comprises: Calculating the cosine similarity between the global features of the phrase sample and the local features of the image sample; Averaging all the cosine similarities above a preset similarity threshold to obtain the local similarity; Performing local feature alignment on the local features of the image sample and the global features of the phrase sample by using the local similarity; The predefined local alignment loss function is used to calculate the local-to-partial alignment loss value in the local feature alignment process.

7. The method according to claim 1, characterized in that Performing mask modeling on the preprocessed remote sensing image sample and the preprocessed text sample respectively, and calculating the mask loss values ​​of the remote sensing image sample and the text sample in the mask modeling process respectively includes: Based on a preset image mask ratio, randomly masking the preprocessed remote sensing image samples to obtain masked remote sensing image samples; Using the visual encoder of the cross-modal retrieval model to extract features from the masked remote sensing image samples, to obtain a plurality of mask feature samples; Processing all the mask feature samples using the visual mask regression head of the cross-modal retrieval model to obtain reconstructed remote sensing image samples; The reconstructed remote sensing image sample and the remote sensing image sample are processed using a predefined image mask loss function to obtain an image mask loss value.

8. The method according to claim 7, characterized in that Also includes: Based on a preset image mask ratio, randomly replacing some words in the text sample with masked specific words to obtain a masked text sample; Using the text encoder of the cross-modal retrieval model to perform feature extraction on the masked text sample to obtain multiple word features; Using the text mask classification head of the cross-modal retrieval model to predict each of the word features, to obtain predicted text; The text sample and the predicted text are processed using a predefined text mask loss function to obtain a text mask loss value.

9. The method according to any one of claims 5 to 8, characterized in that: The predefined global alignment loss function and the predefined local alignment loss function are constructed based on the contrastive learning loss function; Wherein, the predefined image mask loss function is constructed based on a minimum square error function; Wherein, the predefined text mask loss function is constructed based on the cross entropy loss function.

10. A remote sensing image text retrieval method, characterized in that: The method comprises: Preprocessing the remote sensing image and the target text respectively using the trained cross-modal retrieval model to obtain the preprocessed remote sensing image and the preprocessed target text, wherein the trained cross-modal retrieval model is trained based on the training method described in any one of claims 1 to 9; Using the trained cross-modal retrieval model, feature extraction is performed on the pre-processed remote sensing image to obtain image global features and image local features, and feature extraction is performed on the pre-processed target text to obtain text global features and phrase global features; Calculating the global similarity between the image global features and the text global features, and calculating the local similarity between the image local features and the phrase global features using the trained cross-modal retrieval model; The global similarity and the local similarity are weightedly calculated using the trained cross-modal retrieval model to obtain a weighted similarity, and the matching results between the remote sensing image and the target text are screened based on the weighted similarity to obtain a retrieval result between the remote sensing image and the target text.

Images