Multi-scale cross-modal-based satellite remote sensing image text retrieval method and system, terminal and medium

Abstract

The invention belongs to the field of image retrieval, and particularly relates to a multi-scale cross-modal-based satellite remote sensing image text retrieval method and system, a terminal and a medium, and the method comprises the following steps: constructing a multi-source multi-scale cross-modal remote sensing image semantic understanding model; processing a plurality of source satellite remote sensing images by using a multi-source multi-scale cross-modal remote sensing image semantic understanding model to obtain source feature vectors of the source satellite remote sensing images; storing the source feature vectors in a vector library; processing the input image retrieval information or text retrieval information by using the multi-source multi-scale cross-modal remote sensing image semantic understanding model to obtain an image retrieval feature vector or a text retrieval feature vector; and performing similarity comparison on the image retrieval feature vector or the text retrieval feature vector and a source feature vector in a vector library, and screening out a target source satellite remote sensing image. According to the method, the robustness of text and image features is improved, a unified storage and management carrier for multi-source data features is constructed, and the management efficiency and the retrieval precision and efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the field of image retrieval, and specifically relates to a method, system, terminal and medium for satellite remote sensing image text retrieval based on multi-scale cross-modality. Background Art

[0002] As remote sensing image data continues to increase, the ways and means of data acquisition become more diverse. The rapid and efficient retrieval of images containing specific targets and scenes has an important impact on the application of downstream businesses.

[0003] Traditional retrieval methods based on keywords and attributes are single-minded, cumbersome, and unable to fully explore the deep semantic features of images. Currently, technologies such as natural language processing (NLP) and multimodal models are developing rapidly, providing important support for cross-modal remote sensing image semantic retrieval, but there are still some challenges, such as insufficient utilization of multi-level spatial and semantic information in images, and lack of unified storage and management carriers for multi-source data features. Summary of the invention

[0004] To solve the above problems, the present invention provides a method, system, terminal and medium for image and text retrieval based on multi-scale and cross-modal satellite remote sensing images. The multi-level spatial information and semantic information in the multi-source images are fully utilized based on the multi-source and multi-scale cross-modal remote sensing image semantic understanding model to improve the robustness of text and image features. Based on the vector library technology, a unified storage and management carrier for multi-source data features is constructed to improve management efficiency. The combination of the two improves retrieval accuracy and efficiency.

[0005] In a first aspect, the technical solution of the present invention provides a method for image and text retrieval based on multi-scale cross-modal satellite remote sensing images, comprising the following steps: Construct a semantic understanding model for multi-source, multi-scale, and cross-modal remote sensing images; Use a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to process several source satellite remote sensing images and obtain source feature vectors of the source satellite remote sensing images; Store the source feature vector into the vector library; Using a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to process input image retrieval information or text retrieval information to obtain an image retrieval feature vector or a text retrieval feature vector; The image retrieval feature vector or text retrieval feature vector is compared with the source feature vector in the vector library for similarity, and the target source satellite remote sensing image is screened out.

[0006] In an optional implementation, a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model is constructed, specifically including: Build a multimodal pre-training model, which includes an image encoder and a text encoder. The image encoder uses a visual Transformer structure, and the text encoder uses a Transformer structure. Use the machine vision image-text dataset to pre-train the multimodal pre-training model, including encoding images and texts into image feature vectors and text feature vectors respectively, using the InfoNCE loss function to maximize the cosine similarity of N paired image-text pairs in the machine vision image-text dataset, and minimize the cosine similarity of the remaining unpaired image-text pairs, to achieve multimodal feature space alignment of image-text pairs; The pre-built multi-source remote sensing multi-scale image and text dataset is used to fine-tune all parameters of the pre-trained multimodal pre-trained model to obtain the trained multimodal pre-trained model, namely, the multi-source multi-scale cross-modal remote sensing image semantic understanding model.

[0007] In an optional embodiment, the method further includes the step of constructing a multi-source remote sensing multi-scale image and text dataset, specifically including: Label remote sensing image and text data pairs; Use annotated remote sensing image and text data to fine-tune all parameters of the multimodal large language model; The remote sensing image to be annotated is used as the inference data of the multimodal large language model after fine-tuning all parameters to obtain the annotated text of the remote sensing image; Use Kmeans clustering method to retain M annotation texts with different feature distribution for each remote sensing image; Use iterative training to use the data-enhanced remote sensing image-text data pairs as the multimodal large language model fine-tuning data again, and repeat the above steps to enhance the quality of remote sensing image-text data pairs; The final remote sensing image and text data pair is a constructed multi-source remote sensing multi-scale image and text dataset.

[0008] In an optional implementation, a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model is used to process the input text retrieval information to obtain a text retrieval feature vector, specifically including: Use natural language processing technology to process the input text retrieval information to obtain retrieval conditions and scene descriptions; The scene description is input into the text encoder of the multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to obtain the text retrieval feature vector.

[0009] In an optional implementation, a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model is used to process a plurality of source satellite remote sensing images to obtain source feature vectors of the source satellite remote sensing images, specifically including: Preprocess several source satellite remote sensing images to obtain multi-scale image blocks, including multi-scale sampling, image slicing, bit conversion, and noise elimination; A multi-source, multi-scale, cross-modal remote sensing image semantic understanding model is used to encode multi-scale image blocks to obtain the source feature vectors of the source satellite remote sensing images.

[0010] In an optional implementation, storing the source feature vector in a vector library specifically includes: Obtain metadata information of each multi-scale image block; Indexing the source feature vectors and corresponding metadata information of each multi-scale image block; Store the indexed source feature vector into a vector library.

[0011] In an optional implementation, the text retrieval feature vector is compared with the source feature vector in the vector library for similarity, and the target source satellite remote sensing image is screened out, specifically including: Compare the search conditions with the metadata information, select several source feature vectors that match the search conditions, and record them as initial source feature vectors; Compare the text retrieval feature vector with the initial source feature vector for similarity, and filter out the intermediate source feature vector; Use Kmeans clustering method to cluster the intermediate source feature vectors in descending order of similarity; In the category with higher similarity, the category with more than 1 / 2 number is selected as the positive category, which is the final source feature vector retrieved; The source satellite remote sensing image corresponding to the final source feature vector is the target source satellite remote sensing image.

[0012] In a second aspect, the technical solution of the present invention provides a satellite remote sensing image text retrieval system based on multi-scale cross-modality, comprising: Semantic understanding model building module, used to build a semantic understanding model for multi-source, multi-scale, cross-modal remote sensing images; A source feature vector acquisition module is used to process several source satellite remote sensing images using a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to obtain source feature vectors of the source satellite remote sensing images; A vector storage module, used for storing the source feature vector into a vector library; A retrieval feature vector acquisition module is used to process the input image retrieval information or text retrieval information using a multi-source multi-scale cross-modal remote sensing image semantic understanding model to obtain an image retrieval feature vector or a text retrieval feature vector; The target image retrieval module is used to compare the similarity between the image retrieval feature vector or the text retrieval feature vector and the source feature vector in the vector library, and screen out the target source satellite remote sensing image.

[0013] In a third aspect, the technical solution of the present invention provides a terminal, including: A memory device for storing a graphic and text retrieval program based on multi-scale and cross-modal satellite remote sensing images; A processor is used to implement the steps of the satellite remote sensing image image and text retrieval method based on multi-scale cross-modality as described in any of the above items when executing the satellite remote sensing image image and text retrieval program based on multi-scale cross-modality.

[0014] In a fourth aspect, the technical solution of the present invention provides a computer-readable storage medium, on which is stored a satellite remote sensing image graphic and text retrieval program based on multi-scale cross-modality. When the satellite remote sensing image graphic and text retrieval program based on multi-scale cross-modality is executed by a processor, the steps of the satellite remote sensing image graphic and text retrieval method based on multi-scale cross-modality as described in any one of the above items are implemented.

[0015] The present invention provides a satellite remote sensing image graphic and text retrieval method, system, terminal and medium based on multi-scale cross-modality, which have the following beneficial effects compared with the prior art: firstly, a multi-source multi-scale cross-modality remote sensing image semantic understanding model is constructed, which takes into account the multi-modal differences between images and texts, and can make full use of the multi-level spatial information and semantic information in the multi-source images; then, based on the multi-source multi-scale cross-modality remote sensing image semantic understanding model, the source satellite remote sensing image is encoded to obtain the source feature vector; and based on the vector library technology, a unified storage and management carrier for multi-source data features is constructed to improve management efficiency; when performing retrieval, the retrieval feature vector obtained from the image or text retrieval information is also based on the multi-source multi-scale cross-modality remote sensing image semantic understanding model; then, based on the similarity comparison of the feature vectors, the target influence is screened out from the vector library, effectively improving the retrieval accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 The present invention provides a schematic diagram of the architecture of a satellite remote sensing imagery graphic and text retrieval method based on multi-scale and cross-modality.

[0017] Figure 2 It is a schematic flow chart of a method for image and text retrieval based on multi-scale and cross-modal satellite remote sensing images provided by an embodiment of the present invention.

[0018] Figure 3 It is a schematic diagram of the architecture for building a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model.

[0019] Figure 4 It is a schematic diagram of the architecture for constructing a multi-source remote sensing multi-scale image and text dataset.

[0020] Figure 5 The present invention provides a schematic block diagram of a satellite remote sensing image text retrieval system based on multi-scale and cross-modality.

[0021] Figure 6 It is a schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0024] The key terms appearing in the present invention are explained below.

[0025] CLIP: Contrastive Language–Image Pre-training, is a multimodal vision and text learning method.

[0026] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen ImageEncoders and Large Language Models is a multimodal large language model.

[0027] ViT structure: Vision Transformer (ViT), visual Transformer structure, is a deep learning model based on the Transformer architecture, mainly used for image recognition and computer vision tasks.

[0028] Multiscale: Multiscale refers to the use of different scales for observation and processing when analyzing or processing data. For remote sensing images, multiscale methods can obtain image information through different resolutions or spatial scales.

[0029] Remote sensing images: Remote sensing images refer to images of the earth's surface obtained from a distance (such as satellites or drones) through remote sensing technology. They contain information in different bands such as visible light, infrared, and microwaves, and are used to monitor and analyze the geographical environment, meteorological changes, vegetation coverage, water distribution, etc.

[0030] Cross-modality: Cross-modality refers to the process of converting and matching information between different modalities (such as text, images, audio, etc.). In remote sensing imagery, cross-modality usually involves associating and retrieving images (image modality) with text information describing the image content (text modality).

[0031] Image-text retrieval: Image-text retrieval refers to a method of retrieving information based on text descriptions or image content. In cross-modal retrieval, image-text retrieval usually involves two forms: one is to input text for image retrieval, where users enter keywords or descriptions to find images related to them; the other is to input images for text retrieval, where users provide images to find text descriptions related to the image content. The goal of image-text retrieval is to establish a matching relationship between images and texts, thereby achieving accurate retrieval results.

[0032] In view of the problems of current cross-modal remote sensing image semantic retrieval, such as under-utilization of multi-level spatial information and semantic information in images, lack of unified storage and management carrier for multi-source data features, this embodiment provides a satellite remote sensing image text retrieval method based on multi-scale cross-modal. Figure 1 This is a schematic diagram of the architecture of the method, in which a multi-source multi-scale cross-modal remote sensing image semantic understanding model performs image encoding processing on a multi-source remote sensing image to obtain a feature vector, which is then stored in a vector library. Users search for images through images or texts, and use the image encoder of the multi-source multi-scale cross-modal remote sensing image semantic understanding model to encode the input image to obtain a feature vector, which is then queried from the vector library based on the feature vector to implement image search, and the text encoder of the multi-source multi-scale cross-modal remote sensing image semantic understanding model to encode the input text to obtain a feature vector, which is then queried from the vector library based on the feature vector to implement text search. This embodiment fully utilizes the multi-level spatial information and semantic information in multi-source images based on the multi-source multi-scale cross-modal remote sensing image semantic understanding model, improves the robustness of text and image features, and builds a unified storage and management carrier for multi-source data features based on vector library technology to improve management efficiency. The combination of the two improves retrieval accuracy and efficiency.

[0033] Figure 2 1 is a schematic flow chart of a method for multi-scale cross-modal satellite remote sensing image text retrieval based on an embodiment of the present invention. Figure 2The execution subject may be a satellite remote sensing image text retrieval system based on multi-scale cross-modality. The satellite remote sensing image text retrieval method based on multi-scale cross-modality provided in the embodiment of the present invention is executed by a computer device, and accordingly, the satellite remote sensing image text retrieval system based on multi-scale cross-modality runs in the computer device. According to different requirements, the order of the steps in the flowchart can be changed, and some can be omitted.

[0034] like Figure 2 As shown, the method includes the following steps.

[0035] S1, build a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model.

[0036] Multi-source data contains rich information, including multi-level spatial information, semantic information, and temporal information. Constructing a cross-modal learning method for spatial semantics to extract and utilize multi-modal features from multi-source data is the key to improving the accuracy of remote sensing image retrieval.

[0037] This embodiment is based on a pre-built multi-source remote sensing multi-scale image and text dataset, combined with a large visual-language cross-modal pre-trained model, and uses transfer learning to obtain a multi-source multi-scale cross-modal remote sensing image semantic understanding model, which fully integrates the texture, band and multi-level spatial features of multi-source data to improve the accuracy and generalization ability of the model in remote sensing image and text retrieval. The multi-source multi-scale cross-modal remote sensing image semantic understanding model can extract robust image and text feature vectors, and achieve text-to-image alignment by calculating the distance between the two, that is, the similarity. Figure 3 It is a schematic diagram of the architecture for building a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model, which specifically includes the following steps.

[0038] S1.1, build a multimodal pre-training model. The multimodal pre-training model includes an image encoder and a text encoder. The image encoder uses a visual Transformer structure, and the text encoder uses a Transformer structure.

[0039] This embodiment introduces the CLIP model pre-trained with a machine vision image and text dataset, including a dual-branch structure of a pre-trained image encoder and a text encoder, wherein the image encoder uses the ViT structure (Vision Transformer), and this paper uses Vit-base; the text encoder also uses the Transformer structure.

[0040] S1.2, use the machine vision image and text dataset to pre-train the multimodal pre-training model, including encoding the image and text into image feature vectors and text feature vectors respectively, using the InfoNCE loss function to maximize the cosine similarity of N paired image and text pairs in the machine vision image and text dataset, and minimize the cosine similarity of the remaining unpaired image and text pairs, so as to achieve multimodal feature space alignment of image and text pairs.

[0041] In this embodiment, given a batch of image-text data pairs, CLIP encodes the image and text into 512-dimensional image feature vectors and text feature vectors respectively, and uses the InfoNCE loss function to maximize the cosine similarity of the N paired image-text pairs in the batch through training, and minimize the cosine similarity of the N²-N unpaired image-text pairs, so as to achieve multimodal feature space alignment of the image-text pairs.

[0042] The InfoNCE loss function is as follows:

[0043] In the formula, Encode the image; is the text encoding; T is the learnable temperature coefficient; N is the batch size.

[0044] S1.3, use the pre-built multi-source remote sensing multi-scale image and text dataset to fine-tune all parameters of the pre-trained multimodal pre-trained model to obtain the trained multimodal pre-trained model, that is, the multi-source multi-scale cross-modal remote sensing image semantic understanding model.

[0045] This embodiment uses the constructed high-quality multi-source remote sensing multi-scale image and text dataset to fine-tune all parameters of the pre-trained model (where the pre-training weight is clip-rsicd-v2, the batch size is set to 8, the learning rate is set to 5e-6, and the epoch is set to 100) to give full play to the ability of the pre-trained model to migrate downstream, enhance the model's feature alignment ability for remote sensing images and texts, and improve the model's ability to extract multi-scale image features.

[0046] In order to achieve full parameter fine-tuning of the multimodal pre-trained model, this embodiment pre-constructs a multi-source remote sensing multi-scale image and text dataset. However, compared with the training data of CLIP, the sample data that can be used for remote sensing vision-language pre-training model training is very scarce. The production cost of the image-text dataset based on manual annotation is high, and it is difficult to meet the accuracy and generalization requirements of the model. For this reason, this embodiment first annotates a small amount of high-quality remote sensing image-text data pairs, and then introduces the generative cross-modal model BLIP2, and uses a small amount of high-quality multi-source remote sensing image-text data to fine-tune all parameters of BLIP2, enhance the performance of the BLIP2 model in remote sensing image generation, and then use iterative training to continuously enhance the quality of the multi-source remote sensing image-text dataset. Figure 4 This is a schematic diagram of the architecture for constructing a multi-source remote sensing multi-scale image and text dataset, which specifically includes the following steps.

[0047] Step 1: Label remote sensing image and text data pairs.

[0048] Based on the manual annotation method, a small number of high-quality remote sensing image-text data pairs are annotated based on the developed remote sensing image-text annotation specifications based on color and spatial relationships.

[0049] Step 2: Use the annotated remote sensing image and text data to fine-tune all parameters of the multimodal large language model.

[0050] Introducing the generative cross-modal model BLIP2. BLIP-2 is a multimodal model that aims to effectively combine visual and language information, and achieve image-text generation and understanding tasks by reducing the alignment cost between visual encoding and language model. Its design goal is to connect a lightweight visual processor with a pre-trained language model (such as GPT or BERT) to reduce additional training requirements and maintain high performance.

[0051] The structure of BLIP-2 can be divided into the following main parts: Visual encoder: Use ViT (VisionTransformer) to extract features from images and output high-dimensional image representation (1024 dimensions). The visual encoder is a pre-trained CLIP model. Q-Former: Connects the visual encoder and the language model. Extracts important visual information through a small number of learnable queries and outputs features compatible with the language model. The main parameters include the number of queries (64), 12 Transformer layers, and 1024 hidden dimensions. Language model: Use a pre-trained language model (BERT) to process the output of Q-Former for text generation.

[0052] Step 3: Use the remote sensing image to be annotated as the inference data of the multimodal large language model after fine-tuning all parameters to obtain the annotated text of the remote sensing image.

[0053] In this embodiment, a small amount of high-quality multi-source remote sensing image-text data is used to fine-tune all parameters of BLIP2 to enhance the performance of the BLIP2 model in remote sensing image and text generation. The fine-tuning steps are as follows: load the pre-trained BLIP-2 model, set the optimizer to Adam, the learning rate to 1e-5, the weight decay parameter to 0.01, the loss function to use cross-modal contrast loss, and the batchsize to 100; then, use the remote sensing image to be annotated as the inference data of the fine-tuned BLIP2 model to obtain the annotated text of the remote sensing image.

[0054] Step 4: Use the Kmeans clustering method to retain M annotation texts with different feature distributions for each remote sensing image.

[0055] In order to improve the quality and diversity of samples, this embodiment proposes an offline enhancement strategy for remote sensing image and text data samples, and combines the Kmeans clustering method to retain 5 texts with large feature distribution differences for each image.

[0056] Step 5: Use iterative training to use the data-enhanced remote sensing image-text data pairs as the multimodal large language model fine-tuning data again, and repeat the above steps to enhance the quality of the remote sensing image-text data pairs.

[0057] This embodiment uses an iterative training method, and uses the high-quality image-text data pairs that have undergone data enhancement as the fine-tuning data of BLIP2 again. Based on the above steps, the BLIP2 model is continuously iterated to continuously enhance the quality of the multi-source remote sensing image-text dataset.

[0058] Step 6: The final remote sensing image and text data pair is a constructed multi-source remote sensing multi-scale image and text dataset.

[0059] This embodiment uses several enhanced remote sensing image and text data pairs as a constructed multi-source remote sensing multi-scale image and text dataset, and then constructs a multi-source multi-scale cross-modal remote sensing image semantic understanding model based on the dataset.

[0060] S2, uses a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to process several source satellite remote sensing images to obtain the source feature vectors of the source satellite remote sensing images.

[0061] In order to improve the efficiency and accuracy of remote sensing image text retrieval, this embodiment first stores multi-scale image vectors in a database to achieve storage management of massive remote sensing image feature vectors.

[0062] S2.1, preprocessing several source satellite remote sensing images to obtain multi-scale image blocks, including multi-scale sampling, image slicing, bit conversion, and noise elimination.

[0063] Firstly, the remote sensing large-scale images are preprocessed by multi-scale sampling, image slicing, bit conversion, noise elimination, etc. to obtain high-quality multi-scale image blocks.

[0064] S2.2, use the multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to encode the multi-scale image blocks to obtain the source feature vector of the source satellite remote sensing image.

[0065] In an optional embodiment, an image encoder of a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model is used to encode multi-scale image blocks into a 512-dimensional feature vector, which is an image feature vector, i.e., a source feature vector of a source satellite remote sensing image.

[0066] S3, stores the source feature vector into the vector library.

[0067] S3.1, obtain metadata information of each multi-scale image block.

[0068] S3.2, indexing the source feature vectors and corresponding metadata information of each multi-scale image block.

[0069] S3.3, storing the indexed source feature vector into a vector library.

[0070] In this embodiment, in order to improve the retrieval efficiency and accuracy, the source feature vector and the corresponding metadata information are stored in a high-performance vector library to establish an index, thereby forming a basic database for image retrieval. The metadata information includes the location of the image, the time of shooting, etc., which can be obtained when the corresponding image is downloaded from the target website.

[0071] S4, using a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to process the input image retrieval information or text retrieval information to obtain an image retrieval feature vector or a text retrieval feature vector.

[0072] The remote sensing image and text retrieval of this embodiment supports the image search function, that is, the image is input into the image encoder of the multi-source multi-scale cross-modal remote sensing image semantic understanding model to extract the image retrieval feature vector, and supports the image search function by text, using the text encoder of the multi-source multi-scale cross-modal remote sensing image semantic understanding model to process the text retrieval information and extract the text retrieval feature vector. Then, the image retrieval feature vector or the text retrieval feature vector is compared with the source feature vector in the vector library for similarity to obtain the target source satellite remote sensing image.

[0073] In this embodiment, the input text retrieval information includes retrieval conditions (such as time, place, etc.) and scene description. Natural language processing technology is introduced to process the input text retrieval information to obtain retrieval conditions and scene description. The scene description is input into the text encoder of the multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to obtain a text retrieval feature vector.

[0074] S5, performing similarity comparison between the image retrieval feature vector or the text retrieval feature vector and the source feature vector in the vector library, and screening out the target source satellite remote sensing image.

[0075] In this embodiment, the image retrieval feature vector is compared with the source feature vector in the vector library for similarity, and the most similar n vectors and metadata information are returned. Then, the target source satellite remote sensing image that meets the retrieval conditions can be screened out from the n vectors based on the metadata information.

[0076] In this embodiment, the text retrieval feature vector is compared with the source feature vector in the vector library for similarity, and the retrieval results are filtered using the retrieval conditions parsed by the NLP model to obtain n scenes that meet the retrieval requirements at a certain time and place, which specifically includes the following steps.

[0077] Step 1: compare the search conditions with the metadata information, select several source feature vectors that match the search conditions, and record them as initial source feature vectors.

[0078] Step 2: perform a similarity comparison between the text retrieval feature vector and the initial source feature vector to filter out the intermediate source feature vector.

[0079] Step 3: Use the Kmeans clustering method to cluster the intermediate source feature vectors in descending order of similarity.

[0080] Step 4: In the category with higher similarity, select the category with more than 1 / 2 as the positive category, that is, the final source feature vector retrieved.

[0081] Step 5: The source satellite remote sensing image corresponding to the final source feature vector is the target source satellite remote sensing image.

[0082] This embodiment filters scenes that meet the time condition based on the time and place parsed by the NLP model. In an optional implementation, spatial filtering is performed by introducing the administrative area vector range and performing spatial query in the vector database to filter scenes that meet the spatial condition, so as to achieve accurate remote sensing semantic scene query in time and space.

[0083] Affected by the model accuracy, text and image complexity, the n retrieval results obtained may contain wrong scenes. To avoid this problem as much as possible, this embodiment implements an error scene filtering algorithm. The algorithm is based on the following assumptions: for a series of ranges returned, it is assumed that one category is correct, characterized by high similarity. The other category is wrong, with low similarity, and there is a certain gap in similarity with the correct range. Based on the above assumptions, first, retrieve the n retrieval results with the highest similarity to the input text, regard them as the initial retrieval results, sort them in descending order of similarity, and then use the kmeans clustering method to cluster the descending similarity. In the category with higher similarity, take the category with more than 1 / 2 as the positive category, that is, retrieve the correct category, and the other category is eliminated as the negative category. In this way, the wrong retrieval scene filtering is completed. In addition, in order to make the retrieval range more reasonable, if the ranges are adjacent or intersecting, take their intersection as the fused range.

[0084] In the above, an embodiment of a satellite remote sensing image graphic and text retrieval method based on multi-scale cross-modality is described in detail. Based on the satellite remote sensing image graphic and text retrieval method based on multi-scale cross-modality described in the above embodiment, an embodiment of the present invention also provides a satellite remote sensing image graphic and text retrieval system based on multi-scale cross-modality corresponding to the method.

[0085] Figure 5 is a schematic block diagram of the structure of a satellite remote sensing image text retrieval system based on multi-scale cross-modality provided by an embodiment of the present invention. In this embodiment, the satellite remote sensing image text retrieval system based on multi-scale cross-modality 500 can be divided into multiple functional modules according to the functions it performs, such as Figure 5 The functional modules may include: a semantic understanding model building module 510, a source feature vector acquisition module 520, a vector storage module 530, a retrieval feature vector acquisition module 540, and a target image retrieval module 550. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory.

[0086] The semantic understanding model construction module 510 is used to construct a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model.

[0087] The source feature vector acquisition module 520 is used to process a plurality of source satellite remote sensing images using a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to obtain source feature vectors of the source satellite remote sensing images.

[0088] The vector storage module 530 is used to store the source feature vector into a vector library.

[0089] The retrieval feature vector acquisition module 540 is used to process the input image retrieval information or text retrieval information using the multi-source multi-scale cross-modal remote sensing image semantic understanding model to obtain an image retrieval feature vector or a text retrieval feature vector.

[0090] The target image retrieval module 550 is used to perform similarity comparison between the image retrieval feature vector or the text retrieval feature vector and the source feature vector in the vector library, and screen out the target source satellite remote sensing image.

[0091] The satellite remote sensing image graphic and text retrieval system based on multi-scale cross-modality of this embodiment is used to implement the aforementioned satellite remote sensing image graphic and text retrieval method based on multi-scale cross-modality. Therefore, the specific implementation method of the system can be seen in the implementation example part of the satellite remote sensing image graphic and text retrieval method based on multi-scale cross-modality in the previous text. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part, which will not be introduced in detail here.

[0092] In addition, since the satellite remote sensing image graphic retrieval system based on multi-scale cross-modality of this embodiment is used to implement the aforementioned satellite remote sensing image graphic retrieval method based on multi-scale cross-modality, its function corresponds to that of the above-mentioned method and will not be repeated here.

[0093] Figure 6 A schematic diagram of the structure of a terminal 600 provided in an embodiment of the present invention includes: a processor 610, a memory 620 and a communication unit 630. The processor 610 is used to implement the following steps when implementing a multi-scale cross-modal satellite remote sensing image text retrieval program stored in the memory 620: Construct a semantic understanding model for multi-source, multi-scale, and cross-modal remote sensing images; Use a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to process several source satellite remote sensing images and obtain source feature vectors of the source satellite remote sensing images; Store the source feature vector into the vector library; Using a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to process input image retrieval information or text retrieval information to obtain an image retrieval feature vector or a text retrieval feature vector; The image retrieval feature vector or text retrieval feature vector is compared with the source feature vector in the vector library for similarity, and the target source satellite remote sensing image is screened out.

[0094] The present invention also provides a computer storage medium, wherein the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0095] The computer storage medium stores a satellite remote sensing image text retrieval program based on multi-scale cross-modality, and the satellite remote sensing image text retrieval program based on multi-scale cross-modality implements the following steps when executed by the processor: Construct a semantic understanding model for multi-source, multi-scale, and cross-modal remote sensing images; Use a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to process several source satellite remote sensing images and obtain source feature vectors of the source satellite remote sensing images; Store the source feature vector into the vector library; Using a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to process input image retrieval information or text retrieval information to obtain an image retrieval feature vector or a text retrieval feature vector; The image retrieval feature vector or text retrieval feature vector is compared with the source feature vector in the vector library for similarity, and the target source satellite remote sensing image is screened out.

[0096] The above disclosure is only a preferred embodiment of the present invention, but the present invention is not limited thereto. Any non-creative changes that can be thought of by a person skilled in the art, as well as several improvements and modifications made without departing from the principle of the present invention, should fall within the protection scope of the present invention.

Claims

1. A satellite remote sensing image text retrieval method based on multi-scale cross-modality, characterized in that: The following steps are involved: Construct a semantic understanding model for multi-source, multi-scale, and cross-modal remote sensing images; Use a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to process several source satellite remote sensing images and obtain source feature vectors of the source satellite remote sensing images; Store the source feature vector into the vector library; Using a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to process input image retrieval information or text retrieval information to obtain an image retrieval feature vector or a text retrieval feature vector; The image retrieval feature vector or text retrieval feature vector is compared with the source feature vector in the vector library for similarity, and the target source satellite remote sensing image is screened out.

2. The method for image and text retrieval based on multi-scale cross-modal satellite remote sensing images according to claim 1, characterized in that: Construct a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model, including: Build a multimodal pre-training model, which includes an image encoder and a text encoder. The image encoder uses a visual Transformer structure, and the text encoder uses a Transformer structure. Use the machine vision image-text dataset to pre-train the multimodal pre-training model, including encoding images and texts into image feature vectors and text feature vectors respectively, using the InfoNCE loss function to maximize the cosine similarity of N paired image-text pairs in the machine vision image-text dataset, and minimize the cosine similarity of the remaining unpaired image-text pairs, to achieve multimodal feature space alignment of image-text pairs; The pre-built multi-source remote sensing multi-scale image and text dataset is used to fine-tune all parameters of the pre-trained multimodal pre-trained model to obtain the trained multimodal pre-trained model, namely, the multi-source multi-scale cross-modal remote sensing image semantic understanding model.

3. The method for image and text retrieval based on multi-scale cross-modal satellite remote sensing images according to claim 2 is characterized in that: The method also includes the step of constructing a multi-source remote sensing multi-scale image and text dataset, specifically including: Label remote sensing image and text data pairs; Use annotated remote sensing image and text data to fine-tune all parameters of the multimodal large language model; The remote sensing image to be annotated is used as the inference data of the multimodal large language model after fine-tuning all parameters to obtain the annotated text of the remote sensing image; Use Kmeans clustering method to retain M annotation texts with different feature distribution for each remote sensing image; Use iterative training to use the data-enhanced remote sensing image-text data pairs as the multimodal large language model fine-tuning data again, and repeat the above steps to enhance the quality of remote sensing image-text data pairs; The final remote sensing image and text data pair is a constructed multi-source remote sensing multi-scale image and text dataset.

4. The method for image and text retrieval based on multi-scale cross-modal satellite remote sensing images according to claim 2 is characterized in that: The input text retrieval information is processed using a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to obtain a text retrieval feature vector, including: Use natural language processing technology to process the input text retrieval information to obtain retrieval conditions and scene descriptions; The scene description is input into the text encoder of the multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to obtain the text retrieval feature vector.

5. The method for satellite remote sensing image text retrieval based on multi-scale cross-modality according to claim 4 is characterized in that: The multi-source, multi-scale, cross-modal remote sensing image semantic understanding model is used to process several source satellite remote sensing images to obtain the source feature vectors of the source satellite remote sensing images, including: Preprocess several source satellite remote sensing images to obtain multi-scale image blocks, including multi-scale sampling, image slicing, bit conversion, and noise elimination; A multi-source, multi-scale, cross-modal remote sensing image semantic understanding model is used to encode multi-scale image blocks to obtain the source feature vectors of the source satellite remote sensing images.

6. The method for satellite remote sensing image text retrieval based on multi-scale cross-modality according to claim 5, characterized in that: Storing the source feature vector into the vector library includes: Obtain metadata information of each multi-scale image block; Indexing the source feature vectors and corresponding metadata information of each multi-scale image block; Store the indexed source feature vector into a vector library.

7. The method for image and text retrieval based on multi-scale cross-modal satellite remote sensing images according to claim 6, characterized in that: The text retrieval feature vector is compared with the source feature vector in the vector library for similarity, and the target source satellite remote sensing image is screened out, including: Compare the search conditions with the metadata information, select several source feature vectors that match the search conditions, and record them as initial source feature vectors; Compare the text retrieval feature vector with the initial source feature vector for similarity, and filter out the intermediate source feature vector; Use Kmeans clustering method to cluster the intermediate source feature vectors in descending order of similarity; In the category with higher similarity, the category with more than 1 / 2 number is selected as the positive category, which is the final source feature vector retrieved; The source satellite remote sensing image corresponding to the final source feature vector is the target source satellite remote sensing image.

8. A satellite remote sensing imagery and text retrieval system based on multi-scale cross-modality, characterized in that: include: Semantic understanding model building module, used to build a semantic understanding model for multi-source, multi-scale, cross-modal remote sensing images; A source feature vector acquisition module is used to process several source satellite remote sensing images using a multi-source, multi-scale, cross-modal remote sensing image semantic understanding model to obtain source feature vectors of the source satellite remote sensing images; A vector storage module, used for storing the source feature vector into a vector library; A retrieval feature vector acquisition module is used to process the input image retrieval information or text retrieval information using a multi-source multi-scale cross-modal remote sensing image semantic understanding model to obtain an image retrieval feature vector or a text retrieval feature vector; The target image retrieval module is used to compare the similarity between the image retrieval feature vector or the text retrieval feature vector and the source feature vector in the vector library, and screen out the target source satellite remote sensing image.

9. A terminal, characterized in that: include: A memory device for storing a graphic and text retrieval program based on multi-scale and cross-modal satellite remote sensing images; A processor, used to implement the steps of the satellite remote sensing image image and text retrieval method based on multi-scale cross-modality as described in any one of claims 1 to 7 when executing the satellite remote sensing image image and text retrieval program based on multi-scale cross-modality.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores a satellite remote sensing image graphic and text retrieval program based on multi-scale cross-modality. When the satellite remote sensing image graphic and text retrieval program based on multi-scale cross-modality is executed by the processor, the steps of the satellite remote sensing image graphic and text retrieval method based on multi-scale cross-modality as described in any one of claims 1 to 7 are implemented.

Images