A remote sensing image retrieval method, device and medium based on natural language description
By constructing a text library and using BERT and LSTM networks to generate a remote sensing image retrieval model with natural language descriptions, the problem of low accuracy in remote sensing image retrieval was solved, achieving more efficient retrieval of massive remote sensing images and accurate semantic feature capture, thus improving the user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNIV OF GEOSCIENCES (WUHAN)
- Filing Date
- 2024-08-07
- Publication Date
- 2026-05-29
AI Technical Summary
Existing remote sensing image retrieval methods suffer from low retrieval accuracy and are not applicable in scenarios involving massive amounts of remote sensing images. In particular, semantic feature-based retrieval struggles to accurately capture all semantic features, requiring users to further filter the search results.
By constructing a text library and using a BERT pre-trained language model for sentence vector encoding, combined with an LSTM network and a ResNet50 model, a remote sensing image retrieval model with natural language description is generated, and image retrieval is performed using a hybrid representation of sentence vectors and visual feature vectors.
It improves the accuracy and user experience of remote sensing image retrieval, supports multimodal retrieval of images and text, and generates more accurate natural language description sentences that can better capture the semantic relevance between words, thereby improving retrieval accuracy.
Smart Images
Figure CN118885631B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of remote sensing image retrieval, and in particular to a remote sensing image retrieval method, device and medium based on natural language description. Background Technology
[0002] A remote sensing image retrieval system is a system that uses computer technology and algorithms to automatically retrieve images matching user query criteria from a large amount of remote sensing image data. Its purpose is to quickly and accurately find the required image information by effectively organizing, managing, and querying remote sensing image data. Remote sensing image retrieval systems are widely used in resource monitoring, environmental protection, disaster emergency response, urban planning, and agricultural management. In a remote sensing image retrieval system, the retrieval method plays a central role, determining the system's efficiency, accuracy, and user experience.
[0003] The main methods for retrieving remote sensing images include: retrieval based on spatial coordinates, retrieval based on visual features, and retrieval based on semantic features.
[0004] Both spatial coordinate-based and visual feature-based remote sensing image retrieval methods perform retrieval by analyzing the image's texture, color, shape, local features, and geographic location.
[0005] Semantic feature-based retrieval uses deep learning methods to extract high-level features from images. These features can represent the semantic information of the image, such as different types of land features (buildings, roads, forests, farmland, etc.). The retrieval system performs remote sensing image retrieval based on the land feature category input by the user. For example, Chinese patent CN113468371A relates to a method for implementing natural language image retrieval. However, semantic feature-based retrieval has limitations when handling diverse semantic information. The system may struggle to accurately capture all semantic features, and may return multiple regions of interest during retrieval, requiring further filtering by the user. This method has limited accuracy.
[0006] Chinese patent CN118035416A relates to a streaming question-and-answer image matching method and system. It uses a constructed text vector library and image index relation database to perform streaming question-and-answer retrieval to obtain the corresponding text and images. However, this retrieval is not specifically for remote sensing images and suffers from low image matching accuracy. Summary of the Invention
[0007] The purpose of this invention is to address the problems of low retrieval accuracy and unsuitability for applications involving massive amounts of remote sensing images in existing image retrieval methods, and to provide a remote sensing image retrieval method, device, and medium based on natural language description.
[0008] The above-mentioned objective of this application is achieved through the following technical solution:
[0009] S1: Obtain textual description information of remote sensing images and construct a text library; clean and preprocess each sentence in the text library to obtain the sentence ID of each sentence;
[0010] S2: By using a BERT-based pre-trained language model, each sentence in the text database is converted into a vector representation through vector encoding.
[0011] S3: Store the sentence IDs and sentence vector tables in a one-to-one correspondence to construct the dataset;
[0012] S4: Construct a remote sensing image retrieval model based on an LSTM network; train the remote sensing image retrieval model using a dataset;
[0013] S5: Obtain the remote sensing image to be retrieved, and generate a natural language description sentence for the remote sensing image to be retrieved using the trained remote sensing image retrieval model;
[0014] S6: Use the natural language description sentence as the sentence to be retrieved; use a BERT-based pre-trained language model to retrieve remote sensing images similar to the remote sensing image to be retrieved from the dataset.
[0015] Optionally, step S2 includes:
[0016] The sentence is input into a BERT-based pre-trained language model to obtain the word embedding vector for each sentence;
[0017] By averaging the word embedding vectors, a fixed-length sentence vector representation is obtained. ,in Representative word embedding, The first character in the representative sentence Normalized word frequency of each word.
[0018] Optionally, step S4 includes:
[0019] S41: Divide the dataset into training set, validation set and test set according to a preset ratio;
[0020] S42: Extract visual feature vectors from the input images in the dataset using a pre-trained ResNet50 model;
[0021] S43: Construct a vocabulary V that maps all words in the text descriptions of the dataset to fixed-length word embedding vectors;
[0022] S44: Input the word embedding vectors into the remote sensing image retrieval model for training to obtain the results at each time step. Hidden state vector ;
[0023] S45: Add the hidden state vector to the image visual feature vector to obtain the hybrid vector feature; extract and compress the hybrid vector feature to obtain the final representation; generate a natural language description sentence based on the final representation.
[0024] Optionally, step S6 includes:
[0025] S61: Preprocess the sentences to be retrieved using a pre-trained language model;
[0026] S62: Segment the preprocessed sentence to be retrieved into words to obtain the token ID sequence of the sentence to be retrieved;
[0027] Input the token ID sequence into the pre-trained language model to obtain the hidden state vector of each word in the sentence to be retrieved. Average the hidden state vectors of all words in the sentence to be retrieved to obtain the vector representation of the sentence to be retrieved. ,in Representative word embedding, Normalized word frequency of representative words;
[0028] S63: Represent the sentence to be retrieved as a vector Match the sentence vectors stored in the dataset, calculate the dissimilarity between two sentence vectors, and determine the similar sentence vectors in the dataset;
[0029] S64: Based on the sentence ID of similar sentence vectors, find the remote sensing image corresponding to the sentence to be retrieved from the dataset.
[0030] Optionally, step S61 includes:
[0031] The system determines whether each word in the sentence to be retrieved is in the vocabulary of the pre-trained language model. If it is not in the vocabulary, the word is replaced with a preset special tag to obtain the pre-processed sentence to be retrieved.
[0032] Optionally, step S63 includes:
[0033] S63a: Each sentence to be retrieved is a set of a preset number of words. The similarity between two sentences is calculated by minimizing the total cost required to move the sentence to be retrieved to the matching sentences in the dataset.
[0034] make As a word embedding matrix For vocabulary size; For words of 3D encoded vector;
[0035] set up and The sentence to be retrieved and the sentence to be matched are represented using normalized bag-of-words. It is a word exist The number of times it appears in the text divided by The total number of words in the Chinese dictionary, of which Indicates the first The word is in The number of times it appears in; express The number of times the k-th word appears in the text; It is the sum of the number of times all words appear in sentence S;
[0036] set up Let be the Euclidean distance between any two words in the semantic vector space, and let represent the semantic dissimilarity between the two words. For words of 3D encoded vector;
[0037] Introducing an auxiliary matrix ,in , express Chinese words Need to be transferred Chinese words The number of words;
[0038] The distance between the sentence to be retrieved and the sentence to be matched is defined as: moving all words in the sentence to be retrieved (S) to the sentence to be matched. Minimum total cost:
[0039]
[0040]
[0041]
[0042] in ； express The Middle The word was completely transmitted to The total traffic of the Chinese word. express All incoming traffic received by the word in the middle;
[0043] S63b: Obtain similar sentence vectors from the dataset by calculating similarity.
[0044] An electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform a remote sensing image retrieval method based on natural language description.
[0045] A computer-readable storage medium storing instructions that, when executed, perform a remote sensing image retrieval method based on natural language description.
[0046] The beneficial effects of the technical solution provided in this application are:
[0047] 1. This paper defines natural language descriptions for remote sensing images based on a remote sensing image retrieval model. Each image's natural language description is encoded, and its semantic content is mapped to and stored in a vector space. When a user inputs a remote sensing image for image retrieval, the image description is encoded in the same way and then matched with the stored vectors to complete the image retrieval. This addresses the issues of high technical barriers for users and low retrieval accuracy in massive remote sensing image retrieval. It supports both image and text retrieval. When there is a significant difference between the user's input text and the data in the database, a pre-trained language model is used to generate a semantically richer vector representation, which is then converted into a vector for retrieval, improving retrieval accuracy.
[0048] 2. The word embedding vector is input into the remote sensing image retrieval model for training to obtain the hidden state vector at each time step; the hidden state vector is added to the image visual feature vector to obtain the hybrid vector feature; the hybrid vector feature is extracted and compressed to obtain the final representation; and a natural language description sentence is generated using the final representation. Employing multimodal generation of natural language description sentences makes the generation of natural language description sentences more accurate.
[0049] 3. By using dissimilarity calculation, we can better capture the semantic relevance between words, thereby retrieving more similar remote sensing images. Attached Figure Description
[0050] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:
[0051] Figure 1 This is a step diagram of an embodiment of this application;
[0052] Figure 2 This is a comparison chart of Blue scores in the embodiments of this application;
[0053] Figure 3 This is a comparison chart of the f-score evaluation results in the embodiments of this application;
[0054] Figure 4 This is the first search result diagram in the embodiments of this application;
[0055] Figure 5 This is a second search result diagram in an embodiment of this application;
[0056] Figure 6 This is the third search result diagram in the embodiments of this application;
[0057] Figure 7 This is a schematic diagram of the electronic device structure in the embodiments of this application. Detailed Implementation
[0058] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0059] The embodiments of this application provide a remote sensing image retrieval method based on natural language description.
[0060] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a remote sensing image retrieval method based on natural language description in an embodiment of this application, including:
[0061] S1: Obtain textual description information of remote sensing images and construct a text library; clean and preprocess each sentence in the text library to obtain the sentence ID of each sentence;
[0062] Specifically, the text description information of the remote sensing images uses the RSICD and UVA datasets, which contain images and text descriptions. Each sentence in the text library is cleaned and preprocessed to remove punctuation marks and special characters, resulting in a unique sentence ID for each sentence.
[0063] S2: By using a BERT-based pre-trained language model, each sentence in the text database is converted into a vector representation through vector encoding.
[0064] Step S2 includes:
[0065] The sentence is input into a BERT-based pre-trained language model to obtain the word embedding vector for each sentence;
[0066] By averaging the word embedding vectors, a fixed-length sentence vector representation is obtained. ,in Representative word embedding, The first character in the representative sentence Normalized word frequency of each word.
[0067] S3: Store the sentence IDs and sentence vector tables in a one-to-one correspondence to construct the dataset;
[0068] Specifically, sentence IDs and sentence vector tables are stored one-to-one to establish an ID-vector mapping relationship. Finally, the vectors and their index relationships are saved using NumPy or other data storage methods to construct the dataset.
[0069] S4: Construct a remote sensing image retrieval model based on an LSTM network; train the remote sensing image retrieval model using a dataset;
[0070] Step S4 includes:
[0071] S41: Divide the dataset into training set, validation set and test set according to a preset ratio;
[0072] S42: Extract visual feature vectors from the input images in the dataset using a pre-trained ResNet50 model;
[0073] Specifically, the output of the penultimate layer of the ResNet50 model is used as the visual feature of the image and passed to a D-dimensional fully connected layer with ReLU activation to further extract features and obtain a D-dimensional visual feature vector.
[0074] S43: Construct a vocabulary V that maps all words in the text descriptions of the dataset to fixed-length word embedding vectors;
[0075] S44: Input the word embedding vectors into the remote sensing image retrieval model for training to obtain the results at each time step. Hidden state vector ;
[0076] Specifically, at each time step The LSTM unit receives the word embedding vector of the input word at the current time step. And the hidden state of the previous moment. and memory state The hidden state and the memory state are updated through the gating mechanism in the LSTM unit. The hidden state vector at each time step is obtained through the forward propagation of the LSTM. The hidden state vector contains temporal relationships and semantic context information between words.
[0077] S45: Add the hidden state vector to the image visual feature vector to obtain the hybrid vector feature; extract and compress the hybrid vector feature to obtain the final representation; generate a natural language description sentence based on the final representation.
[0078] Specifically, the final representation is passed to a vector with a lexical size dimension V and The activated fully connected layer predicts the next word given image features and the previous word. Each generated word is added to the previously generated sentence until an end marker is generated, thus generating a natural language description sentence.
[0079] Specifically, the various sub-modules of the remote sensing image retrieval model can be optimized end-to-end through backpropagation. A loss function is defined, and at each time step, the cross-entropy loss between the word probability distribution predicted by the remote sensing image retrieval model and the ground truth value is calculated. The losses from all time steps are summed to obtain the total loss for the entire sequence. An optimization algorithm is selected, a reasonable learning rate is set, and an early stopping strategy is adopted to avoid overfitting.
[0080] S5: Obtain the remote sensing image to be retrieved, and generate a natural language description sentence for the remote sensing image to be retrieved using the trained remote sensing image retrieval model;
[0081] Specifically, the remote sensing image retrieval model extracts the visual features of the remote sensing image to be retrieved. These visual features, along with an initial start marker, are input into the model, which iteratively generates the next word until an end marker is generated. Generate natural language description sentences as , This represents the word embedding vector representation of the first word.
[0082] S6: Use the natural language description sentence as the sentence to be retrieved; use a BERT-based pre-trained language model to retrieve remote sensing images similar to the remote sensing image to be retrieved from the dataset.
[0083] Step S6 includes:
[0084] S61: Preprocess the sentences to be retrieved using a pre-trained language model;
[0085] Step S61 includes:
[0086] The system determines whether each word in the sentence to be retrieved is in the vocabulary of the pre-trained language model. If it is not in the vocabulary, the word is replaced with a preset special tag to obtain the pre-processed sentence to be retrieved.
[0087] S62: Segment the preprocessed sentence to be retrieved into words to obtain the token ID sequence of the sentence to be retrieved;
[0088] Input the token ID sequence into the pre-trained language model to obtain the hidden state vector of each word in the sentence to be retrieved. Average the hidden state vectors of all words in the sentence to be retrieved to obtain the vector representation of the sentence to be retrieved. ,in Representative word embedding, Normalized word frequency of representative words;
[0089] S63: Represent the sentence to be retrieved as a vector Match the sentence vectors stored in the dataset, calculate the dissimilarity between two sentence vectors, and determine the similar sentence vectors in the dataset;
[0090] Step S63 includes:
[0091] S63a: Each sentence to be retrieved is a set of a preset number of words. The similarity between two sentences is calculated by minimizing the total cost required to move the sentence to be retrieved to the matching sentences in the dataset.
[0092] make As a word embedding matrix For vocabulary size; For words of 3D encoded vector;
[0093] set up and The sentence to be retrieved and the sentence to be matched are represented using normalized bag-of-words. It is a word exist The number of times it appears in the text divided by The total number of words in the Chinese dictionary, of which Indicates the first The word is in The number of times it appears in; express The number of times the k-th word appears in the text; It is the sum of the number of times all words appear in sentence S;
[0094] set up Let be the Euclidean distance between any two words in the semantic vector space, and let represent the semantic dissimilarity between the two words. For words of 3D encoded vector;
[0095] Introducing an auxiliary matrix ,in , express Chinese words Need to be transferred Chinese words The number of words;
[0096] The distance between the sentence to be retrieved and the sentence to be matched is defined as: moving all words in the sentence to be retrieved (S) to the sentence to be matched. Minimum total cost:
[0097]
[0098]
[0099]
[0100] in ； express The Middle The word was completely transmitted to The total traffic of the Chinese word. express All incoming traffic received by the word in the middle;
[0101] S63b: Obtain similar sentence vectors from the dataset by calculating similarity.
[0102] S64: Based on the sentence ID of similar sentence vectors, find the remote sensing image corresponding to the sentence to be retrieved from the dataset.
[0103] The accuracy of the technical solution is demonstrated through the following experiments:
[0104] Design evaluation metrics:
[0105] Evaluation Metric 1: Blue score (n=1,2,3,4)
[0106]
[0107] in The geometric mean of n-gram precision. ,in yes The number of n-grams between them. It is the total number of n-grams in G.
[0108]
[0109] in For reference, the length of the translation is... The length of the candidate translation.
[0110]
[0111] The BLEU score between the reference translation R and the candidate translation G is calculated as the accuracy. With concise punishment The product of.
[0112] Evaluation metric 2: f-score (β=1,2)
[0113]
[0114]
[0115]
[0116] in The number of images to be retrieved. Represents the set of image tags to be queried. Representing the A set of tags for each retrieved image.
[0117] The Unmanned Aerial Vehicle (UAV) image description dataset was used: it consists of 10 RGB images with a pixel size of 5184×3456 and a spatial resolution of 2cm. Six images were used for training, one for validation, and three for testing. The dataset was divided into 2940 image frames of size 256×256, each with text descriptions provided by three different annotators. The dataset vocabulary size V is 185. Figure 2 and Figure 3 The figures show the average blue score and f-score results of the remote sensing image retrieval method and the multi-label retrieval system proposed in this application on the UAV dataset. Figure 4 and Figure 5 The image shows a comparison of the retrieval performance of the remote sensing image retrieval method proposed in this application with that of a multi-label retrieval system on the UAV dataset. It can be seen that the retrieval results of this application can retrieve more similar images.
[0118] The Remote Sensing Image Description Dataset (RSICD) consists of over 10,000 RS images collected from various maps at different resolutions. Each image has 1 to 5 descriptions. The images are fixed at 224×224 pixels. The vocabulary size V of this dataset is 3323.
[0119] Figure 6 The image shows the retrieval results of the remote sensing image retrieval method proposed in this application on the Remote Sensing Image Description Dataset (RSICD).
[0120] This application also discloses an electronic device. (See reference...) Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.
[0121] The communication bus 502 is used to enable communication between these components.
[0122] The user interface 503 may include a display screen, and optionally, the user interface 503 may also include a standard wired interface or a wireless interface.
[0123] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).
[0124] This application also discloses a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the aforementioned remote sensing image retrieval method based on natural language description.
[0125] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will readily conceive of those skilled in the art upon consideration of the specification and the disclosure of practical truths.
[0126] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.
Claims
1. A remote sensing image retrieval method based on natural language description, characterized in that, The method includes the following steps: S1: Obtain textual description information of remote sensing images and build a text library; Each sentence in the text library is cleaned and preprocessed to obtain the sentence ID of each sentence; S2: By using a BERT-based pre-trained language model, each sentence in the text database is converted into a vector representation through vector encoding. S3: Store the sentence IDs and sentence vector tables in a one-to-one correspondence to construct the dataset; S4: Construct a remote sensing image retrieval model based on an LSTM network; train the remote sensing image retrieval model using a dataset; S5: Obtain the remote sensing image to be retrieved, and generate a natural language description sentence for the remote sensing image to be retrieved using the trained remote sensing image retrieval model; S6: Use the natural language description sentence as the sentence to be retrieved; use a BERT-based pre-trained language model to retrieve remote sensing images similar to the remote sensing image to be retrieved from the dataset; Step S6 includes: S61: Preprocess the sentences to be retrieved using a pre-trained language model; S62: Segment the preprocessed sentence to be retrieved into words to obtain the token ID sequence of the sentence to be retrieved; Input the token ID sequence into the pre-trained language model to obtain the hidden state vector of each word in the sentence to be retrieved. Average the hidden state vectors of all words in the sentence to be retrieved to obtain the vector representation of the sentence to be retrieved. ,in Representative word embedding, Normalized word frequency of representative words; S63: Represent the sentence to be retrieved as a vector Match the sentence vectors stored in the dataset, calculate the dissimilarity between two sentence vectors, and determine the similar sentence vectors in the dataset; Step S63 includes: S63a: Each sentence to be retrieved is a set of a preset number of words. The similarity between two sentences is calculated by minimizing the total cost required to move the sentence to be retrieved to the matching sentences in the dataset. make As a word embedding matrix For vocabulary size; For words of 3D encoded vector; set up and The sentence to be retrieved and the sentence to be matched are represented using normalized bag-of-words. It is a word exist The number of times it appears in the text divided by The total number of words in the Chinese dictionary, of which Indicates the first The word is in The number of times it appears in; express The number of times the k-th word appears in the text; It is the sum of the number of times all words appear in sentence S; set up Let be the Euclidean distance between any two words in the semantic vector space, and let represent the semantic dissimilarity between the two words. For words of 3D encoded vector; Introducing an auxiliary matrix ,in , express Chinese words Need to be transferred Chinese words The number of words; The distance between the sentence to be retrieved and the sentence to be matched is defined as: moving all words in the sentence to be retrieved (S) to the sentence to be matched. Minimum total cost: in ； express The Middle The word was completely transmitted to The total traffic of the Chinese word. express All incoming traffic received by the word in the middle; S63b: Obtain similar sentence vectors from the dataset by calculating similarity; S64: Based on the sentence ID of similar sentence vectors, find the remote sensing image corresponding to the sentence to be retrieved from the dataset.
2. The remote sensing image retrieval method based on natural language description as described in claim 1, characterized in that, Step S2 includes: The sentence is input into a BERT-based pre-trained language model to obtain the word embedding vector for each sentence; By averaging the word embedding vectors, a fixed-length sentence vector representation is obtained. ,in Representative word embedding, The first character in the representative sentence Normalized word frequency of each word.
3. The remote sensing image retrieval method based on natural language description as described in claim 1, characterized in that, Step S4 includes: S41: Divide the dataset into training set, validation set and test set according to a preset ratio; S42: Extract visual feature vectors from the input images in the dataset using a pre-trained ResNet50 model; S43: Construct a vocabulary V that maps all words in the text descriptions of the dataset to fixed-length word embedding vectors; S44: Input the word embedding vectors into the remote sensing image retrieval model for training to obtain the results at each time step. Hidden state vector ; S45: Add the hidden state vector to the image visual feature vector to obtain the hybrid vector feature; extract and compress the hybrid vector feature to obtain the final representation; generate a natural language description sentence based on the final representation.
4. The remote sensing image retrieval method based on natural language description as described in claim 1, characterized in that, Step S61 includes: The system determines whether each word in the sentence to be retrieved is in the vocabulary of the pre-trained language model. If it is not in the vocabulary, the word is replaced with a preset special tag to obtain the pre-processed sentence to be retrieved.
5. An electronic device, characterized in that, The device includes a processor (501), a memory (505), a user interface (503), and a network interface (504). The memory (505) is used to store instructions. The user interface (503) and the network interface (504) are used to communicate with other devices. The processor (501) is used to execute the instructions stored in the memory (505) to cause the electronic device to perform the method as described in any one of claims 1-4.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the method as described in any one of claims 1-4.