A method for quick retrieval of remote sensing images based on hand-drawn sketches

Abstract

This invention discloses a fast remote sensing image retrieval method based on hand-drawn sketches. The method comprises five parts: a tokenization module for feature enhancement, a self-attention module with retrieval tokens, a cross-modal attention module, the definition and usage of a triplet loss function with retrieval tokens, and token selection optimization and retrieval. The model was tested on different cross-modal remote sensing image retrieval datasets and large-scale remote sensing images, verifying the fast retrieval performance of the proposed model for hand-drawn sketches in remote sensing images, and validating the model's zero-learning capability. This invention provides a method for fast retrieval of remote sensing images using hand-drawn sketches. Experimental results show that the proposed method can significantly improve the efficiency and accuracy of remote sensing image retrieval, and possesses good zero-learning capability, enabling effective retrieval of categories not encountered during training.

Description

Technical Field

[0001] This invention relates to the field of cross-modal image retrieval of sketches and remote sensing images, specifically to a fast remote sensing image retrieval method based on hand-drawn sketches, used to find images in remote sensing imagery that match a given sketch type. Background Technology

[0002] In the field of remote sensing imagery, fast and accurate retrieval of images matching a given sketch is crucial for many applications. Traditional image retrieval methods typically rely on low-level visual features (such as color and texture), but these features are difficult to directly apply to hand-drawn sketches because the representation of sketches differs significantly from that of real images. In recent years, visual transformer models have achieved great success in computer vision. Among them, attention mechanisms have been widely used to process image and sequence data, and self-attention mechanisms can model long-range dependencies. Therefore, we propose a fast remote sensing image retrieval method based on hand-drawn sketches to improve retrieval accuracy and efficiency. Summary of the Invention

[0003] To address the shortcomings mentioned in the background section, this invention provides a method for rapid remote sensing image retrieval based on hand-drawn sketches, aiming to solve the technical problems of low accuracy and low efficiency in existing remote sensing image retrieval methods based on hand-drawn sketches.

[0004] The method includes the following steps: Step 1: Tokenization module for feature enhancement; Step 2: Self-attention module with retrieval token; Step 3: Cross-modal attention module; Step 4: Definition and use of the triplet loss function with retrieval token; Step 5: Token selection, optimization, and retrieval.

[0005] Furthermore, in step one, the process involves converting hand-drawn sketches and remotely sensed images into visual tokens. Then, a tokenization module with enhanced features processes the sparsely drawn sketches so that the model can learn more structural information from the sketches. Specifically, a tokenizer is generated by stacking four convolutional layers with different kernel sizes. Each convolutional layer is followed by a non-linear activation function (ReLU). The purpose of this is to expand the receptive field of the visual tokens through layered convolution, thereby preserving more feature information from the sketches. Residual connections are introduced to modify the original tokens, resulting in the following token embedding:

[0006] in, This represents the visual token embedding after feature enhancement. This represents the original visual token embedding of the input. This represents the sketch structure enhancement features output by a four-layer convolutional tokenizer; Furthermore, in step two, a retrieval token is added to the self-attention module to obtain more global image features. Specifically, the retrieval token [RT] is a trainable d-dimensional token embedding [...]. During the inference phase of the model, all visual tokens (including the retrieval token [RT]) interact through a multi-head self-attention module (MSA), and then pass through a multilayer perceptron module (MLP). The overall forward propagation expression of the model is:

[0007]

[0008]

[0009] In formula (2), RT represents the retrieval token, V¹,…,V n For the embedded token after processing, both formulas (3) and (4) introduce residual connections. The layer index is 1, 2, ..., L, where L is the layer number. Both sketches and remotely sensed images will pass through this architecture. The MSA module has three different projection heads. They respectively embed the same token into queries, keys, and values, which are represented by the formula:

[0010] The scaling dot product attention is given by the following formula:

[0011] Furthermore, in step three, a cross-modal attention module is used to establish local visual correspondences between the sketch and the remote sensing image in order to capture their correlation; specifically, this is achieved by exchanging sketch queries. Remote sensing image query New query tensors, key tensors, and value tensors are generated. After the swap, the query tensors, key tensors, and value tensors of the sketch and remote sensing images become... and Cross-modal attention is obtained through the following formula:

[0012] This formula is passed and Multiply, then divide by the square root of the feature dimension. The inner product attention score is calculated, representing the relevance between the query and each key. Then, the softmax function is applied to normalize the attention scores so that their sum equals 1, yielding the attention weights. Finally, the normalized attention weights are compared with the value tensor. Perform dot product calculations to obtain cross-modal attention output.

[0013] Furthermore, in step four, a triad loss function with a retrieval token [RT] is defined and used to train the network proposed by this method. Specifically, it is defined as follows: given a triad... Where Si is a target sketch. These are remote sensing images with the same labels as Si. The target sketch is a remote sensing image with a different label from Si. The goal of this loss function is to minimize the triplets loss to align correctly matched sketch-remote sensing image pairs and to keep the target sketch away from incorrect sketch-remote sensing image pairs. Based on this, we retrieve the token [RT] as a global feature of the sketch and the remote sensing image. The triplets loss is defined as follows:

[0014] in, This represents the global feature representation extracted from the retrieval token, where T represents the number of triplet samples and m represents the interval parameter of the triplet loss. Furthermore, in step five, not all the local visual tokens generated in the previous steps are useful; some may represent background or meaningless regions, and these visual tokens are not important to our retrieval task. Selecting and optimizing these visual tokens can not only reduce the number of tokens but also reduce computational complexity to some extent. We optimize token selection by using the attention score between the retrieval token [RT] and all visual tokens; the attention score is calculated using the following formula:

[0015] This formula is passed and Multiply, then divide by the square root of the feature dimension. The inner product attention score is calculated, and then the softmax function is applied to normalize the obtained attention score to obtain the attention score between the retrieved token [RT] and all visual tokens.

[0016] Based on the attention score *s*, only the top *k* visual tokens are retained, while the rest are discarded, thus achieving token selection. During retrieval, the retrieval tokens from the sketch are used to select those with a certain retention rate. The system uses visual tokens of remote sensing images, and remote sensing images with higher attention scores are returned to complete the rapid retrieval of remote sensing images.

[0017] This invention discloses a fast remote sensing image retrieval method based on hand-drawn sketches. Addressing the challenging task of cross-modal remote sensing image retrieval, it introduces a visual transformer model and proposes a feature-enhanced tokenization method to learn more structural information from sketches, taking into account the sparse strokes. It utilizes a self-attention module with retrieval tokens and a cross-modal attention module specifically designed for the retrieval task. Furthermore, it selects and optimizes generated tokens based on the attention scores between the retrieval token and all visual tokens, and then uses the sketches to quickly retrieve remote sensing images based on these attention scores. This invention achieves good results in large-scale sketch-remote sensing image datasets and large-scale remote sensing image retrieval. It also effectively retrieves novel categories not encountered during training (zero-learning capability), demonstrating high model accuracy, particularly in shape retrieval, and possessing significant practical application value. The overall retrieval process is fast and accurate, providing a new method and approach for research on fast remote sensing image retrieval based on hand-drawn sketches. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the technical process of the present invention. Figure 2 This is a model framework diagram of an embodiment of the present invention; Figure 3 This is a partial category retrieval result display in the visible categories (training phase) of this invention embodiment. The green checkmark in the lower right corner of the image represents a correct retrieval, and the red X in the lower right corner represents an incorrect retrieval. The five selected categories are, in order, baseball field, beach, bridge, football field, and river. Figure 4 This is a partial category retrieval result display in the invisible category (training phase) of this invention embodiment. The green checkmark in the lower right corner of the image represents a correct retrieval, and the red X in the lower right corner represents an incorrect retrieval. The five selected categories are, in order, basketball court, pedestrian crossing, oil and gas field, running track, and tennis court. Figure 5This image shows partial category retrieval results from the large-scale remote sensing image Gaofen-1, as an embodiment of the present invention. A green checkmark in the lower right corner of the image indicates a correct retrieval, while a red X in the lower right corner indicates an incorrect retrieval. The sketch categories used for retrieval are, in order, river, bridge, beach, and tennis court. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 This invention provides a method for rapid retrieval of remote sensing images based on hand-drawn sketches, comprising the following steps: Step 1: Tokenization module for feature enhancement; By converting hand-drawn sketches and remote sensing images into visual tokens, and then processing the sparsely drawn sketches through a feature-enhanced tokenization module, the model can learn more sketch structure information. Step 2: Self-attention module with retrieval token; By incorporating retrieval tokens into the self-attention module, more global image features are obtained. During the model's inference phase, all visual tokens (including retrieval tokens) interact through the multi-head self-attention module, and then pass through the multilayer perceptron module. Step 3: Cross-modal attention module; A cross-modal attention module is used to link the sketch and remote sensing image feature information learned in step two, establishing a local visual correspondence between the sketch and the remote sensing image to capture their correlation. New query tensors, key tensors, and value tensors are generated by exchanging sketch queries and remote sensing image queries. Step 4: Definition and use of the triplet loss function with retrieval token; A triadic loss function with a retrieval token is defined and used to train the proposed network. The goal of this loss function is to minimize the triadic loss to align correctly matched sketch-remote sensing image pairs and to keep the target sketch away from incorrect sketch-remote sensing image pairs. Based on this, the retrieval token is applied as a global feature of both the sketch and the remote sensing image. Step 5: Token selection, optimization, and retrieval; Not all local visual tokens generated in steps one through four are useful; some may represent background or meaningless regions. Selecting and optimizing these visual tokens can not only reduce the number of tokens but also decrease computational complexity to some extent. Token selection optimization is achieved by using the attention score *s* between the retrieval token and all visual tokens; during retrieval, the retrieval tokens from the sketch are used to select those with a certain retention rate. The system uses visual tokens of remote sensing images, and remote sensing images with higher attention scores are returned to complete the rapid retrieval of remote sensing images.

[0021] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0022] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for rapid retrieval of remote sensing images based on hand-drawn sketches, characterized in that, The method unifies hand-drawn sketches and remote sensing images into visual tokens, and sequentially performs sketch structure feature enhancement, global semantic representation extraction, cross-modal local visual association modeling, triad constraint training based on retrieval tokens, and token selection optimization within a unified feature space, thereby achieving rapid remote sensing image retrieval for hand-drawn sketch queries; the method includes the following steps: Step 1: Tokenization Module for Feature Enhancement; Hand-drawn sketches and remote sensing images are converted into visual tokens. The tokenization module for feature enhancement supplements the structural information of sparsely drawn sketches, enabling the model to obtain a more complete representation of the sketch structure. The tokenization module consists of four convolutional layers with different kernel sizes stacked sequentially. Each convolutional layer is followed by a non-linear activation function ReLU to expand the receptive field of the visual tokens and preserve sketch detail features through hierarchical convolution. Based on this, residual connections are introduced to update the original tokens, resulting in the following token embedding: in, This represents the visual token embedding after feature enhancement. This represents the original visual token embedding of the input. This represents the sketch structure enhancement features output by a four-layer convolutional tokenizer; Step 2: Self-attention module with retrieval token; A retrieval token is set in the self-attention module to enhance the global feature representation of the image; the retrieval token [RT] is a trainable d-dimensional token embedding; during the model inference stage, the visual token and the retrieval token [RT] are jointly input into the multi-head self-attention module (MSA) for feature interaction, and then input into the multilayer perceptron module (MLP) to complete feature update. The overall forward propagation expression is: In formula (2), RT represents the retrieval token, V¹,…,V n For the embedded token after processing, both formulas (3) and (4) introduce residual connections. The layer index (values ​​are 1, 2, ..., L), where L is the layer number; both hand-drawn sketches and remotely sensed images use the above architecture for feature encoding; the MSA module contains three types of projection heads. The same token is embedded and mapped to queries, keys, and values, respectively, represented by Q, K, and V as follows: The scaling dot product attention is given by the following formula: Step 3: Cross-modal attention module; A cross-modal attention module is used to establish local visual correspondences between hand-drawn sketches and remotely sensed images to characterize the relationship between them; specifically, the sketch is queried... Remote sensing image query The query tensor, key tensor, and value tensor are swapped and combined to form a new cross-modal input. The resulting query tensor, key tensor, and value tensor are respectively... and Cross-modal attention is represented by the following formula: In this formula, and The attention scores are obtained by multiplying the transposes of the expressions and dividing by the square root of the feature dimensions. Then, the attention scores are normalized using the softmax function to obtain the attention weights. Finally, the attention weights are multiplied by the value tensor. Multiplying them together yields the cross-modal attention output; Step 4: Definition and use of the triad loss function with retrieval token; training the network using the triad loss function with retrieval token [RT]; given a triad... ,in For the target sketch, To and Remote sensing images with the same labels, To and Remote sensing images with different labels; this loss function uses the retrieval token [RT] as a global feature of the sketch and the remote sensing image. By reducing the feature distance between the target sketch and the positive sample remote sensing image, and increasing the feature distance between the target sketch and the negative sample remote sensing image, it achieves the alignment of correct sketch-remote sensing image pairs and the separation of incorrect matching pairs. The triplet loss is defined as: in, This represents the global feature representation extracted from the retrieval token, where T represents the number of triplet samples and m represents the interval parameter of the triplet loss. Step 5: Token Selection, Optimization, and Retrieval; Addressing the issue that the local visual tokens generated in the preceding steps may contain background or invalid regions, token filtering is performed based on the attention score between the retrieved token [RT] and each visual token. This aims to reduce the number of tokens and lower computational complexity while retaining valid visual information. The attention score is calculated as follows: In this formula, the query vector corresponding to token[RT] is retrieved. Multiply by the transpose of the key vector K and divide by the square root of the feature dimension to obtain the inner product attention score; after normalization by the softmax function, the attention score between the retrieval token [RT] and each visual token is obtained. Based on the attention score s, the top k visual tokens are retained, and the remaining visual tokens are discarded, thus completing the token selection; during the retrieval phase, the retrieval tokens from the sketch are used to determine the retention rate. Under constraints, select visual tokens for remote sensing images and sort them by attention score to return candidate remote sensing images, thus completing the fast retrieval of remote sensing images for hand-drawn sketches.

Images