KNN Retrieval Method for Remote Sensing Images Using Visual Language Models

Abstract

A KNN search method for remote sensing images in visual language model is presented. The solution includes five steps: S1 Data preprocessing: Base64 processing is performed on the remote sensing image dataset in the visual language test dataset, and JSONL processing is performed on the text dataset, and after processing, they are uploaded to the database. S2 Image and text data serialization: Serialization is performed on the preprocessed dataset so that it can be read randomly during training. S3 CLIP model training: The serialized data is sent to the visual language model for training, and a trained model is obtained. S4 Image and text feature extraction: Feature extraction is performed by passing the test data through the training model. S5 Obtaining KNN search evaluation results: KNN search evaluation is performed on the feature-extracted files.

Description

[Technical Field]

[0001] The present invention relates to the fields of computer vision, remote sensing science, and natural language processing, and particularly to remote sensing image analysis and remote sensing feature extraction techniques in remote sensing science and technology. More specifically, the present invention relates to a KNN search method that combines remote sensing images and visual language. The method combines deep learning models, multi-head attention mechanisms, contrastive learning, image feature extraction, and CLIP (Contrastive Language-Image Pretraining) models, and is used for remote sensing image search tasks in visual language. It is widely applied in fields such as agriculture, forestry, urban planning, environmental monitoring, geological surveys, and disaster prediction. [Background technology]

[0002] With the rapid growth of image-text pair data and the diversification of vision-language tasks, researchers have introduced a large number of deep learning models into this research field. In recent years, transfer learning has also achieved great success in the field of remote sensing, for tasks such as image classification and object detection, and in natural language processing, for tasks such as question answering and machine translation. The great success of vision-language-based models has promoted research and applications in the field of remote sensing and multimodal representation learning. However, how to effectively transfer these foundational models to specific language scenes remains a challenge. Research on contrastive language-image pretraining (CLIP) has proposed the idea of ​​cross-modal contrastive learning, which improves the versatility and adaptability of models by training them using images and natural language text as joint inputs.

[0003] In the field of remote sensing, self-supervised learning and masked image modeling are used to build foundational models. However, these models primarily learn low-level features and require labeled data for fine-tuning. Furthermore, their lack of language understanding makes them unsuitable for search applications. Remote sensing data, typically captured by satellites or drones, contains a large amount of complex visual information that often needs to be interpreted through manual annotation or complex image processing techniques.

[0004] Chinese Patent Application Publication No. 118736625 describes an image-text pedestrian retrieval method based on fusion keypoint attention guidance, which introduces human body keypoint information into the CLIP model through a cross-attention mechanism and adds an additional supervisory signal to reduce the interference of occlusion and background information, thereby improving the accuracy of pedestrian image-text retrieval and reducing the cost of manual annotation.

[0005] Chinese Patent Application Publication No. 119153120 describes a customized hierarchical search system for microscopic cases based on multimodal learning, which uses a CLIP model to process search queries entered in image format, fuses them with search queries entered in text format, extracts their feature representation information, and compares the images in the history data with CLIP-encoded feature representations to obtain the history image that most closely matches the fused search query. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Chinese Patent Application Publication No. 118736625 [Patent Document 2] Chinese Patent Application Publication No. 119153120 Summary of the Invention [Problem to be solved by the invention]

[0007] Although there has been research on visual language search methods in the past, none of these search methods have been implemented in the field of remote sensing images. This invention collects visual language remote sensing datasets, processes them using Base64 and Jsonl processing, serializes the data, sends the processed data to a CLIP large-scale model for training, obtains a KNN search model and performs feature extraction, and finally implements and evaluates KNN search to obtain a KNN search method. [Means for solving the problem]

[0008] The objective of this invention is to provide a KNN search method for remote sensing images using a visual language model, which trains a remote sensing image dataset through the downstream task of KNN search, and then evaluates the remote sensing image test data through evaluations such as recall.

[0009] To achieve the above objectives, the technical means adopted in the present invention includes the following steps: S1: data preprocessing step, S2: image-text data serialization step, S3: CLIP large-scale model training step, S4: image-text feature extraction step, and S5: KNN search evaluation result acquisition step.

[0010] Step S3 includes the steps of: S31: inputting a remote sensing image into a visual encoder, dividing the image into fixed-size patches, and transforming the image patches into an embedding space; S32: adding position encoding to explicitly introduce the position information of the image patches; S33: processing the layer normalized result using a multi-head self-attention mechanism and a feedforward neural network, and introducing dropout to avoid overfitting or underfitting; S34: activating the MLP using the improved QuickGELU, and outputting a probability distribution result through a fully connected layer and a softmax layer; and S35: training using a standard supervised learning method and optimizing using the Adam algorithm. The formula of the improved QuickGELU activation function is:

number

number

number

number

[0011] Step S4 includes: S41: passing the input image through three convolutional layers and one pooling layer to output a lower-resolution feature map; S42: further extracting more complex high-level features through multiple Bottleneck residual blocks; S43: aggregating spatial information using a self-attention mechanism and converting the feature map into a fixed-length vector; and S44: outputting a vector containing high-dimensional features for use in further search tasks. The convolution layer is a deformable convolution, whose formula is as follows:

number

number

[0012] The method includes steps of: S51: calculating the distance between the feature-extracted vector and a vector in the dataset; S52: determining whether the top K neighbors are related based on the ground truth; and S53: obtaining evaluation results of recall and mean_recall, where the evaluations of recall and mean_recall are defined as follows:

number

number

number

[0013] The present invention has the following outstanding effects and advantages: Compared with conventional activation function algorithms, the improved QuickGELU algorithm of the present invention introduces a dynamically adjusted alpha value, which allows the activation function to dynamically change according to the characteristics of the input data. It also adds processing for negative inputs, which can preserve some features and potentially improve the model's performance in certain input situations. Furthermore, the combination of linear and nonlinear elements allows the activation function to better capture complex patterns in the data.

[0014] Compared to traditional multi-head self-attention mechanisms, our method adds two dropouts between the output of the self-attention mechanism and the MLP, enabling it to process large batches of complex, multi-dimensional data. This allows the model to generalize better, is less prone to overfitting, and achieves high classification accuracy. Compared to standard convolutional layers, our method uses deformable convolutions to enhance network performance and more flexibly handle spatial deformations of feature maps, potentially resulting in improved performance for specific tasks. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a structural schematic diagram of the present invention. [Figure 2] FIG. 1 is a conceptual schematic diagram of the CLIP model of the present invention. [Figure 3] FIG. 1 is a VIT framework diagram of the present invention. [Figure 4] 1 is a training flowchart of the present invention. [Figure 5] The UCM_captions dataset. [Figure 6] FIG. 10 is a diagram showing the training index results of the CLIP model. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, the technical means in the embodiments of the present invention will be described clearly and completely with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and are not all embodiments. Based on the embodiments of the present invention, all other embodiments that can be obtained by those skilled in the art without any creative efforts are all within the protection scope of the present invention.

[0017] In the following description, numerous specific details are set forth to provide a thorough understanding of the present invention; however, the present invention may be embodied in other ways different from those described herein, and those skilled in the art may make similar extensions without violating the spirit of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.

[0018] As shown in Figure 1, this invention discloses a KNN search method for remote sensing images using a visual language model, which includes five steps: S1 Data Preprocessing: Base64 processing is performed on the remote sensing image dataset in the visual language test dataset, and JSONL processing is performed on the text dataset, and then the processed dataset is uploaded to the database. S2 Image Text Data Serialization: Serialization is performed on the preprocessed dataset so that it can be read randomly during training. S3 CN-CLIP Large-Scale Model Training: The serialized data is sent to the visual language model for training and a trained model is obtained. S4 Image and Text Feature Extraction: Feature extraction is performed on the test data through the training model. S5 Obtaining KNN Search Evaluation Results: KNN search evaluation is performed on the feature-extracted file.

[0019] As shown in Figure 2, the training process for the CLIP model of the present invention is divided into the following steps: First, a remote sensing image dataset and a text dataset file are input. Then, the remote sensing image and text data are encoded in the encoder, respectively, to obtain their features. Finally, contrastive learning or other cross-modal operations are performed on the image features and text features to learn the semantic association between the image and text, thereby realizing the model's ability to understand and process cross-modal information.

[0020] As shown in Figure 3, the model pre-training in this invention requires the basic framework of the VIT model, and its implementation process is divided into the following steps: First, divide the image into small patches of equal size and flatten them. Then, encode the image through a Transformer encoder and output its tensor features. Finally, the output from the final layer of the Transformer encoder is passed through a multi-layer perceptron (MLP) head (usually a simple fully connected layer) to perform the final task.

[0021] In some embodiments, the image data Base64 processing in step S1 is as follows: Step S11: Read the remote sensing image data file and convert it into Base64 encoding. Step S12: Scan all files in the remote sensing image folder and write the converted Base64 code of each remote sensing image file into a specified .tsv file.

[0022] In some embodiments, the image text data serialization process in step S2 is as follows: Step S21: Serialization is performed on the Base64 processed image file and text file. Step S22: The serialized data is converted into a high-speed memory-mapped database file (LMDB) with a memory index.

[0023] Example 1 Figure 1 is used in combination to explain the remote sensing image KNN search step based on the visual language model of the present invention, and Figure 5 is used in combination to explain the UCM_captions data used in the present invention. The present invention selected the UCM_captions dataset for experimental processing. This dataset contains remote sensing image datasets of 21 geographic / land use types, with an image resolution of 256 x 256 pixels, covering diverse geographic areas such as urban areas, agricultural areas, forests, and grasslands. Each image represents a geographic category, and each image is accompanied by a text description. These descriptions are manually created and are intended to accurately reflect the content and scene characteristics of the image. Each text description typically includes information about the main objects, environment, and scene contained in the image.

[0024] Furthermore, by combining the specific training flowchart in Figure 4, the following steps are carried out: Step S1: Data preprocessing. Step S11: Image data Base64 processing. First, the remote sensing image data file is read and converted to Base64 encoding. The folder storing the remote sensing images is scanned, and Base64 encoding conversion is performed on each remote sensing image file. Next, the converted Base64 encoding is saved in a single .tsv file according to the specified format. The format is "1680 / 9j / 4AAQSkZJ...YQj73140A / / 2Q==". Finally, it is confirmed that the .tsv file can completely store the encoding information of all image files.

[0025] Step S12: Text data JSONL processing. Extract text description data corresponding to the remote sensing image and convert each record into JSONL format. The record format of each line in the JSONL file is as follows: {"text_id":8410,"text":"It is a river with some plants on one bank and sands on the other.","image_ids":

[1682] }. The processed text data is uploaded to a database for easy use in subsequent training.

[0026] Step S2: Image-Text Data Serialization: The .tsv and .jsonl files generated in step S1 are serialized. The serialized files can be used to randomly read data according to training needs. The serialized data is used to generate a database file in LMDB format. The LMDB file acts as an efficient memory index, making it convenient for subsequent high-speed read and write operations.

[0027] Step S3: Training the CLIP large-scale model. Fine-tune the model training parameters, set the number of training GPUs, the training LMDB data path, and specify the ViT-B-16 visual training scale and text training scale. Set the number of training epochs to 50, the training learning rate to 5e-5, and set the storage path for the weight results. Training can then begin. Figure 6 shows the training index results for the CLIP model of the present invention.

[0028] Step S4: Image and text feature extraction process. The weight file path obtained in Step 3 is written into the feature extraction code, and image and text feature extraction is initiated to obtain an image feature file. Each line stores the features of one image in JSON format, with the following format: {"image_id":1680,"feature":[0.0198,...,-0.017,0.0248]}. The text feature file, with the following format: {"text_id":8410,"feature":[0.1314,...,0.0018,-0.0002]}.

[0029] Step S5: Obtain KNN search evaluation results. Step S51: Write the feature file obtained in KNN search step 4 and perform a KNN search to calculate the top-k recall results for text-to-image (text-figure) and image-to-text (figure-text) searches. The output results are saved in the specified jsonl file. Each line represents the top-k image IDs recalled by one text, and the format is as follows: {"text_id":8410, "image_ids": [1358,1645,1004,1585,1531,1674,1245,1678,765,1165]}. Each line represents the top-k text IDs recalled by one image, and the format is as follows: {"image_id": 1680, "text_ids":[6884,7454,8754,5916,6188,8305,7179,4599,4956,4743]}.

[0030] Step S52: Recall calculation evaluation Write the jsonl file obtained by KNN search, and realize the Recall calculation evaluation based on Recall@1 / 5 / 10, and obtain the following result: {"success":true,"score":40.47619047619048,"scoreJson":{"score":40.47619047619048,"mean_recall":40.47619047619048,"r1":14.285714285714285,"r5":35.714285714285715,"r10":71.42857142857143}}.

[0031] The foregoing are merely specific embodiments of the present application, intended to enable those skilled in the art to understand or realize the present application. Various modifications to these examples will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other examples without departing from the spirit or scope of the present application. Therefore, the present application is not intended to be limited to the examples set forth herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. S1: Data pre-processing step; S2: Image text data serialization step; S3: CLIP large-scale model training step; S4: Image and text feature extraction step; S5: KNN search evaluation result acquisition step; A KNN search method for remote sensing images using a visual language model including: The CLIP large-scale model training step comprises: S31: Input a remote sensing image into a visual encoder, divide the image into patches of fixed size, and transform the image patches into an embedding space; S32: adding position encoding to explicitly introduce position information of the image patch; S33: The layer normalized results are processed by a multi-head self-attention mechanism and a feedforward neural network, and dropout is introduced to avoid overfitting or underfitting; S34: Activating the MLP using the improved QuickGELU and outputting the probability distribution result through a fully connected layer and a softmax layer; S35: Training using standard supervised learning methods and optimizing using the Adam algorithm; The formula for the improved QuickGELU activation function is: [Equation 1] [wherein y represents the output result, x represents the input variable, and σ represents the Sigmoid function, [Equation 2] [Equation 3] [Equation 4] α' represents the parameter after dynamic adjustment, and α is the parameter set at initialization.] The image / text feature extraction step includes: S41: Passing an input image through three convolutional layers and one pooling layer to output a lower resolution feature map; S42: Further extracting more complex high-level features through multiple Bottleneck residual blocks; S43: Aggregating spatial information using a self-attention mechanism and converting the feature map into a fixed-length vector; S44: outputting a vector containing the high-dimensional features for use in further search tasks; The convolution layer is a deformable convolution, whose formula is as follows: [Equation 5] where p is the padding parameter in standard convolution, and C ｏｕｔ and C ｉｎ denote the index of the output and input channels, respectively, and i and j denote the position of the output feature map in the height and width directions, respectively; [Equation 6] are the weights of the convolution kernel. Further, the KNN search evaluation result acquisition step includes: S51: Calculating the distance between the feature extracted vector and the vector in the dataset; S52: determining whether the top K neighbors are related based on the ground truth; S53: obtaining evaluation results of the recall and the average recall; The evaluation of recall and average recall is defined as follows: [Equation 7] [Equation 8] [Equation 9] [Where R is the recall, TP is the number of positive examples predicted as positive, FN is the number of negative examples predicted as negative, R@k (k=1, 5, 10) is the recall score, r(k) is the number of relevant images in the top k returned images, r is the total number of images relevant to the query, and MR represents the average recall, with a higher MR score indicating better retrieval effectiveness.] A KNN search method for remote sensing images using a visual language model, characterized by:

2. The data preprocessing step of S1 S11: Reading a remote sensing image data file and converting it into Base64 encoding; S12: Scan all files in the remote sensing image folder and write the converted Base64 code of each remote sensing image file into a specified .tsv file; The KNN search method for remote sensing images using a visual language model according to claim 1, comprising:

3. The image text data serialization step of S2 includes: S21: Serializing the Base64 processed image file and text file; S22: converting the serialized data into a high-speed memory-mapped database file having a memory index; 3. The KNN search method for remote sensing images using a visual language model according to claim 2, comprising:

Images