Remote sensing image knn retrieval method of visual language model

Abstract

The application discloses a KNN retrieval method of remote sensing images in a visual language model, which comprises five steps: S1, data preprocessing: the remote sensing image data set in the visual language test data set is subjected to base64 processing, and the text data set is subjected to jsonl processing, and the processed data is uploaded to a database; S2, image text serialization: the preprocessed data set is serialized to facilitate random reading during training; S3, CLIP model training: the serialized data is sent to a visual language model for training, and a training model is obtained; S4, image-text feature extraction: the test data is subjected to feature extraction through the training model; and S5, KNN retrieval evaluation: the extracted files are subjected to KNN retrieval evaluation. The application can train the KNN retrieval through the visual language test data set and perform retrieval evaluation, so that relevant or similar content can be quickly and accurately retrieved from massive data.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of computer vision, remote sensing science and natural language processing, in particular to remote sensing image analysis and remote sensing feature extraction technology in remote sensing science and technology, and more particularly, the present application relates to a KNN retrieval method combining remote sensing images and visual language, which combines a deep learning model and a multi-head attention mechanism, contrastive learning, image feature extraction and a CLIP (Contrastive Language-Image Pretraining) model, and is used for remote sensing image retrieval tasks in visual language, and is widely used in the fields of agriculture, forestry, urban planning, environmental monitoring, geological survey, disaster warning, etc. BACKGROUND

[0002] With the explosive growth of image-text pair data and the diversity of visual and language tasks, scholars have introduced a large number of deep learning models in this research field. In addition, in recent years, transfer learning has also achieved great success in the field of remote sensing, such as image classification, object detection, etc. tasks, and in natural language processing for question answering, machine translation, etc. The great success of visual-language base models has promoted the research and application of remote sensing field and multi-modal representation learning. However, how to effectively transfer these base models to specific language scenarios is still a challenge. The CLIP (Contrastive Language-Image Pretraining) study proposes the idea of cross-modal contrastive learning, that is, training image and natural language text as joint input, and then improving the generality and adaptability of the model.

[0003] In the field of remote sensing, self-supervised learning and masked image modeling have been used to build base models. However, these models mainly learn low-level features and need labeled data for fine-tuning. In addition, due to the lack of language understanding, they are not suitable for retrieval applications. Remote sensing data is usually taken by satellites or drones, involving a large amount of complex visual information, which often needs to be interpreted through manual annotation or complex image processing techniques.

[0004] In Chinese patent document CN118736625A, a graph-text pedestrian retrieval method based on fusion key point attention guidance is described, which introduces human key point information to the CLIP model through cross-attention mechanism, adds additional supervision signal, reduces the interference of occlusion and background information, improves the accuracy of pedestrian graph-text retrieval, and reduces the cost of manual annotation.

[0005] A customized microscopic case hierarchical retrieval system based on multi-modal learning is recorded in Chinese patent document CN119153120A. The CLIP model is used to process the retrieval requirements input in the form of images, and after fusion with the retrieval requirements input in the form of text, the feature representation information is extracted, and then compared with the feature representation of the historical image encoded by CLIP to obtain the historical image closest to the fused retrieval requirements.

[0006] Although previous studies have involved visual language retrieval methods, none of them have been implemented in the field of remote sensing images. The present application collects a visual language remote sensing dataset, and performs base64 processing and jsonl processing on it, then serializes the data, and then sends the processed data into the CLIP large model for training, and then obtains the KNN retrieval model and performs feature extraction, finally realizes KNN retrieval and evaluation, and finally obtains a KNN retrieval method. SUMMARY

[0007] The present application aims to provide a visual language model remote sensing image KNN retrieval method, which trains remote sensing image datasets through downstream task KNN retrieval, and then evaluates remote sensing image test data through recall rate and other evaluations.

[0008] To achieve the above purpose, the technical solution adopted by the present application includes the following steps:

[0009] Step S1: data preprocessing: base64 processing of remote sensing image data sets in the test data set, and jsonl processing of text data sets, and uploading the processed data to the database;

[0010] Step S2: image and text data serialization: serializing the preprocessed data set to facilitate random reading during training;

[0011] Step S3: CLIP large model training: sending the serialized data to the visual language model for training to obtain the training model;

[0012] Step S4: image and text feature extraction: feature extraction of test data through the training model;

[0013] Step S5: KNN retrieval evaluation: KNN retrieval evaluation of the extracted feature files.

[0014] The step S1 image data base64 processing is as follows:

[0015] Step S11: read the remote sensing image data file and convert it to base64 encoding;

[0016] Step S12: Traverse all files in the remote sensing image folder, and for each remote sensing image file, convert the base64 encoding and write it to the specified.tsv file;

[0017] The step S2 image text data serialization process is as follows:

[0018] Step S21: Serialize the base64 processed image file and the text file;

[0019] Step S22: Convert the serialized data into a memory indexed LMDB database file;

[0020] The step S3 model pre-training is as follows:

[0021] Step S31: Input the remote sensing image, cut the image into fixed size patches, and convert the image patches into an embedding space;

[0022] Step S32: Add position encoding to explicitly introduce the position information of the tile;

[0023] Step S33: Process the layer normalized result through multi-head self-attention mechanism and feedforward neural network, and introduce dropout to avoid overfitting or underfitting;

[0024] Step S34: Improve the QuickGELU activation MLP, and output the probability distribution result through the fully connected layer and the softmax layer; wherein the formula of the improved QuickGELU activation function is:

[0025] (1)

[0026] Where y represents the output result, x is the input variable, and σ represents the Sigmoid function, i.e.

[0027] (2)

[0028] (3)

[0029] (4)

[0030] represents the dynamically adjusted parameter, wherein α is the parameter set at initialization.

[0031] Step S35: Train through the standard supervised learning method and adjust and optimize in adam.

[0032] The step S4 image-text feature extraction process is as follows:

[0033] Step S41: After the input image passes through three convolutional layers and one pooling layer, a lower resolution feature map is output, wherein the convolutional layer is a variable convolution, and the formula is:

[0034] (5)

[0035] wherein p is a padding parameter in the standard convolution, and respectively represent the output and input channel indexes, and i and j respectively represent the positions of the output feature map in the height and width directions, is the convolution kernel weight.

[0036] Step S42: More complex high-level features are further extracted through a plurality of Bottleneck residual blocks.

[0037] Step S43: The self-attention mechanism is used to aggregate spatial information to change the feature map into a fixed-length vector.

[0038] Step S44: A vector containing high-dimensional features is output for further retrieval tasks.

[0039] The step S5KNN retrieval evaluation is as follows:

[0040] Step S51: The distance between the feature extraction vector and the vector in the data set is calculated;

[0041] Step S52: Based on the ground truth, it is determined whether the top K neighbors are relevant;

[0042] Step S53: The precision, recall and F1-Score are evaluated. The evaluation of the recall is specifically defined as follows:

[0043] (6)

[0044] (7)

[0045] (8)

[0046] wherein R is the recall, TP represents the number of positive classes predicted as positive classes, FN represents the number of negative classes predicted as negative classes, R@k represents the recall point, r(k) represents the number of relevant images in the top k images returned, r represents the total number of query relevant images, and MR represents the average recall. The higher the MR score, the better the retrieval effect.

[0047] The present application has the following excellent effects and advantages:

[0048] Compared with the traditional activation function algorithm, the improved QuickGELU algorithm of the application introduces a dynamically adjusted 'alpha' value, which allows the activation function to dynamically change according to the characteristics of the input data; the processing of negative input is increased, which may help to retain some features, so that the model can perform better under certain input conditions; linear and nonlinear elements are combined to make the activation function better capture complex patterns in the data.

[0049] Compared with the traditional multi-head self-attention mechanism, the application can process large batches of complex data in multiple dimensions by adding two dropouts between the attention output and the MLP, has strong generalization ability of the model, is not easy to cause overfitting problem, and has high classification accuracy. Compared with the standard convolution layer, the application uses a deformable convolution to enhance the performance of the network, which makes it more flexible to handle spatial deformation of feature maps, and may improve the performance of certain tasks. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0051] Figure 1 The structural schematic diagram of the application;

[0052] Figure 2 The CLIP model concept schematic diagram of the application;

[0053] Figure 3 The VIT framework diagram of the application;

[0054] Figure 4 The training flowchart of the application;

[0055] Figure 5 The UCM_captions data set;

[0056] Figure 6 The CLIP model training index result diagram. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0058] In the following description, a lot of specific details are set forth in order to provide a thorough understanding of the present application, however, the present application can be practiced without other different ways from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the present application, therefore, the present application is not limited to the specific embodiments disclosed below.

[0059] As shown in the Figure 1 The application discloses a remote sensing image KNN retrieval method of a visual language model, which comprises the following steps: S1, data preprocessing: the remote sensing image data set in the visual language test data set is subjected to base64 processing, and the text data set is subjected to jsonl processing, and the processed data is uploaded to a database; S2, image and text data serialization: the preprocessed data set is serialized to facilitate random reading during training; S3, CN-CLIP large model training: the serialized data is input into a visual language model for training, and a training model is obtained; S4, image and text feature extraction: test data is subjected to feature extraction through the training model; and S5, KNN retrieval evaluation: the extracted files are subjected to KNN retrieval evaluation.

[0060] As shown in the Figure 2 The CLIP model training process of the application comprises the following steps:

[0061] Firstly, input the remote sensing image data set and the text data set file,

[0062] Secondly, the remote sensing image and the text data are respectively subjected to encoding processing in an encoder, and the features thereof are obtained,

[0063] Finally, the image features and the text features are subjected to contrastive learning or other cross-modal operations to learn the semantic association between the image and the text, so that the understanding and processing ability of the model to the cross-modal information is realized.

[0064] As shown in the Figure 3 The model pre-training in the application needs the basic framework of the VIT model, and the running process thereof comprises the following steps:

[0065] Firstly, the image is cut into small blocks of the same size and is laid flat,

[0066] Secondly, the transformer encoder is subjected to encoding processing, and the tensor features are output,

[0067] Finally, the output of the last layer from the transformer encoder passes through a multi-layer perception (MLP) head, which is usually a simple fully connected layer, for performing the final task.

[0068] Embodiment:

[0069] In combination with Figure 1The application discloses a remote sensing image KNN retrieval method based on a visual language model, which combines Figure 5 The application adopts UCM_captions data. The application selects the UCM_captions data set for experiment processing. The data set contains 21 types of remote sensing image data sets of geographic / land use, the image resolution is 256x256 pixels, and various geographic regions such as urban areas, agricultural areas, forests and grasslands are covered. Each image represents a geographic category, and each image is provided with a text description. The description is artificially written and aims to accurately reflect the content and scene characteristics of the image. Each text description usually includes the main objects, environment and scene information contained in the image.

[0070] In combination with the training specific flowchart, the following steps are implemented: Figure 4

[0071] Step S1: data preprocessing

[0072] Step S11: base64 processing of image data

[0073] Firstly, the remote sensing image data file is read and converted into base64 code. The folder storing the remote sensing image is traversed, and base64 encoding conversion is performed on each remote sensing image file.

[0074] Secondly, the converted base64 code is stored in a.tsv file according to a specified format, which is: 1680 / 9j / 4AAQSkZJ...YQj7314oA / / 2Q==

[0075] Finally, it is ensured that the.tsv file can completely save the encoding information of all image files.

[0076] Step S12: jsonl processing of text data

[0077] The text description data corresponding to the remote sensing image is extracted, and each record is converted into JSONL format. The format of each record in the JSONL file is: {"text_id": 8410, "text": "It is a river with some plants onone bank and sands on the other side.", "image_ids":

[1682] }. The processed text data is uploaded to the database for subsequent training use.

[0078] Step S2: image text data serialization

[0079] ​The.tsv and.jsonl files generated in step S1 are serialized, the serialized files can randomly read data according to the training requirements, and the serialized data is used to generate a database file in the LMDB format, and the LMDB file can be used as an efficient memory index, facilitating subsequent fast access operations.

[0080] Step S3: CLIP large model training

[0081] The training parameters of the model are fine-tuned, the number of training GPUs is set, the LMDB data path for training is set, the ViT-B-16 visual training scale and the text training scale are specified, the number of training steps is 50 rounds, the learning rate for training is set to 5e-5, and the weight result storage path is set.

[0082] As shown in Figure 6 The training index result of the CLIP model of the application is shown.

[0083] Step S4: image-text feature extraction processing

[0084] The weight file path obtained by training in step 3 is written into the feature extraction code to start image-text feature extraction and obtain an image feature file, each row stores the features of a picture in json, and the format is as follows: { "image_id": 1680, "feature": [0.0198,..., -0.017, 0.0248]}, and a text feature file, the format is as follows: { "text_id": 8410, "feature": [0.1314,..., 0.0018, -0.0002]}

[0085] Step S5: KNN retrieval evaluation

[0086] Step S51: KNN retrieval

[0087] The feature file obtained in step 4 is written into KNN retrieval, the top-k recall results of text-to-image and image-to-text retrieval are calculated, and the output results are saved in the specified jsonl file, each row represents the top-k picture id of a text recall, and the format is as follows: { "text_id": 8410, "image_ids": [1358, 1645, 1004, 1585, 1531, 1674, 1245, 1678, 765, 1165]} Each row represents the top-k text id of a picture recall, and the format is as follows: { "image_id": 1680, "text_ids": [6884, 7454, 8754, 5916, 6188, 8305, 7179, 4599, 4956, 4743]}

[0088] Step S52: Recall calculation evaluation

[0089] The jsonl file obtained by KNN retrieval is written and Recall calculation evaluation is implemented according to Recall@1 / 5 / 10, and the result is as follows:

[0090] {"success":true,"score":40.47619047619048,"scoreJson":{"score":40.47619047619048,"mean_recall":40.47619047619048,"r1":14.285714285714285,"r5":35.714285714285715, "r10": 71.42857142857143}}.

[0091] The above described is only the specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features sought to be applied herein.

Claims

1. A KNN retrieval method for remote sensing images based on a visual language model, characterized in that, Includes the following steps: S1: Data preprocessing; S2: Image text data serialization; S3: CLIP Large Model Training; Step S3 CLIP Large Model Training includes: S31: Input remote sensing imagery, divide the image into fixed-size blocks, and convert the image blocks into an embedding space; S32: Add location coding to explicitly import the location information of the tiles; S33: The results after layer normalization are processed by multi-head self-attention mechanism and feedforward neural network, and dropout is introduced to avoid overfitting or underfitting. S34: The improved QuickGELU activation MLP is activated, and the probability distribution results are output through a fully connected layer and a softmax layer; the formula for the improved QuickGELU activation function is: （1） Where y represents the output result, x is the input variable, and σ represents the Sigmoid function, i.e. （2） （3） （4） This represents the parameters after dynamic adjustment, where α is the parameter set during initialization; Step S35: Train using standard supervised learning methods and optimize in Adam; S4: Image and text feature extraction; Step S4, image and text feature extraction processing, includes: S41: After the input image passes through three convolutional layers and one pooling layer, it outputs a low-resolution feature map. The convolutional layers are variable convolutions, and their formulas are as follows: （5） Where p is the padding parameter in standard convolution. and These represent the output and input channel indices, respectively, and i and j represent the positions of the output feature map in the height and width directions, respectively. These are the convolution kernel weights; S42: Further extract more complex high-level features through multiple Bottleneck residual blocks; S43: Use a self-attention mechanism to aggregate spatial information and turn the feature map into a fixed-length vector; S44: Output a vector containing high-dimensional features for further retrieval tasks; S5: KNN retrieval evaluation, step S5 KNN retrieval evaluation includes: S51: Calculate the distance between the extracted feature vector and the vectors in the dataset; S52: Determine whether the first K neighbors are related based on the ground truth; S53: Evaluate its recall and mean_recall; the specific definition of its recall evaluation is as follows: （6） （7） （8） Where R is the recall rate, TP represents the number of positive classes predicted as positive, FN represents the number of negative classes predicted as negative, R@k (k=1,5,10) represents the recall point, r(k) represents the number of relevant images in the first k returned images, r represents the total number of relevant images in the query, and MR represents the average recall rate; the higher the MR score, the better the retrieval effect.

2. The remote sensing image KNN retrieval method based on a visual language model as described in claim 1, characterized in that, The data preprocessing in step S1 includes: S11: Read remote sensing image data files and convert them to base64 encoding; S12: Iterate through all files in the remote sensing image folder, and for each remote sensing image file, convert its base64 encoding and write it to the specified .tsv file.

3. The remote sensing image KNN retrieval method based on a visual language model as described in claim 1, characterized in that, The image text data serialization step S2 includes: S21: Serialize the base64-processed image file and the text file; S22: Convert the serialized data into an LMDB database file with an in-memory index.

Images