Remote sensing image-text retrieval method based on self-adaptive local knowledge filling

Abstract

The invention provides a remote sensing image-text retrieval method based on self-adaptive local knowledge filling. The method comprises the following steps: step 1) creating a remote sensing image-text database; the method specifically comprises the following steps: collecting a plurality of remote sensing images from a plurality of remote sensing data sources to form a remote sensing image-text data set; dividing a plurality of remote sensing images in the remote sensing image-text data set into a training data set, a verification data set and a test data set according to a certain proportion; 2) training a CLIP model by using the remote sensing image-text data set; 3) training an adaptive network by using the remote sensing image-text data set; 4) performing retrieval test on downstream images and texts by using the remote sensing image and text data set; according to the method, text modals are supplemented for the remote sensing field and local knowledge is adaptively supplemented in retrieval by using the multi-modal large language model, and rich and accurate semantic support is provided for image-text retrieval in the remote sensing field.

Description

Technical Field

[0001] The invention relates to a remote sensing image and text retrieval method based on adaptive local knowledge filling, belonging to the field of remote sensing image and text retrieval. Background Art

[0002] With the rapid development of remote sensing technology, we have obtained a huge amount of remote sensing image data; however, these data often lack sufficient text descriptions, which limits the efficiency and accuracy of image and text retrieval; traditional image and text retrieval methods are difficult to effectively utilize the rich information in images, and lack the ability to adapt to the characteristics of different data sets.

[0003] In addition, the current training paradigm of pre-training-fine-tuning-downstream tasks is widely used, including CLIP and CLIP-based model training methods, which leads to the basic requirement of image-text pairs for basic model pre-training; in addition, in recent years, multimodal large language models such as GPT-4V and BLIP2 have emerged in an endless stream, and they have attracted the attention of researchers with their excellent understanding ability and fluent conversation ability; one of the notable features of large models is that a large amount of training data is required to adjust a large number of parameters, so the demand for images and their descriptions when training models is increasing; studies have shown that existing low-quality descriptions can hinder the alignment between visual and language modalities, and higher-quality descriptions without hallucination phenomena are more suitable for the current alignment of image-text modalities; in other words, obtaining higher-quality image-text pairs will play a very important role in downstream tasks such as image-text retrieval.

[0004] Remote sensing image and text retrieval was first explored by Abdullah et al. (Abdullah T, Bazi Y, Al Rahhal MM, et al. TextRS: Deep bidirectional triplet network for matching text to remote sensing images[J]. Remote Sensing, 2020, 12(3): 405.) and Rahhal et al. (Rahhal MMA, Bazi Y, Abdullah T, et al. Deep unsupervised embedding for remote sensing image retrieval using textual cues[J]. Applied Sciences, 2020, 10(24): 8931.), who used CNN to encode images and LSTM to encode text descriptions. In order to supplement the more complete remote sensing semantics, methods such as KCR (Mi L, Li S, Chappuis C, et al. Knowledge-aware cross-modal text-image retrieval for remote sensing images [C] / / Proceedings of the Second Workshop on Complex Data Challenges in Earth Observation (CDCEO 2022). 2022.) introduced knowledge graphs into text descriptions, and Rahhal et al. (Al Rahhal MM, Bazi Y, Alsharif NA, et al. Multilanguage transformer for improved text to remote sensing image retrieval [J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, 15: 9115-9126.) introduced remote sensing image and text retrieval in a multi-language framework.

[0005] Generally speaking, images and texts contain different information densities. Often, the information conveyed by an image requires multiple sentences to describe, and even different subjects may describe the same image from different angles. This is especially true for remote sensing images. The resolution of remote sensing images is lower than that of general domain images, and the receptive field is larger, which leads to the inaccurate association between images and texts, which is not conducive to the development of image-text retrieval tasks. At present, there are also some studies on the global and local information of remote sensing images, such as Yang et al. (Yang R, Wang S, Han Y, et al. Transcending Fusion: A Multi-Scale Alignment Method for Remote Sensing Image-Text Retrieval [J]. arxiv preprint arxiv: 2405.18959, 2024.), Yuan et al. (Yuan Z, Zhang W, Tian C, et al. Remote sensing cross-modal text-image retrieval based on global and local information [J]. IEEE Transactions on Geoscience and Remote Sensing) Sensing, 2022, 60:1-16.), but they did not adaptively change the ratio of local information to global information for a single dataset, but only supplemented some local information.

[0006] Compared with general domain images, remote sensing images usually have lower resolution. This feature is due to the fact that remote sensing images are acquired over a wider range and are intended to cover a larger area of ​​the surface or space. Therefore, remote sensing images have a larger receptive field and relatively lower details and resolution. This low resolution characteristic causes remote sensing images to be insufficient in capturing detailed information, which in turn affects the precise association between the image and its corresponding text. This situation causes the semantic information of remote sensing images to be often vague and unable to accurately reflect the specific content in the image, thus posing a great challenge in image-text retrieval tasks.

[0007] Specifically, the details of objects, terrain or phenomena in remote sensing images are often not as clear as those in high-resolution images, which makes the matching process between images and texts complicated and difficult; since the low resolution of the image limits the expression of details in the image, while the text description requires precise details for effective matching. Summary of the invention

[0008] The present invention proposes a remote sensing image and text retrieval method based on adaptive local knowledge filling, which aims to supplement local knowledge through a multimodal large language model and an adaptive network to improve the performance of image and text retrieval.

[0009] The technical solution of the present invention is a remote sensing image and text retrieval method based on adaptive local knowledge filling, the method comprising:

[0010] Step 1) creating a remote sensing image and text database; specifically comprising: collecting a number of remote sensing images from a number of remote sensing data sources to form a remote sensing image and text data set; dividing a number of remote sensing images in the remote sensing image and text data set into a training data set, a verification data set, and a test data set according to a certain ratio;

[0011] Step 2) Training the CLIP model with remote sensing image and text datasets;

[0012] Step 3) training the adaptive network using remote sensing image and text datasets;

[0013] Step 4) Use the remote sensing image and text dataset to perform retrieval test on downstream images and text.

[0014] Furthermore, the step 2) uses the remote sensing image and text dataset to train the CLIP model, specifically including: using the training dataset obtained in step 1) to train the visual language contrast learning model, and finally obtaining the trained visual language contrast learning model, that is, obtaining an image feature extractor and a text feature extractor.

[0015] Furthermore, the step 3) uses the remote sensing image and text dataset to train the adaptive network, specifically comprising: first, inputting the verification dataset obtained in step 1) into the image feature extractor and text feature extractor obtained in step 2) to extract the splicing features of the image and text; then, using the splicing features of the image and text to train the adaptive network, after 5 layers of fully connected layers, outputting five parameters representing the global and local proportions, and finally obtaining an optimized adaptive network and global and local proportion parameters.

[0016] Furthermore, the step 4) uses the remote sensing image and text dataset to perform a retrieval test on the downstream images and texts, specifically comprising: first, inputting the test dataset obtained in step 1) into the image feature extractor and text feature extractor obtained in step 2) to obtain the test set image features and text features; then, inputting the test set image features and text features into the optimized adaptive network according to step 3), and adjusting the test set data according to the global and local proportion parameters.

[0017] Furthermore, the specific process of step 1) creating a remote sensing image and text database includes:

[0018] Step 1-1) Collect a certain number of remote sensing images (I1, I2, ..., IN );

[0019] Step 1-2) Divide each remote sensing image into four local images: "upper left corner", "lower left corner", "upper right corner", and "lower right corner", denoted as Then the four local images are input into the multimodal large language model respectively, and specific prompts are input for the remote sensing images, and finally four local texts are obtained.

[0020]

[0021] Step 1-3) Input the four local texts corresponding to each remote sensing image into the multimodal large language model, input the prompts that integrate the local knowledge to generate the overall image description, and finally make its reference image output the overall text T of the entire image of each remote sensing image i .

[0022] Furthermore, the specific process of training the CLIP model with the remote sensing image and text dataset in step 2) includes:

[0023] Step 2-1) Obtain the training data set in step 1) and record it as Where N represents the number of samples in the training data set;

[0024] Step 2-2) Use F(I) and G(T) to represent the image features and text features extracted by the image feature extractor and text feature extractor respectively:

[0025]

[0026] The above means {global image features, global text features, [four local image features], [four local text features]} of the i-th sample in the training data set;

[0027] Step 2-3) The image and text similarity of the i-th sample in the training data set is calculated using a common similarity calculation method. The common similarity calculation method is preferably a cosine calculation method cos():

[0028] sim(I i ,T i )=cos(F(I i ),G(T i ))

[0029] Where sim(I i ,T i ) represents the image and text similarity of the i-th sample in the training data set, cos() represents the cosine calculation method, cos(F(I i ),G(T i)) represents the cosine similarity between the image features and text features of the i-th sample in the training data set;

[0030] Step 2-4) Calculate the loss function of the CLIP model as follows:

[0031]

[0032] Among them, L represents the loss function, N represents the sample size of the training data set, and the ultimate goal is to bring the positive sample pairs closer and the negative sample pairs farther apart during the training process through contrastive learning.

[0033] Furthermore, the step 3) of training the adaptive network using the remote sensing image and text dataset specifically includes:

[0034] Step 3-1) Obtain the validation data set obtained in step 1) and also count it as

[0035] Step 3-2) F(I) and G(T) represent the image features and text features extracted by the image feature extractor and text feature extractor respectively, define the adaptive network, and calculate the text features that integrate the local information:

[0036]

[0037] Among them, α represents the proportion of global information, β 11 ,β 12 ,β 21 ,β 22 Respectively represent the proportion of local information, and (α, β 11 ,β 12 ,β 21 ,β 22 )The sum of all parameters is 1;

[0038] Step 3-3) The expression for calculating the image and text similarity of the i-th sample in the validation dataset can be

[0039] It is expressed as:

[0040] sim(I i ,T i ) ‘ =cos(F(I i ), G(T i ) ‘ )

[0041] Where sim(I i ,T i ) ‘ represents the image and text similarity that integrates global and local information, cos() represents the cosine calculation method, cos(F(I i), G(T i ) ‘ ) represents the cosine similarity of image features and text features that integrate global and local information;

[0042] Step 3-4) Train the adaptive network: At this time, fix the CLIP model trained in step 2) and train the adaptive network. The loss function of the adaptive network is still set to:

[0043]

[0044] Furthermore, the specific process of step 4) using the remote sensing image and text data set to perform a retrieval test on the downstream image and text includes:

[0045] Step 4-1) Obtain the test data set obtained in step 1);

[0046] Step 4-2) uses the text feature extractor obtained in step 2) to extract the global and local text features of the test data set, according to the global information ratio and local information ratio obtained in step 3), that is, α and β 11 ,β 12 ,β 21 ,β 22 , generate text features that fuse global and local information in the test data set; use the image feature extractor obtained in step 2) to extract global image features of the test data set;

[0047] Step 4-3) Calculate the image and text retrieval results.

[0048] Furthermore, the step 4-3) calculates the image-text retrieval results, specifically including: first, based on the text features that integrate global and local information, updating the matching strategy between the image and the text, and recalculating the similarity between the image and the text; second, combining the semantic information described in the local text, using the improved text feature calculation method in step 3-2) and the improved similarity calculation method in step 3-3) to more accurately evaluate the similarity between the image and the text, thereby obtaining a more accurate image-text retrieval result.

[0049] Furthermore, in the step 1), several remote sensing images in the remote sensing image and text dataset are divided into a training dataset, a verification dataset, and a test dataset according to a certain ratio, specifically, several remote sensing images in the remote sensing image and text dataset are divided into a training dataset, a verification dataset, and a test dataset according to a ratio of 6:2:2.

[0050] Beneficial effects of the present invention:

[0051] 1) The present invention is based on a remote sensing image and text retrieval method based on adaptive local knowledge filling. It automatically provides local cognition for remote sensing images through a multimodal large language model, completes remote sensing general model training and remote sensing image and text retrieval based on local information, and uses a multimodal large language model to generate corresponding descriptions for remote sensing images. This method effectively supplements the text modality information of remote sensing images; this not only improves the semantic understanding ability of remote sensing images, but also provides more comprehensive text support for the training of remote sensing models, thereby enhancing the model's cognitive ability for remote sensing data; the present invention realizes the use of a multimodal large language model to complete the text modality in the remote sensing field and adaptively supplement local knowledge in retrieval, providing rich and accurate semantic support for image and text retrieval in the remote sensing field;

[0052] 2) The present invention introduces a local knowledge supplement mechanism, considers the lack of local knowledge of remote sensing images, and uses it as knowledge supplement in the training and testing process, thus providing a remote sensing image and text retrieval method based on adaptive local knowledge filling, providing a new idea for improving the image and text retrieval capability of remote sensing models, and making innovative improvements to remote sensing image and text retrieval;

[0053] 3) The present invention proposes to use local knowledge to enrich the semantic content of remote sensing image and text retrieval, thereby providing the retrieval system with more detailed and rich semantic information; this method improves the limitations of traditional retrieval technology in dealing with specific situations, making the retrieval results more accurate and targeted;

[0054] 4) The present invention proposes a method for adaptively supplementing remote sensing image knowledge by injecting unique local information parameters into each remote sensing dataset, so that each dataset can obtain tailored knowledge supplementation; this adaptive mechanism not only improves the relevance and effectiveness of knowledge injection, but also significantly improves the accuracy and practicality of remote sensing image and text retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Attached Figure 1 It is a flow chart of the remote sensing image and text retrieval method based on adaptive local knowledge filling of the present invention.

[0056] Attached Figure 2 It is a schematic diagram of the effect of generating text fused with local information based on a multimodal large language model in the present invention.

[0057] Attached Figure 3 It is a schematic diagram of the effect of training a feature extractor based on contrastive learning in the present invention.

[0058] Attached Figure 4 It is a schematic diagram of the effect of the image and text retrieval based on the adaptive network of the present invention. DETAILED DESCRIPTION

[0059] A remote sensing image and text retrieval method based on adaptive local knowledge filling, the method comprising the following steps:

[0060] Step 1) Create a remote sensing image and text database: collect a number of remote sensing images from a number of remote sensing data sources to form a remote sensing image and text data set; the remote sensing images include geographical area information, time point information, and sensor type; divide a number of remote sensing images in the remote sensing image and text data set into a training data set, a verification data set, and a test data set according to a certain ratio; the remote sensing images in the remote sensing image and text data set cover different geographical areas, time points, and sensor types;

[0061] Step 2) training a CLIP model (Contrastive Language-Image Pre-training) with a remote sensing image and text dataset, specifically comprising: using the training dataset obtained in step 1) to train the visual language contrastive learning model, and finally obtaining a trained visual language contrastive learning model, that is, obtaining an image feature extractor and a text feature extractor;

[0062] Step 3) training the adaptive network with the remote sensing image and text dataset, specifically comprising: first, inputting the verification dataset obtained in step 1) into the image feature extractor and the text feature extractor obtained in step 2) to extract the splicing features of the image and the text; then, using the splicing features of the image and the text to train the adaptive network, after 5 layers of fully connected layers, outputting five parameters representing the global and local proportions, and finally obtaining an optimized adaptive network and global and local proportion parameters;

[0063] Step 4) uses the remote sensing image and text dataset to perform a retrieval test on the downstream images and texts, specifically comprising: first, inputting the test dataset obtained in step 1) into the image feature extractor and text feature extractor obtained in step 2) to obtain the test set image features and text features; then, inputting the test set image features and text features into the optimized adaptive network according to step 3), and adjusting the test set data according to the global and local ratio parameters.

[0064] In the step 1), several remote sensing images in the remote sensing image and text data set are divided into a training data set, a verification data set, and a test data set according to a certain ratio, preferably, several remote sensing images in the remote sensing image and text data set are divided into a training data set, a verification data set, and a test data set according to a ratio of 6:2:2; several remote sensing images are collected from several remote sensing data sources to form a remote sensing image and text data set, preferably, 160,000 remote sensing images are collected from several remote sensing data sources to form a remote sensing image and text data set.

[0065] In view of the scarcity of text data, the present invention adopts advanced multimodal large language models (such as GPT series, Qwen, etc.), and further preferably the multimodal large language model Qwen-7b, to perform in-depth analysis on each remote sensing image, and automatically generate detailed global and local text descriptions, thereby obtaining a remote sensing graphic data set, which covers the key elements, geographical features, environmental changes, etc. in the image; the generated text is corresponded one-to-one with the original remote sensing image to construct a large-scale, high-quality remote sensing graphic database, and the training data set, verification data set and test data set are randomly divided in a ratio of 6:2:2; the remote sensing graphic database in the present invention not only enriches the text modality, but also provides a solid foundation for subsequent training and retrieval.

[0066] The specific process of step 1) creating a remote sensing image and text database includes the following steps:

[0067] Step 1-1) First, collect a certain number of remote sensing images (I1, I2, ..., I N ), the certain number of remote sensing images should cover different geographical areas and scenes to ensure the diversity and coverage of the data;

[0068] Step 1-2) Then, each remote sensing image is divided into four local images: "upper left corner", "lower left corner", "upper right corner", and "lower right corner", recorded as Then the four partial images are input into the multimodal large language model respectively, and a specific prompt is input for the remote sensing image: "You are an expert in the field of remote sensing. This is the upper left corner / lower left corner / upper right corner / lower right corner of a remote sensing image. Please describe this image in one sentence." Finally, four partial texts are obtained.

[0069] Step 1-3) Input the four local texts corresponding to each remote sensing image into the multimodal large language model, and input the prompt "You are an expert in the field of remote sensing. Please refer to the entire remote sensing image and its four local text descriptions to output the description of the entire image" to finally make it output the overall text T of each remote sensing image. i .

[0070] The specific process of step 2) training the CLIP model with the remote sensing image and text data set includes: the CLIP model first extracts features from images and texts respectively, uses a convolutional neural network (CNN) or a vision transformer (VisionTransformer) to process images, and uses a transformer (Transformer) model to process texts; then, the CLIP model maps the feature vectors of the image and text into a shared space, and uses contrastive learning to maximize the similarity between matching image-text pairs, while minimizing the similarity between unmatched image-text pairs; the CLIP model performs well in the image-text matching task through the idea of ​​contrastive learning; the present invention uses the remote sensing image and text data set generated in step 1) as the input for training the CLIP model, and fine-tunes the CLIP model so that it can better understand the unique features of remote sensing images and learn to closely associate images with generated text descriptions.

[0071] The specific process of training the CLIP model with the remote sensing image and text dataset in step 2) includes the following process:

[0072] Step 2-1) Obtain the training data set in step 1) and record it as Where N represents the number of samples in the training data set;

[0073] Step 2-2) Use F(I) and G(T) to represent the image features and text features extracted by the image feature extractor and text feature extractor respectively:

[0074]

[0075] The above means {global image features, global text features, [four local image features], [four local text features]} of the i-th sample in the training data set;

[0076] Step 2-3) lays the computational foundation for subsequent model optimization reasoning. The image and text similarity of the i-th sample in the training data set is calculated using a common similarity calculation method. The common similarity calculation method is preferably the cosine calculation method cos():

[0077] sim(I i ,T i )=cos(F(I i ),G(T i ))

[0078] Where sim(I i ,T i ) represents the image and text similarity of the i-th sample in the training data set, cos() represents the cosine calculation method, cos(F(I i ),G(Ti )) represents the cosine similarity between the image features and text features of the i-th sample in the training data set;

[0079] Step 2-4) Using the above similarity calculation method, the loss function of the CLIP model is calculated as follows:

[0080]

[0081] Among them, L represents the loss function, N represents the sample size of the training data set, and the ultimate goal is to bring the positive sample pairs closer and the negative sample pairs farther apart during the training process through contrastive learning.

[0082] The adaptive network in the adaptive network trained with the remote sensing image and text data set can receive the output of the CLIP model and further process it; in the process of training the adaptive network, the adaptive network will automatically adjust parameters according to the proportion of local and global information. This adjustment is based on the verification data set of the current data set, aiming to achieve personalized optimization for the entire data set. Through multiple iterative training, the parameters of the adaptive network are continuously optimized until satisfactory performance indicators are achieved.

[0083] The input of the adaptive network is the concatenated features of the image and text features of the verification data set; the input passes through five fully connected ReLU layers to compress the features to the last layer, and the last layer has five parameters, representing the proportion of global and local information; the role of the adaptive network is to obtain the proportion of global and local information from the data set; the problem of optimizing the proportion of global and local information is transformed into optimizing the adaptive network; under the constraint of the loss function, the optimized proportion of global and local information is finally obtained.

[0084] The step 3) training the adaptive network with the remote sensing image and text data set specifically comprises the following steps:

[0085] Step 3-1) Obtain the validation data set obtained in step 1) and also count it as

[0086] Step 3-2) F(I) and G(T) represent the image features and text features extracted by the image feature extractor and text feature extractor respectively, define the adaptive network, and calculate the text features that integrate the local information:

[0087]

[0088] Among them, α represents the proportion of global information, β 11 ,β 12 ,β 21 ,β 22 Respectively represent the proportion of local information, and (α, β 11 ,β12 ,β 21 ,β 22 )The sum of all parameters is 1;

[0089] Step 3-3) The expression for calculating the image and text similarity of the i-th sample in the validation dataset can be

[0090] It is expressed as:

[0091] sim(I i ,T i ) ‘ =cos(F(I i ), G(T i ) ‘ )

[0092] Where sim(I i ,T i ) ‘ represents the image and text similarity that integrates global and local information, cos() represents the cosine calculation method, cos(F(I i ), G(T i ) ‘ ) represents the cosine similarity of image features and text features that integrate global and local information;

[0093] Step 3-4) Train the adaptive network: At this time, fix the CLIP model trained in step 2) and train the adaptive network. The loss function of the adaptive network is still set to:

[0094]

[0095] At this time, because the text features incorporate weighted local knowledge, the calculation of the loss function can ultimately obtain the desired local weights after optimization.

[0096] The present invention uses the CLIP model and adaptive network obtained in the training phase to perform image and text retrieval on the test data set; by adjusting the proportion parameters of local and global information, the optimal retrieval effect is achieved, and the retrieval indicator is the recall rate.

[0097] The step 4) uses the remote sensing image and text data set to perform a retrieval test on the downstream image and text: the specific process includes:

[0098] Step 4-1) Obtain the test data set obtained in step 1);

[0099] Step 4-2) uses the text feature extractor obtained in step 2) to extract the global and local text features of the test data set, according to the global information ratio and local information ratio obtained in step 3), that is, α and β 11 ,β 12 ,β 21 ,β22 , generate text features that fuse global and local information in the test data set; use the image feature extractor obtained in step 2) to extract global image features of the test data set;

[0100] Step 4-3) calculates the image-text retrieval results: first, based on the text features that integrate global and local information, the matching strategy between the image and the text is updated, and the similarity between the image and the text is recalculated; secondly, combined with the semantic information described in the local text, the improved text feature calculation method in step 3-2) and the improved similarity calculation method in step 3-3) are used to more accurately evaluate the similarity between the image and the text, thereby obtaining a more accurate image-text retrieval result.

[0101] The invention discloses a remote sensing image and text retrieval method based on a large model adaptively supplementing local knowledge. The multimodal large language model is used to generate corresponding descriptions for remote sensing images, which completes the text modality of remote sensing and is beneficial to remote sensing image cognition and remote sensing model training. It not only proposes to use local knowledge to supplement remote sensing image and text retrieval, providing richer semantics for remote sensing image and text retrieval, but also proposes to adaptively supplement remote sensing image knowledge based on an adaptive network, providing unique local information injection parameters for each remote sensing data set. The method of the invention can better reflect the characteristics of each local area in the image, thereby improving the accuracy and robustness of the image and text retrieval system.

Claims

1. A remote sensing image and text retrieval method based on adaptive local knowledge filling, characterized by include: Step 1) Create a remote sensing image and text database; Specifically, it includes: collecting a number of remote sensing images from a number of remote sensing data sources to form a remote sensing image and text data set; Divide several remote sensing images in the remote sensing image and text dataset into a training dataset, a verification dataset, and a test dataset according to a certain ratio; Step 2) Training the CLIP model with remote sensing image and text datasets; Step 3) training the adaptive network using remote sensing image and text datasets; Step 4) Use the remote sensing image and text dataset to perform retrieval test on downstream images and text.

2. A remote sensing image and text retrieval method based on adaptive local knowledge filling according to claim 1, characterized in that The step 2) uses the remote sensing image and text dataset to train the CLIP model, specifically comprising: using the training dataset obtained in step 1) to train the visual language contrast learning model, and finally obtaining the trained visual language contrast learning model, and obtaining an image feature extractor and a text feature extractor.

3. A remote sensing image and text retrieval method based on adaptive local knowledge filling according to claim 2, characterized in that The step 3) uses the remote sensing image and text dataset to train the adaptive network, specifically comprising: first, inputting the verification dataset obtained in step 1) into the image feature extractor and the text feature extractor obtained in step 2) to extract the splicing features of the image and text; then, using the splicing features of the image and text to train the adaptive network, after 5 layers of fully connected layers, outputting five parameters representing the global and local proportions, and finally obtaining an optimized adaptive network and global and local proportion parameters.

4. A remote sensing image and text retrieval method based on adaptive local knowledge filling according to claim 3, characterized in that The step 4) uses the remote sensing image and text dataset to perform a retrieval test on the downstream images and texts, specifically comprising: first, inputting the test dataset obtained in step 1) into the image feature extractor and text feature extractor obtained in step 2) to obtain the test set image features and text features; then, inputting the test set image features and text features into the optimized adaptive network according to step 3), and adjusting the test set data according to the global and local proportion parameters.

5. The remote sensing image and text retrieval method based on adaptive local knowledge filling according to claim 1 is characterized by: The specific process of step 1) creating a remote sensing image and text database includes: Step 1-1) Collect a certain number of remote sensing images (I1, I2, ..., I N ); Step 1-2) Divide each remote sensing image into four local images: "upper left corner", "lower left corner", "upper right corner", and "lower right corner", denoted as Then the four local images are input into the multimodal large language model respectively, and specific prompts are input for the remote sensing images, and finally four local texts are obtained. Step 1-3) Input the four local texts corresponding to each remote sensing image into the multimodal large language model, input the prompts that integrate the local knowledge to generate the overall image description, and finally make its reference image output the overall text T of the entire image of each remote sensing image i .

6. The remote sensing image and text retrieval method based on adaptive local knowledge filling according to claim 1 is characterized by: The specific process of training the CLIP model with the remote sensing image and text dataset in step 2) includes: Step 2-1) Obtain the training data set in step 1) and record it as Where N represents the number of samples in the training data set; Step 2-2) Use F(I) and G(T) to represent the image features and text features extracted by the image feature extractor and text feature extractor respectively: The above means {global image features, global text features, [four local image features], [four local text features]} of the i-th sample in the training data set; Step 2-3) The image and text similarity of the i-th sample in the training data set is calculated using a common similarity calculation method. The common similarity calculation method is preferably a cosine calculation method cos(): sim(I i ,T i )=cos(F(I i ),G(T i )) Where sim(I i ,T i ) represents the image and text similarity of the i-th sample in the training data set, cos() represents the cosine calculation method, cos(F(I i ),G(T i )) represents the cosine similarity between the image features and text features of the i-th sample in the training data set; Step 2-4) Calculate the loss function of the CLIP model as follows: Among them, L represents the loss function, N represents the sample size of the training data set, and the ultimate goal is to bring the positive sample pairs closer and the negative sample pairs farther apart during the training process through contrastive learning.

7. The remote sensing image and text retrieval method based on adaptive local knowledge filling according to claim 1 is characterized by: The step 3) training the adaptive network with the remote sensing image and text data set specifically includes: Step 3-1) Obtain the validation data set obtained in step 1) and also count it as Step 3-2) F(I) and G(T) represent the image features and text features extracted by the image feature extractor and text feature extractor respectively, define the adaptive network, and calculate the text features that integrate the local information: Among them, α represents the proportion of global information, β 11 ,β 12 ,β 21 ,β 22 Respectively represent the proportion of local information, and (α, β 11 ,β 12 ,β 21 ,β 22 )The sum of all parameters is 1; Step 3-3) The expression for calculating the image and text similarity of the i-th sample in the validation dataset can be expressed as: sim(I i ,T i ) ‘ =cos(F(I i ),G(T i ) ‘ ) Where sim(I i ,T i ) ‘ represents the image and text similarity that integrates global and local information, cos() represents the cosine calculation method, cos(F(I i ), G(T i ) ‘ ) represents the cosine similarity of image features and text features that integrate global and local information; Step 3-4) Train the adaptive network: At this time, fix the CLIP model trained in step 2) and train the adaptive network. The loss function of the adaptive network is still set to:

8. The remote sensing image and text retrieval method based on adaptive local knowledge filling according to claim 1 is characterized by: The specific process of step 4) using the remote sensing image and text data set to perform a retrieval test on the downstream image and text includes: Step 4-1) Obtain the test data set obtained in step 1); Step 4-2) uses the text feature extractor obtained in step 2) to extract the global and local text features of the test data set, according to the global information ratio and local information ratio obtained in step 3), that is, α and β 11 ,β 12 ,β 21 ,β 22 , generate text features that fuse global and local information in the test data set; use the image feature extractor obtained in step 2) to extract global image features of the test data set; Step 4-3) Calculate the image and text retrieval results.

9. A remote sensing image and text retrieval method based on adaptive local knowledge filling according to claim 8, characterized in that The step 4-3) calculates the image-text retrieval results, specifically including: first, based on the text features that integrate global and local information, updating the matching strategy between the image and the text, and recalculating the similarity between the image and the text; second, combining the semantic information described in the local text, using the improved text feature calculation method in step 3-2) and the improved similarity calculation method in step 3-3) to more accurately evaluate the similarity between the image and the text, thereby obtaining a more accurate image-text retrieval result.

10. The remote sensing image and text retrieval method based on adaptive local knowledge filling according to claim 1, characterized in that In the step 1), a plurality of remote sensing images in the remote sensing image and text data set are divided into a training data set, a verification data set, and a test data set according to a certain ratio, specifically, a plurality of remote sensing images in the remote sensing image and text data set are divided into a training data set, a verification data set, and a test data set according to a ratio of 6:2:2.

Images