Method, device and equipment for constructing remote sensing image-text data set and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a remote sensing image-text data set construction method, device and equipment and a storage medium, and the method comprises the steps: obtaining an initial remote sensing image-text data set which comprises a remote sensing image and a remote sensing image description text; performing image reconstruction based on the text description to obtain a scene reconstruction image, and evaluating the quality of the text description to obtain a text evaluation index; calculating the image-text matching degree and the image-text contrast of the remote sensing image and the text description, and calculating the image matching degree of the remote sensing image and the scene reconstruction image; and based on evaluation of at least one of the text evaluation index, the image-text matching degree, the image-text contrast ratio and the image matching degree, obtaining an evaluation result for evaluating the initial remote sensing image-text data set, and based on the evaluation result, constructing a target remote sensing image-text data set. The matching degree of the text description and the original remote sensing image can be accurately measured, the quality of the remote sensing image-text data set is improved, and the problem that multi-modal data alignment is difficult is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for constructing a remote sensing image and text data set. Background Art

[0002] Remote sensing images contain rich geographic environment information. Generating accurate natural language descriptions for remote sensing images and constructing multimodal remote sensing image and text datasets are crucial for training and fine-tuning vertical domain generation models. The matching and fusion of remote sensing images and text descriptions is a complex problem. Related technologies lack effective methods for multimodal data feature alignment, making it difficult to achieve efficient matching and evaluation of image and text data, resulting in low quality of the constructed datasets and affecting the training effect of the model. Summary of the invention

[0003] In view of this, the present invention provides a method, device, equipment and storage medium for constructing a remote sensing image and text dataset, so as to solve the problem that there is a lack of effective methods for multimodal data feature alignment, which makes it difficult to achieve efficient matching and evaluation of image and text data, resulting in low quality of the constructed dataset and affecting the training effect of the model.

[0004] In a first aspect, the present invention provides a method for constructing a remote sensing image and text dataset, the method comprising: obtaining an initial remote sensing image and text dataset, the initial remote sensing image and text dataset comprising a remote sensing image and a text description describing the remote sensing image; performing image reconstruction based on the text description to obtain a scene reconstructed image, evaluating the quality of the text description to obtain a text evaluation index; calculating the image and text matching degree and image and text contrast between the remote sensing image and the text description, and calculating the image matching degree between the remote sensing image and the scene reconstructed image; obtaining an evaluation result of evaluating the initial remote sensing image and text dataset based on an evaluation of at least one of the text evaluation index, the image and text matching degree, the image and text contrast and the image matching degree, and constructing a target remote sensing image and text dataset based on the evaluation result.

[0005] In an optional implementation, the evaluation result of evaluating the initial remote sensing image and text data set based on the evaluation of the image and text matching degree includes: calculating the image and text matching degree index evaluation result E of evaluating the initial remote sensing image and text data set by the following formula: itm ,

[0006]

[0007] Among them, E itm ∈(0,1), α is the first sensitivity parameter, itm pos is the positive image-text matching degree, itm neg is the negative image-text matching degree, [itmpos , itm neg ]∈R 2 The image-text matching degree includes the positive image-text matching degree and the negative image-text matching degree.

[0008] In an optional implementation, the evaluation result of evaluating the initial remote sensing image and text data set based on the evaluation of the image and text contrast comprises: calculating the image and text contrast index evaluation result E of evaluating the initial remote sensing image and text data set by the following formula: itc ,

[0009] E itc =e -β×arccos(itc)

[0010] Among them, E itc ∈(0, 1], β is the second sensitivity parameter, and itc is the image-text contrast.

[0011] In an optional implementation, the evaluation result of evaluating the initial remote sensing image and text data set based on the evaluation of the image matching degree includes: calculating the image matching degree index evaluation result E of evaluating the initial remote sensing image and text data set by the following formula: reg ,

[0012]

[0013] Among them, E reg ∈(0, 1], γ is the third sensitivity parameter, C regen is the image matching degree.

[0014] In an optional implementation, the evaluation result of evaluating the initial remote sensing image and text data set is obtained based on the evaluation of the text evaluation index, including: calculating the text evaluation index evaluation result E of evaluating the initial remote sensing image and text data set by the following formula: cap ,

[0015]

[0016] Among them, E cap ∈(0,1],l token is the length of the encoding sequence described by the text, l ref is the reference encoding sequence length hyperparameter, and decay is the numerical decay hyperparameter.

[0017] In an optional implementation, the evaluation based on the text evaluation index, the image-text matching degree, the image-text contrast and the image matching degree, obtaining the evaluation result of evaluating the initial remote sensing image-text data set, includes: calculating the image-text matching degree index evaluation result E of evaluating the initial remote sensing image-text data set by the following formula: itm ,

[0018]

[0019] Among them, E itm ∈(0,1), α is the first sensitivity parameter, itm pos is the positive image-text matching degree, itm neg is the negative image-text matching degree, [itm pos , itm neg ]∈R 2 , the image-text matching degree includes the positive image-text matching degree and the negative image-text matching degree;

[0020] The image-text contrast index evaluation result E of the initial remote sensing image-text data set is calculated by the following formula: itc ,

[0021] E itc =e -β×arccos(itc)

[0022] Among them, E itc ∈(0, 1], β is the second sensitivity parameter, itc is the image-text contrast;

[0023] The image matching index evaluation result E of the initial remote sensing image and text data set is calculated by the following formula: reg ,

[0024]

[0025] Among them, E reg ∈(0, 1], γ is the third sensitivity parameter, C regen is the image matching degree;

[0026] The text evaluation index evaluation result E of the initial remote sensing image and text data set is calculated by the following formula: cap ,

[0027]

[0028] Among them, E cap ∈(0,1],l token is the length of the encoding sequence described by the text, l ref is the reference encoding sequence length hyperparameter, decay is the numerical decay hyperparameter;

[0029] The comprehensive evaluation result E of the initial remote sensing image and text data set is calculated by the following formula: total ,

[0030] E total =[E itm , E itc , E cap , E reg ]·V weight

[0031] Among them, V weight is a non-negative weight vector.

[0032] In an optional embodiment, the constructing of a target remote sensing image and text dataset based on the evaluation result includes: sorting the initial remote sensing image and text dataset having the evaluation result based on a second target evaluation index, the second target evaluation index including at least one of the following: the image and text matching index evaluation result, the image and text comparison index evaluation result, the image matching index evaluation result, the text evaluation index evaluation result and the comprehensive evaluation result; obtaining a target remote sensing image and a target text description that meet a first preset requirement in the sorting result; and constructing the target remote sensing image and text dataset based on the target remote sensing image and the target text description.

[0033] In an optional embodiment, the aforementioned method for constructing a remote sensing image and text dataset also includes adjusting and training an image generation model for performing the image reconstruction based on the target remote sensing image and text dataset: adjusting and training the image generation model based on different dataset categories in the target remote sensing image and text dataset.

[0034] In an optional embodiment, the aforementioned method for constructing a remote sensing image and text dataset also includes: performing a baseline test on the image generation model after the adjustment training to obtain initial performance parameters, and judging whether the initial performance parameters meet the second preset requirements; when the initial performance parameters do not meet the second preset requirements, obtaining the update weights of the image generation model after the adjustment training, and reconstructing the image based on the updated weights to obtain a reconstructed image of the current scene; calculating the current image matching degree between the current scene reconstructed image and the original remote sensing image corresponding to the current scene reconstructed image, and obtaining a current image matching degree index based on an evaluation of the current image matching degree, and constructing a second remote sensing image and text dataset based on the current image matching degree index; performing a secondary adjustment training on the image generation model after the adjustment training based on the second remote sensing image and text dataset to obtain an image generation model with secondary updated weights; performing the baseline test on the image generation model with secondary updated weights to obtain a second performance parameter, and when the second performance parameter does not meet the second preset requirement, repeating the above steps until the second performance parameter meets the second preset requirement or the number of updates reaches the preset number.

[0035] In a second aspect, the present invention provides a device for constructing a remote sensing image and text dataset, the device comprising: an acquisition module, used to acquire an initial remote sensing image and text dataset, the initial remote sensing image and text dataset comprising a remote sensing image and a text description describing the remote sensing image; a calculation module, used to reconstruct an image based on the text description to obtain a scene reconstructed image, evaluate the quality of the text description, and obtain a text evaluation index; calculate the image and text matching degree and image and text contrast between the remote sensing image and the text description, and calculate the image matching degree between the remote sensing image and the scene reconstructed image; a construction module, used to obtain an evaluation result of evaluating the initial remote sensing image and text dataset based on an evaluation of at least one of the text evaluation index, the image and text matching degree, the image and text contrast, and the image matching degree, and construct a target remote sensing image and text dataset based on the evaluation result.

[0036] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for constructing a remote sensing graphic data set according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for constructing a remote sensing graphic data set according to the first aspect or any corresponding embodiment thereof.

[0038] The method, apparatus, device and storage medium for constructing a remote sensing image and text dataset provided in this embodiment can accurately measure the degree of matching between the text description and the original remote sensing image based on the evaluation of text evaluation indicators, image and text matching, image and text contrast and image matching, thereby improving the quality of the target remote sensing image and text dataset, improving the training effect of the model and overcoming the problem of difficulty in aligning multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of a process for constructing a remote sensing image and text data set provided by an embodiment of the present invention is shown;

[0041] Figure 2 A schematic diagram of the process of adjusting and training an image generation model for image reconstruction in an embodiment of the present invention is shown;

[0042] Figure 3 A schematic flow chart of another method for constructing a remote sensing image and text data set in an embodiment of the present invention is shown;

[0043] Figure 4 A schematic diagram of the structure of a device for constructing a remote sensing image and text data set provided by an embodiment of the present invention is shown;

[0044] Figure 5 A schematic diagram showing the hardware structure of a computer device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0046] The following is an introduction to the relevant concepts involved in the embodiments of the present invention.

[0047] Multimodal Large Models (MLMs) are AI models that can process and understand data in multiple modalities (such as images, text, or audio). These models can achieve richer and more complex tasks by fusing data from different modalities, such as image-text matching, image description generation, or cross-modal retrieval. Multimodal large models are usually based on deep learning technology, pre-trained using large datasets, and fine-tuned in downstream tasks to improve the performance of specific tasks.

[0048] Visual Question Answering (VQA) is an important multimodal task that involves answering natural language questions based on image content. VQA models need to understand both image and text information and establish associations between the two to generate accurate answers. The core of VQA technology lies in designing model architectures that can effectively integrate visual and language features, such as attention mechanisms and joint embedding representations. In practical applications, these technologies can help improve the ability to understand image content and human-computer interaction.

[0049] Adjustment training of large text-image generation models refers to the process of further training on the basis of pre-trained multimodal models through data from specific fields to adapt to specific task requirements. Adjustment training of text-image generation models usually involves two aspects: first, adjustment training through data sets in specific fields so that the model can generate images or texts that are more in line with the style of a specific field; second, optimizing the generation effect of the model so that it can generate high-quality output in specific tasks. This process requires high-quality data sets and effective training methods to ensure that the adjusted and trained model has strong generalization ability and application value.

[0050] There are several problems with the construction of remote sensing image and text datasets and model adjustment and training in related technologies:

[0051] First, data acquisition and annotation are costly. The acquisition and annotation of remote sensing image data requires a lot of manpower and resource investment. Unlike general image annotation tasks, remote sensing images often cover a wide geographical area and contain a variety of geographical features. Remote sensing image annotation requires professional domain knowledge and requires professional annotation by domain experts. This high-cost annotation process limits the rapid construction of high-quality datasets. The open image description generation model in the related technology lacks professional evaluation methods in the remote sensing field, resulting in the inability to effectively screen the quality of the generated text, which in turn causes the generated image-text pairs to be noisy. The noisy remote sensing image-text dataset will further affect the training and adjustment of multimodal large models. Training performance.

[0052] Secondly, it is difficult to align multimodal data. The matching and fusion of remote sensing images and text descriptions is a complex problem. The relevant technologies lack effective methods for aligning multimodal data features, making it difficult to achieve efficient matching and evaluation of image and text data, resulting in low quality of constructed data sets and affecting the training effect of the model. The lack of effective evaluation methods leads to noise in the generated image and text pairs, affecting the purity of the data set and the adjustment and training effect of the model.

[0053] Thirdly, it is difficult to adjust and train the remote sensing image generation model. Due to the lack of high-quality multimodal remote sensing image and text datasets, the adjustment and training of the base text and image model in the remote sensing field in related technologies is difficult to generate images that conform to the remote sensing style. The text input intent is difficult to align accurately, resulting in low quality of the generated remote sensing images, which affects the model reasoning and generation effects.

[0054] According to an embodiment of the present invention, a method embodiment for constructing a remote sensing graphic data set is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0055] In this embodiment, a method for constructing a remote sensing image and text data set is provided, which can be used in a terminal with an operating system, such as a mobile phone, a server, a tablet computer, a laptop computer, or a desktop computer. Figure 1 A schematic diagram of a process for constructing a remote sensing image and text data set provided by an embodiment of the present invention is shown. Figure 1 As shown, the process includes the following steps:

[0056] Step S101, obtaining an initial remote sensing image and text data set, the initial remote sensing image and text data set including a remote sensing image and a text description describing the remote sensing image.

[0057] In this step, the remote sensing images can be classified by type, for example, by city, farmland or natural landscape, to facilitate subsequent processing and use.

[0058] By designing guiding words, based on the pre-trained multimodal model of images and texts, the geographical environment information or features of objects in the aforementioned remote sensing images can be analyzed to generate a text description of the aforementioned remote sensing images, that is, a continuous natural language description. The remote sensing images can be described in batches using consistent guiding words through the method of prompt word guidance to generate corresponding text descriptions. The text description may include geographical environment information, features of objects, meteorological conditions, spatial distribution of elements, etc. in the image to illustrate the layout of remote sensing image elements.

[0059] In this way, through the pre-trained large multimodal model of images and texts, text descriptions of remote sensing images can be automatically generated, which significantly reduces the cost of remote sensing image text description annotation and data acquisition of remote sensing images and text description image and text pairs, and solves the problem of high data acquisition and annotation costs in related technologies. In addition, remote sensing images and text description image and text pairs can also be efficiently constructed.

[0060] The image-text pairs of remote sensing images can be stored in a lightweight data exchange format (JavaScript Object Notation, referred to as json).

[0061] The guide words used to guide the generation of a text description of a remote sensing image may be “Describe this remote sensing image in detail, highlighting the spatial distribution and arrangement of its elements. The description should be between 40 and 80 words”.

[0062] The pre-trained large multimodal model of images and texts can be the visual language model CogVLM, CogVLM2, Contrastive Language-Image Pre-training (CLIP), ViT-B / 32 model or ViT-L / 14 model. CogVLM2 is a large collaborative visual-language model designed for understanding and generating text descriptions related to images. CogVLM2 can extract and express complex information in images, such as geographical environment features, landform types or meteorological conditions, by pre-training on a large amount of images and related text data. CLIP has the ability of multimodal learning, can understand information of two different modalities, image and text, at the same time, and establish a connection between images and text. The ViT-B / 32 model is a visual model based on a transformer neural network (transformer), which can encode the input image into a feature vector. The ViT-L / 14 model has a larger number of parameters than the ViT-B / 32 model. It can also be other models that can quantify the output of image-text matching.

[0063] Remote sensing images can be obtained based on public network datasets, or different types of remote sensing image data can be captured using sensors carried by satellites or drones. Specifically, remote sensing image annotation generation datasets (RSICD for short); or WHU-RS19, 19 categories of remote sensing images obtained from satellite images; or NWPU_VHR_10, a dataset with ten categories of geospatial object detection, including 800 ultra-high resolution remote sensing images; or Remote Sensing Generative Pretrained Model (RSgpt for short) dataset; or Sydney dataset (Sydney_caption) contains 613 images belonging to seven categories; or Remote Sensing Image Dataset (SIRI-WHU-earth) contains 12 categories of scenes, mainly covering urban areas in China, or other methods to obtain remote sensing images.

[0064] Step S102, reconstructing an image based on the text description to obtain a scene reconstructed image, evaluating the quality of the text description to obtain a text evaluation index.

[0065] In this step, a general or domain data-fine-tuned text-based multimodal model can be used to reconstruct the image based on the text description. The prompt words guiding the image reconstruction may include a guiding prefix, the text description in step S101, and a constrained suffix. The guiding prefix can be used to generate background settings such as style. The constrained suffix can be used to explain negative factors, or to constrain the distribution of specific elements.

[0066] The text-based multimodal model can be an image generation model (Stable Diffusion XL, SDXL for short), which includes a two-step generation process, basic image and image refinement, and can accurately understand complex scenes and prompt words. It can also be a DALL·E series model that can parse and understand complex descriptions in natural language and convert them into corresponding images. Or other multimodal models that can generate images based on text descriptions.

[0067] The text evaluation index can be obtained through a pre-trained large image-text multimodal model or a calculation script. Specifically, based on the text encoder (Tokenizer) in the pre-trained large image-text multimodal model, the text description in step S101 can be segmented to obtain a word (token) list to obtain the sequence length l of the text description. token , l token ∈N, the length of the sequence described by the text can be l tokenAs a text evaluation indicator, it can also be serialized based on the GPT series model, or serialized based on the Bidirectional Encoder Representations from Transformers (BERT) model to obtain a text evaluation indicator.

[0068] Step S103, calculating the image-text matching degree and image-text contrast between the remote sensing image and the text description, and calculating the image matching degree between the remote sensing image and the scene reconstruction image.

[0069] In this step, the image-text matching degree can be obtained through a pre-trained image-text multimodal model or a calculation script. Specifically, the image-text matching degree (Image-Text Matching loss, referred to as ITM) can be calculated through a cross-modal pre-training model (Bootstrapping Language-Image Pre-training 2, referred to as BLIP-2), and the result can be obtained as a binary list (logist) [itm pos , itm neg ]∈R 2 , itm pos is the positive image-text matching degree, itm neg It is the negative image-text matching degree.

[0070] Optionally, the similarity between the remote sensing image and the text description features can be calculated by a pre-trained large multimodal model of images and texts, such as cosine similarity, to evaluate the positive image-text matching. The negative image-text matching is obtained by calculating the similarity between the remote sensing image and the text description image-text pairs that do not match the positive sample.

[0071] Based on the pre-trained image-text multimodal large model, the contrastive learning loss (Image-Text Contrastive loss, referred to as ITC) can be calculated to obtain the image-text contrast of the remote sensing image and the text description. The text description can be input into the text encoder, and the remote sensing image can be input into the image encoder. The cosine similarity of the text description and the remote sensing image can be calculated in the same feature space to obtain the image-text contrast itc, itc∈[-1,1].

[0072] Based on the pre-trained multimodal model of images and texts, the cosine similarity of the remote sensing image and the scene reconstruction image can be calculated and used as the image matching degree C regen , C regen ∈[-1, 1].

[0073] Step S104, based on the evaluation of at least one of the text evaluation index, the image-text matching degree, the image-text contrast and the image matching degree, an evaluation result of the initial remote sensing image-text data set is obtained, and a target remote sensing image-text data set is constructed based on the evaluation result.

[0074] In this step, based on the evaluation of text evaluation indicators, image-text matching degree, image-text contrast, image matching degree, text evaluation indicators and image-text matching degree, text evaluation indicators and image-text contrast, text evaluation indicators and image matching degree, image-text matching degree and image-text contrast, image-text matching degree and image matching degree, image-text contrast and image matching degree, text evaluation indicators and image-text matching degree and image-text contrast, text evaluation indicators and image-text matching degree and image-text contrast, text evaluation indicators and image-text matching degree and image matching degree, text evaluation indicators and image-text matching degree and image matching degree, image-text matching degree and image-text contrast and image matching degree, the evaluation result of evaluating the initial remote sensing image-text dataset can be obtained.

[0075] In this way, remote sensing images and text description image-text pairs can be evaluated based on multiple types of indicator data, and each evaluation indicator has a different sensitivity range, which can reflect the quality of text description separately. Multiple types of indicator data can also be normalized into single data, so that the performance of remote sensing images and text description image-text pairs can be evaluated more comprehensively.

[0076] The method for constructing a remote sensing image and text dataset provided in this embodiment can accurately measure the degree of matching between text descriptions and original remote sensing images based on the evaluation of text evaluation indicators, image and text matching, image and text contrast, and image matching, thereby improving the quality of the target remote sensing image and text dataset, improving the training effect of the model, and overcoming the problem of difficulty in aligning multimodal data.

[0077] In an optional implementation, based on the evaluation of the image-text matching degree, an evaluation result of evaluating the initial remote sensing image-text data set is obtained, including: calculating the image-text matching degree index evaluation result E of evaluating the initial remote sensing image-text data set by the following formula: itm ,

[0078]

[0079] Among them, E itm ∈(0,1), α is the first sensitivity parameter, itm pos is the positive image-text matching degree, itm neg is the negative image-text matching degree, [itm pos , itm neg ]∈R 2,The image-text matching degree includes ,positive image-text matching degree and negative image-text matching degree.

[0080] In this embodiment, tanh() is a hyperbolic tangent function, which is a deep learning activation function that can map any real value monotonically increasing to a value in the interval (-1, 1). The image-text matching index evaluation result E itm The closer the value is to 1, the better the evaluation value of the image-text matching index is. In addition, the evaluation value of the image-text matching index is pos -itm neg =0, that is, itm pos and itm neg It is more sensitive near two logical values.

[0081] In this way, by calculating the evaluation results of the image-text matching index, the quality of the remote sensing image and text description image-text pair can be effectively evaluated in terms of image-text matching.

[0082] In some optional implementations, based on the evaluation of the image-text contrast, obtaining the evaluation result of the initial remote sensing image-text data set includes: calculating the image-text contrast index evaluation result E of the initial remote sensing image-text data set by the following formula: itc ,

[0083] E itc =e -β×arccos(itc)

[0084] Among them, E itc ∈(0, 1], β is the second sensitivity parameter, and itc is the image-text contrast.

[0085] In this embodiment, the image-text contrast index itc is converted into the angle θ of the image-text coding in the feature space by the arccosine function arccos(()), which can be expressed by the following formula: θ=arccos(itc), the smaller the angle θ is, the greater the angle E is. itc The closer the value is to 1, the better the alignment effect of the image and text modality is. At the same time, the image and text contrast index evaluation result E itc It is more sensitive to values ​​with smaller angles θ, and can better distinguish image and text contrasts with higher scores.

[0086] In this way, by calculating the evaluation results of the image-text contrast index, the quality of the remote sensing image and the text description image-text pair can be effectively evaluated in terms of image-text contrast.

[0087] In some optional implementations, based on the evaluation of the image matching degree, an evaluation result of evaluating the initial remote sensing image and text data set is obtained, including: calculating the image matching degree index evaluation result E of the initial remote sensing image and text data set by the following formula: reg ,

[0088]

[0089] Among them, E reg ∈(0, 1], γ is the third sensitivity parameter, C regen is the image matching degree.

[0090] In this embodiment, the image matching index evaluation result F reg Calculation and evaluation results of image-text contrast index E itc The evaluation method is similar to that of reg The inputs for calculating cosine similarity are all image modalities. Similarly, the image matching index evaluation result E reg When the angle θ is small, the sensitivity is high, and the reconstructed image with high similarity to the original remote sensing image can be better distinguished.

[0091] In this way, by calculating the evaluation results of the image matching evaluation index, the quality of the remote sensing image and the text description image-text pair can be effectively evaluated in terms of image matching, and the reconstructed image with a high similarity to the original remote sensing image can be better distinguished.

[0092] In some optional implementations, based on the evaluation of the text evaluation index, an evaluation result of evaluating the initial remote sensing image and text data set is obtained, including: calculating the text evaluation index evaluation result E of evaluating the initial remote sensing image and text data set by the following formula: cap ,

[0093]

[0094] Among them, Ecap∈(0,1],l token is the length of the encoding (tokens) sequence of the text description, l ref is the reference encoding (tokens) sequence length hyperparameter, and decay is the numerical decay hyperparameter.

[0095] In this embodiment, l ref The value of can be the upper limit of the token length acceptable to the SDXL model encoder. Too short text descriptions will lead to insufficient scene description information, while too long text description sentences will lead to sparse key information and truncation of input during text description generation. decay can be used to smooth the evaluation results of text evaluation indicators E cap .

[0096] In this way, by calculating the evaluation results of the text evaluation index, the quality of the remote sensing image and text description image-text pair can be effectively evaluated in terms of text description.

[0097] In some optional implementations, the comprehensive evaluation result E of the initial remote sensing image and text data set is calculated by the following formula: total ,

[0098] E total =[E itm , E itc , E cap , E reg ]·V weight

[0099] Among them, V weight is a non-negative weight vector.

[0100] In this way, the comprehensive evaluation result E total Multiple types of indicator data can be normalized into single data, which can provide a more comprehensive performance evaluation of remote sensing images and text description image pairs.

[0101] In some optional embodiments, a target remote sensing image and text dataset is constructed based on the evaluation results, including: sorting the initial remote sensing image and text dataset with the evaluation results based on a second target evaluation index, the second target evaluation index including: at least one of an image and text matching index evaluation result, an image and text comparison index evaluation result, an image matching index evaluation result, a text evaluation index evaluation result and a comprehensive evaluation result; obtaining a target remote sensing image and a target text description that meet a first preset requirement in the sorting results; and constructing a target remote sensing image and text dataset based on the target remote sensing image and the target text description.

[0102] In this embodiment, the initial remote sensing image and text data set includes image and text pairs of remote sensing images and text descriptions. After obtaining the evaluation results of the initial remote sensing image and text data set in the above manner, the evaluation results are saved in a json file of image and text pairs of remote sensing images and text descriptions. In this way, the intermediate remote sensing image data set includes image and text pairs of remote sensing images and text descriptions, and evaluation results corresponding to the above image and text pairs. The evaluation results include at least one of the following: image and text matching index evaluation result E itm , Image-text contrast index evaluation results E itc , Image matching index evaluation results E reg , Text evaluation index evaluation results E cap And the comprehensive evaluation results E total .

[0103] After obtaining the intermediate remote sensing image dataset, the intermediate remote sensing image dataset can be sorted based on the second objective evaluation index to obtain the sorting result. The first preset requirement can be the data cleaning ratio r c , the amount of cleaned data n can be calculated by the following formula clean ,

[0104] nclean =[r c ×max{ids}]

[0105] Among them, ids is the index set of the intermediate remote sensing image dataset, [] is the rounding function, and the amount of cleaned data n is calculated according to the above formula clean , you can delete the intermediate remote sensing image data set after n clean The image-text pairs of remote sensing images and text descriptions and the evaluation results corresponding to the image-text pairs are used to obtain the cleaned intermediate remote sensing images, namely the target remote sensing images and target text descriptions. The evaluation indicators used to evaluate the initial remote sensing image-text dataset can be the same as or different from the indicators selected for the second target evaluation indicators. Each evaluation indicator in the second target evaluation indicator can be used independently as a basis for sorting, and sorting and cleaning can be performed based on any evaluation indicator. The specific selection can be based on the data target of the downstream task.

[0106] The first preset requirement may also be to delete data with obvious grammatical errors, unclear descriptions, or data with inconsistent text descriptions and remote sensing image content. The first preset requirement may also be to delete duplicate remote sensing images and text descriptions. It is also possible to review the cleaned remote sensing images and text descriptions by manual review by designing a questionnaire to further filter out unqualified data. This can further improve the quality of the target remote sensing image and text dataset, and also ensure the uniqueness and diversity of the target remote sensing image and text dataset.

[0107] In this way, by performing data cleaning on the intermediate remote sensing image dataset through any one of the evaluation indicators in the second target evaluation indicators, noise data and low-quality remote sensing images and text description image-text pairs can be removed, ensuring the high quality of the dataset, and constructing a high-quality remote sensing image and text description multimodal dataset. At the same time, it also provides reliable data support for the training and adjustment of the pre-trained image-text multimodal large model, solving the problem of noisy datasets affecting model performance in related technologies.

[0108] In some optional embodiments, the aforementioned method for constructing a remote sensing image and text dataset also includes adjusting and training an image generation model for image reconstruction based on the target remote sensing image and text dataset: adjusting and training the image generation model based on different dataset categories in the target remote sensing image and text dataset.

[0109] Figure 2 FIG. 2 is a schematic diagram showing a flow chart of adjusting and training an image generation model for image reconstruction in an embodiment of the present invention. Figure 2 As shown, in this embodiment, the image generation model can be adjusted and trained by constructing a high-quality multimodal target remote sensing image and text dataset, which can also be called fine-tuning training. The image generation model adjustment training for image reconstruction includes:

[0110] Step S201, load a pre-trained image generation model. The pre-trained image generation model is an image generation model for image reconstruction. By loading the pre-trained image generation model, a basic model framework and parameters for adjusting the training of the image generation model are constructed. Initialize the basic model framework, including loading pre-trained weights and configuring the basic environment of the model. The pre-trained image generation model can be SDXL, and the framework for adjusting the training can be sd-scripts, which is a fine-tuning toolbox for the Stable Diffusion model.

[0111] Step S202, importing the target remote sensing image and text dataset. The target remote sensing image and text dataset is a multimodal remote sensing image and text dataset that has undergone data cleaning. The target remote sensing image and text dataset is preprocessed, including resizing the remote sensing image and standardizing the text description. According to different categories of remote sensing images, the pre-trained image generation model is trained to adjust the categories, so as to learn the unique features and styles of various remote sensing images.

[0112] Step S203: construct and adjust training parameters. The training parameters can be constructed and adjusted through LoRA or DreamBooth.

[0113] Among them, LoRA can reduce the number of parameters through low-rank decomposition, reduce the computational cost and storage requirements of adjusting the training of the pre-trained image generation model, while maintaining or improving the performance of the pre-trained image generation model on specific tasks. This adjustment training method is achieved by introducing a low-rank structure in a specific layer of the pre-trained image generation model, aiming to optimize the response speed and generation effect of the pre-trained image generation model, making it more suitable for specific image reconstruction tasks. DreamBooth can inject custom themes into the pre-trained image generation model through adjustment training, so that the pre-trained image generation model can learn rare or personalized image features.

[0114] You can set an appropriate adjustment training learning rate, that is, fine-tune the learning rate, to retain the existing knowledge structure of the pre-trained image generation model. You can set the batch size (Batch Size) and training cycle (Epochs) according to the specific hardware environment and data set size to achieve the best training efficiency and effect.

[0115] Adjustment training is the process of optimizing the pre-trained image generation model for specific downstream tasks. By adjusting the training process, the pre-trained image generation model can deeply understand the details of specific tasks at extremely low data and hardware costs, and improve the ability to generate and understand modal semantic information for downstream tasks.

[0116] Step S204, adjust the monitoring and optimization of the training process. The training parameters can be adaptively adjusted in response to performance changes during the training process through real-time dynamic adjustment strategies, such as learning rate decay. Real-time data monitoring tools can be used to monitor key performance indicators, such as loss function values ​​or accuracy rates. Parameters can be adjusted and optimized in a timely manner based on the monitoring results to ensure the efficiency and effectiveness of the training process. Specifically, the real-time data monitoring tool can be selected as TensorBoard.

[0117] Step S205, performance evaluation and verification of the pre-trained image generation model after adjustment training. The performance of the pre-trained image generation model after adjustment training can be evaluated on an independent verification set. It can be verified whether the pre-trained image generation model after adjustment training can accurately learn and reproduce specific details such as environmental elements and element spatial distribution in remote sensing images, thereby ensuring the practicality and accuracy of the pre-trained image generation model after adjustment training.

[0118] In this way, by fine-tuning the data in a specific field, the image generation model for image reconstruction can be better adapted to the task of reconstructing image generation, thereby improving the quality and consistency of the reconstructed image. At the same time, the model is adjusted and trained based on the self-built target remote sensing image and text dataset, so that the pre-trained image generation model after adjustment can better generate images that conform to the remote sensing style, improve the ability to accurately align the text input intent, and solve the problem of difficulty in adjusting and training the remote sensing image generation model in related technologies.

[0119] In some optional embodiments, the aforementioned method for constructing a remote sensing image and text data machine also includes: performing a baseline test on the image generation model after adjustment and training to obtain initial performance parameters, and determining whether the initial performance parameters meet the second preset requirements; if the initial performance parameters do not meet the second preset requirements, obtaining the updated weights of the image generation model after adjustment and training, and reconstructing the image based on the updated weights to obtain a reconstructed image of the current scene; calculating the current image matching degree between the current scene reconstructed image and the original remote sensing image corresponding to the current scene reconstructed image, and obtaining a current image matching degree index based on an evaluation of the current image matching degree, and constructing a second remote sensing image and text data set based on the current image matching degree index; performing a secondary adjustment training on the image generation model after adjustment and training based on the second remote sensing image and text data set to obtain an image generation model with secondary updated weights; performing a baseline test on the image generation model with secondary updated weights to obtain second performance parameters, and if the second performance parameters do not meet the second preset requirements, repeating the above steps until the second performance parameters meet the second preset requirements or the number of updates reaches the preset number of times.

[0120] In this embodiment, by obtaining the initial performance parameters, the performance data of the image generation model after adjustment and training before updating can be obtained. Based on the current image matching degree and the current image matching degree index, the performance of the image generation model after adjustment and training in the generation quality and consistency of remote sensing images can be measured. By constructing a second remote sensing image and text data set through data cleaning, low-quality remote sensing images and text description image and text pairs can be removed to construct a higher-quality multimodal remote sensing image and text data set.

[0121] Iterative optimization of the image generation model after adjustment and training can continuously improve the performance of the image generation model after adjustment and training, so that it has stronger image generation capabilities, and the generated reconstructed images have higher image-text matching, further improving the overall performance and effect of the entire system.

[0122] A method for constructing a remote sensing image and text dataset is also provided in an embodiment of the present invention. Figure 3 FIG. 2 is a flow chart of another method for constructing a remote sensing image and text data set in an embodiment of the present invention. Figure 3 As shown in the figure, the methods for constructing remote sensing image and text datasets include:

[0123] Step S310 , by using preset prompt words and based on the pre-trained text-image multimodal large model 34 , a natural language text description 32 of the remote sensing image 31 is generated.

[0124] Figure 3 The original remote sensing images used for the illustrative purposes are from the subset of the category "harbor" in the public dataset. The image format is a 224×224 pixel RGB image. The image description generated after being guided by the preset prompt words is as follows:

[0125] "This picture shows a satellite image of the city. The pier is located in the lower right corner of the harbor, extending into the sea. In front, there are two roads intersecting each other, with multiple lanes on both sides. On the left side, there are several buildings, including restaurants and hotels."

[0126] Step S320, based on the adjusted and trained Vincent graph multimodal large model 35 with SDXL as the base model, a reconstructed image 33 is generated through the prompt words of prefix + image description + constraint suffix. The adjusted and trained Vincent graph multimodal large model 35 with SDXL as the base model can be a SDXL-turbo model after knowledge distillation, or a generative model after updating the SDXL weights.

[0127] Step S330, select the remote sensing image 31 as the image modality input, select the natural language text description 32 generated based on the remote sensing image 31 as the text modality input, calculate the image-text matching degree itm and the image-text contrast itc based on the image modality input and the text modality input, and use itm and itc as the input of the evaluation model.

[0128] In this step, the image-text matching degree itm and the image-text contrast itc can be calculated using the image-text-matching module in BLIP-2 and the match_head parameter of the image-text-matching module instance.

[0129] Step S340, input the natural language text description 32 generated in S310 into the text encoder (Tokenizer) of the adjusted and trained text graph multimodal large model 35, and obtain the output word sequence list tokens. The sequence length l of the text description can be obtained by the len() function. token , the length of the sequence described by the text is l token as input to the evaluation model.

[0130] In this step, tokens can be obtained through the Tokenizer in CLIP.

[0131] Step S350: preprocess the remote sensing image 31 and the reconstructed image 33 to obtain image feature tensors of the same dimension, and calculate the image matching degree C based on the preprocessed remote sensing image and the reconstructed image. regen , the image matching degree C regen as input to the evaluation model.

[0132] In this step, the clip-vit-large-patch14 model can be used as the calculation model for image matching. This model encodes RGB images of different pixels into tensors in the same feature space through CLIP Processor, and then calls CLIP Model to calculate the similarity of the feature tensor to obtain the image matching degree C. regen .

[0133] Step S360, based on the evaluation of at least one of the text evaluation index, the image-text matching degree, the image-text contrast and the image matching degree, an evaluation result of the initial remote sensing image-text data set is obtained, and a target remote sensing image-text data set is constructed based on the evaluation result.

[0134] In this embodiment, a device for constructing a remote sensing image and text data set is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0135] Figure 4 FIG. 1 is a schematic diagram showing the structure of a device for constructing a remote sensing image and text data set provided by an embodiment of the present invention. Figure 4 As shown, this embodiment provides a device for constructing a remote sensing image and text data set, including:

[0136] The acquisition module 401 is used to acquire an initial remote sensing image and text data set, which includes a remote sensing image and a text description describing the remote sensing image.

[0137] The calculation module 402 is used to reconstruct the image based on the text description, obtain the scene reconstructed image, evaluate the quality of the text description, and obtain the text evaluation index; calculate the image-text matching degree and image-text contrast between the remote sensing image and the text description, and calculate the image matching degree between the remote sensing image and the scene reconstructed image.

[0138] The construction module 403 is used to obtain an evaluation result of the initial remote sensing image and text data set based on the evaluation of at least one of the text evaluation index, image and text matching degree, image and text contrast and image matching degree, and to construct the target remote sensing image and text data set based on the evaluation result.

[0139] In some optional implementations, the construction module 403 includes:

[0140] The first construction unit is used to calculate the image-text matching index evaluation result E of the initial remote sensing image-text data set through the following formula: itm ,

[0141]

[0142] Among them, E itm ∈(0,1), α is the first sensitivity parameter, itm pos is the positive image-text matching degree, itm neg is the negative image-text matching degree, [itm pos , itm neg ]∈R2 ,The image-text matching degree includes ,positive image-text matching degree and negative image-text matching degree.

[0143] In some optional implementations, the construction module 403 further includes:

[0144] The second construction unit is used to calculate the image-text contrast index evaluation result E of the initial remote sensing image-text data set by the following formula: itc ,

[0145] E itc =e -β×arccos(itc)

[0146] Among them, E itc ∈(0, 1], β is the second sensitivity parameter, and itc is the image-text contrast.

[0147] In some optional implementations, the construction module 403 further includes:

[0148] The third construction unit is used to calculate the image matching index evaluation result E of the initial remote sensing image and text data set through the following formula: reg ,

[0149]

[0150] Among them, E reg ∈(0, 1], γ is the third sensitivity parameter, C regen is the image matching degree.

[0151] In some optional implementations, the construction module 403 further includes:

[0152] The fourth construction unit is used to calculate the text evaluation index evaluation result E of the initial remote sensing image and text data set through the following formula: cap ,

[0153]

[0154] Among them, E cap ∈(0,1],l token is the encoding sequence length of the text description, l ref is the reference encoding sequence length hyperparameter, and decay is the numerical decay hyperparameter.

[0155] In some optional implementations, the construction module 403 further includes:

[0156] The fifth construction unit is used to calculate the image-text matching index evaluation result E of the initial remote sensing image-text data set through the following formula: itm ,

[0157]

[0158] Among them, E itm ∈(0,1), α is the first sensitivity parameter, itm pos is the positive image-text matching degree, itm neg is the negative image-text matching degree, [itm pos , itm neg ]∈R 2 ,The image-text matching degree includes ,positive image-text matching degree and negative image-text matching degree;

[0159] The image-text contrast index evaluation result E of the initial remote sensing image-text data set is calculated by the following formula: itc ,

[0160] E itc =e -β×arccos(itc)

[0161] Among them, E itc ∈(0, 1], β is the second sensitivity parameter, itc is the image-text contrast;

[0162] The image matching index evaluation result E of the initial remote sensing image and text data set is calculated by the following formula: reg ,

[0163]

[0164] Among them, E reg ∈(0, 1], γ is the third sensitivity parameter, C regen is the image matching degree;

[0165] The text evaluation index evaluation result E of the initial remote sensing image and text dataset is calculated by the following formula: cap ,

[0166]

[0167] Among them, E cap ∈(0,1],l token is the encoding sequence length of the text description, l ref is the reference encoding sequence length hyperparameter, decay is the numerical decay hyperparameter;

[0168] The comprehensive evaluation result E of the initial remote sensing image and text dataset is calculated by the following formula: total ,

[0169] E total =[E itm , E itc , E cap , E reg ]·Vweight

[0170] Among them, V weight is a non-negative weight vector.

[0171] In some optional implementations, the construction module 403 further includes:

[0172] The sixth construction unit is used to sort the initial remote sensing image and text data set with evaluation results based on the second target evaluation index, where the second target evaluation index includes at least one of the following: image and text matching index evaluation result, image and text comparison index evaluation result, image matching index evaluation result and text evaluation index evaluation result; obtain the target remote sensing image and target text description that meet the first preset requirement in the sorting result; and construct the target remote sensing image and text data set based on the target remote sensing image and target text description.

[0173] In some optional implementations, the aforementioned apparatus for constructing a remote sensing image and text dataset further includes: an adjustment training module for adjusting and training the image generation model based on different dataset categories in the target remote sensing image and text dataset.

[0174] In some optional embodiments, the aforementioned device for constructing a remote sensing image and text dataset also includes: an optimization module, which is used to perform a baseline test on the image generation model after adjustment and training, obtain initial performance parameters, and determine whether the initial performance parameters meet the second preset requirements; when the initial performance parameters do not meet the second preset requirements, obtain the updated weights of the image generation model after adjustment and training, and reconstruct the image based on the updated weights to obtain a reconstructed image of the current scene; calculate the current image matching degree between the current scene reconstructed image and the original remote sensing image corresponding to the current scene reconstructed image, obtain the current image matching degree index based on the evaluation of the current image matching degree, and construct a second remote sensing image and text dataset based on the current image matching degree index; perform a secondary adjustment training on the image generation model after adjustment and training based on the second remote sensing image and text dataset to obtain an image generation model with secondary updated weights; perform a baseline test on the image generation model with secondary updated weights to obtain a second performance parameter, and when the second performance parameter does not meet the second preset requirement, repeat the above steps until the second performance parameter meets the second preset requirement or the number of updates reaches the preset number.

[0175] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0176] The device for constructing a remote sensing image and text data set in this embodiment is presented in the form of a functional unit, where the unit refers to an application specific integrated circuit (ASIC) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0177] The embodiment of the present invention also provides a computer device having the above Figure 4 The device shown is used to construct a remote sensing image and text dataset.

[0178] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0179] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0180] The aforementioned memory 20 stores instructions executable by at least one processor 10, so that the aforementioned at least one processor 10 executes the method shown in the above embodiment.

[0181] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0182] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0183] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0184] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (such as a light emitting diode) and a tactile feedback device (such as a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0185] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0186] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0187] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for constructing a remote sensing image and text dataset, characterized in that: The method comprises: Acquire an initial remote sensing image and text data set, wherein the initial remote sensing image and text data set includes a remote sensing image and a text description describing the remote sensing image; Reconstructing an image based on the text description to obtain a scene reconstructed image, and evaluating the quality of the text description to obtain a text evaluation index; Calculating the image-text matching degree and image-text contrast between the remote sensing image and the text description, and calculating the image matching degree between the remote sensing image and the scene reconstruction image; Based on the evaluation of at least one of the text evaluation index, the image-text matching degree, the image-text contrast and the image matching degree, an evaluation result of evaluating the initial remote sensing image-text data set is obtained, and a target remote sensing image-text data set is constructed based on the evaluation result.

2. The method according to claim 1, characterized in that The evaluation result of evaluating the initial remote sensing image and text data set based on the evaluation of the image and text matching degree includes: The image-text matching index evaluation result E of the initial remote sensing image-text data set is calculated by the following formula: itm , Among them, E itm ∈(0,1), α is the first sensitivity parameter, itm pos is the positive image-text matching degree, itm neg is the negative image-text matching degree, [itm pos , itm neg ]∈R 2 The image-text matching degree includes the positive image-text matching degree and the negative image-text matching degree.

3. The method according to claim 1, characterized in that The step of obtaining an evaluation result of the initial remote sensing image and text data set based on the evaluation of the image and text contrast comprises: The image-text contrast index evaluation result E of the initial remote sensing image-text data set is calculated by the following formula: itc , AND itc =and -β×arccos(itc) Among them, E itc ∈(0,1], β is the second sensitivity parameter, and itc is the image-text contrast.

4. The method according to claim 1, characterized in that: The evaluation result of evaluating the initial remote sensing image and text data set based on the evaluation of the image matching degree includes: The image matching index evaluation result E of the initial remote sensing image and text data set is calculated by the following formula: reg , Among them, E reg ∈(0,1], γ is the third sensitivity parameter, C regen is the image matching degree.

5. The method according to claim 1, characterized in that The evaluation result of evaluating the initial remote sensing image and text data set is obtained based on the evaluation of the text evaluation index, including: The text evaluation index evaluation result E of the initial remote sensing image and text data set is calculated by the following formula: cap , Among them, E cap ∈(0,1],l token is the length of the encoding sequence described by the text, l ref is the reference encoding sequence length hyperparameter, and decay is the numerical decay hyperparameter.

6. The method according to claim 1, characterized in that The evaluation based on the text evaluation index, the image-text matching degree, the image-text contrast and the image matching degree, obtaining an evaluation result of evaluating the initial remote sensing image-text data set, includes: The image-text matching index evaluation result E of the initial remote sensing image-text data set is calculated by the following formula: itm , Among them, E itm ∈(0,1), α is the first sensitivity parameter, itm pos is the positive image-text matching degree, itm neg is the negative image-text matching degree, [itm pos , itm neg ]∈R 2 , the image-text matching degree includes the positive image-text matching degree and the negative image-text matching degree; The image-text contrast index evaluation result E of the initial remote sensing image-text data set is calculated by the following formula: itc , AND itc =and -β×arccos(itc) Among them, E itc ∈(0,1], β is the second sensitivity parameter, itc is the image-text contrast; The image matching index evaluation result E of the initial remote sensing image and text data set is calculated by the following formula: reg , Among them, E reg ∈(0,1], γ is the third sensitivity parameter, C regen is the image matching degree; The text evaluation index evaluation result E of the initial remote sensing image and text data set is calculated by the following formula: cap , Among them, E cap ∈(0,1],l token is the length of the encoding sequence described by the text, l ref is the reference encoding sequence length hyperparameter, decay is the numerical decay hyperparameter; The comprehensive evaluation result E of the initial remote sensing image and text data set is calculated by the following formula: total , E total =[E itm ,E itc ,E cap ,E reg ]·V weight Among them, V weight is a non-negative weight vector.

7. The method according to any one of claims 1 to 6, characterized in that The step of constructing a target remote sensing image and text dataset based on the evaluation result includes: Sorting the initial remote sensing image and text data set having the evaluation results based on a second target evaluation index, wherein the second target evaluation index includes at least one of the following: the image and text matching index evaluation result, the image and text comparison index evaluation result, the image matching index evaluation result, the text evaluation index evaluation result, and the comprehensive evaluation result; Obtaining a target remote sensing image and a target text description that meet a first preset requirement in the sorting results; Based on the target remote sensing image and the target text description, the target remote sensing image and text dataset is constructed.

8. The method according to claim 1, characterized in that: The method further includes adjusting and training an image generation model for image reconstruction based on the target remote sensing image and text dataset: Based on different data set categories in the target remote sensing image and text data set, the image generation model is adjusted and trained.

9. The method according to claim 8, characterized in that The method further comprises: Performing a baseline test on the image generation model after the adjustment training to obtain initial performance parameters, and determining whether the initial performance parameters meet the second preset requirements; When the initial performance parameter does not meet the second preset requirement, obtaining the updated weights of the image generation model after the adjustment training, and reconstructing the image based on the updated weights to obtain a reconstructed image of the current scene; Calculating the current image matching degree between the current scene reconstructed image and the original remote sensing image corresponding to the current scene reconstructed image, obtaining a current image matching degree index based on the evaluation of the current image matching degree, and constructing a second remote sensing image and text data set based on the current image matching degree index; Based on the second remote sensing image and text data set, the adjusted and trained image generation model is trained a second time to obtain an image generation model with second updated weights; The baseline test is performed on the image generation model with the second updated weights to obtain a second performance parameter. If the second performance parameter does not meet the second preset requirement, the above steps are repeated until the second performance parameter meets the second preset requirement or the number of updates reaches a preset number.

10. A device for constructing a remote sensing image and text data set, characterized in that: The device comprises: An acquisition module is used to acquire an initial remote sensing image and text data set, wherein the initial remote sensing image and text data set includes a remote sensing image and a text description describing the remote sensing image; A calculation module is used to perform image reconstruction based on the text description to obtain a scene reconstructed image, evaluate the quality of the text description, and obtain a text evaluation index; calculate the image-text matching degree and image-text contrast between the remote sensing image and the text description, and calculate the image matching degree between the remote sensing image and the scene reconstructed image; A construction module is used to obtain an evaluation result of evaluating the initial remote sensing image and text data set based on an evaluation of at least one of the text evaluation index, the image and text matching degree, the image and text contrast and the image matching degree, and to construct a target remote sensing image and text data set based on the evaluation result.

11. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for constructing a remote sensing image and text dataset according to any one of claims 1 to 9 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for constructing a remote sensing image and text dataset according to any one of claims 1 to 9.

Images