Remote sensing question answering system based on comparison-generation type pre-training model

Abstract

The invention discloses a remote sensing question answering system based on a comparison-generation type pre-training model, which comprises the following steps of: firstly, collecting remote sensing images of tasks such as classification, semantic segmentation, target detection, image-text retrieval and image text generation as original data, and uniformly converting the original data into an image-text pair form; the category area proportion is calculated through segmentation processing, a target category and position information thereof are detected, and zero sample classification is performed through a CLIP model. Thirdly, inputting multi-source information into a multi-modal large language model, generating text description of an image according to a question, and completing construction of a remote sensing global visual question and answer data set; and meanwhile, based on the annotation box, cutting the picture and sending the picture into a multi-modal large language model to obtain a remote sensing local visual question and answer data set. Then, pre-training the comparison-generation type model from a coarse stage to a fine stage, so that the comparison-generation type model has global-local level remote sensing knowledge and visual question and answer ability; and finally, realizing question and answer interaction with the user by constructing a user-friendly system interface.

Description

Technical Field

[0001] The present invention relates to a visual question answering system in the computer field, and in particular to the construction of a remote sensing question answering system based on a contrastive-generative pre-training model. Background Art

[0002] Base models are becoming increasingly important in artificial intelligence. Compared with small specialized models tailored for specific tasks or domains, "one-to-one" general base models often show superior performance and generalization capabilities in a wide range of downstream tasks. In recent years, many base models have emerged, such as SimCLR, MAE, Florence, SAM for computer vision, BERT models and GPT, series for natural language processing, and CLIP and Flamingo for visual language learning. In addition, the remote sensing field is also further exploring the direction of remote sensing satellite image base models. So far, the design inspiration of remote sensing base models mainly comes from the success of self-supervised learning (SSL) in the field of computer vision, especially the mask image modeling (MIM) method, especially SatMAE, Scale MAE, ViTAE, Billion Scale MAE, RingMo, GFM and other methods have used MIM methods on large Vision Transformers models and large-scale satellite image datasets, and have achieved good performance.

[0003] However, recent studies have shown that MIM methods, which are the basis of most current remote sensing base models, mainly learn low-level visual features and lack the ability to learn high-level semantics. For example, Kong et al. demonstrated that MIM pre-training essentially learns occlusion-invariant visual features. Park et al. showed that MIM methods are better at learning high-frequency texture features rather than capturing broader global features. Although the low-level features learned by MIM are beneficial for visual recognition tasks in general domains (e.g., natural images in ImageNet), there is no guarantee that the low-level features are also applicable to the remote sensing domain. In addition, all these base models require labeled data and additional fine-tuning stages to adapt to different downstream tasks (e.g., scene classification). Due to the lack of joint modeling of vision and language, they cannot perform zero-shot reasoning like CLIP models. Moreover, recent studies have shown that multimodality plays a key role in building base models for geospatial artificial intelligence (GeoAI). Remote sensing visual language base models can pave the way for many CLIP-based visual language applications in remote sensing scenarios, such as open vocabulary object detection, zero-shot image segmentation, text-to-image generation, and multimodal large language models. Therefore, learning robust features with rich semantics of visual concepts of satellite remote sensing images is of great significance to the study of the basic model of remote sensing visual language. In addition, the remote sensing question answering system based on contrastive-generative pre-training model can improve the operability and application value of remote sensing data, help to better understand and manage various resources and environments on the earth, and thus provide more powerful tools and insights in decision-making and problem solving. This has great potential for scientific research, commercial and government applications in the field of remote sensing. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for constructing a remote sensing question answering system based on a contrastive-generative pre-training model, thereby providing a more accurate solution to the problem of constructing a remote sensing visual question answering system.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] A method for constructing a remote sensing question answering system based on a contrastive-generative pre-training model includes the following steps:

[0007] Step 1: First, collect remote sensing images for tasks such as classification, semantic segmentation, object detection, image-text retrieval, and image-text generation as raw data, and prepare to convert all data into a visual question answering dataset;

[0008] Step 2, segmenting all remote sensing images and calculating the area ratio of all segmentation categories in the current remote sensing image;

[0009] Step 3: Detect the target categories in all remote sensing images, obtain the target detection frame information and the specific location description of the target in the remote sensing image, and crop the image based on the target detection frame to obtain the regional level image;

[0010] Step 4: Perform zero-shot classification of the categories and occurrence probabilities of remote sensing images using the CLIP model;

[0011] Step 5: Input the multi-source information of classification information, area ratio, target detection information, and location information into the multimodal large language model, generate a text description of the image according to the question, and obtain a global visual question answering dataset. At the same time, input the regional level image into the multimodal large language model, generate an answer according to the question, and obtain a regional level visual question answering dataset;

[0012] Step 6: Pre-train the contrastive-generative model in two stages from coarse to fine on the constructed large-scale global-regional remote sensing visual question answering dataset. The first stage data is global-level remote sensing visual question answering data, and the second stage is regional-level remote sensing visual question answering data. The model is equipped with remote sensing knowledge and can perform visual question answering by using dual encoder contrastive learning and encoder-decoder text generation.

[0013] Step 7: Build a system based on the pre-trained contrastive-generative approach to achieve question-answering interaction with users.

[0014] The specific process of step 1 is as follows:

[0015] 1-1. Collect remote sensing classification datasets, including RSC11, NWPU-RESISC45 and RSD46-WHU, target detection datasets xView, VisDrone, DOTA and DIOR, semantic segmentation datasets LoveDA, GID and US3D, and image and text retrieval datasets RSITMD, RSICD and UCM;

[0016] 1-2. Taking the image-text format as the goal, locate the final format that all remote sensing datasets need to be converted.

[0017] The specific process of step 2 is as follows:

[0018] 2-1. Use the ViTAE-RVSA model to segment all remote sensing images so that all the images obtained are mask images;

[0019] 2-2. Based on the masks of all categories in the mask image, calculate the area ratio of each category in the current image.

[0020] As a preferred solution of the present invention, the specific process of step 3 is:

[0021] 3-1. Use the LSKNet model to detect all targets in the remote sensing image and obtain the bounding box (bbox) information of the corresponding target;

[0022] 3-2. Generate text descriptions of the target detection dataset based on bounding box annotations and category labels. The first two texts are generated based on the target location, which is the center point of the bounding box: the first text describes the target in the center of the image, while the second text describes the target at the edge of the image. The remaining three texts are generated based on the number of different target categories present in the image;

[0023] 3-3. The segmentation mask is processed by category, that is, each pixel corresponding to the target category is encoded, the contour points of the connected area of ​​each category in the image mask are identified, and the minimum and maximum values ​​are extracted by sorting the horizontal and vertical coordinates of the contour points, which are expressed as (x min ,y min ) and (x max ,y max ). Finally, a bounding box is defined based on these coordinate positions, and the text description of the corresponding segmented dataset is obtained according to the bounding box-to-text method.

[0024] 3-4. The bounding box (bbox) information of the target is used to construct sentences using rules to obtain a large amount of text descriptions containing the target location information.

[0025] 3-5. Crop the image according to the bounding box (bbox) of the target as region-level image data.

[0026] The specific process of step 4 is as follows:

[0027] 4-1. Collect remote sensing categories and make them into category lists;

[0028] 4-2. Using the RN50 model, the category list is candidate, all remote sensing images are zero-sample classified, and the probabilities of the top eight categories are obtained by arranging them in descending order.

[0029] The specific process of step 5 is as follows:

[0030] 5-1. Input the area ratio information, classification probability, target detection box information, and location information as prompts into the multimodal large language model;

[0031] 5-2. Construct a list of questions and extract one question from it as a prompt to input into the multimodal large language model;

[0032] 5-3. The multimodal large language model is required to generate a text description that conforms to the content of the remote sensing image based on the given multi-source information and questions, so as to obtain a global remote sensing visual question answering dataset;

[0033] 5-4. Input the regional remote sensing images into the multimodal large language model and generate answers based on the questions to obtain the regional remote sensing visual question answering dataset.

[0034] As a preferred solution of the present invention, the specific process of step 6 is:

[0035] 6-1. Dual encoder contrast learning uses InfoNCE loss. For image samples, is a text sample, f I ,f T For image encoder and text encoder, get image and text features The specific loss function is as follows:

[0036]

[0037] Where N is the batch size and τ is a learnable temperature parameter;

[0038] 6-2. The loss function of the encoder-decoder text generation part is as follows:

[0039]

[0040] It means that the text decoder learns to maximize the conditional likelihood of the text y under forward autoregressive factorization;

[0041] 6-3. The final loss function is as follows;

[0042] L CoCa =λ Inf ·L Inf +λ Inf ·L Cap

[0043] 6-4. The first stage of pre-training the model with global remote sensing visual question answering data is used. Based on the first stage, the second stage of pre-training the model with regional remote sensing visual question answering data is used to achieve fine-tuning from coarse to fine.

[0044] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0045] 1. By fusing multi-source remote sensing data and converting them into a remote sensing visual question answering format, the present invention can generate more accurate, natural and diverse text descriptions, breaking through the limitation of insufficient existing remote sensing visual question answering data.

[0046] 2. The present invention uses a two-stage pre-trained contrast-generation model from coarse to fine to perform comparative learning on remote sensing visual question answering data with multi-source information fusion. Compared with other models, the present invention enhances the system's processing capability for complex remote sensing images and improves the robustness and intelligence level of the visual question answering system. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of constructing a remote sensing question-answering system based on a contrastive-generative pre-training model of the present invention.

[0048] Figure 2 This is a schematic diagram of constructing a global remote sensing visual question answering dataset.

[0049] Figure 3 This is a schematic diagram of the construction of a regional remote sensing visual question answering dataset.

[0050] Figure 4 It is a schematic diagram of the architecture of the remote sensing question answering system based on the contrastive-generative pre-trained model of the present invention, which can accommodate different pre-trained models and functions while providing an intuitive user interface. DETAILED DESCRIPTION

[0051] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the attached Figure 1-4 The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be interpreted as limiting the present invention.

[0052] Existing remote sensing basic models based on self-supervised learning mainly learn low-level features and lack understanding of high-level semantics. The lack of understanding of language will result in the model's inability to achieve zero-shot reasoning and require additional fine-tuning stages to adapt to different downstream tasks. To address this limitation, image-text retrieval models for learning image and text feature alignment may have great potential. Early remote sensing retrieval work used CNN to encode images and LSTM to encode text parts. Yuan et al. proposed a new framework using a dynamic fusion module to enable the model to understand the global and local features of satellite images. Rahhal et al. proposed a multilingual framework consisting of language encoders to adapt to remote sensing semantics in different languages. However, the capabilities of these visual language models on downstream tasks other than retrieval have not been verified. Compared with VQA on natural images, research on RSVQA is still in its early stages. Sylvain et al. introduced the first work on VQA on remote sensing data, in which a template-based automated method was designed to construct two remote sensing-oriented datasets. Visual features were extracted using CNN and language features were learned using RNN. After the point-by-point fusion of multimodal features, the answer is predicted by classification. This work provides a dataset and benchmark for further research, paving the way for RSVQA. However, due to the uniqueness of remote sensing images, more specific feature learning algorithms need to be studied and explored, and the available remote sensing VQA data is still very scarce. Based on this problem, the present invention proposes a remote sensing question-answering system based on a contrastive-generative pre-training model, which improves the remote sensing knowledge question-answering ability of the model based on the contrastive-generative pre-training model.

[0053] Combination Figure 1 As shown, the visual question answering system based on remote sensing basic model pre-training of the present invention includes the following steps:

[0054] 1. First, collect remote sensing images for tasks such as classification, semantic segmentation, target detection, image-text retrieval, and image-text generation as raw data, and prepare to convert all data into image-text pairs. The specific process is as follows:

[0055] (1) Collect remote sensing classification datasets, including RSC11, NWPU-RESISC45, and RSD46-WHU, target detection datasets xView, VisDrone, DOTA, and DIOR, semantic segmentation datasets LoveDA, GID, and US3D, and image and text retrieval datasets RSITMD, RSICD, and UCM;

[0056] (2) Taking the image-to-text format as the goal, locate the final format that all remote sensing datasets need to be converted into.

[0057] 2. Segment all remote sensing images and calculate the area ratio of all segmentation categories in the current remote sensing image. The specific process is as follows:

[0058] (1) Use the ViTAE-RVSA model to segment all remote sensing images so that all the images obtained are mask images;

[0059] (2) Based on the masks of all categories in the mask image, calculate the area ratio of each category in the current image.

[0060] 3. Detect the target categories in all remote sensing images, and obtain the target detection frame information and the specific location description of the target in the remote sensing image. The specific process is as follows:

[0061] (1) Use the LSKNet model to detect all targets in the remote sensing image and obtain the bounding box (bbox) information of the corresponding targets;

[0062] (2) Generate text descriptions of the object detection dataset based on bounding box annotations and category labels. The first two texts are generated based on the target location, which is the center point of the bounding box: the first text describes the target in the center of the image, while the second text describes the target at the edge of the image. The remaining three texts are generated based on the number of different target categories present in the image;

[0063] (3) The segmentation mask is processed by category, that is, each pixel corresponding to the target category is encoded, and the contour points of the connected area of ​​each category in the image mask are identified. The horizontal and vertical coordinates of the contour points are sorted to extract the minimum and maximum values, which are represented as and respectively. Finally, a bounding box is defined based on these coordinate positions, and a text description of the corresponding segmented dataset is obtained according to the bounding box-to-text method.

[0064] (4) The bounding box (bbox) information of the target is used to construct sentences using rules to obtain a large amount of text description containing the target location information.

[0065] (5) The image is cropped according to the bounding box (bbox) of the target as region-level image data.

[0066] 4. Use the CLIP model to perform zero-sample classification on the categories and occurrence probabilities of remote sensing images. The specific process is as follows:

[0067] (1) Collect remote sensing categories and compile them into a category list;

[0068] (2) Using the RN50 model, the category list is candidate, all remote sensing images are zero-sample classified, and the probabilities of the top eight categories are obtained by arranging them in descending order.

[0069] 5. Input the multi-source information of classification information, area ratio, target detection information, and location information into the multimodal large language model, generate text description of the image according to the question, and obtain a global visual question answering dataset. At the same time, input the regional level image into the multimodal large language model, generate answers according to the question, and obtain a regional level visual question answering dataset. The specific process is as follows:

[0070] (1) Input area ratio information, classification probability, target detection box information, and location information as prompts into the multimodal large language model;

[0071] (2) A list of questions, from which one question is extracted as a prompt input into the multimodal large language model;

[0072] (3) A multimodal large language model is required to generate a text description that matches the content of the remote sensing image based on the given multi-source information and questions, so as to obtain a global remote sensing visual question-answering dataset.

[0073] (4) The regional-level remote sensing images are input into a multimodal large language model and answers are generated based on the questions to obtain a regional-level remote sensing visual question-answering dataset.

[0074] 6. The large-scale global-regional remote sensing visual question-answering dataset is constructed, and the contrast-generation model is pre-trained in two stages from coarse to fine. The data in the first stage is the global remote sensing visual question-answering data, and the data in the second stage is the regional remote sensing visual question-answering data. The dual encoder contrast learning and encoder-decoder text generation method are used to enable the model to have remote sensing knowledge and perform visual question-answering. The specific process is as follows:

[0075] (1) Dual encoder contrastive learning uses InfoNCE loss, For image samples, is a text sample, f I ,f T For image encoder and text encoder, get image and text features , the specific loss function is as follows:

[0076]

[0077] Where N is the batch size and τ is a learnable temperature parameter;

[0078] (2) The loss function of the encoder-decoder text generation part is as follows:

[0079]

[0080] It means that the text decoder learns to maximize the conditional likelihood of the text y under forward autoregressive factorization;

[0081] (3) The final loss function is as follows;

[0082] L CoCa =λ Inf ·L Inf +λ Inf ·L Cap

[0083] (4) The first stage of pre-training the model with global remote sensing visual question answering data is used, and based on the first stage, the second stage of pre-training the model with regional remote sensing visual question answering data is used to achieve fine-tuning from coarse to fine.

[0084] 7. Based on the pre-trained contrast-generation system, a question-answering interaction with the user is realized. The specific process is as follows:

[0085] (1) Use the Gradio library to build a user-friendly system interface, including model selection, text input, image upload, and result display;

[0086] (2) Django builds a Python web framework that is responsible for processing user requests and interacting with the pre-trained model;

[0087] (3) Implement functions such as model management, task processing, multimodal large language model API calls and result returns, thereby realizing visual question-answering capabilities.

[0088] In summary, the technical solution adopted by the present invention can improve the operability and application value of remote sensing data, help to better understand and manage various resources and environments on the earth, and thus provide more powerful tools and insights in decision-making and problem solving. This has great potential for scientific research, commercial and government applications in the field of remote sensing.

Claims

1. A remote sensing question answering system based on a contrastive-generative pre-trained model, characterized in that: The steps include: Step 1: First, collect remote sensing images for tasks such as classification, semantic segmentation, object detection, image-text retrieval, and image-text generation as raw data, and prepare to convert all data into a visual question answering dataset; Step 2, segmenting all remote sensing images and calculating the area ratio of all segmentation categories in the current remote sensing image; Step 3: Detect the target categories in all remote sensing images, obtain the target detection frame information and the specific location description of the target in the remote sensing image, and crop the image based on the target detection frame to obtain the regional level image; Step 4: Perform zero-shot classification of the categories and occurrence probabilities of remote sensing images using the CLIP model; Step 5: Input the multi-source information of classification information, area ratio, target detection information, and location information into the multimodal large language model, generate a text description of the image according to the question, and obtain a global visual question answering dataset. At the same time, input the regional level image into the multimodal large language model, generate an answer according to the question, and obtain a regional level visual question answering dataset; Step 6: Pre-train the contrastive-generative model in two stages from coarse to fine on the constructed large-scale global-regional remote sensing visual question answering dataset. The first stage data is global-level remote sensing visual question answering data, and the second stage is regional-level remote sensing visual question answering data. The model is equipped with remote sensing knowledge and can perform visual question answering by using dual encoder contrastive learning and encoder-decoder text generation. Step 7: Build a system based on the pre-trained contrastive-generative approach to achieve question-answering interaction with users.

2. The remote sensing question answering system based on the contrastive-generative pre-training model according to claim 1, characterized in that: The specific process of step 1 is as follows: 1-1. Collect remote sensing classification datasets, including RSC11, NWPU-RESISC45 and RSD46-WHU, target detection datasets xView, VisDrone, DOTA and DIOR, semantic segmentation datasets LoveDA, GID and US3D, and image and text retrieval datasets RSITMD, RSICD and UCM; 1-2. Taking the image-text format as the goal, locate the final format that all remote sensing datasets need to be converted.

3. The remote sensing question answering system based on the contrastive-generative pre-training model according to claim 1, characterized in that: The specific process of step 2 is as follows: 2-1. Use the ViTAE-RVSA model to segment all remote sensing images so that all the images obtained are mask images; 2-2. Based on the masks of all categories in the mask image, calculate the area ratio of each category in the current image.

4. The remote sensing question answering system based on the contrastive-generative pre-training model according to claim 1, characterized in that: The specific process of step 3 is as follows: 3-1. Use the LSKNet model to detect all targets in the remote sensing image and obtain the bounding box (bbox) information of the corresponding target; 3-2. Generate text descriptions of the target detection dataset based on bounding box annotations and category labels. The first two texts are generated based on the target location, which is the center point of the bounding box: the first text describes the target in the center of the image, while the second text describes the target at the edge of the image. The remaining three texts are generated based on the number of different target categories present in the image; 3-3. The segmentation mask is processed by category, that is, each pixel corresponding to the target category is encoded, the contour points of the connected area of ​​each category in the image mask are identified, and the minimum and maximum values ​​are extracted by sorting the horizontal and vertical coordinates of the contour points, which are expressed as (x min ,y min ) and (x max ,y max ); finally, a bounding box is defined based on these coordinate positions, and a text description of the corresponding segmented dataset is obtained according to the bounding box-to-text method; 3-4. The bounding box (bbox) information of the target is used to construct sentences using rules to obtain a large amount of text descriptions containing the target location information; 3-5. Crop the image according to the bounding box (bbox) of the target as region-level image data.

5. The remote sensing question answering system based on the contrastive-generative pre-training model according to claim 1, characterized in that: The specific process of step 4 is as follows: 4-1. Collect remote sensing categories and make them into category lists; 4-2. Using the RN50 model, the category list is candidate, all remote sensing images are zero-sample classified, and the probabilities of the top eight categories are obtained by arranging them in descending order.

6. The remote sensing question answering system based on the contrastive-generative pre-training model according to claim 1, characterized in that: The specific process of step 5 is as follows: 5-1. Input the area ratio information, classification probability, target detection box information, and location information as prompts into the multimodal large language model; 5-2. Construct a list of questions and extract one question from it as a prompt to input into the multimodal large language model; 5-3. The multimodal large language model is required to generate a text description that conforms to the content of the remote sensing image based on the given multi-source information and questions, so as to obtain a global remote sensing visual question answering dataset; 5-4. Input the regional remote sensing images into the multimodal large language model and generate answers based on the questions to obtain the regional remote sensing visual question answering dataset.

7. The remote sensing question answering system based on the contrastive-generative pre-training model according to claim 1, characterized in that: The specific process of step 6 is as follows: 6-1. Dual encoder contrast learning uses InfoNCE loss. For image samples, is a text sample, f I ,f T For image encoder and text encoder, get image and text features The specific loss function is as follows: Where N is the batch size and τ is a learnable temperature parameter; 6-2. The loss function of the encoder-decoder text generation part is as follows: It means that the text decoder learns to maximize the conditional likelihood of the text y under forward autoregressive factorization; 6-3. The final loss function is as follows; L CoCa =λ Inf ·L Inf +λ Inf ·L Cap 6-4. The first stage of pre-training the model with global remote sensing visual question answering data is used. Based on the first stage, the second stage of pre-training the model with regional remote sensing visual question answering data is used to achieve fine-tuning from coarse to fine.

8. The remote sensing question answering system based on the contrastive-generative pre-training model according to claim 1, characterized in that: The specific process of step 7 is as follows: 7-1. Use the Gradio library to build a user-friendly system interface, including model selection, text input, image upload, and result display; 7-2. Django builds a Python web framework, which is responsible for processing user requests and interacting with pre-trained models; 7-3. Implement the functions of model management, task processing, multimodal large language model API calls and result return, so as to realize visual question answering capabilities.

Images