Multi-modal large model remote sensing visual question-answering method based on model collaboration

Abstract

The invention provides a multi-modal large model remote sensing visual question-answering method based on model collaboration. The method comprises the following steps: step 1) preparing a visual question-answering data set and constructing an instruction fine tuning data set; 2) forming a plurality of multi-modal remote sensing visual question and answer lightweight pre-training models with different loss ratios; 3) obtaining a multi-modal remote sensing visual question and answer lightweight pre-training model with an optimal weight ratio of comparison loss to generation loss; 4) forming a multi-modal remote sensing visual question and answer large model; 5) respectively obtaining corresponding reasoning results by using the multi-modal remote sensing visual question and answer lightweight pre-training model and the multi-modal remote sensing visual question and answer large model with the optimal weight ratio of the comparison loss and the generation loss; step 6) correcting a reasoning result of the multi-modal remote sensing visual question and answer large model; according to the method, the illusion problem of the multi-modal remote sensing visual question and answer large model can be relieved, and meanwhile, the huge computing power burden caused by collaborative reasoning of multiple large models can be avoided.

Description

Technical Field

[0001] The present invention relates to a multi-modal large model remote sensing visual question answering method based on model collaboration, in particular to the fine-tuning of a multi-modal large model, the collaboration of multi-modal pre-trained models, and the correction of inference results of a multi-modal large model in the remote sensing visual question answering task, belonging to the field of remote sensing multi-modal large models. Background Art

[0002] With the development of the new generation of satellite technology, the number of remote sensing (RS) images is growing rapidly; remote sensing images contain rich information and can be used for various tasks such as object counting, land classification, and target detection; in order to fully utilize the value of remote sensing images as an information source and enable non-expert users to easily access the general information contained in remote sensing images, the visual question answering (VQA) task has been introduced in remote sensing; VQA is a challenging task because the model needs to semantically understand images and questions and reason about the interactions between objects to obtain general information in the images; although the visual question answering task has been popularized in the field of computer vision, due to the complexity of remote sensing images, it is still in its infancy in the remote sensing field; at the same time, large language models have shown superior performance in text-based tasks, and multi-modal large models, based on large language models and incorporating other non-text modal information, can complete various multi-modal tasks; combining the remote sensing VQA task with multi-modal large models is to use the powerful capabilities of large models to promote the development of the visual question answering task in the remote sensing field, mainly including basic question answering such as object type, counting, existence, footprint, quantity comparison, and positional relationship, and question answering that requires complex reasoning by combining basic knowledge; in actual application scenarios, non-expert users can use natural language questions as queries to interact with remote sensing images, making it more convenient to understand the images.

[0003] The VQA task in the field of remote sensing was proposed by Lobry et al. (S. Lobry, D. Marcos, J. Murray, and D. Tuia, “RSVQA: Visual Question Answering for Remote Sensing”). Data,”IEEE Trans. Geosci. Remote. Sens., vol. 58, Art. no. 12, 2020.) first proposed that they established the first large-scale VQA benchmark dataset RSVQA for remote sensing images. The proposed VQA model consists of a convolutional neural network (CNN) for feature extraction and a recurrent neural network (RNN) for question encoding, which predicts the answer by fusing visual information and question representation; in subsequent work, many scholars have conducted relevant research on how to perform effective modal fusion in the field of remote sensing VQA; for example, the CN118469022A invention patent application discloses a multi-source remote sensing visual question answering method and system based on text information guidance, aiming to improve the accuracy of remote sensing visual question answering tasks by fusing text features and image features; later, multimodal large language models emerged, with RSGPT (Y. Hu, J. Yuan, C. Wen, X. Lu, and X. Li, “RSGPT: A Remote Sensing Vision Language Model and Remote sensing large models represented by GeoChat (K.Kuckreja,MSDanish,M.Naseer,A.Das,S.Khan,and FSKhan,“GeoChat:Grounded Large Vision-Language Model for Remote Sensing,”CoRR,vol.abs / 2311.15826,2023.) cover a wider range of remote sensing image understanding tasks.

[0004] However, large multimodal models will make guesses in the absence of information, resulting in inaccurate outputs, which is the well-known "hallucination" problem. In order to alleviate the hallucination problem of large models, scholars have proposed methods such as self-correction of large models, multi-model collaboration and integration, and introduction of external knowledge. However, the integration of multiple large models will incur huge computing power costs. Summary of the invention

[0005] The present invention proposes a multimodal large-model remote sensing visual question answering method based on model collaboration, which aims to solve the problem of how to collaborate lightweight pre-trained models and multimodal large models to alleviate large-model hallucinations in the field of remote sensing visual question answering.

[0006] The technical solution of the present invention is a multimodal large-model remote sensing visual question answering method based on model collaboration, the method comprising:

[0007] Step 1) Prepare a visual question answering dataset and construct an instruction fine-tuning dataset;

[0008] Step 2) Based on the visual question answering dataset, the lightweight pre-training model is trained using contrast loss and generative loss with different weight ratios to achieve synchronous convergence of contrast loss and generative loss, thereby forming several multimodal remote sensing visual question answering lightweight pre-training models with different loss ratios;

[0009] Step 3) determining the optimal weight ratio of the two losses according to the accuracy of the generation results of several multimodal remote sensing visual question answering lightweight pre-trained models with different loss ratios on the validation set, and obtaining a multimodal remote sensing visual question answering lightweight pre-trained model with an optimal weight ratio of contrast loss and generation loss; the optimal weight ratio of the two losses is the optimal weight ratio of contrast loss and generation loss;

[0010] Step 4) Fine-tune the multimodal large model based on the instruction fine-tuning dataset, complete the knowledge injection and instruction alignment in the remote sensing field, and form a multimodal remote sensing visual question answering large model;

[0011] Step 5) for the test data set, using the multimodal remote sensing visual question answering lightweight pre-trained model with the best weight ratio of contrast loss and generation loss in step 3) and the multimodal remote sensing visual question answering large model in step 4) to obtain corresponding reasoning results respectively;

[0012] Step 6) comparing whether the inference results generated by the multimodal remote sensing visual question answering lightweight pre-trained model with the optimal loss ratio and the multimodal remote sensing visual question answering large model are consistent, synthesizing the two inference results and designing prompts to re-input them into the multimodal remote sensing visual question answering large model to correct the inference results of the multimodal remote sensing visual question answering large model; the optimal loss ratio is the optimal weight ratio of the comparison loss and the generation loss;

[0013] Furthermore, the specific process of preparing the visual question answering dataset and constructing the instruction fine-tuning dataset in step 1 includes the following steps:

[0014] Step 1.1) Use the existing remote sensing visual question answering dataset D RSVQA As a visual question answering dataset for a multimodal remote sensing visual question answering lightweight pre-training model, the remote sensing visual question answering dataset D RSVQA The expression is as follows,

[0015]

[0016] N represents the remote sensing visual question answering dataset D RSVQAThe number of samples, Image i is the image of the i-th sample, Question i Represents the problem of the i-th sample, Answer i Represents the answer to the i-th sample, (Image i ,Question i ,Answer i ) represents the i-th image-text pair;

[0017] Step 1.2) Use the remote sensing visual question answering dataset D RSVQA Constructing a multimodal remote sensing visual question answering model instruction fine-tuning dataset, since the remote sensing visual question answering dataset D RSVQA This is a short answer dataset, so a prompt with the content "answer with a word or phrase" is added at the end of each Question to prompt the multimodal remote sensing visual question answering model to generate short answers.

[0018] Furthermore, the specific process of step 2) includes the following steps:

[0019] Step 2.1) Construct a lightweight pre-training model; the overall structure of the lightweight pre-training model includes an image encoder E image , Text Encoder E text , text decoder D text ; Use image encoder E image From the remote sensing visual question answering dataset D RSVQA Extract the first image feature E on the image side from the image of each sample (i.e., image side X) image (X), and for the first image feature E image (X) Perform attention pooling; i and Answer i Concatenate to form text side Y, using text encoder E text Extract the first text feature E on the text side text (Y); The first image feature E on the image side image (X) and the first text feature E on the text side text (T) Interact based on cross attention and obtain image-text fusion feature z, text decoder D text Decode the image-text fusion feature z to restore the corresponding text;

[0020] Step 2.2) Sample positive image-text pairs (x i ,y i ) and negative sample pairs (x i ,y j ); where (x i ,y j) is the normalized encoding of the image in the i-th image-text pair and the text in the j-th image-text pair;

[0021] Step 2.3) Through the first image feature E image (X) and the first text feature E text (Y) Calculate contrast loss Calculate the generation loss by fusing the image and text features z Contrastive loss and generate loss is defined as follows:

[0022]

[0023] Where M is the batch size, σ is the temperature coefficient used to scale the confidence function, T is the total number of tokens in the normalized encoding of the text, and P is θ (y t |y <t ,x) represents the text decoder D text Maximize the likelihood function of image x paired with text y under forward autoregressive factorization;

[0024] Step 2.4) Use the total loss (the sum of contrastive loss and generative loss) and different weight ratios λ Con / λ Cap Train the lightweight pre-trained model constructed in step 2.1) until the lightweight pre-trained model converges; where λ Con is a learnable contrastive loss weight hyperparameter, λ Cap is a learnable hyperparameter of the generated loss weight; during training, by adjusting λ Con and λ Cap The value of controls the contrast loss and generate loss For total loss contribution until the total loss Gradually decreases and tends to be stable; finally, several multimodal remote sensing visual question answering lightweight pre-training models with different loss ratios are obtained.

[0025] Furthermore, the specific process of step 3) includes the following steps:

[0026] Step 3.1) Calculate the accuracy of the inference results of several multimodal remote sensing visual question answering lightweight pre-trained models with different loss ratios on the validation set. The accuracy is the ratio of the correct inference results to the true answers.

[0027] Step 3.2) Draw a weight ratio-accuracy curve, find the weight ratio when the accuracy of the generated answer is closest to the accuracy of the true answer, and determine it as the optimal weight ratio; obtain a multimodal remote sensing visual question answering lightweight pre-training model with an optimal weight ratio of contrast loss and generation loss.

[0028] Furthermore, the specific process of step 4) includes the following steps:

[0029] Step 4.1) Construct a multimodal large model; the overall structure of the multimodal large model includes a visual encoder E of a contrastive language-image pre-training (CLIP) model CLIP and pre-trained Large Language Model (LLM) Vicuna ; Using visual encoder E CLIP From the remote sensing visual question answering dataset D RSVQA Extract the second image feature E from each sample image CLIP (Imagpe i ), using the pre-trained large language model LLM Vicuna From the remote sensing visual question answering dataset D RSVQA Extract the second text feature LLM from each sample question Vicuna (Question i ), and then connect a multi-layer perceptron (MLP, Multilayer Perceptron) projection layer W to transform the second image feature E CLIP (Image i ) and the second text feature LLM Vicuna (Question i ) to align the dimensions, and finally the second image feature E after dimension alignment CLIP (Image i ) and the second text feature LLM Vicuna (Question i ) are combined together as a pre-trained large language model LLM Vicuna Input, generate model response;

[0030] Step 4.2) Calculate the autoregressive loss function Autoregressive loss function Optimize the pre-trained large language model LLM by predicting the conditional probability of each word one by one Vicuna ; Autoregressive loss function The specific definition is as follows:

[0031]

[0032] Among them, L is the sequence length and the target answer is K a , θ is a trainable parameter, K v is an image, K instruct is the instruction in all rounds, K instruct,<i Represents the current predicted word k i All previous rounds of instructions, K a,<i Represents the current predicted word k i Answers from all previous rounds;

[0033] Step 4.3) Fine-tune the pre-trained large language model LLM using Low-Rank Adaptation (LoRA) on the instruction fine-tuning dataset Vicuna , always keep the visual encoder E CLIP The weights of the projection layer W and the pre-trained large language model LLM are updated simultaneously Vicuna Pre-trained weights until the pre-trained large language model LLM Vicuna Converge to form a large multimodal remote sensing visual question answering model.

[0034] Furthermore, the specific process of step 5) includes the following steps:

[0035] Step 5.1) using a multimodal remote sensing visual question answering lightweight pre-trained model with an optimal weight ratio of contrast loss and generation loss to obtain a corresponding inference result; specifically comprising: using the multimodal remote sensing visual question answering lightweight pre-trained model with an optimal weight ratio of contrast loss and generation loss obtained in step 3) to infer the test data set, inputting the images and questions in the test data set into the multimodal remote sensing visual question answering lightweight pre-trained model with an optimal weight ratio of contrast loss and generation loss, and obtaining a corresponding inference result;

[0036] Step 5.2) Use the multimodal remote sensing visual question answering big model to infer and obtain corresponding inference results; specifically including: use the multimodal remote sensing visual question answering big model fine-tuned in step 4) to infer the test data set, input the images and instructions in the test data set into the multimodal remote sensing visual question answering big model, and obtain corresponding inference results.

[0037] Furthermore, the step 6 specifically includes the following steps:

[0038] Step 6.1) Compare whether the inference results generated by the multimodal remote sensing visual question answering lightweight pre-trained model with the best weight ratio of contrast loss and generation loss and the multimodal remote sensing visual question answering large model are consistent, and the two inference results are combined and designed as prompts to input into the multimodal remote sensing visual question answering large model. If they are consistent, they are directly output; if they are inconsistent, a prompt is added at the end of the question: "The possible answer is AnswerM or Answer N ; The true answer may not be included in the possible answers; answer with a word or phrase"; where Answer M is the inference result given by the multimodal remote sensing visual question answering lightweight pre-trained model with the best weight ratio of contrast loss and generation loss. Answer N It is the inference result given by the multimodal remote sensing visual question answering model;

[0039] Step 6.2) The inference result Answer of the multimodal remote sensing visual question answering model after prompt correction final as the final output result.

[0040] Beneficial effects of the present invention:

[0041] 1) The present invention obtains the optimal weight ratio of contrast loss and generation loss in the multimodal remote sensing visual question answering lightweight pre-training model through accuracy, which can effectively improve the accuracy of the generated results while aligning the semantics of images and texts;

[0042] 2) The present invention proposes to use a multimodal remote sensing visual question answering lightweight pre-trained model with an optimal loss ratio to collaborate with a multimodal remote sensing visual question answering large model to realize the reasoning of remote sensing visual question answering tasks, which can effectively avoid the huge computing power cost brought by the integration of multiple large models;

[0043] 3) The present invention proposes to construct a hint including the inference results of a multimodal remote sensing visual question answering lightweight pre-trained model with an optimal loss ratio and the inference results of a multimodal remote sensing visual question answering large model, which can provide feedback from other models to the multimodal remote sensing visual question answering large model, thereby alleviating the hallucination problem of the multimodal remote sensing visual question answering large model;

[0044] 4) The present invention corrects the inference results of the multimodal remote sensing visual question answering large model by feeding back the inference results of the multimodal remote sensing visual question answering lightweight pre-trained model with the best loss ratio to the multimodal remote sensing visual question answering large model. This can alleviate the hallucination problem of the multimodal remote sensing visual question answering large model while avoiding the huge computing power burden brought by the collaborative reasoning of multiple large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is an algorithm flow chart of the multimodal large-model remote sensing visual question answering method based on model collaboration of the present invention.

[0046] Figure 2 It is a network structure diagram of the multimodal remote sensing visual question answering lightweight pre-training model in step 2 of the present invention.

[0047] Figure 3It is a network structure diagram of the multimodal remote sensing visual question answering model in step 4 of the present invention.

[0048] Figure 4 It is a schematic diagram of the collaborative reasoning effect of the model of the present invention. DETAILED DESCRIPTION

[0049] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the drawings. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0050] Combination Figure 1 As shown, the multimodal large-model remote sensing visual question answering method based on model collaboration of the present invention comprises the following steps:

[0051] Step 1: prepare a visual question answering dataset related to the remote sensing visual question answering task and construct an instruction fine-tuning dataset;

[0052] Step 2: Based on the visual question answering dataset, a lightweight pre-training model is trained using contrast loss and generative loss with different weight ratios, and positive and negative image-text pairs are sampled. The contrast loss and generative loss are converged simultaneously to form multiple multimodal remote sensing visual question answering lightweight pre-training models with different loss ratios.

[0053] Step 3, determining the optimal weight ratio of the two losses according to the accuracy of the generation results of several multimodal remote sensing visual question answering lightweight pre-trained models with different loss ratios on the validation set; the optimal weight ratio of the two losses is the optimal weight ratio of the contrast loss and the generation loss; obtaining a multimodal remote sensing visual question answering lightweight pre-trained model with the optimal weight ratio of the contrast loss and the generation loss;

[0054] Step 4: Fine-tune the multimodal large model based on the instruction fine-tuning dataset, complete the knowledge injection and instruction alignment in the remote sensing field, and form a multimodal remote sensing visual question answering large model;

[0055] Step 5: For the test data set, the multimodal remote sensing visual question answering lightweight pre-trained model with the optimal weight ratio of contrast loss and generation loss in step 3 and the multimodal remote sensing visual question answering large model in step 4 are used to obtain corresponding inference results respectively;

[0056] Step 6, compare whether the inference results generated by the multimodal remote sensing visual question answering lightweight pre-trained model with the optimal loss ratio and the multimodal remote sensing visual question answering large model are consistent, integrate the two inference results and design prompts to re-input them into the multimodal remote sensing visual question answering large model, correct the inference results of the multimodal remote sensing visual question answering large model, and alleviate the hallucination problem; the optimal loss ratio is the optimal weight ratio of the contrast loss and the generation loss.

[0057] 1.1 Using the existing remote sensing visual question answering dataset D RSVQA As a visual question answering dataset for a multimodal remote sensing visual question answering lightweight pre-training model, the remote sensing visual question answering dataset D RSVQA The expression is as follows,

[0058]

[0059] N represents the remote sensing visual question answering dataset D RSVQA The number of samples, Image i is the image of the i-th sample, Question i Represents the problem of the i-th sample, Answer i Represents the answer to the i-th sample, (Image i ,Question i ,Answer i ) represents the i-th image-text pair;

[0060] 1.2 Using the Remote Sensing Visual Question Answering Dataset D RSVQA Constructing a multimodal remote sensing visual question answering model instruction fine-tuning dataset, due to D RSVQA This is a short answer dataset, so a prompt with the content "answer with a word or phrase" is added at the end of each Question to prompt the multimodal remote sensing visual question answering model to generate short answers.

[0061] 2.1 Build a lightweight pre-training model; combine Figure 2 As shown, the overall structure includes two encoders (image encoder E image and text encoder E text ) and a decoder (text decoder D text ); using image encoder E image From the remote sensing visual question answering dataset D RSVQA Image in Image i (i.e. image side X) to extract the first image feature E on the image side image (X), and for the first image feature E image (X) Perform attention pooling; i and Answer i Concatenate to form text side Y, using text encoder E text Extract the first text feature E on the text side text (Y); The first image feature E on the image side imgae (X) and the first text feature E on the text side text (Y) Interact based on cross attention and obtain image-text fusion feature z, text decoder D textDecode the image-text fusion feature z to restore the corresponding text;

[0062] 2.2 Sampling positive image-text pairs (x i ,y i ) and negative sample pairs (x i ,y j ); where (x i ,y j ) is the normalized encoding of the image in the i-th image-text pair and the text in the j-th image-text pair; 2.3 Through the first image feature E image (X) and the first text feature E text (Y) Calculate contrast loss Calculate the generation loss by fusing the image and text features z Contrastive loss and generate loss is defined as follows:

[0063]

[0064] Where M is the batch size, σ is the temperature coefficient used to scale the confidence function, T is the total number of tokens in the normalized encoding of the text, and P is θ (y t |y <t ,x) represents the text decoder D text Maximize the likelihood function of image x paired with text y under forward autoregressive factorization;

[0065] 2.4 Use total loss (the sum of contrast loss and generation loss) and different weight ratios λ Con / λ Cap Train the lightweight pre-trained model constructed in step 2.1 until the lightweight pre-trained model converges; where λ Con is a learnable contrastive loss weight hyperparameter, λ Cap is a learnable hyperparameter of the generated loss weight; during training, by adjusting λ Con and λ Cap The value of controls the contrast loss and generate loss For total loss contribution until the total loss Gradually decreases and tends to be stable; finally, multiple multimodal remote sensing visual question answering lightweight pre-training models with different loss ratios are obtained.

[0066] 3.1 Calculate the accuracy of the inference results of multiple multimodal remote sensing visual question answering lightweight pre-trained models with different loss ratios on the validation set. The accuracy is the ratio of the correct inference results to the true answers.

[0067] 3.2 Draw the weight ratio-accuracy curve, find the weight ratio when the accuracy of the generated answer is closest to the accuracy of the true answer, and determine it as the optimal weight ratio; obtain a multimodal remote sensing visual question answering lightweight pre-training model with the optimal weight ratio of contrast loss and generation loss.

[0068] 4.1 Build a multimodal large model, combining Figure 3 As shown in Figure 2, the overall structure mainly includes the visual encoder E of the Contrastive Language-Image Pre-Training (CLIP) model. CLIP and pre-trained Large Language Model (LLM) Vicuna ; Using visual encoder E CLIP Extract the second image feature E CLIP (Image i ), using the pre-trained large language model LLM Vicuna Extract the second text feature LLM Vicuna (Question i ), and then connect a multi-layer perceptron (MLP, Multilayer Perceptron) projection layer W to transform the second image feature E CLIP (Image i ) and the second text feature LLM Vicuna (Question i ) to align the dimensions, and finally the second image feature E after dimension alignment CLIP (Image i ) and the second text feature LLM Vicuna (Question i ) are combined together as a pre-trained large language model LLM Vicuna Input, generate model response; 4.2 Calculate the autoregressive loss function Autoregressive loss function Optimize the pre-trained large language model LLM by predicting the conditional probability of each word one by one Vicuna ; Autoregressive loss function The specific definition of is as follows, where L is the sequence length and the target answer is K a , θ is a trainable parameter, K v is an image, K instruct is the instruction in all rounds, K instruct,<i Represents the current predicted word k i All previous rounds of instructions, K a,<i Represents the current predicted word k i Answers from all previous rounds:

[0069]

[0070] 4.3 Fine-tuning the pre-trained large language model LLM using low-rank adaptation (LoRA) on the instruction fine-tuning dataset Vicuna , always keep the visual encoder E CLIP The weights of the projection layer W and the pre-trained large language model LLM are updated simultaneously Vicuna Pre-trained weights until the pre-trained large language model LLM Vicuna Converge to form a large multimodal remote sensing visual question answering model.

[0071] 5.1 Reasoning of a lightweight pre-trained model for multimodal remote sensing visual question answering with an optimal weight ratio of contrast loss and generation loss; Use the lightweight pre-trained model for multimodal remote sensing visual question answering with an optimal weight ratio of contrast loss and generation loss obtained in step 3 to reason about the test dataset, input the images and questions in the test dataset into the lightweight pre-trained model for multimodal remote sensing visual question answering with an optimal weight ratio of contrast loss and generation loss, and obtain the corresponding reasoning results;

[0072] 5.2 Reasoning of the multimodal remote sensing visual question answering big model: Use the multimodal remote sensing visual question answering big model fine-tuned in step 4 to reason about the test dataset, input the images and instructions in the test dataset into the multimodal remote sensing visual question answering big model, and obtain the corresponding reasoning results.

[0073] 6.1 Combination Figure 4 As shown in the figure, the inference results generated by the multimodal remote sensing visual question answering lightweight pre-trained model with the best weight ratio of contrast loss and generation loss and the multimodal remote sensing visual question answering large model are compared. The two inference results are combined and designed as prompts to input into the multimodal remote sensing visual question answering large model. If they are consistent, they are directly output; if they are inconsistent, a prompt is added at the end of the question: "The possible answer is Answer M or Answer N ; The true answer may not be included in the possible answers; answer with a word or phrase"; where Answer M is the inference result given by the multimodal remote sensing visual question answering lightweight pre-trained model with the best weight ratio of contrast loss and generation loss. Answer N It is the inference result given by the multimodal remote sensing visual question answering model;

[0074] 6.2 The inference result of the multimodal remote sensing visual question answering model after prompt correction Answer final as the final output result.

[0075] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A multimodal large-model remote sensing visual question answering method based on model collaboration, characterized by include: Step 1) Prepare a visual question answering dataset and construct an instruction fine-tuning dataset; Step 2) Based on the visual question answering dataset, the lightweight pre-training model is trained using contrast loss and generative loss with different weight ratios to achieve synchronous convergence of contrast loss and generative loss, thereby forming several multimodal remote sensing visual question answering lightweight pre-training models with different loss ratios; Step 3) according to the accuracy of the generated results of several multimodal remote sensing visual question answering lightweight pre-trained models with different loss ratios on the validation set, determine the optimal weight ratio of the two losses, and obtain a multimodal remote sensing visual question answering lightweight pre-trained model with the optimal weight ratio of contrast loss and generation loss; the optimal weight ratio of the two losses is the optimal weight ratio of contrast loss and generation loss; Step 4) Fine-tune the multimodal large model based on the instruction fine-tuning dataset, complete the knowledge injection and instruction alignment in the remote sensing field, and form a multimodal remote sensing visual question answering large model; Step 5) for the test data set, using the multimodal remote sensing visual question answering lightweight pre-trained model with the best weight ratio of contrast loss and generation loss in step 3) and the multimodal remote sensing visual question answering large model in step 4) to obtain corresponding reasoning results respectively; Step 6) Compare whether the inference results generated by the multimodal remote sensing visual question answering lightweight pre-trained model with the optimal loss ratio and the multimodal remote sensing visual question answering large model are consistent, integrate the two inference results and design prompts to re-input them into the multimodal remote sensing visual question answering large model to correct the inference results of the multimodal remote sensing visual question answering large model; the optimal loss ratio is the optimal weight ratio of the comparison loss and the generation loss.

2. According to the multimodal large-model remote sensing visual question answering method based on model collaboration according to claim 1, it is characterized in that: The specific process of preparing the visual question answering dataset and constructing the instruction fine-tuning dataset in step 1) includes the following steps: Step 1.1) Use the existing remote sensing visual question answering dataset D RSVQA As a visual question answering dataset for a multimodal remote sensing visual question answering lightweight pre-training model, the remote sensing visual question answering dataset D RSVQA The expression is as follows, N represents the remote sensing visual question answering dataset D RSVQA The number of samples, Image i is the image of the i-th sample, Question i Represents the problem of the i-th sample, Answer i Represents the answer to the i-th sample, (Image i ,Question i ,Answer i ) represents the i-th image-text pair; Step 1.2) Use the remote sensing visual question answering dataset D RSVQA A dataset of instruction fine-tuning for building large models for multimodal remote sensing visual question answering.

3. The multimodal large-model remote sensing visual question answering method based on model collaboration according to claim 2 is characterized in that: The specific process of step 2) comprises the following steps: Step 2.1) Construct a lightweight pre-training model; the overall structure of the lightweight pre-training model includes an image encoder E image , Text Encoder E text , text decoder D text ; Use image encoder E image From the remote sensing visual question answering dataset D RSVQA Extract the first image feature E from the image side of each sample in the image image (X), and for the first image feature E image (X) Perform attention pooling; i and Answer i Concatenate to form text side Y, using text encoder E text Extract the first text feature E on the text side text (Y); The first image feature E on the image side image (X) and the first text feature E on the text side text (Y) Interact based on cross attention and obtain image-text fusion feature z, text decoder D text Decode the image-text fusion feature z to restore the corresponding text; Step 2.2) Sample positive image-text pairs (x i ,y i ) and negative sample pairs (x i ,y j ); where (x i ,y j ) is the normalized encoding of the image in the i-th image-text pair and the text in the j-th image-text pair; Step 2.3) Through the first image feature E image (X) and the first text feature E text (Y) Calculate contrast loss Calculate the generation loss by fusing the image and text features z Contrastive loss and generate loss is defined as follows: Where M is the batch size, σ is the temperature coefficient used to scale the confidence function, T is the total number of tokens in the normalized encoding of the text, and P is θ (y t |y <t ,x) represents the text decoder D text Maximize the likelihood function of image x paired with text y under forward autoregressive factorization; Step 2.4) Use the total loss and different weight ratios λ Con / λ Cap Train the lightweight pre-trained model constructed in step 2.1) until the lightweight pre-trained model converges; where λ Con is a learnable contrastive loss weight hyperparameter, λ Cap is a learnable hyperparameter of the generated loss weight; during training, by adjusting λ Con and λ Cap The value of controls the contrast loss and generate loss For total loss contribution until the total loss Gradually decreases and tends to be stable; finally, several multimodal remote sensing visual question answering lightweight pre-training models with different loss ratios are obtained.

4. The multimodal large-model remote sensing visual question answering method based on model collaboration according to claim 1, characterized in that: The specific process of step 3) comprises the following steps: Step 3.1) Calculate the accuracy of the inference results of several multimodal remote sensing visual question answering lightweight pre-trained models with different loss ratios on the validation set. The accuracy is the ratio of the correct inference results to the true answers. Step 3.2) Draw a weight ratio-accuracy curve, find the weight ratio when the accuracy of the generated answer is closest to the accuracy of the true answer, and determine it as the optimal weight ratio; obtain a multimodal remote sensing visual question answering lightweight pre-training model with an optimal weight ratio of contrast loss and generation loss.

5. The multimodal large-model remote sensing visual question answering method based on model collaboration according to claim 2, characterized in that: The specific process of step 4) comprises the following steps: Step 4.1) Construct a multimodal large model; the overall structure of the multimodal large model includes a visual encoder E of a comparison image and text pre-training model CLIP And pre-trained large language model LLM Vicuna ; Using visual encoder E CLIP From the remote sensing visual question answering dataset D RSVQA Extract the second image feature E from each sample image CLIP , using the pre-trained large language model LLM Vicuna From the remote sensing visual question answering dataset D RSVQA Extract the second text feature LLM from each sample question Vicuna , and then connect a multi-layer perceptron projection layer W to transform the second image feature E CLIP and the second text feature LLM Vicuna Perform dimension alignment and finally use the second image feature E after dimension alignment CLIP and the second text feature LLM Vicuna Merge together as a pre-trained large language model LLM Vicuna Input, generate model response; Step 4.2) Calculate the autoregressive loss function Autoregressive loss function Optimize the pre-trained large language model LLM by predicting the conditional probability of each word one by one Vicuna ; Autoregressive loss function The specific definition is as follows: Among them, L is the sequence length and the target answer is K a , θ is a trainable parameter, K v is an image, K instruct is the instruction in all rounds, K instruct,<i Represents the current predicted word k i All previous rounds of instructions, K a,<i Represents the current predicted word k i Answers from all previous rounds; Step 4.3) Pre-train the large language model LLM using low-rank adaptive fine-tuning on the instruction fine-tuning dataset Vicuna , always keep the visual encoder E CLIP The weights of the projection layer W and the pre-trained large language model LLM are updated simultaneously Vicuna Pre-trained weights until the pre-trained large language model LLM Vicuna Converge to form a large multimodal remote sensing visual question answering model.

6. The multimodal large-model remote sensing visual question answering method based on model collaboration according to claim 1, characterized in that: The specific process of step 5) comprises the following steps: Step 5.1) using a multimodal remote sensing visual question answering lightweight pre-trained model with an optimal weight ratio of contrast loss and generation loss to obtain corresponding inference results; Step 5.2) Use the multimodal remote sensing visual question answering large model to infer and obtain the corresponding reasoning results.

7. The multimodal large-model remote sensing visual question answering method based on model collaboration according to claim 1, characterized in that: The step 6 specifically includes the following steps: Step 6.1) Compare the inference results generated by the multimodal remote sensing visual question answering lightweight pre-trained model with the best weight ratio of contrast loss and generation loss and the multimodal remote sensing visual question answering large model to see if they are consistent. Combine the two inference results and design them as prompts to input into the multimodal remote sensing visual question answering large model. If they are consistent, they are directly output; if they are inconsistent, add a prompt at the end of the question: "The possible answer is Answer M or Answer N ; The true answer may not be included in the possible answers; answer with a word or phrase"; where Answer M is the inference result given by the multimodal remote sensing visual question answering lightweight pre-trained model with the best weight ratio of contrast loss and generation loss. Answer N It is the inference result given by the multimodal remote sensing visual question answering model; Step 6.2) The inference result Answer of the multimodal remote sensing visual question answering model after prompt correction final as the final output result.

8. The multimodal large-model remote sensing visual question answering method based on model collaboration according to claim 2, characterized in that: The remote sensing visual question answering dataset D RSVQA Construct an instruction fine-tuning dataset for a multimodal remote sensing visual question answering model, specifically adding a prompt "answer with a word or phrase" at the end of each Question to prompt the multimodal remote sensing visual question answering model to generate short answers.

9. The multimodal large-model remote sensing visual question answering method based on model collaboration according to claim 6, characterized in that: The method of obtaining a corresponding reasoning result by using a multimodal remote sensing visual question answering lightweight pre-trained model with an optimal weight ratio of contrast loss and generation loss specifically includes: using the multimodal remote sensing visual question answering lightweight pre-trained model with an optimal weight ratio of contrast loss and generation loss obtained in step 3) to reason on a test data set, inputting images and questions in the test data set into the multimodal remote sensing visual question answering lightweight pre-trained model with an optimal weight ratio of contrast loss and generation loss, and obtaining a corresponding reasoning result.

10. The multimodal large-model remote sensing visual question answering method based on model collaboration according to claim 6, characterized in that: The method of using the multimodal remote sensing visual question answering big model to infer the corresponding reasoning results specifically includes: using the multimodal remote sensing visual question answering big model fine-tuned in step 4) to infer the test data set, inputting the images and instructions in the test data set into the multimodal remote sensing visual question answering big model, and obtaining the corresponding reasoning results.

Images