Multi-modal large model retrieval question and answer method of cascade architecture

Abstract

The invention discloses a cascade architecture multi-modal large model retrieval question-answering method, and aims to improve the accuracy and efficiency of a question-answering system. The method comprises the following steps: firstly, searching multi-modal information related to query images and problem description from a knowledge base by utilizing a vector search technology through a visual search step; secondly, in the rough rearrangement step, paragraphs highly related to question description are screened out, and irrelevant content is eliminated; then, in the fine rearrangement step, the paragraphs screened out are deeply analyzed through a multi-modal large model, and the paragraphs most likely to contain correct answers are further sorted and selected; and finally, generating a final answer according to the selected paragraph content and the question description in the answer generation step. In addition, through the hybrid multi-modal retrieval strategy and the cascaded coarse-to-fine reordering method, the long-tail problem and the contradiction between efficiency and performance in the reordering stage are solved, and the system performance is remarkably improved. Experimental results show that the method is obviously superior to the existing most advanced method on multiple data sets.

Description

Technical Field

[0001] The present invention relates to multimodal information retrieval question answering technology, and in particular to a multimodal large model retrieval question answering method with a cascade architecture. Background Art

[0002] Question answering systems based on multimodal large models have been one of the research hotspots in the field of artificial intelligence in recent years. Multimodal large language models (MLLMs) are pre-trained on large-scale network data, demonstrating excellent understanding and generation capabilities, and have made increasingly significant progress in their application to question answering systems. However, despite the breakthroughs these models have achieved in many tasks, they still face some significant challenges in practical applications. 1) Limited visual query information: Existing MLLMs can often only utilize limited visual information when processing visual queries, which limits their performance in complex multimodal tasks. For example, in visual question answering tasks, the model may not be able to fully utilize the subtle details in the image. 2) The long tail effect of training data: The data distribution on the Internet is usually long-tailed, that is, a few common patterns occupy most of the data, while a large number of rare patterns are less common. This data distribution leads to poor performance of the model when processing rare patterns. 3) Lack of latest knowledge: The knowledge of pre-trained models is usually derived from the time window of the training data, which means that they may not have the latest information. This is a major limitation for tasks that require real-time knowledge updates (such as news summarization, question answering, etc.).

[0003] To address the above problems, Multimodal Retrieval-Augmented Generation (MRAG) was developed. MRAG enhances the generation capability of MLLMs by dynamically retrieving relevant information from external knowledge bases (e.g., Wikipedia). Specifically, MRAG retrieves relevant text and image information from the knowledge base based on visual clues in the query, thereby providing more contextual support. This approach shows great potential in knowledge-intensive tasks, such as: 1) Visual Question Answering: By retrieving image-related knowledge, the model's ability to understand and answer complex questions is improved. 2) Information Retrieval (InfoSeek): By combining visual and textual information, the accuracy and relevance of information retrieval are improved. Although MRAG has great theoretical advantages, existing methods still have some obvious limitations. 1) Limitations of visual retrieval strategies: a) Unimodal matching: Existing visual retrieval methods usually rely on unimodal matching (e.g., feature vector matching), ignoring the multimodal clues in the knowledge base. For example, the method of feature vector matching using FAISS only considers image features and ignores text information. b) Long-tail problem: Existing retrievers (such as CLIP) perform poorly when dealing with long-tail visual content. Knowledge bases such as Wikipedia contain a large number of long-tail entities that appear less frequently in the training data, making it difficult for the retriever to accurately match them. 2) The efficiency and performance contradiction in the reranking stage: a) The use of dense encoders: Some methods use dense encoders (such as Contriever, BLIP-2, DPR) for paragraph reranking. Although these encoders perform well in performance, they have a high computational cost, which affects the overall efficiency of the system. b) MLLM as a reranker: Other methods directly use MLLM as a reranker. Although MLLM has strong understanding capabilities, its reasoning speed is slow and becomes a bottleneck of the system.

[0004] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0005] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a multi-modal large model retrieval question answering method with a cascade architecture.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A multimodal large model retrieval question answering method with a cascade architecture comprises the following steps:

[0008] 1) Visual search: Retrieve multimodal information related to the query image and question description from the knowledge base, encode the features of the query image and reference images and titles in the knowledge base, and use vector search technology to select the pages most similar to the query image as candidate information sources;

[0009] 2) Rough re-ranking: Filter out the paragraphs with high relevance to the question description from the candidate pages retrieved in step 1), and exclude obviously irrelevant paragraphs by evaluating the relevance score between the question description and the paragraphs;

[0010] 3) Fine re-ranking: Conduct in-depth analysis of the paragraphs selected in step 2), use a multimodal large model to understand the relationship between the query image, question description, and paragraph content, and further rank and select the paragraphs most likely to contain the correct answer;

[0011] 4) Answer generation: Generate the final answer based on the content of the paragraph selected in step 3) and the question description.

[0012] Furthermore, in step 1), multimodal information is retrieved from the knowledge base through visual query, specifically including:

[0013] Extracting features of the query image;

[0014] Matching with reference images and captions in the knowledge base;

[0015] The features of the reference image and title are extracted by the visual encoder and text encoder respectively;

[0016] Matching is performed by jointly embedding the space to obtain the top K candidate pages that are most similar to the query image, and the paragraphs in these pages are merged into a paragraph sequence.

[0017] Furthermore, in step 1), the visual search method further comprises:

[0018] First, feature encoding is performed on the query image;

[0019] The encoded features are then used to calculate similarity with the features of each entry in the knowledge base;

[0020] Select the top K pages with the most similar features to the query image as candidates;

[0021] Each candidate page is selected based on the encoded features of the reference image and title, where the encoded features are extracted by the visual encoder and text encoder respectively;

[0022] The selection of candidate pages uses the FAISS library to perform vector search, ensuring that half of the candidate pages come from similarity matching in the image modality and the other half come from similarity matching in the text modality.

[0023] Furthermore, in step 1), the visual search method further comprises:

[0024] Use the CLIP retriever and fine-tune it. Train CLIP using supervised contrast loss to accommodate a large number of entities and solve the long-tail problem.

[0025] Adjust the loss term of negative samples to overcome the long-tail problem based on the optimal encoding distance to improve the retrieval accuracy of the model when dealing with rare entities;

[0026] Using inter-class contrastive learning loss function with label supervision, matching query images, captions, and reference images are considered to be of the same category to enhance the model's ability to encode long-tail data or domain-specific data;

[0027] A similarity score adjustment mechanism is implemented. For a batch of data, if two samples belong to the same category, the similarity score is maximized; if the two samples do not belong to the same category and the cosine similarity is lower than a set threshold, the similarity score is adjusted to zero or a positive number; preferably, the set threshold is based on the feature space and the number of categories, and is determined by the minimum distance of the binary linear code.

[0028] Furthermore, in step 2), the text encoder Contriver is used as a rough sorter to perform preliminary screening on the retrieved paragraphs to exclude paragraphs that are obviously irrelevant to the query question.

[0029] Further, in step 2), the rough rearrangement method specifically includes:

[0030] Identify the most relevant partial sequence to the query question from a given candidate page;

[0031] Perform a rough ranking to filter out passages that clearly do not answer the question;

[0032] Use the pre-trained Contriver as a rough ranker to evaluate the relevance between the query and the paragraph;

[0033] By calculating the relevance score, select the N parts with the highest scores as the objects for the next step of processing, where N is less than the total number of paragraphs in the candidate page;

[0034] Among them, by constructing a training dataset containing plain text questions and paragraphs, the coarse ranger is fine-tuned to improve the performance in specific domain tasks; in the training dataset, the annotated true part is set as the positive sample, and the hard negative sample sampling technology is used to construct the negative sample, and the non-true part with the highest prediction score of the un-fine-tuned coarse ranger is selected as the hard negative sample; preferably, the InfoNCE loss is used to train the coarse ranger to optimize the accuracy of the relevance evaluation.

[0035] Furthermore, in step 3), a multimodal model MLLM is used as a fine rearranger, and the paragraphs after rough rearrangement and filtering are input into the MLLM, and the MLLM performs the final sorting and generation according to the query question and the paragraph content.

[0036] Furthermore, in step 3), the fine rearrangement method specifically includes:

[0037] Determine the answer probability of each candidate group by calculating the score of the first corresponding word "yes" or "no" output by MLLM;

[0038] In a given partial sequence, the background knowledge with the highest answer probability is selected as the basis for generating the final result;

[0039] The fine-tuning approach is used to train the fine-tuned rearranger, where positive instructions pair questions with background knowledge containing answers, and negative instructions pair questions with other paragraphs on the same page, ensuring that negative samples are mostly from the same page.

[0040] Maintain a 1:1 balance between positive and negative samples in the dataset, train the MLP projection module and large language model module of MLLM to achieve language modeling with causal attention mask;

[0041] By inputting a query image, a question and some background knowledge, as well as previously generated answers, the model predicts the probability of the next token, thereby optimizing the accuracy of fine-grained reranking.

[0042] Furthermore, in step 4), the answer generation method specifically includes:

[0043] The answer generator generates the final answer by combining the finely rearranged background knowledge with the query question;

[0044] Among them, by fine-tuning instructions, the ability of the answer generator to answer knowledge-intensive questions based on context clues is enhanced; template construction instructions are applied to guide the answer generator to generate accurate answers based on context information;

[0045] During the training process, supervised fine-tuning is performed to enable the model to effectively integrate multimodal information and perform language modeling with causal attention masks. By inputting query images, questions, paragraphs, and previously generated answers, the model predicts and generates the next tag.

[0046] A computer program product includes a computer program, and when the computer program is executed by a processor, the multimodal large model retrieval question answering method is implemented.

[0047] The present invention has the following beneficial effects:

[0048] The present invention proposes an innovative cascade architecture for a multimodal large model retrieval question answering method, which significantly improves the performance and efficiency of the question answering system through a series of optimization steps. The method first adopts a visual search step, using vector search technology to retrieve multimodal information closely related to the query image and question description from the knowledge base, thereby overcoming the challenge of limited information utilization of existing models when processing visual queries. Then, through a rough re-ranking step, paragraphs with high relevance to the question description are effectively screened out. This step excludes obviously irrelevant paragraphs by evaluating the relevance score between the question and the paragraph, reducing the computational burden of subsequent processing. In the fine re-ranking step, the multimodal large model is used to deeply analyze the screened paragraphs, further sort and select the paragraphs most likely to contain the correct answer. This step improves the accuracy of the model when processing long-tail visual content by optimizing the retrieval strategy. Finally, in the answer generation step, the final answer is generated according to the selected paragraph content and the question description, ensuring the accuracy and relevance of the answer. The present invention solves the contradiction between efficiency and performance in the re-ranking stage and reduces computational overhead by mixing multimodal retrieval strategies and cascading coarse-to-fine re-ranking methods. Experiments have verified that the method of the present invention significantly improves the accuracy on the EVQA and InfoSeek datasets, and greatly reduces the inference time, which is of great significance for improving the performance and efficiency of multimodal large-model question answering systems.

[0049] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a task flow chart of the multi-modal large model retrieval question answering method with a cascade architecture according to an embodiment of the present invention.

[0051] Figure 2 It is a system framework flow chart of the multimodal large model retrieval question and answer method according to an embodiment of the present invention.

[0052] Figure 3 This is an example of difficult sample sampling according to an embodiment of the present invention.

[0053] Figure 4 1 is a comparison table of experimental results of the embodiments of the present invention. DETAILED DESCRIPTION

[0054] In order to solve the problems existing in the existing MRAG method, the present invention proposes a multimodal large model retrieval question answering method with a cascade architecture and designs an efficient cascaded multimodal retrieval augmented generation (CasRAG) pipeline, which significantly improves the performance and efficiency of the system by optimizing the retrieval strategy and reranking method.

[0055] See also Figure 1 and Figure 2The embodiment of the present invention provides a multi-modal large model retrieval question answering method with a cascade architecture, comprising the following steps:

[0056] Step 1) Visual search: Retrieve multimodal information related to the query image and question description from the knowledge base, by encoding the features of the query image and reference images and titles in the knowledge base, and using vector search technology to select the pages most similar to the query image as candidate information sources.

[0057] In a preferred embodiment, multimodal information is retrieved from a knowledge base through visual query, specifically including: extracting features of the query image; matching with reference images and titles in the knowledge base; extracting features of the reference image and title through a visual encoder and a text encoder respectively; matching through a joint embedding space to obtain the top K candidate pages most similar to the query image, and merging the paragraphs in these pages into a paragraph sequence.

[0058] In a preferred embodiment, the visual search method further includes: first, feature encoding the query image; then using the encoded features to calculate the similarity with the features of each entry in the knowledge base; selecting the top K pages that are most similar to the query image features as candidates; the selection of each candidate page is based on the encoded features of the reference image and the title, wherein the encoded features are extracted by the visual encoder and the text encoder respectively; the selection of candidate pages utilizes the FAISS library for vector search to ensure that half of the candidate pages come from similarity matching in the image modality and the other half come from similarity matching in the text modality.

[0059] In a preferred embodiment, the visual search method further includes: using a CLIP retriever and fine-tuning, training CLIP using supervised contrast loss to adapt to a large number of entities and solve the long-tail problem; adjusting the loss term of negative samples to overcome the long-tail problem based on the optimal encoding distance to improve the retrieval accuracy of the model when dealing with rare entities; using an inter-class contrastive learning loss function with label supervision to treat matching query images, titles, and reference images as the same category to enhance the model's encoding ability for long-tail data or domain-specific data; implementing a similarity score adjustment mechanism, for a batch of data, if two samples belong to the same category, the similarity score is maximized; if the two samples do not belong to the same category and the cosine similarity is lower than a set threshold, the similarity score is adjusted to zero or a positive number; preferably, the set threshold is based on the feature space and the number of categories, and is determined by the minimum distance of the binary linear code.

[0060] Step 2) Rough re-ranking: Filter out paragraphs with high relevance to the question description from the candidate pages retrieved in step 1), and exclude obviously irrelevant paragraphs by evaluating the relevance score between the question description and the paragraphs.

[0061] In a preferred embodiment, a text encoder Contriver is used as a rough sorter to perform preliminary screening on the retrieved paragraphs to exclude paragraphs that are obviously irrelevant to the query question. The rough re-ranking method specifically includes: identifying the most relevant partial sequence to the query question from a given candidate page; performing rough ranking to filter out paragraphs that obviously cannot answer the question; using a pre-trained Contriver as a rough sorter to evaluate the relevance between the query question and the paragraph; by calculating the relevance score, the N parts with the highest scores are selected as the objects for the next step of processing, where N is less than the total number of paragraphs in the candidate page; wherein, by constructing a training data set containing plain text questions and paragraphs, the rough sorter is fine-tuned to improve performance in specific domain tasks; in the training data set, the annotated true part is set as a positive sample, and the negative sample is constructed using the hard negative sample sampling technology, and the non-true part with the highest prediction score of the un-fine-tuned rough sorter is selected as a hard negative sample; preferably, the InfoNCE loss is used to train the rough sorter to optimize the accuracy of the relevance evaluation.

[0062] Step 3) Fine re-ranking: Perform in-depth analysis on the paragraphs selected in step 2), use a multimodal large model to understand the relationship between the query image, question description, and paragraph content, and further sort and select the paragraphs most likely to contain the correct answer.

[0063] In a preferred embodiment, a multimodal model MLLM is used as a fine rearranger, and the paragraphs filtered after rough rearrangement are input into the MLLM, and the MLLM performs the final sorting and generation according to the query question and the paragraph content. The fine rearrangement method specifically includes: determining the answer possibility of each candidate group by calculating the score of the first corresponding word "yes" or "no" output by the MLLM; selecting the background knowledge with the highest answer possibility in a given partial sequence as the basis for generating the final result; training the fine rearranger in a supervised fine-tuning manner, in which positive instructions pair the question with the background knowledge containing the answer, and negative instructions pair the question with other paragraphs on the same page, ensuring that the negative samples are mainly from the same page; maintaining a 1:1 balance ratio of positive and negative samples in the data set, training the MLP projection module and the large language model module of the MLLM to achieve language modeling with causal attention mask; by inputting the query image, the question and part of the background knowledge, as well as the previously generated answer, the model predicts the probability of the next tag, thereby optimizing the accuracy of the fine rearrangement.

[0064] Step 4) Answer generation: Generate the final answer based on the content of the paragraph selected in step 3) and the question description.

[0065] In a preferred embodiment, the answer generation method specifically includes: using the background knowledge after fine-tuning and the information combined with the query question to generate the final answer by the answer generator; wherein, through instruction fine-tuning, the ability of the answer generator to answer knowledge-intensive questions based on context clues is enhanced; template construction instructions are applied to guide the answer generator to generate accurate answers based on context information; during the training process, supervised fine-tuning is performed to enable the model to effectively integrate multimodal information and perform language modeling with causal attention mask, and by inputting query images, questions, paragraphs and previously generated answers, the model predicts and generates the next tag.

[0066] The multimodal large model retrieval question answering method of the cascaded architecture of the embodiment of the present invention significantly improves the performance and efficiency of the system by mixing multimodal retrieval strategies and cascaded coarse-to-fine reranking methods. The method first improves the accuracy of retrieval by combining multimodal matching of reference images and titles, especially the processing of long-tail problems. By fine-tuning the CLIP retriever and optimizing the negative sample loss, the high retrieval accuracy of the model when processing rare entities is ensured. Then, a lightweight text encoder Contriver is used for preliminary text filtering to effectively exclude paragraphs that are obviously irrelevant to the query question, reducing the computational burden. In the fine reranking stage, the powerful understanding and generation capabilities of MLLM are used to conduct in-depth analysis of the paragraphs after preliminary filtering to ensure the accuracy and relevance of the final results. In addition, through instruction fine-tuning and supervised fine-tuning, the answer generator is able to generate accurate answers based on contextual clues while maintaining high efficiency.

[0067] The specific embodiments and experimental verification of the present invention are further described below.

[0068] The present invention proposes a multimodal large model retrieval question answering method with a cascade architecture and designs a MRAG system based on the cascade architecture, which specifically includes two important parts: S1 hybrid multimodal retrieval strategy and S2 cascaded coarse-to-fine reranking method.

[0069] S1. Hybrid multimodal retrieval strategy

[0070] S1.1 Combining reference images and captions

[0071] Multimodal matching: Traditional visual retrieval methods usually rely on unimodal matching (such as feature vector matching), which limits the accuracy of retrieval. This paper proposes a hybrid multimodal retrieval strategy, which significantly improves the accuracy of retrieval by matching the query image with the reference images and titles of Wikipedia pages.

[0072] Specific implementation: During the retrieval process, the features of the query image are first extracted, and then matched with the reference images and titles in the Wikipedia page. The features of the reference image and title are extracted by the visual encoder and text encoder respectively, and finally matched by the joint embedding space.

[0073] S1.2 Custom CLIP Retriever

[0074] Fine-tuning CLIP: To better handle the long-tail problem in the knowledge base, we fine-tune the general CLIP retriever. Specifically, we train CLIP using supervised contrastive loss to adapt to a large number of entities.

[0075] Optimize negative sample loss: During training, adjust the loss term of negative samples based on the optimal encoding distance to overcome the long tail problem. This ensures that the model can maintain high retrieval accuracy even when dealing with rare entities.

[0076] S2. Cascaded coarse-to-fine reordering method

[0077] S2.1 Contriver as a coarse and heavy sorter

[0078] Text filtering: Existing re-ranking methods usually use dense encoders or MLLM, which have high computational costs. This paper proposes a cascaded coarse-to-fine re-ranking method, which first uses Contriver to perform preliminary text filtering to exclude obviously irrelevant content.

[0079] Implementation: Contriver is a lightweight text encoder that can efficiently perform preliminary screening of retrieved paragraphs. Contriver is trained to quickly identify and filter out irrelevant paragraphs based on the query. For example, when processing a query about "architecture", Contriver can effectively exclude paragraphs that are not related to architecture.

[0080] S2.2 MLLM as a fine re-ranker

[0081] Fine re-ranking: Based on the preliminary filtering, MLLM is used for fine re-ranking to ensure the accuracy and relevance of the final results. MLLM has strong understanding and generation capabilities and can generate high-quality answers based on query questions and filtered paragraphs.

[0082] Specific implementation: Input the paragraphs filtered by Contriver into MLLM, and MLLM will perform the final sorting and generation based on the query questions and paragraph content. In this way, the quality of the generated results is guaranteed and the efficiency of the system is significantly improved.

[0083] A detailed description is given below.

[0084] like Figure 1 As shown in the figure, previous visual question answering tasks require the model M to provide an answer A for a given query image I_query and question Q. In Multimodal Retrieval and Generation (MRAG), the correct answer A cannot be obtained based on I_query or Q alone. This requires the model to retrieve external information from the knowledge base memory (KB) to form the final response. KB contains N_page wiki pages P, each page P includes an entity title T_title, a reference image I_ref and N_sec^i segmented paragraphs S, that is, P_i:={T_title; I_ref; S_1,...,S_N_sec^i}, i belongs to N_page. The ideal model M should retrieve the most informative wiki page P_i and its paragraphs S_i,j before generating an answer to assist itself in making the correct answer.

[0085] Use MLLM for multimodal retrieval and generation tasks. Figure 2 As shown, the present invention includes three levels of retrieval steps and an answer generation process, which are respectively executed by four modules, namely, M:={M_v;M_c;M_r;M_g}. 1) Visual Search Search Module M_v uses visual query I_query to retrieve multimodal information, namely T_title and I_ref, from all pages P in the knowledge base KB. Through the features extracted by M_v, the top K candidate pages P: = {P_1,...,P_K} are obtained and all paragraphs in P are connected into a single paragraph sequence S: = {S_1,...S_N_topk}, where N_topk = ∑(i\in K)N_sec^i. 2) Roughly rearrange modules M_c uses a verification module to filter out passages that are obviously not relevant to question Q, thereby reducing the computational burden of the fine rearranger. 3) Fine Rearrangement Module Mr uses MLLM to determine whether the query {I_query; Q} based on paragraph S is answerable. 4) Answer Generation Module M_g utilizes the paragraph S* with the highest confidence score as external knowledge to provide the final answer A.

[0086] S1 hybrid multimodal retrieval strategy

[0087] This innovation is mainly applied in the visual search module M_v. The goal is to find the paragraph S* containing key information. However, given that the knowledge base is too large, direct search is inefficient. Therefore, first retrieve the matching title T_title or reference image I_ref, and then search for the required part within the corresponding page P. CLIP is used as M_v to encode multimodal features, and the FAISS library is used for vector search. Specifically, based on the similarity calculation between the features encoded in the query image I_query and each entry in the knowledge base, the top K pages most similar to the query image are selected as candidates, where K is much smaller than the total number of pages N_page. Each entry consists of the encoded features of the reference image I_ref and the encoded features of the title T_title. Therefore, the top K pages closest to the title or reference image are selected as candidates, half of which are from the image modality and the other half are from the text modality.

[0088] M_v is further fine-tuned to address its limitations in encoding long-tail data or domain-specific data. By treating the matched query image I_query, title T_title and reference image I_ref as the same category, we propose an inter-class contrastive learning loss function with label supervision.

[0089]

[0090] In this process, for a batch of data, if two samples belong to the same category, the similarity score between them will be maximized; conversely, if they do not belong to the same category, the similarity score between them will be minimized. However, severe data imbalance means that minority class samples may always be regarded as negative samples, which prevents the feature representation of minority classes from converging to a stable state. To this end, the similarity measure is modified as follows:

[0091]

[0092] When two samples belong to different categories, if the cosine similarity between them is below a certain threshold -η, the similarity score between them will be adjusted to zero or positive, which can avoid the endless mutual exclusion between negative samples. The goal is to ensure that the minority class is not overly influenced by the majority class. The threshold η is determined by the feature space and the number of categories, and can usually be determined by the minimum distance of the binary linear code.

[0093] S2. Cascaded coarse-to-fine re-ranking method

[0094] This innovation is mainly applied in the rough re-ranking module Mc. Given a candidate page \Tilde{P}, try to identify the most relevant part S* from the partial sequence \Tilde{S}:={S_1,...,S_{N_topk}}. In order to reduce the computational burden, a rough ranking is first performed to filter out paragraphs S that obviously cannot answer the question Q. Specifically, a pre-trained Contriever is used as Mc to evaluate the correlation between Q and S. By calculating the relevance score of each part to the question, the N parts with the highest scores are selected as the objects for the next step, where N is less than N_topk. Mc is further fine-tuned to improve performance in specific domain tasks. A training dataset consisting of plain text questions Q and parts S is constructed. Specifically, the annotated true parts are set as positive samples, and hard negative sample sampling is used to construct negative samples. Specifically, the unfine-tuned Mc is used to obtain the prediction score of the question Q. Then, the part with the highest corresponding score is selected from the non-true part as the hard negative sample (see Figure 3 , green represents the true label, while red represents the hard negative sample. The logit is obtained from the reasoning of the original coarse ranking module Mc on the query question Q). InfoNCE loss is used to train Mc.

[0095] LLaVA-v1.5 is used as Mr for the final fine rearrangement to take advantage of its excellent multimodal understanding ability. Mr can understand the information in the query image I_query, the question Q, and the background knowledge S at the same time. Specifically, the answer probability p of each candidate group (I_query, Q, S) is calculated using the score of the first corresponding word "yes" (or "no") output by Mr. If the background knowledge S can answer the question Q, then the score of "yes" will be higher, and vice versa.

[0096] In this way, the probability of each part S answering question Q can be calculated.

[0097]

[0098] Given a partial sequence {S_1,...,S_N}, all answer possibilities {p_1,...,p_N} are calculated, and the background knowledge S* with the highest p value is selected to generate the final result. Mr_r is further trained in a supervised fine-tuning manner. The positive instruction pairs Q with the background knowledge S containing the answer, while the negative instruction pairs Q with other paragraphs S on the same page P. Note that negative samples are mainly sampled from the same page, which is different from the coarse filtering stage, where the negative part is sampled from other pages. This is because the previous stage has achieved a satisfactory page recall rate, and the main challenge of the system lies in identifying the correct S* from the same page. The dataset maintains a 1:1 balanced ratio of positive and negative samples. Train the MLP projection module and large language model module of LLaVA-v1.5. The training goal of the model is language modeling with causal attention mask. By inputting the query image I_query, the question Q and the part S, and the previously generated answer A_ , the model predicts the probability of the next tag A_i.

[0099]

[0100] Answer Generation

[0101] Given background knowledge S*, an instruction fine-tuning approach is used to give the answer generator M_g the ability to answer knowledge-intensive questions based on contextual clues. Instructions are constructed using templates, combining question Q, answer A, and background knowledge S*. Whether to include the query image I_query at this stage depends on the base model. Through these instructions, M_g is guided to learn how to generate accurate answers based on the provided contextual information. During training, the same supervised fine-tuning method as the fine-tuning module is used to ensure that the model can effectively understand and utilize multimodal information. The training goal is still language modeling with causal attention masking. By inputting the query image, question, part, and previous answers, the model predicts the probability of the next tag.

[0102] Experimental Results

[0103] See also Figure 4 ,Experimental results show that the system of the present invention significantly outperforms the existing state-of-the-art methods on multiple datasets. Specifically: 1) On the EVQA dataset, the accuracy is improved by 10.2%. 2) On the InfoSeek dataset, the accuracy is improved by 11.3%. 3) Efficiency improvement: Through the cascaded coarse-to-fine re-ranking method, the computational overhead of the system is significantly reduced, achieving a nearly 75% reduction in inference time.

[0104] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0105] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0106] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0107] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0108] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0109] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0110] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0111] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0112] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0113] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0114] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0115] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0116] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A multi-modal large model retrieval question answering method with a cascade architecture, characterized in that: The following steps are involved: 1) Visual search: Retrieve multimodal information related to the query image and question description from the knowledge base, encode the features of the query image and reference images and titles in the knowledge base, and use vector search technology to select the pages most similar to the query image as candidate information sources; 2) Rough re-ranking: Filter out the paragraphs with high relevance to the question description from the candidate pages retrieved in step 1), and exclude obviously irrelevant paragraphs by evaluating the relevance score between the question description and the paragraphs; 3) Fine re-ranking: Conduct in-depth analysis of the paragraphs selected in step 2), use a multimodal large model to understand the relationship between the query image, question description, and paragraph content, and further rank and select the paragraphs most likely to contain the correct answer; 4) Answer generation: Generate the final answer based on the content of the paragraph selected in step 3) and the question description.

2. The multimodal large model retrieval question answering method according to claim 1, characterized in that: In step 1), multimodal information is retrieved from the knowledge base through visual query, which specifically includes: Extracting features of the query image; Matching with reference images and captions in the knowledge base; The features of the reference image and title are extracted by the visual encoder and text encoder respectively; Matching is performed by jointly embedding the space to obtain the top K candidate pages that are most similar to the query image, and the paragraphs in these pages are merged into a paragraph sequence.

3. The multimodal large model retrieval question answering method according to claim 2, characterized in that: In step 1), the visual search method further comprises: First, feature encoding is performed on the query image; The encoded features are then used to calculate similarity with the features of each entry in the knowledge base; Select the top K pages with the most similar features to the query image as candidates; Each candidate page is selected based on the encoded features of the reference image and title, where the encoded features are extracted by the visual encoder and text encoder respectively; The selection of candidate pages uses the FAISS library to perform vector search, ensuring that half of the candidate pages come from similarity matching in the image modality and the other half come from similarity matching in the text modality.

4. The multimodal large model retrieval question answering method according to claim 2 or 3, characterized in that: In step 1), the visual search method further comprises: Use the CLIP retriever and fine-tune it. Train CLIP using supervised contrast loss to accommodate a large number of entities and solve the long-tail problem. Adjust the loss term of negative samples to overcome the long-tail problem based on the optimal encoding distance to improve the retrieval accuracy of the model when dealing with rare entities; Using inter-class contrastive learning loss function with label supervision, matching query images, captions, and reference images are considered to be of the same category to enhance the model's ability to encode long-tail data or domain-specific data; A similarity score adjustment mechanism is implemented. For a batch of data, if two samples belong to the same category, the similarity score is maximized; if the two samples do not belong to the same category and the cosine similarity is lower than a set threshold, the similarity score is adjusted to zero or a positive number; preferably, the set threshold is based on the feature space and the number of categories, and is determined by the minimum distance of the binary linear code.

5. The multimodal large model retrieval question answering method according to any one of claims 1 to 4, characterized in that: In step 2), the text encoder Contriver is used as a rough sorter to perform preliminary screening on the retrieved paragraphs to exclude paragraphs that are obviously irrelevant to the query question.

6. The multimodal large model retrieval question answering method according to claim 5, characterized in that: In step 2), the rough rearrangement method specifically includes: Identify the most relevant partial sequence to the query question from a given candidate page; Perform a rough ranking to filter out passages that clearly do not answer the question; Use the pre-trained Contriver as a rough ranker to evaluate the relevance between the query and the paragraph; By calculating the relevance score, select the N parts with the highest scores as the objects for the next step of processing, where N is less than the total number of paragraphs in the candidate page; Among them, by constructing a training dataset containing plain text questions and paragraphs, the coarse ranger is fine-tuned to improve the performance in specific domain tasks; in the training dataset, the annotated true part is set as the positive sample, and the hard negative sample sampling technology is used to construct the negative sample, and the non-true part with the highest prediction score of the un-fine-tuned coarse ranger is selected as the hard negative sample; preferably, the InfoNCE loss is used to train the coarse ranger to optimize the accuracy of the relevance evaluation.

7. The multimodal large model retrieval question answering method according to any one of claims 1 to 6, characterized in that: In step 3), the multimodal model MLLM is used as a fine rearranger, and the paragraphs after rough rearrangement and filtering are input into the MLLM. The MLLM performs the final sorting and generation according to the query question and the paragraph content.

8. The multimodal large model retrieval question answering method according to claim 7, characterized in that: In step 3), the fine rearrangement method specifically includes: Determine the answer probability of each candidate group by calculating the score of the first corresponding word "yes" or "no" output by MLLM; In a given partial sequence, the background knowledge with the highest answer probability is selected as the basis for generating the final result; The fine-tuning approach is used to train the fine-tuned rearranger, where positive instructions pair questions with background knowledge containing answers, and negative instructions pair questions with other paragraphs on the same page, ensuring that negative samples are mostly from the same page. Maintain a 1:1 balance between positive and negative samples in the dataset, train the MLP projection module and large language model module of MLLM to achieve language modeling with causal attention mask; By inputting a query image, a question and some background knowledge, as well as previously generated answers, the model predicts the probability of the next token, thereby optimizing the accuracy of fine-grained reranking.

9. The multimodal large model retrieval question answering method according to any one of claims 1 to 8, characterized in that: In step 4), the answer generation method specifically includes: The answer generator generates the final answer by combining the finely rearranged background knowledge with the query question; Among them, by fine-tuning instructions, the ability of the answer generator to answer knowledge-intensive questions based on context clues is enhanced; template construction instructions are applied to guide the answer generator to generate accurate answers based on context information; During the training process, supervised fine-tuning is performed to enable the model to effectively integrate multimodal information and perform language modeling with causal attention masks. By inputting query images, questions, paragraphs, and previously generated answers, the model predicts and generates the next tag.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multimodal large model retrieval question and answer method as described in any one of claims 1 to 8 is implemented.

Images