A multi-modal large model retrieval question and answer method of a cascade architecture

Abstract

The application discloses a kind of multi-modal big model retrieval question and answer methods of cascade architecture, to improve the accuracy and efficiency of question and answer system.The method comprises the following steps: firstly, through the visual search step, the vector search technology is used to retrieve multi-modal information related to query image and question description from knowledge base;Secondly, the paragraph with higher correlation with question description is filtered out in the rough rearrangement step, and irrelevant content is excluded;Then, the selected paragraph is further sorted and selected in the fine rearrangement step using multi-modal big model in-depth analysis paragraph, to further select the paragraph most likely to contain the correct answer;Finally, the final answer is generated according to the selected paragraph content and question description in the answer generation step.In addition, the application solves the long tail problem and the efficiency and performance contradiction in the reordering stage by using the mixed multi-modal retrieval strategy and the cascade coarse-to-fine reordering method, which significantly improves the system performance.The experimental results show that the application is significantly better than the existing most advanced method on multiple data sets.

Description

Technical Field

[0001] This invention relates to multimodal information retrieval and question answering technology, and in particular to a cascaded architecture multimodal large-scale model retrieval and question answering method. Background Technology

[0002] Question answering systems based on multimodal large models have become a research hotspot in the field of artificial intelligence in recent years. Multimodal large language models (MLLMs), pre-trained on massive amounts of network data, have demonstrated excellent understanding and generation capabilities, achieving increasingly significant progress in question answering applications. However, despite these models' breakthroughs in many tasks, they still face some significant challenges in practical applications: 1) Limited visual query information: Existing MLLMs often only utilize limited visual information when processing visual queries, which limits their performance in complex multimodal tasks. For example, in visual question answering tasks, the model may not be able to fully utilize subtle details in images. 2) Long-tail effect of training data: Data distribution on the internet is typically long-tailed, meaning a few common patterns occupy the majority of the data, while a large number of rare patterns are less common. This data distribution leads to poor model performance when processing rare patterns. 3) Lack of up-to-date knowledge: The knowledge of pre-trained models usually comes from a time window of the training data, meaning they may not have the latest information. This is a major limitation for tasks that require real-time knowledge updates (such as news summarization, question answering, etc.).

[0003] To address the aforementioned issues, Multimodal Retrieval Enhanced Generation (MRAG) has emerged. MRAG enhances the generative capabilities of MLLMs by dynamically retrieving relevant information from external knowledge bases (such as Wikipedia). Specifically, MRAG retrieves relevant text and image information from the knowledge base based on visual cues in the query, thereby providing more contextual support. This method has shown great potential in knowledge-intensive tasks, such as: 1) Visual question answering: improving the model's understanding and answering ability for complex questions by retrieving image-related knowledge. 2) InfoSeek: improving the accuracy and relevance of information retrieval by combining visual and textual information. Despite the significant theoretical advantages of MRAG, existing methods still have some obvious limitations. 1) Limitations of visual retrieval strategies: a) Unimodal matching: Existing visual retrieval methods often rely on unimodal matching (such as feature vector matching), ignoring multimodal cues in the knowledge base. For example, the feature vector matching method using FAISS only considers image features and ignores textual information. b) Long-tail problem: Existing retrieval tools (such as CLIP) perform poorly when handling long-tail visual content. Knowledge bases such as Wikipedia contain a large number of long-tail entities, which appear infrequently in the training data, making it difficult for the retrieval tool to match them accurately. 2) Efficiency and performance contradiction in the reordering stage: a) Use of dense encoders: Some methods use dense encoders (such as Contriever, BLIP-2, DPR) for paragraph reordering. Although these encoders perform well in terms of performance, their computational cost is high, affecting the overall efficiency of the system. b) MLLM as reorderer: Other methods directly use MLLM as the reorderer. Although MLLM has strong understanding capabilities, its inference speed is slow, becoming a bottleneck of the system.

[0004] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a cascaded architecture multimodal large model retrieval and question answering method.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A cascaded architecture multimodal large model retrieval and question answering method includes the following steps:

[0008] 1) Visual search: Retrieve multimodal information related to the query image and question description from the knowledge base. Encode the features of the query image and reference images and titles in the knowledge base, and use vector search technology to select the page most similar to the query image as the candidate information source.

[0009] 2) Coarse rearrangement: Filter paragraphs that are highly relevant to the problem description from the candidate pages retrieved in step 1), and exclude obviously irrelevant paragraphs by evaluating the relevance score between the problem description and the paragraphs;

[0010] 3) Fine-grained rearrangement: The paragraphs selected in step 2) are analyzed in depth. Multimodal large models are used to understand the relationship between the query image, question description and paragraph content, and further sort and select the paragraphs most likely to contain the correct answer;

[0011] 4) Answer generation: Based on the paragraph content selected in step 3) and the question description, generate the final answer.

[0012] Further, in step 1), multimodal information is retrieved from the knowledge base through visual query, specifically including:

[0013] Extract features from the query image;

[0014] Match with reference images and titles in the knowledge base;

[0015] Features of the reference image and the title were extracted using a visual encoder and a text encoder, respectively.

[0016] Matching is performed using the joint embedding space to obtain the top K candidate pages that are most similar to the query image, and the paragraphs in these pages are merged into a paragraph sequence.

[0017] Furthermore, in step 1), the visual search method further includes:

[0018] First, feature encoding is performed on the query image;

[0019] Then, the encoded features are compared with the features of each entry in the knowledge base to calculate similarity.

[0020] Select the top K pages that are most similar to the features of the query image as candidates;

[0021] The selection of each candidate page is based on the encoded features of the reference image and the title, where the encoded features are extracted by the visual encoder and the text encoder, respectively;

[0022] Candidate pages are selected using vector search based on the FAISS library, ensuring that half of the candidate pages are selected based on similarity matching of image modalities and the other half based on similarity matching of text modalities.

[0023] Furthermore, in step 1), the visual search method further includes:

[0024] The CLIP retrieval tool was used and fine-tuned. CLIP was trained using supervised contrastive loss to adapt to a large number of entities, thus addressing the long-tail problem.

[0025] Adjusting the loss term for negative samples, based on the optimal encoding distance, overcomes the long-tail problem and improves the retrieval accuracy of the model when dealing with rare entities;

[0026] Using a label-supervised inter-class contrastive learning loss function, matching query images, titles, and reference images are treated as belonging to the same category to enhance the model's ability to encode long-tail or domain-specific data.

[0027] A similarity score adjustment mechanism is implemented. For a batch of data, if two samples belong to the same category, the similarity score is maximized; if two samples do not belong to the same category and the cosine similarity is lower than a set threshold, the similarity score is adjusted to zero or a positive number. Preferably, the set threshold is based on the feature space and the number of categories, and is determined by the minimum distance of binary linear codes.

[0028] Furthermore, in step 2), the text encoder Contriver is used as a coarse sorter to perform preliminary screening of the retrieved paragraphs in order to exclude paragraphs that are obviously irrelevant to the query question.

[0029] Furthermore, in step 2), the coarse rearrangement method specifically includes:

[0030] Identify the sequence of parts from a given candidate page that is most relevant to the query question;

[0031] Perform a rough ranking to filter out paragraphs that clearly do not answer the question;

[0032] Use a pre-trained Contriver as a coarse-grained sorter to evaluate the relevance between the query question and the paragraph;

[0033] By calculating the relevance score, the N parts with the highest scores are selected as the objects for the next step of processing, where N is less than the total number of paragraphs in the candidate page.

[0034] Specifically, by constructing a training dataset containing plain text questions and paragraphs, the coarse re-ranker is fine-tuned to improve its performance on specific domain tasks. In the training dataset, labeled ground truth parts are set as positive samples, and a hard negative sample sampling technique is used to construct negative samples. The non-ground truth parts with the highest predicted scores by the un-fine-tuned coarse re-ranker are selected as hard negative samples. Preferably, the InfoNCE loss is used to train the coarse re-ranker to optimize the accuracy of relevance assessment.

[0035] Furthermore, in step 3), a multimodal model (MLLM) is used as a fine-grained rearranger. The paragraphs that have been roughly rearranged and filtered are input into the MLLM, which performs the final sorting and generation based on the query question and the paragraph content.

[0036] Furthermore, in step 3), the fine-grained rearrangement method specifically includes:

[0037] The likelihood of an answer for each candidate group is determined by calculating the score of the first corresponding word "yes" or "no" in the MLLM output.

[0038] In a given partial sequence, the background knowledge with the highest probability of answering is selected as the basis for generating the final result;

[0039] A supervised fine-tuning method is used to train the fine-tuning algorithm, where positive instructions match questions with background knowledge containing answers, and negative instructions match questions with other paragraphs on the same page, ensuring that negative samples mainly come from the same page.

[0040] Maintaining a 1:1 balance ratio of positive to negative samples in the dataset, train the MLP projection module and the large language model module of MLLM to achieve language modeling with causal attention mask;

[0041] By inputting a query image, a question, some background knowledge, and previously generated answers, the model predicts the probability of the next label, thereby optimizing the accuracy of fine-grained rearrangement.

[0042] Furthermore, in step 4), the answer generation method specifically includes:

[0043] The final answer is generated by an answer generator by combining information from finely rearranged background knowledge with the query question.

[0044] Among these measures, fine-tuning of instructions enhances the answer generator's ability to answer knowledge-intensive questions based on contextual clues; and the application of template-based instruction construction guides the answer generator to generate accurate answers based on contextual information.

[0045] During training, supervised fine-tuning is performed to enable the model to effectively integrate multimodal information and perform language modeling with causal attention masks. By inputting query images, questions, paragraphs, and previously generated answers, the model predicts and generates the next tag.

[0046] A computer program product includes a computer program that, when executed by a processor, implements the multimodal large-scale model retrieval and question answering method.

[0047] The present invention has the following beneficial effects:

[0048] This invention proposes an innovative cascaded architecture for a multimodal large-model retrieval question-answering method. This method significantly improves the performance and efficiency of the question-answering system through a series of optimization steps. First, a visual search step is employed, utilizing vector search technology to retrieve multimodal information closely related to the query image and question description from the knowledge base, thus overcoming the challenge of limited information utilization in existing models when processing visual queries. Next, a coarse reordering step effectively filters out paragraphs highly relevant to the question description. This step eliminates obviously irrelevant paragraphs by evaluating the relevance score between the question and the paragraphs, reducing the computational burden of subsequent processing. In the fine reordering step, the selected paragraphs are analyzed in depth using a multimodal large-model approach, further ranking and selecting the paragraphs most likely to contain the correct answer. This step improves the model's accuracy when processing long-tail visual content by optimizing the retrieval strategy. Finally, in the answer generation step, the final answer is generated based on the selected paragraph content and question description, ensuring the accuracy and relevance of the answer. This invention resolves the efficiency-performance contradiction in the reordering stage and reduces computational overhead by combining a multimodal retrieval strategy with a cascaded coarse-to-fine reordering method. Experiments have verified that the method of this invention significantly improves the accuracy on the EVQA and InfoSeek datasets and greatly reduces the inference time, which is of great significance for improving the performance and efficiency of multimodal large model question answering systems.

[0049] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0050] Figure 1 This is a task flowchart of the cascaded architecture multimodal large model retrieval and question answering method according to an embodiment of the present invention.

[0051] Figure 2 This is a system framework flowchart of the multimodal large model retrieval question answering method according to an embodiment of the present invention.

[0052] Figure 3 This is an example of difficult sample sampling in an embodiment of the present invention.

[0053] Figure 4 This is a comparison table of experimental results from embodiments of the present invention. Detailed Implementation

[0054] To address the problems existing in current MRAG methods, this invention proposes a cascaded architecture multimodal large model retrieval question answering method and designs an efficient cascaded multimodal retrieval enhancement generation (CasRAG) pipeline. This pipeline significantly improves the system's performance and efficiency by optimizing retrieval strategies and reordering methods.

[0055] See Figure 1 and Figure 2This invention provides a cascaded architecture multimodal large model retrieval and question answering method, comprising the following steps:

[0056] Step 1) Visual search: Retrieve multimodal information related to the query image and question description from the knowledge base. Encode the features of the query image and reference images and titles in the knowledge base, and use vector search technology to select the page most similar to the query image as the candidate information source.

[0057] In a preferred embodiment, retrieving multimodal information from a knowledge base via visual query specifically includes: extracting features of the query image; matching it with reference images and titles in the knowledge base; extracting features of the reference images and titles using a visual encoder and a text encoder, respectively; matching using a joint embedding space to obtain the top K candidate pages most similar to the query image, and merging the paragraphs in these pages into a paragraph sequence.

[0058] In a preferred embodiment, the visual search method further includes: first, feature encoding of the query image; then, similarity calculation using the encoded features and the features of each entry in the knowledge base; selecting the top K pages most similar to the query image features as candidates; the selection of each candidate page is based on the encoded features of the reference image and the title, wherein the encoded features are extracted by the visual encoder and the text encoder respectively; the selection of candidate pages utilizes the FAISS library for vector search to ensure that half of the candidate pages are from image modality similarity matching and the other half are from text modality similarity matching.

[0059] In a preferred embodiment, the visual search method further includes: using and fine-tuning a CLIP retrieval model, training the CLIP using supervised contrastive loss to adapt to a large number of entities and address the long-tail problem; adjusting the loss term for negative samples, based on the optimal encoding distance to overcome the long-tail problem and improve the model's retrieval accuracy when handling rare entities; using a label-supervised inter-class contrastive learning loss function to treat matched query images, titles, and reference images as belonging to the same category to enhance the model's encoding ability for long-tail data or domain-specific data; implementing a similarity score adjustment mechanism, where for a batch of data, if two samples belong to the same category, the similarity score is maximized; if two samples do not belong to the same category and the cosine similarity is below a set threshold, the similarity score is adjusted to zero or a positive number; preferably, the set threshold is based on the feature space and the number of categories and is determined by the minimum distance of binary linear codes.

[0060] Step 2) Rough Reordering: Select paragraphs that are highly relevant to the problem description from the candidate pages retrieved in Step 1), and exclude obviously irrelevant paragraphs by evaluating the relevance score between the problem description and the paragraphs.

[0061] In a preferred embodiment, a text encoder Contriver is used as a coarse reorderer to initially filter the retrieved paragraphs, excluding those clearly irrelevant to the query question. The coarse reordering method specifically includes: identifying the most relevant part sequences to the query question from a given candidate page; performing a coarse ranking to filter out paragraphs that clearly cannot answer the question; using a pre-trained Contriver as a coarse reorderer to evaluate the relevance between the query question and the paragraphs; selecting the N parts with the highest relevance scores as the next processing targets by calculating relevance scores, where N is less than the total number of paragraphs in the candidate page; wherein, by constructing a training dataset containing plain text questions and paragraphs, the coarse reorderer is fine-tuned to improve performance in a specific domain task; in the training dataset, labeled ground truth parts are set as positive samples, and a hard negative sample sampling technique is used to construct negative samples, selecting the non-ground truth parts with the highest predicted scores by the un-fine-tuned coarse reorderer as hard negative samples; preferably, InfoNCE loss is used to train the coarse reorderer to optimize the accuracy of relevance evaluation.

[0062] Step 3) Fine-grained rearrangement: The paragraphs selected in Step 2) are analyzed in depth. Multimodal large models are used to understand the relationship between the query image, question description and paragraph content, and the paragraphs are further sorted and selected to contain the most likely correct answer.

[0063] In a preferred embodiment, a multimodal model (MLLM) is used as a fine-grained reorderer. Paragraphs, after coarse reordering and filtering, are input into the MLLM, which performs final sorting and generation based on the query question and paragraph content. The fine-grained reordering method specifically includes: determining the answer probability of each candidate group by calculating the score of the first corresponding word "yes" or "no" in the MLLM output; selecting the background knowledge with the highest answer probability from a given partial sequence as the basis for generating the final result; training the fine-grained reorderer using supervised fine-tuning, where positive instructions pair the question with background knowledge containing the answer, and negative instructions pair the question with other paragraphs on the same page, ensuring that negative samples mainly come from the same page; maintaining a 1:1 balance between positive and negative samples in the dataset, training the MLLM's MLP projection module and large language model module to achieve language modeling with causal attention masks; and optimizing the accuracy of fine-grained reordering by inputting the query image, question, partial background knowledge, and previously generated answers, and predicting the probability of the next label.

[0064] Step 4) Answer generation: Based on the selected paragraph content in Step 3) and the question description, generate the final answer.

[0065] In a preferred embodiment, the answer generation method specifically includes: using information combining finely rearranged background knowledge with the query question, the answer generator generates the final answer; wherein, through instruction fine-tuning, the answer generator's ability to answer knowledge-intensive questions based on contextual clues is enhanced; template construction instructions are applied to guide the answer generator to generate accurate answers based on contextual information; during training, supervised fine-tuning is performed to enable the model to effectively integrate multimodal information, perform language modeling with causal attention masks, and predict and generate the next tag by inputting the query image, question, paragraph, and previously generated answers.

[0066] The cascaded architecture multimodal large model retrieval question answering method of this invention significantly improves the performance and efficiency of the system by combining multimodal retrieval strategies and cascaded coarse-to-fine reordering methods. First, this method improves retrieval accuracy by combining multimodal matching of reference images and titles, especially for handling long-tail questions. By fine-tuning the CLIP retrieval engine and optimizing negative sample loss, high retrieval accuracy is ensured when handling rare entities. Next, a lightweight text encoder (Contriver) is used for initial text filtering, effectively excluding paragraphs clearly irrelevant to the query question and reducing computational burden. In the fine reordering stage, the powerful understanding and generation capabilities of MLLM are utilized to perform in-depth analysis of the initially filtered paragraphs, ensuring the accuracy and relevance of the final results. Furthermore, through instructional and supervised fine-tuning, the answer generator can generate accurate answers based on contextual clues while maintaining high efficiency.

[0067] The following further describes specific embodiments and experimental verifications of the present invention.

[0068] This invention proposes a cascaded architecture multimodal large model retrieval question answering method and designs a cascaded architecture-based MRAG system, which specifically includes two important parts: S1 hybrid multimodal retrieval strategy and S2 cascaded coarse-to-fine reordering method.

[0069] S1. Hybrid Multimodal Retrieval Strategy

[0070] S1.1 Combining reference images and titles

[0071] Multimodal matching: Traditional visual retrieval methods typically rely on unimodal matching (such as feature vector matching), which limits retrieval accuracy. This invention proposes a hybrid multimodal retrieval strategy that significantly improves retrieval accuracy by matching the query image with reference images and titles on Wikipedia pages.

[0072] Implementation details: During the retrieval process, features of the query image are first extracted, and then matched against reference images and titles on the wiki page. Features of the reference images and titles are extracted using a visual encoder and a text encoder, respectively, and finally matched using a joint embedding space.

[0073] S1.2 Custom CLIP Retrieval Tool

[0074] Fine-tuning CLIP: To better handle the long-tail problem in the knowledge base, this invention fine-tunes the general CLIP retrieval tool. Specifically, CLIP is trained using supervised contrastive loss to adapt to a large number of entities.

[0075] Optimize negative sample loss: During training, the long-tail problem is overcome by adjusting the loss term for negative samples based on the optimal encoding distance. This ensures that the model maintains high retrieval accuracy even when dealing with rare entities.

[0076] S2. Cascaded coarse-to-fine reordering method

[0077] S2.1 Contriver as a coarse sorter

[0078] Text filtering: Existing reordering methods typically use dense encoders or MLLMs, which are computationally expensive. This invention proposes a cascaded coarse-to-fine reordering method, which first uses a Contriver to perform preliminary text filtering, excluding obviously irrelevant content.

[0079] Implementation details: Contriver is a lightweight text encoder that efficiently performs initial filtering of retrieved paragraphs. By training Contriver, it can quickly identify and filter out irrelevant paragraphs based on the query. For example, when processing queries about "buildings," Contriver can effectively exclude paragraphs unrelated to architecture.

[0080] S2.2 MLLM as a fine reorderer

[0081] Fine-grained reordering: Building upon the initial filtering, MLLM is used for fine-grained reordering to ensure the accuracy and relevance of the final results. MLLM has powerful understanding and generation capabilities, enabling it to generate high-quality answers based on the query and the filtered paragraphs.

[0082] Implementation details: The paragraphs filtered by Contriver are input into MLLM, which then performs final sorting and generation based on the query question and paragraph content. This approach ensures both the quality of the generated results and significantly improves system efficiency.

[0083] The following is a detailed description.

[0084] like Figure 1 As shown, previous visual question answering tasks required model M to provide answer A for a given query image I_query and question Q. In Multimodal Retrieval and Generation (MRAG), the correct answer A cannot be derived solely from I_query or Q. This necessitates the model retrieving external information from the knowledge base memory (KB) to form the final response. KB contains N_page wiki pages P, each page P including an entity title T_title, a reference image I_ref, and N_sec^i segmented paragraphs S, i.e., P_i := {T_title; I_ref; S_1,...,S_N_sec^i}, where i belongs to N_page. Ideally, model M should retrieve the most informative wiki page P_i and its segments S_i,j before generating the answer to assist in making the correct response.

[0085] Utilizing MLLM for multimodal retrieval and generation tasks. For example... Figure 2 As shown, this invention comprises three levels of retrieval steps and an answer generation process, executed by four modules respectively, namely M:={M_v;M_c;M_r;M_g}. 1) Visual search Cable module M_v uses visual query I_query to retrieve multimodal information, namely T_title and I_ref, from all pages P in the knowledge base KB. Based on the features extracted by M_v, the top K candidate pages P:={P_1,...,P_K} are obtained, and all paragraphs in P are concatenated into a single paragraph sequence S:={S_1,...S_N_topk}, where N_topk=∑(i\in K)N_sec^i. 2) Roughly rearranged modules M_c uses the verification module to filter out paragraphs that are clearly irrelevant to question Q, thereby reducing the computational burden on the fine rearranger. 3) Fine Rearrangement Module M_r uses MLLM to determine whether a query {I_query; Q} based on paragraph S is answerable. 4) Answer generation module M_g uses the paragraph S* with the highest confidence score as external knowledge to provide the final answer A.

[0086] S1 Hybrid Multimodal Retrieval Strategy

[0087] This innovation is primarily applied to the visual search module M_v. The goal is to find paragraphs S* containing key information. However, given the vast amount of knowledge base content, direct searching is inefficient. Therefore, the system first retrieves matching titles T_title or reference images I_ref, and then searches for the desired portion within the corresponding page P. CLIP is used as M_v to encode multimodal features, and the FAISS library is employed for vector search. Specifically, based on the features encoded in the query image I_query, similarity is calculated between the query image I_query and each entry in the knowledge base. The top K pages most similar to the query image are selected as candidates, where K is much smaller than the total number of pages N_page. Each entry consists of the encoded features of the reference image I_ref and the encoded features of the title T_title. Therefore, the top K pages with the closest titles or reference images are selected as candidates, half from the image modality and the other half from the text modality.

[0088] Further fine-tuning of M_v addresses its limitations in encoding long-tailed or domain-specific data. By treating the matched query image I_query, title T_title, and reference image I_ref as belonging to the same category, this invention proposes a label-supervised inter-class contrastive learning loss function.

[0089]

[0090] In this process, for a batch of data, if two samples belong to the same class, their similarity score is maximized; conversely, if they do not belong to the same class, their similarity score is minimized. However, severe data imbalance means that minority class samples may always be treated as negative samples, which prevents the feature representations of the minority class from converging to a stable state. Therefore, the similarity measurement method is modified as follows:

[0091]

[0092] When two samples belong to different categories, if their cosine similarity is below a certain threshold -η, their similarity score is adjusted to zero or a positive number. This prevents negative samples from endlessly repelling each other. The goal is to ensure that the minority class is not overly influenced by the majority class. The threshold η is determined by the feature space and the number of categories, and is typically determined by the minimum distance of binary linear codes.

[0093] S2. Cascaded coarse-to-fine reordering method

[0094] This innovation is primarily applied to the coarse reordering module M_c. Given a candidate page \Tilde{P}, the task is to identify the most relevant part S* from the partial sequence \Tilde{S}:={S_1,...,S_{N_topk}}. To reduce computational burden, a coarse ranking is first performed to filter out paragraphs S that obviously cannot answer question Q. Specifically, a pre-trained Contriever is used as M_c to evaluate the relevance between Q and S. By calculating the relevance score of each part to the question, the N parts with the highest scores are selected as the next steps, where N is less than N_topk. M_c is further fine-tuned to improve performance on domain-specific tasks. A training dataset consisting of a plain text question Q and parts S is constructed. Specifically, labeled ground truth parts are set as positive samples, and hard negative sample sampling is used to construct negative samples. Specifically, the predicted score of question Q is obtained using the un-fine-tuned M_c. Then, the part with the highest corresponding score from the non-ground truth parts is selected as the hard negative sample (see...). Figure 3 Green represents the true labels, while red represents hard negative samples. The logit is derived from the inference of the original coarse ranking module Mc regarding the query question Q. InfoNCE loss is used to train M_c.

[0095] The final fine-grained ranking is performed using LLaVA-v1.5 as M_r to leverage its excellent multimodal understanding capabilities. M_r can simultaneously understand information from the query image I_query, the question Q, and the background knowledge S. Specifically, the answer probability p for each candidate group (I_query, Q, S) is calculated using the score of the first corresponding word "yes" (or "no") in the M_r output. If the background knowledge S can answer the question Q, then the "yes" score will be higher, and vice versa.

[0096] In this way, the probability of each part S answering question Q can be calculated.

[0097]

[0098] Given a partial sequence {S_1,...,S_N}, calculate all possible responses {p_1,...,p_N} and select the background knowledge S* with the highest p-value to generate the final result. M_r is further trained in a supervised fine-tuning manner. Positive instructions pair Q with the background knowledge S containing the answer, while negative instructions pair Q with other paragraphs S on the same page P. Note that negative samples are primarily sampled from within the same page, unlike the coarse filtering stage where negative parts are drawn from other pages. This is because the previous stages have achieved satisfactory page recall; the main challenge of the system is identifying the correct S* from within the same page. The dataset maintains a 1:1 balance between positive and negative samples. The MLP projection module and the large language model module of LLaVA-v1.5 are trained. The training objective of the model is language modeling with causal attention masks. The input is a query image I_query, a question Q, and a partial S, along with the previously generated answer A_ The model predicts the probability of the next label A_i.

[0099]

[0100] Answer generation

[0101] Given background knowledge S*, an instruction fine-tuning approach is used to empower the answer generator M_g with the ability to answer knowledge-intensive questions based on contextual cues. Instructions are constructed using templates, combining the question Q, the answer A, and the background knowledge S*. Whether the query image I_query is included at this stage depends on the base model. These instructions guide M_g to learn how to generate accurate answers based on the provided contextual information. During training, the same supervised fine-tuning method as the fine-tuning module is used to ensure the model can effectively understand and utilize multimodal information. The training objective remains language modeling with causal attention masks; given the query image, question, part, and previous answers as input, the model predicts the probability of the next tag.

[0102] Experimental results

[0103] See Figure 4 Experimental results show that the system of this invention significantly outperforms existing state-of-the-art methods on multiple datasets. Specifically: 1) On the EVQA dataset, accuracy is improved by 10.2%. 2) On the InfoSeek dataset, accuracy is improved by 11.3%. 3) Efficiency improvement: Through a cascaded coarse-to-fine reordering method, the computational overhead of the system is significantly reduced, achieving a reduction of nearly 75% inference time.

[0104] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0105] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0106] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0107] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0108] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0109] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0111] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0113] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0114] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0115] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0116] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A cascaded architecture multimodal large-scale model retrieval and question answering method, characterized in that, Includes the following steps: 1) Visual search: Retrieve multimodal information related to the query image and question description from the knowledge base. Encode the features of the query image and reference images and titles in the knowledge base, and use vector search technology to select the page most similar to the query image as the candidate information source. 2) Coarse rearrangement: Filter paragraphs that are highly relevant to the problem description from the candidate pages retrieved in step 1), and exclude obviously irrelevant paragraphs by evaluating the relevance score between the problem description and the paragraphs; The coarse rearrangement methods specifically include: Identify the sequence of parts from a given candidate page that is most relevant to the query question; Perform a rough ranking to filter out paragraphs that clearly do not answer the question; Use a coarse-weighted sorter to evaluate the relevance between the query question and the paragraph; By calculating the relevance score, the N parts with the highest scores are selected as the objects for the next step of processing, where N is less than the total number of paragraphs in the candidate page. 3) Fine-grained rearrangement: The paragraphs selected in step 2) are analyzed in depth. Multimodal large models are used to understand the relationship between the query image, question description and paragraph content, and further sort and select the paragraphs most likely to contain the correct answer; 4) Answer generation: Based on the paragraph content selected in step 3) and the question description, generate the final answer.

2. The multimodal large-scale model retrieval and question answering method as described in claim 1, characterized in that, In step 1), multimodal information is retrieved from the knowledge base through visual query, specifically including: Extract features from the query image; Match with reference images and titles in the knowledge base; Features of the reference image and the title were extracted using a visual encoder and a text encoder, respectively. Matching is performed using the joint embedding space to obtain the top K candidate pages that are most similar to the query image, and the paragraphs in these pages are merged into a paragraph sequence.

3. The multimodal large-scale model retrieval and question answering method as described in claim 2, characterized in that, In step 1), the visual search method further includes: First, feature encoding is performed on the query image; Then, the encoded features are compared with the features of each entry in the knowledge base to calculate similarity. Select the top K pages that are most similar to the features of the query image as candidates; The selection of each candidate page is based on the encoded features of the reference image and the title, where the encoded features are extracted by the visual encoder and the text encoder, respectively; Candidate pages are selected using vector search based on the FAISS library, ensuring that half of the candidate pages are selected based on similarity matching of image modalities and the other half based on similarity matching of text modalities.

4. The multimodal large-scale model retrieval and question-answering method as described in claim 2 or 3, characterized in that, In step 1), the visual search method further includes: The CLIP retrieval tool was used and fine-tuned. CLIP was trained using supervised contrastive loss to adapt to a large number of entities, thus addressing the long-tail problem. Adjusting the loss term for negative samples, based on the optimal encoding distance, overcomes the long-tail problem and improves the retrieval accuracy of the model when dealing with rare entities; Using a label-supervised inter-class contrastive learning loss function, matching query images, titles, and reference images are treated as belonging to the same category to enhance the model's ability to encode long-tail or domain-specific data. A similarity score adjustment mechanism is implemented. For a batch of data, if two samples belong to the same category, the similarity score is maximized; if two samples do not belong to the same category and the cosine similarity is lower than the set threshold, the similarity score is adjusted to zero or a positive number.

5. The multimodal large-scale model retrieval and question answering method as described in claim 4, characterized in that, The set threshold is based on the feature space and the number of categories, and is determined by the minimum distance of the binary linear code.

6. The multimodal large-scale model retrieval and question answering method as described in any one of claims 1 to 3, characterized in that, In step 2), a pre-trained text encoder Contriever is used as the coarse sorter to evaluate the relevance between the query question and the paragraphs, and to perform preliminary screening of the retrieved paragraphs to exclude paragraphs that are obviously irrelevant to the query question.

7. The multimodal large-scale model retrieval and question answering method as described in claim 6, characterized in that, Step 2) further includes the coarse rearrangement method as follows: By constructing a training dataset containing plain text questions and paragraphs, the coarse-weighted ranking machine is fine-tuned to improve performance on domain-specific tasks. In the training dataset, labeled ground truth parts are set as positive samples, and a hard negative sample sampling technique is used to construct negative samples. The non-ground truth parts with the highest predicted scores by the un-fine-tuned coarse-weighted ranking machine are selected as hard negative samples.

8. The multimodal large-scale model retrieval and question answering method as described in claim 7, characterized in that, InfoNCE loss is used to train a coarse reorderer to optimize the accuracy of relevance assessment.

9. The multimodal large-scale model retrieval and question answering method as described in any one of claims 1 to 3, characterized in that, In step 3), the Multimodal Model (MLLM) is used as a fine-grained rearranger. The paragraphs that have been coarsely rearranged and filtered are input into the MLLM, which performs the final sorting and generation based on the query question and the paragraph content.

10. The multimodal large-scale model retrieval and question answering method as described in claim 9, characterized in that, In step 3), the fine-grained rearrangement method specifically includes: The likelihood of an answer for each candidate group is determined by calculating the score of the first corresponding word "yes" or "no" in the MLLM output. In a given partial sequence, the background knowledge with the highest probability of answering is selected as the basis for generating the final result; The fine-tuning reordering tool is trained using a supervised fine-tuning approach, where positive instructions pair questions with background knowledge containing answers, and negative instructions pair questions with other paragraphs on the same page. Maintaining a 1:1 balance ratio of positive to negative samples in the dataset, train the MLP projection module and the large language model module of MLLM to achieve language modeling with causal attention mask; By inputting a query image, a question, some background knowledge, and previously generated answers, the model predicts the probability of the next label, thereby optimizing the accuracy of fine-grained rearrangement.

11. The multimodal large-scale model retrieval and question answering method as described in any one of claims 1 to 3, characterized in that, In step 4), the answer generation method specifically includes: The final answer is generated by an answer generator by combining information from finely rearranged background knowledge with the query question. Among these measures, fine-tuning of instructions enhances the answer generator's ability to answer knowledge-intensive questions based on contextual clues; and the application of template-based instruction construction guides the answer generator to generate accurate answers based on contextual information. During training, supervised fine-tuning is performed to enable the model to effectively integrate multimodal information and perform language modeling with causal attention masks. By inputting query images, questions, paragraphs, and previously generated answers, the model predicts and generates the next tag.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal large model retrieval question answering method as described in any one of claims 1 to 11.

Images