Multi-modal multi-scale retrieval enhancement generation method, system and equipment applied to external knowledge questions and answers and medium

Abstract

The invention discloses a multi-modal multi-scale retrieval enhancement generation method, system and device applied to external knowledge questions and answers and a medium. The method comprises question perception, multi-modal multi-scale query fusion coding, dense recall and answer generation. Analyzing a key query phrase from the question through a fine-tuned instruction language model, and accurately positioning a region of interest corresponding to the phrase in an image by using an open set visual positioning model; multi-source information is compressed and distilled into an optimal query vector through a deep fusion network integrating multi-head self-attention and an information bottleneck theory; executing a maximum inner product search to recall related knowledge; guiding the large language model to synthesize all information to generate a final answer; the system, the equipment and the medium directly perform feature fusion in the vector space based on the method, so that challenges such as information loss and cascading errors caused by a traditional normal form can be effectively dealt with, high correlation and high accuracy of retrieval knowledge are ensured, and accurate and reliable image-text questions and answers are realized.

Description

Technical Field

[0001] This invention belongs to the field of multimodal technology at the intersection of computer vision and natural language processing, and specifically relates to a multimodal, multi-scale retrieval enhancement generation method, system, device and medium for external knowledge question answering. Background Technology

[0002] Visual question answering, a key technology integrating computer vision and natural language processing, aims to enable machines to understand visual content and answer related natural language questions. External knowledge-based visual question answering further requires models to use external knowledge bases to reason and generate accurate answers, based on their understanding of visual and textual information. The core challenge of this technology lies in effectively integrating and utilizing non-visual structured knowledge (such as facts, conceptual relationships, and common sense) to handle complex queries that are difficult to answer based solely on images and question text.

[0003] While large language models excel in language understanding and generation, their inherently static training data limits their ability to handle tasks requiring up-to-date information or deep expertise. To address this deficiency, retrieval-enhanced generative techniques have been proposed. These techniques retrieve relevant information fragments from external knowledge bases and input them along with the original query into the language model to generate more accurate and informative answers, thereby mitigating the "illusion" problem caused by outdated or limited knowledge.

[0004] In current research on visual question answering (VQA) for external knowledge, a common paradigm is to preprocess visual information into text descriptions. For example, tools such as object detection, optical character recognition (OCR), or image captioning are used to extract entities, text, and scene descriptions from images. Essentially, this transforms the multimodal VQA problem into a text-centric unimodal knowledge retrieval and question answering task. However, such methods have inherent limitations: visual information inevitably loses rich fine-grained visual details and global semantic information during the text-to-text conversion process, directly impacting the accuracy of knowledge retrieval. Patent application CN120104718A, by constructing adversarial example and noise classification mechanisms, can improve the robustness of large language models in handling pure text retrieval against relevant noise, irrelevant noise, and counterfactual noise. However, this solution is entirely limited to unimodal text processing and cannot cope with the growing demand for multimodal question answering. Patent application CN120632049A pre-segments document pages into "blocks" using a layout analysis model, enabling fine-grained matching at the "block level," which improves the accuracy and traceability of answers to some extent. However, the blocks in this method are predefined structural units, rather than key visual evidence dynamically perceived based on the semantics of the question. When dealing with complex, tightly coupled unstructured content, it cannot adaptively locate and focus on the most critical local visual details in the image. Summary of the Invention

[0005] To overcome the shortcomings of the existing technologies, this invention proposes a multimodal, multi-scale retrieval enhancement generation method, system, device, and medium for external knowledge question answering. By constructing an end-to-end model architecture, without prior text conversion, it aligns visual and textual modalities through a pre-trained CLIP model to achieve multimodal retrieval enhancement, avoiding the information loss risk of the traditional "image-to-text" paradigm. Furthermore, by focusing on key visual evidence and information bottleneck compression techniques, it significantly improves the accuracy and robustness of retrieved knowledge.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A multimodal, multi-scale retrieval enhancement generation method for external knowledge question answering specifically includes the following steps: Step 1, Perform query-guided visual perception: Receive a natural language question and the input raw image, locate and extract key regions of interest in the raw image based on the question; Step 2, perform multimodal multiscale query fusion encoding: extract features and perform deep fusion on the question, original image and region of interest described in Step 1, and generate the optimal query vector after information bottleneck compression; Step 3, perform intensive recall: using the optimal query vector described in Step 2, perform similarity retrieval in a pre-indexed external knowledge base to obtain the Top-K knowledge documents most relevant to the original multimodal query; Step 4, Execute large language model generation: Combine the Top-K knowledge documents described in Step 3 with the original multimodal query to generate the final answer.

[0007] Step 1 specifically includes: Step 1.1, Text Intent Parsing: Perform semantic analysis on the input natural language question to extract one or more key phrases representing the core intent of the question, i.e., the nouns or objects that need to be located and identified in the original image in the visual localization question. Construct a custom dataset consisting of a large number of "image-text question-answer pairs - core phrases" labeled pairs. Select a pre-trained instruction-based fine-tuning large language model as the base model, and fine-tune and train this model on the custom dataset containing "image-text question-answer pairs - core phrases" labeled pairs to obtain the fine-tuned language model. It can directly generate the most suitable core phrase for visual positioning from the question text Q based on a specific instruction template T. : Step 1.2, Visual Localization and Evidence Extraction: Extracting the core phrase... As a text prompt, with the original image Together, they are fed into a pre-trained open set detection visual localization model (Grounding DINO), which searches for targets in the image based on text prompts and outputs bounding boxes of regions that match the search targets, thereby extracting regions of interest; Step 1.2.1: The visual localization model first extracts the visual features of the image internally. and the linguistic features of the text Aligned visual features are obtained through a cross-modal feature enhancer. and the linguistic features of the text ; Step 1.2.2: Receive the enhanced visual features through a Transformer decoder. The model takes a set of learnable object queries as input; these learnable object queries are parameters within the model and are optimized layer by layer iteratively to accurately describe potential targets in the image, ultimately predicting a set of candidate bounding boxes. and its corresponding object embedding Each candidate bounding box Match confidence This is achieved by calculating the candidate bounding box. object embedding Text features The maximum similarity between them is obtained as follows: Step 1.2.3: Set a predefined confidence threshold. Filter out all confidence levels Candidate bounding boxes that exceed this threshold form an optimal bounding box set. ; Step 1.2.4, based on the optimal bounding box set The coordinates of each bounding box in the original image are respectively... Cropping is performed to extract one or more regions of interest (ROIs) that serve as local sources of visual evidence, denoted as... This region of interest (ROI) serves as a source of visual evidence representing local details in subsequent processing.

[0008] Step 2 specifically includes: Step 2.1, Multi-scale Feature Extraction: Text Encoder using a pre-trained cross-modal encoder and image encoder For the question text Q and the original image and areas of interest These three information sources are encoded in parallel to obtain text feature vectors, global visual feature vectors, and local visual feature vectors, respectively. Step 2.1.1, regarding the problem text in Step 1 Encoded, pre-trained weights of the CLIP model are efficiently fine-tuned using low-rank adaptive parameters. Keep frozen, and inject low-rank decomposition matrices A and B into a specific linear layer for a given input. The layer forward propagation update is as follows: in, , For rank and ; Obtain the text feature vector ; Step 2.1.2, process the original image and areas of interest Encode them separately to obtain global visual feature vectors. and local visual feature vectors ; Step 2.1.3 introduces three independent mapping networks to project and transform the visual feature vectors and text feature vectors to align the semantic spaces of different modalities. This allows information from different sources to be directly compared and merged, resulting in three aligned feature vectors within the same semantic space: visual features representing the global context of the scene. Visual features representing local details of an object and text features representing the semantics of the problem ; ; Step 2.2, Deep Feature Fusion: The three aligned feature vectors obtained in Step 2.1, representing "textual features of the problem semantics," "visual features of the global context," and "visual features of local details," are concatenated into an input sequence. ; Inject learnable modality codes (ME) and position codes (PE) into the sequence to form an input representation: ; Input representation It is fed into a deep fusion network consisting of N layers of stacked Transformer Encoders. Each Transformer Encoder layer consists of a multi-head self-attention sublayer and a position feedforward network sublayer, and each sublayer contains residual connections and layer normalization operations. The specific computation process is as follows: the multi-head self-attention sublayer processes the input... The query (Q), key (K), and value (V) spaces are mapped to the query (Q), key (K), and value (V) spaces respectively using trainable linear projection matrices. This process is performed in parallel for multiple attention heads. Then, the attention weights and outputs for each head are calculated: in, The dimension of the key vector is used to concatenate the outputs of all heads and then perform linear projection to obtain the final output. Intermediate features are obtained through residual connections and layer normalization. : Location-based feedforward network sublayers, for intermediate features Perform a nonlinear transformation, and obtain the first layer by residual connection and layer normalization. Layer output: ; in: in, , , , These are the trainable parameters of the network; This makes the visual features representing the global context of the scene... Visual features representing local details of an object and text features representing the semantics of the problem They can perform dynamic, fine-grained information interaction and weighted fusion, ultimately outputting a feature sequence that has undergone deep context fusion. : Step 2.3, the information bottleneck layer optimizes the combined loss function. The training process is ultimately formalized into a contrastive retrieval loss term. and information bottleneck regularization terms Weighted sum of the two items: = Among them, the comparison retrieval loss term Used to improve query vectors The discriminant power; its calculation formula is: Information bottleneck regularization term The encoder output distribution is then measured using KL divergence. With a standard normal distribution The degree of difference between them; functions Formalized as follows: .

[0009] Step 3 specifically includes: Step 3.1, Offline Index Building: Using a Transformer-based document encoder external knowledge base Every document Independently mapped to a fixed-dimensional, normalized dense vector ; Put all document vectors Store in FAISS vector database index This will provide support for subsequent intensive online recalls; Document encoder Single document from external knowledge base Encoded as a dense vector The process is formalized as follows: The entire external knowledge base was transformed into a... indivual A vector space composed of dimensional vectors; Step 3.2, Online Intensive Recall: Using cosine similarity as the query vector With document vectors A measure of relevance between them; for a given query vector With any document in an external knowledge base Similarity score The calculation is as follows: The optimal query vector with the highest similarity score. Submitted to the pre-built FAISS index ;index Perform maximum inner product search Find the vector in the vector space that matches the optimal query vector. closest Given document vectors; to find a subset that... The subset of indexes with the highest similarity scores between documents and queries. : This operation, which performs a maximum inner product search on the FAISS index, returns the indices of the Top-K documents and their corresponding similarity scores; ultimately, the system uses this subset of indices... From the original external knowledge base Recall the corresponding Top-K knowledge documents These documents and their similarity scores are then output as the most relevant external knowledge to the original text and image query, for use in subsequent answer generation.

[0010] Step 4 specifically includes: Step 4.1, Context Construction and Input Concatenation: Combine the Top-K knowledge documents output in Step 3. , and the original query information of the question text Q Integrate them into a single input sequence Construct an input context for downstream large-scale language generation models to understand and process: Each retrieved document (in Each of these can be independently constructed into an input sequence; the original query information With single document content The input sequence is constructed by concatenation. Represented as: [SEP] is a special separator. Step 4.2, Parallel Answer Generation and Probability Calculation: Calculate the K input sequences... Parallel input into pre-trained large-scale language generation models In the middle, for each input An autoregressive process generates a candidate answer. And calculate the conditional generation probability. For the first input sequences The answer with the highest generation probability Formalized as: Then, a set containing K candidate answers and their corresponding generation probabilities is obtained. ; Step 4.3, Joint Probability Reordering and Final Answer Selection: Combine each candidate answer generation probability and the documents it depends on. The original similarity score obtained during the retrieval phase. The joint score of the candidate answers is calculated; specifically, the log generation probability of the candidate answer is added to the log similarity score of the document it depends on. The final answer The candidate answer that maximizes the joint score is: A multimodal, multi-scale retrieval system for external knowledge question answering includes: The question awareness module, used in step 1, is responsible for receiving the natural language question and image input by the user, and locating and extracting key regions of interest in the original image. The multimodal, multiscale query fusion encoding module is used in step 2. It is responsible for feature extraction and deep fusion of the question, original image and region of interest mentioned in step 1, and generates the optimal query vector after information bottleneck compression. The document encoding module is used in step 3 to perform similarity retrieval in a pre-indexed external knowledge base using the generated optimal query vector, and to obtain the Top-K knowledge documents that are most relevant to the original text and image query. The generation module, used in step 4, integrates the original query information of the Top-K knowledge documents output in step 3 with the question text into an input sequence, which is then fed into a pre-trained large-scale language model in parallel to comprehensively understand visual, textual, and external knowledge and generate the final answer.

[0011] Compared with the prior art, the present invention has at least the following beneficial effects: This invention avoids the inherent defects of the existing "image-to-text" paradigm. By constructing a "feature-based vision" architecture that directly processes multimodal information in vector space, it effectively avoids irreversible information loss and cascading error risks in the process of converting images into text.

[0012] This invention introduces a "query-guided visual perception" module, enabling the model to adaptively locate and focus on the most critical visual evidence in an image before deep fusion, based on the semantic intent of the question text. This proactive, top-down attention mechanism effectively suppresses interference from irrelevant background information, significantly improves the signal-to-noise ratio of the input signal, makes the subsequent multimodal fusion process more targeted and accurate, and also enhances the interpretability of the model's decision path.

[0013] In the multimodal fusion stage, this invention sets an information bottleneck layer at the output of the deep fusion network to perform principled, task-oriented compression and distillation of the fused multi-scale features. This design can actively filter and suppress noise and redundancy in multimodal information, generating an optimal query vector with extremely high information density and strong robustness.

[0014] In summary, this invention proposes a multimodal, multi-scale retrieval enhancement generation method for external knowledge question answering, aiming to solve the "illusion" problem caused by the knowledge limitations of large language models and achieve accurate and reliable image-text question answering. This method significantly improves the accuracy of visual information utilization through a query-guided visual perception module; and generates higher-quality query representations through multi-scale input and information bottleneck fusion. By directly fusing features in the vector space, this method effectively addresses the challenges of information loss and cascading errors brought about by the traditional "image-to-text" paradigm, ensuring high relevance and accuracy of retrieved knowledge, and is suitable for building more reliable and interpretable knowledge-intensive multimodal question answering systems. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall process of the multimodal and multiscale retrieval enhancement method proposed in this invention.

[0016] Figure 2 This is a schematic diagram of the multimodal multiscale retrieval module of the present invention.

[0017] Figure 3 This is a schematic diagram of the multimodal and multiscale fusion module of the present invention. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to specific embodiments.

[0019] Step 1: In a specific application instance, the system receives a user's multimodal query, which includes an image. A natural language problem ,like Figure 2 The knowledge perception module is shown below. This step, as the system's "perception layer," aims to mimic human cognitive mechanisms, transforming the model's visual processing flow from passive global perception to an active, top-down attention-focusing process guided by linguistic intent. This process aims to establish a correspondence between text queries and visual space, and to isolate the visual sub-regions that provide the most information for answering the question.

[0020] Step 1.1, Text Intent Parsing: This sub-step aims to parse the core visual query intent from a loosely structured natural language question Q and quantify it into a structured text prompt that can be used by downstream visual models. To achieve this, we formalize this task as a "text-to-text" generation task, where the model's input is a complete natural language question, and the output is its corresponding core phrase. This task is accomplished by constructing a targeted fine-tuning task: We first create a custom dataset consisting of a large number of "VQA question -> core phrase" labeled pairs, where "VQA question" refers to the input "natural language question Q"; then, we use a pre-trained instruction to fine-tune a large language model. This serves as the base model, and the dataset is used to train and fine-tune it. After fine-tuning, the problem-aware model... It can directly generate the most suitable core phrase for visual localization from question Q based on a specific instruction template T. .

[0021] Step 1.2, the core phrase As a text prompt, it is fed along with the original input image I into a pre-trained visual localization model with open-set detection capabilities. This model was chosen to overcome the limitation of traditional closed-set detectors, which can only recognize predefined categories, enabling them to understand and locate visual targets described in any natural language. Internally, this process is systematically decomposed into three stages: feature extraction, cross-modal enhancement, and decoding prediction. First, independent visual and text backbone networks process the input image and text prompt respectively, extracting initial visual feature maps. and the linguistic features of the text Subsequently, these two sets of heterogeneous features are fed into a cross-modal feature enhancer, which is typically composed of multiple Transformer layers. Through self-attention and cross-attention mechanisms, visual and textual features can engage in deep information interaction and alignment, ultimately generating contextualized, mutually perceptive enhanced visual features. and Finally, a Transformer decoder receives a set of learnable object queries and enhanced visual features. By iteratively optimizing the object query into a precise description of potential targets in the image, a set of candidate bounding boxes is ultimately predicted. and its corresponding object embedding Match confidence score for each candidate box. By calculating its object embedding Text features The maximum similarity is obtained. This crucial step ensures a high semantic relevance between the detection results and the input text prompts, and its calculation formula is: in, It is the first Object embedding of candidate boxes It is the first in the enhanced text feature sequence Word embedding, It is the Sigmoid activation function, used to normalize dot product similarity into a confidence score.

[0022] Finally, a predefined confidence threshold is set. Filter out all confidence scores Candidate bounding boxes that exceed this threshold form an optimal bounding box set. Based on the coordinates of each bounding box in this set, the original image is cropped to extract one or more regions of interest sets that serve as local visual evidence sources, denoted as... This area will serve as a source of visual evidence representing local details in subsequent processing. For example... Figure 2 As shown, the model searches and matches within the original image based on the text prompt "jewelry". Figure 2 As the arrow points, the model accurately located the ring area on the person's finger and generated the corresponding bounding box.

[0023] Step 2, serving as the system's "understanding and fusion layer," aims to fuse, interact with, and compress the multi-source, heterogeneous information obtained in Step 1 through a deep network, ultimately generating an optimal query vector. This step is as follows Figure 2 As shown in the query encoder architecture diagram, this module first receives input from the knowledge-aware module, including the question text and image information. The entire architecture consists of a feature extraction and mapping section on the left and a multimodal, multi-scale fusion module on the right. Its specific processing flow is as follows: Step 2.1, Multi-scale Feature Extraction and Mapping: To simultaneously utilize the global context and local details of the image, the system employs a pre-trained cross-modal encoder for text encoding. and image encoder For the question text Q and the original image and areas of interest Three information sources are encoded in parallel. First, the question text Q is encoded to obtain the text feature vector. For the original image and areas of interest Encode them separately to obtain global visual feature vectors. and a set of local visual feature vectors To align the semantic spaces of different modalities, we further introduce three independent, trainable mapping networks to transform and project these features, aligning them in the same high-dimensional vector space. This ensures that information from different sources can be directly compared and merged. After this step, we obtain three feature vectors aligned in the same semantic space: , and .

[0024] Step 2.2, Deep Feature Fusion: such as Figure 3 As shown in the bottom “Multi-Modal Embedding” box, the input feature sequence is first processed here, taking the three aligned feature vectors obtained in the previous step. , , Concatenate into an input sequence At the same time, such as Figure 3 As indicated by the arrows on both sides at the bottom, the system uses addition operations. Learnable modality codes (ME) and positional codes (PE) are injected into the sequence to provide the model with prior knowledge about the source and order of each vector, thus obtaining the input representation. .Will It is fed into a deep fusion network consisting of N layers of stacked Transformer Encoders. In this network, each Transformer layer passes through its multi-head self-attention sublayer, making... , , The three feature sources can dynamically and finely interact and be weighted and fused, ultimately outputting a feature sequence that has undergone deep context fusion. .

[0025] Step 2.3, Information Bottleneck Compression: Compress the deep fusion feature sequence output from Step 2.2. Input is sent to an information bottleneck layer, i.e. Figure 3 The topmost trapezoidal module. This layer is optimized based on the variational information bottleneck theory. This theory is a learning principle whose goal is to learn a random encoder. This allows it to compress input features to the maximum extent while retaining the most critical information for downstream retrieval tasks. Unlike traditional linear output layers, the trapezoidal shape of this module symbolizes compression and purification. Through this optimization, it ultimately achieves... Figure 3 As shown by the arrow at the top, a single optimal query vector with extremely high information density and strong robustness is distilled. .

[0026] Step 3, Intensive Recall: This step acts as the system's "memory retrieval layer," responsible for efficiently locating relevant information from the massive knowledge base using the optimal query vector generated in Step 2. Figure 2 The dense paragraph retrieval module is shown.

[0027] Step 3.1, Offline Index Building: Before the system runs, we first build the external knowledge base. Perform one-time offline preprocessing and index building. For example... Figure 2 As shown on the bottom left, a pre-trained Transformer model is used as the document encoder. Each document in the knowledge base Independently mapped to a fixed-dimensional, normalized dense vector Ultimately, the entire knowledge base was transformed into a... indivual A high-dimensional vector space composed of multidimensional vectors, which is machine-understandable. Subsequently, to support fast retrieval of massive amounts of vectors, we store all document vectors and build a dedicated index based on FAISS. ,like Figure 2 As shown on the right, this provides sublinear time complexity support for subsequent maximum inner product search, thus laying the foundation for the system's high-performance online response.

[0028] Step 3.2, Online Intensive Recall: Generating the Optimal Query Vector Then, the system performs an online intensive recall operation. The core of this operation is to calculate the query vector. With all document vectors in the knowledge base The semantic relevance between them. We use cosine similarity as the query vector. With document vectors A measure of correlation between them. For any document Its relevance score to the query The calculation is as follows: The optimal query vector generated in step 2 Submit to the pre-built FAISS index .index It will efficiently perform a maximum inner product search to find the product of two vectors in the vector space. closest The goal of this process is to find a subset of the indexes. This ensures that the documents in this subset have the highest similarity score to the query. Ultimately, the system uses this index subset as the basis for its decision. From the original external knowledge base Recall the corresponding Top-K knowledge documents And output these documents along with their similarity scores, such as Figure 2 As shown in the bottom right corner, this serves as the external knowledge most relevant to the original text and image query, and is used in subsequent answer generation steps.

[0029] Step 4, Answer Generation: This step, acting as the system's "reasoning and expression layer," is responsible for integrating the retrieved information and generating the final natural language answer, such as... Figure 1 The enhanced generation section is shown.

[0030] Step 4.1. Context Building: For each retrieved document Each input sequence is constructed independently. This sequence is formed by concatenating the original query information with the content of a single document. We represent the query content containing the question text as follows: To clearly distinguish between queries and knowledge, a special separator [SEP] is inserted between them, which is the input sequence constructed for the j-th document. It can be represented as: In this way, we create K independent, context-defined inference paths for the generative model. Each input sequence... Each question contains a complete set of "information needed for the question and answer," that is, what the user wants to ask. And a possible source of knowledge The final constructed form is as follows Figure 1 As shown in the dashed box on the right.

[0031] Step 4.2. Parallel answer generation: We feed the K independent input sequences generated in step 4.1 into a pre-trained large language generation model in parallel. In this model, which acts as an answer generator, it will process each input sequence... An autoregressive process generates a candidate answer. The generation process is performed word-by-word, that is, during the generation of the first word... each word element At that time, the model will simultaneously use the input sequence and the already generated preorder lexicons As a condition, such as Figure 1 As shown in the dashed box on the right.

[0032] At the same time, the model will calculate and generate candidate answers under the given input conditions. Total conditional probability This probability is the product of the conditional probabilities of all words in the answer sequence, and its calculation formula is: During the decoding process, a greedy search strategy is adopted, that is, at each step, the word with the highest probability is selected to form the final candidate answer. .

[0033] After this step, we will obtain a set containing K candidate answers and their corresponding generation probabilities, formally represented as: .

[0034] Step 4.3, Joint Probability Reordering and Final Answer Selection: The decision to generate the final answer depends not only on the output probability of the generative model, but also on the credibility of the retrieval machine that provides the knowledge source.

[0035] Specifically, for each candidate answer We have two dimensions for scoring: retriever confidence, which is the knowledge documents it relies on. The original similarity score obtained in the retrieval phase of step 3 This score reflects the degree of matching between the knowledge source and the user's original query; generator confidence, i.e., the candidate answer... In the document Generation probability under context conditions This score reflects the logical consistency of the answer with the given evidence and the fluency of the text.

[0036] To ensure numerical stability during the calculation process, we typically sum the two scores in logarithmic space to calculate the joint score for each candidate answer. The final answer... This is the candidate answer that maximizes the joint score. The process can be formalized as follows: To quantitatively describe the innovative advantages of this invention, we conducted simulation experiments on the OKVQA dataset. The experiments compared the "feature-based vision" method of this invention with existing "text-based vision" retrieval models. Regarding recall: this metric measures whether the model can find the correct answer among the limited number of returned results. The model of this invention achieved an 83.51% success rate in finding the answer in the first 5 results and an 89.11% success rate in finding the answer in the first 10 results, comprehensively surpassing all baseline models. This verifies that the query vector generated by the multimodal fusion method of this invention has a stronger source-finding capability. Regarding precision: this metric measures how many of the limited number of results returned by the model are truly relevant. The model of this invention achieved an accuracy of 51.27% in the first 5 results and an accuracy of 49.32% in the first 10 results, also significantly outperforming all comparative models. This leading advantage means that the head search results returned by this invention have higher purity and contain fewer false positive irrelevant documents.

[0037] This invention proposes a novel multimodal retrieval enhancement generation framework using the above technical solutions. This framework significantly improves the accuracy of visual information utilization through a query-guided visual perception module; and generates higher-quality query representations through multi-scale input and information bottleneck fusion.

Claims

1. A multimodal, multi-scale retrieval enhancement generation method for external knowledge question answering, characterized in that, Specifically, the following steps are included: Step 1, Perform query-guided visual perception: Receive a natural language question and the input raw image, locate and extract key regions of interest in the raw image based on the question; Step 2, perform multimodal multiscale query fusion encoding: extract features and perform deep fusion on the question, original image and region of interest described in Step 1, and generate the optimal query vector after information bottleneck compression; Step 3, perform intensive recall: using the optimal query vector described in Step 2, perform similarity retrieval in a pre-indexed external knowledge base to obtain the Top-K knowledge documents most relevant to the original multimodal query; Step 4, Execute large language model generation: Combine the Top-K knowledge documents described in Step 3 with the original multimodal query to generate the final answer.

2. The multimodal, multi-scale retrieval enhancement generation method for external knowledge question answering according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1, Text Intent Parsing: Perform semantic analysis on the input natural language question to extract one or more key phrases representing the core intent of the question, i.e., the nouns or objects that need to be located and identified in the original image in the visual localization question. Construct a custom dataset consisting of a large number of "image-text question-answer pairs - core phrases" labeled pairs. Select a pre-trained instruction-based fine-tuning large language model as the base model, and fine-tune and train this model on the custom dataset containing "image-text question-answer pairs - core phrases" labeled pairs to obtain the fine-tuned language model. It can directly generate the most suitable core phrase for visual positioning from the question text Q based on a specific instruction template T. : Step 1.2, Visual Localization and Evidence Extraction: Extracting the core phrase... As a text prompt, with the original image Together, they are fed into a pre-trained open set detection visual localization model (Grounding DINO), which searches for targets in the image based on text prompts and outputs bounding boxes of regions that match the search targets, thereby extracting regions of interest; Step 1.2.1: The visual localization model first extracts the visual features of the image internally. and the linguistic features of the text Aligned visual features are obtained through a cross-modal feature enhancer. and the linguistic features of the text ; Step 1.2.2: Receive the enhanced visual features through a Transformer decoder. The model takes a set of learnable object queries as input; these learnable object queries are parameters within the model and are optimized layer by layer iteratively to accurately describe potential targets in the image, ultimately predicting a set of candidate bounding boxes. and its corresponding object embedding Each candidate bounding box Match confidence This is achieved by calculating the candidate bounding box. object embedding Text features The maximum similarity between them is obtained as follows: Step 1.2.3: Set a predefined confidence threshold. Filter out all confidence levels Candidate bounding boxes that exceed this threshold form an optimal bounding box set. ; Step 1.2.4, based on the optimal bounding box set The coordinates of each bounding box in the original image are respectively... Cropping is performed to extract one or more regions of interest (ROIs) that serve as local sources of visual evidence, denoted as... This region of interest (ROI) serves as a source of visual evidence representing local details in subsequent processing.

3. The multimodal, multi-scale retrieval enhancement generation method for external knowledge question answering according to claim 1, characterized in that, Step 2 specifically includes: Step 2.1, Multi-scale Feature Extraction: Text Encoder using a pre-trained cross-modal encoder and image encoder For the question text Q and the original image and areas of interest These three information sources are encoded in parallel to obtain text feature vectors, global visual feature vectors, and local visual feature vectors, respectively. Step 2.1.1, regarding the problem text in Step 1 Encoded, pre-trained weights of the CLIP model are efficiently fine-tuned using low-rank adaptive parameters. Keep frozen, and inject low-rank decomposition matrices A and B into a specific linear layer for a given input. The layer forward propagation update is as follows: in, , For rank and ; Obtain the text feature vector ; Step 2.1.2, process the original image and areas of interest Encode them separately to obtain global visual feature vectors. and local visual feature vectors ; Step 2.1.3 introduces three independent mapping networks to project and transform the visual feature vectors and text feature vectors to align the semantic spaces of different modalities. This allows information from different sources to be directly compared and merged, resulting in three aligned feature vectors within the same semantic space: visual features representing the global context of the scene. Visual features representing local details of an object and text features representing the semantics of the problem ; ； Step 2.2, Deep Feature Fusion: The three aligned feature vectors obtained in Step 2.1, representing "textual features of the problem semantics," "visual features of the global context," and "visual features of local details," are concatenated into an input sequence. ; Inject learnable modality codes (ME) and position codes (PE) into the sequence to form an input representation: ； Input representation It is fed into a deep fusion network consisting of N layers of stacked Transformer Encoders. Each Transformer Encoder layer consists of a multi-head self-attention sublayer and a position feedforward network sublayer, and each sublayer contains residual connections and layer normalization operations. The specific computation process is as follows: the multi-head self-attention sublayer processes the input... The query (Q), key (K), and value (V) spaces are mapped to the query (Q), key (K), and value (V) spaces respectively using trainable linear projection matrices. This process is performed in parallel for multiple attention heads. Then, the attention weights and outputs for each head are calculated: in, The dimension of the key vector is used to concatenate the outputs of all heads and then perform linear projection to obtain the final output. Intermediate features are obtained through residual connections and layer normalization. : Location-based feedforward network sublayers, for intermediate features Perform a nonlinear transformation, and obtain the first layer by residual connection and layer normalization. Layer output: ； in: in, , , , These are the trainable parameters of the network; This makes the visual features representing the global context of the scene... Visual features representing local details of an object and text features representing the semantics of the problem They can perform dynamic, fine-grained information interaction and weighted fusion, ultimately outputting a feature sequence that has undergone deep context fusion. : Step 2.3, the information bottleneck layer optimizes the combined loss function. The training process is ultimately formalized into a contrastive retrieval loss term. and information bottleneck regularization terms Weighted sum of the two items: = Among them, the comparison retrieval loss term Used to improve query vectors The discriminant power; its calculation formula is: Information bottleneck regularization term The encoder output distribution is then measured using KL divergence. With a standard normal distribution The degree of difference between them; functions Formalized as follows: 。 4. The multimodal, multi-scale retrieval enhancement generation method for external knowledge question answering according to claim 1, characterized in that, Step 3 specifically includes: Step 3.1, Offline Index Building: Using a Transformer-based document encoder external knowledge base Every document Independently mapped to a fixed-dimensional, normalized dense vector ; Put all document vectors Store in FAISS vector database index This will provide support for subsequent intensive online recalls; Document encoder Single document from external knowledge base Encoded as a dense vector The process is formalized as follows: The entire external knowledge base was transformed into a... indivual A vector space composed of dimensional vectors; Step 3.2, Online Intensive Recall: Using cosine similarity as the query vector With document vectors A measure of relevance between them; for a given query vector With any document in an external knowledge base Similarity score The calculation is as follows: The optimal query vector with the highest similarity score Submitted to the pre-built FAISS index ;index Perform maximum inner product search Find the vector in the vector space that matches the optimal query vector. closest Given document vectors; find a subset that... The subset of indexes with the highest similarity scores between documents and queries. : This operation, which performs a maximum inner product search on the FAISS index, returns the indices of the Top-K documents and their corresponding similarity scores; ultimately, the system uses this subset of indices... From the original external knowledge base Recall the corresponding Top-K knowledge documents These documents and their similarity scores are then output as the most relevant external knowledge to the original text and image query, for use in subsequent answer generation.

5. The multimodal, multi-scale retrieval enhancement generation method for external knowledge question answering according to claim 1, characterized in that, Step 4 specifically includes: Step 4.1, Context Construction and Input Concatenation: Combine the Top-K knowledge documents output in Step 3. , and the original query information of the question text Q Integrate them into a single input sequence Construct an input context for downstream large-scale language generation models to understand and process: Each retrieved document (in Each of these can be independently constructed into an input sequence; the original query information With single document content The input sequence is constructed by concatenation. Represented as: [SEP] is a special separator. Step 4.2, Parallel Answer Generation and Probability Calculation: Calculate the K input sequences... Parallel input into pre-trained large-scale language generation models In the middle, for each input An autoregressive process generates a candidate answer. And calculate the conditional generation probability. For the first input sequences The answer with the highest generation probability Formalized as: Then, a set containing K candidate answers and their corresponding generation probabilities is obtained. ; Step 4.3, Joint Probability Reordering and Final Answer Selection: Combine each candidate answer generation probability and the documents it depends on. The original similarity score obtained during the retrieval phase. The joint score of the candidate answers is calculated; specifically, the log generation probability of the candidate answer is added to the log similarity score of the document it depends on. The final answer The candidate answer that maximizes the joint score is: 。 6. A multimodal, multi-scale retrieval system for external knowledge question answering, used to implement the method described in any one of claims 1 to 5, characterized in that, include: The question awareness module, used in step 1, is responsible for receiving the natural language question and image input by the user, and locating and extracting key regions of interest in the original image. The multimodal, multiscale query fusion encoding module is used in step 2. It is responsible for feature extraction and deep fusion of the question, original image and region of interest mentioned in step 1, and generates the optimal query vector after information bottleneck compression. The document encoding module is used in step 3 to perform similarity retrieval in a pre-indexed external knowledge base using the generated optimal query vector, and to obtain the Top-K knowledge documents that are most relevant to the original text and image query. The generation module, used in step 4, integrates the original query information of the Top-K knowledge documents output in step 3 with the question text into an input sequence, which is then fed into a pre-trained large-scale language model in parallel to comprehensively understand visual, textual, and external knowledge and generate the final answer.

7. A multimodal, multi-scale retrieval device for external knowledge question answering, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program, implements multimodal, multiscale retrieval of external knowledge question answering according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables multimodal, multi-scale retrieval of external knowledge questions based on the method described in any one of claims 1 to 5.

Images