Visual question and answer model and method based on multi-modal retrieval enhancement

Through multimodal retrieval enhanced visual question-and-answer model, combined with area feature extraction, multimodal coding and graphic matching modules, the problem of knowledge retrieval in the existing technology ignores visual semantics, realizing the alignment of pictures and text and knowledge retrieval Improved accuracy.

CN119066174APending Publication Date: 2024-12-03SHANXI UNIV
View PDF 0 Cites 9 Cited by

Patent Information

Application Number
CN202411184961.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing visual question and answer models ignore the visual semantic information contained in the picture when searching knowledge, resulting in poor retrieval results and low accuracy of answers, and the lack of contextual information affects the accuracy of the search results.

Method used

Using a visual question-and-answer model based on multimodal retrieval enhancement, the area feature extraction module is used to identify problem keywords and detect image-related areas, and the text and visual features are encoded using multimodal encoding module, and the picture-text matching module is modally aligned. Weak supervision and comparison learning conduct knowledge retrieval and introduce the interpretation and description of problem keywords to improve the accuracy of knowledge retrieval.

Benefits of technology

The alignment of pictures and text modalities is realized, the time and resource consumption of pre-training of picture and text alignment is reduced, and the accuracy and accuracy of knowledge retrieval and answer generation are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066174A_ABST
    Figure CN119066174A_ABST
Patent Text Reader

Abstract

The invention discloses a visual question and answer model and method based on multi-modal retrieval enhancement, and belongs to the field of computer vision and natural language processing. The model comprises a region feature extraction module, a multi-modal coding module, an image-text matching module, a knowledge retrieval module and an answer generation module. According to the method, the region related to the question keyword in the picture is concerned, the whole picture and the text description of the picture region are taken as a bridge of image-text alignment, two modalities of the picture and the text are aligned in the knowledge retrieval process, the semantic association between the picture and the question is enhanced, and the knowledge retrieval and answer generation effects are improved; meanwhile, paraphrasing description of question keywords in a dictionary is introduced into knowledge retrieval and answer generation, and the retrieval and answer generation performance is further enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, natural language processing, and in particular to a visual question answering model and method based on multimodal retrieval enhancement. Background Art

[0002] In recent years, with the emergence of large-scale annotated visual question answering datasets, multimodal learning has made great progress. Visual Question Answering (VQA) is a very challenging task, which requires the VQA system to understand the semantic meaning and association between a given image and a natural language question, and to answer the question based on the knowledge it has mastered. With the development of visual language models, VQA systems have achieved remarkable results in traditional VQA tasks such as VQA and VQA2.0. However, real-world questions often cannot be answered based solely on the content of the image, and certain knowledge is required as support. Therefore, researchers have gradually begun to pay attention to the knowledge-based visual question answering (KBVQA) task, and proposed related datasets such as OK-VQA, AOKVQA, and FVQA, which have promoted the development of visual question answering and multimodal learning.

[0003] KBVQA requires that the VQA system can obtain the required knowledge from external knowledge sources and answer questions accordingly. Therefore, retrieving knowledge related to VQA images and questions becomes the key to solving the KBVQA task. In recent years, most research works have solved the KBVQA task from the perspective of retrieval enhancement. This type of method usually consists of two parts: a retrieval module and a reading module. The retrieval module is responsible for retrieving the required knowledge from external knowledge bases such as ConceptNet, Wikipedia, and Wikidata, and passing it to the generation module to generate the final answer. Among them, the retrieval module is mostly implemented based on dense passage retrieval (DPR) and its variants. DPR uses pre-trained language models such as BERT to encode queries and knowledge sets into separate vector representations, and then selects knowledge with a high matching score with the query by training DPR. In addition, there are some methods that acquire knowledge from large language models such as GPT-3 through contextual learning. The generation module generates answers based on the input knowledge, questions, and images using generation models such as T5, BLIP, and BLIP2.

[0004] Thanks to the powerful encoding ability of language models, solving KBVQA tasks based on retrieval enhancement has been proven to be an effective method. However, most existing methods ignore the visual semantic information contained in images when retrieving knowledge. They usually convert images into text information such as titles and target categories, and then only use a single text modality to retrieve knowledge in the knowledge base. Although some methods introduce visual information in images into knowledge retrieval, these methods require pre-training on large-scale image and text data to align the two modalities of image and text, which is very time-consuming and resource-intensive. In addition, the regional information in the image related to the question is very important for retrieving knowledge and aligning image and text. Many problems often focus on local areas in the image rather than the entire image. At the same time, due to the lack of contextual information, the retrieval module and the generation module cannot understand the background and semantics of the question well, which affects the accuracy of the retrieval results. Summary of the invention

[0005] In view of the problems of poor knowledge retrieval effect and low answering accuracy in current visual question answering models, the present invention provides a visual question answering model and method based on multimodal retrieval enhancement.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A visual question answering model based on multimodal retrieval enhancement includes a region feature extraction module, a multimodal encoding module, a picture-text matching module, a knowledge retrieval module and an answer generation module.

[0008] Furthermore, the regional feature extraction module is used to identify keywords in the question and detect image regions in the picture that are related to the keywords in the question; then, text titles are generated based on the pictures and picture regions, and these titles will serve as text descriptions of the pictures; the multimodal encoding module uses a language model and a visual model to encode text and pictures, and converts text features and visual features into vector representations; the image-text matching module uses the text description of the picture as a bridge and adopts an image-text matching task to align the two modes of picture and text; the knowledge retrieval module uses the question, picture and text description of the picture as queries, uses weakly supervised contrastive learning for training, and retrieves relevant knowledge in the knowledge base based on the input query; the answer generation module generates answers based on the input picture, question, keyword interpretation and retrieved knowledge.

[0009] A visual question answering method based on multimodal retrieval enhancement of the model, the method comprising the following steps:

[0010] Step 1: Identify the keywords in the question through the regional feature extraction module, then obtain the interpretation description of the keywords through WordNet, and further use the GroundingDINO model to detect the image regions related to the keywords in the image, and obtain the text description of the image and these image regions through the BLIP2 model;

[0011] Step 2: Encode questions, keyword explanations, images, image text descriptions, and text paragraphs in the knowledge base through a multimodal encoding module to convert text and visual features into vector representations;

[0012] Step 3: Train the image-text matching module through the image-text matching task to achieve alignment of the two modalities of text and image;

[0013] Step 4: Train the knowledge retrieval module through weakly supervised contrastive learning, and retrieve relevant knowledge in the knowledge base based on questions and pictures;

[0014] Step 5: Based on the retrieved knowledge, answers are generated through the answer generation module.

[0015] Further, the specific process of step 1 is as follows:

[0016] Step 1.1: Identify keywords in the question through the KeyBERT model;

[0017] Step 1.2: Retrieve the keyword’s meaning description W in WordNet;

[0018] Step 1.3: Use the open domain object detection model GroundingDINO to detect the image region R related to the keyword in the image I;

[0019] Step 1.4: Get the image I and the title C of the image region R through the visual language model BLIP2, and the title C will serve as a text description of the image.

[0020] Further, the specific process of step 2 is as follows:

[0021] Step 2.1: Connect the question Q, the image text description C, and the keyword interpretation description W to construct the text query X;

[0022] Step 2.2: Use the ColBERT model to get the vector representation of each token in the text query X and through a fully connected network Will Mapping Among them, L X express The number of tokens included, d, d qRespectively represent the size of the vector dimension, express The vector representation of the i-th text token in ;

[0023] X = [Q; C; W]

[0024]

[0025] Step 2.3: Use the frozen CLIP model to get the vector representation of image I Then through the two-layer multi-layer perceptron Will Mapping Among them, L V express The number of tokens included. express The vector representation of the i-th visual token in ;

[0026]

[0027] Step 2.4: Get the knowledge base through the ColBERT model The vector representation of the text paragraph p in L p express The number of tokens included;

[0028]

[0029] Further, the specific process of step 3 is as follows:

[0030] Step 3.1: Based on the starting position s of the image text description C in the text query, in the vector representation of the text query In the above example, we get the vector representation of the image text description C. Among them, L C express The number of tokens included;

[0031] Step 3.2: Train the image-text matching module through the image-text matching task; first, consider the image text description C corresponding to image I as a positive sample, and consider the text descriptions of other images in the training batch as negative samples; then, use the image-text comparison loss As a training target, the image-text matching module is trained to align the two modalities of image and text;

[0032]

[0033]

[0034] Among them, the sim(·) function is used to measure the similarity between the image text description C and the image I. They represent the vector representation of the image text description C and the image I obtained through the multimodal encoding module, respectively. express The positive sample of express Negative samples of Represents the set of vector representations of other image text descriptions in the same training batch.

[0035] Further, the specific process of step 4 is as follows:

[0036] Step 4.1: Use the DPR model in the knowledge base Retrieve the top-L text paragraphs related to the text query T. If a text paragraph contains the correct answer, the paragraph is considered a positive sample.

[0037] Step 4.2: Use other text paragraphs in the same training batch as negative samples and use contrastive loss To train the knowledge retrieval module to maximize the similarity between text queries and related paragraphs;

[0038]

[0039] in, express The positive sample of express Negative samples, B represents the set of negative samples in the same training batch;

[0040] Step 4.3: Contrasting the weakly supervised loss Comparison loss with images and text Perform joint optimization:

[0041]

[0042] Step 4.4: After the knowledge retrieval module is trained, the vector representation of the text query is and the vector representation of image I Connect in the Knowledge Base Retrieve the Top-K pieces of knowledge related to question Q and image I.

[0043] Further, the specific process of step 5 is as follows:

[0044] Step 5.1: Connect each piece of knowledge with the image I, question Q, and keyword interpretation W, and input them into the visual language model BLIP2;

[0045] Step 5.2: Generate a predicted answer based on the input knowledge, and select the predicted answer with the highest answer confidence as the final answer:

[0046]

[0047] in, represents the retrieved Top-K knowledge set, p k express The kth piece of knowledge in .

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] (1) The present invention uses image-text description as a bridge to achieve alignment of the two modalities of image and text in the process of knowledge retrieval, reducing the time and resource consumption of image-text alignment pre-training on large-scale image-text corpus.

[0050] (2) The present invention enhances the knowledge retrieval process by introducing text descriptions of areas in the image that are relevant to the question keywords, thereby improving the accuracy of image-text alignment and knowledge retrieval.

[0051] (3) The present invention introduces the interpretation description of the question keywords into the knowledge retrieval process, further improving the accuracy of knowledge retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a model framework diagram of the present invention;

[0053] Figure 2 Data samples used in the present invention;

[0054] Figure 3 is a data sample processed by the present invention;

[0055] Figure 4 The test results of the present invention are compared with those of the prior art. DETAILED DESCRIPTION

[0056] In order to gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description of the present invention. However, the present invention has multiple implementations and is not limited to the specific examples listed herein. The presentation of these examples is intended to deepen the comprehensive understanding of the disclosure of the present invention.

[0057] The method of the present invention comprises two main parts: a retriever and a reader. Given a picture-text question-answer pair, the task of the retriever is to retrieve the knowledge required to answer the question from the knowledge base based on the given picture and question, and the reader answers the question based on this knowledge. For the retriever, the present invention first uses a keyword recognition tool to extract keywords in the question, and extracts the interpretation description of the keywords from WordNet. At the same time, based on the given keywords, the present invention identifies the area related to it in the picture, and obtains the text title of the whole picture and the area through the visual language model. Then, the present invention maps the visual features and text features of the question and the picture to a unified representation space, and trains the retriever through comparative learning. At the same time, in order to align the two modes of picture and text, the present invention introduces an image matching module to align the two modes during the knowledge retrieval process. For the reader, the present invention constructs a reader based on the visual language model, and inputs the question, the picture and the relevant knowledge obtained by the retriever into the reader to obtain the answer.

[0058] The present invention is further described in detail below in conjunction with embodiments and drawings. The embodiments are only used to explain the present invention and are not used to limit the protection scope of the present invention.

[0059] Example 1

[0060] Figure 1 The model framework of the present invention is presented, which consists of five core modules: regional feature extraction module, multimodal encoding module, image-text matching module, knowledge retrieval module and answer generation module. The functions of each module are described in detail below:

[0061] Region Feature Extraction Module: This module aims to identify keywords in the question and detect image regions in the image that are related to the keywords in the question. In addition, this module also generates titles for the image and image regions, which will serve as text descriptions of the image.

[0062] Multimodal encoding module, which uses language model and visual model to encode text and pictures and convert pictures and text into vector representation.

[0063] The image-text matching module uses the text description of the image as a bridge to align the two modalities of image and text through the image-text matching task.

[0064] The knowledge retrieval module takes questions, images, and text descriptions of images as queries, uses weakly supervised contrastive learning for training, and retrieves relevant knowledge in the knowledge base based on the input query.

[0065] The answer generation module is used to generate answers based on the input images, questions, keyword interpretations and retrieved knowledge.

[0066] Example 2

[0067] Figure 2 is an example of a visual question answering dataset. Figure 2 As described above, the example consists of an image-text question-answer pair (I, Q, A), where I represents the image, Q represents the question "What is the item with the burners and knobs?", and A represents the answer "stove".

[0068] based on Figure 2 The data sample shown in the figure uses the model described in Example 1 to implement a visual question answering method based on multimodal retrieval enhancement, and the method includes:

[0069] Step 1: Identify the keywords in the question through the regional feature extraction module, then obtain the interpretation description of the keywords through WordNet, and further use the GroundingDINO model to detect the image regions related to the keywords in the image, and obtain the text description of the image and these image regions through the BLIP2 model;

[0070] The specific process is as follows:

[0071] Step 1.1: Identify keywords in question Q, such as “burners” and “knobs”, through the KeyBERT model;

[0072] Step 1.2: Retrieve the paraphrase description W of the key entity in WordNet, such as “burners: the electric range had one large burner and three smaller ones”, “knobs: around handle”;

[0073] Step 1.3: Use the open domain object detection model GroundingDINO to detect the image region R related to the keywords "burners" and "knobs" in the image I.

[0074] Step 1.4: Then, the visual language model BLIP2 is used to obtain the title C of the entire image I and the image region R, such as "kitchen with wooden cabinets and a blue counter top.", "a stove top oven sitting inside of a kitchen." Among them, the title C will serve as a text description of the image.

[0075] Step 2: The multimodal encoding module consists of a text encoder ColBERT and a visual encoder CLIP. First, the question Q, the text description C of the image, and the interpretation description W of the keyword are concatenated to construct a text query T. Then, the image I is used as the visual query. Finally, ColBERT is used to encode the text query

[0076] and Knowledge Base Encode the text paragraph p in , and use CLIP to encode the visual query. The specific steps are as follows:

[0077] Step 2.1: Connect the question Q, the image text description C, and the keyword interpretation description W to construct the text query X;

[0078] X={[CLS] <boq> Q <eoq> <boc> C <eoc> <bow> W <eow>[SEP]}

[0079] Among them, [CLS] and [SEP] represent the start and end symbols of the ColBERT model respectively. <boq> 、 <eoq>They represent the start and end of question Q respectively. <boc> 、 <eoc>They represent the start and end characters of the image text description C respectively. <bow> 、 <eow>They respectively represent the start and end of the keyword interpretation description W.

[0080] Step 2.2: Use the ColBERT model to get the vector representation of each token in the text query X and through a fully connected network Will Mapping

[0081] X = [Q; C; W]

[0082]

[0083] Among them, L X express The number of tokens included, d, d q Respectively represent the size of the vector dimension, express The vector representation of the i-th text token in .

[0084] Step 2.3: Use the frozen CLIP model to get the vector representation of image I Then through two layers of multi-layer perceptron Will Mapping L V express Number of tokens included:

[0085]

[0086] Among them, L V express The number of tokens included. express The vector representation of the i-th visual token in ;.

[0087] Step 2.4: Similar to step 2.2, obtain the knowledge base through the ColBERT model The vector representation of the text paragraph p in L p express The number of tokens included;

[0088]

[0089] Step 3: Train the image-text matching module through the image-text matching task to achieve alignment of the two modalities of text and image. In the image-text matching module, the image text description C is used as a bridge to introduce the image-text matching task to align the two modalities of image and text. The specific contents are as follows:

[0090] Step 3.1: Based on the starting position s of the image text description C in the text query, in the vector representation of the text query In the above example, we get the vector representation of the image text description C. Among them, L S express The number of tokens included;

[0091] Step 3.2: Train the image-text matching module through the image-text matching task; first, consider the image text description C corresponding to image I as a positive sample, and consider the text descriptions of other images in the training batch as negative samples; then, use the image-text comparison loss As a training target, the image-text matching module is trained to align the two modalities of image and text; the calculation process is as follows:

[0092]

[0093] Among them, the sim(·) function is used to measure the similarity between the image text description C and the image I. They represent the vector representation of the image text description C and the image I obtained through the multimodal encoding module, respectively. express The positive sample of express Negative samples of Represents the set of vector representations of other image text descriptions in the same training batch.

[0094] Step 4: Train the knowledge retrieval module through weakly supervised contrastive learning, and retrieve relevant knowledge from the knowledge base based on questions and pictures; the specific contents are as follows:

[0095] Step 4.1: Use the DPR model in the knowledge base Retrieve the top-L text paragraphs related to the text query T. If a text paragraph contains the correct answer, the paragraph is considered a positive sample.

[0096] Step 4.2: Similar to step 3.3, other text paragraphs in the same training batch are used as negative samples and contrast loss is used. To train the knowledge retrieval module to maximize the similarity between the text query and the relevant paragraphs; the calculation process is as follows:

[0097]

[0098] in, represents a positive sample, represents negative samples, and B represents the set of negative samples in the same training batch.

[0099] Step 4.3: Contrasting the weakly supervised loss Comparison loss with images and text Perform joint optimization:

[0100]

[0101] Step 4.4: After the knowledge retrieval module is trained, the vector representation of the text query is and the vector representation of image I Connect in the Knowledge Base Retrieve the Top-K pieces of knowledge related to question Q and image I.

[0102] Step 5: Based on the retrieved Top-K pieces of knowledge, generate relevant answers through the answer generation module. The specific process is as follows:

[0103] Step 5.1: Connect each piece of knowledge with the image I, question Q, and keyword interpretation W, and input them into the visual language model BLIP2;

[0104] Step 5.2: Generate a predicted answer based on the input knowledge, and select the predicted answer with the highest answer confidence as the final answer:

[0105]

[0106] in, represents the retrieved Top-K knowledge set, p k express The kth piece of knowledge in .

[0107] Example 3

[0108] Figure 3 The data after the visual question answering dataset is processed by the present invention and the examples of the retrieved knowledge.

[0109] The present invention first uses the regional feature extraction module in Example 1 to process the given picture and question. Then, the processed data is encoded by the trained multimodal encoding module, and the trained knowledge retrieval module is used to search for relevant knowledge in the knowledge base. Finally, this knowledge will be input into the reading module to generate the final answer. The specific process is as follows:

[0110] Step 1: Get the keyword "horse" in the question "What breed is this horse?" through the regional feature extraction module, and then get the definition of "horse" "solid-hoofedherbivorousquadruped domesticated since prehistoric times" through WordNet.

[0111] Step 2: Based on the keyword "horse", Figure 3 Detect image regions related to "horse" in a given image ( Figure 3 Then, through BLIP2, we get the picture title "a person petting the nose of a brown and white horse" and the picture area title "a horse standing on top of a grasscovered field".

[0112] Step 3: Concatenate the question "What breed is this horse?", the image title "a person petting thenose of a brown and white horse", the image region title "a horse standing on top of agrass covered field", and the keyword interpretation "horse: solid-hoofed herbivorous quadruped domesticated since prehistoric times" as a text query and input it into the text encoder ColBERT to obtain the vector representation of the text query;

[0113] Step 4: Input the image into the visual encoder CLIP to obtain the vector representation of the image;

[0114] Step 5: Connect the vector representation of the text query and the image, input them into the knowledge retrieval module, calculate their similarity with each piece of knowledge in the knowledge base, and select the top-5 pieces of knowledge with the highest similarity: "palomino horses are a colorbreed, exhibiting a yellow or gold coat...", "mustang horse breed information, history, videos, pictures horse facts history...", "the arabian or arab horsearabic dmg is a breed of horse…”, “arabian horse conformation and our breed standard…”, “appaloosa type spots in the arabian also seem to be athrowback to the particolor…”.

[0115] Step 6: Input the top 5 pieces of knowledge obtained in step 5 into the answer generation module together with the question, picture, and keyword interpretation. This module generates the corresponding answer and corresponding answer probability for each piece of knowledge. The results are as follows: "palomino, 0.89", "mustang, 0.24", "arabian, 0.38", "arabian, 0.53", "appaloosa, 0.12". Finally, "palomino" with the highest answer probability is selected as the final answer.

[0116] Example 4

[0117] Figure 4 The comparison results of the present invention and the existing methods on the test data set are shown, and the test indicators are Recall@5 and EM Score, where Recall@5 is used to evaluate the performance of knowledge retrieval and EM Score is used to evaluate the performance of answer generation.

[0118] Recall@5 refers to the probability that the top 5 retrieved knowledge items contain at least one relevant piece of knowledge.

[0119]

[0120] Among them, S represents the answer set, H(p k ,S) means if the retrieved knowledge p k Contains any answer in S, then H(p k ,S) is 1, otherwise it is 0.

[0121] EM score means that if the predicted answer is any answer in S, the predicted answer is considered correct.

[0122] like Figure 4 As shown, the Recall@5 and EM score of the best method FLMR are 89.32 and 67.85 respectively. FLMR is pre-trained on a large-scale image-text corpus to align the two modalities of image and text, which requires a lot of time and computing resources. Compared with FLMR, the present invention uses the text description of the picture as a bridge to achieve the alignment of the picture and text in the process of knowledge retrieval, reducing the time and computing power consumption caused by pre-training. Compared with FLMR, the Recall@5 and EM score of the present invention are increased by 0.49 and 1.47 respectively, indicating that it is effective to align the picture and text through the picture text description. In addition, when the paraphrase description of the question keywords is removed (w / o WordNet), the Recall@5 and EM score of the present invention decrease by 0.55 and 1.31 respectively, indicating that the paraphrase description of the keywords can further improve the performance of knowledge retrieval and answer generation.

[0123] In summary, the present invention proposes a visual question answering model and method based on multimodal retrieval enhancement. This method uses the text description of the image as a bridge for image-text alignment, and realizes the alignment of the image and text in the knowledge retrieval process, without the need for pre-training on large-scale image-text corpus, thereby reducing the time and computing power consumption caused by pre-training, while improving the performance of knowledge retrieval and answer generation. In addition, the present invention introduces the interpretation description of the question keywords into knowledge retrieval and answer generation, further improving the accuracy of knowledge retrieval and answer generation.

[0124] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.< / eow> < / bow> < / eoc> < / boc> < / eoq> < / boq> < / eow> < / bow> < / eoc> < / boc> < / eoq> < / boq>

Claims

1. A visual question answering model based on multimodal retrieval enhancement, characterized by: It includes regional feature extraction module, multimodal encoding module, image-text matching module, knowledge retrieval module and answer generation module.

2. The visual question answering model based on multimodal retrieval enhancement according to claim 1, characterized in that: The region feature extraction module is used to identify keywords in the question and detect the image region in the picture that is related to the keyword in the question; then, a text title is generated based on the picture and the picture region, and these titles will serve as text descriptions of the picture; The multimodal encoding module encodes text and pictures using a language model and a visual model, and converts text features and visual features into vector representations; The image-text matching module uses the text description of the image as a bridge and adopts the image-text matching task to align the two modalities of image and text; The knowledge retrieval module uses questions, images, and text descriptions of images as queries, uses weakly supervised contrastive learning for training, and retrieves relevant knowledge in the knowledge base based on the input query; The answer generation module generates answers based on the input pictures, questions, keyword definitions and retrieved knowledge.

3. A visual question answering method based on multimodal retrieval enhancement using any one of the models described in claims 1 to 2, characterized in that: The method comprises the following steps: Step 1: Identify the keywords in the question through the regional feature extraction module, then obtain the interpretation description of the keywords through WordNet, and further use the GroundingDINO model to detect the image regions related to the keywords in the image, and obtain the text description of the image and these image regions through the BLIP2 model; Step 2: Encode questions, keyword explanations, images, image text descriptions, and text paragraphs in the knowledge base through a multimodal encoding module to convert text and visual features into vector representations; Step 3: Train the image-text matching module through the image-text matching task to achieve alignment of the two modalities of text and image; Step 4: Train the knowledge retrieval module through weakly supervised contrastive learning, and retrieve relevant knowledge in the knowledge base based on questions and pictures; Step 5: Based on the retrieved knowledge, answers are generated through the answer generation module.

4. The visual question answering method based on multimodal retrieval enhancement according to claim 3, characterized in that: The specific process of step 1 is as follows: Step 1.1: Identify keywords in the question through the KeyBERT model; Step 1.2: Retrieve the keyword’s meaning description W in WordNet; Step 1.3: Use the open domain object detection model GroundingDINO to detect the image region R related to the keyword in the image I; Step 1.4: Get the image I and the title C of the image region R through the visual language model BLIP2, and the title C will serve as a text description of the image.

5. The visual question answering method based on multimodal retrieval enhancement according to claim 3, characterized in that: The specific process of step 2 is as follows: Step 2.1: Connect the question Q, the image text description C, and the keyword interpretation description W to construct the text query X; Step 2.2: Use the ColBERT model to get the vector representation of each token in the text query X And through a fully connected network Will Mapping Among them, L X express The number of tokens included, d, d q Respectively represent the size of the vector dimension, express The i-th text toke n The vector representation of ; X = [Q; C; W] Step 2.3: Use the frozen CLIP model to get the vector representation of image I Then through the two-layer multi-layer perceptron Will Mapping Among them, L V express The number of tokens included. express The vector representation of the i-th visual token in ; Step 2.4: Get the knowledge base through the ColBERT model The vector representation of the text paragraph p in L p express The number of tokens included; 6. The visual question answering method based on multimodal retrieval enhancement according to claim 3, characterized in that: The specific process of step 3 is as follows: Step 3.1: Based on the starting position s of the image text description C in the text query, in the vector representation of the text query In the above example, we get the vector representation of the image text description C. Among them, L C express Toke included n quantity; Step 3.2: Train the image-text matching module through the image-text matching task; first, consider the image text description C corresponding to image I as a positive sample, and consider the text descriptions of other images in the training batch as negative samples; then, use the image-text comparison loss As a training target, the image-text matching module is trained to align the two modalities of image and text; Among them, the sim(·) function is used to measure the similarity between the image text description C and the image I. They represent the vector representation of the image text description C and the image I obtained through the multimodal encoding module, respectively. express The positive sample of express Negative samples of Represents the set of vector representations of other image text descriptions in the same training batch.

7. The visual question answering method based on multimodal retrieval enhancement according to claim 3, characterized in that: The specific process of step 4 is as follows: Step 4.1: Use the DPR model in the knowledge base Retrieve the top-L text paragraphs related to the text query T. If a text paragraph contains the correct answer, the paragraph is considered a positive sample. Step 4.2: Use other text paragraphs in the same training batch as negative samples and use contrastive loss To train the knowledge retrieval module to maximize the similarity between text queries and related paragraphs; in, express The positive sample of express Negative samples, B represents the set of negative samples in the same training batch; Step 4.3: Contrasting the weakly supervised loss Comparison loss with images and text Perform joint optimization: Step 4.4: After the knowledge retrieval module is trained, the vector representation of the text query is and the vector representation of image I Connect in the Knowledge Base Retrieve the Top-K pieces of knowledge related to question Q and image I.

8. The visual question answering method based on multimodal retrieval enhancement according to claim 3, characterized in that: The specific process of step 5 is as follows: Step 5.1: Connect each piece of knowledge with the image I, question Q, and keyword interpretation W, and input them into the visual language model BLIP2; Step 5.2: Generate a predicted answer based on the input knowledge, and select the predicted answer with the highest answer confidence as the final answer: in, Represents the retrieved Top-K knowledge set, pk represents The kth piece of knowledge in .

Citation Information

Cited By

  • Visual knowledge question-answering method based on question-enhanced knowledge retrieval network

    CN119938864A

  • Retrieval enhancement generation method based on multi-modal document

    CN119988542A

  • A multimodal document-based retrieval augmentation generation method

    CN119988542B

  • Pictographic phrase paraphrase mining method and device, equipment, medium and product

    CN120068854A

  • Dialogue method and device based on multi-modal large model, equipment and storage medium

    CN120448507A