A Visual Knowledge Reasoning Question Answering Method Enhanced by Multi-Source Heterogeneous Knowledge Joint

Abstract

This invention discloses a visual knowledge reasoning question-answering method enhanced by multi-source heterogeneous knowledge. First, a multimodal model is used to generate text descriptions of images and candidate answers to questions as auxiliary information for multi-source heterogeneous knowledge retrieval, improving the quality of the retrieved knowledge. Specifically, four types of heterogeneous knowledge are retrieved: Wikipedia knowledge, relevant webpage text knowledge, relevant webpage image knowledge, and large-scale language model knowledge. These multi-source heterogeneous knowledge complement each other, contributing to the performance improvement of the visual question-answering system. Based on this multi-source heterogeneous knowledge, a heterogeneous knowledge fusion reasoning model is trained, improving the quality and accuracy of the question-answering model's answers. This enables the question-answering model to answer questions requiring external knowledge, expanding the application scenarios of the question-answering model.

Description

Technical Field

[0001] This invention belongs to the field of visual question answering, specifically involving the joint use of multi-source heterogeneous knowledge for visual question answering reasoning, improving the generalization performance and quality of visual question answering methods, and enabling them to answer visual questions that require external knowledge. Background Technology

[0002] Visual Question Answering (VQA) is an artificial intelligence task that combines computer vision and natural language processing techniques to enable computers to understand and answer questions related to images. In a VQA task, given an image and a natural language question related to that image, the model needs to understand the content of the image and answer the question in text form.

[0003] Traditional visual question answering models typically rely solely on knowledge learned from their training data to answer questions. These models learn patterns and associations through training on large datasets of images and corresponding question answers. However, they often lack the ability to use external knowledge sources or reasoning to answer questions beyond the scope of their training data. To overcome this issue, several new approaches have emerged in recent years aimed at integrating external knowledge into visual question answering models. By connecting the model to these external knowledge sources, richer context and reasoning capabilities can be provided, enabling the model to answer questions more effectively. The following are the roles of different types of external knowledge in visual question answering:

[0004] 1. Wikipedia Knowledge: Wikipedia is an online encyclopedia containing a vast amount of knowledge, providing detailed information on a wide range of topics. In visual question answering, the system can use Wikipedia to obtain background knowledge related to the question, thereby providing a more accurate and comprehensive answer.

[0005] 2. Relevant Webpage Text Knowledge: Besides Wikipedia, there is a wealth of textual information online, covering various fields and topics. The system can utilize search engines or relevant databases to retrieve webpage text related to the question and extract key information to support the answer.

[0006] 3. Relevant Webpage Image Knowledge: In visual question answering, images are an important source of information. The system can utilize the image search function of a search engine to obtain images related to the current image, and then use computer vision technology to analyze and understand the relevant web images to obtain their features, content, and contextual information. Furthermore, the system can retrieve relevant images from a related image database to provide more references and support.

[0007] 4. Large-scale language model knowledge: Large-scale language models such as GPT-3.5 also provide extensive knowledge bases. These models learn rich semantic and grammatical knowledge through training on large amounts of text. In visual question answering, by using appropriate prompts, large-scale language models can provide external knowledge relevant to the question, which is beneficial for answering the question more accurately.

[0008] This invention uses four existing models: VinVL[1], BLIP[2], DPR[3], and VL-T5[4]. VinVL is an improved visual language modeling approach that enhances the joint understanding between vision and language by re-examining and improving visual representations. This invention uses the VinVL model to generate textual descriptions of images. BLIP is a framework designed to unify visual language understanding and generation tasks. It is a pre-training method designed to enhance the model's ability to understand and generate natural language descriptions related to visual content. This invention uses the BLIP model to generate candidate answers to questions. DPR is a text retrieval method that aims to find paragraphs or sentences related to questions from a large amount of text. This invention uses the DPR model to retrieve Wikipedia knowledge related to questions. The VL-T5 model aims to unify visual and language tasks by taking images and text as input and generating relevant text as output. This invention uses the VL-T5 structure as a heterogeneous knowledge fusion reasoning model. In addition to the above models, this invention also uses the FAISS[5] library. FAISS is a library for efficient similarity search and clustering, primarily used to solve similarity search problems in vector spaces. This invention uses the FAISS library to accelerate the calculation of the similarity between the feature vector of the problem and the feature vector of Wikipedia.

[0009] Traditional visual question answering methods focus solely on the image and the question itself, only answering questions related to the visual content of the image and unable to address those requiring external knowledge. Recent methods have improved upon traditional visual question answering by incorporating Wikipedia articles as external knowledge to aid in answering questions. However, a limitation is that Wikipedia articles cannot effectively cover the external knowledge required for answering questions. Therefore, constructing multi-source, heterogeneous external knowledge for images and questions to more comprehensively cover the external knowledge needed for answering questions is a pressing issue. Furthermore, effectively retrieving external knowledge relevant to the question is also a crucial problem.

[0010] [1] Zhang P, Li X, Hu X, et al. Vinvl: Revisiting visual representations in vision-language models[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021:5579-5588.

[0011] [2] Li J, Li D, Xiong C, et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation[C] / / International Conference on Machine Learning. PMLR, 2022:12888-12900.

[0012] [3] Karpukhin V, B, Min S, et al. Dense passage retrieval for open-domain question answering[J]. arXiv preprint arXiv:2:004.04906, 2020.

[0013] [4] Cho J, Lei J, Tan H, et al. Unifying vision-and-language tasks via text generation[C] / / International Conference on Machine Learning. PMLR, 2021:1931-1942.

[0014] [5] Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data. Summary of the Invention

[0015] To address the shortcomings of existing technologies, this invention provides a visual knowledge reasoning and question-answering method that leverages multi-source heterogeneous knowledge joint enhancement. This invention mainly comprises three points:

[0016] 1. Generate auxiliary information required for multi-source heterogeneous knowledge retrieval, including text descriptions of images and candidate answers to questions. Use the image text description generation module to generate text descriptions of images. Use the multimodal visual question answering module to generate candidate answers to questions.

[0017] 2. Based on image and text descriptions, questions, and candidate answers, multi-source heterogeneous knowledge retrieval is performed, specifically including four types of heterogeneous knowledge: Wikipedia knowledge, web page text knowledge, web page image knowledge, and large-scale language model knowledge. Wikipedia knowledge is retrieved using the Wikipedia search engine module, web page text and web page image knowledge are retrieved using a browser, and large-scale language model knowledge is obtained by constructing appropriate prompts.

[0018] 3. After acquiring four types of heterogeneous knowledge, a suitable loss function is set on the visual question answering dataset, and a visual question answering model for heterogeneous knowledge fusion reasoning is obtained through training.

[0019] A visual knowledge reasoning and question answering method jointly enhanced by multi-source heterogeneous knowledge includes the following steps:

[0020] Step 1: Generate a text description for the image.

[0021] The image text description generation module used in step 1 is VinVL. For the input image, it is represented as a triple (w, q, v), where w is the feature vector of a text word, q is the feature vector of an image region label word, and v is the feature vector of the image region. The generation process for v and q is as follows: Given an image containing K target regions, Faster-RCNN is used to extract the features (v′, z) of each region, where... v′ is the image feature vector, and z is the image location feature vector. Then, v′ and z are concatenated to form a location-sensitive feature vector. Simultaneously, Faster R-CNN is used to obtain the label for each image region.

[0022] Step 2: Generate candidate answers to the question.

[0023] The multimodal visual question-answering module used in step 2 is BLIP. First, the image... Converted into a series of 2D blocks Where (H, W) are the width and height of the image, C is the number of channels in the image, and (P, P) is the resolution of each image patch. Then, the image is linearly mapped to dimension D using Equation-2. For each image patch feature, the corresponding location feature information E also needs to be added. posThen, the final visual features are obtained through an encoder.

[0024]

[0025] z′ l =MSA(LN(z) l-1 ))+z l-1 , l=1...L (3)

[0026] z l =MLP(LN(z′) l ))+z′ l , l=1...L (4)

[0027]

[0028] In the above formula, MSA refers to Multihead self-attention, and LN refers to Layer normalization.

[0029] In the Transformer-Encoder-based text encoder, the generated image visual features are injected by inserting a cross-attention layer between the self-attention layer and the forward network layer, while the text features of the question are generated using the text encoder.

[0030] After obtaining the text features of the question, candidate answers are generated using a Transformer-Decoder-based text decoder.

[0031] Step 3: Search for relevant Wikipedia information.

[0032] The Wikipedia search engine module used in step 3 is DPR. First, an article-dense vector encoder E is used. P The entire Wikipedia article is encoded, generating 768-dimensional feature vectors, which are then indexed using FAISS. When a query is input, the query-dense vector encoder E is used. Q (.) Encode the query, then use the similarity sim(q, p) = E Q (q) T E P (p) Retrieves relevant Wikipedia knowledge. The dense vector encoder is implemented using BERT, and the similarity function uses the L2 distance function.

[0033] Step 4: Retrieve relevant web page text information.

[0034] The search engine used in step 4 was Google Chrome. The query format used was {caption}.{question}? {candidate answer}. After obtaining the preprocessed webpage text, the text encoder used was BERT.

[0035] Step 5: Retrieve relevant web page image information.

[0036] The image search engine used in step 5 is Google Chrome. The query format used is {caption}{candidate answer}. After obtaining the webpage images and corresponding text descriptions, the image encoder and text encoder used are the image encoder and text encoder from CLIP, respectively.

[0037] Step 6: Acquire knowledge of large-scale language models.

[0038] The large-scale language model used in step 6 is ChatGPT, which is accessed using the API provided by OpenAI.

[0039] Step 7: Train a heterogeneous knowledge fusion reasoning model based on four types of heterogeneous knowledge.

[0040] In step 7, the heterogeneous knowledge fusion reasoning model adopts the VL-T5 structure. The visual feature extraction model used is ViT.

[0041] The heterogeneous knowledge fusion reasoning model consists of a bidirectional multimodal encoder and an autoregressive text decoder. Its function is to perform knowledge fusion reasoning based on a given image and question, as well as retrieved multi-source heterogeneous knowledge, to generate an answer.

[0042] The model's training data is based on a visual question-answering dataset that requires external knowledge. Each training sample consists of an image, a question, and an answer. For each training sample, multi-source heterogeneous knowledge related to the training sample is retrieved according to the methods in steps 1 to 6 to form the final training sample.

[0043] The model's input includes visual and textual input. It processes images and retrieved web page images, using a visual feature extraction model to obtain image features and features from relevant web page images. The visual input for the heterogeneous knowledge fusion reasoning model is as follows; the text input for the heterogeneous knowledge fusion reasoning model is as follows: Wikipedia knowledge: Webpage text knowledge:

[0044] Large language model knowledge: {l ans , l exp}, {Question};

[0045] The model is trained using the following loss function until it converges.

[0046]

[0047] Where v refers to the visual features of the input image, x refers to the input text, and y refers to the generated answer.

[0048] The beneficial effects of this invention are as follows: This invention proposes a visual knowledge reasoning question-answering method enhanced by multi-source heterogeneous knowledge. Compared with traditional visual question-answering methods, this method can effectively retrieve relevant external knowledge and utilize multiple heterogeneous external knowledge for joint reasoning. It can solve the problem of traditional visual question-answering methods being unable to answer questions due to a lack of external knowledge, effectively improving the accuracy and quality of question answering, and expanding the application scenarios of visual question answering. Attached Figure Description

[0049] Figure 1 Flowchart of the retrieval auxiliary information generation process of this invention;

[0050] Figure 2 The flowchart of knowledge retrieval and fusion reasoning in this invention. Detailed Implementation

[0051] The detailed parameters of the present invention will be further described below.

[0052] A visual knowledge reasoning and question answering method jointly enhanced by multi-source heterogeneous knowledge includes the following steps:

[0053] Step 1: Generate a text description for the image.

[0054] The input image is preprocessed, including resizing, standardization, and cropping, to better adapt to the model's input requirements;

[0055] Meaningful feature representations are extracted from images. These feature representations are typically a set of vectors or matrices representing key information in the image. The image text description generation module can determine the length of the generated sequence based on certain conditions, such as generating a special termination symbol or reaching a predetermined maximum length.

[0056] The image text description generation module used in step 1 is VinVL. For the input image, it is represented as a triple (w, q, v), where w is the feature vector of a text word, q is the feature vector of an image region label word, and v is the feature vector of the image region. The generation process for v and q is as follows: Given an image containing K target regions, Faster-RCNN is used to extract the features (v′, z) of each region, where... v′ is the image feature vector, and z is the image location feature vector. Then, v′ and z are concatenated to form a location-sensitive feature vector. Simultaneously, Faster R-CNN is used to obtain the label for each image region.

[0057] Step 2: Generate candidate answers to the question.

[0058] Image features and question features are obtained using an image encoder and a question encoder, respectively. The image and question features are then fused based on an attention mechanism to obtain the input of the question answer generator.

[0059] A question-and-answer generator uses fused features as input to generate answers relevant to the question. The generator maps the input features to a probability distribution of answers. Based on this probability distribution, the answer with the highest probability is selected as the final candidate answer.

[0060] The multimodal visual question-answering module used in step 2 is BLIP. First, the image... Converted into a series of 2D blocks Where (H, W) are the width and height of the image, C is the number of channels in the image, and (P, P) is the resolution of each image patch. Then, the image is linearly mapped to dimension D using Equation-2. For each image patch feature, the corresponding location feature information E also needs to be added. pos Then, the final visual features are obtained through an encoder.

[0061]

[0062] z′ l =MSA(LN(z) l-1 ))+z l-1 , l=1...L (3)

[0063] z l =MLP(LN(z′) l ))+z′ l , l=1...L (4)

[0064]

[0065] In the above formula, MSA refers to Multihead self-attention, and LN refers to Layer normalization.

[0066] In the Transformer-Encoder-based text encoder, the generated image visual features are injected by inserting a cross-attention layer between the self-attention layer and the forward network layer, while the text features of the question are generated using the text encoder.

[0067] After obtaining the text features of the question, candidate answers are generated using a Transformer-Decoder-based text decoder.

[0068] Step 3: Search for relevant Wikipedia information.

[0069] Prepare the Wikipedia dataset. This includes converting Wikipedia articles into a format usable by the model; the articles will be segmented into paragraphs and converted into vector representations.

[0070] Based on the image text descriptions and candidate answers generated in steps 1 and 2, the input to the Wikipedia search engine module is constructed. The Wikipedia search engine module will process the input and convert it into a vector representation, and calculate the similarity score between the input vector and the Wikipedia document vector.

[0071] Wikipedia documents are sorted based on similarity scores to determine which documents are most relevant to the query; documents with higher similarity scores are ranked higher.

[0072] The top-K1 documents with the highest similarity were selected as the relevant Wikipedia entries.

[0073] The Wikipedia search engine module used in step 3 is DPR. First, an article-dense vector encoder E is used. P The entire Wikipedia article is encoded, generating 768-dimensional feature vectors, which are then indexed using FAISS. When a query is input, the query-dense vector encoder E is used. Q (.) Encode the query, then use the similarity sim(q, p) = E Q (q) T E P (p) Retrieves relevant Wikipedia knowledge. The dense vector encoder is implemented using BERT, and the similarity function uses the L2 distance function.

[0074] Step 4: Retrieve relevant web page text information.

[0075] Based on the image text descriptions and candidate answers generated in steps 1 and 2, the query input for the text search engine is constructed. The text search engine is used to retrieve relevant webpage fragments. A webpage fragment parsing script is used to parse pure webpage text knowledge from HTML webpages. A webpage fragment filtering script is used to filter out incomplete, excessively long or short, or non-English text knowledge, resulting in filtered text knowledge.

[0076] After obtaining the preprocessed webpage fragment text, a text encoder is used to encode it to obtain the webpage fragment text features. Encode the concatenated string of image text description and question using the same method to obtain feature f. cq Calculation features and f cq The cosine similarity between them is used to obtain the similarity score.

[0077] Select the top-K2 webpage text fragments with the highest relevance as relevant webpage text knowledge.

[0078] Step 5: Retrieve relevant web page image information.

[0079] Based on the image text descriptions and candidate answers generated in steps 1 and 2, the query input for the image search engine is constructed. The image search engine is used to retrieve relevant images containing image text descriptions, and a webpage image parsing script is used to parse all images and their corresponding descriptions from an HTML webpage.

[0080] After obtaining the webpage image, the image and its text description are encoded using an image encoder and a text encoder respectively to obtain the webpage image feature vector. and webpage image text description feature vector The feature vector f of the input image is obtained using the same method. m and the text description feature vector f of the input image c Calculate f m and Similarity between f c and Similarity between The final relevance scores for the webpage images are as follows:

[0081] Finally, the top-K3 images with the highest relevance were selected as the relevant web page image knowledge.

[0082] Step 6: Acquire knowledge of large-scale language models.

[0083] Based on the image text descriptions and candidate answers generated in steps 1 and 2, the large-scale language model input is constructed according to the following format: “{Caption}{Question}?1.Please rewrite this question.2.Please give an answer.3.Please give an explanation for the answer.” The final large-scale language model knowledge is derived from the output answer. ans and explanation l exp constitute.

[0084] Step 7: Train a heterogeneous knowledge fusion reasoning model based on four types of heterogeneous knowledge;

[0085] The heterogeneous knowledge fusion reasoning model consists of a bidirectional multimodal encoder and an autoregressive text decoder. Its function is to perform knowledge fusion reasoning based on a given image, question, and retrieved multi-source heterogeneous knowledge to generate an answer. The model's training data is based on a visual question-answering dataset that requires external knowledge; each training sample consists of an image, a question, and an answer. For each training sample, relevant multi-source heterogeneous knowledge is retrieved according to steps 1 to 6 to form the final training sample.

[0086] The model's input includes visual and textual input. It processes images and retrieved web page images, using a visual feature extraction model to obtain image features and features from relevant web page images. The visual input for the heterogeneous knowledge fusion reasoning model is as follows; the text input for the heterogeneous knowledge fusion reasoning model is as follows:

[0087] Wikipedia knowledge: Webpage text knowledge:

[0088] Large language model knowledge: {l ans , l exp}, {Question};

[0089] The model is trained using the following loss function until it converges.

[0090]

[0091] Where v refers to the visual features of the input image, x refers to the input text, and y refers to the generated answer.

Claims

1. A multi-source heterogeneous knowledge joint enhancement visual knowledge reasoning question and answer method, characterized in that, Comprising the following steps: Step 1: generating a text description of the image; Preprocessing the input image, Extracting feature representations from the image; feature representations represent key information in the image; the image text description generation module determines the length of the generated sequence according to the conditions; Step 2: generating candidate answers to the question; Using image and question encoders to obtain image features and question features respectively, and fusing image and question features based on an attention mechanism to obtain the input of the question answer generator; Through the question answer generator, using the fused features as input, generate an answer related to the question; The question answer generator maps the input features to a probability distribution of an answer; according to the generated answer probability distribution, select the answer with the highest probability as the final candidate answer to the question; Step 3: retrieving relevant Wikipedia knowledge; Prepare the Wikipedia dataset; Based on the image text description and candidate answer generated in steps 1 and 2, construct the input of the Wikipedia retriever module, which will process and convert the input into a vector representation, and calculate the similarity score between the input vector and the Wikipedia document vector; According to the similarity score, sort the Wikipedia documents to determine which documents are most relevant to the query, and the higher the similarity score, the higher the ranking; Select the top-K1 documents with the highest similarity as relevant Wikipedia knowledge; Step 4: retrieving relevant web page text knowledge; Based on the image text description and candidate answer generated in steps 1 and 2, construct the query input of the text search engine; Use the text search engine to retrieve relevant web page snippets; parse the HTML web page to obtain pure web page text knowledge through the web page snippet parsing script; filter the text knowledge through the web page snippet filter script to form filtered text knowledge; After obtaining the preprocessed webpage segment text, encode it using a text encoder to obtain webpage segment text features Encode the spliced string of the image text description and the question in the same way to obtain features f cq ; Calculate the cosine similarity between features and f cq , to get similarity Select the top-K2 web page snippet texts with the highest relevance as relevant web page text knowledge; Step 5: retrieving relevant web page image knowledge; Based on the image text description and candidate answer generated in steps 1 and 2, construct the query input of the image search engine; use the image search engine to retrieve relevant images containing image text descriptions, and parse the HTML web page to obtain all images and corresponding image descriptions through the web page image parsing script; After obtaining the webpage image, the image and the image text description are encoded using an image encoder and a text encoder respectively to obtain the webpage image feature vector and the webpage image text description feature vector Through the same method, the feature vector f of the input image is obtained m and the text description feature vector f of the input image c ; the similarity between f m and is calculated f c and is calculated The final correlation score of the webpage image is as follows: Finally, select the top-K3 images with the highest relevance as relevant web page image knowledge; Step 6: obtaining large-scale language model knowledge; Based on the image text description and candidate answers generated in step 1 and step 2, construct the input of the large-scale language model; the knowledge of the large-scale language model is output by the answer l ans and explanation l exp constitute; Step 7: training a heterogeneous knowledge fusion reasoning model based on the four types of heterogeneous knowledge; The heterogeneous knowledge fusion reasoning model consists of a bidirectional multi-modal encoder and a self-attention text decoder; The training data of the model is based on the visual question answering dataset that requires external knowledge, and each training sample consists of an image, a question and an answer. For each training sample, retrieve the relevant multi-source heterogeneous knowledge of the training sample according to the methods of steps 1 to 6 to form the final training sample; The input of the model includes visual input and text input: processing the image and the retrieved web image, using a visual feature extraction model to obtain the image features and the features of the related web image As the visual input of the heterogeneous knowledge fusion reasoning model; the text input of the heterogeneous knowledge fusion reasoning model is as follows: Wikipedia knowledge: Web page text knowledge: Large language model knowledge:{l ans ,l exp}, Question}; The model is trained according to the following loss function until the model converges: Where v refers to the input image visual feature, x refers to the input text, and y refers to the generated answer.

2. The visual knowledge reasoning question answering method of claim 1, wherein The text description of the image generated in step 1 comprises: The image text description generation module used in step 1 is VinVL; for the input image, it is represented as a triple (w, q, v), where w is the feature vector of the text word, q is the feature vector of the image region label word, and v is the feature vector of the image region; where v and q are generated as follows: Given an image containing K target regions, use Faster-RCNN to extract the feature (v ′ ,z) of each region, wherein is the image feature vector, z is the image location feature vector; the spliced v ′ and z constitute the location-sensitive feature vector; meanwhile, the label of each image region is obtained by using Faster-RCNN.

3. The visual knowledge reasoning question answering method of claim 1 or 2, wherein, The candidate answer generation module used in step 2 is BLIP; The multi-modal visual question answering module used in step 2 is BLIP; transforming the image into a series of 2D blocks where (H, W) is the width and height of the image, C is the number of channels of the image, (P, P) is the resolution of each image patch, where Linearly map the image into D dimensions by Formula-2; for each image patch feature, the corresponding location feature information E is also needed pos Get the final visual feature through the encoder; z' l = MSA(LN(z l-1 ))+ z l-1 , l = 1...L (3) z l = MLP(LN(z' (z l ))+ z' l , l = 1...L (4) y = LN(z L ) (5) MSA in the above formula refers to Multihead self-attention, and LN refers to Layernorm layer; In the text encoder based on Transformer-Encoder, the generated image visual features are injected by inserting a cross-attention layer between the self-attention layer and the forward network layer, and the text features of the question are generated using the text encoder; After obtaining the text features of the question, a text decoder based on Transformer-Decoder is used to generate the candidate answer.

4. The visual knowledge reasoning question answering method of claim 1, wherein, The relevant Wikipedia knowledge retrieval module used in step 3 is DPR; The dense vector encoder is implemented by BERT, and the similarity function uses the L2 distance function. Using article dense vector encoder E P (.) Encode all Wikipedia articles, resulting in 768-dimensional feature vectors, index these feature vectors using FAISS; When a query is entered, the query intensive vector encoder E Q (.) encodes the query, and obtains relevant Wikipedia knowledge by similarity sim(q,p) = E Q (q) T E P (p) The text encoder used in step 4 is BERT; 5. The visual knowledge reasoning question answering method of claim 1, wherein, The image encoder and text encoder used in step 5 are the image encoder and text encoder in CLIP, respectively; The large-scale language model used in step 6 is ChatGPT. In step 7, the heterogeneous knowledge fusion reasoning model adopts the VL-T5 structure; the visual feature extraction model used is ViT.

6. The visual knowledge reasoning question answering method of claim 1, wherein, ​

Images