Visual question and answer method based on knowledge retrieval enhancement

Abstract

The invention provides a visual question and answer method based on knowledge retrieval enhancement, and relates to the field of computer vision and natural language processing, and the method comprises the following steps: respectively encoding a multi-modal query and a knowledge text by using a multi-modal encoder; calculating the similarity between the multi-modal query and the feature vector of the knowledge text by using a vector dot product, and retrieving to obtain a plurality of related external knowledge; the method comprises the following steps of: extracting the most useful visual representation from a frozen image encoder by using a lightweight visual Transform module; using a T5 encoder to independently encode the joint input of a plurality of knowledge paragraphs, images and questions; the first N most representative marks are connected in series and input into a T5 decoder for joint decoding, and finally a correct answer is generated. According to the method, the problems of visual information loss and insufficient knowledge utilization in the existing method are solved, and the performance of the system in the knowledge-intensive visual question and answer task is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and natural language processing, and in particular to a visual question answering method based on knowledge retrieval enhancement. Background Art

[0002] Visual Question Answering (VQA) is one of the basic research tasks in the field of vision and language, and is an important way to realize user-oriented human-computer interactive visual systems. A VQA system takes an image and a natural language question about the image as input, and gives the correct answer to the question after recognition and reasoning. The main challenge of the visual question answering task is to deeply understand the semantic information of the input image and the question and build cross-modal associations to achieve answer prediction. The knowledge-based visual question answering (KB-VQA) task extends the VQA task to solve more complex visual questions, that is, those that require external knowledge to answer correctly. For the knowledge-based visual question answering task, previous works first use a pre-trained visual-language model to fuse image and question information in a multimodal space, and then fuse the multimodal representation of the image and question with the retrieved textual knowledge, for example: directly fuse the textual knowledge with the multimodal features of the image and the question, or fuse the output of the visual-language model with the knowledge graph through a graph convolutional network. However, when injecting textual knowledge into the multimodal space, a lot of knowledge information is lost, resulting in lower performance. Therefore, TRiG proposed a paradigm shift for the KB-VQA task, that is, first converting images into plain text, and then using rich pre-trained language models in the natural language space for knowledge retrieval and generative question answering. RA-VQA also uses the "image to plain text" approach, but it builds a new joint training framework to simultaneously train the knowledge retriever and the answer generator, and achieves better results. However, this paradigm has obvious shortcomings - the loss of visual information is inevitable in the process of converting images to plain text. In order to solve the problem of visual information loss, some methods try to supplement visual features. For example, TwO and REVIVE both add visual features of images in the knowledge retrieval stage and the answer generation stage; RA-VQA-v2 proposes the FLMR method to improve the performance of knowledge retrieval. Specifically, FLMR integrates more fine-grained, token-level visual features and text features into multi-dimensional embeddings. When calculating the relevance score, FLMR considers the interaction between each pair of token embeddings, including cross-modal interactions between text and images, thereby achieving a more fine-grained relevance evaluation. However, the additional supplementation of visual features on the basis of image conversion to plain text will increase the complexity of the system. On the other hand, there may be information redundancy between the visual information of the coexisting visual modality and textual modality, which in turn introduces additional unnecessary burdens to the system. The existing technology has the following three main problems: 1. Insufficient utilization of multiple knowledge paragraphs: Current generators often only use one retrieved knowledge paragraph for answer generation, failing to fully utilize multiple related knowledge paragraphs, limiting the system’s performance in knowledge-intensive tasks. 2. Complicated image-to-text conversion process: This process usually includes image object detection, title generation, and OCR text detection. The steps are complicated and rely on multiple pre-trained models, which reduces the efficiency of the system. In addition, the quality of the generated text information limits the quality of the final question and answer. 3. Loss of visual information: In the process of converting images to text, important visual information (such as the relative position information of objects, visual features, etc.) is often lost, causing the system to make errors when answering image-related questions.

[0003] Based on the above problems, the present invention proposes a visual question answering method based on knowledge retrieval enhancement. The method includes a knowledge retriever for multimodal queries and a cross-modal retrieval enhancement generator. By directly utilizing complete image information (rather than object labels, keywords and titles) and collecting evidence from multiple paragraphs to generate answers, the method overcomes the problems of visual information loss and insufficient knowledge utilization in existing methods, and significantly improves the performance of the system in knowledge-intensive visual question answering tasks. Summary of the invention

[0004] The purpose of the present invention is to propose a visual question answering method based on knowledge retrieval enhancement, aiming to solve the problems of insufficient knowledge utilization, image information loss and high system complexity in the prior art. The method includes a knowledge retriever for multimodal queries and a cross-modal retrieval enhancement generator, which combines multimodal information processing and knowledge retrieval mechanisms. Through the collaborative work of the knowledge retriever and the cross-modal generator, the performance of the visual question answering system in complex problems is effectively improved, especially in scenarios where external knowledge assistance is required.

[0005] The technical principle of this method is to combine multimodal information processing technology and the retrieval mechanism of external knowledge base to generate more accurate question-answering results by jointly encoding and reasoning images and question texts. It is particularly suitable for complex visual question-answering systems that need to combine image content with external knowledge to answer questions, and is widely used in scenarios such as intelligent question-answering systems, image understanding, human-computer interaction, and intelligent assistants.

[0006] The technical solution of the present invention is to provide a system for enhancing visual question answering for knowledge retrieval, comprising the following steps: Step 1: According to the image I and question text Q input by the user, multiple relevant knowledge paragraphs are retrieved from the preset external knowledge base; S1.1, the multimodal query consisting of the image I and the question text Q is encoded by the multimodal retriever to obtain the multimodal query vector representation of the image and the question, and the knowledge paragraphs in the knowledge base are encoded to obtain the dense vector representation of the knowledge paragraphs; S1.2, using vector dot product similarity to calculate the relevance score between the multimodal query and the knowledge paragraph, and retrieving the K most relevant knowledge paragraphs based on the relevance score; Step 2: Based on the retrieved K relevant knowledge paragraphs, image I and question text Q, the correct answer to the question is generated through a cross-modal retrieval-enhanced generator; S2.1, through the Q-Former module, which is a learning module, extracts the most useful visual representation related to the problem from the frozen image encoder, and dynamically adjusts the visual features extracted from the image encoder during the training process; S2.2, concatenate the extracted visual features with the features of the question text and the feature vectors of the K retrieved relevant knowledge paragraphs, and input these joint features into the language generation model T5 based on the encoder-decoder structure for independent encoding and joint decoding; S2.3, in the T5 language generation model, each retrieved knowledge paragraph is encoded separately; and the information of multiple knowledge paragraphs is processed comprehensively through joint decoding to generate the final answer to the question; this process simultaneously considers the interaction of visual features, question text features and multiple related knowledge paragraphs to ensure that the generated answer has high accuracy and relevance; S2.4, improves the accuracy of visual question answering systems in complex scenarios by comprehensively considering multimodal information of vision, text, and external knowledge in the reasoning process.

[0007] The retriever in step 1 includes a frozen image encoder and a Q-Former module for encoding the multimodal query and the knowledge text respectively; The Q-Former module consists of two Transformer submodules that share a self-attention layer: (1) a visual Transformer that interacts with the frozen image encoder for visual feature extraction, and (2) a text Transformer that can act as a text encoder. When the input is a multimodal query When , Q-Former jointly encodes the input image I and the question text Q. The visual Transformer submodule and the text Transformer submodule perform synchronous processing through a shared self-attention mechanism to ensure the effective fusion of image and text information, and finally obtain the encoded feature vector set of the multimodal query. When the input is a text paragraph from a knowledge corpus, the text Transformer submodule in Q-Former independently encodes the knowledge text and generates a feature vector representation of the text paragraph; After obtaining the feature vectors of the multimodal query and the knowledge text respectively, the vector dot product is used to calculate the pairwise similarity between each feature vector in the multimodal query and the feature vector of the knowledge text. By calculating the similarity score matrix, the highest similarity score is selected as the similarity measure between the multimodal query and the knowledge paragraph, ensuring that the system can select the knowledge information most relevant to the query. The calculation formula is as follows: in, Represents a multimodal query The similarity with the text knowledge p, represents the i-th feature vector of the multimodal query, t p The feature vector representing the text knowledge, represents the vector dot product operation.

[0008] The specific functional steps of the retrieval enhancement generator in step 2 are as follows: (1) Visual representation extraction and mapping: A learnable lightweight visual Transformer module is used to extract the most useful visual representation from the frozen image encoder, and a fully connected FC layer is used to linearly project the extracted visual representation so that the dimension of the visual representation matches the hidden dimension of the language generation model T5, thereby ensuring that the visual information can be seamlessly embedded in the processing flow of the T5 model; (2) Joint input construction: connect the question with each paragraph to form K input texts Where T i =“Question:Q Context:p i ”, that is, in question Q and each relevant paragraph p i The prefixes “Question:” and “Context:” are added before the text. Then, the mapped visual features are prepended to each input text T i Before embedding, form the joint input of each paragraph with the question text and visual features; (3) Independent encoding of multiple paragraphs: In the T5 encoder, each joint input is independently encoded to obtain K encoded representation vectors. These vectors capture the contextual relationship between each paragraph and the question, while combining visual information to improve the model's ability to understand complex multimodal queries. (4) Efficient joint decoding: After the encoding stage is completed, the first N most representative tokens are taken from each encoded representation vector and concatenated to form a compact input sequence to reduce the input sequence length of the T5 decoder. In this way, the processing time of the decoder is effectively shortened and the computational efficiency of the generation process is significantly improved. (5) Answer generation: The optimized encoded representation vector is input into the T5 decoder, which combines visual features, question text, and context paragraph information to generate the final answer to the question. By considering multimodal information and relevant context at the same time, the generator can generate more accurate and relevant answers.

[0009] The beneficial effect of the present invention is that it can significantly improve the performance of the question-answering system in visual question-answering tasks, especially in complex scenarios based on external knowledge. By enhancing the processing capabilities of multimodal information and effectively utilizing multiple knowledge segments, the present invention can not only improve the accuracy of question-answering, but also provide more efficient solutions in knowledge-intensive tasks. The method has broad application prospects and is suitable for fields such as intelligent question-answering systems, intelligent assistants, image understanding, and human-computer interaction. Compared with the prior art, the present invention has the following innovations and advantages: 1. Make full use of multiple knowledge paragraphs: The present invention can consider multiple relevant knowledge paragraphs when generating answers, and make full use of the information in the external knowledge base, thereby greatly improving the performance of the system in knowledge-intensive tasks. 2. Avoiding loss of visual information: By directly utilizing the visual features of the image instead of converting the image into text, the present invention effectively avoids the problem of visual information loss caused by image conversion in traditional methods, enabling the system to handle more complex visual problems. 3. Simplify system processes and improve efficiency: The present invention reduces the introduction of redundant information and the complexity of the system by optimizing the knowledge retrieval and generation process, while improving the decoding efficiency of the generator. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. Figure 1 It is a schematic diagram of the framework of the present invention; Figure 2 It is a model structure diagram of a knowledge retriever for multimodal queries; Figure 3 It is a retrieval flow chart of a knowledge retriever for multimodal queries; Figure 4 This is the model structure diagram of the cross-modal retrieval enhanced generator. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] The present invention is applied to a system for knowledge retrieval and enhanced visual question answering, comprising the following steps: Step 1: According to the image I and question text Q input by the user, multiple relevant knowledge paragraphs are retrieved from the preset external knowledge base; S1.1, the multimodal query consisting of the image I and the question text Q is encoded by the multimodal retriever to obtain the multimodal query vector representation of the image and the question, and the knowledge paragraphs in the knowledge base are encoded to obtain the dense vector representation of the knowledge paragraphs; S1.2, using vector dot product similarity to calculate the relevance score between the multimodal query and the knowledge paragraph, and retrieving the K most relevant knowledge paragraphs based on the relevance score; Step 2: Based on the retrieved K relevant knowledge paragraphs, image I and question text Q, the correct answer to the question is generated through a cross-modal retrieval-enhanced generator; S2.1, through the Q-Former module, which is a learning module, extracts the most useful visual representation related to the problem from the frozen image encoder, and dynamically adjusts the visual features extracted from the image encoder during the training process; S2.2, concatenate the extracted visual features with the features of the question text and the feature vectors of the K retrieved relevant knowledge paragraphs, and input these joint features into the language generation model T5 based on the encoder-decoder structure for independent encoding and joint decoding; S2.3, in the T5 language generation model, each retrieved knowledge paragraph is encoded separately; and the information of multiple knowledge paragraphs is processed comprehensively through joint decoding to generate the final answer to the question; this process simultaneously considers the interaction of visual features, question text features and multiple related knowledge paragraphs to ensure that the generated answer has high accuracy and relevance; S2.4, improves the accuracy of visual question answering systems in complex scenarios by comprehensively considering multimodal information of vision, text, and external knowledge in the reasoning process.

[0013] The retriever in step 1 includes a frozen image encoder and a Q-Former module, which are used to encode the multimodal query (i.e., visual features, question text features) and the knowledge text respectively; The Q-Former module is a lightweight Transformer with 188M parameters. It uses a set of learnable query vectors to interact with the frozen image encoder to extract the most important visual information related to the current question text from the image. The Q-Former module consists of two Transformer submodules that share a self-attention layer: (1) a visual Transformer that interacts with the frozen image encoder for visual feature extraction, and (2) a text Transformer that can act as a text encoder. When the input is a multimodal query When , Q-Former jointly encodes the input image I and the question text Q. The visual Transformer submodule and the text Transformer submodule perform synchronous processing through a shared self-attention mechanism to ensure the effective fusion of image and text information, and finally obtain the encoded feature vector set of the multimodal query. When the input is a text paragraph from a knowledge corpus, the text Transformer submodule in Q-Former independently encodes the knowledge text and generates a feature vector representation of the text paragraph; After obtaining the feature vectors of the multimodal query and the knowledge text respectively, the vector dot product is used to calculate the pairwise similarity between each feature vector in the multimodal query and the feature vector of the knowledge text. By calculating the similarity score matrix, the highest similarity score is selected as the similarity measure between the multimodal query and the knowledge paragraph, ensuring that the system can select the knowledge information most relevant to the query. The calculation formula is as follows: in, Represents a multimodal query The similarity with the text knowledge p, represents the i-th feature vector of the multimodal query, t p The feature vector representing the text knowledge, represents the vector dot product operation.

[0014] The retrieval enhancement generator in step 2 includes a frozen image encoder, a lightweight visual Transformer, and a language generation model T5. The specific functional steps are as follows: (1) Visual representation extraction and mapping: A learnable lightweight visual Transformer module is used to extract the most useful visual representation from the frozen image encoder, and a fully connected FC layer is used to linearly project the extracted visual representation so that the dimension of the visual representation matches the hidden dimension of the language generation model T5, thereby ensuring that the visual information can be seamlessly embedded in the processing flow of the T5 model; (2) Joint input construction: connect the question with each paragraph to form K input texts Where T i =“Question:Q Context:p i ”, that is, in question Q and each relevant paragraph p i The prefixes “Question:” and “Context:” are added before the text (this part is similar to the code program description, so it is retained in the specification and deleted in the claims); then, the mapped visual features are prepended to each input text T i Before embedding, form the joint input of each paragraph with the question text and visual features; (3) Independent encoding of multiple paragraphs: In the T5 encoder, each joint input is independently encoded to obtain K encoded representation vectors. These vectors capture the contextual relationship between each paragraph and the question, while combining visual information to improve the model's ability to understand complex multimodal queries. (4) Efficient joint decoding. After the encoding stage is completed, the first N most representative tokens are taken from each encoded representation vector, and these tokens are concatenated to form a compact input sequence to reduce the input sequence length of the T5 decoder. In this way, the processing time of the decoder is effectively shortened and the computational efficiency of the generation process is significantly improved (this part is a description of the beneficial effects and is retained in the specification part); (5) Answer generation: The optimized encoded representation vector is input into the T5 decoder. During the processing, the decoder combines the visual features, the question text, and the information of the context paragraph to generate the final answer to the question. By considering multimodal information and relevant context at the same time, the generator can generate more accurate and relevant answers (this part is a description of the beneficial effects and is retained in the specification part).

[0015] The specific embodiment provides a visual question answering method based on knowledge retrieval enhancement, which mainly includes two modules: a knowledge retriever for multimodal queries and a cross-modal retrieval enhancement generator; Given a text question Q about an image I, the knowledge retriever module treats the image I and the text question Q as a multimodal query and from external knowledge corpus Retrieve the The most relevant K paragraphs. The formula is as follows: like Figure 2 , 3 As shown in Figure 2, the specific process of knowledge retrieval includes: Step 1: When the input is a multimodal query When , Q-Former jointly encodes the input image I and question Q, and the visual Transformer submodule and the text Transformer submodule perform synchronous processing through a shared self-attention mechanism to ensure the effective fusion of image and text information, and finally obtain the encoded feature vector set of the multimodal query; Step 2: When the input is a text paragraph from the knowledge corpus, the text transformer submodule in Q-Former independently encodes the knowledge text and generates a feature vector representation of the text paragraph. Step 3: After obtaining the feature vectors of the multimodal query and the knowledge text respectively, the vector dot product is used to calculate the pairwise similarity between each feature vector in the multimodal query and the feature vector of the knowledge text. By calculating the similarity score matrix, the highest similarity score is selected as the similarity measure between the multimodal query and the knowledge paragraph, ensuring that the system can select the knowledge information most relevant to the query. The calculation formula is as follows: in, Represents a multimodal query The similarity with the text knowledge p. represents the i-th feature vector of the multimodal query, t p A feature vector representing textual knowledge. · Represents a vector dot product operation; Step 4: In the inference phase, first use the text transformer module in the trained Q-Former to pre-extract the knowledge corpus The vector representation t of each paragraph p in p Next, we use the FAISS system to index all paragraph representations, thus enabling sublinear nearest neighbor search. At runtime, given a multimodal query Calculate its representation And retrieve the relevance from the pre-extracted paragraph vector library The largest first K paragraphs; After retrieving the K most relevant paragraphs for each image-question pair in the dataset, the image, question, and multiple retrieved paragraphs are input into the cross-modal retrieval-enhanced generator to generate a natural language answer. The formula is as follows: Step 5, such as Figure 4As shown in Figure 1, the retrieval-enhanced answer generation process includes: visual representation extraction and mapping: a learnable lightweight visual Transformer module is used to extract the most useful visual representation from the frozen image encoder, and a fully connected (FC) layer is used to linearly project the extracted visual representation so that the dimension of the visual representation matches the hidden dimension of the language generation model T5, thereby ensuring that the visual information can be seamlessly embedded in the processing flow of the T5 model; Step 6: Joint input construction: Separately in question Q and each related paragraph p i After adding the prefixes "Question:" and "Context:", the question is connected to each paragraph to form K input texts. Where T i =“Question:QContext:p i Then, the mapped visual features are prepended to each input text T i Before embedding, form the joint input of each paragraph with the question text and visual features; Step 7, multi-paragraph independent encoding: In the T5 encoder, each joint input is independently encoded to obtain K encoded representation vectors. These vectors capture the contextual relationship between each paragraph and the question, while combining visual information to improve the model's ability to understand complex multimodal queries; Step 8, efficient joint decoding: In the classic FiD structure, the K encoded feature vectors are directly concatenated and then input into the T5 decoder for decoding. Because this model only performs multi-paragraph fusion in the decoder, it is called Fusion-in-Decoder. FiD can generate answers by integrating information from multiple different sources, so it is the most effective way to solve knowledge-intensive tasks. However, practice and research have shown that due to the long input sequence of the decoder, the decoding efficiency is very low, which leads to low overall efficiency of the FiD system and high resource consumption; In order to improve the generation efficiency, after the encoding stage is completed, the first N most representative tokens are taken from each encoded representation vector, and these tokens are connected in series to form a compact input sequence to reduce the input sequence length of the T5 decoder. In this way, the processing time of the decoder is effectively shortened and the computational efficiency of the generation process is significantly improved; Step 9, answer generation: The optimized encoded representation vector is input into the T5 decoder, which combines visual features, question text, and context paragraph information to generate the final answer to the question. By considering multimodal information and relevant context at the same time, the generator can generate more accurate and relevant answers.

[0016] The visual question answering method based on knowledge retrieval enhancement provided in the embodiment can not only improve the accuracy of question answering, but also provide more efficient solutions in knowledge-intensive tasks by enhancing the processing capability of multimodal information and effectively utilizing multiple knowledge paragraphs.

[0017] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Visual question answering method based on knowledge retrieval enhancement, a system applied to knowledge retrieval enhanced visual question answering, characterized by , including the following steps: Step 1: According to the image I and question text Q input by the user, multiple relevant knowledge paragraphs are retrieved from the preset external knowledge base; S1.1, the multimodal query consisting of the image I and the question text Q is encoded by the multimodal retriever to obtain the multimodal query vector representation of the image and the question, and the knowledge paragraphs in the knowledge base are encoded to obtain the dense vector representation of the knowledge paragraphs; S1.2, using vector dot product similarity to calculate the relevance score between the multimodal query and the knowledge paragraph, and retrieving the K most relevant knowledge paragraphs based on the relevance score; Step 2: Based on the retrieved K relevant knowledge paragraphs, image I and question text Q, the correct answer to the question is generated through a cross-modal retrieval-enhanced generator; S2.1, through a lightweight Transformer module, extracts the most useful visual representation related to the problem from the frozen image encoder, and dynamically adjusts the visual features extracted from the image encoder during training; S2.2, concatenate the extracted visual features with the features of the question text and the feature vectors of the K retrieved relevant knowledge paragraphs, and input these joint features into the language generation model T5 based on the encoder-decoder structure for independent encoding and joint decoding; S2.3, in the T5 language generation model, each retrieved knowledge paragraph is encoded separately; and Through joint decoding, the information of multiple knowledge paragraphs is processed comprehensively to generate the final answer to the question; this process simultaneously considers the interaction of visual features, question text features and multiple related knowledge paragraphs to ensure that the generated answer has high accuracy and relevance; S2.4, improves the accuracy of visual question answering systems in complex scenarios by comprehensively considering multimodal information of vision, text, and external knowledge in the reasoning process.

2. The visual question answering method based on knowledge retrieval enhancement according to claim 1 is characterized in that ,The retriever described in step 1 includes a frozen image encoder and a Q-Former module for encoding ,multimodal queries and knowledge text respectively; The Q-Former module consists of two Transformer submodules that share a self-attention layer: (1) a visual Transformer that interacts with the frozen image encoder for visual feature extraction, and (2) a text Transformer that can act as a text encoder. When the input is a multimodal query When , Q-Former jointly encodes the input image I and the question text Q. The visual Transformer submodule and the text Transformer submodule perform synchronous processing through a shared self-attention mechanism to ensure the effective fusion of image and text information, and finally obtain the encoded feature vector set of the multimodal query. When the input is a text paragraph from a knowledge corpus, the text Transformer submodule in Q-Former independently encodes the knowledge text and generates a feature vector representation of the text paragraph; After obtaining the feature vectors of the multimodal query and the knowledge text respectively, the vector dot product is used to calculate the pairwise similarity between each feature vector in the multimodal query and the feature vector of the knowledge text. By calculating the similarity score matrix, the highest similarity score is selected as the similarity measure between the multimodal query and the knowledge paragraph, ensuring that the system can select the knowledge information most relevant to the query. The calculation formula is as follows: in, Represents a multimodal query The similarity with the text knowledge p, represents the i-th feature vector of the multimodal query, t p The feature vector representing the text knowledge, represents the vector dot product operation.

3. The visual question answering method based on knowledge retrieval enhancement according to claim 1 is characterized in that , The specific functional steps of the retrieval enhancement generator described in step 2 are as follows: (1) Visual representation extraction and mapping: A learnable lightweight visual Transformer module is used to extract the most useful visual representation from the frozen image encoder, and a fully connected FC layer is used to linearly project the extracted visual representation so that the dimension of the visual representation matches the hidden dimension of the language generation model T5, thereby ensuring that the visual information can be seamlessly embedded in the processing flow of the T5 model; (2) Joint input construction: connect the question with each paragraph to form K input texts Where T i ="Question:QContext:p i "; Then, the mapped visual features are prepended to each input text T i Before embedding, form the joint input of each paragraph with the question text and visual features; (3) Independent encoding of multiple paragraphs: In the T5 encoder, each joint input is independently encoded to obtain K encoded representation vectors. These vectors capture the contextual relationship between each paragraph and the question, while combining visual information to improve the model's ability to understand complex multimodal queries. (4) Efficient joint decoding: After the encoding stage is completed, the first N most representative symbols are taken from each encoded representation vector and these symbols are concatenated to form a compact input sequence to reduce the input sequence length of the T5 decoder; (5) Answer generation: The optimized encoded representation vector is input into the T5 decoder. During the processing, the decoder combines the visual features, question text and context paragraph information to generate the final answer to the question.

Images