A multimodal document-based retrieval augmentation generation method

Abstract

The application discloses a kind of based on multimodal document's retrieval enhancement generation method, this method includes the following steps: S1, data construction;S2, the feature extraction of multimodal knowledge retriever;S3, the feature mapping of multimodal knowledge retriever;S4, the correlation degree calculation of multimodal knowledge retriever;S5, multimodal answer generation: large language model produces text reply according to multimodal input.The application uses the multimodal document of picture and text combination as knowledge carrier, and designs a kind of multimodal retrieval enhancement generation scheme.Compared with the existing end-to-end model scheme, the scheme is based on retrieval enhancement generation framework, which ensures the accuracy and explainability of the answer.Compared with the retrieval enhancement generation scheme using text document as knowledge carrier, the scheme adds visual information to the document to construct a multimodal document, and improves the knowledge retriever and answer generator to utilize the multimodal document, thereby improving the accuracy of knowledge-intensive visual question answering task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of knowledge-intensive visual question answering, and particularly relates to a retrieval-enhanced generation method based on multi-modal documents. BACKGROUND

[0002] Knowledge-intensive visual question answering is a sub-task of visual question answering. Visual question answering is a task of answering related natural language questions according to given visual content, such as answering questions about elements on a picture according to a picture. It is a typical multi-modal task involving deep learning, natural language processing and multi-modal learning. Knowledge-intensive visual question answering is more difficult than traditional visual question answering because it requires not only understanding of visual content but also knowledge related to the visual content to correctly answer the question. This task has potential application scenarios in production and life, such as allowing users to ask knowledge-related questions about objects in a photo based on a photo taken by the user, and developing related technologies to facilitate user information queries.

[0003] Existing technical solutions for knowledge-intensive visual question answering can be divided into two categories: end-to-end models and retrieval-augmented generation (RAG, Retrieval-Augmented Generation) systems. The end-to-end model solution uses a single model, such as a multi-modal large language model, to process visual question answering as a whole. The composition of the multi-modal large language model usually includes a visual encoder, a large language model and a visual-to-language mapping module. The visual encoder is used to encode visual content into visual features, which are then converted to the input space of the large language model through the mapping module. After being trained with a large amount of visual language corpus and instructions, the multi-modal large language model can generate appropriate text responses based on visual content and text instructions, has the ability of visual question answering, and can correctly answer a part of knowledge-intensive visual question answering problems.

[0004] In order to explicitly utilize knowledge when answering knowledge-intensive visual question answering problems and improve the accuracy of the results and the explainability of the system, a retrieval-augmented generation system can be constructed. Retrieval-augmented generation is a method of improving the performance of knowledge-intensive tasks by adding external retrieved knowledge to the input of a large language model. In specific implementation, the architecture of the retrieval-augmented generation system is usually composed of a knowledge retriever and an answer generator. Existing retrieval-augmented generation systems usually use text documents as knowledge carriers. The knowledge retriever matches the user input with the knowledge documents through certain algorithms and sorts them according to the relevance, returning a set of documents with high relevance. The answer generator usually uses a multi-modal large language model, which can utilize the problem information and the retrieved documents when generating answers, thereby achieving the purpose of utilizing knowledge.

[0005] The end-to-end model scheme mainly relies on the ability of the multi-modal large language model itself to answer the question, including the ability of visual understanding, the ability of utilizing knowledge reserve and the language generation ability. The knowledge-intensive visual question answering involves rich visual entities and entity knowledge, which is difficult for a single model to accurately remember and utilize, so the answer accuracy of the end-to-end model scheme is usually low. Moreover, since it is usually a black box model, it is difficult to track and improve the specific logic of the model to generate answers, and the explainability is poor.

[0006] The existing retrieval enhancement generation system usually uses a text document as a carrier of knowledge. When applied to knowledge-intensive image-text question answering, the knowledge retriever needs to retrieve the corresponding text document by utilizing visual information and text questions, and there is a modal difference between the two sides, which requires the retriever to have strong cross-modal retrieval ability, so the retriever trained using the text document as the knowledge base may be suboptimal. When generating an answer, since the retriever may return multiple high-relevance documents, the answer generator needs to utilize the correct document to generate an answer at this time, which also requires the answer generator to have strong cross-modal information screening ability. Therefore, the retrieval enhancement generation system using a text document has a bottleneck in accuracy when processing knowledge-intensive image-text question answering.

[0007] The information disclosed in this BACKGROUND section is only for the purpose of enhancing the understanding of the general background of the application, and should not be considered as acknowledging or implying in any form that this information constitutes prior art known to those skilled in the art. SUMMARY

[0008] In view of the problems in the prior art, the purpose of the present application is to provide a retrieval enhancement generation method based on multi-modal documents.

[0009] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0010] A retrieval enhancement generation method based on multi-modal documents, the retrieval enhancement generation method comprising the following steps:

[0011] S1, data construction: constructing a multi-modal knowledge base composed of multi-modal documents;

[0012] S2, feature extraction of multi-modal knowledge retriever: the question picture and question text in the user input are respectively encoded into features by an image encoder and a text encoder, wherein the picture extracts global features and local features, and the text extracts text features; the document picture in the multi-modal document extracts global features, and the document text extracts text features;

[0013] S3, Feature mapping of the multi-modal knowledge retriever: mapping the features extracted in step S2 to the same dimension; the global features and the local features of the picture are mapped using a multi-layer perception network and a Transformer network respectively, and the text is mapped using a linear layer;

[0014] S4, Correlation calculation of the multi-modal knowledge retriever: each feature on the user input side is respectively multiplied with all the features of the document, and the maximum value is taken as the correlation of the input feature, and then all the correlations are summed up as the final correlation;

[0015] S5, Multi-modal answer generation: the large language model generates a text reply according to the multi-modal input.

[0016] Further, step S1 is specifically:

[0017] First, match all the text documents with the corresponding entity pictures to form multi-modal documents; according to the accuracy of the source from high to low, try to extract the encyclopedia page picture, use the encyclopedia picture search and use the general picture search method in turn until the entity picture is found; finally, each text document in the original knowledge base is matched with the corresponding entity picture to form a multi-modal knowledge base.

[0018] Further, in step S4, for the retrieval of multi-modal documents, a mask interaction strategy for multi-modal features is designed to optimize performance, that is, the dot product value of the local features of the picture on the user input side and the global features of the picture on the document side is masked as negative infinity, in order to reduce the interference of irrelevant features on the correlation calculation; Specifically: for the question text q and the question picture I , if a multi-modal document contains text d and picture I d , the correlation r of the two can be calculated as:

[0019] ;

[0020] Wherein, the matrix Q is q and I the feature matrix obtained after user input side feature extraction and feature mapping, D is d and I d the feature matrix obtained after document side feature extraction and feature mapping, l Q and l DThe number of features contained by Q and D respectively; wherein mask represents a mask interaction strategy, which sets the dot product value of the local features of the user input side picture and the global features of the document side picture to negative infinity, and keeps other values unchanged.

[0021] Further, the process of answer generation in step S5 can be defined in form as follows: let {image} represent the feature sequence of the image after mapping, {text} represent the text sequence, and the answer prediction process of the model be represented as:

[0022]

[0023] Wherein, the parameters of the model are θ , {text} o , which is obtained by the model predicting the word sequence autoregressively, contains a total of L words; {image} q and {text} q respectively represent the question picture and the question text; {image} d 1 and {text} d 1 respectively represent the picture and the text of the multi-modal document with the highest relevance, and so on, and a plurality of multi-modal documents are put in; {text} inst represents the reading instruction, which prompts the model to answer according to the previous input; {text} o,<i represents a plurality of words that have been generated, {text} o,i represents the word that is currently being generated.

[0024] By adopting the above technical solution, the present application has the following beneficial effects:

[0025] The present application uses a multi-modal document combined by a picture and text as a knowledge carrier, and designs a multi-modal retrieval enhanced generation scheme. Compared with the existing end-to-end model scheme, the present scheme is based on a retrieval enhanced generation framework, which guarantees the accuracy and explainability of the answer; compared with the retrieval enhanced generation scheme using a text document as a knowledge carrier, the present scheme adds visual information to the document to construct a multi-modal document, and improves the knowledge retriever and the answer generator to utilize the multi-modal document, thereby improving the accuracy of the knowledge-intensive visual question and answer task. BRIEF DESCRIPTION OF DRAWINGS

[0026] ​In order to make the technical solutions in the specific embodiments or prior art of the present application clearer, the drawings needed in the specific embodiments or prior art description will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0027] Figure 1 A multi-modal retrieval enhancement generation framework provided by the present application is shown in the figure.

[0028] Figure 2 A multi-modal answer generator provided by the present application is shown in the figure. EMBODIMENTS

[0029] The technical solutions of the present application will be described below in detail with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0030] The specific embodiments of the present application will be described in detail below with reference to the drawings. It should be understood that the specific embodiments described here are only used to illustrate and explain the present application, and are not used to limit the present application.

[0031] For knowledge-intensive graphic-text question and answer tasks, the accuracy is low and the explainability is poor when using an end-to-end model. When using a retrieval enhancement generation system with text documents as knowledge carriers, there is a modal difference between the visual content and the text question input by the user and the text documents to be retrieved, so there is a bottleneck in accuracy.

[0032] The present application is inspired by the process of recognizing entities in a picture by comparing entity pictures. A retrieval enhancement generation system based on multi-modal documents is proposed. The present application aims to use multi-modal documents composed of entity pictures and text documents as knowledge carriers to eliminate the modal difference between user input and knowledge carriers, and to design corresponding knowledge retriever and answer generator to utilize multi-modal documents in order to improve the accuracy in knowledge-intensive visual question and answer.

[0033] Data construction part: the scheme of the present application first needs to construct a multi-modal knowledge base composed of multi-modal documents. Since the current knowledge-intensive visual question answering knowledge base is usually composed of text documents, the present application first needs to match all the text documents with corresponding entity pictures to form multi-modal documents. The present application tries to extract encyclopedia page pictures, uses encyclopedia picture search and uses general picture search in order of decreasing accuracy of source until the entity picture is found. Finally, the present application pairs each text document in the original knowledge base with a corresponding entity picture to form a multi-modal knowledge base.

[0034] The multi-modal retrieval enhanced generation framework of the present application is shown in Figure 1 The framework of the present application consists of a multi-modal knowledge retriever and a multi-modal answer generator, wherein the multi-modal knowledge retriever consists of three parts: feature extraction, feature mapping and relevance calculation; the multi-modal answer generator generates answers by reading multiple multi-modal documents. The four parts will be described in more detail below.

[0035] Feature extraction of multi-modal knowledge retriever: the question picture and question text in the user input are encoded into features by image encoder and text encoder respectively, wherein the picture extracts global features and local features, and the text extracts text features. The used encoders are pre-trained and can extract rich meaning features from the corresponding modal input. For the document side, since it is a multi-modal document, in addition to extracting the features of the text part, the present application additionally extracts the global features of the document picture.

[0036] Feature mapping of multi-modal knowledge retriever: in order to perform subsequent relevance calculation, the features just extracted need to be mapped to the same dimension. The global features and local features of the picture are mapped using multi-layer perceptron and Transformer network respectively, and the text is mapped using linear layer. After mapping, all features have the same dimension.

[0037] Relevance calculation of multi-modal knowledge retriever: the relevance calculation of user input and multi-modal document generally follows the relevance calculation method of late interaction. Each feature of the user input side is respectively multiplied with all the features of the document, and the maximum value is taken as the relevance of the input feature, and then all the relevance is summed up as the final relevance. For multi-modal document retrieval, the present application designs a mask interaction strategy for multi-modal features to optimize performance, i.e. the dot product value of the local features of the picture on the user input side and the global features of the picture on the document side is masked as negative infinity, in order to reduce the interference of irrelevant features on the relevance calculation. Specifically:

[0038] For the retrieval of multi-modal documents, a mask interaction strategy for multi-modal features is designed to optimize performance, that is, the dot product value of the local features of the picture on the user input side and the global features of the picture on the document side is masked as negative infinity, so as to reduce the interference of irrelevant features on the relevance calculation; Specifically, for the question text q and question picture I in the user input d , if a multi-modal document contains text I and picture r d , the relevance of the two can be calculated as:

[0039] ;

[0040] Wherein, the matrix Q is q and I the feature matrix obtained through user input side feature extraction and feature mapping, D is d and I d the feature matrix obtained through document side feature extraction and feature mapping, l Q and l D are the number of features contained in Q and D respectively; wherein mask represents the mask interaction strategy, which sets the dot product value of the local features of the picture on the user input side and the global features of the picture on the document side to negative infinity, and other values remain unchanged.

[0041] Multi-modal answer generator: Figure 2 The model structure of the multi-modal answer generator is shown in more detail. When constructing the input of the multi-modal large language model, the picture is mapped into the input space of the large language model through the visual encoder and the visual-to-language mapping module, and is interleaved with the text to form a multi-modal input. The large language model generates a text reply according to the multi-modal input. When training the model as an answer generator, the application further increases a plurality of multi-modal documents with high relevance retrieved after the input of the question picture and the question text, forms an input of interleaved pictures and texts, and constructs a reading example to train the model to generate an answer.

[0042] The process of answer generation can be defined as follows:

[0043] Let {image} represent the feature sequence after image mapping, and {text} represent the text sequence. The answer prediction process of the model is:

[0044] ;

[0045] Wherein, the parameters of the model are θ , {text} odenotes the answer text sequence that needs to be predicted, which is obtained by the model predicting the word sequence autoregressively, and contains a total of L words. q and q denote the question picture and question text, respectively. d 1 and d 1 denote the picture and text of the top-ranked multimodal document, and so on, and a plurality of multimodal documents are put in. inst denotes the reading instruction, which prompts the model to answer according to the previous input. o,<i denotes the plurality of words that have been generated, o,i denotes the word that is currently being generated.

[0046] The present application will be further illustrated with an example. When the user uses the multimodal query (MultimodalQuery) in Figure 1 to inquire when the building in the picture opens, the multimodal retriever of the present application will find the top K multimodal documents (Top K Multimodal Documents) from the knowledge base. The first multimodal document corresponds to the correct building “BMW Welt”, and contains its opening date October 17, 2007. The second multimodal document corresponds to the incorrect building “Europa building” and contains its planned completion year 2012. The multimodal answer generator will generate an answer according to the multimodal query and the top K multimodal documents, as shown in Figure 2 , the model gives the correct answer, which is 2007. The process ends.

[0047] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A retrieval enhancement generation method based on multimodal documents, characterized in that: The retrieval enhancement generation method comprises the following steps: S1. Data construction: Construct a multimodal knowledge base consisting of multimodal documents; S2. Feature extraction of the multimodal knowledge retriever: The question image and question text in the user input are encoded into features by the image encoder and text encoder respectively. The image extracts global features and local features, and the text extracts text features. The document image in the multimodal document extracts global features, and the document text extracts text features. S3, Feature Mapping of Multimodal Knowledge Retriever: Map the features extracted in step S2 to the same dimension; the global features and local features of the image are mapped using a multi-layer perceptron and a Transformer network respectively, while the text is mapped using a linear layer; S4. Relevance calculation of the multimodal knowledge retriever: Each feature on the user input side is dot-multiplied with all the features of the document, and the maximum value is taken as the relevance corresponding to the feature on the input side. The final relevance is then summed up. S5. Multimodal answer generation: A large language model generates text responses based on multimodal input; In step S4, for the retrieval of multimodal documents, a mask interaction strategy for multimodal features is designed to optimize the performance, that is, the dot product value of the local features of the user input side image and the global features of the document side image is masked to negative infinity, in order to reduce the interference of irrelevant features on the relevance calculation; specifically: for the question text in the user input q and the problem picture I , if a multimodal document contains text d and pictures I d , the correlation between the two r Calculated as: ； Among them, the matrix Q is q and I The feature matrix obtained by feature extraction and feature mapping on the user input side is D d and I d The feature matrix obtained by feature extraction and feature mapping on the document side, l Q and l D are the number of features contained in Q and D respectively; Where mask represents the mask interaction strategy, which sets the dot product value of the local features of the user input side image and the global features of the document side image to negative infinity, and keeps other values ​​unchanged.

2. The retrieval enhancement generation method based on multimodal documents according to claim 1 is characterized in that: Step S1 is specifically as follows: First, all text documents are matched with corresponding entity images to form multimodal documents. Then, encyclopedia page images are extracted in descending order of source accuracy, using encyclopedia image search and general image search until the entity image is found. Finally, each text document in the original knowledge base is paired with a corresponding entity image to form a multimodal knowledge base.

3. The retrieval enhancement generation method based on multimodal documents according to claim 1 is characterized in that: The answer generation process in step S5 is formally defined as follows: {image} represents the feature sequence after image mapping, {text} represents the text sequence, and the answer prediction process of the model is expressed as: ; The parameters of the model are θ ,{text} o Represents the answer text sequence that needs to be predicted, which is obtained by the model autoregressively predicting the word sequence, including L word; {image} q and {text} q Represents the question image and question text respectively; {image} d 1 and {text} d 1 Represent the image and text of the most relevant multimodal document respectively, and so on into several multimodal documents; {text} inst Represents a reading instruction, and its prompt model answers based on the previous input; {text} o,<i Indicates several words that have been generated, {text} o,i Indicates the word currently being generated.

Images