Image-text question and answer method, system and device based on multi-mode RAG and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to an image-text question answering method, system and device based on multi-modal RAG and a storage medium, and the method comprises the following steps: 1) extracting multi-modal information from a PDF document, representing the multi-modal information as dense vectors, and storing the dense vectors in a text vector database and an image vector database; 2) obtaining a semantic embedding vector of the problem text and a multi-modal embedding vector of the problem text; obtaining a semantic embedding vector of the description text of the problem image and a multi-mode embedding vector of the problem image; 3) performing coarse screening in the text vector database and the image vector database by using the semantic embedding vector and the multi-modal embedding vector, finding out coarse screening text data and coarse screening image data, performing multi-modal fine ranking on the coarse screening text data and the coarse screening image data, and obtaining text data and image data which are retrieved and recalled; and 4) generating a final answer by the multi-modal large language model. According to the method, multi-modal data in a long document can be effectively analyzed, and information most relevant to a problem can be accurately retrieved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to a picture-text question-answering method, system, device and storage medium, and in particular to a picture-text question-answering method, system, device and storage medium based on multimodal RAG. Background Art

[0002] At present, document question answering technology is mainly divided into large language model methods based on text information and large visual language model methods based on images.

[0003] Among them, the large language model method based on text information mainly uses OCR (Optical Character Recognition) to recognize the text in the document and convert the chart content in the document into machine-readable text information, which facilitates the large language model to process and understand the document. However, OCR can only extract text information from the image and cannot effectively capture other important visual content in the image, such as charts, graphics, or data visualization. Therefore, when faced with documents with mixed text and images, OCR has difficulty in comprehensively understanding the text and image information, which limits its application effect in multimodal question-answering systems.

[0004] Image-based visual large language model methods often face the problem of insufficient access to textual information. The core information of a document is usually in text form. Purely visual methods tend to overlook detailed textual content and rely solely on image features (such as the document's layout and charts), potentially missing key information. Furthermore, user questions may involve abstract concepts, logical reasoning, or causal relationships. These types of questions require semantic analysis of the textual content, which pure visual methods cannot handle. Visual features are also easily affected by document formatting, fonts, and background, resulting in reduced system robustness.

[0005] Retrieval-Augmented Generation (RAG) technology effectively alleviates the knowledge limitations of large language models (LLMs) by combining information retrieval and generation models, and is widely used in question-answering systems, web page generation and other fields. Traditional RAG systems mainly rely on text modalities and enhance the generation results by retrieving text fragments from external knowledge bases. However, with the popularization of multimodal data, traditional RAG has significant defects in the following aspects: 1. Single-modal dependence: only processing text information, ignoring the semantic associations of visual elements such as images and tables, resulting in one-sided semantic understanding in complex scenarios. 2. Severe information loss: When converting images into text descriptions, key information such as spatial layout and chart details may be lost. 3. Low retrieval efficiency: No efficient index structure is designed for multimodal data, and cross-modal retrieval latency is high.

[0006] Based on this, there are three problems in the existing multimodal long document question answering: 1. Large models cannot process the text and image data of documents that are dozens of pages long. When faced with long documents, existing question answering systems are unable to effectively process long documents with large amounts of text and many images due to input length limitations. Traditional models can often only process limited context, resulting in an inability to fully understand the document content, which in turn affects the accuracy and generation quality of question answering. 2. Insufficient fusion of multimodal data. Existing RAG systems rely on a single modality (such as plain text or images) to parse documents, and do not fully combine text and image information, resulting in incomplete recalled information, which in turn affects the question answering results. 3. Lack of fine-ranking strategies for multimodal data. Existing solutions lack the joint screening of multimodal features in the fine-ranking stage of recall, resulting in the inability to select the most useful information from multimodal data, and the final question answering results are highly biased.

[0007] Therefore, how to effectively parse multimodal data in long documents and accurately retrieve the information most relevant to the question is a key technical problem that needs to be urgently solved in multimodal long document question answering. Summary of the Invention

[0008] In order to overcome the defects of the prior art, the present invention proposes a picture-text question-answering method, system, device and storage medium based on multimodal RAG. On the basis of traditional RAG (which only processes text), it expands the processing capability of multimodal data such as images, improves the accuracy and richness of the generated content by fusing multimodal information, and makes up for the defects of traditional RAG.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A multimodal RAG-based image-text question-answering method, characterized by comprising the following steps:

[0011] 1) PDF document multimodal parsing and storage: Extract multimodal information from PDF documents and use vector models to represent the extracted multimodal information as dense vectors, which are stored in text vector databases and image vector databases;

[0012] 2) User Question Preprocessing: For the text portion of the user question, it is input into the semantic vector model and the multimodal vector model respectively to obtain the semantic embedding vector of the question text and the multimodal embedding vector of the question text. For the image portion of the user question, a multimodal large language model is used to generate a description text of the question image. The description text of the question image is input into the semantic vector model to obtain the semantic embedding vector of the description text of the question image. At the same time, the image portion of the user question is input into the multimodal vector model to obtain the multimodal embedding vector of the question image.

[0013] 3) Retrieval and recall: Using the semantic embedding vector of the question text, the multimodal embedding vector of the question text, the semantic embedding vector of the description text of the question image, and the multimodal embedding vector of the question image, a coarse screening is performed in the text vector database and the image vector database using similarity search to find the coarse screened text data and coarse screened image data that are most relevant to the user's question, and the coarse screened text data and coarse screened image data are subjected to multimodal fine sorting, and the text data and image data for retrieval and recall are obtained based on the multimodal fine sorting results;

[0014] 4) Question and answer generation: The retrieved text data, image data, and description text of the user question and question image are input into the multimodal large language model, and the multimodal large language model generates the final answer.

[0015] Preferably, the multimodal parsing and storage of the PDF document in step 1) specifically includes:

[0016] 11) Parsing: Extract latex-formatted formula text, text, formatted table text and table titles, images and image titles, metadata, and images and image blocks corresponding to each page of the PDF document from the PDF document;

[0017] 12) Data Processing and Vectorization: Generate semantic embedding vectors based on the extracted latex-formatted formula text, text, formatted table text and table titles, and images and image titles using a semantic vector model. Generate multimodal embedding vectors based on the extracted images, the images corresponding to each page of the PDF document, and the divided image blocks using a multimodal vector model.

[0018] 13) Storage: The generated semantic embedding vector and its corresponding metadata are stored in a text vector database, and the generated multimodal embedding vector and its corresponding metadata are stored in an image vector database.

[0019] Preferably, the analysis in step 11) specifically includes:

[0020] 111) Layout Parsing: Use open source layout analysis tools to parse PDF documents to obtain formulas in the form of images, text in the form of images, tables and table titles in the form of images, and images and image titles. Use the formula extraction model to extract formula text in latex format from formulas in the form of images, use the OCR model to extract text from text in the form of images, and use the image-to-structured table text model to extract formatted table text from tables in the form of images.

[0021] 112) Image parsing: using a PDF to image tool to convert each page of the PDF document into an image, and dividing the image into a number of image blocks according to the aspect ratio of the image;

[0022] 113) Metadata Extraction: Extract the page number of each page of a PDF document as metadata.

[0023] Preferably, the data processing and vectorization in step 12) specifically include:

[0024] 121) For the text in the PDF document and the latex format formula text, the corresponding semantic embedding vector is generated using the semantic vector model;

[0025] 122) For tables in PDF documents, after concatenating the formatted table text and table title, a semantic embedding vector is generated using a semantic vector model;

[0026] 123) For images in PD documents, we first design prompt words and use a multimodal large language model to analyze the image content, generate a description of the image content, and then use a semantic vector model to generate a semantic embedding vector for the image content description text and the image title; at the same time, we use the multimodal vector model to directly generate a multimodal embedding vector for the image;

[0027] 124) For each image and each image block corresponding to each page of the PDF document, a corresponding multimodal embedding vector is generated using a multimodal vector model.

[0028] Preferably, in 112), the image is regularly divided into several image blocks according to the aspect ratio of the image, specifically: if the image is a square image, a 2x2 grid is used for segmentation; if the image is a horizontally long image, a 1xN grid is used for horizontal segmentation, N=2,3,4; if the image is a vertically long image, an Nx1 grid is used for vertical segmentation, N=2,3,4.

[0029] Preferably, the step 3) of performing multimodal fine sorting on the coarsely screened text data and the coarsely screened image data and obtaining retrieved text data and image data according to the multimodal fine sorting results specifically includes:

[0030] 31) Finding document pages whose text data and image data have been roughly screened based on the metadata of the roughly screened text data and the roughly screened image data, and using the roughly screened text data and the roughly screened image data corresponding to these document pages as the text data and image data to be retrieved;

[0031] 32) re-sorting the remaining coarse-screened text data with the help of a refined sorting tool, and extracting the remaining coarse-screened text data with a high ranking as the text data for retrieval and recall;

[0032] 33) For the remaining coarse-screened image data, perform secondary semantic vector rearrangement and secondary image vector rearrangement, and integrate them, and use the remaining coarse-screened image data with the highest ranking after integration as the image data for retrieval and recall.

[0033] Preferably, the step 33) specifically includes:

[0034] 331) Secondary semantic vector rearrangement: Input the remaining coarse-screened image data and the topic of the user's question into the multimodal large language model, and generate a description of the topic of the user's question by the multimodal large language model. Based on the description of the topic of the user's question, a new semantic embedding vector is obtained using the semantic vector model, and the similarity between the new semantic embedding vector and the semantic embedding vector of the question text is calculated to obtain a semantic similarity score;

[0035] 332) Secondary image vector rearrangement: Based on the remaining coarse-screened image data and the subject of the user's question, a new multimodal embedding vector is obtained using a multimodal vector model, and the similarity between the new multimodal embedding vector and the multimodal embedding vector of the question text is calculated to obtain a multimodal similarity score;

[0036] 333) Re-ranking result integration: Add the semantic similarity score and multimodal similarity score of the same coarse-screened image data to obtain a secondary re-ranking score, and use the remaining coarse-screened image data with the highest secondary re-ranking score as the image data for retrieval and recall.

[0037] In addition, the present invention also provides a multimodal RAG-based graphic question-and-answer system, which is characterized by comprising:

[0038] PDF document multimodal parsing and storage module, which is used to extract multimodal information from PDF documents and use vector models to represent the extracted multimodal information as dense vectors, which are stored in text vector databases and image vector databases;

[0039] A user question preprocessing module is used to preprocess user questions. For the text portion of the user question, it is input into the semantic vector model and the multimodal vector model respectively to obtain the semantic embedding vector of the question text and the multimodal embedding vector of the question text; for the image portion of the user question, a description text of the question image is generated using a multimodal large language model, and the description text of the question image is input into the semantic vector model to obtain the semantic embedding vector of the description text of the question image. At the same time, the image portion of the user question is input into the multimodal vector model to obtain the multimodal embedding vector of the question image;

[0040] A retrieval and recall module is configured to use the semantic embedding vector of the question text, the multimodal embedding vector of the question text, the semantic embedding vector of the description text of the question image, and the multimodal embedding vector of the question image to perform a rough screening in the text vector database and the image vector database using similarity retrieval, respectively, to find the rough screened text data and rough screened image data that are most relevant to the user's question, perform multimodal fine sorting on the rough screened text data and the rough screened image data, and obtain retrieved and recalled text data and image data based on the multimodal fine sorting results;

[0041] The question-answer generation module is used to input the retrieved text data, image data, and description text of the user question and question image into the multimodal large language model, and the multimodal large language model generates the final answer.

[0042] Furthermore, the present invention also provides a multimodal RAG-based graphic question-and-answer device, which is characterized by comprising:

[0043] one or more processors;

[0044] a memory for storing one or more programs;

[0045] When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal RAG-based graphic question-and-answer method as described above.

[0046] A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the multimodal RAG-based graphic question-and-answer method as described above are implemented.

[0047] The multimodal RAG-based image-text question-answering method, system, device, and storage medium of the present invention combine the powerful feature extraction capabilities of a multimodal large language model with the precise retrieval capabilities of RAG technology. Compared with the existing technology, it has one or more of the following beneficial technical effects:

[0048] 1. Multimodal feature extraction: This paper uses technologies such as multimodal large language models, semantic vector models, and multimodal vector models to simultaneously parse the text and image information of PDF documents, understand the document content from multiple perspectives, and fully mine the information of each modality in the document.

[0049] 2. Complex PDF processing capabilities: This invention uses minerU to parse PDF. MinerU's layout parsing supports complex structures such as multiple columns and nested tables, which can better extract information from PDFs. At the same time, with the help of technologies such as multimodal large language models, semantic vector models, and multimodal vector models, it can process complex text and image information in PDFs, providing powerful PDF parsing capabilities.

[0050] 3. Multimodal question and answer: Based on its own multimodal processing capabilities, the present invention allows users to input multimodal query information such as images and text, expanding the capabilities of the question and answer system.

[0051] 4. Two-stage multimodal recall: Based on user questions and answers, the text and image information in the database are retrieved and sorted separately, and the data for the first recall is obtained through rough screening. In the multimodal fine sorting stage, highly relevant data is obtained by integrating the text and image data recalled in the rough screening. By re-processing and re-sorting the text data and the image data in combination with the query information, multimodal data highly relevant to the query is recalled again. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the multimodal RAG-based picture-text question-answering method of the present invention.

[0053] Figure 2 It is a schematic diagram of multimodal parsing and storage of PDF documents in the present invention.

[0054] Figure 3 It is a schematic diagram of image analysis in the present invention.

[0055] Figure 4 It is a schematic diagram of the question-answer generation process in the present invention.

[0056] Figure 5 It is a schematic diagram of the composition of the multimodal RAG-based picture-text question-answering system of the present invention. DETAILED DESCRIPTION

[0057] Before describing in detail any embodiment of the present invention, it should be understood that the present invention is not limited in its application to the construction and arrangement details of the components set forth in the following description or illustrated in the following figures. The present invention is capable of other embodiments and can be practiced or carried out in various ways. In addition, it should be understood that the words and terms used herein are for descriptive purposes and should not be considered restrictive. The use of "including" or "having" and their variations herein is intended to cover the items and their equivalents set forth below and additional items. Unless otherwise specified or limited, the terms "mounted", "connected", "supported" and "coupled" and their variations are used broadly and cover direct mounting and indirect mounting, connection, support and coupling. In addition, "connected" and "coupled" are not limited to physical or mechanical connections or couplings.

[0058] Furthermore, on the first hand, in the disclosure of the present invention, the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore the above terms cannot be understood as limitations on the present invention; on the second hand, the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" cannot be understood as a limitation on the quantity.

[0059] In order to overcome the defects of the prior art, the present invention provides a picture-text question-answering method based on multimodal RAG, which is an enhanced artificial intelligence technology that combines multimodal information retrieval and generation technology. On the basis of traditional RAG (which only processes text), it expands the processing capability of multiple modal data such as images, and improves the accuracy and richness of generated content by fusing multimodal information, thus making up for the defects of traditional RAG.

[0060] Figure 1 FIG. 4 shows a flow chart of the multimodal RAG-based picture-text question-answering method of the present invention. Figure 1 As shown, the multimodal RAG-based picture-text question-answering method of the present invention includes the following steps:

[0061] 1. Multimodal parsing and storage of PDF documents.

[0062] This step aims to obtain text information and image information from the existing numerous PDF documents, and to establish a corresponding database based on the obtained text information and image information, so that when users ask questions, corresponding retrieval results can be provided according to user questions, thereby helping to produce answers.

[0063] In the present invention, multimodal information (including formulas, texts, pictures, tables, etc.) is extracted from PDF documents, and the extracted multimodal information is represented as dense vectors using a vector model and stored in a text vector database and an image vector database. Figure 2 As shown in the figure, multimodal parsing and storage of PDF documents specifically include:

[0064] 1. Analysis.

[0065] Extract latex format formula text, text, formatted table text and table title, pictures and picture titles, metadata, and images and divided image blocks corresponding to each page of the PDF document from PDF documents.

[0066] Specifically, the analysis includes:

[0067] (1) Layout analysis.

[0068] Use open source layout analysis tools (such as MinerU) to parse the PDF document to obtain formulas in the form of pictures, text in the form of pictures, tables and table titles in the form of pictures, pictures and picture titles.

[0069] At the same time, the formula extraction model is used to extract formula text in latex format from the formula in picture form, the OCR model is used to obtain text from text in picture form, and the picture-to-structured table text model is used to extract formatted table text from the table in picture form.

[0070] Among them, different types of data include different fields. For example, the fields of text include: type (text, indicating text), text (text string), text_level (whether it is a title), page_idx (page number); the fields of picture include: type (image, indicating picture), img_path (picture path), img_caption (picture title), img_footnote (picture footnote), page_idx (page number); the fields of table include: type (table, indicating table), table_path (table path), table_caption (table title), table_footnote (table footnote), table_body (table body), page_idx (page number), etc.

[0071] For formulas, the present invention uses MinerU's built-in yolo_v8_mfd and unimernet_small formula extraction models to convert the formulas into Latex format and embed them into a text type (type=text) string to form a Latex-formatted formula text. For tables, the table body is identified as HTML format using MinerU's built-in rapid_table table content extraction model (i.e., an image-to-structured table text model) and converted into structured table text.

[0072] (2) Image analysis.

[0073] For a PDF document, the present invention further utilizes a PDF to image tool to convert each page of the PDF document into an image, and regularly divides the image into a plurality of image blocks according to the aspect ratio of the image.

[0074] Specifically, if Figure 3As shown, the present invention uses pdf2image to convert each page of a PDF document into a high-resolution image. To more effectively extract information from the high-resolution image, the present invention follows the AnyRes dynamic image segmentation strategy of LLaVA-1.5, automatically selecting an appropriate segmentation mode based on the image's aspect ratio to regularly divide the high-resolution image into several image blocks. This process dynamically segments the input image according to the following strategy: a 2x2 grid is used for approximately square images; a 1xN grid is used for horizontal segmentation (N=2, 3, 4) for horizontally long images; and an Nx1 grid is used for vertically long images (N=2, 3, 4). The segmented image blocks include some overlapping edges while preserving the original image. Finally, the converted image and the segmented image blocks are combined to form several images.

[0075] (3) Metadata extraction.

[0076] In the present invention, when performing layout parsing and image parsing, the page number of each page of the PDF document is extracted as metadata to facilitate identification of the information source of the parsed text and image.

[0077] 2. Data processing and vectorization.

[0078] In the present invention, a semantic embedding vector can be generated based on the extracted latex format formula text, text, formatted table text and table title, and pictures and picture titles using a semantic vector model, and a multimodal embedding vector can be generated based on the extracted pictures and the images corresponding to each page of the PDF document and the divided image blocks using a multimodal vector model.

[0079] The core task of this step is to process the parsing results into vectors. Specifically:

[0080] For text (including text parsed from layout and latex format formula text), the present invention directly uses a semantic vector model (such as BGE-m3, etc.) to generate a corresponding semantic embedding vector.

[0081] For tables, the present invention concatenates and integrates the formatted table text and table title, and then uses a semantic vector model (such as BGE-m3, etc.) to generate a corresponding semantic embedding vector.

[0082] For images parsed from PDFs, the present invention first designs prompt words (for example, "Please provide adetailed explanation of the content, including specific numerical values, text, symbols, and other character records that appear."). It then uses an MLLM model (multimodal large language model) to deeply analyze the image content and generate a detailed description of the image content. It then uses a semantic vector model to generate semantic embedding vectors for the image title and the text describing the image content. Furthermore, the present invention uses a multimodal vector model (such as BGE-VL) to directly generate a multimodal embedding vector for the image.

[0083] For the original image and each image block corresponding to each page of PDF, the present invention directly uses a multimodal vector model (such as BGE-VL, etc.) to generate a corresponding multimodal embedding vector.

[0084] 3. Storage.

[0085] The generated semantic embedding vectors and their corresponding metadata are stored in a text vector database, and the generated multimodal embedding vectors and their corresponding metadata are stored in an image vector database. Storing the corresponding metadata facilitates subsequent retrieval and integration. The text vector database and image vector database can be based on the Milvus database.

[0086] Through the above step 1, the text vector database and image vector database for retrieval and query have been obtained. Next, based on the text vector database and image vector database, relevant information can be retrieved and recalled according to the user's question, and answers can be generated based on the question and information. The specific process is as follows: Figure 4 shown.

[0087] 2. User problem preprocessing.

[0088] like Figure 4 As shown, in the present invention, the user question is pre-processed first. The present invention allows users to query using multimodal information such as text and images (i.e., multimodal query), and through dual-path information processing, the present invention simultaneously captures the semantic and visual information in the query:

[0089] For the text part of the user's question, it is input into the semantic vector model and the multimodal vector model respectively to obtain the semantic embedding vector of the question text and the multimodal embedding vector of the question text, so as to facilitate the use of the semantic embedding vector of the question text to query the text data in the text vector database, and use the multimodal embedding vector of the question text to query the image data in the image vector database;

[0090] For the image part of the user's question, a multimodal large language model is used to generate a description text of the question image, and the description text is input into the semantic vector model to obtain the semantic embedding vector of the description text of the question image. At the same time, the image part of the user's question is input into the multimodal vector model to obtain the multimodal embedding vector of the question image, so as to use the semantic embedding vector of the description text of the question image to query the text data in the text vector database, and use the multimodal embedding vector of the question image to query the image data in the image vector database.

[0091] 3. Retrieval and recall.

[0092] After preprocessing the user question, the semantic embedding vector of the question text, the multimodal embedding vector of the question text, the semantic embedding vector of the description text of the question image, and the multimodal embedding vector of the question image are used to perform coarse screening in the text vector database and the image vector database using similarity retrieval respectively, to find the coarse-screened text data (i.e., Top N text data) and coarse-screened image data (i.e., TopN image data) that are most relevant to the user question (for example, similarity greater than 85%, etc.), and a multimodal fine sorting strategy is used to perform multimodal fine sorting on the coarse-screened text data and coarse-screened image data, and the retrieved text data and image data (i.e., Top K recalled data) are obtained based on the multimodal fine sorting results.

[0093] The multimodal fine sorting of the coarsely screened text data and the coarsely screened image data and obtaining the retrieved text data and image data according to the multimodal fine sorting results specifically includes:

[0094] 1. Based on the metadata of the coarse-screened text data and coarse-screened image data, find the document pages whose text data and image data have been coarsely screened. These documents are considered to be highly relevant to the user's question. The coarse-screened text data and coarse-screened image data corresponding to these document pages are used as the text data and image data for retrieval and recall.

[0095] 2. For some document pages, only text data is retrieved after rough screening but no image data is retrieved. In this case, a fine ranking tool (such as BGE-reranker) is used to rerank the remaining rough-screened text data. The remaining rough-screened text data with a high ranking (for example, the top 20%, top 30%, etc.) is extracted as the text data for retrieval.

[0096] 3. For the remaining rough-screened image data (for some document pages, only image data but no text data was retrieved after rough screening), perform secondary semantic vector re-ranking and secondary image vector re-ranking, and integrate them. The remaining rough-screened image data that ranks high after integration (for example, top 20%, top 30%, etc.) is used as the image data for retrieval recall. This specifically includes:

[0097] (1) Secondary semantic vector rearrangement: The remaining coarse-screened image data and the topic of the user's question are input into the multimodal large language model. The multimodal large language model generates a description of the topic of the user's question. Based on the description of the topic of the user's question, a new semantic embedding vector is obtained using the semantic vector model. The similarity between the new semantic embedding vector and the semantic embedding vector of the question text is calculated to obtain a semantic similarity score.

[0098] (2) Secondary image vector rearrangement: Based on the remaining coarse-screened image data and the topic of the user's question, a new multimodal embedding vector is obtained using a multimodal vector model. The similarity between the new multimodal embedding vector and the multimodal embedding vector of the question text is calculated to obtain a multimodal similarity score.

[0099] (3) Integration of rearrangement results: The semantic similarity score and multimodal similarity score of the same coarse-screened image data are added together to obtain a secondary rearrangement score. The remaining coarse-screened image data with a high secondary rearrangement score (for example, the top 20%, the top 30%, etc.) are used as the image data for retrieval and recall.

[0100] Of course, when determining the number of image data to be retrieved and recalled, the maximum number of image inputs of the multimodal large language model also needs to be considered, so that the number of recalled image data does not exceed the maximum number of image inputs of the multimodal large language model.

[0101] 4. Question and answer generation.

[0102] In the present invention, the retrieved text data, image data, and description text of the user question and question image are input into a multimodal large language model, and the final answer is generated by the multimodal large language model.

[0103] Figure 5 FIG. 1 shows a schematic diagram of the composition of the multimodal RAG-based picture-text question answering system of the present invention. Figure 5 As shown, the multimodal RAG-based picture-text question answering system of the present invention includes:

[0104] 1. PDF document multimodal parsing and storage module.

[0105] The PDF document multimodal parsing and storage module is used to extract multimodal information from the PDF document, and use a vector model to represent the extracted multimodal information as a dense vector, which is stored in a text vector database and an image vector database.

[0106] 2. User question preprocessing module.

[0107] The user question preprocessing module is used to preprocess user questions, wherein the text part of the user question is input into the semantic vector model and the multimodal vector model respectively to obtain the semantic embedding vector of the question text and the multimodal embedding vector of the question text; for the image part of the user question, a description text of the question image is generated using a multimodal large language model, the description text of the question image is input into the semantic vector model to obtain the semantic embedding vector of the description text of the question image, and at the same time, the image part of the user question is input into the multimodal vector model to obtain the multimodal embedding vector of the question image.

[0108] 3. Retrieval and recall module.

[0109] The retrieval and recall module is used to use the semantic embedding vector of the question text, the multimodal embedding vector of the question text, the semantic embedding vector of the description text of the question image, and the multimodal embedding vector of the question image to perform coarse screening in the text vector database and the image vector database using similarity retrieval respectively, to find the coarse-screened text data and coarse-screened image data that are most relevant to the user's question, to perform multimodal fine sorting on the coarse-screened text data and coarse-screened image data, and to obtain the retrieved and recalled text data and image data based on the multimodal fine sorting results.

[0110] 4. Question and answer generation module.

[0111] The question-answer generation module is used to input the retrieved text data, image data, and description text of the user question and question image into the multimodal large language model, and the multimodal large language model generates the final answer.

[0112] In addition, the present invention also provides a graphic and text question and answer device based on multimodal RAG, which includes: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the graphic and text question and answer method based on multimodal RAG as described above.

[0113] Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal RAG-based graphic question-and-answer method as described above.

[0114] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art may, based on the principles of the present invention, modify or replace the technical solutions of the present invention with equivalents without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A multimodal RAG-based image-text question answering method, characterized in that: The following steps are involved: 1) PDF document multimodal parsing and storage: Extract multimodal information from PDF documents and use vector models to represent the extracted multimodal information as dense vectors, which are stored in text vector databases and image vector databases; 2) User Question Preprocessing: For the text portion of the user question, it is input into the semantic vector model and the multimodal vector model respectively to obtain the semantic embedding vector of the question text and the multimodal embedding vector of the question text. For the image portion of the user question, a multimodal large language model is used to generate a description text of the question image. The description text of the question image is input into the semantic vector model to obtain the semantic embedding vector of the description text of the question image. At the same time, the image portion of the user question is input into the multimodal vector model to obtain the multimodal embedding vector of the question image. 3) Retrieval and recall: Using the semantic embedding vector of the question text, the multimodal embedding vector of the question text, the semantic embedding vector of the description text of the question image, and the multimodal embedding vector of the question image, a coarse screening is performed in the text vector database and the image vector database using similarity search to find the coarse screened text data and coarse screened image data that are most relevant to the user's question, and the coarse screened text data and coarse screened image data are subjected to multimodal fine sorting, and the text data and image data for retrieval and recall are obtained based on the multimodal fine sorting results; 4) Question and answer generation: The retrieved text data, image data, and description text of the user question and question image are input into the multimodal large language model, and the multimodal large language model generates the final answer.

2. The multimodal RAG-based picture-text question-answering method according to claim 1, characterized in that: The multimodal parsing and storage of the PDF document in step 1) specifically includes: 11) Parsing: Extract latex-formatted formula text, text, formatted table text and table titles, images and image titles, metadata, and images and image blocks corresponding to each page of the PDF document from the PDF document; 12) Data Processing and Vectorization: Generate semantic embedding vectors based on the extracted latex-formatted formula text, text, formatted table text and table titles, and images and image titles using a semantic vector model. Generate multimodal embedding vectors based on the extracted images, the images corresponding to each page of the PDF document, and the divided image blocks using a multimodal vector model. 13) Storage: The generated semantic embedding vector and its corresponding metadata are stored in a text vector database, and the generated multimodal embedding vector and its corresponding metadata are stored in an image vector database.

3. The multimodal RAG-based picture-text question-answering method according to claim 2, characterized in that: The analysis in step 11) specifically includes: 111) Layout Parsing: Use open source layout analysis tools to parse PDF documents to obtain formulas in the form of images, text in the form of images, tables and table titles in the form of images, and images and image titles. Use the formula extraction model to extract formula text in latex format from formulas in the form of images, use the OCR model to extract text from text in the form of images, and use the image-to-structured table text model to extract formatted table text from tables in the form of images. 112) Image parsing: using a PDF to image tool to convert each page of the PDF document into an image, and dividing the image into a number of image blocks according to the aspect ratio of the image; 113) Metadata Extraction: Extract the page number of each page of a PDF document as metadata.

4. The multimodal RAG-based picture-text question-answering method according to claim 3, characterized in that: The data processing and vectorization in step 12) specifically include: 121) For the text in the PDF document and the latex format formula text, the corresponding semantic embedding vector is generated using the semantic vector model; 122) For tables in PDF documents, after concatenating the formatted table text and table title, a semantic embedding vector is generated using a semantic vector model; 123) For images in PD documents, we first design prompt words and use a multimodal large language model to analyze the image content, generate a description of the image content, and then use a semantic vector model to generate a semantic embedding vector for the image content description text and the image title; at the same time, we use the multimodal vector model to directly generate a multimodal embedding vector for the image; 124) For each image and each image block corresponding to each page of the PDF document, a corresponding multimodal embedding vector is generated using a multimodal vector model.

5. The multimodal RAG-based picture-text question-answering method according to claim 3, characterized in that: In the step 112), the image is regularly divided into a number of image blocks according to the aspect ratio of the image: if the image is a square image, a 2x2 grid is used for segmentation; if the image is a horizontally long image, a 1xN grid is used for horizontal segmentation, where N=2, 3, or 4; if the image is a vertically long image, an Nx1 grid is used for vertical segmentation, where N=2, 3, or 4.

6. The multimodal RAG-based picture-text question-answering method according to any one of claims 1 to 5, characterized in that: The step 3) of performing multimodal fine sorting on the coarsely screened text data and coarsely screened image data and obtaining retrieved text data and image data according to the multimodal fine sorting results specifically includes: 31) Finding document pages whose text data and image data have been roughly screened based on the metadata of the roughly screened text data and the roughly screened image data, and using the roughly screened text data and the roughly screened image data corresponding to these document pages as the text data and image data to be retrieved; 32) re-sorting the remaining coarse-screened text data with the help of a refined sorting tool, and extracting the remaining coarse-screened text data with a high ranking as the text data for retrieval and recall; 33) For the remaining coarse-screened image data, perform secondary semantic vector rearrangement and secondary image vector rearrangement, and integrate them, and use the remaining coarse-screened image data with the highest ranking after integration as the image data for retrieval and recall.

7. The multimodal RAG-based picture-text question-answering method according to claim 6, characterized in that: The step 33) specifically includes: 331) Secondary semantic vector rearrangement: Input the remaining coarse-screened image data and the topic of the user's question into the multimodal large language model, and generate a description of the topic of the user's question by the multimodal large language model. Based on the description of the topic of the user's question, a new semantic embedding vector is obtained using the semantic vector model, and the similarity between the new semantic embedding vector and the semantic embedding vector of the question text is calculated to obtain a semantic similarity score; 332) Secondary image vector rearrangement: Based on the remaining coarse-screened image data and the subject of the user's question, a new multimodal embedding vector is obtained using a multimodal vector model, and the similarity between the new multimodal embedding vector and the multimodal embedding vector of the question text is calculated to obtain a multimodal similarity score; 333) Re-ranking result integration: Add the semantic similarity score and multimodal similarity score of the same coarse-screened image data to obtain a secondary re-ranking score, and use the remaining coarse-screened image data with the highest secondary re-ranking score as the image data for retrieval and recall.

8. A multimodal RAG-based picture-text question answering system, characterized in that: include: PDF document multimodal parsing and storage module, which is used to extract multimodal information from PDF documents and use vector models to represent the extracted multimodal information as dense vectors, which are stored in text vector databases and image vector databases; A user question preprocessing module is used to preprocess user questions. For the text portion of the user question, it is input into the semantic vector model and the multimodal vector model respectively to obtain the semantic embedding vector of the question text and the multimodal embedding vector of the question text; for the image portion of the user question, a description text of the question image is generated using a multimodal large language model, and the description text of the question image is input into the semantic vector model to obtain the semantic embedding vector of the description text of the question image. At the same time, the image portion of the user question is input into the multimodal vector model to obtain the multimodal embedding vector of the question image; A retrieval and recall module is configured to use the semantic embedding vector of the question text, the multimodal embedding vector of the question text, the semantic embedding vector of the description text of the question image, and the multimodal embedding vector of the question image to perform a rough screening in the text vector database and the image vector database using similarity retrieval, respectively, to find the rough screened text data and rough screened image data that are most relevant to the user's question, perform multimodal fine sorting on the rough screened text data and the rough screened image data, and obtain retrieved and recalled text data and image data based on the multimodal fine sorting results; The question-answer generation module is used to input the retrieved text data, image data, and description text of the user question and question image into the multimodal large language model, and the multimodal large language model generates the final answer.

9. A multimodal RAG-based graphic question-and-answer device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal RAG-based image-text question-answering method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the multimodal RAG-based picture-text question-answering method as described in any one of claims 1 to 7 are implemented.

Images