Multi-modal retrieval enhancement generation technology-based vertical industry document intelligent analysis and question answering method

Abstract

The invention relates to a vertical industry document intelligent analysis and question-answering method based on a multi-modal retrieval enhancement generation technology, which can comprehensively process various data in a document and remarkably improve the overall analysis capability of a large model on the document. By deeply analyzing multi-modal data such as texts, tables and charts in the document, the method enables document analysis to be more comprehensive and accurate. According to the method, a vector database and a graph database are combined for efficiently storing and retrieving information. The combination not only improves the ability of the system to understand the context, but also significantly improves the accuracy of retrieval. Meanwhile, the output retrieval result can provide evidence for the result obtained by the user by using the method, so that the output result of the method has acceptable accuracy in various vertical fields.

Description

Technical Field

[0001] The present invention relates to the field of data retrieval and analysis, and in particular to a vertical industry document intelligent parsing and question-answering method based on multimodal retrieval enhancement generation technology. Background Art

[0002] In recent years, with the rapid development of vertical industries, the data in related documents has expanded from single text data to multimodal data including text, tables and charts, which makes it more difficult to parse and retrieve information in these documents. Document parsing is a complex task. The various non-text data in it carry a lot of key information and are of great value to people, but they are often overlooked due to the complexity of processing. If all data can be fully processed when parsing a document, when a user asks a question about the document, it can be searched in all the data, and the large language model can be used to integrate the search results and generate responses, which will significantly improve the accuracy of the answer. At the same time, this method will also enhance the visualization and decision-making process of vertical field data.

[0003] Traditional document analysis methods rely on manual processing and single-modal data, which is time-consuming and error-prone. Existing work often ignores tables and charts in documents, which poses challenges to the extraction and comprehensive application of document data. At the same time, the limitations of large models in vertical field decision-making include limited context windows and easy hallucinations, which make it impossible for large models to process the entire document at once and people doubt the reliability of the intelligent question-answering results they produce.

[0004] Therefore, comprehensive processing of data in documents and optimization of retrieval methods are the current research focuses of intelligent document parsing and question answering in vertical industries. At present, the combination of large language models (LLMs) and multimodal retrieval-augmented generation (RAG) is the main means of document parsing and question answering. This method not only makes the parsing more comprehensive but also the answers more accurate. For the sake of simplicity, "large language model" can also be referred to as "large model". Summary of the invention

[0005] In view of the fact that the original data processing methods are not comprehensive in document parsing due to the diversity and complexity of data presentation in existing vertical industry documents, and the problems of insufficient understanding of context and insufficient accuracy of traditional single data storage and retrieval methods, the present invention proposes a vertical industry document intelligent parsing and question-answering method based on multimodal retrieval enhanced generation technology, which can optimize the computer's comprehensive processing and retrieval of all data in the document, and alleviate the pressure brought by the increasing number of vertical field documents and complex data processing.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The vertical industry document intelligent parsing and question-answering method based on multimodal retrieval enhanced generation technology includes the following steps:

[0008] Step 1: Collect a comprehensive document data set in Chinese and English formats, where the document contains natural language text, tables, and chart data, and preprocess the document data in the comprehensive document data set to obtain preprocessed data;

[0009] Step 2: The preprocessed data is processed by the embedding model and then stored in the vector database. The preprocessed data is extracted by the large language model and the relationships between entities are stored in the graph database.

[0010] Step 3: After the input question is embedded by the embedding model and the large language model extracts entities and relationships between entities, the query vector and query graph are obtained;

[0011] Step 4: Input the graph to be queried into the graph database to query and retrieve relevant content to obtain relevant graph data, input the vector to be queried into the vector database to query and retrieve relevant content, re-evaluate and sort the retrieval results of the vector database by the re-ranking model to obtain relevant context data, integrate the relevant context data and the relevant graph data and input them into the large language model;

[0012] Step 5: The large language model performs a fact-checking process on the relevant context data and relevant graph data based on the query vector and the query graph, and then analyzes and integrates them to obtain the final retrieval result.

[0013] Furthermore, in step 1, the preprocessing method is to preprocess the document to extract individual components, use OCR to extract text from the scanned document to obtain natural language text data, use an object detection model to identify and extract tables and charts in the document as images, and use a visual language model to convert each image into Markdown format.

[0014] Furthermore, the specific method of processing the preprocessed data by the embedding model includes the following steps:

[0015] Step 2.1, dividing the preprocessed data set into blocks to obtain multiple block data;

[0016] Step 2.2: Each block of data is embedded into the Milvus vector database, and the semantic relationship between different data types is captured during embedding.

[0017] Furthermore, the large language model adopts the Tongyi Qianwen series large model.

[0018] Furthermore, the integration method in step 4 is to merge the same contents in the relevant context data and the relevant graph data.

[0019] The beneficial effects of the present invention are:

[0020] First, the method of the present invention can comprehensively process various data in the document, significantly improving the overall analysis ability of the large model for the document. By deeply analyzing multimodal data such as text, tables, and charts in the document, this method makes document parsing more comprehensive and accurate. Secondly, the method combines vector databases and graph databases for efficient storage and retrieval of information. This combination not only improves the system's ability to understand the context, but also significantly improves the accuracy of retrieval. At the same time, the output retrieval results can provide evidence for the results obtained by the user using this method, so that the output results of this method have acceptable accuracy in various vertical fields.

[0021] The present invention is described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the overall flow chart of the present invention;

[0023] Figure 2 It is a flowchart for intelligent analysis of questions and answers;

[0024] Figure 3 It is a flow chart for the conversion of multimodal data;

[0025] Figure 4 Flowchart for multimodal document analysis. DETAILED DESCRIPTION

[0026] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0027] The method of the present invention improves the existing document analysis method by adding the use of visual language models to convert images of tables and charts into text format for further processing and adopting a variant of the BERT (Bidirectional Encoder Representations from Transformers) model as an embedding model. The workflow diagram of the method is shown in the figure below. Figure 1 .

[0028] 1. Preprocess vertical domain documents to extract individual components.

[0029] 2. Tables and charts extracted from documents are processed using a visual language model for comprehensive interpretation and transformation.

[0030] 3. Once all elements (text, tables, and charts) are converted into a structured format, all data is stored in a vector database and a graph database using an embedding model and a large model, respectively.

[0031] 4. After the above stages, enter the RAG pipeline.

[0032] 1. Functional design

[0033] 1. Pretreatment process

[0034] First, we collected a comprehensive dataset of documents in Chinese and English formats in a ratio of 6:4. These documents contain natural language text, tables, and charts. We preprocessed the documents to extract individual components, used OCR to extract text from scanned documents, and used object detection models to identify and extract tables and charts into images in preparation for subsequent processing.

[0035] 2. Intelligent analysis of the question-answering process

[0036] In order to improve the process of intelligent analysis and question answering of vertical documents and to provide interpretable evidence for the intelligent question answering results, the present invention designs a process of first converting and storing and then retrieving and generating. After retrieving relevant context fragments from the vector library, the reordering model is used to reorder these fragments. Finally, the reordered context sequence and the relevant graph data retrieved from the graph database are sent to the large model together to improve the generation accuracy. The idea is as follows Figure 2 .

[0037] Based on this analytical question-answering idea, the detailed processes of the first-change storage-then-retrieval generation process and the test process in the multimodal RAG pipeline are introduced below:

[0038] 1. First store and then retrieve the generation process

[0039] Initially, we used an embedding model to convert document preprocessed data and queries into embedding vectors. These embedding vectors are stored in a vector database, and then combined with a reranking model to re-evaluate and fine-tune the top candidate pairs. The retrieval process is then enhanced using a knowledge graph index, which is stored in a graph database and enhances retrieval by leveraging structured data and entity relationships. For the large model, we use the Tongyi Qianwen series of large models, which are effective in Chinese processing and are known for their powerful natural language understanding and generation capabilities. After integrating these advanced models, we selected our settings to check the accuracy and relevance of complex queries and RAG tasks. We selected some Chinese and English test datasets to test on this RAG pipeline, and evaluated using large models and manual inspections, and finally obtained higher accuracy than the traditional RAG pipeline.

[0040] 2. Testing process

[0041] The purpose of the test process is to verify the accuracy of the test results. First, the unannotated test set documents are input to obtain the test results. Various indicators, including fidelity, response relevance, contextual precision and contextual recall, are used to evaluate the test results, and finally they are manually checked. The results show that this method is an extremely effective method to improve the accuracy and efficiency of document analysis, especially in interpreting tables and charts.

[0042] 3. Post-processing

[0043] The present invention ultimately outputs various indicators for each question and predicted response, including fidelity, response relevance, context precision and context recall, and also outputs data such as the source and related context of the predicted response. The predicted response source and related context serve as interpretable evidence of the prediction results.

[0044] The specific implementation details of conversion, storage, retrieval and generation are introduced in the detailed description. The multimodal RAG idea proposed in this paper has significant advantages over the traditional RAG method using a single modality. The multimodal RAG method can not only comprehensively analyze and summarize various data such as charts, tables and texts in the document, but also significantly improve the overall analysis ability of the document. When processing complex data, this method can output the retrieval source and related context, provide users with a reliable basis, and ensure the credibility of the retrieval results. By integrating different types of data sources, multimodal RAG can provide more comprehensive and accurate analysis results, providing strong support for users in processing and understanding document information.

[0045] 2. Detailed description

[0046] Here are the implementation details of combining multimodal RAG with large models:

[0047] This model uses a visual language model to process documents containing text, tables, and charts into text format, divides the processed text format and the original text data into blocks, and then uses the bge-m3 embedding model to store them in the Milvus vector database through dense embedding and sparse embedding, and uses a large model to extract entities and relationships from the text and store them in the Nebula Graph database. When a user queries, the system uses these embeddings to perform hybrid retrieval, reranks the retrieved results using the bge-reranker-large model, and searches in the graph database at the same time, finally intersecting the results of the vector database with the results of the graph database, and combining the retrieved context with the query in the Qwen-14b large model to generate a response.

[0048] Since documents nowadays mostly use pictures, tables, and text to represent data information, and traditional models are not sufficient for related efficient processing, the basic architecture of multimodal large models has gradually been applied to various vertical industries. We use OCR to extract text from scanned documents, use object detection models to identify and extract tables and charts as images, and then use MiniCPM-Llama3-V-2-5 to process each image and convert them into Markdown format. These converted texts are then embedded into a unified representation together with natural language text using vector embeddings. These vector embeddings capture the semantic relationships between different data types for easy comprehensive analysis. The conversion process is as follows: Figure 3 shown.

[0049] In addition, the model also builds a knowledge graph (graph database) by using the big model to extract relevant entities and their relationships. The knowledge graph not only contains important information in the document, but also reveals the complex associations therein to form a structured knowledge system. After entering the RAG pipeline, the RAG system builds a knowledge base by subdividing the entire document into smaller parts. This segmentation method enables the system to process and store a large amount of document information more efficiently and quickly retrieve relevant content when needed. After entering the RAG pipeline, the RAG system builds a knowledge base by subdividing the entire document into smaller parts. When the user enters a query, by combining vector data library construction and knowledge graph, the system can not only understand and interpret the document more comprehensively, but also provide users with more reliable and detailed analysis results, greatly improving the credibility of the big model. The test results of this method on the document dataset show that this pipeline significantly improves the performance of the model in processing and analyzing complex document information, and the test results are good.

[0050] The complete multimodal document analysis process is as follows:

[0051] Step 1: Input the preprocessed dataset.

[0052] 1.1 Divide the preprocessed dataset into blocks.

[0053] Step 2: Store the data into the database.

[0054] 2.1 Embed each block of data into the Milvus vector database.

[0055] 2.3 The entities and relationships of the preprocessed dataset are stored in the Nebula Graph database through the big model.

[0056] Step 3: Search the database based on the question entered by the user.

[0057] 3.1 The query is also converted into an embedding vector through the embedding model.

[0058] 3.2 Use large models to extract relations and entities in the problem.

[0059] 3.3 Compute the similarity between the query embedding generated in 3.1 and other segments stored in the vector database, and retrieve the top k relevant context segments.

[0060] 3.4 Optimize the results of 3.3 through the bge-reranker-large reranking model.

[0061] 3.5 Use the relationships and entities generated in 3.2 to retrieve related entities and relationships in the graph database;

[0062] Step 4: Input the results of 3.4 and 3.5 into the big model together with the embedding vectors, relations and entities corresponding to the query and the preset prompt template. The big model fact-checks the results from both the vectors from 3.4 and the knowledge graph from 3.5 and merges them to ensure accuracy.

[0063] The above is an example of the best implementation of the present invention, and the parts not described in detail are common knowledge of ordinary technicians in the field. The protection scope of the present invention shall be based on the content of the claims, and any equivalent transformation based on the technical enlightenment of the present invention is also within the protection scope of the present invention.

Claims

1. A vertical industry document intelligent parsing and question-answering method based on multimodal retrieval enhanced generation technology, characterized in that: The following steps are involved: Step 1: Collect a comprehensive document data set in Chinese and English formats, where the document contains natural language text, tables, and chart data, and preprocess the document data in the comprehensive document data set to obtain preprocessed data; Step 2: The preprocessed data is processed by the embedding model and then stored in the vector database. The preprocessed data is extracted by the large language model and the relationships between entities are stored in the graph database. Step 3: After the input question is embedded by the embedding model and the large language model extracts entities and relationships between entities, the query vector and query graph are obtained; Step 4: Input the graph to be queried into the graph database to query and retrieve relevant content to obtain relevant graph data, input the vector to be queried into the vector database to query and retrieve relevant content, re-evaluate and sort the retrieval results of the vector database by the re-ranking model to obtain relevant context data, integrate the relevant context data and the relevant graph data and input them into the large language model; Step 5: The large language model analyzes the data integrated in the previous step according to the query vector and the query graph to obtain the final retrieval result.

2. The vertical industry document intelligent parsing and question-answering method based on multimodal retrieval enhanced generation technology according to claim 1 is characterized in that: In the step 1, the preprocessing method is to preprocess the document to extract individual components, use OCR to extract text from the scanned document to obtain natural language text data, use an object detection model to identify and extract tables and charts in the document as images, and use a visual language model to convert each image into Markdown format.

3. The vertical industry document intelligent parsing and question-answering method based on multimodal retrieval enhanced generation technology according to claim 1 is characterized in that: The specific method of embedding the model to process the preprocessed data includes the following steps: Step 2.1, dividing the preprocessed data set into blocks to obtain multiple block data; Step 2.2: Each block of data is embedded into the Milvus vector database, and the semantic relationship between different data types is captured during embedding.

4. The vertical industry document intelligent parsing and question-answering method based on multimodal retrieval enhanced generation technology according to claim 1 is characterized in that: The large language model adopts the Tongyi Qianwen series large model.

5. The vertical industry document intelligent parsing and question-answering method based on multimodal retrieval enhanced generation technology according to claim 1 is characterized in that: The integration method in step 4 is to merge the same contents in the relevant context data and the relevant graph data.

Images