A visual information extraction method based on a multi-modal large model

Abstract

A kind of visual information extraction method based on multimodal large model, comprising the following steps: (1) using three kinds of special data, i.e.text positioning data, visual reference data and chain thinking data, fine-tuning multimodal large model;(2) user uploads target picture and inputs letter prompt, according to situation triggers template library retrieval and adds context information;(3) multimodal large model generates preliminary reply result and generates final reply result in combination with OCR letter recognition result;(4) dialogue history and meta information will be integrated and saved to template library;The method of the application realizes the accurate output of result text box, completes the error correction of large model output result in cooperation with special OCR model, improves the accuracy and reliability of visual information extraction.

Description

Technical Field

[0001] This invention relates to a visual information extraction method based on a multimodal large model, belonging to the field of information technology, and particularly to the field of visual information extraction technology. Background Technology

[0002] With the rapid development of internet technology and the increasing demand for information exchange, a large number of documents are now digitized, stored, and distributed in image form. In various application scenarios, such as receipt understanding, card recognition, automatic paper scoring, and document matching, it is necessary to extract key information from document images. This process is called Visual Information Extraction (VIE), which involves mining, analyzing, and extracting information from visually rich documents. For example, in receipt understanding, VIE algorithms can extract information such as store names, product details, and prices from receipt images. Unlike extracting information from plain text, the results of VIE are not solely determined by the text content. They are also influenced by other visual factors such as document layout, font style, block colors, charts, and graphics.

[0003] Analyzing and processing visually rich documents is a challenging task. Existing visual information extraction technologies have room for improvement in terms of accuracy and reliability of document OCR, lack ease of use, and suffer from poor generalization ability, high cost, and cumbersome processes.

[0004] How to effectively utilize large-scale model technology, fully explore the potential capabilities of large-scale models, and solve the aforementioned problems in existing visual information extraction is a technical challenge that urgently needs to be addressed in the field of visual information extraction technology. Summary of the Invention

[0005] In view of this, the purpose of this invention is to develop a visual information extraction method based on a multimodal large model, which can achieve technical goals such as high reliability, strong interpretability, strong versatility, and ease of use.

[0006] To achieve the above objectives, this invention proposes a visual information extraction method based on a multimodal large model, the method comprising the following steps:

[0007] (1) Use three types of specialized data, namely text location data, visual reference data and chain thinking data, to fine-tune the multimodal large model to ensure that it can accurately understand the content of document images and extract key information from them;

[0008] (2) After the multimodal model is fine-tuned, the user uploads the target image and enters text prompts to the multimodal model. If the text prompts do not contain the trigger word "answer according to the template library", then proceed directly to step (3). Otherwise, the multimodal model retrieves the most relevant template from the template library and adds the relevant context information to the current dialogue before proceeding to step (3).

[0009] (3) The target image is first subjected to optical character recognition (OCR) to recognize the text of the entire image, thereby obtaining the text content and the coordinates of its location; combined with relevant context information, a complete graphic prompt is formed, and then submitted to the multimodal large model for processing to generate a preliminary response result; the preliminary response result includes not only the text content, but also the coordinate information of the text; the preliminary response result is then combined with the aforementioned OCR text recognition result and after judgment and error correction processing, the final response result is generated.

[0010] (4) When the user enters "make template" in the following dialogue, the multimodal big model will provide a detailed description of the image entered by the user, including the document type, title, main information, etc., and save these descriptions together with the user's question as the meta-information of the template so that accurate retrieval and matching can be performed in step (2); then, the user's dialogue history will be saved; the multimodal big model will ask the user to confirm whether the content of the model's reply is accurate; if the user finds that the reply is incorrect, it can be manually corrected; finally, the manually corrected dialogue history and the meta-information summarized by the model will be integrated and saved in the template library.

[0011] The purpose of the text localization data is to enable the multimodal large model to output accurate text coordinates, thereby facilitating subsequent error correction and improving overall reliability. The text localization data comes from pre-defined datasets including DocVQA-ZH, XFUND-ZH, and pre-defined business image data. After PaddleOCR recognition processing, the text content and its corresponding text box coordinates are obtained. To allow the multimodal large model to adapt to diverse instructions, fixed prompts are not used during fine-tuning; a pre-defined question is presented as "...". Please provide the bounding box coordinates of the region described in this sentence: XXX", while the answer to the aforementioned multimodal large model is " <ref>XXX< / ref> <box> [[x1, y1, x2, y2]]< / box> ”;

[0012] The purpose of the visual reference data is to enhance the multimodal large model's ability to understand bounding boxes, allowing it to focus on text marked with prominently colored boxes. This enables users to guide the multimodal large model simply by selecting or marking the target text region on the image, without providing the target text region coordinates. The visual reference data comes from pre-defined datasets including DocVQA-ZH, XFUND-ZH, and pre-defined business image data. After PaddleOCR identifies the text content and text boxes, OpenCV-Python is used to mark these text boxes with prominent colors in the image. A pre-defined problem is described as " "Please identify the content marked in the prominent color box in the image," there is no fixed prompt; the answer to the aforementioned multimodal large model is " <ref> XXX< / ref> <box> [[x1, y1, x2, y2]]< / box> ”;

[0013] The purpose of the chain-of-thought data is to improve the performance of multimodal large models in complex document scenarios through chain-of-thought reasoning. The chain-of-thought used is the layout chain, which includes three steps: problem analysis, relevant region division, and answer determination. The chain-of-thought data is constructed based on pre-set datasets including DocVQA-ZH, XFUND-ZH, and set business image data, and is formed by using ChatGLM with manual annotation.

[0014] The aforementioned multimodal large model adopts InternVL2, whose architecture is a combination of a visual encoder ViT module, a projection connection layer MLP module, and a large language model LLM module.

[0015] A two-stage fine-tuning strategy is used to fine-tune the aforementioned multimodal large model;

[0016] In the first stage, the ViT module and LLM module are frozen. Using the text location data and visual reference data, only the parameters of the MLP module are fine-tuned and trained to enhance the multimodal large model's understanding of text location and its ability to resolve visual references. This helps ensure that the multimodal large model can accurately locate text elements in the document and understand the marked content.

[0017] In the second stage, while keeping the ViT and LLM modules frozen, the parameters of the small adapter layers attached to the ViT and LLM modules of the multimodal large model are fine-tuned using the chained thinking data and low-rank adaptive adaptation technology. This not only reduces the number of extra parameters, but also effectively avoids the risk of overfitting, while keeping the core functions of the multimodal large model unchanged.

[0018] The template library consists of user dialogue history instances and meta-information summarized by the multimodal large model; each dialogue history instance contains a specific type of document image, user questions for specific key information and model responses, and the meta-information includes descriptions of the document image and questions; the model responses have all been manually corrected to ensure the accuracy of the results;

[0019] When "answer based on template library" is triggered, the multimodal big model retrieves the most relevant template content from the template library as follows: The multimodal big model summarizes the information of the current target image and the question content, mainly involving elements such as document type and title content, and then uses the BM25 algorithm to search and match the most relevant template instance in the meta-information part of the template library.

[0020] The specific content of the multimodal big model adding relevant contextual information to the current dialogue is as follows: after finding a matching instance, the multimodal big model will attach the dialogue history of the matching instance to the context of the current dialogue in order to perform in-context learning.

[0021] In step (3), the specific content of the judgment and error correction process of the preliminary response result is: using the coordinate information in the multimodal large model response result to match the text box obtained by the previous OCR, and selecting the matching result with the largest intersection-union ratio (IoU) value that is greater than the preset threshold.

[0022] The beneficial effects of this invention are as follows: This invention uses a multimodal large model to achieve accurate result text box output, and cooperates with a dedicated OCR model to correct errors in the output results of the large model, thus avoiding the illusion problem of the large model to a certain extent and improving the accuracy and reliability of the results; This invention enables the multimodal large model to understand visual reference marks on images, avoiding the difficulty of model understanding based solely on text descriptions in complex document contexts, and facilitating the model's focus on target areas; This invention adds manually corrected historical dialogues to the current dialogue context, utilizing the small-sample learning characteristics of the large model itself to improve the information extraction capability of unknown document data, eliminating the need to train multiple models for different types of documents, making it efficient, convenient, and cost-effective; This invention fully leverages the thinking chain capability of the multimodal large model, significantly improving the accuracy of model information extraction by introducing intermediate reasoning steps, while also enhancing the interpretability of the results. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a visual information extraction method based on a multimodal large model proposed in this invention.

[0024] Figure 2 This is an example image of the set business image data used in the embodiments of the present invention.

[0025] Figure 3 This is an example diagram of the visual reference data used in the embodiments of the present invention.

[0026] Figure 4 This is a schematic diagram illustrating the two-stage fine-tuning of a large model using three types of specialized data in an embodiment of the present invention.

[0027] Figure 5 This is a schematic diagram of the template library and template library retrieval in an embodiment of the present invention.

[0028] Figure 6 This is a schematic diagram of the judgment and error correction process of the preliminary response result in step (3) of the embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0030] See Figure 1 This invention introduces a visual information extraction method based on a multimodal large model, which includes the following steps:

[0031] (1) Use three types of specialized data, namely text location data, visual reference data and chain thinking data, to fine-tune the multimodal large model to ensure that it can accurately understand the content of document images and extract key information from them;

[0032] (2) After the multimodal model is fine-tuned, the user uploads the target image and enters text prompts to the multimodal model. If the text prompts do not contain the trigger word "answer according to the template library", then proceed directly to step (3). Otherwise, the multimodal model retrieves the most relevant template from the template library and adds the relevant context information to the current dialogue before proceeding to step (3).

[0033] (3) The target image is first subjected to optical character recognition (OCR) to recognize the text of the entire image, thereby obtaining the text content and the coordinates of its location; combined with relevant context information, a complete graphic prompt is formed, and then submitted to the multimodal large model for processing to generate a preliminary response result; the preliminary response result includes not only the text content, but also the coordinate information of the text; the preliminary response result is then combined with the aforementioned OCR text recognition result and after judgment and error correction processing, the final response result is generated.

[0034] (4) When the user enters "make template" in the following dialogue, the multimodal big model will provide a detailed description of the image entered by the user, including the document type, title, main information, etc., and save these descriptions together with the user's question as the meta-information of the template so that accurate retrieval and matching can be performed in step (2); then, the user's dialogue history will be saved; the multimodal big model will ask the user to confirm whether the content of the model's reply is accurate; if the user finds that the reply is incorrect, it can be manually corrected; finally, the manually corrected dialogue history and the meta-information summarized by the model will be integrated and saved in the template library.

[0035] The purpose of the text localization data is to enable the multimodal large model to output accurate text coordinates, thereby facilitating subsequent error correction and improving overall credibility. The text localization data comes from pre-defined datasets including DocVQA-ZH (DocVQA-ZH is a public dataset from an insurance text visual cognition question-answering competition, containing question-answer pairs targeting key information) and XFUND-ZH (XFUND-ZH originates from the Chinese subset of the academic open-source dataset XFUND, containing entity relationship pairs and their coordinate frames, as well as defined business image data. The data in both DocVQA-ZH and XFUND-ZH datasets are Chinese document images rich in visual information, such as cards, invoices, and forms).

[0036] See Figure 2 In this embodiment, the set business image data mainly includes several types of data used in the experiment of this invention, such as settlement slips, nameplates, etc., and the content or coordinate boxes of the information to be extracted are manually marked.

[0037] After being processed by PaddleOCR, the above data yields the text content and its corresponding text box coordinates. For example, the recognition... Figure 2 The partial result obtained is “[[797.0, 36.0], [1150.0, 20.0], [1151.0,59.0], [799.0, 75.0]], ('ZXDU58W121', 0.9939807057380676)”; In order to enable the multimodal large model to adapt to diverse instructions, fixed prompts are not used during fine-tuning, for example, for Figure 2 The data shown includes a pre-defined optional question format: " Please provide the bounding box coordinates of the region described in the sentence: "Outdoor power system", while the answer to the multimodal large model is " <ref> Outdoor power system< / ref> <box> [[346.0, 143.0, 657.0, 175.0 ]]< / box> ”;

[0038] The purpose of the visual reference data is to enhance the multimodal large model's ability to understand bounding boxes, allowing it to focus on text marked with a brightly colored box (in this embodiment, a red box is used). This enables users to guide the multimodal large model simply by selecting or marking the target text region on the image, without providing the coordinates of the target text region. The visual reference data comes from a pre-defined dataset including DocVQA-ZH, XFUND-ZH, and pre-defined business image data. After PaddleOCR identifies the text content and text boxes in these datasets, OpenCV-Python is used to mark these text boxes in the image with a bright color.

[0039] For example, for Figure 3 The data shown includes a pre-defined optional question format: " "Please identify the content marked in the red box in the image," there is no fixed prompt; the answer to the multimodal large model is " <ref> Outdoor power system< / ref> <box> [[346.0, 143.0, 657.0, 175.0]]< / box> ”;

[0040] The purpose of the chain-of-thought data is to improve the performance of multimodal large models in complex document scenarios through chain-of-thought reasoning. The chain-of-thought used is the layout chain, which includes three steps: problem analysis, relevant region division, and answer determination. The chain-of-thought data is constructed based on pre-set datasets including DocVQA-ZH, XFUND-ZH, and set business image data, and is formed by using ChatGLM with manual annotation.

[0041] Taking a table scenario as an example, the question might be "What is the total confirmed amount (excluding tax)?" The model's answer would be: "This is a question about table understanding. The question asks for the value of the field 'Total confirmed amount (excluding tax)'; the relevant area is in the left-hand area of ​​the table in the document, at coordinates [x1, y1, x2, y2]; the answer is in the same column as 'Confirmed Amount (excluding tax)' and in the same row as 'Total,' and the content is..." <ref> XXX< / ref> <box> [[x3, y3, x4, y4]]< / box> ".

[0042] See Figure 4 The multimodal large model mentioned above adopts InternVL2, whose architecture is a combination of visual encoder ViT module + projection connection layer MLP module + large language model LLM module;

[0043] A two-stage fine-tuning strategy is used to fine-tune the aforementioned multimodal large model;

[0044] In the first stage, the ViT module and LLM module are frozen. Using the text location data and visual reference data, only the parameters of the MLP module are fine-tuned and trained to enhance the multimodal large model's understanding of text location and its ability to resolve visual references. This helps ensure that the multimodal large model can accurately locate text elements in the document and understand the marked content.

[0045] In the second stage, while keeping the ViT and LLM modules frozen, the parameters of the small adapter layers attached to the ViT and LLM modules of the multimodal large model are fine-tuned using the chained thinking data and low-rank adaptive adaptation technology. This not only reduces the number of extra parameters, but also effectively avoids the risk of overfitting, while keeping the core functions of the multimodal large model unchanged.

[0046] Low-rank adaptation is a lightweight adaptive mechanism that fine-tunes parameters by adding small adapters to existing modules. This approach not only reduces the number of extra parameters but also effectively avoids the risk of overfitting while maintaining the core functionality of the model.

[0047] See Figure 5 The template library consists of user dialogue history instances and meta-information summarized by the multimodal big model; each dialogue history instance contains a specific type of document image, user questions for specific key information and model responses, and the meta-information includes descriptions of the document image and questions; the model responses have all been manually corrected to ensure the accuracy of the results;

[0048] See Figure 5 When the "answer based on template library" is triggered, the multimodal big model retrieves the most relevant template content from the template library as follows: The multimodal big model summarizes the information of the current target image and the question content, mainly involving elements such as document type and title content, and then uses the BM25 algorithm (Best Match 25) to search and match the most relevant template instance in the meta-information part of the template library.

[0049] The specific content of the multimodal big model adding relevant contextual information to the current dialogue is as follows: after finding a matching instance, the multimodal big model will attach the dialogue history of the matching instance to the context of the current dialogue in order to perform in-context learning.

[0050] See Figure 6 In step (3), the specific content of the judgment and error correction process for the preliminary response result is as follows: the multimodal large model response result is "green electricity consumption certificate". Its coordinate information is matched with the text box obtained by the previous OCR. The matching result with the largest Intersection over Union (IoU) value and greater than the preset threshold is selected. The first item of the OCR result, "Certificate No.", has an IoU of 0 with the large model result, while the second item, "green electricity consumption certificate", has the highest IoU. Therefore, the latter is selected. After obtaining the matching result, the text content in the OCR result is used to replace the large model response result to complete the error correction.

[0051] The inventors conducted extensive experiments on the method proposed in this invention. Various types of document images were selected as test samples, including settlement slips, invoices, and nameplate images, to test the adaptability of the method in different application scenarios. The experiments were divided into two main parts, depending on whether a template library was used.

[0052] The first part focuses on testing the performance improvements of fine-tuning training on multimodal large models. In this phase, experiments were conducted using single images to ask questions. Results showed that the fine-tuned model significantly improved its ability to output text bounding boxes and greatly enhanced its fine-grained text understanding capabilities. Notably, the model can also understand human-marked visual cues, performing particularly well when handling complex documents. By incorporating chained thinking data, the fine-tuned model performs detailed step-by-step analysis during output, resulting in a significant improvement in response accuracy compared to the original model.

[0053] The second part further tests the practical effectiveness of the entire method. In this process, multiple rounds of question-and-answer sessions were conducted beforehand to expand the template library. Experimental results show that the multimodal large model is highly effective in learning from few samples of context, quickly correcting erroneous responses with only a few samples. Compared to models without contextual learning support, this context-learning approach significantly improves the accuracy of the response results. Furthermore, in practical applications, text recognition errors in the multimodal large model may occur, leading to biases in the initial response results. However, by combining OCR recognition results with the initial response results from the multimodal large model, the method can effectively identify and correct these errors. This mechanism demonstrated good performance in the experiments, identifying and correcting text errors to ensure the accuracy of the final response results.

[0054] The above experimental results prove that the method proposed in this invention is effective.

Claims

1. A method for visual information extraction based on a multimodal large model, characterized in that: The method includes the following steps: (1) Use three types of specialized data, namely text location data, visual reference data and chain thinking data, to fine-tune the multimodal large model to ensure that it can accurately understand the content of document images and extract key information from them; (2) After the multimodal model is fine-tuned, the user uploads the target image and enters text prompts to the multimodal model. If the text prompts do not contain the trigger word "answer according to the template library", then proceed directly to step (3). Otherwise, the multimodal model retrieves the most relevant template from the template library and adds the relevant context information to the current dialogue before proceeding to step (3). (3) The target image is first subjected to optical character recognition (OCR) to recognize the text of the entire image, thereby obtaining the text content and the coordinates of its location; combined with relevant context information, a complete graphic prompt is formed, and then submitted to the multimodal large model for processing to generate a preliminary response result; the preliminary response result includes not only the text content, but also the coordinate information of the text. The preliminary response result, combined with the aforementioned OCR text recognition result, undergoes error correction processing to generate the final response result; (4) When the user enters "make template" in the following dialogue, the multimodal big model will provide a detailed description of the image entered by the user, including the document type, title and main information, and save these descriptions together with the user's question as the meta-information of the template so that accurate retrieval and matching can be performed in step (2); then, the user's dialogue history will be saved; the multimodal big model will ask the user to confirm whether the content of the model's reply is accurate; if the user finds that the reply is incorrect, it will be manually corrected; finally, the manually corrected dialogue history and the meta-information summarized by the model will be integrated and saved in the template library.

2. The visual information extraction method based on a multimodal large model according to claim 1, characterized in that: The purpose of the text localization data is to enable the multimodal large model to output accurate text coordinates, thereby facilitating subsequent error correction and improving overall reliability. The text localization data comes from pre-defined datasets including DocVQA-ZH, XFUND-ZH, and pre-defined business image data. These datasets are processed by PaddleOCR to obtain the text content and its corresponding text box coordinates. To allow the multimodal large model to adapt to diverse instructions, fixed prompts are not used during fine-tuning; a pre-defined question is presented as "...". Please provide the bounding box coordinates of the region described in this sentence: XXX”, while the answer to the question about the multimodal large model is “ <ref> XXX< / ref> <box> [[x1, y1, x2, y2]]< / box> ”; The purpose of the visual reference data is to enhance the multimodal large model's ability to understand bounding boxes, allowing it to focus on text marked with prominently colored boxes. This enables users to guide the multimodal large model simply by selecting or marking the target text region on the image, without providing the target text region's coordinates. The visual reference data comes from pre-defined datasets including DocVQA-ZH, XFUND-ZH, and pre-defined business image data. After PaddleOCR identifies the text content and text boxes in these datasets, OpenCV-Python is used to mark these text boxes with prominent colors in the image. A pre-defined problem is described as " "Please identify the content marked in the prominent color box in the image," there is no fixed prompt; the answer to the question about the multimodal large model is " <ref> XXX< / ref> <box> [[x1, y1, x2, y2]]< / box> ”; The purpose of the chain-of-thought data is to improve the performance of multimodal large models in complex document scenarios through chain-of-thought reasoning. The chain-of-thought used is the layout chain, which includes three steps: problem analysis, relevant region division, and answer determination. The chain-of-thought data is constructed based on pre-set datasets including DocVQA-ZH, XFUND-ZH, and set business image data, and is formed by using ChatGLM with manual annotation.

3. The visual information extraction method based on a multimodal large model according to claim 1, characterized in that: The multimodal large model mentioned above adopts InternVL2, and its architecture is a combination of a visual encoder ViT module, a projection connection layer MLP module, and a large language model LLM module. A two-stage fine-tuning strategy is used to fine-tune the aforementioned multimodal large model; In the first stage, the ViT module and LLM module are frozen. Using the text location data and visual reference data, only the parameters of the MLP module are fine-tuned and trained to enhance the multimodal large model's understanding of text location and its ability to resolve visual references. This helps ensure that the multimodal large model can accurately locate text elements in the document and understand the marked content. In the second stage, while keeping the ViT and LLM modules frozen, the parameters of the small adapter layers attached to the ViT and LLM modules of the multimodal large model are fine-tuned using the chained thinking data and low-rank adaptive adaptation technology. This not only reduces the number of extra parameters, but also effectively avoids the risk of overfitting, while keeping the core functions of the multimodal large model unchanged.

4. The visual information extraction method based on a multimodal large model according to claim 1, characterized in that: The template library consists of user dialogue history instances and meta-information summarized by the multimodal large model; each dialogue history instance contains a specific type of document image, user questions for specific key information and model responses, and the meta-information includes descriptions of the document image and questions; the model responses have all been manually corrected to ensure the accuracy of the results; When "answer based on template library" is triggered, the multimodal big model retrieves the most relevant template content from the template library as follows: The multimodal big model summarizes the information of the current target image and the question content, involving document type, title and content, and then uses the BM25 algorithm to search and match the most relevant template instance in the meta-information part of the template library; The specific content of the multimodal big model adding relevant contextual information to the current dialogue is as follows: after finding a matching instance, the multimodal big model will attach the dialogue history of the matching instance to the context of the current dialogue in order to perform in-context learning.

5. The visual information extraction method based on a multimodal large model according to claim 1, characterized in that: In step (3), the specific content of the judgment and error correction process of the preliminary response result is: using the coordinate information in the multimodal large model response result to match the text box obtained by the previous OCR, and selecting the matching result with the largest intersection-union ratio (IoU) value that is greater than the preset threshold.

Images